Constraints on the list of reference images for sub-images

Abstract

Methods and apparatuses for processing video are described. The processing may include video encoding, decoding, or transcoding. An example video processing method includes performing a conversion between a video including a current picture that includes a current sub-picture and a bitstream of the video, the current sub-picture including a current strip, wherein the bitstream conforms to format rules that, in response to a condition associated with at least a first picture and the current sub-picture, do not allow an active entry in a reference picture list of the current strip to include the first picture that is before a second picture in decoding order, the second picture including a progressive temporal sub-layer access sub-picture.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority and benefit of U.S. Provisional Patent Application No. 62 / 992,724, filed on Mar. 20, 2020, and International Patent Application No. PCT / US2021 / 022990, filed on Mar. 18, 2021. All of the above - mentioned patent applications are hereby incorporated by reference in their entirety. Technical Field

[0003] This patent document relates to image and video encoding, decoding, and transcoding. Background Art

[0004] In the Internet and other digital communication networks, digital video occupies the largest bandwidth. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth demand for digital video use is expected to continue to grow. Summary of the Invention

[0005] This document discloses techniques that can be used by video encoders and decoders to process encoded and decoded representations of video using various syntax rules.

[0006] In one example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including one or more sub - pictures and a bitstream of the video, where the bitstream conforms to format rules that define the syntax of network abstraction layer (NAL) units in a specified bitstream, and where the format rules specify that NAL units of video coding layer (VCL) NAL unit types include content associated with a particular type of picture or a particular type of sub - picture.

[0007] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture including sub - pictures and a bitstream of the video, where the bitstream conforms to format rules, and where the format rules specify that a sub - picture is a random - access - type sub - picture in response to the sub - picture being a pre - sub - picture of an intra - random - access - point sub - picture.

[0008] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture including sub - pictures and a bitstream of the video, where the bitstream conforms to format rules, and where the format rules specify that one or more random - access - skip pre - sub - pictures are not present in the bitstream in response to the one or more random - access - skip pre - sub - pictures being associated with an instantly - decoded - refresh sub - picture.

[0009] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture having sub-pictures and a bitstream of the video, where the bitstream conforms to a format rule that specifies that in response to one or more random access decodable pre-sub-pictures being associated with an instantly decodable refresh sub-picture having a type of network abstraction layer (NAL) unit indicating that the instantly decodable refresh sub-picture is not associated with a pre-picture, one or more random access decodable pre-sub-pictures do not exist in the bitstream.

[0010] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture having two adjacent sub-pictures and a bitstream of the video, where the bitstream conforms to a format rule that specifies that two adjacent sub-pictures having different types of network abstraction layer (NAL) units have a syntax element with the same first value, and the syntax element indicates whether each of the two adjacent sub-pictures in a coded layer video sequence is considered a picture.

[0011] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture having two adjacent sub-pictures and a bitstream of the video, where the format rule specifies that the two adjacent sub-pictures include a first adjacent sub-picture having a first sub-picture index and a second adjacent sub-picture having a second sub-picture index, and where the format rule specifies that in response to a first syntax element associated with the first sub-picture index indicating that the first adjacent sub-picture is not considered a picture or a second syntax element associated with the second sub-picture index indicating that the second adjacent sub-picture is not considered a picture, the two adjacent sub-pictures have the same type of network abstraction layer (NAL) unit.

[0012] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a picture having one or more sub-pictures and a bitstream of the video, where the bitstream conforms to a format rule that specifies that in response to a syntax element indicating that each picture of a video with a reference picture parameter set (PPS) has multiple video coding layer (VCL) network abstraction layer (NAL) units that do not have the same type of VCL NAL unit, a picture is allowed to include more than two different types of VCL NAL units.

[0013] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures having one or more sub-pictures and a bitstream of the video, where the bitstream conforms to a format rule that specifies that a post-sub-picture sequentially associated with an intra random access point sub-picture or a progressive decoding refresh sub-picture is after the intra random access point sub-picture or the progressive decoding refresh sub-picture.

[0014] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that, in response to: (1) a second-order sub-picture being before an intra-random access point sub-picture, (2) the sub-picture and the intra-random access point sub-picture having the same first value for a layer to which a network abstraction layer (NAL) unit of the sub-picture and the intra-random access point sub-picture belongs, and (3) the sub-picture and the intra-random access point sub-picture having the same second value of a sub-picture index, the sub-picture is before the intra-random access point sub-picture and one or more random access decodable pre-sub-pictures associated with the intra-random access point sub-picture in a first order.

[0015] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that a random access skip pre-sub-picture associated with a clean random access sub-picture is before one or more random access decodable pre-sub-pictures associated with the clean random access sub-picture in an order.

[0016] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that a random access skip pre-sub-picture associated with a clean random access sub-picture is after one or more intra-random access point sub-pictures before the clean random access sub-picture in a second order after the one or more intra-random access point sub-pictures before the clean random access sub-picture in a first order.

[0017] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that, in response to: (1) a syntax element indicating that a coded layer video sequence conveys a picture representing a frame, and (2) a current sub-picture being a pre-sub-picture associated with an intra-random access point sub-picture, the current sub-picture is before one or more non-pre-sub-pictures associated with the intra-random access point sub-picture in a decoding order.

[0018] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that, in response to a picture being a pre-picture of an intra random access point picture, one or more types of network abstraction layer (NAL) units of all video coding layer (VCL) NAL units in the picture include RADL_NUT or RASL_NUT.

[0019] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each including a plurality of sub-pictures and a bitstream of the video, wherein the bitstream conforms to a format rule that specifies that, in response to: (1) at least one sub-picture being before a progressive decoding refresh sub-picture in a second order, (2) the at least one sub-picture and the progressive decoding refresh sub-picture having the same first value for a layer to which network abstraction layer (NAL) units of the at least one sub-picture and the progressive decoding refresh sub-picture belong, and (3) the at least one sub-picture and the progressive decoding refresh picture having the same second value of a sub-picture index, the at least one sub-picture is before the progressive decoding refresh sub-picture and one or more sub-pictures associated with the progressive decoding refresh sub-picture in a first order.

[0020] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture including a current sub-picture, the current sub-picture including a current strip, and a bitstream of the video, wherein the bitstream conforms to a format rule that, in response to: (a) a first picture having the same temporal identifier and the same layer identifier of network abstraction layer (NAL) units as the current sub-picture, and (b) the current sub-picture being after a progressive temporal sub-layer access sub-picture in decoding order, and (c) the current sub-picture and the progressive temporal sub-layer access sub-picture having the same temporal identifier, the same layer identifier, and the same sub-picture index, the format rule does not allow an active entry in a reference picture list of the current strip to include the first picture that is before a second picture in decoding order, the second picture including the progressive temporal sub-layer access sub-picture.

[0021] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture including a current sub-picture, the current sub-picture including a current strip, and a bitstream of the video, wherein the bitstream conforms to a format rule that, in response to the current sub-picture not being a specific type of sub-picture, the format rule does not allow an active entry in a reference picture list of the current strip to include a first picture generated by a decoding process that generates an unavailable reference picture.

[0022] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture that includes a current sub-picture and a bitstream of the video, the sub-picture including a current slice, wherein the bitstream conforms to format rules, and in response to the current sub-picture not being a particular type of sub-picture, the format rules do not allow an entry in the reference picture list of the current slice to include a first picture generated by a decoding process that generates an unavailable reference picture.

[0023] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture that includes a current sub-picture and a bitstream of the video, the current sub-picture including a current slice, wherein the bitstream conforms to format rules, and in response to (a) a first picture including a previous intra-random access point sub-picture, the previous intra-random access point sub-picture being before the current sub-picture in a second order, (b) the previous intra-random access point sub-picture having the same layer identifier of a network abstraction layer (NAL) unit and the same sub-picture index as the current sub-picture, and (c) the current sub-picture being a pure random access sub-picture, the format rules do not allow an entry in the reference picture list of the current slice to include the first picture that is before the current picture in a first order or a second order.

[0024] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture that includes a current sub-picture and a bitstream of the video, the current sub-picture including a current slice, wherein the bitstream conforms to format rules, and in response to (a) the current sub-picture being associated with an intra-random access point sub-picture, and (b) the current sub-picture being after the intra-random access point sub-picture in a first order, the format rules do not allow an active entry in the reference picture list of the current slice to include the first picture that is before the current picture in a first order or a second order.

[0025] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture that includes a current sub-picture and a bitstream of the video, the current sub-picture including a current slice, wherein the bitstream conforms to format rules, and in response to: (a) the current sub-picture being after an intra-random access point sub-picture in a first order, and (b) the current sub-picture being after one or more preceding sub-pictures associated with an IRAP sub-picture in a first order and a second order, the format rules do not allow an entry in the reference picture list of the current slice to include the first picture that is before the current picture including the intra-random access point sub-picture associated with the current sub-picture in a first order or a second order.

[0026] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including a current picture containing a current sub-picture, the current sub-picture including a current strip, and a bitstream of the video, wherein the bitstream conforms to format rules that specify that in response to the current sub-picture being a random access decodable prefix sub-picture, the reference picture list of the current strip does not include active entries of any one or more of the following: a first picture including a random access skipped prefix sub-picture and a second picture that is before a third picture including an intra random access point sub-picture in decoding order.

[0027] In another example aspect, a video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each containing one or more sub-pictures and a codec representation of the video. The codec representation conforms to format rules that specify that one or more pictures including one or more sub-pictures are included in the codec representation according to network abstraction layer (NAL) units, wherein a type of NAL unit indicated in the codec representation includes a coded strip of a particular type of picture or a coded strip of a particular type of sub-picture.

[0028] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each containing one or more sub-pictures and a codec representation of the video, wherein the codec representation conforms to format rules that specify that two adjacent sub-pictures having different network abstraction layer unit types will have the same indication of the sub-picture regarded as a picture flag.

[0029] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each containing one or more sub-pictures and a codec representation of the video, wherein the codec representation conforms to format rules that define an order of a first type of sub-picture and a second type of sub-picture, wherein the first sub-picture is a postfix sub-picture or a prefix sub-picture or a random access skipped prefix (RASL) sub-picture type, and the second sub-picture is a RASL type or a random access decodable prefix (RADL) type or an instant decoding refresh (IDR) type or a gradual decoding refresh (GDR) type sub-picture.

[0030] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a video including one or more pictures each containing one or more sub-pictures and a codec representation of the video, wherein the codec representation conforms to format rules that define conditions for allowing or not allowing a first type of sub-picture to coexist with a second type of sub-picture.

[0031] In yet another exemplary aspect, a video encoder device is disclosed. The video encoder includes a processor configured to implement the above-described method.

[0032] In yet another exemplary aspect, a video decoder device is disclosed. The video decoder includes a processor configured to implement the above-described method.

[0033] In yet another exemplary aspect, a computer-readable medium having code stored thereon is disclosed. The code embodies one of the methods described herein in the form of processor-executable code.

[0034] These and other features will be described throughout this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 An example of raster scan strip segmentation of a picture is shown, where the picture is divided into 12 slices and 3 raster scan strips.

[0036] Figure 2 An example of rectangular strip segmentation of a picture is shown, where the picture is divided into 24 slices (6 slice columns and 4 slice rows) and 9 rectangular strips.

[0037] Figure 3 An example of a picture divided into slices and rectangular strips is shown, where the picture is divided into 4 slices (2 slice columns and 2 slice rows) and 4 rectangular strips.

[0038] Figure 4 A picture divided into 15 slices, 24 strips, and 24 sub-pictures is shown.

[0039] Figure 5 is a block diagram of an example video processing system.

[0040] Figure 6 is a block diagram of a video processing device.

[0041] Figure 7 is a flowchart of an example method of video processing.

[0042] Figure 8 is a block diagram showing a video coding / decoding system according to some embodiments of the present disclosure.

[0043] Figure 9 is a block diagram showing an encoder according to some embodiments of the present disclosure.

[0044] Figure 10 is a block diagram showing a decoder according to some embodiments of the present disclosure.

[0045] Figures 11 to 31 is a flowchart of an example method of video processing. DETAILED DESCRIPTION

[0046] The use of chapter headings in this document is for ease of understanding and does not limit the applicability of the technologies and embodiments disclosed in each chapter to that chapter. Additionally, the use of H.266 terminology in some descriptions is merely for ease of understanding and does not limit the scope of the disclosed technologies. Therefore, the technologies described herein are also applicable to other video codec protocols and designs. In this document, editorial changes to the text are shown relative to the current draft of the VVC specification by means of left and right square brackets (e.g., [[]]), where the deleted text within the square brackets represents the stricken text, and bold italic represents the added text.

[0047] 1. Initial Discussion

[0048] This document is related to video coding and decoding technologies. Specifically, it is about the definition of sub - picture types and the relationships in terms of decoding order, output order, and prediction relationships between different types of sub - pictures in single - layer and multi - layer contexts. The key is to clearly specify the meaning of the hybrid sub - picture types within a picture through a set of constraints on the decoding order, output order, and prediction relationships. These ideas can be applied, either alone or in various combinations, to any video coding standard or non - standard video codec that supports multi - layer video coding and decoding, such as the Versatile Video Coding (VVC) being developed.

[0049] 2. Abbreviations

[0050] APS (Adaptation Parameter Set) Adaptive Parameter Set

[0051] AU (Access Unit) Access Unit

[0052] AUD (Access Unit Delimiter) Access Unit Delimiter

[0053] AVC (Advanced Video Coding) Advanced Video Coding

[0054] CLVS (Coded Layer Video Sequence) Coded Layer Video Sequence

[0055] CPB (Coded Picture Buffer) Coded Picture Buffer

[0056] CRA (Clean Random Access) Clean Random Access

[0057] CTU (Coding Tree Unit) Coding Tree Unit

[0058] CVS (Coded Video Sequence) Coded Video Sequence

[0059] DCI (Decoding Capability Information) Decoding Capability Information

[0060] DPB (Decoded Picture Buffer) Decoded Picture Buffer

[0061] EOB (End Of Bitstream) End Of Bitstream

[0062] EOS (End Of Sequence) End Of Sequence

[0063] GDR (Gradual Decoding Refresh) Gradual Decoding Refresh

[0064] HEVC (High Efficiency Video Coding) High Efficiency Video Coding

[0065] HRD (Hypothetical Reference Decoder) Hypothetical Reference Decoder

[0066] IDR (Instantaneous Decoding Refresh) Instantaneous Decoding Refresh

[0067] JEM (Joint Exploration Model) Joint Exploration Model

[0068] MCTS (Motion-Constrained Tile Sets) Motion-Constrained Tile Sets

[0069] NAL (Network Abstraction Layer) Network Abstraction Layer

[0070] OLS (Output Layer Set) Output Layer Set

[0071] PH (Picture Header) Picture Header

[0072] PPS (Picture Parameter Set) Picture Parameter Set

[0073] PTL (Profile, Tier and Level) Profile, Tier and Level

[0074] PU (Picture Unit) Picture Unit

[0075] RADL (Random Access Decodable Leading (Picture)) Random Access Decodable Leading (Picture)

[0076] RAP (Random Access Point) Random Access Point

[0077] RASL (Random Access Skipped Leading (Picture)) Random Access Skipped Leading (Picture)

[0078] RBSP (Raw Byte Sequence Payload) Raw Byte Sequence Payload

[0079] RPL (Reference Picture List) Reference Picture List

[0080] SEI (Supplemental Enhancement Information) Supplemental Enhancement Information

[0081] SPS (Sequence Parameter Set) Sequence Parameter Set

[0082] STSA (Step - wise Temporal Sublayer Access) Step - wise Temporal Sublayer Access

[0083] SVC (Scalable Video Coding) Scalable Video Coding

[0084] VCL (Video Coding Layer) Video Coding Layer

[0085] VPS (Video Parameter Set) Video Parameter Set

[0086] VTM (VVC Test Model) VVC Test Model

[0087] VUI (Video Usability Information) Video Usability Information

[0088] VVC (Versatile Video Coding) Versatile Video Coding

[0089] 3. Introduction to Video Coding

[0090] Video coding standards have evolved mainly through the development of well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, ISO / IEC developed MPEG-1 and MPEG-4 Visual standards, and the two organizations jointly developed H.262 / MPEG-2 video and H.264 / MPEG-4 Advanced Video Coding (AVC) and H.265 / HEVC standards. Since H.262, video coding standards have been based on a hybrid video coding structure, in which temporal prediction plus transform coding is utilized. To explore future video coding technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and applied them to a reference software called the Joint Exploration Model (JEM). At the same time, JVET meetings are held quarterly, and the goal of the new coding standard is to reduce the bit rate by 50% compared to HEVC. The new video coding standard was officially named Versatile Video Coding (VVC) at the JVET meeting in April 2018, and the first version of the VVC Test Model (VTM) was also released at that time. Due to the continuous efforts in VVC standardization, new coding technologies have been adopted into the VVC standard at each JVET meeting. Then, the working draft of VVC and the test model VTM are updated after each meeting. The VVC project now aims to be technically completed (FDIS) at the meeting in July 2020.

[0091] 3.1. Picture segmentation schemes in HEVC

[0092] HEVC includes four different picture segmentation schemes, namely regular slices, dependent slices, tiles, and Wavefront Parallel Processing (WPP), which can be applied to Maximum Transfer Unit (MTU) size matching, parallel processing, and reduced end-to-end latency.

[0093] Regular slices are similar to those in H.264 / AVC. Each regular slice is encapsulated in its own NAL unit, and picture prediction (intra-sample prediction, motion information prediction, coding mode prediction) and entropy coding dependencies across slice boundaries are disabled. Therefore, a regular slice can be reconstructed independently of other regular slices within the same picture (although there may still be interdependencies due to loop filter operations).

[0094] Regular slices are the only tool that can be used for parallelization and are also available in H.264 / AVC in a nearly identical form. Parallelization based on regular slices does not require much inter-processor or inter-core communication (except for the inter-processor or inter-core data sharing for motion compensation when decoding predicted coded pictures, which is usually much heavier than inter-processor or inter-core data sharing due to intra-picture prediction). However, for the same reason, the use of regular slices incurs a large amount of coding and decoding overhead due to the bit cost of the slice headers and the lack of prediction across slice boundaries. In addition, due to the intra-picture independence of regular slices and each regular slice being encapsulated in its own NAL unit, regular slices (compared to other tools mentioned below) also serve as a key mechanism for bitstream segmentation to match the MTU size requirements. In many cases, the goals of parallelization and MTU size matching pose conflicting requirements for the slice layout in a picture. The recognition of this situation led to the development of the parallelization tools mentioned below.

[0095] Dependent slices have short slice headers and allow the bitstream to be segmented at tree-block boundaries without breaking any intra-picture prediction. Basically, dependent slices provide fragmentation of regular slices into multiple NAL units to provide reduced end-to-end latency by allowing a portion of a regular slice to be sent out before the encoding of the entire regular slice is completed.

[0096] In WPP, a picture is segmented into single-row coded tree blocks (CTBs). Entropy decoding and prediction are allowed to use data from CTBs in other segments. Through parallel decoding of CTB rows, parallel processing is possible, where the start of decoding of a CTB row is delayed by two CTBs to ensure that data related to CTBs above and to the right of the main CTB can be obtained before the main CTB being decoded. Using this staggered start (which looks like a wavefront when graphically represented), as many processors / cores can be parallelized as there are CTB rows in the picture. Since intra-picture prediction between adjacent tree-block rows within a picture is permitted, the inter-processor / inter-core communication required to enable intra-picture prediction can be substantial. Compared to when not applied, WPP segmentation does not result in the generation of additional NAL units, so WPP is not a tool for MTU size matching. However, if MTU size matching is required, regular slices can be used together with WPP with a certain degree of coding and decoding overhead.

[0097] Slices define the horizontal and vertical boundaries that divide a picture into slice columns and slice rows. Slice columns extend from the top to the bottom of the picture. Similarly, slice rows extend from the left to the right of the picture. The number of slices in a picture can be simply derived by multiplying the number of slice columns by the number of slice rows.

[0098] Before decoding the top-left CTB of the next slice in the raster scan order of the picture's slices, the scan order of the CTBs is changed to be local within the slice (in the raster scan order of the slice's CTBs). Similar to conventional strips, slices break the prediction dependency and entropy decoding dependency within the picture. However, they do not need to be included in separate NAL units (the same as WPP in this regard); thus, slices cannot be used for MTU size matching. Each slice can be processed by one processor / core, and in the case where a strip spans multiple slices, the inter-processor / inter-core communication required for intra-picture prediction between the processing units decoding adjacent slices is limited to transmitting the shared strip header and loop filtering related to the sharing of reconstructed samples and metadata. When a strip contains more than one slice or WPP segment, the byte offset of the entry point of each slice or WPP segment except the first one in the strip is signaled in the strip header.

[0099] For simplicity, HEVC has specified restrictions on the application of four different picture partitioning schemes. A given coded video sequence cannot include both slices and wavefronts for most profiles specified in HEVC. For each strip and slice, one or both of the following conditions must be met: 1) all coded tree blocks in the strip belong to the same slice; 2) all coded tree blocks in the slice belong to the same strip. Finally, a wavefront segment exactly contains one CTB row, and when WPP is being used, if a strip starts within a CTB row, it must end within the same CTB row.

[0100] The latest revision to HEVC is specified in the JCT-VC output file JCTVC-AC1005, J. Boyce, A. Ramasubramonian, R. Skupin, G. J. Sullivan, A. Tourapis, Y.-K. Wang (editors), "HEVC Additional Supplemental Enhancement Information (Draft 4)", publicly released on October 24, 2017:

[0101] http: / / phenix.int-evry.fr / jct / doc_end_user / documents / 29_Macau / wg11 / JCTVC-AC1005-v2.zip. By including this amendment, HEVC specifies three MCTS-related SEI messages, namely the temporal MCTS SEI message, the MCTS extraction information set SEI message, and the MCTS extraction information nested SEI message.

[0102] The time-domain MCTS SEI message indicates the presence of MCTS in the bitstream and signals the MCTS. For each MCTS, the motion vectors are restricted to point to full-sample positions within the MCTS and fractional-sample positions of full-sample positions within the MCTS that are only required for interpolation, and the use of motion vector candidates for time-domain motion vector prediction derived from blocks outside the MCTS is not allowed. In this way, each MCTS can be independently decoded in the absence of slices not included in the MCTS.

[0103] The MCTS extraction information set SEI message provides supplementary information (specified as part of the semantics of the SEI message) that can be used in MCTS sub-bitstream extraction to generate a compliant bitstream for an MCTS group. This information consists of a number of extraction information sets, each defining a number of MCTS groups and containing the RBSP bytes that will replace the VPS, SPS, and PPS during the MCTS sub-bitstream extraction process. When extracting the sub-bitstream according to the MCTS sub-bitstream extraction process, the parameter sets (VPS, SPS, and PPS) need to be rewritten or replaced, and the slice header needs to be slightly updated because one or all of the syntax elements related to the slice address (including first_slice_segment_in_pic_flag and slice_segment_address) usually need to have different values.

[0104] 3.2. Picture segmentation in VVC

[0105] In VVC, a picture is divided into one or more slice rows and one or more slice columns. A slice is a sequence of CTUs that cover a rectangular region of the picture. The CTUs within a slice are scanned in raster scan order.

[0106] A strip consists of an integer number of complete slices or an integer number of consecutive complete CTU rows within a slice of the picture.

[0107] Two strip modes are supported, namely the raster scan strip mode and the rectangular strip mode. In the raster scan strip mode, a strip contains the complete sequence of slices in the slice raster scan of the picture. In the rectangular strip mode, a strip contains a number of complete slices that together form a rectangular region of the picture, or a number of consecutive complete CTU rows of a single slice that together form a rectangular region of the picture. The slices within a rectangular strip are scanned in slice raster scan order within the rectangular region corresponding to the strip.

[0108] A sub-picture contains one or more strips that together cover a rectangular region of the picture.

[0109] Figure 1 An example of the raster scan strip segmentation of a picture is shown, where the picture is divided into 12 slices and 3 raster scan strips.

[0110] Figure 2 An example of rectangular strip segmentation of a picture is shown, where the picture is divided into 24 slices (6 slice columns and 4 slice rows) and 9 rectangular strips.

[0111] Figure 3 An example of a picture segmented into slices and rectangular strips is shown, where the picture is divided into 4 slices (2 slice columns and 2 slice rows) and 4 rectangular strips.

[0112] Figure 4 An example of sub - picture segmentation of a picture is shown, where the picture is segmented into 18 slices, 12 slices on the left (each covering a strip with 4x4 CTUs) and 6 slices on the right (each covering 2 vertically stacked strips with 2x2 CTUs), resulting in a total of 24 strips and 24 sub - pictures of different dimensions (each strip is a sub - picture).

[0113] 3.3. Change of Picture Resolution within a Sequence

[0114] In AVC and HEVC, unless a new sequence using a new SPS starts with an IRAP picture, the spatial resolution of a picture cannot be changed. VVC enables changing the picture resolution within a sequence at positions where IRAP pictures are not encoded, and IRAP pictures are always intra - coded and decoded. This feature is sometimes referred to as reference picture resampling (RPR), because when the reference picture has a different resolution from the current picture being decoded, this feature requires resampling of the reference pictures used for inter - prediction.

[0115] The scaling ratio is restricted to be greater than or equal to 1 / 2 (2 - fold downsampling from the reference picture to the current picture) and less than or equal to 8 (8 - fold upsampling). Three sets of resampling filters with different frequency cut - offs are specified to handle various scaling ratios between the reference picture and the current picture. The three sets of resampling filters are applied to scaling ratios in the ranges from 1 / 2 to 1 / 1.75, from 1 / 1.75 to 1 / 1.25, and from 1 / 1.25 to 8 respectively. Each set of resampling filters has 16 phases for luminance and 32 phases for chrominance, the same as in the case of motion - compensated interpolation filters. In fact, the normal MC interpolation process is a special case of the resampling process with a scaling ratio range from 1 / 1.25 to 8. The horizontal and vertical scaling ratios are derived based on the picture width and height and the left, right, up, and down scaling offsets specified for the reference picture and the current picture.

[0116] Other aspects of the VVC design that support features different from HEVC include: i) The picture resolution and the corresponding consistency window are signaled in the PPS instead of the SPS, while the maximum picture resolution is signaled in the SPS. ii) For a single-layer bitstream, each picture buffer (the storage slot for a decoded picture in the DPB) occupies the buffer size required to store a decoded picture with the maximum picture resolution.

[0117] 3.4. Scalable Video Coding (SVC) in General and in VVC

[0118] Scalable Video Coding (SVC, sometimes also referred to as scalability in video coding) refers to video coding in which a base layer (BL), sometimes referred to as a reference layer (RL), and one or more scalable enhancement layers (EL) are used. In SVC, the base layer can carry video data with a base-level quality. One or more enhancement layers can carry additional video data to support, for example, higher spatial, temporal, and / or signal-to-noise (SNR) levels. Enhancement layers can be defined relative to previously encoded layers. For example, the bottom layer can be used as the BL, and the top layer can be used as the EL. Intermediate layers can act as ELs or RLs, or both. For example, an intermediate layer (e.g., a layer that is neither the lowest nor the highest layer) can be an EL of the layer below (e.g., the base layer or any intermediate enhancement layer) and at the same time an RL for one or more enhancement layers above the intermediate layer. Similarly, in the multi-view or 3D extension of the HEVC standard, there can be multiple views, and the information of one view can be used to code (e.g., encode or decode) the information of another view (e.g., motion estimation, motion vector prediction, and / or other redundancies).

[0119] In SVC, parameters used by the encoder or decoder are grouped into parameter sets based on the coding level (e.g., video level, sequence level, picture level, slice level, etc.) where they can be utilized. For example, parameters that can be utilized by one or more coded video sequences in different layers of the bitstream can be included in the Video Parameter Set (VPS), and parameters utilized by one or more pictures in a coded video sequence can be included in the Sequence Parameter Set (SPS). Similarly, parameters utilized by one or more slices in a picture can be included in the Picture Parameter Set (PPS), and other parameters specific to a single slice can be included in the slice header. Similarly, an indication of which (which) parameter set a particular layer uses at a given time can be provided at various coding levels.

[0120] Due to the support for reference picture resampling (RPR) in VVC, the support for bitstreams containing multiple layers, e.g., two layers with SD and HD resolutions in VVC, can be designed without any additional signaling processing level codec tools because the upsampling required for spatial scalability support can be achieved using only the RPR upsampling filter. However, for scalability support, a high level of syntax change is required (compared to non - scalability support). Scalability support is specified in VVC version 1. Different from the scalability support in any early video codec standard (including extensions of AVC and HEVC), the VVC scalability is designed to be as friendly as possible to single - layer decoder designs. The decoding capabilities of multi - layer bitstreams are specified in a way as if there were only a single layer in the bitstream. For example, decoding capabilities such as DPB size are specified independently of the number of layers in the bitstream to be decoded. Basically, a decoder designed for single - layer bitstreams can decode multi - layer bitstreams with little change. Compared to the multi - layer extension designs of AVC and HEVC, the HLS aspect has been significantly simplified at the cost of some flexibility. For example, an IRAP AU needs to include pictures of each layer present in the CVS.

[0121] 3.5. Random Access and Its Support in HEVC and VVC

[0122] Random access means accessing and decoding a bitstream starting from a picture that is not the first picture of the bitstream in decoding order. To support tuning and channel switching in broadcast / multicast and multi - party video conferencing, searching in local playback and streaming, and stream adaptation in streaming, the bitstream needs to include frequent random access points, which are usually intra - coded pictures but can also be inter - coded pictures (e.g., in the case of progressive decoding refresh).

[0123] HEVC includes signaling of Intra Random Access Point (IRAP) pictures in the NAL unit header by NAL unit type. Three types of IRAP pictures are supported, namely Instantaneous Decoder Refresh (IDR), Clean Random Access (CRA), and Broken Link Access (BLA) pictures. IDR pictures constrain the inter-picture prediction structure to not reference any pictures prior to the current group-of-pictures (GOP), which is typically referred to as a closed-GOP random access point. By allowing certain pictures to reference pictures prior to the current GOP, CRA pictures are less restrictive, and in the case of random access, all pictures are discarded. CRA pictures are typically referred to as open-GOP random access points. BLA pictures typically result from the concatenation of two bitstreams or parts thereof in a CRA picture, for example during stream switching. To enable better system use of IRAP pictures in the system, a total of six different NAL units are defined to signal the attributes of IRAP pictures, which can be used to better match the stream access point types defined in the ISO base media file format (ISOBMFF), which is used for random access support in Dynamic Adaptive Streaming over HTTP (DASH).

[0124] VVC supports three types of IRAP pictures, two types of IDR pictures (one type with or the other type without associated RADL pictures), and one type of CRA picture. These are basically the same as in HEVC. The BLA picture type in HEVC is not included in VVC, mainly for two reasons: i) The basic function of BLA pictures can be achieved by CRA pictures plus the Sequence End NAL unit, the presence of which indicates that the subsequent pictures start a new CVS in a single-layer bitstream. ii) During the development of VVC, it was desired to specify fewer NAL unit types than in HEVC, such as by using five bits instead of six bits for the NAL unit type field in the NAL unit header to indicate.

[0125] Another key difference in random access support between VVC and HEVC is that GDR is supported in a more standardized way in VVC. In GDR, the decoding of the bitstream can start from an inter-coded picture, and although not the entire picture area can be correctly decoded at the beginning, after multiple pictures, the entire picture area will be correct. AVC and HEVC also support GDR, using recovery point SEI messages to signal GDR random access points and recovery points. In VVC, a new NAL unit type is specified to indicate GDR pictures, and the recovery point is signaled in the picture header syntax structure. Starting with GDR pictures allows for CVS and bitstreams. This means that the entire bitstream can be allowed to contain only inter-coded pictures without a single intra-coded picture. The main benefit of specifying GDR support in this way is to provide consistent behavior for GDR. GDR-enabled encoders smooth the bitrate of the bitstream by distributing intra-coded strips or blocks across multiple pictures instead of intra-coding the entire picture, allowing for a significant reduction in end-to-end latency, which is considered more important today as ultra-low latency applications such as wireless displays, online games, and drone-based applications become increasingly popular.

[0126] Another GDR-related feature in VVC is virtual boundary signaling. The boundary between the refreshed area (i.e., the correctly decoded area) and the non-refreshed area at the picture between a GDR picture and its recovery point can be signaled as a virtual boundary, and when signaled, the loop filter across the boundary will not be applied, so there will be no decoding mismatch for some samples at or near the boundary. This can be useful when the application determines to display the correctly decoded area during GDR processing.

[0127] IRAP pictures and GDR pictures can be collectively referred to as random access point (RAP) pictures.

[0128] 3.6. Reference Picture Management and Reference Picture List (RPL)

[0129] Reference picture management is a core function essential for any video coding scheme using inter-frame prediction. It manages the storage and removal of reference pictures in the decoded picture buffer (DPB) and places the reference pictures in the RPL in their correct order.

[0130] The reference picture management in HEVC, which includes reference picture marking and removal from the decoded picture buffer (DPB) as well as reference picture list construction (RPLC), is different from that in AVC. Instead of the reference picture marking mechanism based on a sliding window plus adaptive memory management control operation (MMCO) in AVC, HEVC specifies a reference picture management and marking mechanism based on the so-called reference picture set (RPS), and thus RPLC is based on the RPS mechanism. The RPS consists of a set of reference pictures associated with a picture, including all reference pictures in decoding order before the associated picture, which can be used for inter prediction of the associated picture or any picture after the associated picture in decoding order. The reference picture set includes five reference picture lists. The first three lists contain all reference pictures that can be used for inter prediction of the current picture and can be used for inter prediction of one or more pictures after the current picture in decoding order. The other two lists consist of all reference pictures that are not used in the inter prediction of the current picture but can be used for inter prediction of one or more pictures after the current picture in decoding order. The RPS provides "intra-coded" signaling notification of the DPB state, rather than "inter-coded" signaling notification as in AVC, mainly to improve fault tolerance. The RPLC process in HEVC is based on the RPS, by signaling an index for each reference index to an RPS subgroup; this process is simpler than the RPLC process in AVC.

[0131] The reference picture management in VVC is more similar to HEVC than to AVC, but is simpler and more robust. As in those standards, two RPLs, list 0 and list 1, are derived, but they are not based on the reference picture set concept used in HEVC or the automatic sliding window process used in AVC; instead, they are signaled more directly. The reference pictures in the RPL are listed as active and inactive entries, and only the active entries can be used as reference indices for inter prediction of the CTUs of the current picture. The inactive entries indicate other pictures to be hosted in the DPB for reference of other pictures arriving later in the bitstream.

[0132] 3.7. Parameter Sets

[0133] AVC, HEVC, and VVC specify parameter sets. The types of parameter sets include SPS, PPS, APS, and VPS. AVC, HEVC, and VVC all support SPS and PPS. VPS was introduced starting from HEVC and is included in both HEVC and VVC. APS is not included in AVC or HEVC, but is included in the latest VVC draft text.

[0134] The SPS is designed to carry sequence-level header information, while the PPS is designed to carry picture-level header information that does not change frequently. By using the SPS and PPS, it is not necessary to repeat the information that does not change frequently for each sequence or picture, thus avoiding redundant signaling of this information. In addition, the use of the SPS and PPS enables out-of-band transmission of important header information, which not only avoids the need for redundant transmission but also improves fault tolerance.

[0135] The VPS is introduced to carry sequence-level header information that is common to all layers in a multi-layer bitstream.

[0136] The APS is introduced to carry picture-level or slice-level information that requires a significant number of bits to encode and decode, can be shared by multiple pictures, and can have a significant number of different variations in the sequence.

[0137] 3.8. Related Definitions in VVC

[0138] The related definitions in the latest VVC text (JVET-Q2001-vE / v15) are as follows.

[0139] In accordance with Previous (when present) has the same value as the nuh_layer_id of a specific picture.

[0140] Is Of

[0141] Each Includes those with a nal_unit_type equal to CRA_NUT

[0142] In accordance with Consists of Followed by zero or more that are not Of Includes all subsequent But does not include any subsequent ones that are Of

[0143] Each layer in the CVS has And each In Is Of

[0144] Each current in is of

[0145] is of

[0146] Each VCL NAL unit includes an

[0147] encoded / decoded picture that is of

[0148] each including an

[0149] Each layer in the CVS has and each in is of

[0150] is of

[0151] All with the same value of nal_unit_type within the range (inclusive) from IDR_W_RADL to CRA_NUT and in the same and in the order output from before

[0152] from for output from the DPB (for ).

[0153] is of

[0154] each including those with nal_unit_type equal to RADL_NUT

[0155] is of

[0156] each including those with nal_unit_type equal to RASL_NUT

[0157] is of

[0158] Each VCL NAL unit includes those with nal_unit_type equal to STSA_NUT

[0159] Note – An STSA picture does not use a picture with the same TemporalId as the STSA picture for inter prediction reference. A picture with the same TemporalId as the STSA picture that comes after the STSA picture in decoding order does not use a picture with the same TemporalId as the STSA picture that comes before the STSA picture in decoding order for inter prediction reference. The STSA picture enables switching from the immediately lower sublayer to the sublayer containing the STSA picture at the STSA picture. The TemporalId of the STSA picture must be greater than 0.

[0160] a rectangular region of one or more slices within.

[0161] According to at after and non-IRAP that is not an STSA picture

[0162] Note – The post pictures related to an IRAP picture are also after the IRAP picture in decoding order. A picture that is after the associated IRAP picture in output order and before the associated IRAP picture in decoding order is not allowed.

[0163] 3.9. NAL unit header syntax and semantics in VVC

[0164] In the latest VVC text (in JVET-Q2001-vE / v15), the NAL unit header syntax and semantics are as follows.

[0165] 7.3.1.2 NAL Unit Header Syntax

[0166]

[0167]

[0168] 7.4.2.2 NAL Unit Header Semantics

[0169] shall be equal to 0.

[0170] shall be equal to 0. The value of nuh_reserved_zero_bit may be specified by ITU-T|ISO / IEC in the future. The decoder shall ignore (i.e., remove and discard from the bitstream) NAL units for which nuh_reserved_zero_bit is equal to 1.

[0171] Identifies the layer to which the VCL NAL unit belongs or the layer to which the non-VCL NAL unit applies. The value of nuh_layer_id shall be in the range 0 to 55, inclusive. Other values of nuh_layer_id are reserved for future use by ITU-T|ISO / IEC.

[0172] The value of nuh_layer_id shall be the same for all VCL NAL units of a coded picture. The value of nuh_layer_id for a coded picture or a PU is the value of nuh_layer_id of the VCL NAL units of the coded picture or PU.

[0173] The value of nuh_layer_id for AUD, PH, EOS, and FD NAL units is subject to the following constraints:

[0174] – If nal_unit_type is equal to AUD_NUT, then nuh_layer_id shall be equal to vps_layer_id[0].

[0175] – Otherwise, when nal_unit_type is equal to PH_NUT, EOS_NUT, or FD_NUT, nuh_layer_id shall be equal to the nuh_layer_id of the associated VCL NAL unit.

[0176] Note 1 – The value of nuh_layer_id for DCI, VPS, and EOB NAL units is not constrained.

[0177] The value of nal_unit_type shall be the same for all pictures of a CVSS AU.

[0178] Specify the NAL unit type, i.e., the type of the RBSP data structure contained in the NAL unit as specified in Table 5.

[0179] NAL units with unspecified semantics where nal_unit_type is in the range UNSPEC_28..UNSPEC_31 (inclusive) shall not affect the decoding process specified in this specification.

[0180] Note 2 – NAL unit types in the range UNSPEC_28..UNSPEC_31 can be used as determined by the application. The decoding process for these values of nal_unit_type is not specified in this specification. Since different applications may use these NAL unit types for different purposes, special care must be taken in practice when designing an encoder that generates NAL units with these nal_unit_type values and when designing a decoder that interprets the content of NAL units with these nal_unit_type values. This specification does not define any management of these values. These nal_unit_type values may only be applicable in contexts where "conflicts" (i.e., different definitions of the meaning of the content of NAL units with the same nal_unit_type value) are unimportant, or impossible, or are managed (e.g., defined or managed in a control application or transport specification, or through the context of controlling bitstream distribution).

[0181] For purposes other than determining the amount of data in the DU of the bitstream (as specified in Appendix C), the decoder shall ignore (remove and discard from the bitstream) the content of all NAL units that use the reserved values of nal_unit_type.

[0182] Note 3 – This requirement allows for future definition of compatible extensions to this specification.

[0183] Table 5 – NAL unit type codes and NAL unit type classifications

[0184]

[0185]

[0186] Note 4 – A clean random access (CRA) picture can have associated RASL or RADL pictures present in the bitstream.

[0187] Annotation 5 – An Instantaneous Decoding Refresh (IDR) picture with nal_unit_type equal to IDR_N_LP has no associated previous picture in the bitstream. An IDR picture with nal_unit_type equal to IDR_W_RADL has no associated RASL picture present in the bitstream, but may have an associated RADL picture in the bitstream.

[0188] The nal_unit_type values of all VCL NAL units in a sub-picture shall be the same. The sub-picture is said to have the same NAL unit type as the VCL NAL units of the sub-picture.

[0189] For the VCL NAL units of any particular picture, the following applies:

[0190] – If the mixed_nalu_types_in_pic_flag is equal to 0, the value of nal_unit_type shall be the same for all VCL NAL units of the picture, and the picture or PU is considered to have the same NAL unit type as the coded slice NAL units of the picture or PU.

[0191] – Otherwise (mixed_nalu_types_in_pic_flag equal to 1), the picture shall have at least two sub-pictures, and the VCL NAL units of the picture shall have exactly two different nal_unit_type values, as follows: The VCL NAL units of at least one sub-picture of the picture shall all have a nal_unit_type with a specific value equal to STSA_NUT, RADL_NUT, RASL_NUT, IDR_W_RADL, IDR_N_LP or CRA_NUT, while the VCL NAL units of the other sub-pictures in the picture shall all have a different specific value equal to TRAIL_NUT, RADL_NUT, RASL_NUT.

[0192] For a single-layer bitstream, the following constraints apply:

[0193] – Except for the first picture in the bitstream in decoding order, each picture is considered to be associated with the previous IRAP picture in decoding order.

[0194] – When a picture is the previous picture of an IRAP picture, it shall be a RADL or RASL picture.

[0195] – When a picture is the subsequent picture of an IRAP picture, it shall not be a RADL or RASL picture.

[0196] – There shall be no RASL picture associated with an IDR picture in the bitstream.

[0197] – There shall be no RADL pictures associated with an IDR picture having a nal_unit_type equal to IDR_N_LP in the bitstream.

[0198] Annex 6 – Random access can be performed at the position of an IRAP PU by discarding all PUs before the IRAP PU (and correctly decoding the IRAP picture and all subsequent non-RASL pictures in decoding order), provided that each parameter set is available when referenced (either in the bitstream or by an external means not specified in this specification).

[0199] – Any picture before an IRAP picture in decoding order shall be before the IRAP picture in output order and shall be before any RADL picture associated with the IRAP picture in output order.

[0200] – Any RASL picture associated with a CRA picture in output order shall be before any RADL picture associated with the CRA picture.

[0201] – Any RASL picture associated with a CRA picture in output order shall be after any IRAP picture before the CRA picture in decoding order.

[0202] – If field_seq_flag is equal to 0 and the current picture is a prefix picture associated with an IRAP picture, then in decoding order it shall be before all non-prefix pictures associated with the same IRAP picture in decoding order. Otherwise, let picA and picB be the first and last prefix pictures associated with the IRAP picture in decoding order, respectively. There shall be at most one non-prefix picture before picA in decoding order and no non-prefix pictures between picA and picB in decoding order.

[0203] Minus 1 specifies the temporal identifier of the NAL unit.

[0204] The value of nuh_temporal_id_plus1 shall not be equal to 0.

[0205] The derivation of the variable TemporalId is as follows:

[0206] TemporalId = nuh_temporal_id_plus1 - 1 (36)

[0207] When nal_unit_type is in the range from IDR_W_RADL to RSV_IRAP_12, inclusive, TemporalId shall be equal to 0.

[0208] When nal_unit_type is equal to STSA_NUT and vps_independent_layer_flag[GeneralLayerIdx[nuh_layer_id]] is equal to 1, TemporalId shall not be equal to 0.

[0209] For all VCL NAL units of an AU, the value of TemporalId shall be the same. The value of TemporalId for a coded picture, PU, or AU is the value of TemporalId for the VCL NAL units of the coded picture, PU, or AU. The value of TemporalId for a sublayer representation is the maximum value of TemporalId for all VCL NAL units in the sublayer representation.

[0210] The value of TemporalId for non-VCL NAL units is subject to the following constraints:

[0211] – If nal_unit_type is equal to DCI_NUT, VPS_NUT, or SPS_NUT, then TemporalId shall be equal to 0, and the TemporalId of the AU containing the NAL unit shall be equal to 0.

[0212] – Otherwise, if nal_unit_type is equal to PH_NUT, then TemporalId shall be equal to the TemporalId of the PU containing the NAL unit.

[0213] – Otherwise, if nal_unit_type is equal to EOS_NUT or EOB_NUT, then TemporalId shall be equal to 0.

[0214] – Otherwise, if nal_unit_type is equal to AUD_NUT, FD_NUT, PREFIX_SEI_NUT, or SUFFIX_SEI_NUT, then TemporalId shall be equal to the TemporalId of the AU containing the NAL unit.

[0215] – Otherwise, when nal_unit_type is equal to PPS_NUT, PREFIX_APS_NUT, or SUFFIX_APS_NUT, TemporalId shall be greater than or equal to the TemporalId of the PU containing the NAL unit.

[0216] Annotation 7 – When the NAL unit is a non-VCL NAL unit, the value of TemporalId is equal to the minimum value of the TemporalId values of all AUs applicable to the non-VCL NAL unit. When nal_unit_type is equal to PPS_NUT, PREFIX_APS_NUT, or SUFFIX_APS_NUT, TemporalId can be greater than or equal to the TemporalId of the AU containing it, because all PPSs and APSs can be included at the beginning of the bitstream (e.g., when they are transmitted out-of-band and the receiver places them at the beginning of the bitstream), where the TemporalId of the first decoded picture is equal to 0.

[0217] 3.10. Mixed NAL unit types in picture

[0218] 7.4.3.4 Picture parameter set semantics ...

[0220] Each picture that references a PPS and has a value of 1 for mixed_nalu_types_in_pic_flag has more than one VCL NAL unit, the VCL NAL units do not have the same nal_unit_type value, and the picture is not an IRAP picture. A value of 0 for mixed_nalu_types_in_pic_flag specifies that each picture that references a PPS has one or more VCL NAL units, and the VCL NAL units of each picture that references a PPS have the same nal_unit_type value.

[0221] When no_mixed_nalu_types_in_pic_constraint_flag is equal to 1, the value of mixed_nalu_types_in_pic_flag shall be equal to 0.

[0222] For each slice with a nal_unit_type value nalUnitTypeA in the range from IDR_W_RADL to CRA_NUT (inclusive), in a picture picA that also contains one or more slices with another value of nal_unit_type (i.e., the value of mixed_nalu_types_in_pic_flag for picture picA is equal to 1), the following applies:

[0223] – The slice shall belong to the subpicture for which the corresponding value of subpic_treated_as_pic_flag[i] is equal to 1

[0224] subpicA.

[0225] – A strip shall not belong to picA sub - picture that contains VCL NAL units and nal_unit_type is not equal to nalUnitTypeA.

[0226] – If nalUnitTypeA is equal to CRA, for all subsequent PUs in CLVS after the current picture in decoding order and output order, neither RefPicList[0] nor RefPicList[1] of the strips in subpicA of these PUs shall include any picture that precedes picA in decoding order in the active entries.

[0227] – Otherwise (i.e., nalUnitTypeA is equal to IDR_W_RADL or IDR_N_LP), for all PUs in CLVS after the current picture in decoding order, neither RefPicList[0] nor RefPicList[1] of the strips in subpicA of these PUs shall include any picture that precedes picA in decoding order in the active entries.

[0228] Note 1 – mixed_nalu_types_in_pic_flag being equal to 1 indicates that the picture referring to the PPS contains strips with different NAL unit types, e.g., a coded picture resulting from a sub - picture bitstream merge operation, where the encoder must ensure that the bitstream structure matches and further alignment of the original bitstream parameters. An example of such alignment is as follows: when the value of sps_idr_rpl_present_flag is equal to 0 and mixed_nalu_types_in_pic_flag is equal to 1, the picture referring to the PPS cannot have strips with nal_unit_type equal to IDR_W_RADL or IDR_N_LP. ...

[0230] 3.11. Picture Header Structure Syntax and Semantics in VVC

[0231] In the latest VVC text (in JVET - Q2001 - vE / v15), the picture header structure syntax and semantics most relevant to the technical solution of this article are as follows.

[0232] 7.3.2.7 Picture Header Structure Syntax

[0233]

[0234]

[0235] 7.4.3.7 Picture Header Structure Semantics

[0236] The PH syntax structure contains information common to all slices of the coded pictures associated with the PH syntax structure.

[0237] Being equal to 1 specifies that the current picture is a GDR or IRAP picture. gdr_or_irap_pic_flag being equal to 0 specifies that the current picture may or may not be a GDR or IRAP picture.

[0238] Being equal to 1 specifies that the picture associated with PH is a GDR picture. gdr_pic_flag being equal to 0 specifies that the picture associated with PH is not a GDR picture. When absent, the value of gdr_pic_flag is inferred to be equal to 0. When gdr_enabled_flag is equal to 0, the value of gdr_pic_flag shall be equal to 0.

[0239] Note 1 – When gdr_or_irap_pic_flag is equal to 1 and gdr_pic_flag is equal to 0, the picture associated with PH is an IRAP picture. ...

[0241] Specifies the picture order count modulo MaxPicOrderCntLsb of the current picture. The length of the ph_pic_order_cnt_lsb syntax element is log2_max_pic_order_cnt_lsb_minus4 + 4 bits. The value of ph_pic_order_cnt_lsb shall be in the range of 0 to MaxPicOrderCntLsb−1, inclusive.

[0242] Affects the output of the previously decoded pictures in the DPB after decoding a CLVSS picture that is not the first picture in the bitstream specified in Annex C.

[0243] Specify the recovery point for decoding pictures in the output order. If the current picture is a GDR picture associated with PH, and there is a picture picA in the CLVS with PicOrderCntVal that is after the current GDR picture in the decoding order, and PicOrderCntVal is equal to the PicOrderCntVal of the current GDR picture plus the value of recovery_poc_cnt, then the picture picA is called a recovery point picture. Otherwise, the first picture with a PicOrderCntVal greater than the PicOrderCntVal of the current picture plus the value of recovery_poc_cnt in the output order is called a recovery point picture. The recovery point picture should not be before the current GDR picture in the decoding order. The value of recovery_poc_cnt should be in the range of 0 to MaxPicOrderCntLsb - 1 (inclusive).

[0244] When the current picture is a GDR picture, the variable RpPicOrderCntVal is derived as follows:

[0245] RpPicOrderCntVal = PicOrderCntVal + recovery_poc_cnt (81)

[0246] Note 2 – When gdr_enabled_flag is equal to 1 and the PicOrderCntVal of the current picture is greater than or equal to the RpPicOrderCntVal of the associated GDR picture, the current and subsequent decoded pictures in the output order exactly match the corresponding pictures generated by starting the decoding process from the previous IRAP picture (when it exists) that is before the associated GDR picture in the decoding order. ...

[0248] 3.12 Constraints on RPL in VVC

[0249] In the latest VVC text (in JVET - Q2001 - vE / v15), the VVC constraints on RPL are as follows (as part of the decoding process of the reference picture list construction in VVC clause 8.3.2).

[0250] 8.3.2 Decoding process of reference picture list construction ...

[0252] For each i equal to 0 or 1, the first NumRefIdxActive[i] entries in RefPicList[i] are called the active entries in RefPicList[i], and the other entries in RefPicList[i] are called the non - active entries in RefPicList[i].

[0253] Annotation 2 – A particular picture may be referred to by an entry in RefPicList[0] and an entry in RefPicList[1] simultaneously. It is also possible that a particular picture is referred to by more than one entry in RefPicList[0] or more than one entry in RefPicList[1].

[0254] Annotation 3 – The active entries in RefPicList[0] and the active entries in RefPicList[1] together refer to all the reference pictures that can be used for inter - prediction of the current picture and one or more pictures after the current picture in decoding order. The non - active entries in RefPicList[0] and the non - active entries in RefPicList[1] together refer to all the reference pictures that are not used for inter - prediction of the current picture but can be used for inter - prediction of one or more pictures after the current picture in decoding order.

[0255] Annotation 4 – There may be one or more entries in RefPicList[0] or RefPicList[1] equal to "no reference picture" because the corresponding picture does not exist in the DPB. Each non - active entry equal to "no reference picture" in RefPicList[0] or RefPicList[0] should be ignored. An unexpected picture loss should be inferred for each active entry equal to "no reference picture" in RefPicList[0] or RefPicList[1].

[0256] One requirement for bitstream conformance is to apply the following constraints:

[0257] – For each i equal to 0 or 1, num_ref_entries[i][RplsIdx[i]] should not be less than NumRefIdxActive[i].

[0258] – The picture referred to by each active entry in RefPicList[0] or RefPicList[1] should exist in the DPB, and the TemporalId should be less than or equal to the TemporalId of the current picture.

[0259] – The picture referred to by each entry in RefPicList[0] or RefPicList[1] should not be the current picture, and the non_reference_picture_flag should be equal to 0.

[0260] – The STRP entry in RefPicList[0] or RefPicList[1] of the strip of the picture and the LTRP entry in RefPicList[0] or RefPicList[1] of the same strip or different strips of the same picture shall not refer to the same picture.

[0261] – There shall be no LTRP entry in RefPicList[0] or RefPicList[1] where the difference between the PicOrderCntVal of the current picture and the PicOrderCntVal of the picture referred to by the entry is greater than or equal to 2. 24 of the LTRP entry.

[0262] – Let setOfRefPics be the unique set of pictures referred to by all entries in RefPicList[0] having the same nuh_layer_id as the current picture and all entries in RefPicList[1] having the same nuh_layer_id as the current picture. The number of pictures in setOfRefPics shall be less than or equal to MaxDpbSize - 1 (inclusive), where MaxDpbSize is specified in Clause A.4.2 and setOfRefPics shall be the same for all strips of the picture.

[0263] – When the nal_unit_type of the current strip is equal to STSA_NUT, there shall be no active entry in RefPicList[0] or RefPicList[1] where the TemporalId is equal to the TemporalId of the current picture and the nuh_layer_id is equal to the nuh_layer_id of the current picture.

[0264] – When the current picture is a picture after the STSA picture in decoding order and the TemporalId of the STSA picture is equal to the TemporalId of the current picture and the nuh_layer_id is equal to the nuh_layer_id of the current picture, there shall be no picture before the STSA picture in decoding order where the TemporalId of the STSA picture is equal to the TemporalId of the current picture and the nuh_layer_id is equal to the nuh_layer_id of the current picture, which is included as an active entry in RefPicList[0] or RefPicList[1].

[0265] – When the current picture is a CRA picture, there shall be no picture referred to by an entry in RefPicList[0] or RefPicList[1] in output order or in decoding order that is before any previous IRAP picture (when present) in decoding order.

[0266] – When the current picture is a post-picture, there shall be no picture involved in the active entry in RefPicList[0] or RefPicList[1] generated by the decoding process of generating unavailable reference pictures associated with the current picture.

[0267] – When the current picture is a post-picture that is after one or more pre-pictures associated with the same IRAP picture (if any) both in decoding order and in output order, there shall be no picture involved in the entry in RefPicList[0] or RefPicList[1] generated by the decoding process of generating unavailable reference pictures associated with the current picture.

[0268] – When the current picture is a recovery point picture or a picture after the recovery point picture in output order, there shall be no entry in RefPicList[0] or RefPicList[1] that contains a picture generated by the decoding process of generating unavailable reference pictures of the GDR picture of the recovery point picture.

[0269] – When the current picture is a post-picture, there shall be no picture involved in the active entry in RefPicList[0] or RefPicList[1] that is before the associated IRAP picture either in output order or in decoding order.

[0270] – When the current picture is a post-picture that is after one or more pre-pictures associated with the same IRAP picture (if any) both in decoding order and in output order, there shall be no picture involved in the entry in RefPicList[0] or RefPicList[1] that is before the associated IRAP picture either in output order or in decoding order.

[0271] – When the current picture is a RADL picture, there shall be no

[0272] any of the following active entries:

[0273] о RASL picture

[0274] о A picture generated by the decoding process of generating unavailable reference pictures

[0275] о A picture before the associated IRAP picture in decoding order

[0276] – For each ILRP entry in RefPicList[0] or RefPicList[1] of the strip of the current picture, the picture involved shall be in the same AU as the current picture.

[0277] – For each ILRP entry in RefPicList[0] or RefPicList[1] of the strip of the current picture, the picture involved shall exist in the DPB and the nuh_layer_id shall be less than the nuh_layer_id of the current picture.

[0278] – Each ILRP entry in RefPicList[0] or RefPicList[1] of the strip shall be an active entry. ...

[0280] 3.13. Setting of PictureOutputFlag

[0281] In the latest VVC text (in JVET-Q2001-vE / v15), the specification for setting the value of the variable PictureOutputFlag is as follows (as part of the decoding process in Clause 8.1.2 of the coded picture).

[0282] 8.1.2 Decoding Process of Coded Picture

[0283] The decoding process specified in this clause applies to each coded picture in BitstreamToDecode, referred to as the current picture and represented by the variable CurrPic.

[0284] Depending on the value of chroma_format_idc, the number of sample arrays of the current picture is as follows:

[0285] – If chroma_format_idc is equal to 0, the current picture consists of 1 sample array S L composed.

[0286] – Otherwise (chroma_format_idc is not equal to 0), the current picture consists of 3 sample arrays S L 、S Cb 、S Cr composed.

[0287] The decoding process of the current picture takes the syntax elements and uppercase variables from Clause 7 as input. When interpreting the semantics of each NAL unit and each syntax element in the remaining part of Clause 8, the term "bitstream" (or a part thereof, such as the CVS of the bitstream) refers to BitstreamToDecode (or a part thereof).

[0288] Depending on the value of separate_colour_plane_flag, the structure of the decoding process is as follows:

[0289] – If separate_colour_plane_flag is equal to 0, the decoding process is called once and the current picture is taken as the output.

[0290] – Otherwise (separate_colour_plane_flag is equal to 1), the decoding process is called three times. The input to the decoding process is all the NAL units of the coded pictures with the same colour_plane_id value. The decoding process for the NAL units with a specific colour_plane_id value is specified as if the bitstream only contained a CVS in a monochrome colour format with that specific colour_plane_id value. The output of each of the three decoding processes is assigned to one of the 3 sample arrays of the current picture, and the NAL units with colour_plane_id equal to 0, 1, and 2 are assigned to S L 、S Cb and S Cr .

[0291] Note – When separate_colour_plane_flag is equal to 1 and chroma_format_idc is equal to 3, the derived variable ChromaArrayType is equal to 0. In the decoding process, the value of this variable is evaluated, resulting in the same operation as for a monochrome picture (when chroma_format_idc is equal to 0).

[0292] For the current picture CurrPic, the decoding process operates as follows:

[0293] 1. Decode the NAL units as specified in Clause 8.2.

[0294] 2. The processing in Clause 8.3 specifies the following decoding processes using the syntax elements in the slice header layer and above layers:

[0295] – Derive the variables and functions related to picture order counting as specified in Clause 8.3.1. It only needs to be called for the first slice of the picture.

[0296] – At the start of the decoding process for each slice of a non-IDR picture, call the decoding process for reference picture list construction specified in Clause 8.3.2 to derive reference picture list 0 (RefPicList[0]) and reference picture list 1 (RefPicList[1]).

[0297] – Invoke the decoding process marked with the reference picture in Clause 8.3.3, where the reference picture can be marked as "not used for reference" or "used for long-term reference". It only needs to be invoked for the first strip of the picture.

[0298] – When the current picture is a CRA picture with NoOutputBeforeRecoveryFlag equal to 1 or a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, invoke the decoding process for generating unavailable reference pictures specified in Subclause 8.3.4. It only needs to be invoked for the first strip of the picture.

[0299] – PictureOutputFlag is set as follows:

[0300] – If one of the following conditions is true, then PictureOutputFlag is set to be equal to 0:

[0301] – The current picture is a RASL picture, and the NoOutputBeforeRecoveryFlag of the associated IRAP picture is equal to 1.

[0302] – gdr_enabled_flag is equal to 1, and the current picture is a GDR picture with NoOutputBeforeRecoveryFlag equal to 1.

[0303] – gdr_enabled_flag is equal to 1, the current picture is associated with a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, and the PicOrderCntVal of the current picture is less than the RpPicOrderCntVal of the associated GDR picture.

[0304] – sps_video_parameter_set_id is greater than 0, ols_mode_idc is equal to 0, and the current AU contains a picture picA that satisfies all of the following conditions:

[0305] – The PictureOutputFlag of PicA is equal to 1.

[0306] – The nuh_layer_id nuhLid of PicA is greater than the nuh_layer_id of the current picture.

[0307] – PicA belongs to the output layer of OLS (i.e., OutputLayerIdInOls[TargetOlsIdx][0] is equal to nuhLid).

[0308] – The sps_video_parameter_set_id is greater than 0, the ols_mode_idc is equal to 2, and the ols_output_layer_flag[TargetOlsIdx][GeneralLayerIdx[nuh_layer_id]] is equal to 0.

[0309] – Otherwise, PictureOutputFlag is set to be equal to pic_output_flag.

[0310] 3. The processing regulations in Clauses 8.4, 8.5, 8.6, 8.7, and 8.8 use the decoding process of the syntax elements in all syntax structure layers. The requirement for bitstream consistency is that the coded and decoded slices of a picture will contain the slice data of each CTU of the picture, such that dividing the picture into slices and dividing the slices into CTUs each forms a segmentation of the picture.

[0311] 4. After all slices of the current picture have been decoded, the currently decoded picture is marked as "for short-term reference", and each ILRP entry in RefPicList[0] or RefPicList[1] is marked as "for short-term reference".

[0312] 4. Technical problems solved by the disclosed technical solution

[0313] The existing designs in the latest VVC text (JVET-Q2001-vE / v15) have the following problems:

[0314] 1) Since different types of sub-pictures are allowed to be mixed in a picture, it is confusing to refer to the content of a NAL unit of VCL NAL unit type as the coded and decoded slice of a specific type of picture. For example, a NAL unit with a nal_unit_type equal to CRA_NUT is the coded and decoded slice of a CRA picture only when the nal_unit_type of all slices of the picture is equal to CRA_NUT; when the nal_unit_type of one slice of the picture is not equal to CRA_NUT, then the picture is not a CRA picture.

[0315] 2) Currently, if a sub-picture contains VCL NAL units with nal_unit_type in the range from IDR_W_RADL to CRA_NUT (including the endpoints), the value of subpic_treated_as_pic_flag[] for the sub-picture needs to be equal to 1, and the mixed_nalu_types_in_pic_flag for the picture is equal to 1. In other words, for an IRAP sub-picture mixed with another type of sub-picture in a picture, the value of subpic_treated_as_pic_flag[] needs to be equal to 1. However, with the support of more mixing of VCL NAL unit types, this requirement is insufficient.

[0316] 3) Currently, at most two different types of VCL NAL units (and two different types of sub-pictures) are allowed in a picture.

[0317] 4) In single-layer and multi-layer contexts, there is a lack of constraints on the output order of post sub-pictures relative to associated IRAP or GDR sub-pictures.

[0318] 5) Currently, it is stipulated that when a picture is a pre-picture of an IRAP picture, it should be a RADL or RASL picture. This constraint, together with the definition of pre- / RADL / RASL pictures, does not allow the mixing of RADL and RASL NAL unit types within a picture generated by the mixing of two CRA pictures and their associated RADL and RASL pictures that are not AU-aligned.

[0319] 6) In single-layer and multi-layer contexts, there is a lack of constraints on the sub-picture type of pre-sub-pictures (i.e., the NAL unit type of VCL NAL units in the sub-picture).

[0320] 7) In single-layer and multi-layer contexts, there is a lack of constraints on whether a RASL sub-picture can exist and be associated with an IDR sub-picture.

[0321] 8) In single-layer and multi-layer contexts, there is a lack of constraints on whether a RADL sub-picture can exist and be associated with an IDR sub-picture with nal_unit_type equal to IDR_N_LP.

[0322] 9) In single-layer and multi-layer contexts, there is a lack of constraints on the relative output order between sub-pictures before an IRAP sub-picture in decoding order and RADL sub-pictures associated with the IRAP sub-picture.

[0323] 10) In single-layer and multi-layer contexts, there is a lack of constraints on the relative output order between sub-pictures before a GDR sub-picture in decoding order and sub-pictures associated with the GDR sub-picture.

[0324] 11) In the single-layer and multi-layer contexts, there is a lack of constraint on the relative output order between the RASL sub-pictures associated with the CRA sub-pictures and the RADL sub-pictures associated with the CRA sub-pictures.

[0325] 12) In the single-layer and multi-layer contexts, there is a lack of constraint on the relative output order between the RASL sub-pictures associated with the CRA sub-pictures and the IRAP sub-pictures that are before the CRA sub-pictures in the decoding order.

[0326] 13) In the single-layer and multi-layer contexts, there is a lack of constraint on the relative decoding order between the associated non-preceding pictures and the preceding pictures of the IRAP pictures.

[0327] 14) In the single-layer and multi-layer contexts, there is a lack of constraint on the RPL active entries of the sub-pictures that are after the STSA sub-pictures in the decoding order.

[0328] 15) In the single-layer and multi-layer contexts, there is a lack of constraint on the RPL entries of the CRA sub-pictures.

[0329] 16) In the single-layer and multi-layer contexts, there is a lack of constraint on the RPL active entries of the sub-pictures that refer to the pictures generated by the decoding process for generating unavailable reference pictures.

[0330] 17) In the single-layer and multi-layer contexts, there is a lack of constraint on the RPL entries of the sub-pictures that refer to the pictures generated by the decoding process for generating unavailable reference pictures.

[0331] 18) In the single-layer and multi-layer contexts, there is a lack of constraint on the RPL active entries of the sub-pictures that are associated with the IRAP pictures and are after the IRAP pictures in the output order.

[0332] 19) In the single-layer and multi-layer contexts, there is a lack of constraint on the RPL entries of the sub-pictures that are associated with the IRAP pictures and are after the IRAP pictures in the output order.

[0333] 20) In the single-layer and multi-layer contexts, there is a lack of constraint on the RPL active entries of the RADL sub-pictures.

[0334] 5. Examples of Technical Solutions and Embodiments

[0335] To solve the above problems and other problems, the following summarized methods are disclosed. These items should be considered as examples for explaining general concepts and should not be construed in a narrow manner. In addition, these items can be used alone or in any combination.

[0336] 1) To solve Problem 1, instead of designating the content of NAL units with VCL NAL unit type as "coding strip of a specific type of picture", it is designated as "coding strip of a specific type of picture or sub - picture". For example, the content of an NAL unit with nal_unit_type equal to CRA_NUT is designated as "coding strip of a CRA picture or sub - picture".

[0337] a. In addition, one or more of the following terms are defined: associated GDR sub - picture, associated IRAP sub - picture, CRA sub - picture, GDR sub - picture, IDR sub - picture, IRAP sub - picture, pre - sub - picture, RADL sub - picture, RASL sub - picture, STSA sub - picture, post - sub - picture.

[0338] 2) To solve Problem 2, constraints are added to require that subpic_treated_as_pic_flag[] of any two adjacent sub - pictures of different NAL unit types should be equal to 1.

[0339] a. In one example, the constraint is specified as follows: for any two adjacent sub - pictures with sub - picture indices i and j in a picture, when subpic_treated_as_pic_flag[i] or subpic_treated_as_pic_flag[j] is equal to 0, the two sub - pictures should have the same NAL unit type.

[0340] a. Optionally, it is required that when subpic_treated_as_pic_flag[i] of any sub - picture with sub - picture index i is equal to 0, all sub - pictures in the picture should have the same NAL unit type (i.e., all VCL NAL units in the picture should have the same NAL unit type, i.e., the value of mixed_nalu_types_in_pic_flag should be equal to 0). And this means that when all sub - pictures make their corresponding subpic_treated_as_pic_flag[] equal to 1, mixed_nalu_types_in_pic_flag can only be equal to 1.

[0341] 3) To solve Problem 3, when mixed_nalu_types_in_pic_flag is equal to 1, a picture can be allowed to contain more than two different types of VCL NAL units.

[0342] 4) To solve Problem 4, it is specified that post - sub - pictures should be after the associated IRAP or GDR sub - pictures in the output order.

[0343] 5) To solve Problem 5, in order to allow the mixing of RADL and RASL NAL unit types in pictures resulting from the mixing of two CRA pictures and their non-AU-aligned associated RADL and RASL pictures, the existing constraint that the preceding picture specifying an IRAP picture should be a RADL or RASL picture is changed as follows: When a picture is the preceding picture of an IRAP picture, the nal_unit_type value of all VCL NAL units in the picture should be equal to RADL_NUT or RASL_NUT. In addition, in the decoding process of pictures using a mixed nal_unit_type value of RADL_NUT and RASL_NUT, when the layer containing the picture is the output layer, the PictureOutputFlag of the picture is set to be equal to pic_output_flag.

[0344] In this way, although when the NoOutputBeforeRecoveryFlag associated with the CRA picture is equal to 1, the guarantee of the "correctness" of the RASL sub-pictures of the "intermediate values" in such pictures is also appropriate, but it is actually not necessary. However, by requiring all output pictures to meet the correct constraints of the decoder, the RADL sub-pictures in such pictures can be guaranteed. The unnecessary parts of the guarantee are insignificant and do not increase the complexity of implementing a conforming encoder or decoder. In this case, it would be useful to add a NOTE to clarify that although such RASL sub-pictures associated with CRA pictures with a NoOutputBeforeRecoveryFlag equal to 1 can be output by the decoding process, they are not intended for display and thus are not applied to display.

[0345] 6) To solve Problem 6, it is specified that when a sub-picture is the preceding sub-picture of an IRAP sub-picture, it should be a RADL or RASL sub-picture.

[0346] 7) To solve Problem 7, it is specified that there should be no RASL sub-pictures associated with IDR sub-pictures in the bitstream.

[0347] 8) To solve Problem 8, it is specified that there should be no RADL sub-pictures associated with IDR sub-pictures with a nal_unit_type equal to IDR_N_LP in the bitstream.

[0348] 9) To solve problem 9, it is stipulated that any sub-picture with a nuh_layer_id equal to a specific value layerId and a sub-picture index equal to a specific value subpicIdx shall, in the output order, be before the IRAP sub-picture and all its associated RADL sub-pictures, and this any sub-picture shall, in the decoding order, be before the IRAP sub-picture with a nuh_layer_id equal to layerId and a sub-picture index equal to subpicIdx.

[0349] 10) To solve problem 10, it is stipulated that any sub-picture with a nuh_layer_id equal to a specific value layerId and a sub-picture index equal to a specific value subpicIdx shall, in the output order, be before the GDR sub-picture and all its associated sub-pictures, and this any sub-picture shall, in the decoding order, be before the GDR sub-picture with a nuh_layer_id equal to layerId and a sub-picture index equal to subpicIdx.

[0350] 11) To solve problem 11, it is stipulated that any RASL sub-picture associated with a CRA sub-picture shall, in the output order, be before any RADL sub-picture associated with the CRA sub-picture.

[0351] 12) To solve problem 12, it is stipulated that any RASL sub-picture associated with a CRA sub-picture shall, in the output order, be after any IRAP sub-picture that is before the CRA sub-picture in the decoding order.

[0352] 13) To solve problem 13, it is stipulated that if field_seq_flag is equal to 0, and the current sub-picture with a nuh_layer_id equal to a specific value layerId and a sub-picture index equal to a specific value subpicIdx is a pre-sub-picture associated with an IRAP sub-picture, then it shall, in the decoding order, be before all non-pre-sub-pictures associated with the same IRAP sub-picture; otherwise, let subpicA and subpicB be the first and last pre-sub-pictures associated with the IRAP sub-picture in the decoding order, respectively. There shall be at most one non-pre-sub-picture with a nuh_layer_id equal to layerId and a sub-picture index equal to subpicIdx before subpicA in the decoding order, and there shall be no non-pre-pictures with a nuh_layer_id equal to layerId and a sub-picture index equal to subpicIdx between picA and picB in the decoding order.

[0353] 14) To solve problem 14, it is stipulated that when the current subpicture with a TemporalId equal to a specific value tId, a nuh_layer_id equal to a specific value layerId, and a subpicture index equal to a specific value subpicIdx is a subpicture that is after the STSA subpicture with a TemporalId equal to tId, a nuh_layer_id equal to layerId, and a subpicture index equal to subpicIdx in decoding order, there should be no picture with a TemporalId equal to tId and a nuh_layer_id equal to layerId that is before the picture containing the STSA subpicture in decoding order and is included as an active entry in RefPicList[0] or RefPicList[1].

[0354] 15) To solve problem 15, it is stipulated that when the current subpicture with a nuh_layer_id equal to a specific value layerId and a subpicture index equal to a specific value subpicIdx is a CRA subpicture, there should be no picture referred to by an entry in RefPicList[0] or RefPicList[1] that is before any picture in output order or in decoding order before the previous IRAP subpicture (when it exists) with a nuh_layer_id equal to layerId and a subpicture index equal to subpicIdx.

[0355] 16) To solve problem 16, it is stipulated that when the current subpicture with a nuh_layer_id equal to a specific value layerId and a subpicture index equal to a specific value subpicIdx is not a subpicture of a RASL subpicture associated with a CRA picture with a NoOutputBeforeRecoveryFlag equal to 1, a GDR subpicture of a GDR picture with a NoOutputBeforeRecoveryFlag equal to 1, or a recovery picture of a GDR picture with a NoOutputBeforeRecoveryFlag equal to 1 and a nuh_layer_id equal to layerId, there should be no picture involved in the active entries in RefPicList[0] or RefPicList[1] generated by the decoding process of generating unavailable reference pictures.

[0356] 17) To solve problem 17, it is stipulated that when the current sub - picture with a nuh_layer_id equal to a specific value layerId and a sub - picture index equal to a specific value subpicIdx is not a CRA sub - picture of a CRA picture with a NoOutputBeforeRecoveryFlag equal to 1, a sub - picture before the same CRA sub - picture of a CRA picture with a NoOutputBeforeRecoveryFlag equal to 1 in decoding order, a pre - sub - picture associated with the CRA sub - picture of a CRA picture with a NoOutputBeforeRecoveryFlag equal to 1, a GDR sub - picture of a GDR picture with a NoOutputBeforeRecoveryFlag equal to 1, or a sub - picture of a recovery picture of a GDR picture with a NoOutputBeforeRecoveryFlag equal to 1 and a nuh_layer_id equal to layerId, there should be no picture involved by an entry in RefPicList[0] or RefPicList[1] generated by a decoding process that generates an unavailable reference picture.

[0357] 18) To solve problem 18, it is stipulated that when the current sub - picture is associated with an IRAP sub - picture and is after the IRAP sub - picture in output order, there should be no picture involved by an active entry in RefPicList[0] or RefPicList[1] before the picture containing the associated IRAP sub - picture in output order or in decoding order.

[0358] 19) To solve problem 19, it is stipulated that when the current sub - picture is associated with an IRAP sub - picture, after the IRAP sub - picture in output order, and after the pre - sub - picture associated with the same IRAP sub - picture (if any) in both decoding order and output order, there should be no picture involved by an entry in RefPicList[0] or RefPicList[1] before the picture containing the associated IRAP sub - picture in output order or in decoding order.

[0359] 20) To solve problem 20, it is stipulated that when the current sub - picture is a RADL sub - picture, there should not be any of the following active entries in RefPicList[0] or RefPicList[1]:

[0360] a. A picture containing a RASL sub - picture

[0361] b. A picture before the picture containing the associated IRAP sub - picture in decoding order

[0362] 6. Embodiments

[0363] The following are some example embodiments in terms of the technical solutions summarized in Section 5 above, which can be applied to the VVC specification. The modified text is based on the latest VVC text in JVET-Q2001-vE / v15. Most of the relevant parts added or modified are highlighted in bold italics, and some of the deleted parts are highlighted in double brackets (e.g., [[ ]]) in a prominent way, and the deleted text is between the double brackets. There are also some other changes that are editorial or not part of this technical solution and are thus not highlighted.

[0364] 6.1. First Embodiment

[0365] This embodiment is for Items 1, 1a, 2, 2a, 4, and 6 to 20.

[0366] 3 Definitions ...

[0368] The previous GDR with nuh_layer_id equal to layerId (when it exists), There is no with nuh_layer_id equal to layerId between this picture and a specific

[0369]

[0370] The previous with nuh_layer_id equal to layerId (when it exists), There is no GDR picture with nuh_layer_id equal to layerId between this picture and a specific one.

[0371]

[0372] Each picture with nal_unit_type equal to CRA_NUT .

[0373]

[0374] Each layer in the CVS has one and each current in All are GDR pictures

[0375] Each VCL NAL unit includes one with a nal_unit_type equal to GDR_NUT

[0376]

[0377] Each includes one with a nal_unit_type equal to IDR_W_RADL or IDR_N_LP

[0378]

[0379] All includes the same value of nal_unit_type within the range from IDR_W_RADL to CRA_NUT (inclusive)

[0380] Before in the

[0381] order output from the DPB in the

[0382] Each VCL NAL unit includes one with a nal_unit_type equal to RADL_NUT

[0383]

[0384] Each includes one with a nal_unit_type equal to RASL_NUT

[0385] Random Access Skip Leading (RASL) sub - picture: Each VCL NAL unit includes a sub - picture with a nal_unit_type equal to RASL_NUT.

[0386] Each including those with a nal_unit_type equal to STSA_NUT

[0387]

[0388] each unit includes those with a nal_unit_type equal to TRAIL_NUT

[0389] Note – Post pictures associated with IRAP or GDR pictures are also after the IRAP or GDR pictures in decoding order. Pictures that are after the associated IRAP or GDR pictures in output order and before the associated IRAP or GDR pictures in decoding order are not allowed.

[0390]

[0391] ...

[0393] 7.4.2.2 NAL Unit Header Semantics ...

[0395] Specifies the NAL unit type, i.e., the type of the RBSP data structure contained in the NAL unit as specified in Table 5.

[0396] NAL units with unspecified semantics where nal_unit_type is in the range UNSPEC_28..UNSPEC_31 (inclusive) shall not affect the decoding processes specified in this specification.

[0397] Note 2 – NAL unit types in the range UNSPEC_28..UNSPEC_31 may be used as determined by the application. The decoding processes for these values of nal_unit_type are not specified in this specification. Since different applications may use these NAL unit types for different purposes, care must be taken in practice when designing encoders that produce NAL units with these nal_unit_type values and when designing decoders that interpret the content of NAL units with these nal_unit_type values. This specification does not define any management of these values.

[0398] These nal_unit_type values may only apply in contexts where "conflicts" (i.e., different definitions of the meaning of NAL unit content with the same nal_unit_type value) are unimportant, or impossible, or are managed (e.g., defined or managed in control applications or transmission specifications, or through the context of the controlled bitstream distribution).

[0399] For purposes other than determining the amount of data in the DU of the bitstream (as specified in Annex C), the decoder shall ignore (remove and discard from the bitstream) the content of all NAL units using reserved values of nal_unit_type.

[0400] Note 3 – This requirement allows for future definition of compatible extensions to this specification.

[0401] Table 5 – NAL unit type codes and NAL unit type classifications

[0402]

[0403]

[0404]

[0405] Note 4 – A clean random access (CRA) picture may have associated RASL or RADL pictures present in the bitstream.

[0406] Note 5 – An instantaneous decoding refresh (IDR) picture with a nal_unit_type equal to IDR_N_LP has no associated preceding pictures in the bitstream. An IDR picture with a nal_unit_type equal to IDR_W_RADL has no associated RASL pictures present in the bitstream, but may have associated RADL pictures in the bitstream.

[0407] The nal_unit_type values of all VCL NAL units in a sub-picture shall be the same. The sub-picture is said to have the same NAL unit type as the VCL NAL units of the sub-picture.

[0408]

[0409] For the VCL NAL units of any particular picture, the following applies:

[0410] – If the mixed_nalu_types_in_pic_flag is equal to 0, the value of nal_unit_type shall be the same for all VCL NAL units of the picture, and the picture or PU is considered to have the same NAL unit type as the VCL NAL units of the picture or PU.

[0411] – Otherwise (when mixed_nalu_types_in_pic_flag equals 1), the picture shall have at least two sub - pictures, and the VCL NAL units of the picture shall have exactly two different nal_unit_type values as follows: the VCL NAL units of at least one sub - picture of the picture shall all have a specific value equal to a nal_unit_type of STSA_NUT, RADL_NUT, RASL_NUT, IDR_W_RADL, IDR_N_LP, or CRA_NUT, while the VCL NAL units of the other sub - pictures in the picture shall all have a different specific value equal to a nal_unit_type of TRAIL_NUT, RADL_NUT, RASL_NUT.

[0412] One requirement for bit - stream conformance is to apply the following constraints:

[0413] – The post - picture in output order shall be after the associated IRAP or GDR picture.

[0414]

[0415] – When the picture is a pre - picture of an IRAP picture, it shall be a RADL or RASL picture.

[0416]

[0417] – There shall be no RASL picture associated with an IDR picture in the bit - stream.

[0418]

[0419] – There shall be no RADL picture associated with an IDR picture having a nal_unit_type equal to IDR_N_LP in the bit - stream.

[0420] Note 6 – Random access can be performed at the position of an IRAP PU (and the IRAP picture and all subsequent non -

[0421] RASL pictures can be decoded correctly in decoding order) by discarding all PUs before the IRAP PU, provided that each parameter set is available when referenced (either in the bit - stream or by an external means not specified in this specification).

[0422]

[0423] – Any picture (with nuh_layer_id equal to the specific value layerId) before an IRAP picture with nuh_layer_id equal to layerId in decoding order shall be before the IRAP picture and all its associated RADL pictures in output order.

[0424]

[0425] – Any picture (with nuh_layer_id equal to the specific value layerId) before a GDR picture with nuh_layer_id equal to layerId in decoding order shall be before the GDR picture and all its associated pictures in output order.

[0426]

[0427] – Any RASL picture associated with a CRA picture in output order shall be before any RADL picture associated with the CRA picture.

[0428]

[0429] – Any RASL picture associated with a CRA picture in output order shall be after any IRAP picture before the CRA picture in decoding order.

[0430]

[0431] – If field_seq_flag is equal to 0 and the current picture with nuh_layer_id equal to the specific value layerId is a leading picture associated with an IRAP picture, it shall be before all non-leading pictures associated with the same IRAP picture in decoding order. Otherwise, let picA and picB be the first and last leading pictures associated with the IRAP picture in decoding order. There shall be at most one non-leading picture with nuh_layer_id equal to layerId before picA in decoding order, and there shall be no non-leading pictures with nuh_layer_id equal to layerId between picA and picB in decoding order.

[0432]

[0433] ...

[0435] 7.4.3.4 Picture Parameter Set Semantics ...

[0437] It is stipulated that each picture that refers to the PPS and is equal to 1 has more than one VCL NAL unit The VCL NAL units do not have the same nal_unit_type value [[, and the picture is not an IRAP picture]]. The mixed_nalu_types_in_pic_flag being equal to 0 stipulates that each picture that refers to the PPS has one or more VCL NAL units, and the VCL NAL units of each picture that refers to the PPS have the same nal_unit_type value.

[0438] When no_mixed_nalu_types_in_pic_constraint_flag is equal to 1, the value of mixed_nalu_types_in_pic_flag shall be equal to 0.

[0439] [[For each slice with a nal_unit_type value nalUnitTypeA within the range of IDR_W_RADL to CRA_NUT (inclusive) in a picture picA that also contains one or more slices with another value of nal_unit_type (i.e., the value of mixed_nalu_types_in_pic_flag of picture picA is equal to 1), the following applies:

[0440] – The slice shall belong to the subpicture subpicA for which the value of the corresponding subpic_treated_as_pic_flag[i] is equal to 1.

[0441] – The slice shall not belong to the subpicture of picA that contains VCL NAL units and whose nal_unit_type is not equal to nalUnitTypeA.

[0442] – If nalUnitTypeA is equal to CRA, for all subsequent PUs in the CLVS after the current picture in decoding order and output order, neither RefPicList[0] nor RefPicList[1] of the slices in subpicA of these PUs shall include any picture that precedes picA in decoding order in the active entries.

[0443] – Otherwise (i.e., nalUnitTypeA is equal to IDR_W_RADL or IDR_N_LP), for all PUs in the CLVS after the current picture in decoding order, neither RefPicList[0] nor RefPicList[1] of the slices in subpicA of these PUs shall include any picture that precedes picA in decoding order in the active entries.]]

[0444] Annotation 1 – When mixed_nalu_types_in_pic_flag equals 1, it indicates that the picture referring to the PPS contains slices with different NAL unit types. For example, a coded picture resulting from a sub-picture bitstream merge operation, where the encoder must ensure that the bitstream structure matches and further alignment of the original bitstream parameters. An example of such alignment is as follows: When the value of sps_idr_rpl_present_flag equals 0 and mixed_nalu_types_in_pic_flag equals 1, the picture referring to the PPS shall not have slices with nal_unit_type equal to IDR_W_RADL or IDR_N_LP. ...

[0446] 7.4.3.7 Picture Header Structure Semantics ...

[0448] Specifies the recovery points for decoding pictures in output order.

[0449]

[0450]

[0451] If the current picture is a [[PH - associated]] GDR picture, and there is a picture picA in the CLVS that is after the current GDR picture in decoding order and has a PicOrderCntVal equal to [[the value of PicOrderCntVal of the current GDR picture plus recovery_poc_cnt]], then the picture picA is called a recovery point picture. Otherwise, the first picture with a PicOrderCntVal greater than [[the value of PicOrderCntVal of the current picture plus recovery_poc_cnt]] in output order is called a recovery point picture. The recovery point picture shall not be before the current GDR picture in decoding order. The value of recovery_poc_cnt shall be in the range of 0 to MaxPicOrderCntLsb - 1 (inclusive).

[0452] [[When the current picture is a GDR picture, the variable RpPicOrderCntVal is derived as follows:

[0453] RpPicOrderCntVal = PicOrderCntVal + recovery_poc_cnt (81)]

[0454] Annotation 2 – When gdr_enabled_flag is equal to 1 and the PicOrderCntVal of the current picture is greater than or equal to [[RpPicOrderCntVal]] of the associated GDR picture, the current and subsequent decoded pictures in the output order exactly match the corresponding pictures generated by starting the decoding process from the previous IRAP picture (when present) before the associated GDR picture in the decoding order. ...

[0456] 8.3.2 Decoding process for reference picture list construction ...

[0458] One requirement for bitstream consistency is to apply the following constraints:

[0459] – For each i equal to 0 or 1, num_ref_entries[i][RplsIdx[i]] shall not be less than NumRefIdxActive[i].

[0460] – The pictures involved in each active entry in RefPicList[0] or RefPicList[1] shall exist in the DPB, and the TemporalId shall be less than or equal to the TemporalId of the current picture.

[0461] – The pictures involved in each entry in RefPicList[0] or RefPicList[1] shall not be the current picture, and the non_reference_picture_flag shall be equal to 0.

[0462] – The STRP entries in RefPicList[0] or RefPicList[1] of the strip of a picture and the LTRP entries in RefPicList[0] or RefPicList[1] of the same strip or different strips of the same picture shall not involve the same picture.

[0463] – There shall not be an LTRP entry in RefPicList[0] or RefPicList[1] where the difference between the PicOrderCntVal of the current picture and the PicOrderCntVal of the picture involved in the entry is greater than or equal to 2 24 of the LTRP entry.

[0464] – Let setOfRefPics be the unique set of pictures involved by all the entries in RefPicList[0] that have the same nuh_layer_id as the current picture and all the entries in RefPicList[1] that have the same nuh_layer_id as the current picture. The number of pictures in setOfRefPics shall be less than or equal to MaxDpbSize - 1 (inclusive), where MaxDpbSize is specified in Clause A.4.2 and setOfRefPics shall be the same for all slices of the picture.

[0465] – When the nal_unit_type of the current slice is equal to STSA_NUT, there shall be no active entry in RefPicList[0] or RefPicList[1] whose TemporalId is equal to the TemporalId of the current picture and whose nuh_layer_id is equal to the nuh_layer_id of the current picture.

[0466] – When the current picture is a picture that comes after the STSA picture in decoding order and the TemporalId of the STSA picture is equal to the TemporalId of the current picture and the nuh_layer_id is equal to the nuh_layer_id of the current picture, there shall be no picture that comes before the STSA picture in decoding order and whose TemporalId is equal to the TemporalId of the current picture and whose nuh_layer_id is equal to the nuh_layer_id of the current picture and that is included as an active entry in RefPicList[0] or RefPicList[1].

[0467]

[0468]

[0469] – When the current picture with nuh_layer_id equal to a specific value layerId is a CRA picture, there shall be no picture involved by an entry in RefPicList[0] or RefPicList[1] that comes before (in output order or in decoding order) any previous IRAP picture (if any) in decoding order with nuh_layer_id equal to layerId.

[0470]

[0471] – When the current picture with nuh_layer_id equal to a specific value layerId is not a RASL picture associated with a CRA picture with NoOutputBeforeRecoveryFlag equal to 1, a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, or a recovery picture of a GDR picture with NoOutputBeforeRecoveryFlag equal to 1 and nuh_layer_id equal to layerId, there shall be no picture involved by the active entries in RefPicList[0] or RefPicList[1] generated by a decoding process that generates unavailable reference pictures.

[0472]

[0473] – When the current picture with nuh_layer_id equal to a specific value layerId is not a CRA picture with NoOutputBeforeRecoveryFlag equal to 1, a picture before the pre-picture associated with the same CRA picture with NoOutputBeforeRecoveryFlag equal to 1 in decoding order, a pre-picture associated with a CRA picture with NoOutputBeforeRecoveryFlag equal to 1, a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, or a recovery picture of a GDR picture with NoOutputBeforeRecoveryFlag equal to 1 and nuh_layer_id equal to layerId, there shall be no picture involved by the entries in RefPicList[0] or RefPicList[1] generated by a decoding process that generates unavailable reference pictures.

[0474]

[0475] – When the current picture is associated with an IRAP picture and is after the IRAP picture in output order, there shall be no picture involved by the active entries in RefPicList[0] or RefPicList[1] before the associated IRAP picture in output order or in decoding order.

[0476]

[0477] – When the current picture is associated with an IRAP picture, it shall be after the IRAP picture in output order and after the preceding pictures associated with the same IRAP picture (if any) in both decoding order and output order. There shall be no picture referred to by an entry in RefPicList[0] or RefPicList[1] that is before the associated IRAP picture in either output order or decoding order.

[0478]

[0479] – When the current picture is a RADL picture, there shall be no

[0480] of any of the following active entries:

[0481] о RASL pictures

[0482] о pictures that are before the associated IRAP picture in decoding order

[0483]

[0484] – Each picture referred to by an ILRP entry in RefPicList[0] or RefPicList[1] of the slice of the current picture shall be in the same AU as the current picture.

[0485] – Each picture referred to by an ILRP entry in RefPicList[0] or RefPicList[1] of the slice of the current picture shall exist in the DPB, and the nuh_layer_id shall be less than the nuh_layer_id of the current picture.

[0486] – Each ILRP entry in RefPicList[0] or RefPicList[1] of the slice shall be an active entry.

[0487] Figure 5FIG. 0 is a block diagram showing an example video processing system 1900 in which the various techniques disclosed herein may be implemented. Various implementations may include some or all components of system 1900. System 1900 may include an input 1902 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8- or 10-bit multi-component pixel values), or may be in a compressed or encoded format. Input 1902 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, passive optical network (PON), etc., and wireless interfaces such as Wi-Fi or cellular interfaces.

[0488] System 1900 may include a codec component 1904, which may implement the various codec or encoding methods described in this document. The codec component 1904 may reduce the average bit rate of the video from the output of input 1902 to the output of the codec component 1904 to produce a codec representation of the video. Thus, codec techniques are sometimes referred to as video compression or video transcoding techniques. The output of the codec component 1904 may be stored or transmitted through communication via a connection represented by component 1906. Component 1908 may use the stored or communicated bitstream (or codec) representation of the video received at input 1902 to generate pixel values or a displayable video to be sent to a display interface 1910. The process of generating a user-visible video from the bitstream representation is sometimes referred to as video decompression. Additionally, although some video processing operations are referred to as "codec" operations or tools, it should be understood that encoding tools or operations are used at the encoder, and the corresponding decoding tools or operations to reverse the encoding results will be performed by the decoder.

[0489] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), or DisplayPort, etc. Examples of storage interfaces include Serial Advanced Technology Attachment (SATA), PCI, IDE interfaces, etc. The techniques described in this document may be implemented in various electronic devices, such as mobile phones, laptop computers, smartphones, or other devices capable of performing digital data processing and / or video display.

[0490] Figure 6is a block diagram of a video processing apparatus 3600. The apparatus 3600 can be used to implement one or more of the methods described herein. The apparatus 3600 can be implemented in a smart phone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. The apparatus 3600 can include one or more processors 3602, one or more memories 3604, and video processing hardware 3606. The (multiple) processors 3602 can be configured to implement one or more of the methods described in this document. The (multiple) memories 3604 can be used to store data and code for implementing the methods and techniques described herein. The video processing hardware 3606 can be used to implement some of the techniques described in this document in hardware circuitry.

[0491] Figure 8 is a block diagram showing an example video codec system 100 that can utilize the techniques of the present invention.

[0492] As Figure 8 shown, the video codec system 100 can include a source device 110 and a destination device 120. The source device 110 generates encoded video data that can be referred to as a video encoding device. The destination device 120 can decode the encoded video data generated by the source device 110, which can be referred to as a video decoding device.

[0493] The source device 110 can include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0494] The video source 112 can include sources such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of these sources. The video data can include one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a bitstream. The bitstream can include a sequence of bits that form a codec representation of the video data. The bitstream can include coded pictures and associated data. A coded picture is a codec representation of a picture. The associated data can include a sequence parameter set, a picture parameter set, and other syntax structures. The I / O interface 116 can include a modulator / demodulator (modem) and / or a transmitter. The encoded video data can be directly sent to the destination device 120 via the I / O interface 116 over a network 130a. The encoded video data can also be stored on a storage medium / server 130b for access by the destination device 120.

[0495] The destination device 120 can include an I / O interface 126, a video decoder 124, and a display device 122.

[0496] The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain the encoded video data from the source device 110 or the storage medium / server 130b. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to the user. The display device 122 may be integrated with the target device 120 or may be external to the target device 120, which is configured to connect to an external display device.

[0497] The video encoder 114 and the video decoder 124 may operate according to video compression standards, such as the High Efficiency Video Coding (HEVC) standard, the Versatile Video Coding (VVC) standard, and other current and / or future standards.

[0498] Figure 9 is a block diagram showing an example of a video encoder 200, and the video encoder 200 may be Figure 8 the video encoder 114 in the system 100 shown in

[0499] The video encoder 200 may be configured to perform any or all of the techniques of the present invention. In Figure 9 the example of, the video encoder 200 includes a plurality of functional components. The techniques described in the present invention may be shared among various components of the video encoder 200. In some examples, a processor may be configured to perform any or all of the techniques described in the present disclosure.

[0500] The functional components of the video encoder 200 may include a splitting unit 201, a prediction unit 202 including a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205, an intra prediction unit 206, a residual generation unit 207, a transformation unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transformation unit 211, a reconstruction unit 212, a buffer 213, and an entropy coding unit 214.

[0501] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra block copy (IBC) unit. The IBC unit may perform prediction in the IBC mode, where at least one reference picture is the picture in which the current video block is located.

[0502] In addition, some components (e.g., the motion estimation unit 204 and the motion compensation unit 205) may be highly integrated but are shown separately in Figure 9 the example of for explanatory purposes.

[0503] The splitting unit 201 can split an image into one or more video blocks. The video encoder 200 and the video decoder 300 can support various video block sizes.

[0504] The mode selection unit 203 can select one of the coding / decoding modes (intra or inter) (e.g., based on an error result), and provide the resulting intra or inter coded block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 203 can select a combination of intra and inter prediction (CIIP) modes, where the prediction is based on an inter prediction signal and an intra prediction signal. In the case of inter prediction, the mode selection unit 203 can also select the resolution of the motion vector for a block (e.g., sub-pixel or integer pixel accuracy).

[0505] To perform inter prediction on a current video block, the motion estimation unit 204 can generate motion information for the current video block by comparing one or more reference frames from the buffer 213 with the current video block. The motion compensation unit 205 can determine a predicted video block for the current video block based on the motion information of pictures other than the picture associated with the current video block from the buffer 213 and decoded samples.

[0506] The motion estimation unit 204 and the motion compensation unit 205 can perform different operations on the current video block, e.g., depending on whether the current video block is in an I-slice, a P-slice, or a B-slice.

[0507] In some examples, the motion estimation unit 204 can perform uni-directional prediction on the current video block, and the motion estimation unit 204 can search for a reference video block of the current video block in the reference pictures of list 0 or list 1. The motion estimation unit 204 can then generate a reference index indicating the reference picture in list 0 or list 1 that contains the reference video block and a motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 204 can output the reference index, a prediction direction indicator, and the motion vector as the motion information of the current video block. The motion compensation unit 205 can generate a predicted video block for the current block based on the reference video block indicated by the motion information of the current video block.

[0508] In other examples, the motion estimation unit 204 may perform bidirectional prediction on a current video block. The motion estimation unit 204 may search for a reference video block of the current video block in the reference pictures in list 0, and may also search for another reference video block of the current video block in the reference pictures in list 1. The motion estimation unit 204 may then generate a reference index and a motion vector. The reference index indicates the reference pictures in list 0 and list 1 that contain the reference video block, and the motion vector indicates the spatial displacement between the reference video block and the current video block. The motion estimation unit 204 may output the reference index and the motion vector of the current video block as the motion information of the current video block. The motion compensation unit 205 may generate a predicted video block of the current video block based on the reference video block indicated by the motion information of the current video block.

[0509] In some examples, the motion estimation unit 204 may output a set of complete motion information for the decoding process of the decoder.

[0510] In some examples, the motion estimation unit 204 may not output a set of complete motion information for the current video. Instead, the motion estimation unit 204 may signal the motion information of the current video block by referring to the motion information of another video block. For example, the motion estimation unit 204 may determine that the motion information of the current video block is similar enough to the motion information of an adjacent video block.

[0511] In one example, the motion estimation unit 204 may indicate a value in a syntax structure associated with the current video block. The value indicates to the video decoder 300 that the current video block has the same motion information as another video block.

[0512] In another example, the motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0513] As described above, the video encoder 200 may predictively signal motion vectors. Two examples of predictive signaling techniques that may be implemented by the video encoder 200 include advanced motion vector prediction (AMVP) and merge mode signaling.

[0514] The intra prediction unit 206 may perform intra prediction on the current video block. When the intra prediction unit 206 performs intra prediction on the current video block, the intra prediction unit 206 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a predicted video block and various syntax elements.

[0515] The residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., indicated by a negative sign) the predicted video block of the current video block from the current video block. The residual data for the current video block may include residual video blocks corresponding to different sample components of the samples in the current video block.

[0516] In other examples, the current video block may not have residual data for the current video block, such as in the skip mode, and the residual generation unit 207 may not perform the subtraction operation.

[0517] The transform processing unit 208 may generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0518] After the transform processing unit 208 generates the transform coefficient video block associated with the current video block, the quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values associated with the current video block.

[0519] The inverse quantization unit 210 and the inverse transform unit 211 may respectively apply inverse quantization and inverse transform to the transform coefficient video block to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 212 may add the reconstructed residual video block to the corresponding samples of one or more predicted video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current block for storage in the buffer 213.

[0520] After the reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce the block effect in the video block.

[0521] The entropy coding unit 214 may receive data from other functional components of the video encoder 200. When the entropy coding unit 214 receives data, the entropy coding unit 214 may perform one or more entropy coding operations to generate entropy-coded data and output a bitstream including the entropy-coded data.

[0522] Some embodiments of the disclosed technology include making a decision or determination to enable a video processing tool or mode. In an example, when a video processing tool or mode is enabled, the encoder will use or implement the tool or mode in the processing of video blocks, but may not necessarily modify the resulting bitstream based on the use of the tool or mode. That is, when a video processing tool or mode is enabled based on a decision or determination, the conversion from video blocks to a bitstream (or bitstream representation) of the video will use the video processing tool or mode. In another example, when a video processing tool or mode is enabled, the decoder will process the bitstream knowing that the bitstream has been modified based on the video processing tool or mode. That is, the conversion from the bitstream of the video to blocks of the video will be performed using the video processing tool or mode enabled based on the decision or determination.

[0523] Figure 10 is a block diagram showing an example of a video decoder 300, which may be Figure 8 the video decoder 114 in the system 100 shown.

[0524] The video decoder 300 may be configured to perform any or all of the techniques of the present invention. In Figure 10 an example, the video decoder 300 includes a plurality of functional components. The techniques described in this disclosure may be shared among the various components of the video decoder 300. In some examples, a processor may be configured to perform any or all of the techniques described in this disclosure.

[0525] In Figure 10 an example, the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, a reconstruction unit 306, and a buffer 307. In some examples, the video decoder 300 may perform a decoding pass generally opposite to the encoding process described for the video encoder 200 ( Figure 9 ).

[0526] The entropy decoding unit 301 may retrieve the encoded bitstream. The encoded bitstream may include entropy-encoded video data (e.g., encoded video data blocks). The entropy decoding unit 301 may decode the entropy-encoded video data, and from the entropy-decoded video data, the motion compensation unit 302 may determine motion information, including motion vectors, motion vector precision, reference picture list indices, and other motion information. The motion compensation unit 302 may determine such information (e.g.) by performing AMVP and merge mode.

[0527] The motion compensation unit 302 may produce motion-compensated blocks, possibly performing interpolation based on an interpolation filter. An identifier of the interpolation filter to be used with sub-pixel precision may be included in the syntax elements.

[0528] The motion compensation unit 302 may use an interpolation filter such as that used by the video encoder 200 during the encoding of video blocks to calculate the interpolation of sub-integer pixels of a reference block. The motion compensation unit 302 may determine the interpolation filter used by the video encoder 200 according to the received syntax information, and use the interpolation filter to generate a prediction block.

[0529] The motion compensation unit 302 may use some syntax information to determine the size of the blocks for encoding the (multiple) frames and / or (multiple) slices of an encoded video sequence, the partitioning information describing how each macroblock of a picture of the encoded video sequence is partitioned, the mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-frame encoded block, and other information for decoding the encoded video sequence.

[0530] The intra prediction unit 303 may form a prediction block from spatially adjacent blocks using, for example, an intra prediction mode received in the bitstream. The inverse quantization unit 303 inverse quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 303 applies an inverse transform.

[0531] The reconstruction unit 306 may add a residual block to the corresponding prediction block generated by the motion compensation unit 202 or the intra prediction unit 303 to form a decoded block. If necessary, a deblocking filter may also be applied to filter the decoded block in order to remove blocking artifacts. The decoded video block is then stored in the buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction and also generates the decoded video for presentation on a display device.

[0532] Next, a list of technical solutions preferred by some embodiments is provided.

[0533] The following technical solutions illustrate example embodiments of the techniques discussed in the previous section (e.g., item 1).

[0534] 1. A video processing method (e.g., Figure 7 the method 700 shown in), comprising: performing (702) a conversion between a video including one or more pictures including one or more sub-pictures and an encoded and decoded representation of the video, wherein the encoded and decoded representation format rules, the format rules stipulating that one or more pictures including one or more sub-pictures are included in the encoded and decoded representation according to network abstraction layer (NAL) units, and wherein one type of NAL unit indicated in the encoded and decoded representation includes an encoded slice of a specific type picture or an encoded slice of a specific type sub-picture.

[0535] The following technical solutions illustrate example embodiments of the techniques discussed in the previous section (e.g., item 2).

[0536] 2. A video processing method, comprising: performing a conversion between a video including one or more pictures each including one or more sub - pictures and a codec representation of the video, wherein the codec representation conforms to format rules that specify that two adjacent sub - pictures having different network abstraction layer unit types will have the same indication of the sub - picture regarded as a picture flag.

[0537] The following technical solutions show example embodiments of the technologies discussed in the previous sections (e.g., items 4, 5, 6, 7, 9, 1, 11, 12).

[0538] 3. A video processing method, comprising: performing a conversion between a video including one or more pictures each including one or more sub - pictures and a codec representation of the video, wherein the codec representation conforms to format rules that define an order of a first type of sub - picture and a second type of sub - picture, wherein the first sub - picture is a post - sub - picture or a pre - sub - picture or a random access skipped - pre (RASL) sub - picture type, and the second sub - picture is a RASL type or a random access decodable - pre (RADL) type or an instantly decodable refresh (IDR) type or a gradually decodable refresh (GDR) type sub - picture.

[0539] 4. The method according to technical solution 3, wherein the rule specifies that, in the output order, the post - sub - picture is after the random access point or the GDR sub - picture within the associated frame.

[0540] 5. The method according to technical solution 3, wherein the rule specifies that when the picture is a pre - picture of an intra - frame random access point picture, the nal_unit_type value of all network abstraction layer units in the picture is equal to RADL_NUT or RASL_NUT.

[0541] 6. The method according to technical solution 3, wherein the rule specifies that a given sub - picture that is a pre - sub - picture of an IRAP sub - picture must also be a RADL or a RASL sub - picture.

[0542] 7. The method according to technical solution 3, wherein the rule specifies that a given sub - picture that is a RASL sub - picture is not allowed to be associated with an IDR sub - picture.

[0543] 8. The method according to technical solution 3, wherein the rule specifies that a given sub - picture having the same layer id and sub - picture index as an IRAP sub - picture must be before the IRAP sub - picture and all its associated RADL sub - pictures in the output order.

[0544] 9. The method according to Technical Solution 3, wherein the rule stipulates that a given sub-picture having the same layer id and sub-picture index as the GDR sub-picture must be before the GDR sub-picture and all its associated RADL sub-pictures in the output order.

[0545] 10. The method according to Technical Solution 3, wherein the rule stipulates that a given sub-picture that is a RASL sub-picture associated with a CRA sub-picture is before all RADL sub-pictures associated with the CRA sub-picture in the output order.

[0546] 11. The method according to Technical Solution 3, wherein the rule stipulates that a given sub-picture that is a RASL sub-picture associated with a CRA sub-picture is before all IRAP sub-pictures associated with the CRA sub-picture in the output order.

[0547] 12. The method according to Technical Solution 3, wherein the rule stipulates that if a given sub-picture is a pre-sub-picture of an IRAP sub-picture, then the given sub-picture is before all non-pre-sub-pictures associated with the IRAP picture in the decoding order.

[0548] The following technical solutions illustrate example embodiments of the techniques discussed in the previous sections (e.g., items 8, 14, 15).

[0549] 13. A video processing method, comprising: performing a conversion between a video including one or more pictures each including one or more sub-pictures and a codec representation of the video, wherein the codec representation conforms to format rules that define conditions for allowing or not allowing a first type of sub-picture to coexist with a second type of sub-picture.

[0550] 14. The method according to Technical Solution 13, wherein the rule stipulates that if there is an IDR sub-picture of network abstraction layer type IDR_N_LP, the codec representation is not allowed to have a RADP sub-picture.

[0551] 15. The method according to Technical Solution 13, wherein the rule does not allow a picture to be included in a reference list of a picture including a stepwise temporal sub-layer access (STSA) sub-picture such that the picture is before the picture including the STSA sub-picture.

[0552] 16. The method according to Technical Solution 13, wherein the rule does not allow a picture to be included in a reference list of a picture including an intra random access point (IRAP) sub-picture such that the picture is before the picture including the IRAP sub-picture.

[0553] 17. The method according to any one of Technical Solutions 1 to 16, wherein the conversion includes encoding the video into a codec representation.

[0554] 18. The method according to any one of technical solutions 1 to 16, wherein the conversion includes decoding the codec representation to generate pixel values of the video.

[0555] 19. A video decoding device, including a processor configured to implement the method described in one or more of technical solutions 1 to 18.

[0556] 20. A video encoding device, including a processor configured to implement the method described in one or more of technical solutions 1 to 18.

[0557] 21. A computer program product having computer code stored thereon, which when run by a processor causes the processor to implement the method described in any one of technical solutions 1 to 18.

[0558] 22. The method, device or system described in this document.

[0559] In the technical solutions described herein, the encoder can conform to the format rules by generating a codec representation according to the format rules. In the technical solutions described herein, the decoder can use the format rules to parse the syntax elements in the codec representation, and understand the presence and absence of the syntax elements according to the format rules to generate the decoded video.

[0560] Figure 11 is a flowchart of an example method 1100 for video processing. Operation 1102 includes performing a conversion between a video including one or more pictures including one or more sub-pictures and a bitstream of the video, wherein the bitstream conforms to format rules that govern the syntax of network abstraction layer (NAL) units in the specified bitstream, and wherein the format rules specify that NAL units of video coding layer (VCL) NAL unit types include content associated with a particular type of picture or a particular type of sub-picture.

[0561] In some embodiments of method 1100, the content of the NAL unit of VCL NAL unit type indicates that the coded strip is associated with a pure random access picture or a pure random access sub-picture. In some embodiments of method 1100, the pure random access sub-picture is an intra random access point sub-picture where each VCL NAL unit has a pure random access type. In some embodiments of method 1100, the content of the NAL unit of VCL NAL unit type indicates that the coded strip is associated with an associated progressive decoding refresh picture or an associated progressive decoding refresh sub-picture. In some embodiments of method 1100, the associated progressive decoding refresh sub-picture is, in decoding order, a previous progressive decoding refresh sub-picture having an identifier of the layer to which the VCL NAL unit belongs or the layer to which the non-VCL NAL unit is applied equal to a first specific value and a second specific value of a sub-picture index, and wherein, in decoding order, between the previous progressive decoding refresh sub-picture and a specific sub-picture having the first specific value of the identifier and the second specific value of the sub-picture index, there is no intra random access point sub-picture having the first specific value of the identifier and the second specific value of the sub-picture index.

[0562] In some embodiments of method 1100, the content of the NAL unit of VCL NAL unit type indicates that the coded strip is associated with an associated intra random access point picture or an associated intra random access point sub-picture. In some embodiments of method 1100, the associated intra random access point sub-picture is, in decoding order, a previous intra random access point sub-picture having an identifier of the layer to which the VCL NAL unit belongs or the layer to which the non-VCL NAL unit is applied equal to a first specific value and a second specific value of a sub-picture index, and wherein, in decoding order, between the previous intra random access point sub-picture and a specific sub-picture having the first specific value of the identifier and the second specific value of the sub-picture index, there is no progressive decoding refresh sub-picture having the first specific value of the identifier and the second specific value of the sub-picture index. In some embodiments of method 1100, the content of the NAL unit of VCL NAL unit type indicates that the coded strip is associated with an instantly decoded refresh picture or an instantly decoded refresh sub-picture. In some embodiments of method 1100, the instantly decoded refresh sub-picture is an intra random access point sub-picture where each VCL NAL unit has an instantly decoded refresh type.

[0563] In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded slice is associated with a previous picture or a previous sub - picture. In some embodiments of method 1100, a previous sub - picture is a sub - picture that is before the intra - random - access - point sub - picture within the associated frame in output order. In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded slice is associated with a random - access - decodable previous picture or a random - access - decodable previous sub - picture. In some embodiments of method 1100, a random - access - decodable previous sub - picture is a sub - picture where each VCL NAL unit has a random - access - decodable - previous type. In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded slice is associated with a random - access - skipped previous picture or a random - access - skipped previous sub - picture. In some embodiments of method 1100, a random - access - skipped previous sub - picture is a sub - picture where each VCL NAL unit has a random - access - skipped - previous type. In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded slice is associated with a progressive - temporal - layer - access picture or a progressive - temporal - layer - access sub - picture.

[0564] In some embodiments of method 1100, a progressive - temporal - layer - access sub - picture is a sub - picture where each VCL NAL unit has a progressive - temporal - layer - access type. In some embodiments of method 1100, the content of a NAL unit of VCL NAL unit type indicates that the coded slice is associated with a subsequent picture or a subsequent sub - picture. In some embodiments of method 1100, a subsequent sub - picture is a sub - picture where each VCL NAL unit has a subsequent type.

[0565] Figure 12 is a flow chart of an example method 1200 for video processing. Operation 1202 includes performing a conversion between a video including a picture that includes sub - pictures and a bit - stream of the video, where the bit - stream conforms to format rules, and where the format rules specify that, in response to the sub - picture being a previous sub - picture of an intra - random - access - point sub - picture, the sub - picture is a random - access type of sub - picture.

[0566] In some embodiments of method 1200, the random - access type of the sub - picture is a random - access - decodable previous sub - picture. In some embodiments of method 1200, the random - access type of the sub - picture is a random - access - skipped previous sub - picture.

[0567] Figure 13It is a flowchart of an example method 1300 for video processing. Operation 1302 includes performing a conversion between a video including a picture that includes sub-pictures and a bitstream of the video, where the bitstream conforms to format rules, and where the format rules specify that in response to one or more random access skip leading sub-pictures being associated with an instant decoding refresh sub-picture, one or more random access skip leading sub-pictures do not exist in the bitstream.

[0568] Figure 14 It is a flowchart of an example method 1400 for video processing. Operation 1402 includes performing a conversion between a video including a picture that includes sub-pictures and a bitstream of the video, where the bitstream conforms to format rules that specify that in response to one or more random access decodable leading sub-pictures being associated with an instant decoding refresh sub-picture having a type of network abstraction layer (NAL) unit indicating that the instant decoding refresh sub-picture is not associated with a leading picture, one or more random access decodable leading sub-pictures do not exist in the bitstream.

[0569] Figure 15 It is a flowchart of an example method 1500 for video processing. Operation 1502 includes performing a conversion between a video including a picture that includes two adjacent sub-pictures and a bitstream of the video, where the bitstream conforms to format rules that specify that two adjacent sub-pictures having different types of network abstraction layer (NAL) units have a syntax element with the same first value, the syntax element indicating whether each of the two adjacent sub-pictures in a coded layer video sequence is considered a picture.

[0570] In some embodiments of method 1500, the format rules specify that the syntax element of two adjacent sub-pictures indicates that each of the two adjacent sub-pictures in a coded layer video sequence is considered a picture.

[0571] Figure 16 It is a flowchart of an example method 1600 for video processing. Operation 1602 includes performing a conversion between a video including a picture that includes two adjacent sub-pictures and a bitstream of the video, where the format rules specify that the two adjacent sub-pictures include a first adjacent sub-picture having a first sub-picture index and a second adjacent sub-picture having a second sub-picture index, and where the format rules specify that in response to a first syntax element associated with the first sub-picture index indicating that the first adjacent sub-picture is not considered a picture or a second syntax element associated with the second sub-picture index indicating that the second adjacent sub-picture is not considered a picture, the two adjacent sub-pictures have the same type of network abstraction layer (NAL) unit.

[0572] In some embodiments of method 1600, a picture includes a plurality of sub - pictures including two adjacent sub - pictures, and wherein the formatting rule specifies that in response to a sub - picture from the plurality of sub - pictures having a syntax element indicating that the sub - picture is not considered a picture, the plurality of sub - pictures have the same type of NAL unit. In some embodiments of method 1600, a picture includes a plurality of sub - pictures including two adjacent sub - pictures, and wherein the formatting rule specifies that in response to the plurality of sub - pictures having a corresponding syntax element indicating that each of the plurality of sub - pictures in the CLVS is considered a picture, the syntax element indicates that each picture of the video with a reference picture parameter set (PPS) has a plurality of VCL NAL units that do not have the same type of video coding layer (VCL) NAL unit.

[0573] Figure 17 is a flowchart of an example method 1700 for video processing. Operation 1702 includes performing a conversion between a video including a picture that includes one or more sub - pictures and a bit - stream of the video, wherein the bit - stream conforms to a formatting rule that specifies that in response to a syntax element indicating that each picture of the video with a reference picture parameter set (PPS) has a plurality of VCL NAL units that do not have the same type of video coding layer (VCL) network abstraction layer (NAL) unit, allowing the picture to include more than two different types of VCL NAL units.

[0574] Figure 18 is a flowchart of an example method 1800 for video processing. Operation 1802 includes performing a conversion between a video including one or more pictures that include one or more sub - pictures and a bit - stream of the video, wherein the bit - stream conforms to a formatting rule that specifies that a post - sub - picture associated in sequence with an intra - random access point sub - picture or a progressive decoding refresh sub - picture is after the intra - random access point sub - picture or the progressive decoding refresh sub - picture.

[0575] In some embodiments of method 1800, the sequence is the output sequence.

[0576] Figure 19It is a flowchart of an example method 1900 for video processing. Operation 1902 includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, where the bitstream conforms to a format rule that specifies that, in response to: (1) the second-order sub-picture being before the intra-random access point sub-picture in a frame, (2) the sub-picture and the intra-random access point sub-picture having the same first value for the layer to which the network abstraction layer (NAL) unit of the sub-picture and the intra-random access point sub-picture belongs, and (3) the sub-picture and the intra-random access point sub-picture having the same second value of the sub-picture index, the sub-picture is before the intra-random access point sub-picture and one or more random access decodable pre-sub-pictures associated with the intra-random access point sub-picture in a first order.

[0577] In some embodiments of method 1800, the first order is the output order. In some embodiments of method 1800, the second order is the decoding order.

[0578] Figure 20 It is a flowchart of an example method 2000 for video processing. Operation 2002 includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, where the bitstream conforms to a format rule that specifies that a random access skip pre-sub-picture associated with a pure random access sub-picture is before one or more random access decodable pre-sub-pictures associated with the pure random access sub-picture in an order.

[0579] In some embodiments of method 2000, the order is the output order.

[0580] Figure 21 It is a flowchart of an example method 2100 for video processing. Operation 2102 includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, where the bitstream conforms to a format rule that specifies that a random access skip pre-sub-picture associated with a pure random access sub-picture is after one or more intra-random access point sub-pictures before the pure random access sub-picture in a second order in a first order.

[0581] In some embodiments of method 2100, the first order is the output order. In some embodiments of method 2100, the second order is the decoding order.

[0582] Figure 22is a flowchart of an example method 2200 for video processing. Operation 2202 includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, where the bitstream conforms to a format rule that specifies that, in response to: (1) a syntax element indicating that a coded video sequence conveys a picture representing a frame, and (2) the current sub-picture being a pre-sub-picture associated with an intra-random access point sub-picture, the current sub-picture is before one or more non-pre-sub-pictures associated with the intra-random access point sub-picture in decoding order.

[0583] In some embodiments of method 2200, in response to: (1) a syntax element indicating that a coded video sequence conveys a picture representing a field, and (2) the current sub-picture not being a pre-sub-picture, the format rule specifies that there is at most one non-pre-sub-picture before a first pre-sub-picture associated with an intra-random access point sub-picture in decoding order, and there is no non-pre-picture between the first pre-sub-picture and the last pre-sub-picture associated with the intra-random access point sub-picture in decoding order, where the current sub-picture, the at most one non-pre-sub-picture, and the non-pre-picture have the same first value for a layer to which network abstraction layer (NAL) units of the current sub-picture, the at most one non-pre-sub-picture, and the non-pre-picture belong, and where the current sub-picture, the at most one non-pre-sub-picture, and the non-pre-picture have the same second value of a sub-picture index.

[0584] Figure 23 is a flowchart of an example method 2300 for video processing. Operation 2302 includes performing a conversion between a video including one or more pictures each including one or more sub-pictures and a bitstream of the video, where the bitstream conforms to a format rule that specifies that, in response to a picture being a pre-picture of an intra-random access point picture, one or more types of NAL units of all video coding layer (VCL) network abstraction layer (NAL) units in the picture include RADL_NUT or RASL_NUT.

[0585] In some embodiments of method 2300, the format rule specifies that, in response to: (1) one or more types of NAL units of all VCL NAL units in a picture include RADL_NUT and RASL_NUT, and (2) the layer including the picture is an output layer, a variable of the picture is set to be equal to a value of a picture output flag.

[0586] Figure 24It is a flowchart of an example method 2400 for video processing. Operation 2402 includes performing a conversion between a video including one or more pictures each including a plurality of sub-pictures and a bitstream of the video, where the bitstream complies with format rules that specify that, in response to: (1) at least one sub-picture being before a gradually decoded refresh sub-picture in a second order, (2) the at least one sub-picture and the gradually decoded refresh sub-picture having the same first value for a layer to which NAL units of the at least one sub-picture and the gradually decoded refresh sub-picture belong, and (3) the at least one sub-picture and the gradually decoded refresh picture having the same second value of sub-picture index, the at least one sub-picture is before the gradually decoded refresh sub-picture and one or more sub-pictures associated with the gradually decoded refresh sub-picture in a first order.

[0587] In some methods of embodiment 2400, the first order is the output order. In some methods of embodiment 2400, the second order is the decoding order.

[0588] Figure 25 It is a flowchart of an example method 2500 for video processing. Operation 2502 includes performing a conversion between a video including a current picture including a current sub-picture, the current sub-picture including a current strip, and a bitstream of the video, where the bitstream complies with format rules in response to: (a) a first picture having the same temporal identifier as the current sub-picture and the same layer identifier of a network abstraction layer (NAL) unit, and (b) the current sub-picture being after a gradually temporal sub-layer access sub-picture in decoding order, and (c) the current sub-picture and the gradually temporal sub-layer access sub-picture having the same temporal identifier, the same layer identifier, and the same sub-picture index, and the format rules do not allow an active entry in a reference picture list of the current strip to include the first picture before a second picture in decoding order, the second picture including the gradually temporal sub-layer access sub-picture.

[0589] In some methods of embodiment 2500, the reference picture list includes a list 0 reference picture list. In some methods of embodiment 2500, the reference picture list includes a list 1 reference picture list.

[0590] Figure 26 It is a flowchart of an example method 2600 for video processing. Operation 2602 includes performing a conversion between a video including a current picture including a current sub-picture, the current sub-picture including a current strip, and a bitstream of the video, where the bitstream complies with format rules in response to the current sub-picture not being a sub-picture of a specific type, and the format rules do not allow an active entry in a reference picture list of the current strip to include a first picture generated by a decoding process that generates an unavailable reference picture.

[0591] In some embodiments of method 2600, the current sub-picture is not a random access skip pre-sub-picture associated with a pure random access sub-picture of a pure random access picture, and the pure random access sub-picture has a flag value equal to 1 indicating no output before recovery. In some embodiments of method 2600, the current sub-picture is not a progressive decoding refresh sub-picture of a progressive decoding refresh picture, and the progressive decoding refresh sub-picture has a flag value equal to 1 indicating no output before recovery. In some embodiments of method 2600, the current sub-picture is not a sub-picture of a recovery picture of a progressive decoding refresh picture, and the sub-picture of the recovery picture of the progressive decoding refresh picture has a flag value equal to 1 indicating no output before recovery and has the same layer identifier of the network abstraction layer (NAL) unit as the current sub-picture. In some embodiments of method 2600, the reference picture list includes the list 0 reference picture list. In some embodiments of method 2600, the reference picture list includes the list 1 reference picture list.

[0592] Figure 27 is a flowchart of an example method 2700 for video processing. Operation 2702 includes performing a conversion between a video including a current picture that includes a current sub-picture and a bitstream of the video, where the current sub-picture includes a current slice, and where the bitstream conforms to format rules that, in response to the current sub-picture not being a particular type of sub-picture, do not allow an entry in the reference picture list of the current slice to include a first picture generated by a decoding process that generates an unavailable reference picture.

[0593] In some embodiments of method 2700, the current picture is not a pure random access sub-picture of a pure random access picture, and the pure random access sub-picture has a flag value equal to 1 indicating no output before recovery. In some embodiments of method 2700, the current sub-picture is not a sub-picture before one or more pre-sub-pictures associated with a pure random access sub-picture of a pure random access picture in decoding order, and the one or more pre-sub-pictures before the sub-picture have a flag value equal to 1 indicating no output before recovery. In some embodiments of method 2700, the current sub-picture is not a pre-sub-picture associated with a pure random access sub-picture of a pure random access picture, and the pre-sub-picture has a flag value equal to 1 indicating no output before recovery. In some embodiments of method 2700, the current sub-picture is not a progressive decoding refresh sub-picture of a progressive decoding refresh picture, and the progressive decoding refresh sub-picture has a flag value equal to 1 indicating no output before recovery.

[0594] In some embodiments of method 2700, the current sub-picture is not a sub-picture of the recovered picture of the gradually decoded and refreshed picture, the sub-picture having a flag value indicating no output before recovery equal to 1 and having the same layer identifier of the network abstraction layer (NAL) unit as the current sub-picture. In some embodiments of method 2700, the reference picture list includes the list 0 reference picture list. In some embodiments of method 2700, the reference picture list includes the list 1 reference picture list.

[0595] Figure 28 is a flowchart of an example method 2800 for video processing. Operation 2802 includes performing a conversion between a video including a current picture that includes a current sub-picture and a bitstream of the video, the current sub-picture including a current slice, wherein the bitstream complies with format rules, in response to (a) a first picture including a previous intra-random access point sub-picture, the previous intra-random access point sub-picture being before the current sub-picture in a second order, (b) the previous intra-random access point sub-picture having the same layer identifier of the network abstraction layer (NAL) unit and the same sub-picture index as the current sub-picture, and (c) the current sub-picture being a pure random access sub-picture, the format rules not allowing an entry in the reference picture list of the current slice to include the first picture before the current picture in a first order or a second order.

[0596] In some embodiments of method 2800, the first order includes the output order. In some embodiments of method 2800, the second order includes the decoding order. In some embodiments of method 2800, the reference picture list includes the list 0 reference picture list. In some embodiments of method 2800, the reference picture list includes the list 1 reference picture list.

[0597] Figure 29 is a flowchart of an example method 2900 for video processing. Operation 2902 includes performing a conversion between a video including a current picture that includes a current sub-picture and a bitstream of the video, the current sub-picture including a current slice, wherein the bitstream complies with format rules, in response to (a) the current sub-picture being associated with an intra-random access point sub-picture, and (b) the current sub-picture being after the intra-random access point sub-picture in a first order, the format rules not allowing an active entry in the reference picture list of the current slice to include the first picture before the current picture in a first order or a second order.

[0598] In some embodiments of method 2900, the first order includes the output order. In some embodiments of method 2900, the second order includes the decoding order. In some embodiments of method 2900, the reference picture list includes the list 0 reference picture list. In some embodiments of method 2900, the reference picture list includes the list 1 reference picture list.

[0599] Figure 30 is a flowchart of an example method 3000 for video processing. Operation 3002 includes performing a conversion between a video including a current picture that includes a current sub-picture and a bitstream of the video, the current sub-picture including a current slice, wherein the bitstream conforms to format rules, in response to: (a) the current sub-picture being after an intra-random access point sub-picture in a first order, (b) the current sub-picture being after one or more preceding sub-pictures associated with an IRAP sub-picture in a first order and a second order, the format rules not allowing an entry in a reference picture list of the current slice to include a first picture that is before a current picture including an intra-random access point sub-picture associated with the current sub-picture in the first order or the second order.

[0600] In some embodiments of method 3000, the first order includes an output order. In some embodiments of method 3000, the second order includes a decoding order. In some embodiments of method 3000, the reference picture list includes a list 0 reference picture list. In some embodiments of method 3000, the reference picture list includes a list 1 reference picture list.

[0601] Figure 31 is a flowchart of an example method 3100 for video processing. Operation 3102 includes performing a conversion between a video including a current picture that includes a current sub-picture and a bitstream of the video, the current sub-picture including a current slice, wherein the bitstream conforms to format rules, the format rules specifying that in response to the current sub-picture being a random access decodable preceding sub-picture, the reference picture list of the current slice does not include any active entry of any one or more of: a first picture including a random access skipped preceding sub-picture and a second picture that is before a third picture including an associated intra-random access point sub-picture in a decoding order.

[0602] In some embodiments of method 3100, the reference picture list includes a list 0 reference picture list. In some embodiments of method 3100, the reference picture list includes a list 1 reference picture list.

[0603] In this document, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during the conversion from the pixel representation of a video to the corresponding bitstream representation, and vice versa. For example, the bitstream representation of the current video block may correspond to bits co-located or propagated at different positions in the bitstream defined by the syntax. For example, a macroblock may be encoded based on the transformed and coded / decoded error residual values and also using bits in the headers and other fields in the bitstream. Further, during the conversion, the decoder may parse the bitstream based on this determination, knowing the possible presence or absence of some fields, as described in the above technical solution. Similarly, the encoder may determine to include or not include certain syntax fields and generate the coded / decoded representation accordingly by including or not including the syntax fields from the coded / decoded representation.

[0604] The disclosed and other technical solutions, examples, embodiments, modules, and functional operations described herein may be implemented in digital electronic circuitry or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in combinations of one or more of them. The disclosed embodiments and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by, or to control the operation of, a data processing apparatus. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances affecting a machine-readable propagated signal, or one or more of them. The term "data processing apparatus" includes all apparatus, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver apparatus.

[0605] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. The program can be stored in a part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language file), in a single file dedicated to the program, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program can be deployed on one or more computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0606] The processes and logical flows described in this specification can be performed by one or more programmable processors that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logical flows can also be performed by special-purpose logic circuitry, and the apparatus can also be implemented as special-purpose logic circuitry, e.g., an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0607] For example, processors suitable for the execution of a computer program include both general and special purpose microprocessors, and any one or more of any type of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing the instructions and one or more storage devices for storing the instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, e.g., magnetic disks, magneto-optical disks, or optical disks, or be operatively coupled to one or more mass storage devices to receive data therefrom or transfer data thereto, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable hard disks; magneto-optical disks; and CD ROM and DVD ROM disks. The processor and the memory can be supplemented by, or incorporated in, special-purpose logic circuitry.

[0608] Although this patent document contains many details, it should not be construed as limiting any subject matter or the scope of any claims, but rather as a description of the features of particular embodiments of a particular technology. Certain features described in the context of separate embodiments of this patent document may also be implemented in combination in a single embodiment. Conversely, the various functions described in the context of a single embodiment may also be implemented separately in multiple embodiments, or in any suitable sub-combination. Additionally, although the above features may be described as acting in certain combinations, and even initially claimed as such, in some cases, one or more features from a claimed combination may be removed from the combination, and the claimed combination may be directed to a sub-combination or a variant of a sub-combination.

[0609] Likewise, although operations are depicted in the drawings in a particular order, this should not be construed as requiring that such operations be performed in the particular order shown or in sequential order to achieve the desired result, or that all illustrated operations be performed. Additionally, the separation of various system components in the embodiments of this patent document should not be construed as required in all embodiments.

[0610] Only some implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and illustrated in this patent document.

Claims

1. A video processing method, comprising: Performing a conversion between a video including a current picture containing a current sub - picture and a bit - stream of the video, wherein the current sub - picture includes a current slice, wherein the bit - stream conforms to a first format rule, and in the following cases, the first format rule does not allow an active entry in the reference picture list of the current slice to include a first picture that is before a second picture in decoding order, and the second picture includes a progressive temporal sub - layer access sub - picture: (a) The first picture has the same temporal identifier as the current sub - picture and the same layer identifier of the network abstraction layer (NAL) unit, and (b) In decoding order, the current sub - picture is after the progressive temporal sub - layer access sub - picture, and (c) The current sub - picture and the progressive temporal sub - layer access sub - picture have the same temporal identifier, the same layer identifier, and the same sub - picture index.

2. The method according to claim 1, wherein, The reference picture list includes a list 0 reference picture list.

3. The method according to claim 1, wherein, The reference picture list includes a list 1 reference picture list.

4. The method according to claim 1, wherein, The bit - stream also conforms to a second format rule, and in the following cases, the second format rule does not allow an active entry in the reference picture list of the current slice to include a third picture generated by a decoding process that generates an unavailable reference picture for a pure - random - access picture containing a pure - random - access sub - picture associated with the current sub - picture: The current sub - picture is not a random - access skip - leading sub - picture associated with the pure - random - access sub - picture in the pure - random - access picture, and the pure - random - access sub - picture has a flag value indicating no output before recovery equal to 1.

5. The method according to claim 1, wherein, The bit - stream also conforms to a third format rule, and when a condition group related to the current picture is satisfied, the third format rule does not allow an active entry in the reference picture list of the current slice to include a fourth picture, wherein the fourth picture is generated by a decoding process that generates an unavailable reference picture for an intra - random - access - point picture containing an intra - random - access - point sub - picture associated with the current sub - picture, or for a gradually - decoded - refresh picture containing a gradually - decoded - refresh sub - picture associated with the current sub - picture.

6. The method according to claim 5, wherein, The condition group includes: The current picture is not a pure - random - access sub - picture in a pure - random - access picture, and the pure - random - access sub - picture has a flag value indicating no output before recovery equal to 1.

7. The method according to claim 5, wherein The condition group further includes: The current sub - picture is not a sub - picture associated with a pure - random - access sub - picture of a pure - random - access picture, and the pure - random - access sub - picture has a flag value indicating no output before recovery equal to 1, wherein in decoding order, the pure - random - access picture is before one or more leading pictures associated with the pure - random - access picture.

8. The method according to claim 5, wherein The condition group further includes: The current sub - picture is not a leading sub - picture associated with a pure - random - access sub - picture in a pure - random - access picture, and the pure - random - access sub - picture has a flag value indicating no output before recovery equal to 1.

9. The method according to claim 5, wherein The condition group further includes: the current sub-picture is not a gradually decoded and refreshed sub-picture in the gradually decoded and refreshed picture, and the gradually decoded and refreshed sub-picture has a flag value equal to 1 indicating no output before recovery.

10. The method according to claim 5, wherein, The condition group further includes: the current sub-picture is not a sub-picture of the recovery picture in the gradually decoded and refreshed picture, and the sub-picture of the recovery picture in the gradually decoded and refreshed picture has a flag value equal to 1 indicating no output before recovery and has the same layer identifier of the NAL unit as the current sub-picture.

11. The method according to claim 1, wherein, The bitstream also conforms to a fourth format rule, and the fourth format rule does not allow an active entry in the reference picture list of the current slice to include a fifth picture when the current sub-picture is after the intra-random access point sub-picture in the first order and the second order, the fifth picture is before a sixth picture including the intra-random access point sub-picture in the first order and the second order, and wherein, the intra-random access point sub-picture has the same layer identifier and the same sub-picture index as the current sub-picture.

12. The method according to claim 1, wherein, The conversion includes encoding the video into the bitstream.

13. The method according to claim 1, wherein, The conversion includes decoding the video from the bitstream.

14. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein, When executed by the processor, the instructions cause the processor to: perform a conversion between a video including a current picture including a current sub-picture and a bitstream of the video, wherein the current sub-picture includes a current slice, wherein, the bitstream conforms to a first format rule, and in the following cases, the first format rule does not allow an active entry in the reference picture list of the current slice to include a first picture before a second picture in the decoding order, and the second picture includes a progressive temporal sub-layer access sub-picture: (a) The first picture has the same temporal identifier and the same layer identifier of the network abstraction layer (NAL) unit as the current sub-picture, and (b) in the decoding order, the current sub-picture is after the progressive temporal sub-layer access sub-picture, and (c) the current sub-picture and the progressive temporal sub-layer access sub-picture have the same temporal identifier, the same layer identifier and the same sub-picture index.

15. The device according to claim 14, wherein, The reference picture list includes a list 0 reference picture list.

16. The apparatus according to claim 14, wherein, The reference picture list includes a list 1 reference picture list.

17. The device according to claim 14, wherein The bitstream also conforms to a second format rule, and in the following cases, the second format rule does not allow an active entry in the reference picture list of the current slice to include a third picture generated by a decoding process that generates an unavailable reference picture for a pure random access picture including a pure random access sub-picture associated with the current sub-picture: The current sub-picture is not a random access skip pre-sub-picture associated with the pure random access sub-picture in the pure random access picture, and the pure random access sub-picture has a flag value equal to 1 indicating no output before recovery.

18. The apparatus according to claim 14, wherein, The bitstream also conforms to a third format rule, and when a condition group related to the current picture is satisfied, the third format rule does not allow an active entry in the reference picture list of the current slice to include a fourth picture, Among them, the fourth picture is generated by a decoding process, and the decoding process is an intra random access point picture including an intra random access point sub-picture associated with the current sub-picture, or a progressive decoding refresh picture including a progressive decoding refresh sub-picture associated with the current sub-picture, generating an unavailable reference picture.

19. A non-transitory computer-readable storage medium storing instructions, the instructions causing a processor to: Perform a conversion between a video including a current picture containing a current sub-picture and a bitstream of the video, wherein, The current sub-picture includes a current strip, Among them, the bitstream conforms to a first format rule, and in the following cases, the first format rule does not allow an active entry in the reference picture list of the current strip to include a first picture that is before a second picture in decoding order, and the second picture includes a progressive temporal sub-layer access sub-picture: (a) The first picture has the same temporal identifier as the current sub-picture and the same layer identifier of the network abstraction layer (NAL) unit, and (b) The current sub-picture is after the progressive temporal sub-layer access sub-picture in the decoding order, and (c) The current sub-picture and the progressive temporal sub-layer access sub-picture have the same temporal identifier, the same layer identifier, and the same sub-picture index.

20. A non-transitory computer-readable recording medium stores a bitstream of a video generated by a method executed by a video processing device, wherein, The method includes: Generating the bitstream for the video including the current picture including the current sub-picture, where the current sub-picture includes a current strip, Among them, the bitstream conforms to a first format rule, and in the following cases, the first format rule does not allow an active entry in the reference picture list of the current strip to include a first picture that is before a second picture in decoding order, and the second picture includes a progressive temporal sub-layer access sub-picture: (a) The first picture has the same temporal identifier as the current sub-picture and the same layer identifier of the network abstraction layer (NAL) unit, and (b) The current sub-picture is after the progressive temporal sub-layer access sub-picture in the decoding order, and (c) The current sub-picture and the progressive temporal sub-layer access sub-picture have the same temporal identifier, the same layer identifier, and the same sub-picture index.

21. A method for storing a bitstream of a video, including: Generating the bitstream for the video including the current picture including the current sub-picture, where the current sub-picture includes a current strip; and Storing the bitstream in a non-transitory computer-readable recording medium, Among them, the bitstream conforms to a first format rule, and in the following cases, the first format rule does not allow an active entry in the reference picture list of the current strip to include a first picture that is before a second picture in decoding order, and the second picture includes a progressive temporal sub-layer access sub-picture: (a) The first picture has the same temporal identifier as the current sub-picture and the same layer identifier of the network abstraction layer (NAL) unit, and (b) The current sub-picture is after the progressive temporal sub-layer access sub-picture in the decoding order, and (c) The current sub-picture and the step-by-step time-domain sub-layer access sub-picture have the same time-domain identifier, the same layer identifier, and the same sub-picture index.

Images