Methods, apparatus and storage media for processing video data

Abstract

The processing may include video encoding, decoding, or transcoding. In one example, the video processing method includes performing a conversion between a video and a video bitstream comprising one or more layers, according to a format rule, wherein the format rule specifies that the reference image referred to by each inter-layer reference image entry in the reference image list of the current image of the current layer satisfies a constraint, wherein the constraint is at least one of the following: (a) the reference image is an intra-frame random access (IRAP) image, or (b) the reference image has a temporal identifier less than or equal to a specific value based on the maximum allowed value of video layers that the current layer's strip can reference, wherein the maximum allowed value is indicated in the syntax element.

Description

[0001] Cross-references to related applications

[0002] This application promptly claims priority and benefit to U.S. Provisional Patent Application No. 63 / 016,122, filed April 27, 2020. The entire disclosure of the aforementioned application is incorporated herein by reference as part of the disclosure of this application. Technical Field

[0003] The patent document relates to image and video encoding and decoding. Background Technology

[0004] Digital video consumes the largest share of bandwidth in the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth demand for digital video is expected to continue to grow. Summary of the Invention

[0005] This document discloses techniques that can be used by video encoders and decoders to process the codec representation of video using control information useful for decoding the codec representation.

[0006] In one example aspect, a video processing method is disclosed. The method includes: performing a conversion between a video comprising one or more images containing one or more sub-images and a codec representation of the video. The codec representation conforms to a format rule specifying that the one or more images comprising one or more sub-images are included in a codec representation according to a Network Abstraction Layer (NAL) unit, wherein the type NAL unit indicated in the codec representation includes either a codec stripe of a specific type of image or a codec stripe of a specific type of sub-image.

[0007] In another example, a different video processing method is disclosed. This method includes performing a conversion between a video and a codec representation of a video comprising one or more images containing one or more sub-images, wherein the codec representation conforms to a format rule specifying that two adjacent sub-images with different Network Abstraction Layer Unit (BIL) types will have the same indication of the sub-images as image markers.

[0008] In another example, a different video processing method is disclosed. This method includes: performing a conversion between a video and a video codec representation comprising one or more images containing one or more sub-images, wherein the codec representation conforms to a format rule defining the order of sub-images of a first type and sub-images of a second type, wherein the first sub-image is a tail sub-image, a leading sub-image, or a random access skip leading (RASL) sub-image type, and the second sub-image is a RASL type, a random access decodable leading (RADL) type, an instantaneous decode refresh (IDR) type, or a gradually decode refresh (GDR) type sub-image.

[0009] In another example, a different video processing method is disclosed. This method includes performing a conversion between a video comprising one or more images containing one or more sub-images and a video codec representation, wherein the codec representation conforms to a format rule that defines conditions allowing or disallowing the co-occurrence of sub-images of a first type with sub-images of a second type.

[0010] In another example, a different video processing method is disclosed. The method includes: performing a conversion between a video and a video bitstream comprising multiple layers including one or more images, according to a format rule, wherein the format rule specifies that the reference image referred to by each inter-layer reference image entry in the reference image list of the current image's stripe of the current layer satisfies a constraint, wherein the constraint is at least one of the following: (a) the reference image is an intra-frame random access (IRAP) image, or (b) the reference image has a temporal identifier less than or equal to a specific value, which is based on a maximum permissible value of video layers that the stripe from the current layer can reference, wherein the maximum permissible value is indicated in a syntax element.

[0011] In another example, a different video processing method is disclosed. This method includes performing a conversion between a video and a bitstream, comprising a video image, according to a format rule, wherein the format rule specifies the use of one or more of a first syntax structure indicating the position of a vertical virtual boundary and a second syntax structure indicating the position of a horizontal virtual boundary to indicate the virtual boundaries of a video region of the video image, wherein the format rule specifies that the numerical values ​​indicated by the first syntax element in the first syntax structure and the second syntax element in the second syntax structure are 1 less than the positions of the vertical and horizontal virtual boundaries counted from the top left position of the video image.

[0012] In yet another example, a video encoder apparatus is disclosed. The video encoder includes a processor configured to implement the methods described above.

[0013] In yet another example, a video decoder apparatus is disclosed. The video decoder includes a processor configured to implement the methods described above.

[0014] In yet another example, a computer-readable medium on which code is stored is disclosed. This code implements one of the methods described herein in the form of processor-executable code.

[0015] These and other features will be described in this document. Attached Figure Description

[0016] Figure 1 An example of raster scan strip segmentation of an image is shown, where the image is divided into 12 tiles and 3 raster scan strips.

[0017] Figure 2 An example of rectangular strip segmentation of an image is shown, where the image is divided into 24 slices (6 slice columns and 4 slice rows) and 9 rectangular strips.

[0018] Figure 3 An example of an image divided into slices and rectangular strips is shown, where the image is divided into 4 slices (2 slice columns and 2 slice rows) and 4 rectangular strips.

[0019] Figure 4 The image is shown as being divided into 15 slices, 24 strips, and 24 sub-images.

[0020] Figure 5 This is a block diagram of an example video processing system.

[0021] Figure 6 This is a block diagram of a video processing device.

[0022] Figure 7 This is a flowchart of an example method for video processing.

[0023] Figure 8 This is a block diagram illustrating a video encoding / decoding system according to some embodiments of the present disclosure.

[0024] Figure 9 This is a block diagram illustrating an encoder according to some embodiments of the present disclosure.

[0025] Figure 10 This is a block diagram illustrating a decoder according to some embodiments of the present disclosure.

[0026] Figure 11 This is a flowchart of an example method for video processing based on some implementations of the disclosed technology.

[0027] Figure 12 This is a flowchart of an example method for video processing based on some implementations of the disclosed technology. Detailed Implementation

[0028] Chapter headings are used in this document for ease of understanding, not to limit the applicability of the techniques and embodiments disclosed in each chapter to that chapter only. Furthermore, the use of H.266 terminology in some descriptions is merely for ease of understanding, not to limit the scope of the disclosed techniques. Therefore, the techniques described herein are also applicable to other video codec protocols and designs. In this document, edits are displayed in the text relative to the current draft of the VVC specification, with strikethrough indicating deleted text and highlighting (including bold and italic) indicating added text.

[0029] 1. Introduction

[0030] This document relates to video codec technology. Specifically, it concerns the definition of subpicture types in single-layer and multi-layer contexts, and their relationships regarding decoding order, output order, and prediction relationships between different types of subpictures. The key is to clearly specify the meaning of mixed subpicture types within a picture through a set of constraints on decoding order, output order, and prediction relationships. These ideas can be applied individually or in various combinations to any video codec standard or non-standard video codec that supports multi-layer video codecs, such as the Multi-Functional Video Codec (VVC) under development.

[0031] 2. Abbreviation

[0032] APS Adaptive Parameter Set

[0033] AU Access Unit

[0034] AUD Access Unit Delimiter

[0035] AVC Advanced Video Codec

[0036] CLVS codec layer video sequence

[0037] CPB encoded image buffer

[0038] CRA (Clean Random Access)

[0039] CTU (Codec Tree Unit)

[0040] CVS codec video sequence

[0041] DCI decoding capability information

[0042] DPB Decoding Image Buffer

[0043] EOB End of Bitstream

[0044] End of EOS sequence

[0045] GDR gradually decoded and refreshed

[0046] HEVC High-Efficiency Video Encoding and Decoding

[0047] HRD Hypothetical Reference Decoder

[0048] IDR Instant Decoding and Refresh

[0049] JEM Collaborative Exploration Mode

[0050] MCTS Motion Constraint Pieces

[0051] NAL Network Abstraction Layer

[0052] OLS Output Layer Set

[0053] PH image header

[0054] PPS Image Parameter Set

[0055] PTL refers to profiles, tiers, and levels.

[0056] PU Image Unit

[0057] RADL Random Access Decodable Preamble (Image)

[0058] RAP Random Access Point

[0059] RASL Random Access Skip Preamble (Image)

[0060] RBSP raw byte sequence payload

[0061] RPL Reference Image List

[0062] SEI Supplemental Enhancement Information

[0063] SPS Sequence Parameter Set

[0064] STSA Stepwise Temporal Sublayer Access

[0065] SVC Scalable Video Codec

[0066] VCL (Video Codec Layer)

[0067] VPS Video Parameter Set

[0068] VTM VVC Test Model

[0069] VUI Video Availability Information

[0070] VVC Multi-Functional Video Encoding and Decoding

[0071] 3. Preliminary Discussion

[0072] Video coding standards have primarily evolved through the development of well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, while ISO / IEC developed MPEG-1 and MPEG-4 Visual. These two organizations jointly developed the H.262 / MPEG-2 video and H.264 / MPEG-4 Advanced Video Coding (AVC) and H.265 / HEVC standards. Since H.262, video coding standards have been based on a hybrid video coding architecture, utilizing time prediction plus transform coding. To explore future video coding technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Group (JVET) in 2015. Since then, JVET has adopted many new methods and incorporated them into reference software called the Joint Exploration Model (JEM). JVET meetings are held quarterly, and the goal of the new coding standard is to reduce the bitrate by 50% compared to HEVC. The new video coding standard was officially named Universal Video Codec (VVC) at the JVET meeting in April 2018, and the first version of the VVC Test Model (VTM) was also released at that time. Due to ongoing efforts to standardize VVC, new codec technologies have been adopted into the VVC standard at every JVET meeting. The VVC working draft and the VTM test model are updated after each meeting. The current goal of the VVC project is to achieve Technical Finalization (FDIS) at the meeting in July 2020.

[0073] 3.1. Image Segmentation Schemes in HEVC

[0074] HEVC includes four different image segmentation schemes: regular striping, subordinate striping, slice, and wavefront parallel processing (WPP), which can be applied to maximum transmission unit (MTU) size matching, parallel processing, and reduced end-to-end latency.

[0075] The regular stripes are similar to those in H.264 / AVC. Each regular stripe is encapsulated in its own NAL unit, and intra-image prediction (intra-sample prediction, motion information prediction, coding pattern prediction) and entropy coding dependencies across stripe boundaries are disabled. Therefore, regular stripes can be reconstructed independently of other regular stripes within the same image (although interdependencies may still exist due to loop filtering operations).

[0076] Regular striping is the only tool available for parallelization, and it is also available in almost the same form in H.264 / AVC. Parallelization based on regular striping requires minimal inter-processor or inter-core communication (except for inter-processor or inter-core data sharing for motion compensation when decoding predictive encode-decode images, which is typically much heavier than inter-processor or inter-core data sharing due to intra-image prediction). However, for the same reason, using regular striping results in significant encoding / decoding overhead due to the bit cost of the stripe header and the lack of prediction across stripe boundaries. Furthermore, due to the intra-image independence of regular striping and the fact that each regular stripe is encapsulated in its own NAL unit, regular striping (compared to other tools mentioned below) also serves as a key mechanism for bitstream segmentation to match MTU size requirements. In many cases, the goals of parallelization and MTU size matching present conflicting requirements for stripe layout in images. This recognition led to the development of the parallelization tools mentioned below.

[0077] Slave stripes have short stripe headers and allow the bitstream to be split at tree block boundaries without disrupting any in-picture predictions. Essentially, slave stripes provide the ability to fragment regular stripes into multiple NAL units, thereby reducing end-to-end latency by allowing a portion of the regular stripe to be sent before the entire regular stripe's encoding is complete.

[0078] In WPP, images are segmented into single-row codec tree blocks (CTBs). Entropy decoding and prediction are allowed to use data from CTBs in other partitions. Parallel processing is possible through parallel decoding of CTB rows, where the start of decoding a CTB row is delayed by two CTBs, ensuring that data related to the top of the CTB and the right side of the subject CTB is available before the subject CTB is decoded. Using this staggered start (which looks like a wavefront when graphically represented), parallelization can use as many processors / cores as the number of CTB rows contained in the image. Because intra-image prediction between adjacent tree block rows within an image is allowed, the inter-processor / inter-core communication required to implement intra-image prediction can be substantial. WPP partitioning does not generate additional NAL units compared to when it is not applied, therefore WPP is not a tool for MTU size matching. However, if MTU size matching is required, regular striping can be used with WPP, but with some encoding / decoding overhead.

[0079] A slice defines the horizontal and vertical boundaries that divide an image into slice columns and rows. Slice columns extend from the top to the bottom of the image. Similarly, slice rows extend from the left to the right of the image. The number of slices in an image can be simply derived by multiplying the number of slice columns by the number of slice rows.

[0080] Before decoding the top-left CTB of the next slice in the order of slice raster scans of the image, the scan order of the CTBs is changed to be local within the slice (in the order of slice CTB raster scans). Similar to regular stripes, slices break the intra-image prediction dependencies and entropy decoding dependencies. However, they do not need to be included in a single NAL unit (the same as WPP in this respect); therefore, slices cannot be used for MTU size matching. Each slice can be processed by one processor / core, and the inter-processor / inter-core communication required between processing units decoding intra-image predictions of adjacent slices is limited to transmitting a shared stripe header when the stripe spans more than one slice, and loop filtering associated with the sharing of reconstructed samples and metadata. When a stripe includes more than one slice or WPP segment, the entry point byte offset of each slice or WPP segment in the stripe, except for the first slice or WPP segment, is signaled in the stripe header.

[0081] For simplicity, HEVC specifies restrictions on the application of four different image segmentation schemes. A given codec video sequence cannot simultaneously include both slices and wavefronts of most of the levels specified in the HEVC standard. For each strip and slice, one or both of the following conditions must be met: 1) All coded tree blocks in a strip belong to the same slice; 2) All coded tree blocks in a slice belong to the same strip. Finally, a wavefront segment contains exactly one CTB line, and when using WPP, if a strip begins at a CTB line, it must end at the same CTB line.

[0082] The latest modifications to HEVC are specified in the JCT-VC output file JCTVC-AC1005 "HEVC Additional Supplemental Enhancement Information (Draft 4)" published on October 24, 2017 by J. Boyce, A. Ramasubramonian, R. Skupin, G. J. Sullivan, A. Tourapis, and Y.-K. Wang (eds.), and are publicly available here: http: / / phenix.int-evry.fr / jct / doc_end_user / documents / 29_Macau / wg11 / JCTVC-AC1005-v2.zip. Including this modification, HEVC specifies three MCTS-related SEI messages: the Time MCTS SEI message, the MCTS Extracted Information Set SEI message, and the MCTS Extracted Information Nested SEI message.

[0083] The temporal MCTS SEI message indicates the presence of an MCTS in the bitstream and signals the MCTS. For each MCTS, motion vectors are restricted to pointing to full-sampled positions within the MCTS and fractional-sampled positions that require interpolation only from full-sampled positions within the MCTS, and motion vector candidates derived from blocks outside the MCTS for temporal motion vector prediction are not allowed. In this way, each MCTS can be decoded independently, and there are no slices not included in the MCTS.

[0084] The MCTS Extraction Information Set (SEI) message provides supplementary information (specified as part of the SEI message semantics) that can be used in MCTS sub-bitstream extraction to generate a bitstream conforming to the MCTS set. This information consists of multiple extraction information sets, each defining multiple MCTS sets and containing RBSP bytes that will replace the VPS, SPS, and PPS during the MCTS sub-bitstream extraction process. When extracting a sub-bitstream according to the MCTS sub-bitstream extraction process, the parameter sets (VPS, SPS, and PPS) need to be rewritten or replaced, and the slice header needs to be slightly updated because one or all slice address-related syntax elements (including first_slice_segment_in_pic_flag and slice_segment_address) typically need to have different values.

[0085] 3.2. Image Segmentation in VVC

[0086] In VVC, an image is divided into one or more slice rows and one or more slice columns. A slice is a sequence of CTUs covering a rectangular area of ​​the image. The CTUs in a slice are scanned in raster scan order within that slice.

[0087] A strip consists of an integer number of consecutive complete CTU lines from an integer number of complete slices or images.

[0088] Two stripe modes are supported: raster scan stripe mode and rectangular stripe mode. In raster scan stripe mode, a stripe contains a complete sequence of stripes in a raster scan of the image. In rectangular stripe mode, a stripe contains multiple complete slices that together form a rectangular area of ​​the image, or multiple consecutive complete CTU rows of a single slice that together form a rectangular area of ​​the image. Slices within a rectangular stripe are scanned in slice raster scan order within the rectangular area corresponding to that stripe.

[0089] A sub-image contains one or more stripes that collectively cover a rectangular area of ​​the image.

[0090] Figure 1 An example of raster scan strip segmentation of an image is shown, where the image is divided into 12 slices and 3 raster scan strips.

[0091] Figure 2 An example of rectangular strip segmentation of an image is shown, where the image is divided into 24 strips (6 strip columns and 4 strip rows) and 9 rectangular strips.

[0092] Figure 3 An example of an image divided into slices and rectangular strips is shown, where the image is divided into 4 slices (2 slice columns and 2 slice rows) and 4 rectangular strips.

[0093] Figure 4 An example of sub-image segmentation of an image is shown, where the image is segmented into 18 pieces: 12 pieces on the left-hand side, each covering a 4×4 CTU strip, and 6 pieces on the right-hand side, each covering a 2×2 CTU strip, forming two vertically stacked strips, resulting in a total of 24 strips and 24 sub-images of different dimensions (each strip being a sub-image).

[0094] 3.3. Changes in image precision within a sequence

[0095] In AVC and HEVC, the spatial precision of a picture cannot be changed unless a new sequence with a new SPS begins with an IRAP picture. VVC allows changes to the picture precision within a sequence at locations where IRAP pictures are not encoded; IRAP pictures are always intra-frame encoded and decoded. This feature is sometimes called Reference Picture Resampling (RPR) because it requires resampling the reference picture used for inter-frame prediction when the reference picture has a different precision than the current picture being decoded.

[0096] The scaling ratio is limited to greater than or equal to 1 / 2 (2x downsampling from the reference image to the current image) and less than or equal to 8 (8x upsampling). Three sets of resampling filters with different frequency cutoffs are specified to handle various scaling ratios between the reference and current images. The three sets of resampling filters are applied to scaling ratios ranging from 1 / 2 to 1 / 1.75, from 1 / 1.75 to 1 / 1.25, and from 1 / 1.25 to 8, respectively. Each set of resampling filters has 16 phases for luma and 32 phases for chroma, the same as in motion-compensated interpolation filters. In fact, the normal MC interpolation process is a special case of the resampling process where the scaling ratio ranges from 1 / 1.25 to 8. The horizontal and vertical scaling ratios are derived based on the image width and height, as well as the left, right, top, and bottom scaling offsets specified for the reference and current images.

[0097] Other aspects of the VVC design that support this feature and differ from HEVC include: i) Picture precision and the corresponding consistency window are signaled in the PPS instead of the SPS, where maximum picture precision is signaled. ii) For a single-layer bitstream, each picture storage (the slot in the DPB used to store a decoded picture) occupies the buffer size required to store the decoded picture with maximum picture precision.

[0098] 3.4. Scalable Video Codec (SVC) in General and VVC

[0099] Scalable Video Coding (SVC, sometimes also called Scalability in Video Coding) refers to video coding and decoding using a base layer (BL) (sometimes called a reference layer (RL)) and one or more scalable enhancement layers (EL). In SVC, the base layer can carry video data with a basic quality level. One or more enhancement layers can carry additional video data to support, for example, higher spatial, temporal, and / or signal-to-noise ratio (SNR) levels. Enhancement layers can be defined relative to previously encoded layers. For example, the bottom layer can be used as a BL, while the top layer can be used as an EL. Intermediate layers can be used as ELs or RLs, or both. For example, an intermediate layer (e.g., a layer that is neither the lowest nor the highest layer) can be an EL of the layer below the intermediate layer (such as a base layer or any intermediate enhancement layer) and simultaneously used as an RL of one or more enhancement layers above the intermediate layer. Similarly, in the multi-view or 3D extension of the HEVC standard, there can be multiple views, and information from one view can be used to codec (e.g., encode or decode) information from another view (e.g., motion estimation, motion vector prediction, and / or other redundancy).

[0100] In SVC, parameters used by the encoder or decoder are grouped into parameter sets based on the codec level (e.g., video level, sequence level, picture level, stripe level, etc.), and these parameters may be utilized at that codec level. For example, parameters that can be utilized by one or more codec video sequences at different layers in a bitstream can be included in the Video Parameter Set (VPS), and parameters that can be utilized by one or more pictures in a codec video sequence can be included in the Sequence Parameter Set (SPS). Similarly, parameters used by one or more stripes in a picture can be included in the Picture Parameter Set (PPS), and other parameters specific to a single strip can be included in the stripe header. Likewise, indications of which parameter set(s) a particular layer uses at a given time can be provided at various codec levels.

[0101] Because VVC supports Reference Picture Resampling (RPR), it's possible to design support for bitstreams containing multiple layers (e.g., two layers with SD and HD precision in VVC) without requiring any additional signal processing-level codec tools, as the upsampling required for spatial scalability support can be achieved using only RPR upsampling filters. However, scalability support requires a higher level of syntax changes (compared to no scalability support). Scalability support is specified in VVC version 1. Unlike scalability support in any earlier video codec standards (including extensions to AVC and HEVC), VVC's scalability is designed to be as friendly as possible to single-layer decoder designs. The decoding capability of a multi-layer bitstream is specified as if there were only one layer in the bitstream. For example, decoding capabilities such as DPB size are specified in a way that is independent of the number of layers in the bitstream to be decoded. Essentially, a decoder designed for a single-layer bitstream doesn't require many changes to decode multi-layer bitstreams. Compared to the multi-layer extension designs of AVC and HEVC, HLS is significantly simplified at the expense of some flexibility. For example, IRAP AU requires a picture of every layer present in CVS.

[0102] 3.5. Random Access and its Support in HEVC and VVC

[0103] Random access refers to accessing and decoding the bitstream starting with the image that is not the first image in the decoding order. To support tuning and channel switching in broadcast / multicast and multi-party video conferencing, searching in local playback and streaming, and stream adaptation in streaming, the bitstream needs to include frequent random access points. These access points are typically intra-frame codec images, but can also be inter-frame codec images (e.g., in the case of gradual decoding refresh).

[0104] HEVC includes signaling for Intra-Frame Random Access Point (IRAP) pictures in the NAL unit header via NAL unit type. Three types of IRAP pictures are supported: Instantaneous Decoder Refresh (IDR), Full Random Access (CRA), and Broken Link Access (BLA) pictures. IDR pictures constrain the inter-frame picture prediction structure to not reference any pictures preceding the current Group of Pictures (GOP), and are often referred to as Closed GOP IRAP pictures. CRA pictures are less restrictive by allowing some pictures to reference pictures preceding the current GOP; in the case of random access, all pictures are discarded. CRA pictures are often referred to as Open GOP IRAP pictures. BLA pictures are typically derived from the concatenation of two bitstreams or a portion thereof in a CRA picture, for example, during stream switching. To enable the system to better utilize IRAP pictures, a total of six different NAL units are defined to signal the properties of IRAP pictures. This can be used to better match the stream access point types defined in the ISO Basic Media File Format (ISOBMFF), which are used for random access support in Dynamic Adaptive Streaming (DASH) over HTTP.

[0105] VVC supports three types of IRAP pictures, two types of IDR pictures (one type has an associated RADL picture, and the other does not), and one type of CRA picture. These are essentially the same as HEVC. The BLA picture type in HEVC is not included in VVC, primarily for two reasons: i) The basic functionality of a BLA picture can be achieved by adding a sequence NAL unit end to a CRA picture, the presence of which indicates that a new CVS begins in a single-layer bitstream. ii) During the development of VVC, it was desirable to specify fewer NAL unit types than in HEVC, as shown by using 5 bits instead of 6 bits for the NAL unit type field in the NAL unit header.

[0106] Another key difference in random access support between VVC and HEVC is the more canonical support for GDR in VVC. In GDR, bitstream decoding can begin with an inter-frame encoded picture, although not the entire picture region may be correctly decoded initially, but after several pictures, the entire picture region will be correctly decoded. AVC and HEVC also support GDR, using Recovery Point (SEI) messages to signal GDR random access points and recovery points. In VVC, a new NAL unit type is specified to indicate GDR pictures, and recovery points are signaled in the picture header syntax structure. This allows CVS and bitstreams to begin with GDR pictures. This means that an entire bitstream can contain only inter-frame encoded pictures, without any single intra-frame encoded pictures. The main benefit of specifying GDR support in this way is providing consistent behavior for GDR. GDR enables encoders to smooth the bitrate of a bitstream by distributing intra-frame encoded stripes or blocks across multiple images, rather than intra-frame encoding and decoding the entire image, thus significantly reducing end-to-end latency. This is considered more important than ever today as ultra-low latency applications such as wireless displays, online gaming, and drone-based applications become increasingly popular.

[0107] Another GDR-related feature in VVC is virtual boundary signaling. The boundary between the refreshed area (i.e., the correctly decoded area) and the unrefreshed area at the GDR image and its recovery point can be signaled as a virtual boundary. When signaled, loop filtering across the boundary is not applied, thus preventing decoding mismatches at or near the boundary. This is useful when the application decides to display the correctly decoded area during the GDR process.

[0108] IRAP images and GDR images can be collectively referred to as Random Access Point (RAP) images.

[0109] 3.6. Reference Image Management and Reference Image List (RPL)

[0110] Reference picture management is a core function required by any video codec scheme that uses inter-frame prediction. It manages storing reference pictures in the decode picture buffer (DPB), removing reference pictures from the decode picture buffer (DPB), and placing reference pictures into the RPL in the correct order.

[0111] HEVC's reference picture management differs from AVC's, including reference picture tagging and removal from the decoded picture buffer (DPB) and reference picture list construction (RPLC). Instead of AVC's sliding window-based reference picture tagging mechanism with Adaptive Memory Management Control Operations (MMCO), HEVC specifies a reference picture management and tagging mechanism based on a so-called Reference Picture Set (RPS), and therefore RPLC is based on the RPS mechanism. An RPS consists of a reference picture set associated with a picture (consisting of all reference pictures preceding the associated picture in decoding order), which can be used for inter-frame prediction of the associated picture or any picture following the associated picture in decoding order. The reference picture set consists of five reference picture lists. The first three lists contain all reference pictures that can be used for inter-frame prediction of the current picture and for inter-frame prediction of one or more pictures following the current picture in decoding order. The other two lists consist of all reference pictures that are not used for inter-frame prediction of the current picture but can be used for inter-frame prediction of one or more pictures following the current picture in decoding order. RPS provides "intra-frame encoding / decoding" signaling in the DPB state, instead of "inter-frame encoding / decoding" signaling as in AVC, primarily to improve error resilience. HEVC's RPLC procedure is based on RPS, notifying the index by signaling a subset of the RPS for each reference index; this process is simpler than the RPLC procedure in AVC.

[0112] VVC's reference picture management is more similar to HEVC than AVC's, but simpler and more robust. As in those standards, two Reference Picture Sets (RPLs), List 0 and List 1, are derived, but they are not based on the reference picture set concept used in HEVC or the automatic sliding window process used in AVC; instead, they are signaled more directly. Reference pictures used for RPLs are listed as active and inactive entries, and only active entries can be used as reference indices for inter-frame prediction of the current picture's CTU. Inactive entries indicate other pictures to be saved in the DPB for reference by other pictures arriving later in the bitstream.

[0113] 3.7. Parameter Set

[0114] AVC, HEVC, and VVC specify parameter sets. Parameter set types include SPS, PPS, APS, and VPS. All versions of AVC, HEVC, and VVC support SPS and PPS. VPS was introduced with HEVC and is included in both HEVC and VVC. APS is not included in AVC or HEVC, but is included in the latest VVC draft text.

[0115] The Sequence-Level Prefix (SPS) is designed to carry sequence-level header information, while the Picture-Level Prefix (PPS) is designed to carry infrequently changing picture-level header information. Using SPS and PPS, infrequently changing information does not need to be repeated for each sequence or picture, thus avoiding redundant signaling. Furthermore, the use of SPS and PPS enables out-of-band transmission of important header information, thereby not only avoiding the need for redundant transmission but also improving error resilience.

[0116] The VPS was introduced to carry sequence-level header information shared by all layers in a multi-layer bitstream.

[0117] The purpose of APS is to carry such image-level or strip-level information, which requires a considerable number of bits to encode and decode, can be shared by multiple images, and can have a considerable number of different variations in the sequence.

[0118] 3.8. Relevant Definitions in VVC

[0119] The relevant definitions in the latest VVC text (JVET-Q2001-vE / v15) are as follows.

[0120] Previous IRAP images (if present) with the same nuh_layer_id value as the specific image, in the order of decoding.

[0121] In it yes of

[0122] In each of them The nal_unit_type is equal to that of CRA_NUT. .

[0123] Sequence, which includes in decoding order Followed by zero or more non- of Including all subsequent But not including Any subsequent AU.

[0124] Each layer in the CVS has a PU, and the encoded / decoded image in each PU is... of

[0125] The encoded / decoded image in each current PU is of

[0126] Encoding and decoding images within it is of

[0127] The nal_unit_type of each VCL NAL unit is equal to GDR_NUT.

[0128] In it yes of

[0129] Each The nal_unit_type is equal to IDR_W_RADL or IDR_N_LP

[0130] Each layer in CVS has And each The encoded and decoded images in the image are AU.

[0131] Encoding and decoding images within it is of

[0132] all Have the same value for nal_unit_type in the range IDR_W_RADL to CRA_NUT (inclusive).

[0133] (leading picture): with In the same layer and by exist Previous images.

[0134] from The order of output (for output from...) Output ).

[0135] In it yes PU.

[0136] Each The nal_unit_type is equal to that of RADL_NUT.

[0137] Encoding and decoding images within it is of

[0138] Each The nal_unit_type is equal to the RASL_NUT.

[0139] In it yes of

[0140] Each The nal_unit_type is equal to that of STSA_NUT.

[0141] Note – STSA images do not use images with the same TemporalId as the STSA image for inter-frame prediction reference. Images that follow the STSA image in decoding order and have the same TemporalId as the STSA image do not use images that precede the STSA image in decoding order and have the same TemporalId as the STSA image for inter-frame prediction reference. STSA images enable switching upwards from the immediately adjacent lower sublayer to the sublayer containing the STSA image. The TemporalId of the STSA image must be greater than 0.

[0142] One or more A rectangular area.

[0143] (trailing picture): in the output order Subsequent non-IRAP And it is not an STSA image.

[0144] Note – Tail images associated with an IRAP image must also follow the IRAP image in decoding order. Images that are both output-ordered after the associated IRAP image and decoded-ordered before the associated IRAP image are not allowed.

[0145] 3.9. NAL Unit Header Syntax and Semantics in VVC

[0146] In the latest VVC text (in JVET-Q2001-vE / v15), the NAL unit header syntax and semantics are as follows.

[0147] 7.3.1.2 NAL Unit Header Syntax

[0148]

[0149] 7.4.2.2 NAL Unit Header Semantics

[0150] It should be equal to 0.

[0151] It should be equal to 0. The value of nuh_reserved_zero_bit 1 may be specified by ITU-T|ISO / IEC in the future. The decoder will ignore (i.e., remove and discard) NAL units where nuh_reserved_zero_bit is equal to 1.

[0152] Specifies the layer identifier to which the VCL NAL element belongs, or the layer identifier to which a non-VCL NAL element applies. The value of nuh_layer_id should be in the range of 0 to 55 (inclusive). Other values ​​for nuh_layer_id are reserved for future use by ITU-T|ISO / IEC.

[0153] The value of nuh_layer_id should be the same for all VCL NAL units of the codec image. The nuh_layer_id value of the codec image or PU is the nuh_layer_id value of the VCL NAL unit of the codec image or PU.

[0154] The nuh_layer_id value of AUD, PH, EOS, and FD NAL cells is constrained as follows:

[0155] – If nal_unit_type equals AUD_NUT, then nuh_layer_id should equal vps_layer_id[0].

[0156] Otherwise, when nal_unit_type is equal to PH_NUT, EOS_NUT or FD_NUT, nuh_layer_id should be equal to the nuh_layer_id of the associated VCL NAL unit.

[0157] Note 1 – The value of nuh_layer_id for DCI, VPS, and EOB NAL units is unrestricted.

[0158] The value of nal_unit_type should be the same for all images in CVSS AU.

[0159] Specify the NAL cell type, i.e., the type of RBSP data structure contained in the NAL cell as specified in Table 5.

[0160] NAL units have a nal_unit_type in the range UNSPEC_28..UNSPEC_31 (including endpoints), but their semantics are not specified and should not affect the decoding process specified in this specification.

[0161] Note 2 – NAL unit types within the range UNSPEC_28..UNSPEC_31 can be used, as determined by the application. The decoding process for these values ​​of nal_unit_type is not specified in this specification. Because different applications may use these NAL unit types for different purposes, special care must be taken when designing encoders that generate NAL units with these nal_unit_type values, and when designing decoders that interpret the content of NAL units with these nal_unit_type values. This specification does not define any management of these values. These nal_unit_type values ​​may only apply in contexts where “conflicts” (i.e., different definitions of the meaning of NAL unit content with the same nal_unit_type value) are not important, impossible, or managed (e.g., in environments that control application or transport specifications or by controlling bitstream distribution).

[0162] For purposes other than determining the amount of data in the DU of the bitstream (as specified in Appendix C), the decoder will ignore (remove and discard) the contents of all NAL units using the reserved value of nal_unit_type.

[0163] Note 3 – This requirement allows for the definition of compatible extensions to this specification in the future.

[0164] Table 5 – NAL Unit Type Codes and NAL Unit Type Categories

[0165]

[0166]

[0167] Note 4 – A fully random access (CRA) picture can have an associated RASL or RADL picture that exists in the bitstream.

[0168] Note 5 – An Instantaneous Decode Refresh (IDR) picture with nal_unit_type equal to IDR_N_LP does not have an associated preamble picture present in the bitstream. An IDR picture with nal_unit_type equal to IDR_W_RADL does not have an associated RASL picture present in the bitstream, but may have an associated RADL picture present in the bitstream.

[0169] The value of nal_unit_type should be the same for all VCL NAL units in a sub-image. A sub-image is considered to have the same NAL unit type as the VCL NAL units in the sub-image.

[0170] For any specific image's VCL NAL unit, the following applies:

[0171] – If mixed_nalu_types_in_pic_flag equals 0, then the value of nal_unit_type should be the same for all VCL NAL units of the picture, and the picture or PU is considered to have the same NAL unit type as the VCL NAL unit of the picture or PU.

[0172] Otherwise (if mixed_nalu_types_in_pic_flag equals 1), the image should have at least two sub-images, and the VCL NAL units of the image should have exactly two different nal_unit_type values, as follows: the VCL NAL units of at least one sub-image of the image should each have a specific value of nal_unit_type, equal to STSA_NUT, RADL_NUT, RASL_NUT, IDR_W_RADL, IDR_N_LP, or CRA_NUT, while the VCL NAL units of the other sub-images in the image should each have a different specific value of nal_unit_type, equal to TRAIL_NUT, RADL_NUT, or RASL_NUT.

[0173] For single-layer bitstreams, the following constraints apply:

[0174] – Except for the first image in the bitstream in decoding order, each image is treated as associated with a previous IRAP image in decoding order.

[0175] – When the image is a leading image of an IRAP image, it should be a RADL or RASL image.

[0176] – When the image is the tail image of an IRAP image, it should not be a RADL or RASL image.

[0177] – The bitstream should not contain RASL images associated with IDR images.

[0178] – There should be no RADL images in the bitstream associated with an IDR image whose nal_unit_type is equal to IDR_N_LP.

[0179] Note 6 – By discarding all PUs preceding the IRAP PU, it is possible to perform random access at the location of the IRAP PU (and correctly decode the IRAP picture and all subsequent non-RASL pictures in the decoding order), provided that each parameter set is available when referenced (either in the bitstream or via an external means not specified in this specification).

[0180] - Any image that precedes the IRAP image in decoding order should precede the IRAP image in output order, and should precede any RADL image associated with the IRAP image in output order.

[0181] - Any RASL images associated with a CRA image should precede any RADL images associated with a CRA image in the output order.

[0182] – Any RASL images associated with a CRA image should be output in the order of output and followed by any IRAP images that are decoded before the CRA image.

[0183] – If field_seq_flag equals 0, and the current image is a preceding image associated with an IRAP image, then it should precede all non-preceding images associated with the same IRAP image in decoding order. Otherwise, assuming picA and picB are the first and last preceding images associated with the IRAP image in decoding order, there should be at most one non-preceding image preceding picA in decoding order, and no non-preceding images between picA and picB in decoding order.

[0184] Decrement by 1 specifies the time-domain identifier of the NAL cell.

[0185] The value of nuh_temporal_id_plus1 should not be equal to 0.

[0186] The variable TemporalId is derived as follows:

[0187] TemporalId=nuh_temporal_id_plus1-1 (36)

[0188] When nal_unit_type is in the range IDR_W_RADL to RSV_IRAP_12 (inclusive), TemporalId should be equal to 0.

[0189] TemporalId should not be equal to 0 when nal_unit_type equals STSA_NUT and vps_independent_layer_flag[GeneralLayerIdx[nuh_layer_id]] equals 1.

[0190] The TemporalId value should be the same for all VCL NAL units in the AU. The TemporalId value of the codec image, PU, ​​or AU is the TemporalId value of the VCL NAL unit in the codec image, PU, ​​or AU. The TemporalId value of the sublayer representation is the maximum value of the TemporalId of all VCL NAL units in the sublayer representation.

[0191] The TemporalId value of non-VCL NAL cells is constrained as follows:

[0192] – If nal_unit_type is equal to DCI_NUT, VPS_NUT, or SPS_NUT, then TemporalId should be equal to 0, and the TemporalId of the AU containing the NAL unit should be equal to 0.

[0193] Otherwise, if nal_unit_type is equal to PH_NUT, then TemporalId should be equal to the TemporalId of the PU containing the NAL unit.

[0194] Otherwise, if nal_unit_type is equal to EOS_NUT or EOB_NUT, then TemporalId should be equal to 0.

[0195] Otherwise, if nal_unit_type is equal to AUD_NUT, FD_NUT, PREFIX_SEI_NUT, or SUFFIX_SEI_NUT, then TemporalId should be equal to the TemporalId of the AU containing the NAL unit.

[0196] Otherwise, when nal_unit_type is equal to PPS_NUT, PREFIX_APS_NUT or SUFFIX_APS_NUT, TemporalId should be greater than or equal to the TemporalId of the PU containing the NAL unit.

[0197] Note 7 – When the NAL unit is a non-VCL NAL unit, the TemporalId value is equal to the minimum TemporalId value of all AUs applied to the non-VCL NAL unit. When nal_unit_type is equal to PPS_NUT, PREFIX_APS_NUT, or SUFFIX_APS_NUT, the TemporalId can be greater than or equal to the TemporalId of the AU, because all PPS and APS can be included at the beginning of the bitstream (e.g., when they are transmitted out of band and the receiver places them at the beginning of the bitstream), where the first codec picture has a TemporalId equal to 0.

[0198] 3.10. Mixed NAL unit types within the image

[0199] 7.4.3.4 Image Parameter Set Semantics ...

[0201] A value of 1 indicates that each picture in the reference PPS has more than one VCL NAL unit, the VCL NAL units do not have the same value of nal_unit_type, and the picture is not an IRAP picture. A value of 0 for mixed_nalu_types_in_pic_flag indicates that each picture in the reference PPS has one or more VCL NAL units, and the VCL NAL units of each picture in the reference PPS have the same value of nal_unit_type.

[0202] When no_mixed_nalu_types_in_pic_constraint_flag equals 1, the value of mixed_nalu_types_in_pic_flag should be equal to 0.

[0203] For each stripe in image picA with a nal_unit_type value nalUnitTypeA in the range IDR_W_RADL to CRA_NUT (inclusive), image picA also contains one or more stripes with another nal_unit_type value, i.e., the value of mixed_nalu_types_in_pic_flag of image picA is equal to 1, the following applies:

[0204] – The stripe should belong to the subpicA whose corresponding subpic_treated_as_pic_flag[i] value is 1.

[0205] – A stripe should not belong to a subpico of a picA containing a VCL NAL unit whose nal_unit_type is not equal to nalUnitTypeA.

[0206] – If nalUnitTypeA equals CRA, then for all subsequent PUs after the current picture in CLVS in decoding and output order, the RefPicList[0] and RefPicList[1] of the stripes in subpicA of these PUs should not include any picture in valid entries that precedes picA in decoding order.

[0207] – Otherwise (i.e., nalUnitTypeA equals IDR_W_RADL or IDR_N_LP), for all PUs in the CLVS that are in the decoding order after the current picture, the RefPicList[0] or RefPicList[1] of the stripes in the subpicA of these PUs should not include any picture in the valid entries that are in the decoding order before the picA.

[0208] Note 1 – `mixed_nalu_types_in_pic_flag` equal to 1 indicates that the reference PPS image contains stripes with different NAL unit types, such as encoder-decoder images derived from sub-picture bitstream merging operations. The encoder must ensure bitstream structure matching and further alignment of parameters from the original bitstream. An example of this alignment is as follows: when `sps_idr_rpl_present_flag` is equal to 0 and `mixed_nalu_types_in_pic_flag` is equal to 1, the reference PPS image cannot have stripes with `nal_unit_type` equal to `IDR_W_RADL` or `IDR_N_LP`. ...

[0210] 3.11. VVC Image Header Structure Syntax and Semantics

[0211] In the latest VVC text (in JVET-Q2001-vE / v15), the image header structure syntax and semantics most relevant to this invention are as follows.

[0212] 7.3.2.7 Image Header Structure Syntax

[0213]

[0214] 7.4.3.7 Image header structure and semantics

[0215] The PH syntax structure contains information shared by all stripes of the encoded / decoded image associated with the PH syntax structure.

[0216] A value of 1 indicates that the current image is a GDR or IRAP image. A value of 0 indicates that the current image may or may not be a GDR or IRAP image.

[0217] A value of 1 indicates that the image associated with the PH is a GDR image. A value of 0 indicates that the image associated with the PH is not a GDR image. When it does not exist, the value of gdr_pic_flag is inferred to be 0. When gdr_enabled_flag is 0, the value of gdr_pic_flag should be 0.

[0218] Note 1 – When gdr_or_irap_pic_flag equals 1 and gdr_pic_flag equals 0, the PH-related image is an IRAP image. ...

[0220] Specifies the image order count modulo MaxPicOrderCntLsb for the current image. The length of the ph_pic_order_cnt_lsb syntax element is log2_max_pic_order_cnt_lsb_minus4+4 bits. The value of ph_pic_order_cnt_lsb should be in the range of 0 to MaxPicOrderCntLsb-1 (inclusive).

[0221] This affects the output of previously decoded images in the DPB after decoding a CLVSS image, which is not the first image in the bitstream specified in Appendix C.

[0222] Specifies the recovery point for the decoded images in output order. If the current image is a GDR image associated with the PH, and there exists an image picA in CLVS that follows the current GDR image in decoding order, with PicOrderCntVal equal to the PicOrderCntVal of the current GDR image plus the value of recovery_poc_cnt, then image picA is called the recovery point image. Otherwise, the first image in output order whose PicOrderCntVal is greater than the PicOrderCntVal of the current image plus the value of recovery_poc_cnt is called the recovery point image. The recovery point image should not precede the current GDR image in decoding order. The value of recovery_poc_cnt should be in the range of 0 to MaxPicOrderCntLsb-1 (inclusive).

[0223] When the current image is a GDR image, the variable RpPicOrderCntVal is derived as follows:

[0224] RpPicOrderCntVal=PicOrderCntVal+recovery_poc_cnt (81)

[0225] Note 2 – When gdr_enabled_flag equals 1 and the current image’s PicOrderCntVal is greater than or equal to the associated GDR image’s RpPicOrderCntVal, the current and subsequent decoded images in output order are completely matched with the corresponding images generated by starting the decoding process from the previous IRAP image (if present) that precedes the associated GDR image in decoding order. ...

[0227] 3.12. Limitations on RPL in VVC

[0228] In the latest VVC documentation (in JVET-Q2001-vE / v15), the constraints on RPL in VVC are as follows (as part of the decoding process for constructing the reference picture list in section 8.3.2 of VVC). 8.3.2 Decoding Process for Constructing the Reference Picture List ...

[0230] For each i equal to 0 or 1, the first NumRefIdxActive[i] entries in RefPicList[i] are considered valid entries in RefPicList[i], and the other entries in RefPicList[i] are considered invalid entries in RefPicList[i].

[0231] Note 2 – A particular image may be referred to by both an entry in RefPicList[0] and an entry in RefPicList[1]. A particular image may also be referred to by more than one entry in RefPicList[0] or more than one entry in RefPicList[1].

[0232] Note 3 – Valid entries in RefPicList[0] and RefPicList[1] together refer to all reference images that can be used for inter-frame prediction of the current image and one or more images following the current image in decoding order. Invalid entries in RefPicList[0] and RefPicList[1] together refer to all reference images that are not used for inter-frame prediction of the current image but can be used for inter-frame prediction of one or more images following the current image in decoding order.

[0233] Note 4 – There may be one or more entries in RefPicList[0] or RefPicList[1] that are equal to “No reference image” because the corresponding image does not exist in the DPB. Each invalid entry in RefPicList[0] or RefPicList[0] that is equal to “No reference image” should be ignored. An unexpected image loss should be inferred for each valid entry in RefPicList[0] or RefPicList[1] that is equal to “No reference image”.

[0234] The requirement for bitstream consistency applies to the following constraints:

[0235] – For each i equal to 0 or 1, num_ref_entries[i][RplsIdx[i]] should not be less than NumRefIdxActive[i].

[0236] The image referred to by each valid entry in –RefPicList[0] or RefPicList[1] should exist in DPB and should have a TemporalId less than or equal to the current image.

[0237] The image referred to by each entry in –RefPicList[0] or RefPicList[1] should not be the current image and should have a non_reference_picture_flag equal to 0.

[0238] – A STRP entry in RefPicList[0] or RefPicList[1] of a stripe of an image and an LTRP entry in RefPicList[0] or RefPicList[1] of the same stripe or different stripe of the same image should not refer to the same image.

[0239] – There should be no LTRP entry in RefPicList[0] or RefPicList[1], for which the difference between the PicOrderCntVal of the current image and the PicOrderCntVal of the image referred to by the entry is greater than or equal to 2. 24 .

[0240] – Let setOfRefPics be the set of unique images referred to by all entries in RefPicList[0] that have the same nuh_layer_id as the current image and all entries in RefPicList[1] that have the same nuh_layer_id as the current image. The number of images in setOfRefPics should be less than or equal to MaxDpbSize-1 (inclusive), where MaxDpbSize is specified in Clause A.4.2, and setOfRefPics should be the same for all stripes of the images.

[0241] – When the nal_unit_type of the current stripe is equal to STSA_NUT, there should be no valid entries in RefPicList[0] or RefPicList[1] that have a TemporalId equal to the TemporalId of the current image and a nuh_layer_id equal to the nuh_layer_id of the current image.

[0242] – When the current image is a picture that is after the STSA picture in the decoding order and has a TemporalId equal to the TemporalId of the current image and a nuh_layer_id equal to the nuh_layer_id of the current image, there should be no picture that is before the STSA picture in the decoding order and has a TemporalId equal to the TemporalId of the current image and a nuh_layer_id equal to the nuh_layer_id of the current image. Such picture should be included as a valid entry in RefPicList[0] or RefPicList[1].

[0243] – When the current image is a CRA image, there should be no image referred to by an entry in RefPicList[0] or RefPicList[1] that precedes any preceding IRAP image (if it exists) in output or decoding order.

[0244] – When the current image is a tail image, there should be no image generated by the decoding process of an unavailable reference image used to generate an IRAP image associated with the current image, as referred to by a valid entry in RefPicList[0] or RefPicList[1].

[0245] – When the current image is a tail image following one or more preceding images (if any) associated with the same IRAP image in both the decoding and output order, there should be no image generated by the decoding process of the unavailable reference image used to generate the IRAP image associated with the current image, as referred to by an entry in RefPicList[0] or RefPicList[1].

[0246] – When the current image is a recovery point image or an image that follows the recovery point image in the output order, there should be no entries in RefPicList[0] or RefPicList[1] containing images generated by the decoding process of an unavailable reference image used to generate the GDR image for the recovery point image.

[0247] – When the current image is the tail image, there should be no image referred to by a valid entry in RefPicList[0] or RefPicList[1] that precedes the associated IRAP image in output or decoding order.

[0248] – When the current image is the tail image of one or more preceding images (if any) associated with the same IRAP image in decoding and output order, there should be no image referred to by an entry in RefPicList[0] or RefPicList[1] that precedes the associated IRAP image in output or decoding order.

[0249] – When the current image is a RADL image, there should be no valid entries of the following in RefPicList[0] or RefPicList[1]:

[0250] oRASL image

[0251] o The image generated by the decoding process used to generate unusable reference images.

[0252] o Images preceding the associated IRAP images in the decoding order

[0253] – The image referred to by each ILRP entry in RefPicList[0] or RefPicList[1] of the current image strip should be in the same AU as the current image.

[0254] – The image referred to by each ILRP entry in RefPicList[0] or RefPicList[1] of the current image strip should exist in the DPB and should have a nuh_layer_id that is less than the nuh_layer_id of the current image.

[0255] – Each ILRP entry in the stripe’s RefPicList[0] or RefPicList[1] should be a valid entry. ...

[0257] 3.13. Setting PictureOutputFlag

[0258] In the latest VVC documentation (in JVET-Q2001-vE / v15), the specification for setting the value of the variable PictureOutputFlag is as follows (as part of the decoding process in Clause 8.1.2 for encoding and decoding pictures).

[0259] 8.1.2 Image Decoding Processing

[0260] The decoding process specified in this clause applies to each encoded / decoded image in BitstreamToDecode, referred to as the current image and represented by the variable CurrPic.

[0261] Depending on the value of chroma_format_idc, the number of sample points in the current image array is as follows:

[0262] – If chroma_format_idc equals 0, then the current image consists of a 1-sample array S L composition.

[0263] Otherwise (chroma_format_idc is not equal to 0), the current image consists of a 3-sample array S L S Cb S Cr composition.

[0264] The current image decoding process takes the syntax elements and uppercase variables from Item 7 as input. When interpreting the semantics of each NAL unit and each syntax element in the remainder of Item 8, the term "bitstream" (or a portion thereof, such as CVS of a bitstream) refers to BitstreamToDecode (or a portion thereof).

[0265] Depending on the value of separate_colour_plane_flag, the decoding process is constructed as follows:

[0266] – If separate_colour_plane_flag equals 0, the decoding process is called once, and the current image is output.

[0267] Otherwise (if separate_colour_plane_flag equals 1), the decoding process is called three times. The input to the decoding process is all NAL units of the encoded and decoded images with the same colour_plane_id value. The decoding process for NAL units with a specific colour_plane_id value is specified as a CVS (Content Value for Monochrome) color format containing only that specific colour_plane_id value in the bitstream. The output of each of the three decoding processes is assigned to one of the three sample arrays of the current image, with NAL units having colour_plane_ids equal to 0, 1, and 2 respectively assigned to S... L S Cb and S Cr .

[0268] Note – When separate_colour_plane_flag equals 1 and chroma_format_idc equals 3, the variable ChromaArrayType equals 0. During decoding, the value of this variable is evaluated, resulting in the same operation as for monochrome images (when chroma_format_idc equals 0).

[0269] For the current image CurrPic, the decoding process is as follows:

[0270] 1. Decoding of the NAL unit is specified in Clause 8.2.

[0271] 2. The procedures in Clause 8.3 use syntax elements at the stripe header level and above to specify the following decoding procedures:

[0272] – The variables and functions related to image sequence counting are derived as specified in Clause 8.3.1. This only needs to be called for the first stripe of the image.

[0273] – At the start of the decoding process for each strip of a non-IDR image, the decoding process for constructing the reference image list specified in Clause 8.3.2 is invoked to derive reference image list 0 (RefPicList[0]) and reference image list 1 (RefPicList[1]).

[0274] – The decoding process for the reference picture marker in Clause 8.3.2 is invoked, where the reference picture can be marked as “not used for reference” or “used for long-term reference.” This only needs to be invoked for the first strip of the picture.

[0275] – When the current image is a CRA image with NoOutputBeforeRecoveryFlag equal to 1 or a GDR image with NoOutputBeforeRecoveryFlag equal to 1, the decoding procedure for generating the unavailable reference image specified in sub-clause 8.3.4 is invoked, which only needs to be invoked for the first strip of the image.

[0276] –PictureOutputFlag is set as follows:

[0277] – If one of the following conditions is true, PictureOutputFlag is set to 0:

[0278] – The current image is a RASL image, and the NoOutputBeforeRecoveryFlag of the associated IRAP image is equal to 1.

[0279] –gdr_enabled_flag equals 1, and the current image is a GDR image with NoOutputBeforeRecoveryFlag equal to 1.

[0280] –gdr_enabled_flag equals 1, the current image is associated with a GDR image where NoOutputBeforeRecoveryFlag equals 1, and the PicOrderCntVal of the current image is less than the RpPicOrderCntVal of the associated GDR image.

[0281] –sps_video_parameter_set_id is greater than 0, ols_mode_idc is equal to 0, and the current AU contains an image picA that meets all of the following conditions:

[0282] –PicA’s PictureOutputFlag is equal to 1.

[0283] –PicA’s nuh_layer_id nuhLid is greater than the nuh_layer_id of the current image.

[0284] –PicA belongs to the output layer of OLS (i.e., OutputLayerIdInOls[TargetOlsIdx][0] is equal to nuhLid).

[0285] –sps_video_parameter_set_id is greater than 0, ols_mode_idc is equal to 2, and ols_output_layer_flag[target ols idx][general layeridx[nuh_layer_id]] is equal to 0.

[0286] Otherwise, PictureOutputFlag is set to equal pic_output_flag.

[0287] 3. The procedures in Clauses 8.4, 8.5, 8.6, 8.7, and 8.8 use syntax elements from all syntax structure layers to specify the decoding process. The requirement for bitstream consistency is that the encoding / decoding stripes of an image should contain stripe data for each CTU of the image, such that the image is divided into stripes, and each stripe is divided into CTUs, each forming a partition of the image.

[0288] 4. After all stripes of the current image have been decoded, the currently decoded image is marked as “for short-term reference”, and each ILRP entry in RefPicList[0] or RefPicList[1] is marked as “for short-term reference”.

[0289] 4. The technical problem solved by the disclosed technical solution

[0290] The existing design in the latest VVC documentation (in JVET-Q2001-vE / v15) has the following issues:

[0291] 1) Because it allows mixing sub-images of different types within an image, it is confusing to refer to the contents of NAL units with VCL NAL unit type as codec stripes of a specific type of image. For example, a NAL unit with nal_unit_type equal to CRA_NUT is a codec stripe of a CRA image only if all stripes of the image have nal_unit_type equal to CRA_NUT; if a stripe of the image has nal_unit_type not equal to CRA_NUT, then the image is not a CRA image.

[0292] 2) Currently, if a subpicture contains VCL NAL units with a nal_unit_type in the range IDR_W_RADL to CRA_NUT (inclusive), and for the picture, mixed_nalu_types_in_pic_flag is equal to 1, then for the subpicture, the value of subpic_treated_as_pic_flag[] needs to be equal to 1. In other words, for an IRAP subpicture in a picture that is mixed with another type of subpicture, the value of subpic_treated_as_pic_flag[] needs to be equal to 1. However, with the support for more mixing of VCLNAL unit types, this requirement is insufficient.

[0293] 3) Currently, only two different types of VCL NAL units (and two different types of sub-images) are allowed within an image.

[0294] 4) In single-layer and multi-layer contexts, there is a lack of constraints on the output order of tail sub-images relative to associated IRAP or GDR sub-images.

[0295] 5) Currently, it is specified that when an image is a preceding image of an IRAP image, it should be a RADL or RASL image. This constraint, along with the definition of preceding / RADL / RASL images, disallows the mixing of RADL and RASL NAL unit types within an image resulting from the mixing of two CRA images and their non-AU aligned associated RADL and RASL images.

[0296] 6) In single-layer and multi-layer contexts, there is a lack of constraints on the sub-image type (i.e., the NAL unit type of the VCL NAL unit in the sub-image) for the preceding sub-image.

[0297] 7) In both single-layer and multi-layer contexts, there is a lack of constraints on whether RASL sub-images can exist and be associated with IDR sub-images.

[0298] 8) In both single-layer and multi-layer contexts, there is a lack of constraints on whether RADL subpicks can exist and be associated with IDR subpicks whose nal_unit_type is equal to IDR_N_LP.

[0299] 9) In both single-layer and multi-layer contexts, there is a lack of constraints on the relative output order between sub-pictures that precede IRAP sub-pictures in the decoding order and RADL sub-pictures associated with IRAP sub-pictures.

[0300] 10) In both single-layer and multi-layer contexts, there is a lack of constraints on the relative output order between sub-images that precede the GDR sub-image in the decoding order and sub-images associated with the GDR sub-image.

[0301] 11) In both single-layer and multi-layer contexts, there is a lack of constraints on the relative output order between RASL sub-images associated with CRA sub-images and RADL sub-images associated with CRA sub-images.

[0302] 12) In both single-layer and multi-layer contexts, there is a lack of constraints on the relative output order between RASL sub-images associated with CRA sub-images and IRAP sub-images that precede CRA sub-images in decoding order.

[0303] 13) In both single-layer and multi-layer contexts, there is a lack of constraints on the relative decoding order between associated non-preamble and preamble images of IRAP images.

[0304] 14) In both single-layer and multi-layer contexts, there is a lack of constraints on valid RPL entries for sub-images following the STSA sub-image in the decoding order.

[0305] 15) In both single-layer and multi-layer contexts, there is a lack of constraints on RPL entries for CRA subgraphs.

[0306] 16) In both single-layer and multi-layer contexts, there is a lack of constraints on valid RPL entries referring to sub-images generated by the decoding process used to generate unusable reference images.

[0307] 17) In both single-layer and multi-layer contexts, there is a lack of constraints on RPL entries referring to sub-images generated by the decoding process used to generate unusable reference images.

[0308] 18) In both single-layer and multi-layer contexts, there is a lack of constraints on valid RPL entries for sub-images that are associated with the IRAP image and follow the IRAP image in the output order.

[0309] 19) In both single-layer and multi-layer contexts, there is a lack of constraints on RPL entries for sub-images that are associated with the IRAP image and follow the IRAP image in the output order.

[0310] 20) In both single-layer and multi-layer contexts, there is a lack of constraints on valid RPL entries for RADL sub-images.

[0311] 5. Examples of solutions and implementation methods

[0312] To address the aforementioned and other issues, methods outlined below are disclosed. These items should be considered as examples for explaining general concepts, and not interpreted in a narrow sense. Furthermore, these items can be used individually or in combination in any way.

[0313] 1) To address issue 1, instead of specifying the content of a NAL unit with the VCL NAL unit type as "code-decode stripe for a specific type of picture", it is specified as "code-decode stripe for a specific type of picture or subpicture". For example, the content of a NAL unit whose nal_unit_type is equal to CRA_NUT is specified as "code-decode stripe for a CRA picture or subpicture".

[0314] a. In addition, define one or more of the following terms: associated GDR sub-picture, associated IRAP sub-picture, CRA sub-picture, GDR sub-picture, IDR sub-picture, IRAP sub-picture, leading sub-picture, RADL sub-picture, RASL sub-picture, STSA sub-picture, and tail sub-picture.

[0315] 2) To solve problem 2, add a constraint that requires any two adjacent subpics with different NAL cell types to have subpic_treated_as_pic_flag[] equal to 1.

[0316] a. In one example, the constraint is specified as follows: For any two adjacent subpics in an image that have subpic indexes i and j, the two subpics should have the same NAL cell type when either subpic_treated_as_pic_flag[i] or subpic_treated_as_pic_flag[j] is equal to 0.

[0317] a. Alternatively, it is required that when any subpic with subpic index i has subpic_treated_as_pic_flag[i] equal to 0, all subpics in the picture should have the same NAL unit type (i.e., all VCL NAL units in the picture should have the same NAL unit type, i.e., the value of mixed_nalu_types_in_pic_flag should be equal to 0). This means that when all subpics have their corresponding subpic_treated_as_pic_flag[] equal to 1, mixed_nalu_types_in_pic_flag can only be equal to 1.

[0318] 3) To solve problem 3, when mixed_nalu_types_in_pic_flag equals 1, it is permissible for an image to contain more than two different types of VCL NAL units.

[0319] 4) To resolve issue 4, specify that the tail sub-image should be placed after the associated IRAP or GDR sub-image in the output order.

[0320] 5) To address issue 5, and to allow mixing of RADL and RASL NAL unit types within an image generated from the mixing of two CRA images and their associated non-AU aligned RADL and RASL images, the existing constraint that the preceding image of an IRAP image should be a RADL or RASL image is modified as follows: When the image is the preceding image of an IRAP image, the nal_unit_type value of all VCL NAL units in the image should be equal to RADL_NUT or RASL_NUT. Furthermore, in the decoding process of images with mixednal_unit_type values ​​of RADL_NUT and RASL_NUT, when the layer containing the image is an output layer, the PictureOutputFlag of the image is set to equal pic_output_flag.

[0321] Thus, by requiring all output images to be corrected to conform to the decoder's constraints, RADL sub-images in such images can be guaranteed. While a guarantee of the "correctness" of the "mid-valued" RASL sub-images in such images is appropriate when the associated CRA image has NoOutputBeforeRecoveryFlag equal to 1, it is actually unnecessary. The unnecessary parts of the guarantee are irrelevant and do not increase the complexity of implementing conformance to the encoder or decoder. In this case, a note clarifies that although such RASL sub-images associated with CRA images where NoOutputBeforeRecoveryFlag equals 1 can be output by the decoding process, they are not intended for display and therefore should not be used for display.

[0322] 6) To solve problem 6, it is stipulated that when a sub-image is the preceding sub-image of an IRAP sub-image, it should be a RADL or RASL sub-image.

[0323] 7) To solve problem 7, it is stipulated that there should be no RASL sub-images associated with IDR sub-images in the bitstream.

[0324] 8) To solve problem 8, it is stipulated that there should be no RADL sub-pictures in the bitstream that are associated with IDR sub-pictures whose nal_unit_type is equal to IDR_N_LP.

[0325] 9) To address problem 9, it is stipulated that, in the decoding order, any sub-image whose nuh_layer_id is equal to a specific value layerId and whose sub-image index is equal to a specific value subpicIdx, preceding the IRAP sub-image whose nuh_layer_id is equal to layerId and whose sub-image index is equal to subpicIdx, should be in the output order before the IRAP sub-image and all its associated RADL sub-images.

[0326] 10) To address problem 10, it is stipulated that, in the decoding order, any sub-image whose nuh_layer_id is equal to a specific value layerId and whose sub-image index is equal to a specific value subpicIdx, preceding the GDR sub-image whose nuh_layer_id is equal to layerId and whose sub-image index is equal to subpicIdx, should be in the output order before the GDR sub-image and all its associated sub-images.

[0327] 11) To address problem 11, it is stipulated that any RASL subpicks associated with a CRA subpick should precede any RADL subpicks associated with a CRA subpick in the output order.

[0328] 12) To address problem 12, it is stipulated that any RASL subpicks associated with CRA subpicks should be output after any IRAP subpicks, and any IRAP subpicks should be decoded before the CRA subpicks.

[0329] 13) To address problem 13, it is stipulated that if field_seq_flag equals 0, and (where nuh_layer_id equals the specific value layerId and the subpic index equals the specific value subpicIdx) the current subpic is a preceding subpic associated with an IRAP subpic, then it should precede all non-preceding subpics associated with the same IRAP subpic in decoding order; otherwise, assuming subpicA and subpicB are the first and last preceding subpics associated with the IRAP subpic in decoding order, there should be at most one non-preceding subpic with nuh_layer_id equal to layerId and subpic index equal to subpicIdx preceding subpicA in decoding order, and there should be no non-preceding subpic with nuh_layer_id equal to layerId and subpic index equal to subpicIdx between picA and picB in decoding order.

[0330] 14) To solve problem 14, it is stipulated that when the current sub-image whose TemporalId is equal to a specific value tId, nuh_layer_id is equal to a specific value layerId, and subpic index is equal to a specific value subpicIdx, and is a sub-image that follows the STSA sub-image whose TemporalId is equal to tId, nuh_layer_id is equal to layerId, and subpic index is equal to subpicIdx in the decoding order, there should be no image whose TemporalId is equal to tId and nuh_layer_id is equal to layerId in the decoding order that precedes the image containing the STSA sub-image, and such image should be included as a valid entry in RefPicList[0] or RefPicList[1].

[0331] 15) To address problem 15, it is stipulated that when the current subpick with nuh_layer_id equal to a specific value layerId and subpick index equal to a specific value subpicIdx is a CRA subpick, there should be no image referred to by an entry in RefPicList[0] or RefPicList[1] that precedes any image in output order or decoding order that contains an IRAP subpick with nuh_layer_id equal to layerId and subpick index equal to subpicIdx in decoding order (if such an image exists).

[0332] 16) To address problem 16, it is stipulated that when the current subpic is not a RASL subpic associated with a CRA subpic of a CRA image with NoOutputBeforeRecoveryFlag equal to 1, a GDR subpic of a GDR image with NoOutputBeforeRecoveryFlag equal to 1, or a recovery subpic of a GDR image with NoOutputBeforeRecoveryFlag equal to 1 and nuh_layer_id equal to layerId, there should be no image generated by the decoding process for generating unusable reference images as referred to by a valid entry in RefPicList[0] or RefPicList[1].

[0333] 17) To address problem 17, it is stipulated that when the current subpic is not a CRA subpic of a CRA image with NoOutputBeforeRecoveryFlag equal to 1, a subpic of a CRA image with NoOutputBeforeRecoveryFlag equal to 1, a subpic of a CRA image with NoOutputBeforeRecoveryFlag equal to 1, a subpic of a CRA image with NoOutputBeforeRecoveryFlag equal to 1, a subpic of a GDR image with NoOutputBeforeRecoveryFlag equal to 1, or a subpic of a GDR image with NoOutputBeforeRecoveryFlag equal to 1 and nuh_layer_id equal to layerId, there should be no image generated by the decoding process for generating an unusable reference image as referred to by an entry in RefPicList[0] or RefPicList[1].

[0334] 18) To address problem 18, it is stipulated that when the current subpicture is associated with an IRAP subpicture and is after the IRAP subpicture in output order, there should be no picture referred to by a valid entry in RefPicList[0] or RefPicList[1] that is before the picture containing the associated IRAP subpicture in output order or decoding order.

[0335] 19) To address problem 19, it is stipulated that when the current subpicture is associated with an IRAP subpicture, it should be after the IRAP subpicture in output order and after the preceding subpicture (if any) associated with the same IRAP subpicture in both decoding and output order. Therefore, there should be no image referred to by an entry in RefPicList[0] or RefPicList[1] that precedes the image containing the associated IRAP subpicture in output or decoding order.

[0336] 20) To solve problem 20, it is stipulated that when the current sub-image is a RADL sub-image, there should be no valid entry in either RefPicList[0] or RefPicList[1] for any of the following:

[0337] a. Images containing RASL sub-images

[0338] b. Images preceding the images containing associated IRAP sub-images in the decoding order.

[0339] 6. Example

[0340]

[0341] 6.1. First Embodiment

[0342] This embodiment applies to projects 1, 1a, 2, 2a, 4, and 6 through 20.

[0343] 3 Definitions ...

[0345] nuh_layer_id is equal to the previous layerId (when present) and in a specific There are no instances where nuh_layer_id equals layerId. nuh_layer_id is equal to the previous nuh_layer_id of layerId (when present) with specific There are no instances where nuh_layer_id equals layerId.

[0346] The nal_unit_type is equal to that of CRA_NUT.

[0347]

[0348] Each layer in CVS has And each current The encoded and decoded images in the image are all

[0349] The nal_unit_type of each VCL NAL unit is equal to the image of GDR_NUT.

[0350]

[0351] The nal_unit_type of each VCL NAL unit is an IRAP image that is equal to IDR_W_RADL or IDR_N_LP.

[0352]

[0353] all Having the same nal_unit_type value in the range from IDR_W_RADL to CRA_NUT (inclusive).

[0354] In order of output Previous images.

[0355] Output from DPB

[0356] The nal_unit_type of each VCL NAL unit is equal to the image of RADL_NUT.

[0357]

[0358] The nal_unit_type of each VCL NAL unit is equal to the image of RASL_NUT.

[0359]

[0360] The nal_unit_type of each VCL NAL unit is equal to STSA_NUT.

[0361]

[0362] The nal_unit_type of each VCL NAL unit is equal to the image of TRAIL_NUT.

[0363] Note – Tail images associated with an IRAP or GDR image must also follow the IRAP or GDR image in decoding order. Images that are both following the associated IRAP or GDR image in output order and preceding the associated IRAP or GDR image in decoding order are not allowed.

[0364]

[0365] Note – Tail images associated with IRAP or GDR sub-images must also follow the IRAP or GDR sub-image in decoding order. Sub-images that are both following the associated IRAP or GDR sub-image in output order and preceding the associated IRAP or GDR sub-image in decoding order are not allowed. ...

[0367] 7.4.2.2 NAL Unit Header Semantics ...

[0369] Specify the NAL cell type, i.e., the type of RBSP data structure contained in the NAL cell as specified in Table 5.

[0370] NAL units have a nal_unit_type in the range UNSPEC_28..UNSPEC_31 (including endpoints), but their semantics are not specified and should not affect the decoding process specified in this specification.

[0371] Note 2 – NAL unit types within the range UNSPEC_28..UNSPEC_31 can be used, as determined by the application. The decoding process for these values ​​of nal_unit_type is not specified in this specification. Because different applications may use these NAL unit types for different purposes, special care must be taken when designing encoders that generate NAL units with these nal_unit_type values, and when designing decoders that interpret the content of NAL units with these nal_unit_type values. This specification does not define any management of these values. These nal_unit_type values ​​may only apply in contexts where “conflicts” (i.e., different definitions of the meaning of NAL unit content with the same nal_unit_type value) are not important, impossible, or managed (e.g., defined or managed in control of the application or transport specification, or managed by the environment controlling the distribution of bitstreams).

[0372] For purposes other than determining the amount of data in the DU of the bitstream (as specified in Appendix C), the decoder will ignore (remove and discard) the contents of all NAL units using the reserved value of nal_unit_type.

[0373] Note 3 – This requirement allows for the definition of compatible extensions to this specification in the future.

[0374] Table 5 – NAL Unit Type Codes and NAL Unit Type Categories

[0375]

[0376]

[0377]

[0378] Note 4 – A fully random access (CRA) picture can have an associated RASL or RADL picture that exists in the bitstream.

[0379] Note 5 – An Instantaneous Decode Refresh (IDR) picture with nal_unit_type equal to IDR_N_LP does not have an associated preamble picture present in the bitstream. An IDR picture with nal_unit_type equal to IDR_W_RADL does not have an associated RASL picture present in the bitstream, but may have an associated RADL picture present in the bitstream.

[0380] The value of nal_unit_type should be the same for all VCL NAL units in a sub-image. A sub-image is considered to have the same NAL unit type as the VCL NAL units in the sub-image.

[0381]

[0382] For any specific image's VCL NAL unit, the following applies:

[0383] – If mixed_nalu_types_in_pic_flag equals 0, then the value of nal_unit_type should be the same for all VCL NAL units of the picture, and the picture or PU is considered to have the same NAL unit type as the VCL NAL unit of the picture or PU.

[0384] Otherwise (if mixed_nalu_types_in_pic_flag equals 1), the picture should have at least two subpicks, and the VCL NAL units of the picture should have exactly two different nal_unit_type values, as follows: the VCL NAL units of at least one subpick of the picture should each have a specific value of nal_unit_type, equal to STSA_NUT, RADL_NUT, RASL_NUT, IDR_W_RADL, IDR_N_LP, or CRA_NUT, while the VCL NAL units of the other subpicks in the picture should each have a different specific value of nal_unit_type, equal to TRAIL_NUT, RADL_NUT, or RASL_NUT.

[0385] The requirement for bitstream consistency applies to the following constraints:

[0386] – The tail image should be placed after the associated IRAP or GDR image in the output order.

[0387]

[0388] – When the image is a leading image of an IRAP image, it should be a RADL or RASL image.

[0389]

[0390] – The bitstream should not contain RASL images associated with IDR images.

[0391]

[0392] – There should be no RADL images in the bitstream associated with an IDR image whose nal_unit_type is equal to IDR_N_LP.

[0393] Note 6 – By discarding all PUs preceding the IRAP PU, random access can be performed at the location of the IRAP PU (and the IRAP picture and all subsequent non-RASL pictures can be correctly decoded in the decoding order), provided that each parameter set is available when referenced (either in the bitstream or via an external means not specified in this specification).

[0394]

[0395] - Any image whose nuh_layer_id equals a specific value of layerId, preceding the IRAP image whose nuh_layer_id is equal to layerId, should be in the output order before the IRAP image and all its associated RADL images.

[0396]

[0397] - In decoding order, any image whose nuh_layer_id equals a specific value of layerId should precede the GDP image and all its associated images in the output order.

[0398]

[0399] - Any RASL images associated with a CRA image should precede any RADL images associated with a CRA image in the output order.

[0400]

[0401] – Any RASL images associated with a CRA image should be output in the order of output and followed by any IRAP images that are decoded before the CRA image.

[0402]

[0403]

[0404] – If field_seq_flag equals 0 and the current image with nuh_layer_id equal to a specific value layerId is a preceding image associated with an IRAP image, then it should precede all non-preceding images associated with the same IRAP image in decoding order. Otherwise, assuming picA and picB are the first and last preceding images associated with an IRAP image in decoding order, there should be at most one non-preceding image with nuh_layer_id equal to layerId preceding picA in decoding order, and there should be no non-preceding image with nuh_layer_id equal to layerId between picA and picB in decoding order.

[0405]

[0406] 7.4.3.4 Image Parameter Set Semantics ...

[0408] An expression equal to 1 indicates that each image in the reference PPS has more than one VCL NAL unit. The VCL NAL units do not have the same nal_unit_type value [[, and the image is not an IRAP image]]. A mixed_nalu_types_in_pic_flag equal to 0 indicates that each image in the reference PPS has one or more VCL NAL units, and that the VCL NAL units of each image in the reference PPS have the same nal_unit_type value.

[0409] When no_mixed_nalu_types_in_pic_constraint_flag equals 1, the value of mixed_nalu_types_in_pic_flag should be equal to 0.

[0410] For each stripe in image picA that has a nal_unit_type value nalUnitTypeA in the range IDR_W_RADL to CRA_NUT (inclusive) (i.e., the value of mixed_nalu_types_in_pic_flag in image picA is equal to 1), image picA also contains one or more stripes with another nal_unit_type value, the following applies:

[0411] – The stripe should belong to the subpicA whose corresponding subpic_treated_as_pic_flag[i] value is 1.

[0412] – A stripe should not belong to a subpico of a picA containing a VCL NAL unit whose nal_unit_type is not equal to nalUnitTypeA.

[0413] – If nalUnitTypeA equals CRA, then for all subsequent PUs in CLVS after the current picture in the decoding and output order, neither RefPicList[0] nor RefPicList[1] of the stripes in subpicA of these PUs should include any picture in valid entries that precedes picA in the decoding order.

[0414] – Otherwise (i.e., nalUnitTypeA equals IDR_W_RADL or IDR_N_LP), for all PUs in the CLVS following the current picture in decoding order, neither RefPicList[0] nor RefPicList[1] of the stripes in subpicA of these PUs should include any picture preceding picA in decoding order among the valid entries. ...

[0416] 7.4.3.7 Image Header Structure Semantics ...

[0418] Specify the recovery points for the decoded images in the output order.

[0419]

[0420] If the current image is a GDR image [[associated with PH]], and exists in CLVS after the current GDR image in decoding order, and PicOrderCntVal equals ... If the image picA is the sum of the current GDR image's PicOrderCntVal and the recovery_poc_cnt value, then image picA is called the recovery point image. Otherwise, if PicOrderCntVal is greater than... The first image in the output order of [[the current image's PicOrderCntVal plus the value of recovery_poc_cnt]] is called the recovery point image. The recovery point image should not precede the current GDR image in the decoding order. The value of recovery_poc_cnt should be in the range of 0 to MaxPicOrderCntLsb-1 (inclusive).

[0421] When the current image is a GDR image, the variable RpPicOrderCntVal is derived as follows:

[0422] RpPicOrderCntVal=PicOrderCntVal+recovery_poc_cnt(81)]]

[0423] Note 2 – When gdr_enabled_flag equals 1 and the current image's PicOrderCntVal is greater than or equal to the associated GDR image's When [[RpPicOrderCntVal]] is used, the current and subsequent decoded images in the output order are completely matched with the corresponding images generated by starting the decoding process from the previous IRAP image (if present) that precedes the associated GDR image in the decoding order. ...

[0425] 8.3.2 Decoding process based on the reference image list ...

[0427] The requirement for bitstream consistency applies to the following constraints:

[0428] – For each i equal to 0 or 1, num_ref_entries[i][RplsIdx[i]] should not be less than NumRefIdxActive[i].

[0429] –The image referred to by each valid entry in RefPicList[0] or RefPicList[1] should exist in the DPB and should have a TemporalId less than or equal to the TemporalId of the current image.

[0430] –The image referred to by each entry in RefPicList[0] or RefPicList[1] should not be the current image, and non_reference_picture_flag should be equal to 0.

[0431] – A STRP entry in RefPicList[0] or RefPicList[1] of a strip of an image and an LTRP entry in RefPicList[0] or RefPicList[1] of different strips of the same strip or the same image should not refer to the same image.

[0432] – There should be no LTRP entry in RefPicList[0] or RefPicList[1], for which the difference between the PicOrderCntVal of the current image and the PicOrderCntVal of the image referred to by the entry is greater than or equal to 2. 24 .

[0433] – Let setOfRefPics be the set of unique images referred to by all entries in RefPicList[0] that have the same nuh_layer_id as the current image and all entries in RefPicList[1] that have the same nuh_layer_id as the current image. The number of images in setOfRefPics should be less than or equal to MaxDpbSize-1 (inclusive), where MaxDpbSize is specified in Clause A.4.2, and setOfRefPics should be the same for all stripes of the image.

[0434] – When the nal_unit_type of the current stripe is equal to STSA_NUT, there should be no valid entries in RefPicList[0] or RefPicList[1] that have a TemporalId equal to the TemporalId of the current image and a nuh_layer_id equal to the nuh_layer_id of the current image.

[0435] – When the current image is a picture that is after the STSA picture in the decoding order and has a TemporalId equal to the TemporalId of the current image and a nuh_layer_id equal to the nuh_layer_id of the current image, there should be no picture that is before the STSA picture in the decoding order and has a TemporalId equal to the TemporalId of the current image and a nuh_layer_id equal to the nuh_layer_id of the current image. Such picture should be included as a valid entry in RefPicList[0] or RefPicList[1].

[0436]

[0437] – When the current image for which numh_layer_id equals a specific value layerId is a CRA image, there should be no image referred to by an entry in RefPicList[0] or RefPicList[1] that precedes any IRAP image for which numh_layer_id equals a specific value layerId in output or decoding order (if such an image exists).

[0438]

[0439] – When the current image for which nuh_layer_id equals a specific value layerId is not a RASL image associated with a CRA image where NoOutputBeforeRecoveryFlag equals 1, a GDR image where NoOutputBeforeRecoveryFlag equals 1, or a GDR image where NoOutputBeforeRecoveryFlag equals 1 and nuh_layer_id equals layerId, there should be no image generated by the decoding process used to generate the unusable reference image, as referred to by a valid entry in RefPicList[0] or RefPicList[1].

[0440]

[0441]

[0442] – When the current sub-image of nuh_layer_id equal to a specific value layerId is not a CRA image with NoOutputBeforeRecoveryFlag equal to 1, a preceding image associated with the same CRA image with NoOutputBeforeRecoveryFlag equal to 1 in the decoding order, a preceding image associated with a CRA image with NoOutputBeforeRecoveryFlag equal to 1, a GDR image with NoOutputBeforeRecoveryFlag equal to 1, or a GDR image with NoOutputBeforeRecoveryFlag equal to 1 and nuh_layer_id equal to layerId, there should be no image generated by the decoding process used to generate an unusable reference image as referred to by an entry in RefPicList[0] or RefPicList[1].

[0443]

[0444] – When the current image is associated with an IRAP image and is after the IRAP image in output order, there should be no image referred to by a valid entry in RefPicList[0] or RefPicList[1] that is before the associated IRAP image in output or decoding order.

[0445]

[0446] – When the current image is associated with an IRAP image, is after the IRAP image in output order, and is after the preceding image (if any) associated with the same IRAP image in both decoding and output order, there should be no image referred to by an entry in RefPicList[0] or RefPicList[1] that is before the associated IRAP image in output or decoding order.

[0447]

[0448] – When the current image is a RADL image, there should be no valid entries of the following in RefPicList[0] or RefPicList[1]:

[0449] oRASL image

[0450] o Images preceding the associated IRAP images in the decoding order

[0451]

[0452] – The image referred to by each ILRP entry in RefPicList[0] or RefPicList[1] of the current image strip should be in the same AU as the current image.

[0453] – The image referred to by each ILRP entry in RefPicList[0] or RefPicList[1] of the current image strip should exist in the DPB and should have a nuh_layer_id that is less than the nuh_layer_id of the current image.

[0454] – Each ILRP entry in the stripe’s RefPicList[0] or RefPicList[1] should be a valid entry.

[0455] Figure 5 This is a block diagram illustrating an example video processing system 1900, in which various techniques disclosed herein can be implemented. Various implementations may include some or all of the components of system 1900. System 1900 may include an input 1902 for receiving video content. The video content may be received in a raw or uncompressed format, such as 8 or 10-bit multi-component pixel values, or it may be in a compressed or encoded format. Input 1902 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces (such as Ethernet, Passive Optical Networking (PON), etc.) and wireless interfaces (such as Wi-Fi or cellular interfaces).

[0456] System 1900 may include codec component 1904, which may implement various codec or encoding methods described in this document. Codec component 1904 may reduce the average bit rate of the video from input 1902 to the output of codec component 1904 to produce a codec representation of the video. Therefore, codec techniques are sometimes referred to as video compression or video transcoding techniques. As indicated by component 1906, the output of codec component 1904 may be stored or transmitted via connected communication. Component 1908 may use the stored or communicated bitstream (or codec) representation of the video received at input 1902 to generate pixel values ​​or displayable video sent to display interface 1910. The process of generating user-visible video from the bitstream representation is sometimes referred to as video decompression. Furthermore, although some video processing operations are referred to as “codec” operations or tools, it should be understood that codec tools or operations are used at the encoder, and corresponding decoding tools or operations, the opposite of the encoding result, will be performed by the decoder.

[0457] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or DisplayPort. Examples of storage interfaces include SATA (Serial Advanced Technology Accessory), PCI, IDE, etc. The technologies described in this document can be implemented in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of performing digital data processing and / or video display.

[0458] Figure 6 This is a block diagram of a video processing apparatus 3600. Apparatus 3600 can be used to implement one or more methods described herein. Apparatus 3600 can be implemented in smartphones, tablets, computers, Internet of Things (IoT) receivers, etc. Apparatus 3600 may include one or more processors 3602, one or more memories 3604, and video processing hardware 3606. Processor 3602 can be configured to implement one or more methods described in this document. Memory (multiple memories) 3604 can be used to store data and code for implementing the methods and techniques described herein. Video processing hardware 3606 can be used to implement some of the techniques described in this document in hardware circuitry.

[0459] Figure 8 This is a block diagram illustrating an example video codec system 100 that can utilize the techniques disclosed herein.

[0460] like Figure 8 As shown, the video encoding / decoding system 100 may include a source device 110 and a destination device 120. The source device 110 generates encoded video data and may be referred to as a video encoding device. The destination device 120 can decode the encoded video data generated by the source device 110 and may be referred to as a video decoding device.

[0461] The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0462] Video source 112 may include sources such as video capture devices, interfaces for receiving video data from video content providers, and / or computer graphics systems used to generate video data, or combinations of these sources. Video data may include one or more pictures. Video encoder 114 encodes the video data from video source 112 to generate a bitstream. The bitstream may include a sequence of bits forming a codec representation of the video data. The bitstream may include codec pictures and associated data. A codec picture is a codec representation of a picture. Associated data may include sequence parameter sets, picture parameter sets, and other syntax structures. I / O interface 116 may include a modulator / demodulator (modem) and / or a transmitter. Encoded video data may be transmitted directly to destination device 120 via network 130a through I / O interface 116. Encoded video data may also be stored on storage medium / server 130b for access by destination device 120.

[0463] Destination device 120 may include I / O interface 126, video decoder 124 and display device 122.

[0464] I / O interface 126 may include a receiver and / or a modem. I / O interface 126 may acquire encoded video data from source device 110 or storage medium / server 130b. Video decoder 124 may decode the encoded video data. Display device 122 may display the decoded video data to a user. Display device 122 may be integrated with destination device 120, or it may be external to destination device 120, which is configured to interface with an external display device.

[0465] The video encoder 114 and the video decoder 124 can operate according to video compression standards, such as the High Efficiency Video Coding (HEVC) standard, the Multi-Function Video Coding (VVM) standard, and other current and / or further standards.

[0466] Figure 9 This is a block diagram illustrating an example of a video encoder 200, which may be... Figure 8 The video encoder 114 in the system 100 shown.

[0467] The video encoder 200 can be configured to perform any or all of the techniques disclosed herein. Figure 9In the example, the video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video encoder 200. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0468] The functional components of the video encoder 200 may include a segmentation unit 201, a prediction unit 202 that may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205 and an intra-frame prediction unit 206, a residual generation unit 207, a transform unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transform unit 211, a reconstruction unit 212, a buffer 213 and an entropy coding unit 214.

[0469] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra-block copy (IBC) unit. The IBC unit can perform prediction in IBC mode, where at least one reference picture is the picture containing the current video block.

[0470] Furthermore, some components, such as the motion estimation unit 204 and the motion compensation unit 205, can be highly integrated, but for illustrative purposes, in Figure 9 The examples are shown separately.

[0471] The segmentation unit 201 can segment an image into one or more video blocks. The video encoder 200 and the video decoder 300 can support various video block sizes.

[0472] The mode selection unit 203 may, for example, select one of the coding modes (intra-frame or inter-frame) based on the error result, and provide the resulting intra-frame or inter-frame coded block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 203 may select a combination of intra-frame and inter-frame prediction (CIIP) modes, where the prediction is based on the inter-frame prediction signal and the intra-frame prediction signal. In the case of inter-frame prediction, the mode selection unit 203 may also select the precision of the motion vector for the block (e.g., sub-pixel or integer pixel precision).

[0473] To perform inter-frame prediction on the current video block, motion estimation unit 204 can generate motion information for the current video block by comparing one or more reference frames from buffer 213 with the current video block. Motion compensation unit 205 can determine the predicted video block for the current video block based on motion information and decoded samples from images other than those associated with the current video block from buffer 213.

[0474] The motion estimation unit 204 and the motion compensation unit 205 can perform different operations on the current video block, for example, depending on whether the current video block is in an I-band, P-band, or B-band.

[0475] In some examples, motion estimation unit 204 can perform unidirectional prediction on the current video block, and can search for a reference video block for the current video block in the reference images of list 0 or list 1. Motion estimation unit 204 can then generate a reference index indicating a reference image in list 0 or list 1 containing the reference video block, and a motion vector indicating the spatial displacement between the current video block and the reference video block. Motion estimation unit 204 can output the reference index, prediction direction indicator, and motion vector as motion information for the current video block. Motion compensation unit 205 can generate a predicted video block for the current block based on the reference video block indicated by the motion information of the current video block.

[0476] In other examples, motion estimation unit 204 can perform bidirectional prediction on the current video block. Motion estimation unit 204 can search for a reference video block for the current video block in the reference images in list 0, and can also search for another reference video block for the current video block in the reference images in list 1. Motion estimation unit 204 can then generate a reference index and a motion vector, where the reference index indicates a reference image in list 0 or list 1 containing the reference video block, and the motion vector indicates the spatial displacement between the reference video block and the current video block. Motion estimation unit 204 can output the reference index and motion vector of the current video block as motion information for the current video block. Motion compensation unit 205 can generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0477] In some examples, the motion estimation unit 204 can output the complete set of motion information for the decoder's decoding processing.

[0478] In some examples, motion estimation unit 204 may not output the complete set of motion information for the current video. Instead, motion estimation unit 204 can signal the motion information of the current video block by referencing the motion information of another video block. For example, motion estimation unit 204 may determine that the motion information of the current video block is sufficiently similar to the motion information of neighboring video blocks.

[0479] In one example, the motion estimation unit 204 may indicate a value in the syntax structure associated with the current video block that indicates to the video decoder 300 that the current video block has the same motion information as another video block.

[0480] In another example, motion estimation unit 204 can identify another video block and motion vector difference (MVD) in the syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. Video decoder 300 can use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0481] As described above, the video encoder 200 can predictively signal motion vectors. Two examples of predictive signaling techniques that can be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and Combined Mode Signaling.

[0482] Intra-prediction unit 206 can perform intra-prediction on the current video block. When intra-prediction unit 206 performs intra-prediction on the current video block, it can generate prediction data for the current video block based on decoded samples from other video blocks in the same frame. The prediction data for the current video block can include the predicted video block and various syntax elements.

[0483] The residual generation unit 207 can generate residual data for the current video block by subtracting (e.g., indicated by a negative sign) the predicted video block of the current video block from the current video block. The residual data for the current video block may include residual video blocks corresponding to different sample components of the samples in the current video block.

[0484] In other examples, there may be no residual data for the current video block, for example, in skip mode, and the residual generation unit 207 may not perform the subtraction operation.

[0485] The transform processing unit 208 can generate one or more transform coefficient video blocks of the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0486] After the transform processing unit 208 generates a transform coefficient video block associated with the current video block, the quantization unit 209 can quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0487] Inverse quantization unit 210 and inverse transform unit 211 can apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct the residual video block from the transform coefficient video block. Reconstruction unit 212 can add the reconstructed residual video block to the corresponding samples from one or more predicted video blocks generated by prediction unit 202 to produce a reconstructed video block associated with the current block, which is then stored in buffer 213.

[0488] After the video block is reconstructed by the reconstruction unit 212, a loop filtering operation can be performed to reduce the video block effect in the video block.

[0489] Entropy encoding unit 214 can receive data from other functional components of video encoder 200. When entropy encoding unit 214 receives data, it can perform one or more entropy encoding operations to generate entropy encoded data and output a bit stream including the entropy encoded data.

[0490] Figure 10 This is a block diagram illustrating an example of a video decoder 300, which may be... Figure 8 The video decoder 114 in the system 100 shown.

[0491] The video decoder 300 can be configured to perform any or all of the technologies disclosed herein. Figure 10 In the example, the video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video decoder 300. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0492] exist Figure 10 In the example, the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra-frame prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, a reconstruction unit 306, and a buffer 307. In some examples, the video decoder 300 can perform functions typically associated with the video encoder 200. Figure 9 The decoding process is the inverse of the encoding process described.

[0493] Entropy decoding unit 301 can retrieve encoded bitstreams. The encoded bitstreams may include entropy-coded video data (e.g., encoded blocks of video data). Entropy decoding unit 301 can decode the entropy-coded video data, and motion compensation unit 302 can determine motion information from the entropy-decoded video data, including motion vectors, motion vector precision, reference image list index, and other motion information. Motion compensation unit 302 can determine this information, for example, by performing AMVP and merging modes.

[0494] Motion compensation unit 302 can generate motion compensation blocks, possibly performing interpolation based on an interpolation filter. The syntax elements can include identifiers of the interpolation filters to be used with sub-pixel precision.

[0495] The motion compensation unit 302 can use interpolation filters, such as those used by the video encoder 200 during the encoding of video blocks, to calculate interpolated values ​​for sub-integer pixels of the reference block. The motion compensation unit 302 can determine the interpolation filter used by the video encoder 200 based on the received syntax information and use the interpolation filter to generate a prediction block.

[0496] The motion compensation unit 302 can use some syntax information to determine the size of the blocks of (multiple) frames and / or (multiple) strips used to encode the encoded video sequence, segmentation information describing how each macroblock of the image of the encoded video sequence is segmented, a pattern indicating how each segment is encoded, one or more reference frames (and a list of reference frames) for each inter-frame coded block, and other information for decoding the encoded video sequence.

[0497] Intra-prediction unit 303 can use, for example, an intra-prediction mode received in the bitstream to form prediction blocks from spatially adjacent blocks. Inverse quantization unit 303 inverse quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 301. Inverse transform unit 303 applies an inverse transform.

[0498] The reconstruction unit 306 can add the residual block to the corresponding prediction block generated by the motion compensation unit 202 or the intra-frame prediction unit 303 to form a decoded block. If necessary, a deblocking filter can also be applied to filter the decoded block to remove block artifacts. The decoded video block is then stored in a buffer 307, which provides a reference block for subsequent motion compensation / intra-frame prediction and also generates decoded video for presentation on a display device.

[0499] The following is a list of preferred embodiments.

[0500] The first set of clauses illustrates example embodiments of the techniques discussed in the preceding section (e.g., item 1).

[0501] 1. A video processing method (e.g., Figure 7 The method 700 described in the text includes: performing (702) a conversion between a video and a video codec representation comprising one or more images containing one or more sub-images, wherein the codec representation conforms to a format rule specifying that one or more images comprising one or more sub-images are included in a codec representation according to a Network Abstraction Layer (NAL) unit, wherein the type NAL unit indicated in the codec representation includes a codec stripe of a specific type of image or a codec stripe of a specific type of sub-image.

[0502] The following clauses illustrate example embodiments of the techniques discussed in the preceding section (e.g., item 2).

[0503] 2. A video processing method, comprising: performing a conversion between a video and a video codec representation comprising one or more images containing one or more sub-images, wherein the codec representation conforms to a format rule specifying that two adjacent sub-images with different Network Abstraction Layer Unit types will have the same indication of the sub-images as image markers.

[0504] The following clauses illustrate example embodiments of the techniques discussed in the preceding sections (e.g., items 4, 5, 6, 7, 9, 1, 11, 12).

[0505] 3. A video processing method, comprising: performing a conversion between a video and a video codec representation comprising one or more images containing one or more sub-images, wherein the codec representation conforms to a format rule defining the order of sub-images of a first type and sub-images of a second type, wherein the first sub-image is a tail sub-image, a leading sub-image, or a random access skip leading (RASL) sub-image type, and the second sub-image is a RASL type, a random access decodable leading (RADL) type, an instantaneous decode refresh (IDR) type, or a gradually decode refresh (GDR) type sub-image.

[0506] 4. The method according to Clause 3, wherein the rule specifies that the tail sub-picture follows the associated intra-frame random access point or GDR sub-picture in the output order.

[0507] 5. The method described in Clause 3, wherein the rule specifies that when the image is a preceding image of an intra-frame random access point image, the nal_unit_type value of all network abstraction layer units in the image is equal to RADL_NUT or RASL_NUT.

[0508] 6. The method described in Clause 3, wherein the rule specifies that a given subpicture which is a preceding subpicture of an IRAP subpicture must also be a RADL or RASL subpicture.

[0509] 7. The method described in Clause 3, wherein the rule specifies that a given sub-image as a RASL sub-image is not allowed to be associated with an IDR sub-image.

[0510] 8. The method described in Clause 3, wherein the rule specifies that a given subpicture having the same layer id and subpicture index as the IRAP subpicture must precede the IRAP subpicture and all its associated RADL subpictures in the output order.

[0511] 9. The method described in Clause 3, wherein the rule specifies that a given sub-image with the same layer id and sub-image index as the GDR sub-image must precede the GDR sub-image and all its associated RADL sub-images in the output order.

[0512] 10. The method according to Clause 3, wherein the rule specifies that a given subpick as a RASL subpick associated with a CRA subpick precedes all RADL subpicks associated with the CRA subpick in the output order.

[0513] 11. The method according to Clause 3, wherein the rule specifies that a given sub-picture, as a RASL sub-picture associated with a CRA sub-picture, precedes all IRAP sub-pictures associated with the CRA sub-picture in the output order.

[0514] 12. The method described in Clause 3, wherein the rule specifies that a given subpicture is a preceding subpicture associated with an IRAP subpicture, then the given subpicture precedes all non-preceding subpictures associated with the IRAP subpicture in the decoding order.

[0515] The following clauses illustrate example embodiments of the techniques discussed in the preceding sections (e.g., items 8, 14, 15).

[0516] 13. A video processing method, comprising: performing a conversion between a video and a video codec representation comprising one or more images containing one or more sub-images, wherein the codec representation conforms to a format rule that defines conditions that allow or disallow the appearance of a first type of sub-image with a second type of sub-image.

[0517] 14. The method described in Clause 13, wherein the rule stipulates that in the presence of an IDR subpicture of network abstraction layer type IDR_N_LP, the codec representation is not allowed to have a RADP subpicture.

[0518] 15. The method described in Clause 13, wherein the rule disallows including an image in a reference list of images containing a Stepwise Temporal Sublayer Access (STSA) sub-image, such that the image precedes the image containing the STSA sub-image.

[0519] 16. The method according to Clause 13, wherein the rule disallows including a picture in the reference list of pictures containing an Intra-Random Access Point (IRAP) subpicture, such that the picture precedes the picture containing the IRAP subpicture.

[0520] 17. The method according to any one of Clauses 1 to 16, wherein the conversion includes encoding the video into a codec representation.

[0521] 18. The method according to any one of clauses 1 to 16, wherein the conversion includes decoding the codec representation to generate pixel values ​​of the video.

[0522] 19. A video decoding apparatus, including a processor configured to implement one or more of the methods described in Clauses 1 to 18.

[0523] 20. A video encoding apparatus, including a processor configured to implement one or more of the methods described in Clauses 1 to 18.

[0524] 21. A computer program product having computer code stored thereon, which, when executed by a processor, causes the processor to perform the method described in any one of clauses 1 to 18.

[0525] 22. The methods, apparatus or systems described in this document.

[0526] The second set of clauses illustrates example embodiments of the techniques discussed in the previous section (e.g., items 8-11).

[0527] 1. A method for video processing (e.g., such as...) Figure 11 The method 1100 shown includes: performing a conversion between a video and a video bitstream comprising one or more layers, according to a format rule, and wherein the format rule specifies that the reference picture referred to by each inter-layer reference picture entry in the reference picture list of the current picture of the current layer satisfies a constraint, wherein the constraint is at least one of the following: (a) the reference picture is an intra-frame random access (IRAP) picture, or (b) the reference picture has a temporal identifier less than or equal to a specific value, which is the maximum permissible value of video layers that can be referenced according to the current layer's strip, wherein the maximum permissible value is indicated in the syntax element.

[0528] 2. The method described in Clause 1, wherein one or more indices of the syntax element are layer indices.

[0529] 3. The method described in clause 1 or 2, wherein the syntax element is a two-dimensional syntax element.

[0530] 4. The method according to any one of clauses 1 to 3, wherein the time domain identifier is TemporalId and the syntax element is max_tid_il_ref_pics_plus1[i][j], where i and j are integers.

[0531] 5. The method according to any one of clauses 1 to 3, wherein the specific value corresponds to the value of Max(0, max_tid_il_ref_pics_plus1[currLayerIdx][refLayerIdx]-1), wherein currLayerIdx and refLayerIdx are equal to GeneralLayerIdx[nuh_layer_id] and GeneralLayerIdx[refpicLayerId], respectively, GeneralLayerIdx[] specifies the layer index of the layer having a specific layer identifier corresponding to nuh_layer_id, and refpicLayerId is the layer identifier of the image.

[0532] 6. The method according to Clause 5, wherein the format rule further specifies that the image referred to by each inter-layer reference image entry exists in the decoded image buffer and has fewer layer identifiers than the current image's layer identifier.

[0533] 7. The method according to any one of clauses 1 to 3, wherein the specific value corresponds to the value of max_tid_il_ref_pics_plus1[currLayerIdx][refLayerIdx], wherein currLayerIdx and refLayerIdx are equal to GeneralLayerIdx[nuh_layer_id] and GeneralLayerIdx[refpicLayerId], respectively, GeneralLayerIdx[] specifies the layer index of the layer having a specific layer identifier corresponding to nuh_layer_id, and refpicLayerId is the layer identifier of the image.

[0534] 8. The method according to Clause 7, wherein the rule further specifies that the image referred to by each inter-layer reference image entry exists in the decoded image buffer and has fewer layer identifiers than the current image's layer identifier.

[0535] 9. The method according to any one of Clauses 1 to 8, wherein the conversion includes encoding the video into a bitstream.

[0536] 10. The method according to any one of clauses 1 to 8, wherein the conversion includes decoding video from a bitstream.

[0537] 11. The method according to any one of clauses 1 to 8, wherein the conversion includes generating a bitstream from video, and the method further includes storing the bitstream in a non-transitory computer-readable recording medium.

[0538] 12. A video processing apparatus, comprising a processor configured to implement the method described in any one or more of clauses 1 to 11.

[0539] 13. A method for storing a video bitstream, comprising the method of any one of clauses 1 to 11, and further comprising storing the bitstream to a non-transitory computer-readable recording medium.

[0540] 14. A computer-readable medium storing program code that, when executed, causes a processor to perform any one or more of the methods described in clauses 1 to 11.

[0541] 15. A computer-readable medium for storing a bit stream generated according to any of the above methods.

[0542] 16. A video processing apparatus for storing bit streams, wherein the video processing apparatus is configured to implement the method described in any one or more of clauses 1 to 11.

[0543] The third set of clauses illustrates example embodiments of the techniques discussed in the previous section (e.g., item 12).

[0544] 1. A method for video processing (e.g., such as...) Figure 12 The method 1200 shown includes: performing a conversion between a video and a bitstream, including a video image, according to a format rule, wherein the format rule specifies the use of one or more of a first syntax structure indicating the position of a vertical virtual boundary and a second syntax structure indicating the position of a horizontal virtual boundary to indicate the virtual boundaries of a video region of the video image, wherein the format rule specifies that the values ​​indicated by the first syntax element in the first syntax structure and the second syntax element in the second syntax structure are 1 less than the positions of the vertical and horizontal virtual boundaries counted from the top left position of the video image.

[0545] 2. The method according to Clause 1, wherein the first syntax element includes sps_virtual_boundary_pos_x_minus1[i] and the second syntax element includes sps_virtual_boundary_pos_y_minus1[i], where i is an integer.

[0546] 3. The method according to Clause 1, wherein the first syntax element includes ph_virtual_boundary_pos_x_minus1[i] and the second syntax element includes ph_virtual_boundary_pos_y_minus1[i], where i is an integer.

[0547] 4. The method according to any one of clauses 1 to 3, wherein at least one of a first syntax element and a second syntax element is encoded or decoded using a descriptor, said descriptor being ue(v) or u(v).

[0548] 5. The method according to any one of clauses 1-3, wherein the first syntax element includes sps_virtual_boundary_pos_x_minus1[i] plus 1, sps_virtual_boundary_pos_x_minus1[i] plus 1 specifies the position of the i-th vertical virtual boundary divided by 8 in luminance samples, and has a value in the range of 0 to Ceil(pic_width_max_in_luma_samples÷8)-2, wherein pic_width_max_in_luma_samples indicates the maximum width of the picture in luminance samples.

[0549] 6. The method according to any one of clauses 1-3, wherein the second syntax element includes sps_virtual_boundary_pos_y_minus1[i] plus 1, sps_virtual_boundary_pos_y_minus1[i] plus 1 specifies the position of the i-th horizontal virtual boundary divided by 8 in luminance samples, and has a value in the range of 0 to Ceil(pic_height_max_in_luma_samples÷8)-2, wherein pic_height_max_in_luma_samples indicates the maximum height of the picture in luminance samples.

[0550] 7. The method according to any one of clauses 1-3, wherein the first syntax element includes ph_virtual_boundary_pos_x_minus1[i] plus 1, ph_virtual_boundary_pos_x_minus1[i] plus 1 specifies the position of the i-th vertical virtual boundary divided by 8 in luminance samples, and has a value in the range of 0 to Ceil(pic_width_in_luma_samples÷8)-2, wherein pic_width_in_luma_samples indicates the width of the picture in luminance samples.

[0551] 8. According to the method described in Clause 7, wherein for the range of i from 0 to NumVerVirtualBoundaries-1, the list VirtualBoundariesPosX[i] in units of luminance samples, specifying the position of the vertical virtual boundary, is derived using the following condition: VirtualBoundariesPosX[i] = (sps_virtual_boundaries_present_flag?(sps_virtual_boundary_pos_x_minus1[i]+1):(ph_virtual_boundary_pos_x_minus1[i]+1))*8.

[0552] 9. The method according to Clause 8, wherein the distance between any two vertical virtual boundaries is greater than or equal to CtbSizeY luminance samples, where CtbSizeY indicates the size of the codec tree unit.

[0553] 10. The method according to any one of clauses 1-3, wherein the second syntax element includes ph_virtual_boundary_pos_y_minus1[i] plus 1 specifying the position of the i-th horizontal virtual boundary divided by 8 in luminance samples, and has a value in the range of 0 to Ceil(pic_height_in_luma_samples÷8)-2, wherein pic_height_in_luma_samples indicates the height of the picture in luminance samples.

[0554] 11. The method according to Clause 10, wherein for i ranging from 0 to NumVerVirtualBoundaries-1, the list VirtualBoundariesPosX[i] in units of luminance samples, specifying the location of the vertical virtual boundary, is derived using the following condition: VirtualBoundariesPosY[i] = (sps_virtual_boundaries_present_flag?(sps_virtual_boundary_pos_y_minus1[i]+1):(ph_virtual_boundary_pos_y_minus1[i]+1))*8.

[0555] 12. The method according to Clause 11, wherein the distance between any two horizontal virtual boundaries is greater than or equal to CtbSizeY luminance samples, where CtbSizeY indicates the size of the codec tree unit.

[0556] 13. The method according to any one of Clauses 1 to 12, wherein the vertical virtual boundary and the horizontal virtual boundary correspond to the boundary between the refreshed area and the unrefreshed area within the video frame.

[0557] 14. The method according to any one of clauses 1 to 13, wherein the vertical virtual boundary and the horizontal virtual boundary correspond to the boundary where no loop filtering is applied during the conversion.

[0558] 15. The method according to any one of clauses 1 to 14, wherein the conversion includes encoding the video into a bitstream.

[0559] 16. The method according to any one of Clauses 1 to 14, wherein the conversion includes decoding video from a bitstream.

[0560] 17. The method according to clauses 1 to 14, wherein the conversion includes generating a bitstream from video, and the method further includes storing the bitstream in a non-transitory computer-readable recording medium.

[0561] 18. A video processing apparatus, comprising a processor configured to implement the method described in any one or more of Clauses 1 to 17.

[0562] 19. A method for storing a video bitstream, comprising the method of any one of clauses 1 to 17, and further comprising storing the bitstream to a non-transitory computer-readable recording medium.

[0563] 20. A computer-readable medium storing program code that, when executed, causes a processor to perform any one or more of the methods described in clauses 1 to 17.

[0564] 21. A computer-readable medium for storing a bit stream generated according to any of the above methods.

[0565] 22. A video processing apparatus for storing a bitstream representation, wherein the video processing apparatus is configured to implement the method described in any one or more of clauses 1 to 17.

[0566] In the terms described herein, an encoder can conform to a format rule by generating a codec representation according to the format rule. In the terms described herein, a decoder can use the format rule to parse the syntax elements into a codec representation based on the presence or absence of the syntax elements known according to the format rule, in order to produce a decoded video.

[0567] In this document, the term "video processing" can refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm can be applied during the conversion from the pixel representation of a video to the corresponding bitstream representation, and vice versa. As defined in the syntax, the bitstream representation of the current video block can, for example, correspond to bits that are co-located or scattered at different locations within the bitstream. For example, a macroblock can be encoded based on the error residual values ​​of the transformation and encoding, and also using bits in the header and other fields of the bitstream. Furthermore, during the conversion, the decoder can, based on this determination, parse the bitstream knowing that some fields may or may not be present, as described in the solutions above. Similarly, the encoder can determine whether certain syntax fields are included or excluded, and generate the codec representation accordingly by including or excluding syntax fields from the codec representation.

[0568] The disclosed and other solutions, examples, embodiments, modules, and functional operations described in this document can be implemented in digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in combinations thereof. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by or control of the operation of a data processing apparatus. The computer-readable medium can be a combination of a machine-readable storage device, a machine-readable storage substrate, a memory device, a substance that implements a machine-readable propagating signal, or a combination thereof. The term "data processing apparatus" includes all means, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may include code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination thereof. The propagating signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0569] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language, including compiled or interpreted languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suited to a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored as a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), as a single file dedicated to the program in question, or as portions of multiple collaborative files (e.g., a file storing portions of one or more modules, subroutines, or code). Computer programs can be deployed to execute on a single computer or on multiple computers located in one place or distributed across multiple locations and interconnected via a communication network.

[0570] The processes and logic flows described in this document can be executed by one or more programmable processors that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by dedicated logic circuitry, and the devices can be implemented as dedicated logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits).

[0571] For example, processors suitable for executing computer programs include both general-purpose and special-purpose microprocessors, as well as any one or more processors in any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, to receive data from or transfer data to, or both. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. The processor and memory may be supplemented or incorporated therein by dedicated logic circuitry.

[0572] While this patent document contains numerous details, these details should not be construed as limiting the scope of any subject matter or claimed content, but rather as descriptions of features characteristic of specific embodiments of a particular art. Certain features described in this patent document within the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0573] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order or sequence shown, or requiring all illustrated operations to be performed to obtain the desired result. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[0574] Only some implementation methods and examples are described, and other implementation methods, enhancements and variations can be made based on the content described and shown in this patent document.

Claims

1. A method for processing video data, comprising: Perform conversion between a video containing one or more images and the bitstream of the video according to format rules. The format rules specify that the first image referred to by each inter-layer reference image entry in the reference image list of the current image's strip in the current layer must satisfy the constraints. The constraints include at least one of the following: (a) the first image is an intra-frame random access (IRAP) image, or (b) the temporal identifier of the first image is less than the value of the first syntax element corresponding to the index of the current layer and the index of the reference layer, and The first syntax element is included in the bitstream.

2. The method according to claim 1, wherein, The first syntax element is a two-dimensional syntax element, and the first syntax element corresponding to the index of the current layer and the index of the reference layer is one of a plurality of first syntax elements included in the bitstream that correspond to different indices of the current layer and different indices of the reference layer, respectively.

3. The method according to claim 2, wherein, The first syntax element corresponding to the index of the current layer and the index of the reference layer is represented as max_tid_il_ref_pics_plus1[currLayerIdx][refLayerIdx], where currLayerIdx is equal to GeneralLayerIdx[nuh_layer_id] and refLayerIdx is equal to GeneralLayerIdx[refpicLayerId], where GeneralLayerIdx[] indicates the layer index of the layer with a specific layer identifier, nuh_layer_id is the layer identifier of the current image, and refpicLayerId is the layer identifier of the first image.

4. The method according to claim 3, wherein, The format rules also specify that the first image referred to by each inter-layer reference image entry exists in the decoded image buffer and has a layer identifier smaller than the layer identifier of the current image.

5. The method according to claim 1, wherein, The value of the first syntax element indicates the maximum allowed value of the temporal identifier of the video layer that the current layer's strip can reference.

6. The method according to claim 1, wherein, The formatting rules also specify the use of one or more of a second syntax element indicating the location of the vertical virtual boundary and a third syntax element indicating the location of the horizontal virtual boundary to indicate the virtual boundary of the video area of ​​the second image. The formatting rules further stipulate that the numerical values ​​indicated by the second syntax element are each 1 less than the position of the vertical virtual boundary, and The formatting rules further stipulate that the numerical values ​​indicated by the third syntax element are each 1 less than the position of the horizontal virtual boundary.

7. The method according to claim 6, wherein, The second syntax element includes sps_virtual_boundary_pos_x_minus1[i], and the third syntax element includes sps_virtual_boundary_pos_y_minus1[i]. In this context, incrementing 1 by sps_virtual_boundary_pos_x_minus1[i] indicates dividing the position of the i-th vertical virtual boundary in units of luminance samples by 8. Here, sps_virtual_boundary_pos_x_minus1[i] has a value in the range of 0 to Ceil(pic_width_max_in_luma_samples÷8)-2, where pic_width_max_in_luma_samples indicates the maximum width of the image in luminance samples. Here, `sps_virtual_boundary_pos_y_minus1[i]` incremented by 1 indicates the position of the i-th horizontal virtual boundary in units of luminance samples, divided by 8, and Among them, sps_virtual_boundary_pos_y_minus1[i] has a value in the range of 0 to Ceil(pic_height_max_in_luma_samples÷8)-2, where pic_height_max_in_luma_samples indicates the maximum height of the picture in units of luminance samples.

8. The method according to claim 7, wherein, The second syntax element also includes ph_virtual_boundary_pos_x_minus1[i], and the third syntax element also includes ph_virtual_boundary_pos_y_minus1[i]. In this context, incrementing ph_virtual_boundary_pos_x_minus1[i] by 1 indicates the position of the i-th vertical virtual boundary in units of luminance samples, divided by 8. Wherein, ph_virtual_boundary_pos_x_minus1[i] has a value in the range of 0 to Ceil(pic_width_in_luma_samples÷8)-2, where pic_width_in_luma_samples indicates the width of the image in luminance samples. Here, ph_virtual_boundary_pos_y_minus1[i] plus 1 indicates the position of the i-th horizontal virtual boundary in units of luminance samples divided by 8, and Wherein, ph_virtual_boundary_pos_y_minus1[i] has a value in the range of 0 to Ceil(pic_height_in_luma_samples÷8)-2, where pic_height_in_luma_samples indicates the height of the picture in units of luminance samples.

9. The method according to claim 8, wherein, When the syntax flag represented by `sps_virtual_boundaries_present_flag` is true, for the range `i` from 0 to `NumVerVirtualBoundaries-1`, the list `VirtualBoundariesPosX[i]` indicating the positions of the vertical virtual boundaries in units of luminance samples is equal to `(sps_virtual_boundary_pos_x_minus1[i]+1)`. When the syntax flag represented by `sps_virtual_boundaries_present_flag` is false, `VirtualBoundariesPosX[i]` is equal to `(ph_virtual_boundary_pos_x_minus1[i]+1)*8`, and... Wherein, the distance between any two vertical virtual boundaries is greater than or equal to CtbSizeY luminance samples, where CtbSizeY indicates the size of the codec tree unit.

10. The method according to claim 8, wherein, When the syntax flag represented by `sps_virtual_boundaries_present_flag` is true, for the range `i` from 0 to `NumVerVirtualBoundaries-1`, the list `VirtualBoundariesPosY[i]` indicating the positions of the horizontal virtual boundaries in units of luminance samples is equal to `(sps_virtual_boundary_pos_y_minus1[i]+1)`. When the syntax flag represented by `sps_virtual_boundaries_present_flag` is false, `VirtualBoundariesPosY[i]` is equal to `(ph_virtual_boundary_pos_y_minus1[i]+1)*8`, and... Wherein, the distance between any two horizontal virtual boundaries is greater than or equal to CtbSizeY luminance samples, where CtbSizeY indicates the size of the codec tree unit.

11. The method according to claim 8, wherein, The second and third syntax elements are encoded and decoded using a descriptor, which is ue(v).

12. The method according to claim 6, wherein, Loop filtering is not applied to the vertical virtual boundary and the horizontal virtual boundary.

13. The method according to claim 6, wherein, The vertical virtual boundary and the horizontal virtual boundary correspond to the boundary between the refreshed area and the non-refreshed area within the second image.

14. The method according to claim 1, wherein, The conversion includes encoding the video into the bitstream.

15. The method according to claim 1, wherein, The conversion includes decoding the video from the bitstream.

16. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein, When the instruction is executed by the processor, the processor: Perform conversion between a video containing one or more images and the bitstream of the video according to format rules. The format rules specify that the first image referred to by each inter-layer reference image entry in the reference image list of the current image's strip in the current layer must satisfy the constraints. The constraints include at least one of the following: (a) the first image is an intra-frame random access (IRAP) image, or (b) the temporal identifier of the first image is less than the value of the first syntax element corresponding to the index of the current layer and the index of the reference layer, and The first syntax element is included in the bitstream.

17. The apparatus according to claim 16, wherein, The first syntax element is a two-dimensional syntax element, and the first syntax element corresponding to the index of the current layer and the index of the reference layer is one of a plurality of first syntax elements included in the bitstream that correspond to different indices of the current layer and different indices of the reference layer, respectively. The first syntax element corresponding to the index of the current layer and the index of the reference layer is represented as max_tid_il_ref_pics_plus1[currLayerIdx][refLayerIdx], where currLayerIdx is equal to GeneralLayerIdx[nuh_layer_id] and refLayerIdx is equal to GeneralLayerIdx[refpicLayerId], where GeneralLayerIdx[] indicates the layer index of the layer with a specific layer identifier, nuh_layer_id is the layer identifier of the current image, and refpicLayerId is the layer identifier of the first image. The format rules further specify that the first image referred to by each inter-layer reference image entry exists in the decoded image buffer and has a layer identifier smaller than the layer identifier of the current image.

18. A non-transitory computer-readable storage medium for storing instructions, said instructions causing a processor to: Perform conversion between a video containing one or more images and the bitstream of the video according to format rules. in, The formatting rules stipulate that the first image referred to by each inter-layer reference image entry in the reference image list of the current image's strip in the current layer must satisfy the constraints. The constraints include at least one of the following: (a) the first image is an intra-frame random access (IRAP) image, or (b) the temporal identifier of the first image is less than the value of the first syntax element corresponding to the index of the current layer and the index of the reference layer, and The first syntax element is included in the bitstream.

19. The non-transitory computer-readable storage medium according to claim 18, wherein, The first syntax element is a two-dimensional syntax element, and the first syntax element corresponding to the index of the current layer and the index of the reference layer is one of a plurality of first syntax elements included in the bitstream that correspond to different indices of the current layer and different indices of the reference layer, respectively. The first syntax element corresponding to the index of the current layer and the index of the reference layer is represented as max_tid_il_ref_pics_plus1[currLayerIdx][refLayerIdx], where currLayerIdx is equal to GeneralLayerIdx[nuh_layer_id] and refLayerIdx is equal to GeneralLayerIdx[refpicLayerId], where GeneralLayerIdx[] indicates the layer index of the layer with a specific layer identifier, nuh_layer_id is the layer identifier of the current image, and refpicLayerId is the layer identifier of the first image. The format rules further specify that the first image referred to by each inter-layer reference image entry exists in the decoded image buffer and has a layer identifier smaller than the layer identifier of the current image.

20. A method for storing a video bitstream, comprising: Generate a video bitstream containing one or more images according to the format rules, and The bitstream is stored in a non-transitory computer-readable storage medium. The format rules specify that the first image referred to by each inter-layer reference image entry in the reference image list of the current image's strip in the current layer must satisfy the constraints. The constraints include at least one of the following: (a) the first image is an intra-frame random access (IRAP) image, or (b) the temporal identifier of the first image is less than the value of the first syntax element corresponding to the index of the current layer and the index of the reference layer, and The first syntax element is included in the bitstream.

21. A video processing method, comprising: Perform conversion between video and video bitstream that includes multiple layers and one or more images, according to format rules, and The format rules specify that the reference images referred to by each inter-layer reference image entry in the reference image list of the current image's strip in the current layer must satisfy the constraints. Wherein, the constraint is at least one of the following: (a) the reference picture is an intra-frame random access (IRAP) picture, or (b) the reference picture has a temporal identifier less than or equal to a specific value, the specific value being based on the maximum allowed value of a video layer that can be referenced from the stripe of the current layer, wherein the maximum allowed value is indicated in a syntax element, wherein the syntax element is a two-dimensional syntax element.

22. The method according to claim 21, wherein, One or more indices of the syntax element are layer indices.

23. The method according to claim 21, wherein, The time-domain identifier is TemporalId, and the syntax element is max_tid_il_ref_pics_plus1[i][j], where i and j are integers.

24. The method according to claim 21, wherein, The specific value corresponds to the value of Max(0,max_tid_il_ref_pics_plus1[currLayerIdx][refLayerIdx]-1), where currLayerIdx and refLayerIdx are equal to GeneralLayerIdx[nuh_layer_id] and GeneralLayerIdx[refpicLayerId], respectively. GeneralLayerIdx[] specifies the layer index of the layer with a specific layer identifier corresponding to nuh_layer_id, and refpicLayerId is the layer identifier of the image.

25. The method according to claim 24, wherein, The format rules also stipulate that the image referred to by each inter-layer reference image entry exists in the decoded image buffer and has a layer identifier smaller than the layer identifier of the current image.

26. The method according to claim 21, wherein, The specific value corresponds to the value of max_tid_il_ref_pics_plus1[currLayerIdx][refLayerIdx], where currLayerIdx and refLayerIdx are equal to GeneralLayerIdx[nuh_layer_id] and GeneralLayerIdx[refpicLayerId], respectively. GeneralLayerIdx[] specifies the layer index of the layer with a specific layer identifier corresponding to nuh_layer_id, and refpicLayerId is the layer identifier of the image.

27. The method according to claim 26, wherein, The rule also stipulates that the image referred to by each inter-layer reference image entry exists in the decoded image buffer and has a layer identifier smaller than the layer identifier of the current image.

28. The method according to any one of claims 21 to 27, wherein, The conversion includes encoding the video into the bitstream.

29. The method according to any one of claims 21 to 27, wherein, The conversion includes decoding the video from the bitstream.

30. The method according to any one of claims 21 to 27, wherein, The conversion includes generating the bitstream from the video, and the method further includes storing the bitstream in a non-transitory computer-readable recording medium.

31. A video processing apparatus comprising a processor configured to implement the method of any one of claims 21 to 30.

32. A method for storing a video bitstream, comprising the method of any one of claims 21 to 30, and further comprising storing the bitstream to a non-transitory computer-readable recording medium.

33. A computer-readable medium storing program code that, when executed, causes a processor to perform the method of any one of claims 21 to 30.

Images