Syntax for dependent random access point indication in video bitstream

Abstract

A system, method, and apparatus for encoding, decoding, or transcoding visual media data using syntax for indicating dependent random access points in a video bitstream is described. An example method for processing visual media data includes performing conversion between visual media data and a bitstream of the visual media data according to a format rule, wherein the format rule specifies that a supplemental enhancement information (SEI) message referring to a dependent random access point (DRAP) picture is included in the bitstream, and wherein the format rule further specifies that the SEI message includes a syntax element indicating a number of intra random access point (IRAP) pictures or dependent random access point (DRAP) pictures within the same codec layer video sequence (CLVS) as the DRAP picture.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is filed under applicable patent law and / or in accordance with the Paris Convention to claim priority to and the benefit of U.S. Provisional Patent Application No. 63 / 084,953, filed on September 29, 2020. The entire disclosure of the foregoing application is incorporated by reference as a part of the disclosure of this application for all purposes under the law. Technical Field

[0003] This patent document relates to digital video coding and decoding technology, including video encoding, transcoding or decoding. Background Art

[0004] Digital video accounts for the largest use of bandwidth on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, bandwidth demand for digital video usage is expected to continue to grow. Summary of the Invention

[0005] This document discloses techniques that can be used by video encoders and decoders to process codec representations of videos or images according to file formats.

[0006] In one example aspect, a method for processing visual media data is disclosed. The method includes performing conversion between the visual media data and a bitstream comprising multiple layers of the visual media data according to a format rule, wherein the format rule provides for including a supplemental enhancement information (SEI) message in the bitstream to indicate that a decoder is permitted to decode 1) dependent random access point (DRAP) pictures in a layer associated with the SEI message and / or 2) pictures included in the layer and following the DRAP pictures in decoding order and output order without decoding other pictures in the layer except intra random access point (IRAP) pictures associated with the DRAP pictures.

[0007] In another example aspect, another method of processing visual media data is disclosed. The method includes performing conversion between the visual media data and a bitstream of the visual media data according to a format rule, wherein the format rule specifies whether and how a second type of supplemental enhancement information (SEI) message, different from a first type of SEI message, is included in the bitstream, and wherein the first type of SEI message and the second type of SEI message indicate a first type of dependent random access point (DRAP) picture and a second type of DRAP picture, respectively.

[0008] In another example aspect, another method of processing visual media data is disclosed. The method includes performing conversion between visual media data and a bitstream of the visual media data according to a format rule, wherein the format rule provides that a supplemental enhancement information (SEI) message referring to a dependent random access point (DRAP) picture is included in the bitstream, and wherein the format rule further provides that the SEI message includes a syntax element that indicates a number of intra random access point (IRAP) pictures or DRAP pictures within the same codec layer video sequence (CLVS) as the dependent random access point (DRAP) picture.

[0009] In yet another exemplary aspect, a video processing device is disclosed, wherein the video processing device includes a processor configured to implement the above method.

[0010] In yet another exemplary aspect, a method for storing visual media data in a file comprising one or more bitstreams is disclosed. The method corresponds to the above method and further includes storing the one or more bitstreams in a non-transitory computer-readable recording medium.

[0011] In yet another exemplary aspect, a computer-readable medium storing a bitstream is disclosed. The bitstream is generated according to the above method.

[0012] In yet another example aspect, a video processing device storing a bitstream is disclosed, wherein the video processing device is configured to implement the above method.

[0013] In yet another example aspect, a computer-readable medium is disclosed on which a bitstream conforms to a file format generated according to the above method.

[0014] These and other features are described throughout this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of an example video processing system.

[0016] Figure 2 It is a block diagram of a video processing device.

[0017] Figure 3 is a flow chart of an example method of video processing.

[0018] Figure 4 is a block diagram illustrating a video encoding and decoding system according to some embodiments of the present disclosure.

[0019] Figure 5 is a block diagram illustrating an encoder according to some embodiments of the present disclosure.

[0020] Figure 6is a block diagram illustrating a decoder according to some embodiments of the present disclosure.

[0021] Figures 7 to 9 is a flow chart of an example method of processing visual media data based on some embodiments of the disclosed technology. DETAILED DESCRIPTION

[0022] Section headings are used in this document to facilitate understanding and do not limit the applicability of the techniques and embodiments disclosed in each section to only that section. In addition, the use of H.266 terminology in some descriptions is intended solely to facilitate understanding and is not intended to limit the scope of the disclosed techniques. As such, the techniques described herein are also applicable to other video codec protocols and designs. In this document, editorial changes to text relative to the current draft of the VVC specification are indicated by strikethrough indicating deleted text and highlighting (including bold italics) indicating added text.

[0023] 1. Preliminary Discussion

[0024] This document relates to video codec technology. Specifically, this document relates to support for cross-random access point (RAP) references in video codecs based on Supplementary Enhancement Information (SEI) messages. These ideas can be applied alone or in various combinations to any video codec standard or non-standard video codec, such as the recently finalized Versatile Video Codec (VVC).

[0025] 2. Abbreviation

[0026] ACT Adaptive Color Transformation

[0027] ALF Adaptive Loop Filter

[0028] AMVR Adaptive Motion Vector Resolution

[0029] APS Adaptive Parameter Set

[0030] AU Access Unit

[0031] AUD Access Unit Delimiter

[0032] AVC Advanced Video Codec (Rec.ITU-T H.264 | ISO / IEC 14496-10)

[0033] B. Bidirectional Prediction

[0034] BCW Bidirectional Prediction with CU-level Weights

[0035] BDOF Bidirectional Optical Flow

[0036] BDPCM Block-based delta pulse coding modulation

[0037] BP buffer period

[0038] CABAC Context-based Adaptive Binary Arithmetic Coding

[0039] CB codec block

[0040] CBR Constant Bit Rate

[0041] CCALF Cross-Component Adaptive Loop Filter

[0042] CLVS codec layer video sequence

[0043] CLVSS codec layer video sequence starts

[0044] CPB codec picture buffer

[0045] CRA clean random access

[0046] CRC Cyclic Redundancy Check

[0047] CTB Codec Tree Block

[0048] CTU Codec Tree Unit

[0049] CU codec unit

[0050] CVS codec video sequence

[0051] CVSS Codec Video Sequence Start

[0052] DPB decoded picture buffer

[0053] DCI decoding capability information

[0054] DRAP relies on random access points

[0055] DU decoding unit

[0056] DUI decoding unit information

[0057] EG Index Golomb

[0058] EGk k-order index Columbus

[0059] EOB End of bitstream

[0060] EOS sequence ends

[0061] FD fill data

[0062] FIFO First In First Out

[0063] FL fixed length

[0064] GBR Green, Blue and Red

[0065] GCI General Constraints Information

[0066] GDR Gradual Decode Refresh

[0067] GPM geometric partitioning mode

[0068] HEVC High Efficiency Video Codec (Rec.ITU-T H.265 | ISO / IEC 23008-2)

[0069] HRD Hypothesized Reference Decoder

[0070] HSS Hypothetical Flow Scheduler

[0071] Intra-I frame

[0072] IBC Intra Block Copy

[0073] IDR Instant Decode Refresh

[0074] ILRP inter-layer reference image

[0075] IRAP Intra-frame random access point

[0076] LFNST Low-Frequency Non-Separable Transform

[0077] LPS Least Likely Symbol

[0078] LSB Least Significant Bit

[0079] LTRP Long Term Reference Picture

[0080] LMCS Luminance Mapping with Chroma Scaling

[0081] MIP matrix-based intra prediction

[0082] MPS Maximum Probable Symbol

[0083] MSB Most Significant Bit

[0084] MTS Multi-Transformation Selection

[0085] MVP Motion Vector Prediction

[0086] NAL Network Abstraction Layer

[0087] OLS output layer set

[0088] OP operating point

[0089] OPI Operating Point Information

[0090] P prediction

[0091] PH picture header

[0092] POC picture sequence counting

[0093] PPS Picture Parameter Set

[0094] PROF uses optical flow prediction refinement

[0095] PT picture timing

[0096] PU picture unit

[0097] QP quantization parameter

[0098] RADL Random Access Decodable Boot (Image)

[0099] RAP Random Access Point

[0100] RASL Random Access Skip Boot (image)

[0101] RBSP Raw Byte Sequence Payload

[0102] RGB red, green, and blue

[0103] RPL Reference Image List

[0104] SAO Sample Adaptive Offset

[0105] SAR sample aspect ratio

[0106] SEI Supplemental Enhancement Information

[0107] SH Strip Header

[0108] SLI sub-picture level information

[0109] SODB data bit string

[0110] SPS sequence parameter set

[0111] STRP Short-Term Reference Picture

[0112] STSA Stepwise Temporal Sublayer Access

[0113] TR Truncated Rice

[0114] TU Transform Unit

[0115] VBR variable bit rate

[0116] VCL video codec layer

[0117] VPS Video Parameter Set

[0118] VSEI Generic Supplementary Enhancement Information (Rec.ITU-T H.274 | ISO / IEC 23002-7)

[0119] VUI Video Availability Information

[0120] VVC (Versatile Video Codec) (Rec. ITU-T H.266 | ISO / IEC 23090-3)

[0121] 3. Video Codec Introduction

[0122] 3.1. Video Codec Standards

[0123] Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, ISO / IEC developed MPEG-1 and MPEG-4 Visual, and the two organizations jointly developed H.262 / MPEG-2 Video, H.264 / MPEG-4 Advanced Video Codec (AVC), and H.265 / HEVC. Since H.262, video codec standards have been based on a hybrid video codec architecture that utilizes temporal prediction plus transform coding. To explore future video codec technologies beyond HEVC, the Joint Video Exploration Team (JVET) was jointly established by VCEG and MPEG in 2015. Since then, many new approaches have been adopted by JVET and incorporated into reference software called the Joint Exploration Model (JEM). When the Versatile Video Codec (VVC) project was officially launched, JVET was later renamed the Joint Video Experts Team (JVET). VVC is a new codec standard finalized by JVET at its 19th meeting that ended on July 1, 2020, with the goal of reducing bit rate by 50% compared to HEVC.

[0124] The Versatile Video Codec (VVC) standard (ITU-T H.266 | ISO / IEC 23090-3) and the associated Versatile Supplementary Enhancement Information (VSEI) standard (ITU-T H.274 | ISO / IEC 23002-7) have been designed for the widest range of applications, including traditional uses such as television broadcasting, video conferencing, or playback from storage media, as well as newer and more advanced use cases such as adaptive bitrate streaming, video region extraction, composition and merging of content from multiple codec video bitstreams, multi-view video, scalable layered codecs, and viewport-adaptive 360° immersive media.

[0125] Picture Order Count (POC) in HEVC and VVC

[0126] In HEVC and VVC, the POC is essentially used as a picture ID for identification of pictures in many parts of the decoding process, including DPB management, part of which is reference picture management.

[0127] For the newly introduced PH, in VVC, the information of the least significant bit (LSB) of the POC is signaled in the PH, unlike HEVC, where it is signaled in the SH. This LSB is used to derive the POC value and has the same value for all slices of a picture. VVC also allows the signaling of the most significant bit (MSB) periodicity value of the POC in the PH to enable the derivation of the POC value without tracking the POC MSB, which relies on the POC information of earlier decoded pictures. For example, this allows mixing IRAPs and non-IRAPs within an AU in a multi-layer bitstream. An additional difference between the POC signaling of HEVC and VVC is that in HEVC, the POC LSB is not signaled for IDR pictures, which caused some disadvantages during the later development of the multi-layer extension of HEVC to be able to mix IDR and non-IDR pictures within an AU. Therefore, in VVC, the POC LSB information is signaled for every picture, including IDR pictures. Signaling of the POC LSB information of IDR pictures also makes it easier to support merging IDR pictures and non-IDR pictures from different bitstreams into one picture, otherwise processing the POC LSB in the merged picture would require some complex design.

[0128] 3.3. Random Access and Support in HEVC and VVC

[0129] Random access refers to accessing and decoding a bitstream starting from a picture that is not the first picture in the bitstream in decoding order. To support tuning and channel switching in broadcast / multicast and multi-party video conferencing, seeking in local playback and streaming, and stream adaptation in streaming, the bitstream needs to include frequent random access points, which are usually intra-coded pictures but can also be inter-coded pictures (for example, in the case of gradual decoding refresh).

[0130] HEVC includes signaling of intra random access point (IRAP) pictures in the NAL unit header via the NAL unit type. Three types of IRAP pictures are supported, namely instantaneous decoder refresh (IDR), clean random access (CRA), and broken link access (BLA) pictures. IDR pictures constrain the inter-picture prediction structure to not reference any pictures before the current group of pictures (GOP), traditionally known as closed GOP random access points. CRA pictures have fewer restrictions by allowing certain pictures to reference pictures before the current GOP, and in the case of random access, all pictures are discarded. CRA pictures are traditionally known as open GOP random access points. BLA pictures typically originate from the splicing of two bitstreams or parts thereof at a CRA picture, such as during stream switching. In order to enable the system to better use IRAP pictures, a total of six different NAL units are defined to signal the properties of IRAP pictures, which can be used to better match the stream access point types defined in the ISO Base Media File Format (ISOBMFF), which is used to support Dynamic Adaptive Streaming over HTTP (DASH).

[0131] VVC supports three types of IRAP pictures, two types of IDR pictures (one type with or without an associated RADL picture), and one type of CRA picture. These are essentially the same as in HEVC. VVC does not include the BLA picture type in HEVC for two main reasons: i) the basic functionality of a BLA picture can be implemented with a CRA picture plus a sequence end NAL unit, the presence of which indicates that the subsequent picture starts a new CVS in a single-layer bitstream; ii) during the development of VVC, it was desired to specify fewer NAL unit types than in HEVC, and this was indicated by using five bits instead of six bits for the NAL unit type field in the NAL unit header.

[0132] Another key difference between VVC and HEVC in terms of random access support is that GDR is supported in a more standardized way in VVC. In GDR, decoding of the bitstream can start from an inter-frame coded picture, and although not the entire picture area can be correctly decoded at the beginning, after several pictures, the entire picture area will be correct. AVC and HEVC also support GDR, using the recovery point SEI message for signaling GDR random access points and recovery points. In VVC, a new NAL unit type is specified to indicate GDR pictures, and the recovery point is signaled in the picture header syntax structure. CVS and bitstreams are allowed to start from GDR pictures. This means that the entire bitstream is allowed to contain only inter-frame coded pictures, without a single intra-frame coded picture. The main benefit of specifying GDR support in this way is to provide consistent behavior for GDR. GDR enables the encoder to smooth the bitrate of the bitstream by distributing intra-coded slices or blocks across multiple pictures, rather than intra-coding the entire picture, thereby significantly reducing end-to-end latency, which is considered more important today than before as wireless displays, online gaming, and drone-based applications become more popular.

[0133] Another GDR-related feature in VVC is virtual boundary signaling. On the picture between the GDR picture and its recovery point, the boundary between the refresh area (i.e., the correctly decoded area) and the unrefreshed area can be signaled as a virtual boundary, and when signaled, in-loop filtering across the boundary will not be applied, so there will be no decoding mismatch of some samples at or near the boundary. This may be useful when the application determines that the correctly decoded area is displayed during the GDR process.

[0134] IRAP pictures and GDR pictures may be collectively referred to as random access point (RAP) pictures.

[0135] 3.4.VUI and SEI messages

[0136] VUI is a syntax structure sent as part of SPS (and possibly in VPS for HEVC). The information carried by VUI does not affect the decoding process of the specification, but the information may be important for correctly rendering the encoded and decoded video.

[0137] SEI assists with processes related to decoding, display, or other purposes. Like VUI, SEI does not affect the decoding process of the specification. SEI is carried in SEI messages. Decoder support of SEI messages is optional. However, SEI messages do affect bitstream conformance (for example, if the syntax of SEI messages in the bitstream does not conform to the specification, the bitstream is non-conforming), and some SEI messages are required in the HRD specification.

[0138] The VUI syntax structure and most SEI messages used with VVC are not specified in the VVC specification, but are specified in the VSEI specification. The SEI information required for HRD conformance testing is specified in the VVC specification. VVC v1 defines 5 SEI messages related to HRD conformance testing, and VSEI v1 specifies 20 additional SEI messages. The SEI messages carried in the VSEI specification do not directly affect conforming decoder behavior and have been defined so that they can be used in a codec-agnostic manner, allowing VSEI to be used with other video codec standards besides VVC in the future. The VSEI specification does not specifically refer to VVC syntax element names, but rather to variables whose values ​​are set in the VVC specification.

[0139] Compared to HEVC, VVC's VUI syntax structure only focuses on information related to the correct rendering of pictures, and does not contain any timing information or bitstream restriction indications. In VVC, VUI is signaled in SPS, which contains a length field before the VUI syntax structure, signaling the length of the VUI payload in bytes. This enables the decoder to easily skip information, and more importantly, allows for convenient future VUI syntax extensions by adding new syntax elements directly to the end of the VUI syntax structure in a manner similar to the SEI message syntax extension.

[0140] The VUI grammatical structure contains the following information:

[0141] The content is interlaced or progressive;

[0142] Whether the content contains frame-packed stereoscopic video or projected omnidirectional video;

[0143] Sample aspect ratio;

[0144] Whether the content is suitable for overscan display;

[0145] Color description, including primaries, matrix, and transfer characteristics, which is particularly important for signaling ultra-high-definition (UHD) and high-definition (HD) color spaces and high dynamic range (HDR);

[0146] • Chroma position compared to luma (clarifies signaling for progressive content compared to HEVC).

[0147] When the SPS does not contain any VUI, this information is considered unspecified and must be conveyed by external means or specified by the application if the content of the bitstream is intended to be presented on a display.

[0148] Table 1 lists all SEI messages specified for VVC v1, along with the specifications containing their syntax and semantics. Of the 20 SEI messages specified in the VSEI specification, many are inherited from HEVC (e.g., padding payload and user data SEI messages). Some SEI messages are critical for the correct processing or presentation of the codec's video content. This is, for example, the case of the primary display color volume, content light level information, or alternative transfer characteristics SEI messages that are particularly relevant for HDR content. Other examples include equirectangular projection, sphere rotation, region packing, or omnidirectional viewport SEI messages, which are relevant for the signaling and processing of 360° video content.

[0149] Table 1: SEI message list in VVC v1

[0150]

[0151]

[0152] New SEI messages specified for VVC v1 include a frame-field information SEI message, a sample aspect ratio information SEI message, and a sub-picture level information SEI message.

[0153] The frame-field information SEI message contains information indicating how the associated picture should be displayed (such as field parity or frame repetition period), the source scan type of the associated picture, and whether the associated picture is a copy of a previous picture. In previous video codec standards, this information, together with the timing information of the associated picture, could be used for signaling in the picture timing SEI message. However, it was observed that frame-field information and timing information are two different pieces of information that are not necessarily signaled together. A typical example is to signal timing information at the system level, but signal frame-field information in the bitstream. Therefore, it was decided to remove the frame-field information from the picture timing SEI message and signal it in a dedicated SEI message instead. This change also makes it possible to modify the syntax of the frame-field information to convey additional and clearer instructions to the display, such as pairing fields together, or more values ​​for frame repetition.

[0154] The sample aspect ratio SEI message can signal different sample aspect ratios for different pictures in the same sequence, and the corresponding information contained in the VUI applies to the entire sequence. When using the reference picture resampling function with a scaling factor, this may result in different pictures in the same sequence having different sample aspect ratios.

[0155] The sub-picture level information SEI message provides level information of a sub-picture sequence.

[0156] 3.5. Cross-RAP reference

[0157] A video coding and decoding method based on Cross RAP Reference (CRR), also known as External Decoding Refresh (EDR), is proposed in JVET-M0360, JVET-N0119, JVET-O0149 and JVET-P0114.

[0158] The basic idea of ​​this video coding method is as follows. Instead of coding random access points as intra-coded IRAP pictures (except the first picture in the bitstream), they are coded using inter prediction to overcome the unavailability of earlier pictures if the random access points are coded as IRAP pictures. The trick is to provide a limited number of earlier pictures, usually representing different scenes of the video content, via a separate video bitstream, which can be called an external device. Such early pictures are called external pictures. Therefore, each external picture can be used for inter prediction reference by pictures across random access points. The codec efficiency gain comes from coding random access points as inter-frame predicted pictures and having more available reference pictures for pictures that follow the EDR picture in decoding order.

[0159] As described below, the bitstream encoded and decoded using this video codec can be used in applications based on ISOBMFF and DASH.

[0160] DASH content preparation operations

[0161] 1) Video content is encoded into one or more representations, each with a specific spatial resolution, temporal resolution, and quality.

[0162] 2) Each specific representation of video content is represented by a main stream and optionally an external stream. The main stream contains codec pictures, which may or may not contain EDR pictures. When the main stream includes at least one EDR picture, the external stream also exists and contains the external pictures. When the main stream does not contain EDR pictures, the external stream does not exist.

[0163] 3) Each main stream is carried in a Main Stream Representation (MSR). Each EDR picture in the MSR is the first picture in a segment.

[0164] 4) Each external stream (if any) is carried in an ExternalStreamRepresentation (ESR).

[0165] 5) For each segment in the MSR that starts with an EDR picture, there is a segment in the corresponding ESR with the same segment start time derived from the MPD, carrying the external pictures required to decode the EDR picture and the subsequent pictures in decoding order in the bitstream carried by the MSR.

[0166] 6) The MSR of the same video content is included in one Adaptation Set (AS). The ESR of the same video content is included in one AS.

[0167] DASH streaming operations

[0168] 1) The client obtains the MPD of the DASH media representation, parses the MPD, selects the MSR, and determines the starting presentation time of the content to be consumed.

[0169] 2) The client requests segments of the MSR, starting with the segment that includes a picture whose presentation time is equal to (or close enough to) the start presentation time.

[0170] a. If the first picture in the starting segment is an EDR picture, then the corresponding segment in the associated ESR (with the same segment start time derived from the MPD) is also requested, preferably before requesting the MSR segment. Otherwise, no segments of the associated ESR are requested.

[0171] 3) When switching to a different MSR, the client requests switching to the segments of the MSR starting from the first segment whose segment start time is greater than the start time of the last segment requested to be switched from the MSR.

[0172] If the first picture in the starting segment of the MSR is an EDR picture, the corresponding segment in the ESR will also be requested, preferably before the MSR segment is requested. Otherwise, no segment in the ESR will be requested.

[0173] 4) When operating continuously on the same MSR (after decoding the starting segment after a search or stream switch operation), there is no need to request any segments of the associated ESR, including when requesting any segment starting with an EDR picture.

[0174] 3.6.DRAP Indication SEI Message

[0175] The VSEI specification includes the DRAP indication SEI message, which is as follows:

[0176]

[0177] A picture associated with a Dependent Random Access Point (DRAP) indication SEI message is called a DRAP picture.

[0178] The presence of a DRAP indication SEI message indicates that the constraints on picture order and picture reference specified in this clause apply. These constraints enable the decoder to correctly decode a DRAP picture and the pictures following it in decoding order and output order without decoding any other pictures except the DRAP picture's associated IRAP picture.

[0179] The presence of a DRAP indication SEI message indicates the following constraints, which all apply:

[0180] –DRAP picture is a trailing picture;

[0181] – The temporal sublayer identifier of a DRAP picture is equal to 0;

[0182] – A DRAP picture does not include any pictures in its reference picture list active entries except its associated IRAP picture;

[0183] – Any picture that follows a DRAP picture in decoding order and output order does not include in its valid entries in a reference picture list any picture that precedes the DRAP picture in decoding order or output order, except for the DRAP picture's associated IRAP picture.

[0184] 4. Technical Problems Solved by the Disclosed Technical Solutions

[0185] The functionality of the DRAP indication SEI message can be considered as a subset of the CRR method.For simplicity, the picture associated with the DRAP indication SEI message is called a type 1 DRAP picture.

[0186] From a coding perspective, although the CRR method proposed in JVET-P0114 or earlier JVET contributions is not adopted by VVC, the encoder can still encode the video bitstream in such a way that some pictures depend only on the associated IRAP pictures for inter-frame prediction reference (such as type 1 DRAP pictures indicated by the DRAP SEI message), and some other pictures (for example, called type 2 DRAP pictures) depend only on some pictures in the picture set consisting of the associated IRAP pictures and some other (type 1 or type 2) DRAP pictures.

[0187] However, given a VVC bitstream, it is not known whether such Type 2 DRAP pictures exist in the bitstream. Furthermore, even when it is known that such Type 2 DRAP pictures exist in the bitstream, in order to compose media files from ISOBMFF and DASH media representations based on such VVC bitstream to implement CRR or EDR streaming operation, file and DASH media representation editors will need to parse and deduce a large amount of information, including POC values ​​and valid entries in reference picture lists, to determine whether a particular picture is a Type 2 DRAP picture, and if so, which earlier IRAP or DRAP picture is required to randomly access from that particular picture, so that the appropriate set of pictures can be included in separate, time-synchronized file tracks and DASH representations.

[0188] Another problem is that the semantics of the DRAP indication SEI message only applies to single-layer bitstreams.

[0189] 5. Solution List

[0190] To solve the above problems, and others, the following methods are disclosed. These items should be considered as examples to explain the general concept and should not be interpreted narrowly. In addition, these items can be applied individually or in any combination.

[0191] 1) In one example, the semantics of the DRAP indication SEI message are changed so that the SEI message can be applied to a multi-layer bitstream, i.e., the semantics enable the decoder to correctly decode the DRAP picture (i.e., the picture associated with the DRAP indication SEI message) and the pictures in the same layer and following the DRAP in decoding order and output order, without decoding any other pictures in the same layer except the associated IRAP picture of the DRAP picture.

[0192] a. For example, a DRAP picture is required not to include any pictures in the same layer in the valid entries of its reference picture list except for its associated IRAP picture.

[0193] b. In one example, any picture in the same layer and following the DRAP picture in decoding order and output order is required not to include in its valid entries in the reference picture list any picture in the same layer and preceding the DRAP picture in decoding order or output order, except for the DRAP picture's associated IRAP picture.

[0194] 2) In one example, the RAP picture ID for a DRAP picture is signaled in a DRAP indication SEI message to specify the identifier of the RAP picture, which may be an IRAP picture or a DRAP picture.

[0195] a. In one example, a presence flag indicating whether the RAP picture ID is present in the DRAP indication is signaled, and when the flag is equal to a specific value, such as 1, the RAP picture ID is signaled in the DRAP indication SEI message, and when the flag is equal to another value, such as 0, the RAP picture ID is not signaled in the DRAP indication SEI message.

[0196] 3) In one example, a DRAP picture associated with a DRAP indication SEI message is allowed to refer to the associated IRAP picture or the previous picture in decoding order, which is a GDR picture with ph_recovery_poc_cnt equal to 0 for inter prediction reference.

[0197] 4) In one example, a new SEI message is named, for example, a Type 2 DRAP indication SEI message, and each picture associated with the new SEI message is referred to as a special type of picture, for example, a Type 2 DRAP picture.

[0198] 5) In one example, it is specified that type 1 DRAP pictures (associated with DRAP indication SEI messages) and type 2 DRAP pictures (associated with type 2 DRAP indication SEI messages) are collectively referred to as DRAP pictures.

[0199] 6) In one example, the type 2 DRAP indication SEI message includes a RAP picture ID, e.g., denoted as RapPicId, to specify an identifier of a RAP picture, which may be an IRAP picture or a DRAP picture, and a syntax element (e.g., t2drap_num_ref_rap_pics_minus1) indicating the number of IRAP or DRAP pictures that are within the same CLVS as the type 2 DRAP picture and that can be included in valid entries in the reference picture list of the type 2 DRAP picture.

[0200] a. In one example, the syntax element indicating this number (eg, t2drap_num_ref_rap_pics_minus1) is coded as u(3) using 3 bits.

[0201] b. Optionally, a syntax element indicating this number (eg, t2drap_num_ref_rap_pics_minus1) is coded as ue(v).

[0202] 7) In one example, for the RAP picture ID of a DRAP picture, in a DRAP indication SEI message or a type 2 DRAP indication SEI message, one or more of the following methods apply:

[0203] a. In one example, the syntax element signaling the RAP picture ID uses a 16-bit codec, u(16).

[0204] i. Alternatively, the syntax element signaling the RAP picture ID is coded using ue(v).

[0205] b. In one example, instead of signaling the RAP picture ID in the DRAP indication SEI message, the POC value of the DRAP picture is signaled using, for example, se(v) or i(32).

[0206] i. Optionally, the POC delta relative to the POC value of the associated IRAP picture is signaled, for example using ue(v) or u(16).

[0207] 8) In one example, each IRAP or DRAP picture specified as an IRAP or DRAP is associated with a RAP picture ID RapPicId.

[0208] a. In one example, the value of RapPicId that specifies an IRAP picture is inferred to be equal to 0.

[0209] b. In one example, it is specified that the RapPicId values ​​of any two IRAP or DRAP pictures in a CLVS should be different.

[0210] c. In addition, in one example, the value of RapPicId of IRAP and DRAP pictures within a CLVS will increase as the decoding order of the IRAP or DRAP pictures increases.

[0211] d. Furthermore, in one example, within the same CLVS, the RapPicId of a DRAP picture should be 1 greater than the RapPicId of the preceding IRAP or DRAP picture in decoding order.

[0212] 9) In one example, the Type 2 DRAP indication SEI message also includes a list of RAP picture IDs,

[0213] One list for each IRAP or DRAP picture that is within the same CLVS as a type 2 DRAP picture and that can be included in a valid entry in the reference picture list of the type 2 DRAP picture.

[0214] a. In one example, each of the RAP picture ID lists is coded to be the same as the RAP picture ID of the DRAP picture associated with the Type 2 DRAP indication SEI message.

[0215] b. Optionally, the values ​​of the RAP picture ID list are required to increase in increasing order of the value of the list index i, and the UE(v) codec uses the increment between the RapPicId value of the i-th DRAP pic and 1) the RapPicId value of the i-1th DRAP or IRAP pic (when i is greater than 0) or 2) the increment between the RapPicId value of the i-th DRAP pic and 0 (when i is equal to 0).

[0216] c. Optionally, each of the RAP picture IDs in the list is encoded to represent the POC value of the RAP picture, for example, encoded as se(v) or i(32).

[0217] d. Optionally, each of the RAP picture ID lists is coded to represent a POC delta relative to the POC value of the associated IRAP picture, for example signaled using ue(v), u(16).

[0218] e. Optionally, each in the RAP picture ID list is encoded to represent the POC delta between the POC value of the current picture and 1) the POC value of the i-1th DRAP or IRAP pic (when i is greater than 0) or 2) the POC value of the IRAP picture (when i is equal to 0), for example using ue(v) or u(16).

[0219] f. Optionally, in addition, it is required that for any two values ​​of list index values ​​i and j, for a list of RAP picture IDs, when i is less than j, the i-th IRAP or DRAP picture shall precede the j-th IRAP or DRAP picture in decoding order.

[0220] 6. Examples

[0221] The following are some example embodiments of some of the inventive aspects summarized in Section 5 above that can be applied to the VSEI specification. The changed text is based on the latest VSEI text in JVET-S2007-v7. Most relevant sections that have been added or modified are highlighted by bold italics, and some deleted sections are marked by double brackets (e.g., [[a]] indicates deletion of the letter "a"). There may be some other changes of an editorial nature and therefore are not highlighted.

[0222] 6.1. First embodiment

[0223] This embodiment is a change to the existing DRAP indication SEI message.

[0224] 6.1.1. Dependent Random Access Point Indication SEI Message Syntax

[0225]

[0226] 6.1.2. Dependence on Random Access Point Indication SEI Message Semantics

[0227] A picture associated with a Dependent Random Access Point (DRAP) indication SEI message is called a Type 1 DRAP picture.

[0228] Type 1 DRAP pictures and type 2 DRAP pictures (associated with type 2 DRAP indication SEI messages) are collectively referred to as DRAP pictures.

[0229] The presence of a DRAP indication SEI message indicates that the constraints on picture order and picture references specified in this subclause apply. These constraints enable the decoder to correctly decode a type 1 DRAP picture and pictures that follow it in the same layer in decoding order and output order, without decoding any other pictures in the same layer except the associated IRAP pictures of the type 1 DRAP picture.

[0230] The constraints indicated by the presence of the DRAP indication SEI message are as follows, and all of these constraints apply:

[0231] – Type 1 DRAP pictures are post-images.

[0232] – The temporal sublayer identifier of a Type 1 DRAP picture is equal to 0.

[0233] - Except for its associated IRAP picture, a type 1 DRAP picture does not include any pictures in the same layer that are valid entries in its reference picture list.

[0234] Any picture in the same layer and following a type 1 DRAP picture in decoding order and output order does not include in its valid entries in the reference picture list any picture in the same layer and preceding the type 1 DRAP picture in decoding order or output order, except for the associated IRAP picture of the type 1 DRAP picture.

[0235] drap_rap_id_in_clvs specifies the RAP picture ID of the type 1 Drap picture, expressed as RapPicId.

[0236] Each IRAP picture or DRAP picture, as an IRAP or DRAP, is associated with a RapPicId. The RapPicId value of an IRAP picture is inferred to be equal to 0. The RapPicId values ​​of any two IRAP pictures or DRAP pictures in a CLVS should be different.

[0237] 6.2. Second embodiment

[0238] This embodiment is for a new Type 2 DRAP indication SEI message.

[0239] 6.2.1. Type 2 DRAP Indication SEI Message Syntax

[0240]

[0241]

[0242] 6.2.2. Type 2 DRAP Indication SEI Message Semantics

[0243] A picture associated with a Type 2 DRAP indication SEI message is referred to as a Type 2 DRAP picture.

[0244] Type 1 DRAP pictures (associated with DRAP indication SEI messages) and type 2 DRAP pictures are collectively referred to as DRAP pictures.

[0245] The presence of a Type 2 DRAP indication SEI message indicates that the constraints on picture order and picture references specified in this subclause apply. These constraints enable a decoder to correctly decode a Type 2 DRAP picture and pictures in the same layer that follow it in decoding order and output order, without decoding any other pictures in the same layer, except for the picture list referenceablePictures, which consists of a list of IRAP pictures or DRAP pictures in decoding order identified by the t2drap_ref_rap_id[i] syntax element within the same CLVS.

[0246] The presence of a Type 2 DRAP indication SEI message indicates the following constraints, which all apply:

[0247] – Type 2DRAP pictures are post-images;

[0248] – The temporal sublayer identifier of a type 2 DRAP picture is equal to 0;

[0249] – a type 2 DRAP picture does not include any pictures in the same layer among the valid entries in its reference picture list, except referenceablePictures;

[0250] – any picture in the same layer and following a type 2 DRAP picture in decoding order and output order does not include in its reference picture list any pictures in the same layer and preceding the type 2 DRAP picture in decoding order or output order, except referenceablePictures;

[0251] - any picture in the list referenceablePictures that does not include in its valid entries in the reference picture list any picture that is at the same layer and is not a picture at an earlier position in the list referenceablePictures.

[0252] NOTE – Therefore, the first picture in referenceablePictures, even if it is a DRAP picture rather than an IRAP picture, does not include any pictures of the same layer in its valid entries in the reference picture list.

[0253] t2drap_rap_id_in_clvs specifies the RAP picture identifier of the type 2drap picture, denoted as RapPicId.

[0254] Each IRAP picture or DRAP picture, as an IRAP or DRAP, is associated with a RapPicId. The RapPicId value of an IRAP picture is inferred to be equal to 0. The RapPicId values ​​of any two IRAP pictures or DRAP pictures in a CLVS should be different.

[0255] In bitstreams conforming to this version of this specification, t2drap_reserved_zero_13bits shall be equal to 0. Other values ​​of t2drap_reserved_zero_13bits are reserved for future use by ITU-T | ISO / IEC. Decoders shall ignore the value of t2drap_reserved_zero_13bits.

[0256] t2drap_num_ref_rap_pics_minus1 plus 1 indicates the number of IRAP pictures or DRAP pictures that are within the same CLVS as the type 2 DRAP picture and that can be included in valid entries of the reference picture list of the type 2 DRAP picture.

[0257] t2drap_ref_rap_id[i] indicates the RapPicId of the i-th IRAP picture or DRAP picture that is within the same CLVS as the type-2 DRAP picture and can be included in a valid entry of the reference picture list of the type-2 DRAP picture.

[0258] Figure 11 is a block diagram illustrating an example video processing system 1900 in which the various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of system 1900. System 1900 may include an input 1902 for receiving video content. The video content may be received in a raw or uncompressed format, such as 8 or 10-bit multi-component pixel values, or may be in a compressed or encoded format. Input 1902 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, a Passive Optical Network (PON), and wireless interfaces such as Wi-Fi or a cellular interface.

[0259] System 1900 may include a codec component 1904 that can implement the various codecs or encoding methods described in this document. Codec component 1904 can reduce the average bit rate of the video from input 1902 to the output of codec component 1904 to produce a coded representation of the video. Codec technology is therefore sometimes referred to as video compression or video transcoding technology. The output of codec component 1904 can be stored or transmitted via a communication connection such as represented by component 1906. The stored or communicated bitstream (or codec) representation of the video received at input 1902 can be used by component 1908 to generate pixel values ​​or transmit to a displayable video of display interface 1910. The process of generating user-viewable video from the bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "codec" operations or tools, it will be understood that the codec tools or operations are used at the encoder, and the corresponding decoding tools or operations that reverse the codec results will be performed by the decoder.

[0260] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or DisplayPort, etc. Examples of storage interfaces include SATA (Serial Advanced Technology Attachment), PCI, IDE interfaces, etc. The technology described in this document may be embodied in various electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of performing digital data processing and / or video display.

[0261] Figure 236 is a block diagram of a video processing device 3600. Device 3600 can be used to implement one or more methods described herein. Device 3600 can be embodied in a smartphone, tablet computer, computer, Internet of Things (IoT) receiver, etc. Device 3600 may include one or more processors 3602, one or more memories 3604, and video processing hardware 3606. Processor(s) 3602 can be configured to implement one or more methods described in this document. Memory(s) 604 can be used to store data and code for implementing the methods and techniques described herein. Video processing hardware 606 can be used to implement some of the techniques described in this document in hardware circuitry. In some embodiments, video processing hardware 3606 can be at least partially included in processor 3602 (e.g., a graphics coprocessor).

[0262] Figure 4 is a block diagram illustrating an example video coding system 100 that may utilize the techniques of this disclosure.

[0263] like Figure 4 As shown, the video encoding and decoding system 100 may include a source device 110 and a target device 120. The source device 110 generates encoded video data, wherein the source device 110 may be referred to as a video encoding device. The target device 120 may decode the encoded video data generated by the source device 110, wherein the target device 120 may be referred to as a video decoding device.

[0264] Source device 110 may include a video source 112 , a video encoder 114 , and an input / output (I / O) interface 116 .

[0265] The video source 112 may include a source such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of these sources. The video data may include one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a bitstream. The bitstream may include a sequence of bits that form a codec representation of the video data. The bitstream may include a codec picture and associated data. The codec picture is a codec representation of the picture. Associated data may include a sequence parameter set, a picture parameter set, and other syntax structures. The I / O interface 116 may include a modulator / demodulator (modem) and / or a transmitter. The encoded video data may be transmitted directly to the target device 120 via the network 130a via the I / O interface 116. The encoded video data may also be stored on a storage medium / server 130b for access by the target device 120.

[0266] Target device 120 may include an I / O interface 126 , a video decoder 124 , and a display device 122 .

[0267] I / O interface 126 may include a receiver and / or a modem. I / O interface 126 may obtain coded video data from source device 110 or storage medium / server 130b. Video decoder 124 may decode the coded video data. Display device 122 may display the decoded video data to a user. Display device 122 may be integrated with target device 120, or may be external to target device 120 configured to interface with an external display device.

[0268] The video encoder 114 and the video decoder 124 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVM) standard, and other current and / or additional standards.

[0269] Figure 5 is a block diagram illustrating an example of a video encoder 200, which may be Figure 4 The video encoder 114 in the system 100 is shown.

[0270] Video encoder 200 may be configured to perform any or all of the techniques of this disclosure. Figure 5 In the example of , video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of video encoder 200. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0271] The functional components of the video encoder 200 may include a segmentation unit 201, a prediction unit 202 (which may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205 and an intra-frame prediction unit 206), a residual generation unit 207, a transform unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transform unit 211, a reconstruction unit 212, a buffer 213 and an entropy coding unit 214.

[0272] In other examples, the video encoder 200 may include more, fewer, or different functional components. In an example, the prediction unit 202 may include an intra block copy (IBC) unit. The IBC unit may perform prediction in IBC mode, where at least one reference picture is a picture in which the current video block is located.

[0273] Furthermore, some components such as the motion estimation unit 204 and the motion compensation unit 205 may be highly integrated, but for the purpose of explanation, they are shown in FIG. Figure 5 In the example, they are shown separately.

[0274] The segmentation unit 201 may segment a picture into one or more video blocks. The video encoder 200 and the video decoder 300 may support various video block sizes.

[0275] The mode selection unit 203 can select one of the coding modes (e.g., intra or inter) based on the error result, and provide the resulting intra-coded block or inter-coded block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 203 can select a combination of intra and inter prediction mode (CIIP), where prediction is based on an inter prediction signal and an intra prediction signal. In the case of inter prediction, the mode selection unit 203 can also select the resolution of the motion vector of the block (e.g., sub-pixel or integer pixel precision).

[0276] To perform inter-frame prediction on the current video block, the motion estimation unit 204 may generate motion information for the current video block by comparing the current video block with one or more reference frames from the buffer 213. The motion compensation unit 205 may determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from the buffer 213 other than the picture associated with the current video block.

[0277] Motion estimation unit 204 and motion compensation unit 205 may perform different operations on the current video block, eg, depending on whether the current video block is in an I slice, a P slice, or a B slice.

[0278] In some examples, motion estimation unit 204 may perform unidirectional prediction on the current video block, and motion estimation unit 204 may search for a reference picture in list 0 or list 1 for a reference video block of the current video block. Motion estimation unit 204 may then generate a reference index indicating a reference picture in list 0 or list 1, the reference index including the reference video block and a motion vector indicating a spatial displacement between the current video block and the reference video block. Motion estimation unit 204 may output the reference index, the prediction direction indicator, and the motion vector as motion information for the current video block. Motion compensation unit 205 may generate a predicted video block for the current block based on the reference video block indicated by the motion information for the current video block.

[0279] In other examples, the motion estimation unit 204 may perform bidirectional prediction on the current video block. The motion estimation unit 204 may search for a reference video block for the current video block in the reference pictures in list 0 and may also search for another reference video block for the current video block in list 1. The motion estimation unit 204 may then generate a reference index indicating the reference pictures in list 0 and list 1 containing the reference video block and a motion vector indicating the spatial displacement between the reference video block and the current video block. The motion estimation unit 204 may output the reference index and motion vector for the current video block as motion information for the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0280] In some examples, motion estimation unit 204 may output a complete motion information set for use in a decoding process by a decoder.

[0281] In some examples, motion estimation unit 204 may not output a complete set of motion information for the current video. Instead, motion estimation unit 204 may reference motion information of another video block to signal the motion information of the current video block. For example, motion estimation unit 204 may determine that the motion information of the current video block is sufficiently similar to the motion information of a neighboring video block.

[0282] In one example, motion estimation unit 204 may indicate a value in a syntax structure associated with the current video block that indicates to video decoder 300 that the current video block has the same motion information as another video block.

[0283] In another example, the motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0284] As discussed above, the video encoder 200 may predictively signal motion vectors.Two examples of predictive signaling techniques that may be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and Merge mode signaling.

[0285] The intra-frame prediction unit 206 can perform intra-frame prediction on the current video block. When the intra-frame prediction unit 206 performs intra-frame prediction on the current video block, the intra-frame prediction unit 206 can generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block can include the predicted video block and various syntax elements.

[0286] The residual generation unit 207 can generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) the predicted video block(s) of the current video block from the current video block. The residual data for the current video block may include residual video blocks corresponding to different sample components of the samples in the current video block.

[0287] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 207 may not perform a subtraction operation.

[0288] Transform processing unit 208 may generate one or more transform coefficient video blocks for a current video block by applying one or more transforms to a residual video block associated with the current video block.

[0289] After transform processing unit 208 generates a transform coefficient video block associated with the current video block, quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0290] The inverse quantization unit 210 and the inverse transform unit 211 may apply inverse quantization and inverse transform, respectively, to the transform coefficient video block to reconstruct a residual video block from the transform coefficient video block. The reconstruction unit 212 may add the reconstructed residual video block to corresponding samples of one or more predicted video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current block for storage in the buffer 213.

[0291] After the reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

[0292] The entropy coding unit 214 may receive data from other functional components of the video encoder 200. When the entropy coding unit 214 receives the data, the entropy coding unit 214 may perform one or more entropy coding operations to generate entropy-coded data and output a bitstream including the entropy-coded data.

[0293] Figure 6 is a block diagram illustrating an example of a video decoder 300, which may be Figure 4 The video decoder 114 in the system 100 is shown.

[0294] Video decoder 300 may be configured to perform any or all of the techniques of this disclosure. Figure 6 In the example of FIG, video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of video decoder 300. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0295] exist Figure 6 In the example of FIG, the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra-frame prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, a reconstruction unit 306, and a buffer 307. In some examples, the video decoder 300 can perform the same operations as those generally performed for the video encoder 200 ( Figure 5 ) is the opposite of the encoding process described in .

[0296] The entropy decoding unit 301 can retrieve a coded bitstream. The coded bitstream can include entropy-encoded video data (e.g., coded blocks of video data). The entropy decoding unit 301 can decode the entropy-encoded video data, and based on the entropy-decoded video data, the motion compensation unit 302 can determine motion information including motion vectors, motion vector precision, reference picture list index, and other motion information. The motion compensation unit 302 can determine such information, for example, by implementing AMVP and Merge modes.

[0297] The motion compensation unit 302 may generate a motion compensated block and may perform interpolation based on an interpolation filter. An identifier of the interpolation filter to be used with sub-pixel precision may be included in a syntax element.

[0298] Motion compensation unit 302 may calculate interpolated values ​​of sub-integer pixels of a reference block using interpolation filters such as those used by video encoder 200 during encoding of the video block. Motion compensation unit 302 may determine the interpolation filters used by video encoder 200 based on received syntax information and use the interpolation filters to generate a prediction block.

[0299] The motion compensation unit 302 may use some syntax information to determine the size of blocks used to encode frame(s) and / or slice(s) of the coded video sequence, partitioning information describing how each macroblock of a picture of the coded video sequence is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-coded block, and other information used to decode the coded video sequence.

[0300] The intra prediction unit 303 can form a prediction block from spatially adjacent blocks using, for example, an intra prediction mode received in the bitstream. The inverse quantization unit 303 inversely quantizes, i.e., dequantizes, the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 303 applies an inverse transform.

[0301] The reconstruction unit 306 can add the residual block to the corresponding prediction block generated by the motion compensation unit 202 or the intra-frame prediction unit 303 to form a decoded block. If necessary, a deblocking filter can also be applied to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in the buffer 307 to provide reference blocks for subsequent motion compensation / intra-frame prediction and also to generate decoded video for presentation on a display device.

[0302] The following provides a list of preferred solutions for some embodiments.

[0303] A first set of solutions is provided below, which illustrate example embodiments of the techniques discussed in the previous section (eg, item 1).

[0304] 1. A video processing method (e.g., Figure 3 ), comprising: performing (702) a conversion between a video comprising multiple layers and a codec representation of the video, wherein the codec representation is organized according to a format rule; wherein the format rule provides for Supplemental Enhancement Information (SEI) to be included in the codec representation, wherein the SEI information carries information sufficient to enable a decoder to decode Dependent Random Access Point (DRAP) pictures and / or decode pictures in a layer in decoding order and output order without decoding other pictures in the layer, except for Intra Random Access Pictures (IRAPs) of the DRAP pictures.

[0305] 2. The method according to solution 1, wherein a DRAP picture excludes any pictures in the layer from the reference picture list except IRAP.

[0306] The following solution illustrates an example embodiment of the technique discussed in the previous section (eg, item 2).

[0307] 3. A video processing method, comprising: performing conversion between a video comprising multiple layers and a codec representation of the video, wherein the codec representation is organized according to a format rule; wherein the format rule specifies that a supplemental enhancement information (SEI) message is included in the codec representation of a dependent random access point (DRAP) picture, wherein the SEI message includes an identifier of the random access point (RAP) picture.

[0308] 4. The method according to solution 3, wherein RAP is an intra random access picture.

[0309] 5. The method according to solution 3, wherein the RAP is a dependent random access picture (DRAP).

[0310] The following solution illustrates an example embodiment of the technique discussed in the previous section (eg, item 3).

[0311] 6. The method according to solution 5, wherein a DRAP picture is allowed to refer to an associated intra random access picture or a previous picture in decoding order, which is a progressive decoding refresh picture.

[0312] The following solutions illustrate example embodiments of the techniques discussed in the previous section (eg, items 4-6).

[0313] 7. A video processing method comprising: performing conversion between a video comprising multiple layers and a codec representation of the video, wherein the codec representation is organized according to format rules; wherein the format rules specify whether and how a type 2 supplemental enhancement information (SEI) message referring to a dependent random access picture (DRAP) is included in the codec representation.

[0314] 8. The method according to solution 7, wherein the format rules specify that a Type 2 SEI message and each picture associated with the message be treated as a special type of picture.

[0315] 9. A method according to solution 7, wherein the format rules specify that the type 2 SEI message includes an identifier of a random access picture (RAP) called a type 2 RAP picture, and a syntax element indicating the number of pictures in the same codec video layer as the random access picture, so that the picture is included in the valid reference picture list of the type 2 RAP picture.

[0316] 10. The method according to any of solutions 1-9, wherein converting comprises generating a codec representation from the video.

[0317] 11. The method according to any of solutions 1-9, wherein converting comprises decoding the codec representation to generate the video.

[0318] 12. A video decoding device, comprising a processor, wherein the processor is configured to implement the method described in one or more of solutions 1 to 11.

[0319] 13. A video encoding device comprising a processor configured to implement the method described in one or more of solutions 1 to 11.

[0320] 14. A computer program product having computer code stored thereon, which, when executed by a processor, causes the processor to implement the method described in any one of solutions 1 to 11.

[0321] 15. A computer-readable medium storing a codec representation generated according to any one of solutions 1 to 11.

[0322] 16. The methods, apparatus, or systems described in this document.

[0323] The second set of solutions provides example embodiments of the techniques discussed in the previous section (eg, items 1, 1.a, 1.b, 2, 2.a, 3).

[0324] 1. A method for processing visual media data (e.g. Figure 7 The method 710 shown includes: performing (712) conversion between visual media data and a bitstream comprising multiple layers of visual media data according to a format rule; wherein the format rule provides that a supplemental enhancement information (SEI) message is included in the bitstream to indicate that a decoder is allowed to decode 1) dependent random access point (DRAP) pictures in a layer associated with the SEI message and / or 2) pictures included in the layer and following the DRAP pictures in decoding order and output order without having to decode other pictures in the layer other than intra random access point (IRAP) pictures associated with the DRAP pictures.

[0325] 2. The method according to solution 1, wherein, except for IRAP pictures, a DRAP picture excludes pictures in the layer from the valid entries of the reference picture list of the DRAP picture.

[0326] 3. A method according to solution 1, wherein the first picture included in the layer and following the DRAP picture in decoding order and output order excludes the second picture included in the layer and preceding the DRAP picture in decoding order and output order from the valid entries of the reference picture list of the first picture, except for the IRAP picture.

[0327] 4. The method according to solution 1, wherein the format rule further stipulates that the SEI message includes an identifier of a random access point (RAP) picture.

[0328] 5. The method according to solution 4, wherein the RAP picture is an IRAP picture or a DRAP picture.

[0329] 6. The method according to solution 4, wherein the format rule further stipulates that a presence flag indicating the presence of an identifier of a RAP picture in the SEI message is included in the bitstream.

[0330] 7. The method according to solution 6, wherein the presence flag having a value equal to the first value indicates that the identifier of the RAP picture is present in the SEI message.

[0331] 8. The method according to solution 6, wherein the presence flag having a value equal to the second value indicates that the identifier of the RAP picture is omitted from the SEI message.

[0332] 9. The method according to solution 1, wherein a DRAP picture is allowed to refer to an IRAP picture or a previous picture in decoding order, where the previous picture is a Progressive Decoding Refresh (GDR) picture whose decoded picture recovery point is equal to 0 in output order.

[0333] 10. The method according to any one of solutions 1 to 9, wherein the bitstream is a universal video codec bitstream.

[0334] 11. The method of any one of solutions 1 to 10, wherein performing the conversion comprises generating a bitstream from the visual media data.

[0335] 12. The method of any one of solutions 1 to 10, wherein performing the conversion comprises reconstructing the visual media data from the bitstream.

[0336] 13. A device for processing visual media data, comprising a processor and a non-volatile memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: perform conversion between visual media data and a bitstream comprising multiple layers of visual media data according to format rules; wherein the format rules provide that a supplemental enhancement information (SEI) message is included in the bitstream to indicate that a decoder is allowed to decode 1) dependent random access point (DRAP) pictures in a layer associated with the SEI message and / or 2) pictures included in the layer and following the DRAP pictures in decoding order and output order without having to decode other pictures in the layer except intra random access point (IRAP) pictures associated with the DRAP pictures.

[0337] 14. The apparatus of solution 13, wherein, except for IRAP pictures, a DRAP picture excludes pictures in the layer from valid entries in the reference picture list of the DRAP picture.

[0338] 15. An apparatus according to solution 13, wherein a first picture included in the layer and following the DRAP picture in decoding order and output order excludes a second picture included in the layer and preceding the DRAP picture in decoding order and output order from the valid entries of the reference picture list of the first picture, except for the IRAP picture.

[0339] 16. The apparatus of solution 13, wherein the bitstream is a universal video codec bitstream.

[0340] 17. A non-transitory computer-readable storage medium storing instructions that cause a processor to: perform conversion between visual media data and a bitstream comprising multiple layers of visual media data according to format rules; wherein the format rules specify that a supplemental enhancement information (SEI) message is included in the bitstream to indicate that a decoder is allowed to decode 1) dependent random access point (DRAP) pictures in a layer associated with the SEI message and / or 2) pictures included in the layer and following the DRAP pictures in decoding order and output order without having to decode other pictures in the layer except intra random access point (IRAP) pictures associated with the DRAP pictures.

[0341] 18. The non-transitory computer-readable storage medium of solution 17, wherein the bitstream is a universal video codec bitstream.

[0342] 19. A non-transitory computer-readable storage medium storing a bitstream of visual media data generated by a method performed by a visual media data processing device, wherein the method includes: determining that a supplemental enhancement information (SEI) message is included in the bitstream, allowing a decoder to decode 1) a dependent random access point (DRAP) picture in a layer associated with the SEI message and / or 2) a picture included in the layer and following the DRAP picture in decoding order and output order without having to decode other pictures in the layer except for the intra-frame random access point (IRAP) picture associated with the DRAP picture; and generating a bitstream based on the determination.

[0343] 20. The non-transitory computer-readable storage medium of solution 19, wherein the bitstream is a universal video codec bitstream.

[0344] 21. A visual media data processing device, comprising a processor configured to implement the method described in any one or more of solutions 1 to 12.

[0345] 22. A method for storing a bitstream of visual media data, comprising the method of any one of solutions 1 to 12, further comprising storing the bitstream in a non-transitory computer-readable storage medium.

[0346] 23. A computer-readable medium storing program code, which, when executed, causes a processor to implement the method of any one or more of solutions 1 to 12.

[0347] 24. A computer-readable medium storing a bitstream generated according to any of the above methods.

[0348] 25. A visual media data processing device storing a bitstream, wherein the visual media data processing device is configured to implement the method described in any one or more of solutions 1 to 12.

[0349] 26. A computer-readable medium, wherein a bit stream on the medium complies with the format rules according to any one of solutions 1 to 12.

[0350] The third set of solutions provides example embodiments of the techniques discussed in the previous section (eg, items 4 to 8).

[0351] 1. A method for processing visual media data (e.g. Figure 8 The method 800 shown includes performing 802 conversion between visual media data and a bitstream of the visual media data according to a format rule, wherein the format rule specifies whether and how a second type of supplemental enhancement information (SEI) message that is different from a first type of SEI message is included in the bitstream, and wherein the first type of SEI message and the second type of SEI message indicate a first type of dependent random access point (DRAP) picture and a second type of DRAP picture, respectively.

[0352] 2. The method according to solution 1, wherein the format rule further stipulates that the second type of SEI message includes a random access point (RAP) picture identifier.

[0353] 3. The method according to solution 1, wherein, for a DRAP picture of the first type or a DRAP picture of the second type, a random access point (RAP) picture identifier is included in the bitstream.

[0354] 4. The method according to solution 3, wherein the RAP picture identifier is encoded and decoded as u(16), where u(16) is an unsigned integer using 16 bits, or the RAP picture identifier is encoded and decoded as ue(v), where ue(v) is an unsigned integer using an exponential Golomb code.

[0355] 5. The method according to solution 1, wherein the format rule further stipulates that the first type of SEI message or the second type of SEI message includes information about a picture order count (POC) value of the first type of DRAP picture or the second type of DRAP picture.

[0356] 6. The method according to solution 1, wherein the format rule further stipulates that each IRAP picture or DRAP picture is associated with a random access point (RAP) picture identifier.

[0357] 7. The method according to solution 6, wherein the format rule further stipulates that the value of the RAP picture identifier of the IRAP picture is inferred to be equal to 0.

[0358] 8. The method according to solution 6, wherein the format rule further stipulates that the values ​​of the RAP picture identifiers of any two IRAP or DRAP pictures in a codec layer video sequence (CLVS) are different from each other.

[0359] 9. The method according to solution 6, wherein the format rule further stipulates that the values ​​of the RAP picture identifiers of the IRAP or DRAP pictures in the codec layer video sequence (CLVS) are incremented according to the decoding order of the IRAP pictures or DRAP pictures.

[0360] 10. The method of solution 6, wherein the format rule further stipulates that the value of the RAP picture identifier of the DRAP picture is one greater than the value of the previous IRAP picture or DRAP picture in decoding order within the codec layer video sequence (CLVS).

[0361] 11. The method of any one of solutions 1 to 10, wherein performing the conversion comprises generating a bitstream from the visual media data.

[0362] 12. The method of any one of solutions 1 to 10, wherein performing the conversion comprises reconstructing the visual media data from the bitstream.

[0363] 13. A device for processing visual media data, comprising a processor and a non-volatile memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: perform conversion between visual media data and a bitstream of visual media data according to format rules, wherein the format rules specify whether and how a second type of supplemental enhancement information (SEI) message, which is different from a first type of SEI message, is included in the bitstream, and wherein the first type of SEI message and the second type of SEI message indicate a first type of dependent random access point (DRAP) picture and a second type of DRAP picture, respectively.

[0364] 14. The apparatus of solution 13, wherein the format rule further specifies that the second type of SEI message includes a random access point (RAP) picture identifier.

[0365] 15. An apparatus according to solution 13, wherein, for a first type of DRAP picture or a second type of DRAP picture, a random access point (RAP) picture identifier is included in the bitstream, the RAP picture identifier is encoded and decoded as u(16), where u(16) is an unsigned integer using 16 bits, or the RAP picture identifier is encoded and decoded as ue(v), where ue(v) is an unsigned integer using an exponential Golomb code.

[0366] 16. The apparatus of solution 13, wherein the format rules further specify that each IRAP picture or DRAP picture is associated with a random access point (RAP) picture identifier, and the format rules further specify that the value of the RAP picture identifier of the IRAP picture is inferred to be equal to 0.

[0367] 17. A non-transitory computer-readable storage medium storing instructions that cause a processor to: perform conversion between visual media data and a bitstream of visual media data according to format rules, wherein the format rules specify whether and how a second type of supplemental enhancement information (SEI) message that is different from a first type of SEI message is included in the bitstream, and wherein the first type of SEI message and the second type of SEI message indicate a first type of dependent random access point (DRAP) picture and a second type of DRAP picture, respectively.

[0368] 18. A non-transitory computer-readable storage medium according to solution 17, wherein the format rules further specify that the second type of SEI message includes a random access point (RAP) picture identifier, and wherein, for a first type of DRAP picture or a second type of DRAP picture, the random access point (RAP) picture identifier is included in the bitstream, the RAP picture identifier is encoded and decoded as u(16), which is an unsigned integer using 16 bits, or is encoded and decoded as ue(v), which is an unsigned integer using an exponential Golomb code, and wherein the format rules further specify that each IRAP picture or DRAP picture is associated with the random access point (RAP) picture identifier, and the format rules further specify that the value of the RAP picture identifier of the IRAP picture is inferred to be equal to 0.

[0369] 19. A non-transitory computer-readable storage medium storing a bitstream of visual media data generated by a method performed by a visual media data processing device, wherein the method includes determining whether and how a second type of supplemental enhancement information (SEI) message different from a first type of SEI message is included in the bitstream; and generating the bitstream based on the determination.

[0370] 20. A non-transitory computer-readable storage medium according to solution 19, wherein the format rules further specify that the second type of SEI message includes a random access point (RAP) picture identifier, and wherein, for a first type of DRAP picture or a second type of DRAP picture, the random access point (RAP) picture identifier is included in the bitstream, the RAP picture identifier is encoded and decoded as u(16), which is an unsigned integer using 16 bits, or is encoded and decoded as ue(v), which is an unsigned integer using an exponential Golomb code, and wherein the format rules further specify that each IRAP picture or DRAP picture is associated with the random access point (RAP) picture identifier, and the format rules further specify that the value of the RAP picture identifier of the IRAP picture is inferred to be equal to 0.

[0371] 21. A visual media data processing device, comprising a processor configured to implement the method described in any one or more of solutions 1 to 12.

[0372] 22. A method for storing a bitstream of visual media data, comprising the method of any one of solutions 1 to 12, further comprising storing the bitstream in a non-transitory computer-readable storage medium.

[0373] 23. A computer-readable medium storing program code, which, when executed, causes a processor to implement the method of any one or more of solutions 1 to 12.

[0374] 24. A computer-readable medium storing a bitstream generated according to any of the above methods.

[0375] 25. A visual media data processing device storing a bitstream, wherein the video processing device is configured to implement any one or more of the methods described in solutions 1 to 12.

[0376] 26. A computer-readable medium, wherein a bit stream on the medium complies with the format rules according to any one of solutions 1 to 12.

[0377] The fourth group of solutions provides example embodiments of the techniques discussed in the previous section (eg, items 6 and 9).

[0378] 1. A method for processing visual media data (e.g. Figure 9The method 900 shown comprises: performing 902 conversion between visual media data and a bitstream of the visual media data according to a format rule, wherein the format rule specifies that a supplemental enhancement information (SEI) message referring to a dependent random access point (DRAP) picture is included in the bitstream, and wherein the format rule further specifies that the SEI message includes a syntax element, wherein the syntax element indicates the number of intra random access point (IRAP) pictures or dependent random access point (DRAP) pictures within the same codec layer video sequence (CLVS) as the DRAP picture.

[0379] 2. The method according to solution 1, wherein an IRAP picture or a DRAP picture is allowed to be included in a valid entry of a reference picture list of a DRAP picture.

[0380] 3. The method according to solution 1, wherein the syntax element is encoded and decoded as u(3), where u(3) is an unsigned integer using 3 bits, or the syntax element is encoded and decoded as ue(v), where ue(v) is an unsigned integer using an exponential Golomb code.

[0381] 4. The method according to solution 1, wherein the format rule further stipulates that the SEI message also includes a list of IRAP pictures or random access point (RAP) picture identifiers of DRAP pictures in the same codec layer video sequence (CLVS) as the DRAP picture.

[0382] 5. The method according to solution 4, wherein an IRAP picture or a DRAP picture is allowed to be included in a valid entry of a reference picture list of a DRAP picture.

[0383] 6. The method according to solution 4, wherein each of the list of RAP picture identifiers is encoded to be the same as the RAP picture identifier of the DRAP picture associated with the SEI message.

[0384] 7. The method according to solution 4, wherein the identifiers in the list have values ​​corresponding to the i-th RAP picture, i being equal to or greater than 0, and wherein the values ​​of the RAP picture identifiers increase in increasing order of the value i.

[0385] 8. A method according to solution 7, wherein each identifier in the decoding list is encoded using the ue(v) of the delta between the value of the i-th DRAP picture identifier and 1) the value of the (i-1)th DRAP picture or IRAP picture identifier, where i is greater than 0, or the ue(v) of the delta between 2) 0, where i is equal to 0.

[0386] 9. The method according to solution 4, wherein each identifier in the list is encoded to represent a picture order count (POC) value of a RAP picture.

[0387] 10. The method according to solution 4, wherein each identifier in the list is encoded to represent picture order count (POC) delta information relative to a POC value of the IRAP picture associated with the SEI message.

[0388] 11. A method according to solution 4, wherein each identifier in the list is encoded to represent the POC delta information between the picture order count (POC) value of the current picture and 1) the POC value of the (i-1)th DRAP picture or IRAP picture, where i is greater than 0, or 2) the POC delta information between the POC values ​​of the IRAP picture associated with the SEI message.

[0389] 12. The method of solution 4, wherein the list comprises identifiers corresponding to the i-th RAP picture, the j-th RAP picture, where i is less than j, and wherein the i-th RAP picture precedes the j-th RAP picture in decoding order.

[0390] 13. The method of any one of solutions 1 to 12, wherein performing the conversion comprises generating a bitstream from the visual media data.

[0391] 14. The method of any of solutions 1 to 12, wherein performing the conversion comprises reconstructing the visual media data from the bitstream.

[0392] 15. A device for processing visual media data, comprising a processor and a non-volatile memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: perform conversion between visual media data and a bitstream of visual media data according to format rules, wherein the format rules specify that a supplemental enhancement information (SEI) message referring to a dependent random access point (DRAP) picture is included in the bitstream, and wherein the format rules further specify that the SEI message includes a syntax element, wherein the syntax element indicates the number of intra random access point (IRAP) pictures or dependent random access point (DRAP) pictures within the same codec layer video sequence (CLVS) as the DRAP picture.

[0393] 16. An apparatus according to solution 15, wherein an IRAP picture or a DRAP picture is allowed to be included in a valid entry of a reference picture list of a DRAP picture, wherein the syntax element is encoded as u(3), where u(3) is an unsigned integer using 3 bits, or the syntax element is encoded as ue(v), where ue(v) is an unsigned integer using an exponential Golomb code, wherein the format rule further stipulates that the SEI message also includes a list of random access point (RAP) picture identifiers of IRAP pictures or DRAP pictures within the same codec layer video sequence (CLVS) as the DRAP picture, wherein the IRAP picture or the DRAP picture is allowed to be included in a valid entry of a reference picture list of the DRAP picture, and wherein each identifier in the list of RAP picture identifiers is encoded as the same as the RAP picture identifier of the DRAP picture associated with the SEI message.

[0394] 17. A non-transitory computer-readable storage medium storing instructions that cause a processor to: perform conversion between visual media data and a bitstream of visual media data according to format rules, wherein the format rules specify that a supplemental enhancement information (SEI) message referring to a dependent random access point (DRAP) picture is included in the bitstream, and wherein the format rules further specify that the SEI message includes a syntax element that indicates the number of intra random access point (IRAP) pictures or dependent random access point (DRAP) pictures within the same codec layer video sequence (CLVS) as the DRAP picture.

[0395] 18. A non-transitory computer-readable storage medium according to solution 17, wherein an IRAP picture or a DRAP picture is allowed to be included in a valid entry of a reference picture list of a DRAP picture, wherein the syntax element is encoded as u(3), where u(3) is an unsigned integer using 3 bits, or the syntax element is encoded as ue(v), where ue(v) is an unsigned integer using an exponential Golomb code, wherein the format rule further stipulates that the SEI message also includes a list of random access point (RAP) picture identifiers of IRAP pictures or DRAP pictures within the same codec layer video sequence (CLVS) as the DRAP picture, wherein the IRAP picture or the DRAP picture is allowed to be included in a valid entry of a reference picture list of the DRAP picture, and wherein each identifier in the list of RAP picture identifiers is encoded as the same as the RAP picture identifier of the DRAP picture associated with the SEI message.

[0396] 19. A non-transitory computer-readable storage medium storing a bitstream of visual media data generated by a method executed by a visual media data processing device, wherein the method includes: determining that a supplemental enhancement information (SEI) message referring to a dependent random access point (DRAP) picture is included in the bitstream; and generating the bitstream based on the determination.

[0397] 20. A non-transitory computer-readable storage medium according to solution 19, wherein an IRAP picture or a DRAP picture is allowed to be included in a valid entry of a reference picture list of a DRAP picture, wherein the syntax element is encoded as u(3), where u(3) is an unsigned integer using 3 bits, or the syntax element is encoded as ue(v), where ue(v) is an unsigned integer using an exponential Golomb code, wherein the format rule further stipulates that the SEI message also includes a list of random access point (RAP) picture identifiers of IRAP pictures or DRAP pictures within the same codec layer video sequence (CLVS) as the DRAP picture, wherein the IRAP picture or the DRAP picture is allowed to be included in a valid entry of a reference picture list of the DRAP picture, and wherein each identifier in the list of RAP picture identifiers is encoded as the same as the RAP picture identifier of the DRAP picture associated with the SEI message.

[0398] 21. A visual media data processing device, comprising a processor configured to implement the method described in any one or more of solutions 1 to 14.

[0399] 22. A method for storing a bitstream of visual media data, comprising the method of any one of solutions 1 to 14, further comprising storing the bitstream in a non-transitory computer-readable storage medium.

[0400] 23. A computer-readable medium storing program code, which, when executed, causes a processor to implement the method of any one or more of solutions 1 to 14.

[0401] 24. A computer-readable medium storing a bitstream generated according to any of the above methods.

[0402] 25. A visual media data processing device for storing a bitstream, wherein the video processing device is configured to implement any one or more of the methods described in solutions 1 to 14.

[0403] 26. A computer-readable medium, wherein a bit stream on the medium complies with the format rules according to any one of solutions 1 to 14.

[0404] In the solutions described herein, the visual media data corresponds to a video or image. In the solutions described herein, an encoder can conform to the format rules by generating a codec representation according to the format rules. In the solutions described herein, a decoder can use the format rules to parse syntax elements in the codec representation to produce a decoded video, while being aware of the presence and absence of syntax elements according to the format rules.

[0405] In this document, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during conversion from a pixel representation of a video to a corresponding bitstream representation, or vice versa. The bitstream representation of the current video block may, for example, correspond to bits that are collocated or dispersed in different places within the bitstream, as defined by the syntax. For example, a macroblock may be encoded based on an error residual value after transformation and encoding and also using bits from headers and other fields in the bitstream. Furthermore, during conversion, the decoder may, based on this determination, parse the bitstream knowing that some fields may or may not be present, as described in the above solution. Similarly, the encoder may determine whether to include or not include certain syntax fields and generate the codec representation accordingly by including the syntax fields or excluding the syntax fields from the codec representation.

[0406] The disclosed and other solutions, examples, embodiments, modules, and functional operations described in this document can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware (including the structures disclosed in this document and their structural equivalents), or in a combination of one or more thereof. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium, which are used to be executed by a data processing device or to control the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances that affect a machine-readable propagated signal, or a combination of one or more thereof. The term "data processing device" includes all devices, equipment, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, a device may also include code that creates an operating environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof. A propagated signal is an artificially generated signal, such as a machine-generated electrical signal, an optical signal, or an electromagnetic signal, that is generated to encode information for transmission to a suitable receiver device.

[0407] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language (including compiled or interpreted languages), and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or code portions). A computer program can be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0408] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special purpose logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0409] Processors suitable for running computer programs include, for example, general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, a processor will receive instructions and data from read-only memory or random access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, or be operatively coupled to receive data from or transfer data to or from such one or more mass storage devices. However, a computer does not require such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0410] Although this patent document contains many details, these details should not be interpreted as limitations on any subject matter or the scope of what may be claimed, but rather as descriptions of features specified for particular embodiments of particular technologies. Certain features described in this patent document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable subcombination. Furthermore, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be excluded from the combination in some cases, and the claimed combination may be directed to a subcombination or variation of the subcombination.

[0411] Similarly, while operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown, or in sequential order, or that all illustrated operations be performed, in order to achieve desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0412] Only a few implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A method for processing visual media data, comprising: performing conversion between visual media data and a bitstream of said visual media data according to format rules, wherein the format rule specifies that a supplemental enhancement information (SEI) message referencing a second type of dependent random access point (DRAP) picture is included in the bitstream, and the second type of DRAP picture is different from the first type of DRAP picture, The format rule further stipulates that the SEI message includes a syntax element, which indicates the number of intra random access point (IRAP) pictures or second type dependent random access point (DRAP) pictures in the same coding layer video sequence (CLVS) as the second type DRAP picture, allowing the IRAP picture or the second type DRAP picture to be included in the valid entry of the reference picture list of the second type DRAP picture. The first type of DRAP picture is a picture that depends on an IRAP picture and is associated with a first type of SEI message, and the first type of SEI message is a first DRAP indication SEI message. The second type of DRAP picture is a picture that is allowed to depend on an IRAP picture or another DRAP picture and is associated with a second type of SEI message, and the second type of SEI message is a second DRAP indication SEI message.

2. The method according to claim 1, wherein The syntax element is encoded and decoded as u(3), where u(3) is an unsigned integer using 3 bits, or the syntax element is encoded and decoded as ue(v), where ue(v) is an unsigned integer using an exponential Golomb code.

3. The method according to claim 1 or 2, wherein: The format rule further stipulates that the SEI message also includes a list of random access point (RAP) picture identifiers of the IRAP pictures or the second type of DRAP pictures within the same codec layer video sequence (CLVS) as the second type of DRAP pictures.

4. The method according to claim 3, wherein: Each identifier in the list of RAP picture identifiers is encoded to be the same as a RAP picture identifier of a DRAP picture of the second type associated with the SEI message.

5. The method according to claim 3, wherein: The identifiers in the list have values ​​corresponding to the i-th RAP picture, i being equal to or greater than 0, and wherein the values ​​of the RAP picture identifiers increase in increasing order of the value i.

6. The method according to claim 5, wherein: Each identifier in the list is encoded using the ue(v) of the increment between the value of the i-th second-type DRAP picture identifier and 1) the value of the (i-1)-th second-type DRAP picture or IRAP picture identifier, where i is greater than 0, or the ue(v) of the increment between and 2) 0, where i is equal to 0.

7. The method according to claim 4, wherein: Each identifier in the list is encoded to represent a picture order count (POC) value of a RAP picture.

8. The method according to claim 4, wherein Each identifier in the list is encoded to represent picture order count (POC) delta information relative to a POC value of the IRAP picture associated with the SEI message.

9. The method according to claim 4, wherein: Each identifier in the list is encoded to represent the POC delta information between the picture order count (POC) value of the current picture and 1) the POC value of the (i-1)th second type DRAP picture or IRAP picture, where i is greater than 0, or 2) the POC delta information between the POC values ​​of the IRAP picture associated with the SEI message.

10. The method according to claim 3, wherein: The list includes identifiers corresponding to an i-th RAP picture, a j-th RAP picture, where i is less than j, and where the i-th RAP picture precedes the j-th RAP picture in decoding order.

11. The method according to claim 1 or 2, wherein: The performing of the conversion includes generating the bitstream from the visual media data.

12. The method according to claim 1 or 2, wherein: The performing of the conversion includes reconstructing the visual media data from the bitstream.

13. An apparatus for processing visual media data, comprising a processor and a non-transitory memory having instructions thereon, wherein: The instructions, when executed by the processor, cause the processor to: performing conversion between visual media data and a bitstream of said visual media data according to format rules, wherein the format rule specifies that a supplemental enhancement information (SEI) message referencing a second type of dependent random access point (DRAP) picture is included in the bitstream, and the second type of DRAP picture is different from the first type of DRAP picture, The format rule further stipulates that the SEI message includes a syntax element, which indicates the number of intra random access point (IRAP) pictures or second type dependent random access point (DRAP) pictures in the same coding layer video sequence (CLVS) as the second type DRAP picture, allowing the IRAP picture or the second type DRAP picture to be included in the valid entry of the reference picture list of the second type DRAP picture. The first type of DRAP picture is a picture that depends on an IRAP picture and is associated with a first type of SEI message, and the first type of SEI message is a first DRAP indication SEI message. The second type of DRAP picture is a picture that is allowed to depend on an IRAP picture or another DRAP picture and is associated with a second type of SEI message, and the second type of SEI message is a second DRAP indication SEI message.

14. The device according to claim 13, wherein The syntax element is encoded as u(3), where u(3) is an unsigned integer using 3 bits, or the syntax element is encoded as ue(v), where ue(v) is an unsigned integer using an exponential Golomb code, The format rule further stipulates that the SEI message also includes a list of random access point (RAP) picture identifiers of the IRAP picture or the second type of DRAP picture in the same codec layer video sequence (CLVS) as the second type of DRAP picture, Each identifier in the list of RAP picture identifiers is encoded and decoded to be the same as the RAP picture identifier of the second type of DRAP picture associated with the SEI message.

15. A non-transitory computer-readable storage medium storing instructions that cause a processor to: performing conversion between visual media data and a bitstream of said visual media data according to format rules, in, The format rule specifies that a supplemental enhancement information (SEI) message referencing a second type of dependent random access point (DRAP) picture is included in the bitstream, and the second type of DRAP picture is different from the first type of DRAP picture, The format rule further stipulates that the SEI message includes a syntax element, which indicates the number of intra random access point (IRAP) pictures or second type dependent random access point (DRAP) pictures in the same coding layer video sequence (CLVS) as the second type DRAP picture, allowing the IRAP picture or the second type DRAP picture to be included in the valid entry of the reference picture list of the second type DRAP picture. The first type of DRAP picture is a picture that depends on an IRAP picture and is associated with a first type of SEI message, and the first type of SEI message is a first DRAP indication SEI message. The second type of DRAP picture is a picture that is allowed to depend on an IRAP picture or another DRAP picture and is associated with a second type of SEI message, and the second type of SEI message is a second DRAP indication SEI message.

16. The non-transitory computer-readable storage medium of claim 15, wherein: The syntax element is encoded as u(3), where u(3) is an unsigned integer using 3 bits, or the syntax element is encoded as ue(v), where ue(v) is an unsigned integer using an exponential Golomb code, The format rule further stipulates that the SEI message also includes a list of random access point (RAP) picture identifiers of the IRAP picture or the second type of DRAP picture in the same codec layer video sequence (CLVS) as the second type of DRAP picture, Each identifier in the list of RAP picture identifiers is encoded and decoded to be the same as the RAP picture identifier of the second type of DRAP picture associated with the SEI message.

17. A non-transitory computer-readable storage medium storing a bit stream of visual media data, the bit stream being generated by a method performed by a visual media data processing apparatus, wherein: The method comprises: determining that a supplemental enhancement information (SEI) message referencing a second type of dependent random access point (DRAP) picture is included in the bitstream, wherein the second type of DRAP picture is different from the first type of DRAP picture; and generating the bitstream based on the determination; The SEI message includes a syntax element indicating the number of intra random access point (IRAP) pictures or second type dependent random access point (DRAP) pictures in the same codec layer video sequence (CLVS) as the second type DRAP picture, allowing the IRAP picture or the second type DRAP picture to be included in a valid entry in the reference picture list of the second type DRAP picture. The first type of DRAP picture is a picture that depends on an IRAP picture and is associated with a first type of SEI message, and the first type of SEI message is a first DRAP indication SEI message. The second type of DRAP picture is a picture that is allowed to depend on an IRAP picture or another DRAP picture and is associated with a second type of SEI message, and the second type of SEI message is a second DRAP indication SEI message.

18. The non-transitory computer-readable storage medium of claim 17, wherein: The syntax element is encoded as u(3), where u(3) is an unsigned integer using 3 bits, or the syntax element is encoded as ue(v), where ue(v) is an unsigned integer using an exponential Golomb code, It is further provided that the SEI message also includes a list of random access point (RAP) picture identifiers of the IRAP picture or the second type of DRAP picture in the same codec layer video sequence (CLVS) as the second type of DRAP picture, Each identifier in the list of RAP picture identifiers is encoded and decoded to be the same as the RAP picture identifier of the second type of DRAP picture associated with the SEI message.

19. A method for storing a video bitstream, comprising: determining that a supplemental enhancement information (SEI) message referencing a second type of dependent random access point (DRAP) picture is included in the bitstream, wherein the second type of DRAP picture is different from the first type of DRAP picture; and generating the bitstream based on the determination; storing the bitstream in a non-transitory computer-readable recording medium; The SEI message includes a syntax element indicating the number of intra random access point (IRAP) pictures or second type dependent random access point (DRAP) pictures in the same codec layer video sequence (CLVS) as the second type DRAP picture, allowing the IRAP picture or the second type DRAP picture to be included in a valid entry in the reference picture list of the second type DRAP picture. The first type of DRAP picture is a picture that depends on an IRAP picture and is associated with a first type of SEI message, and the first type of SEI message is a first DRAP indication SEI message. The second type of DRAP picture is a picture that is allowed to depend on an IRAP picture or another DRAP picture and is associated with a second type of SEI message, and the second type of SEI message is a second DRAP indication SEI message.

Images