DRAP and EDRAP in ISOBMFF

Abstract

A method of processing video data. The method comprises: generating a sequence of samples (sample Seq), the sequence of samples comprising a most recent previous streaming access point (SAP) of type 1, type 2 or type 3, a first dependent random access point (DRAP) sample (sample A), and all samples following the first DRAP sample in a decoding order and an output order in a track comprising at least one second DRAP sample (sample B); for each second DRAP sample in the sequence of samples, ensuring that all data used to process the second DRAP sample is accessible in the reference sample entry, in the second DRAP sample itself, or in one of the samples prior to the second DRAP sample in the decoding order and present in the sequence of samples; and performing conversion between a video including video data and a bitstream of the video data based on the data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 390,184, filed on July 18, 2022. The above patent application is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to the generation, storage and consumption of digital audio-visual media information in file format. Background Art

[0004] Digital video occupies the largest share of bandwidth used on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, the demand for bandwidth used by digital video is likely to continue to grow. Summary of the invention

[0005] The present disclosure relates to media file formats. In particular, the present disclosure relates to improved support for dependent random access points (DRAPs) and extended dependent random access points (EDRAPs) in the International Organization for Standardization (ISO) Base Media File Format (ISOBMFF). These ideas can be applied alone or in various combinations to media files according to any media file format, for example, ISOBMFF and file formats derived from ISOBMFF.

[0006] A first aspect relates to a method for processing video data, comprising: generating a sample sequence (sampleSeq), the sample sequence comprising a most recent previous stream access point (SAP) of type 1, type 2 or type 3, a first dependent random access point (DRAP) sample (sampleA), and all samples that follow the first DRAP sample in both decoding order and output order in a track including at least one second DRAP sample (sampleB); for each second DRAP sample in the sample sequence, ensuring that all data for processing the second DRAP sample is accessible; and performing conversion between a video comprising video data and a bitstream of the video data based on the data.

[0007] A second aspect relates to another method for processing video data, comprising: generating a sample sequence (sampleSeq), the sample sequence comprising the most recent previous stream access point (SAP) of type 1, type 2 or type 3, a first extended dependent random access point (EDRAP) sample (sampleA) identified by ref_edrap_idx_delta[i] in decoding order, where i is in the range of 0 to num_ref_edrap_samples-1, including 0 and num_ref_edrap_samples-1, and all samples that follow the first EDRAP sample in both decoding order and output order in a track including at least one second EDRAP sample (sampleB); for each second EDRAP sample in the sample sequence, ensuring that all data for processing the second EDRAP sample is accessible; and based on the data, performing conversion between a video comprising video data and a bitstream of the video data.

[0008] A third aspect relates to an apparatus for processing video data, comprising: a processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of the first or second aspect.

[0009] A fourth aspect relates to a non-transitory computer-readable medium, which includes a computer program product for use by a video codec device, the computer program product including computer executable instructions stored on the non-transitory computer-readable medium, so that when executed by a processor, the video codec device performs the method of the first or second aspect.

[0010] A fifth aspect relates to a non-transitory computer-readable recording medium storing a bit stream of a video generated by a method performed by a video processing device, wherein the method includes the method of the first or second aspect.

[0011] A sixth aspect relates to a method for storing a bitstream of a video comprising the method of the first or second aspect, wherein performing conversion between a video comprising video data and a bitstream of the video data comprises generating the bitstream, and wherein the method further comprises storing the bitstream in a non-temporary computer-readable recording medium.

[0012] For clarity, any of the foregoing embodiments may be combined with any or more of the other foregoing embodiments to create new embodiments within the scope of the present disclosure.

[0013] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] For a more complete understanding of the present disclosure, reference is now made to the following brief description, taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0015] Figure 1 An example of encoding and decoding of an intra random access point (IRAP) picture is shown.

[0016] Figure 2 An example of a picture received by a decoder is shown.

[0017] Figure 3 An example of encoding and decoding of a DRAP picture is shown.

[0018] Figure 4 An example of a picture received by a decoder is shown.

[0019] Figure 5 An example of encoding and decoding of an EDRAP picture is shown.

[0020] Figure 6 An example of a picture received by a decoder is shown.

[0021] Figure 7 An example of EDRAP-based streaming is shown.

[0022] Figure 8 An example of a picture received by a decoder is shown.

[0023] Fig. 9 is a block diagram illustrating an exemplary video processing system.

[0024] Fig.10 is a block diagram of an exemplary video processing device.

[0025] Fig.11 is a flow chart of an exemplary method of video processing.

[0026] Fig.12 is a block diagram illustrating an exemplary video coding system.

[0027] Fig.13 is a block diagram illustrating an exemplary encoder.

[0028] Fig.14 is a block diagram illustrating an exemplary decoder.

[0029] Fig.15 is a schematic diagram of an exemplary encoder. DETAILED DESCRIPTION

[0030] It should be understood at the outset that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or yet to be developed. The present disclosure should not be limited in any way to the illustrative implementations, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but may be modified within the full scope of the appended claims and their equivalents.

[0031] The section headings used in this document are for ease of understanding and do not limit the applicability of the techniques and embodiments disclosed in each section to only that section. In addition, the techniques described herein are applicable to other video codec protocols and designs.

[0032] 1. Summary.

[0033] The present disclosure relates to media file formats. In particular, the present disclosure relates to improved support for Dependent Random Access Points (DRAPs) and Extended Dependent Random Access Points (EDRAPs) in the ISO Base Media File Format (ISOBMFF). These ideas can be applied alone or in various combinations to media files according to any media file format, for example, ISOBMFF and file formats derived from ISOBMFF.

[0034] 2. Background.

[0035] 2.1 Video Codec Standards

[0036] Video codec standards have evolved primarily through the development of the well-known International Telecommunication Union-Telecommunication Standardization Sector (ITU-T) and ISO / International Electrotechnical Commission (IEC) standards. ITU-T developed H.261 and H.263, ISO / IEC developed Moving Picture Experts Group (MPEG)-1 and MPEG-4 Video, and the two organizations jointly developed H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Codec (AVC) and H.265 / HEVC standards. Since H.262, video codec standards have been based on a hybrid video codec structure that utilizes temporal prediction plus transform codecs. In order to explore future video codec technologies beyond HEVC, the Video Codec Experts Group (VCEG) and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and input them into a reference software called the Joint Exploration Model (JEM). JVET was later renamed the Joint Video Experts Team (JVET) when the Versatile Video Codec (VVC) project officially began. VVC is a new codec standard that aims to reduce bit rate by 50% compared to HEVC. The standard was finalized by JVET at its 19th meeting that ended on July 1, 2020.

[0037] The VVC standard (ITU-T H.266 | ISO / IEC 23090-3) and the associated Versatile Supplementary Enhancement Information (VSEI) standard (ITU-T H.274 | ISO / IEC 23002-7) have been designed for the widest range of applications, including traditional uses such as television broadcasting, video conferencing or playback from storage media, as well as newer and more advanced use cases such as adaptive bitrate streaming, video region extraction, composition and merging of content from multiple codec video bitstreams, multi-view video, scalable layered codecs and viewport-adaptive 360° immersive media.

[0038] 2.2 File format standards.

[0039] Streaming media applications are typically based on Internet Protocol (IP), Transmission Control Protocol (TCP), and Hypertext Transfer Protocol (HTTP) delivery methods, and typically rely on file formats such as ISOBMFF. One such streaming system is Dynamic Adaptive Streaming over HTTP (DASH). In order to use a video format with ISOBMFF and DASH, file format specifications specific to that video format, such as the AVC file format and the HEVC file format, will be needed for encapsulating video content in ISOBMFF tracks and DASH representations and segments. Important information about the video bitstream (e.g., profiles, tiers, and levels, and many other information) will need to be exposed as file format level metadata and / or DASH Media Presentation Description (MPD) for content selection purposes, e.g., for selecting appropriate media segments, for initialization at the start of a streaming session, and for stream adaptation during a streaming session. Similarly, in order to use an image format with ISOBMFF, file format specifications specific to that image format, such as the AVC image file format and the HEVC image file format, will be useful.

[0040] 2.3 Random access and its support in HEVC and VVC.

[0041] Random access refers to accessing and decoding a bitstream starting from a picture that is not the first picture of the bitstream in decoding order. To support tuning and channel switching in broadcast / multicast and multi-party video conferencing, seeking in local playback and streaming, and stream adaptation in streaming, the bitstream needs to include frequent random access points, which are usually intra-coded pictures, but can also be inter-coded pictures (for example, in the case of progressive decoding refresh).

[0042] HEVC includes the signaling of intra-frame random access point (IRAP) pictures in the network abstraction layer (NAL) unit header via NAL unit types. Three types of IRAP pictures are supported, namely instantaneous decoder refresh (IDR), clean random access (CRA), and broken link access (BLA) pictures. IDR pictures constrain the inter-picture prediction structure to not reference any pictures before the current group of pictures (GOP), which is often referred to as a closed GOP random access point. CRA pictures are less restrictive by allowing certain pictures to reference pictures before the current GOP, and all pictures are discarded in the case of random access. CRA pictures are often referred to as open GOP random access points. BLA pictures usually originate from the concatenation of two bitstreams or parts thereof in a CRA picture, such as during stream switching. In order to enable the system to better use IRAP pictures, a total of six different NAL units are defined to signal the properties of IRAP pictures, which can be used to better match the stream access point types defined in ISOBMFF for random access support in DASH.

[0043] VVC supports three types of IRAP pictures, two types of IDR pictures (one type with or the other type without an associated random access decodable preamble (RADL) picture), and one type of CRA picture. These are essentially the same as HEVC. HEVC's BLA picture type is not included in VVC, primarily for two reasons: i) The basic functionality of BLA pictures can be achieved with CRA pictures plus the end of a sequence NAL unit, the presence of which indicates that subsequent pictures start a new coded video sequence (CVS) in a single-layer bitstream. ii) During the development of VVC, it was desirable to specify fewer NAL unit types than HEVC, as indicated by the use of 5 bits instead of 6 bits for the NAL unit type field in the NAL unit header.

[0044] Another key difference between VVC and HEVC in terms of random access support is that VVC supports gradual decoding refresh (GDR) in a more standardized way. In GDR, decoding of the bitstream can start from an inter-frame coded picture. Although not the entire picture area can be correctly decoded at the beginning, after multiple pictures, the entire picture area will be correct. AVC and HEVC also support GDR, using the recovery point supplementary enhancement information (SEI) message to signal GDR random access points and recovery points. In VVC, a new NAL unit type is specified to indicate GDR pictures, and the recovery point is signaled in the picture header syntax structure. CVS and bitstreams are allowed to start with GDR pictures. This means that the entire bitstream is allowed to include only inter-frame coded pictures, without a single intra-frame coded picture. The main benefit of specifying GDR support in this way is to provide consistent behavior for GDR. GDR enables the encoder to smooth the bitrate of the bitstream by distributing intra-coded strips or blocks across multiple pictures, instead of intra-coding the entire picture, allowing for a significant reduction in end-to-end latency, which is considered more important today than before as ultra-low latency applications such as wireless displays, online gaming, drone-based applications, etc. become increasingly popular.

[0045] Another GDR-related feature of VVC is virtual boundary signaling. The boundary between the refresh area (i.e., the correctly decoded area) and the unrefreshed area at the picture between the GDR picture and its recovery point can be signaled as a virtual boundary, and when signaled, loop filtering across the boundary will not be applied, so there will be no decoding mismatch of some samples at or near the boundary. This may be useful when the application determines to display the correctly decoded area during the GDR process.

[0046] IRAP pictures and GDR pictures may be collectively referred to as random access point (RAP) pictures.

[0047] 2.4 Dependent Random Access Point (DRAP) Sample Group.

[0048] ISOBMFF includes features of the DRAP sample set. The revised specifications for the DRAP sample set features are provided below.

[0049] 10.8 Dependent Random Access Point (DRAP) Sample Group.

[0050] 10.8.1 Definitions

[0051] A dependent random access point (DRAP) sample is a sample such that all samples following it in decoding order and output order can be correctly decoded if the most recent previous SAP sample of type 1, 2, or 3 is available for reference.

[0052] NOTE 1 The most recent SAP sample can be a synchronized sample or tagged by a SAP sample group.

[0053] NOTE 2 DRAP samples can be used only in combination with SAP samples of type 1, 2 and 3. This is to enable the functionality of creating a decodable sequence of samples by concatenating the preceding SAP sample and the DRAP sample and the samples following the DRAP sample in decoding order and output order.

[0054] 10.8.2 Syntax

[0055] class VisualDRAPEntry()

[0056] extends VisualSampleGroupEntry('drap'){

[0057] unsigned int(3)DRAP_type;

[0058] unsigned int(29)reserved=0;

[0059] }

[0060] 10.8.3 Semantics

[0061] DRAP_type is a non-negative integer. When DRAP_type is in the range of 1 to 3, it indicates the SAP_type (as specified in Appendix I) to which the DRAP sample should have corresponded if the DRAP sample did not depend on the most recent previous SAP. Other type values ​​are reserved.

[0062] reserved SHALL be equal to 0. The semantics of this subclause applies only to sample group description entries with reserved equal to 0. When parsing this sample group, the parser shall allow and ignore sample group description entries with reserved greater than 0.

[0063] 2.5. Video encoding, decoding, storage and streaming based on Extended Dependent Random Access Point (EDRAP)

[0064] 2.5.1. Concept and Standard Support

[0065] This paper describes the concepts of video encoding, decoding, storage and streaming based on EDRAP and Figure 1-8 The application (eg, adaptive streaming) determines the frequency of the random access point (RAP), for example, the RAP period is 1s or 2s. Figure 1 As shown, by convention, RAP is provided through the encoding and decoding of IRAP pictures. Figure 1 Includes various Instantaneous Decoder Refresh (IDR) pictures and Clean Random Access (CRA) pictures.

[0066] Note that inter prediction references to non-critical pictures between RAP pictures are not shown, and the output order is from left to right.

[0067] When randomly accessed from CRA6, the decoder receives and correctly decodes the picture, such as Figure 2 shown.

[0068] like Figure 3 As shown, the DRAP method provides improved coding efficiency by allowing DRAP pictures (and subsequent pictures) to refer to previous IRAP pictures for inter-frame prediction. Note that inter-frame prediction of non-critical pictures between RAP pictures is not shown, and the output order is from left to right.

[0069] When randomly accessed from DRAP6, the decoder receives and correctly decodes the picture, such as Figure 4 shown.

[0070] like Figure 5 As shown, the EDRAP method provides more flexibility by allowing EDRAP pictures (and subsequent pictures) to reference several earlier RAP pictures (IRAP or EDRAP). Note that inter-prediction of non-critical pictures between RAP pictures is not shown, and the output order is from left to right.

[0071] When randomly accessed from EDRAP6, the decoder receives and correctly decodes the picture, such as Figure 6 shown.

[0072] Figure 7-8 The ERAP-based video stream is shown in Figure 1. When segments starting from EDRAP6 are randomly accessed or switched to this segment, the decoder receives and correctly decodes these segments, such as Figure 8 shown.

[0073] At the time of this disclosure, EDRAP-based video codec is supported by EDRAP indication SEI messages in a developing amendment to the VSEI standard, the storage part is supported by EDRAP sample groups and associated external stream track references in a developing amendment to the ISOBMFF standard, and the streaming part is supported by main stream representation (MSR) and external stream representation (ESR) descriptors in a developing amendment to the DASH standard.

[0074] 2.5.2.EDRAP Indication SEI Message

[0075] An amendment to the VSEI standard is under development. The latest draft specification for this amendment includes the specification of the EDRAP indication SEI message.

[0076] The syntax and semantics of the EDRAP indication SEI message are as follows.

[0077]

[0078] A picture associated with an EDRAP indication SEI message is called an EDRAP picture.

[0079] The presence of the EDRAP indication SEI message indicates the constraints on picture order and picture references specified in this subclause. These constraints enable the decoder to correctly decode the EDRAP picture and the pictures in the same layer and after the EDRAP picture in both decoding order and output order, without decoding any other pictures in the same layer except for the picture referenceablePictures list, which consists of a list of IRAP or EDRAP pictures in decoding order within the same codec layer video sequence (CLVS) and identified by the edrap_ref_rap_id[i] syntax element.

[0080] The constraints indicated by the presence of the EDRAP indication SEI message (which shall all apply) are as follows:

[0081] EDRAP pictures are trailing pictures.

[0082] EDRAP pictures have a temporal sub-layer identifier equal to 0.

[0083] Except for referenceablePictures, an EDRAP picture does not include any pictures in the same layer in its active entry in the reference picture list.

[0084] Except for referenceablePictures, any picture in the same layer and following the EDRAP picture in both decoding order and output order does not include in its active entry in the reference picture list any picture in the same layer and preceding the EDRAP picture in either decoding order or output order.

[0085] Any picture in the list referenceablePictures does not include in its active entry in the reference picture list any picture that is in the same layer and is not a picture at an earlier position in the list referenceablePictures.

[0086] NOTE - Therefore, the first picture in referenceablePictures, even when it is an EDRAP picture instead of an IRAP picture, does not include any pictures from the same layer in the active entry of its reference picture list.

[0087] edrap_rap_id_minus1 plus 1 specifies the RAP picture identifier of the EDRAP picture, denoted as RapPicId.

[0088] Each IRAP or EDRAP picture is associated with a RapPicId value. The RapPicId value of an IRAP picture is inferred to be equal to 0. The RapPicId values ​​of any two EDRAP pictures associated with the same IRAP picture shall be different.

[0089] edrap_leading_pictures_decodable_flag equal to 1 specifies that both of the following constraints apply:

[0090] - Any picture in the same layer and following an EDRAP picture in decoding order shall follow in output order any picture in the same layer and preceding the EDRAP picture in decoding order.

[0091] —Except for referenceablePictures, any picture in the same layer and following the EDRAP picture in decoding order and preceding the EDRAP picture in output order shall not include in its active entry in the reference picture list any picture in the same layer and preceding the EDRAP picture in decoding order.

[0092] edrap_leading_pictures_decodable_flag equal to 0 imposes no such constraint.

[0093] In bitstreams conforming to this version of this specification, edrap_reserved_zero_12bits shall be equal to 0. Other values ​​of edrap_reserved_zero_12bits are reserved for use by ITU-T | ISO / IEC. A decoder shall ignore the value of edrap_reserved_zero_12bits.

[0094] edrap_num_ref_rap_pics_minus1 plus 1 indicates the number of IRAP or EDRAP pictures that are within the same CLVS as the EDRAP picture and that can be included in the active entry of the reference picture list of the EDRAP picture.

[0095] edrap_ref_rap_id[i] indicates the RapPicId of the ith RAP picture that may be included in the active entry of the reference picture list of the EDRAP picture. The ith RAP picture shall be an IRAP picture associated with the current EDRAP picture or an EDRAP picture associated with the same IRAP picture as the current EDRAP picture.

[0096] 2.5.3. EDRAP sample groups and associated external stream track references.

[0097] An amendment to the ISOBMFF standard is under development. The latest draft specification for this amendment includes specifications for the EDRAP sample set and the associated external flow track reference.

[0098] The specifications of these two ISOBMFF features are as follows.

[0099] 3.1 Definition

[0100] EDRAP sample - a sample for which all subsequent samples in decoding and output order can be correctly decoded, provided that the most recent previous SAP sample of type 1, 2, or 3 and zero or more previous EDRAP samples are available when decoding the sample and subsequent samples.

[0101] Item Note 1. The most recent previous SAP sample and zero or more previous EDRAP samples of type 1, 2, or 3 as described above are referred to as the previous SAP and EDRAP samples required by the EDRAP sample.

[0102] Random access - decoding of an elementary stream starts from a specific access unit, rather than decoding any access unit earlier in the decoding order of the elementary stream

[0103] Item Note 1. Synchronous samples and SAP provide random access capability.

[0104] 3.2 Abbreviations

[0105] EDRAP Extended Dependent Random Access Point

[0106] 8.3.3.4 Associated external stream track reference.

[0107] A track reference of type 'aest' (meaning "associated external stream track") may be included in a video track.

[0108] When a video track has a track reference of type 'aest', the following applies:

[0109] - A video track shall have at least one sample associated with an EDRAP sample group.

[0110] -The referenced track should meet the following constraints:

[0111] - Each sample in the reference track should be identified as a sync sample.

[0112] - The header flags track_in_movie and track_in_preview of the reference track shall both be equal to 0.

[0113] -The reference track should use a constrained scheme as follows:

[0114] - The scheme_type field in the SchemeTypeBox in the RestrictedSchemeInfoBox is equal to 'tspt', and the value of the mode field in the SamplePackingInformationBox is equal to 1.

[0115] -Bit 0 of the Flags field of SchemeTypeBox is equal to 0, making the value of (Flags & 0x000001) equal to 0.

[0116] - For each sample sampleA in the video track associated with the EDRAP sample group, there shall be one and only one sample sampleB in the reference track that has the same decoding time as sampleA, and multiple consecutive samples starting from sampleB in the reference track shall contain all media data of the previous SAP and EDRAP samples required by sampleA. The consecutive samples in the reference track shall precede sampleC corresponding to another sample in the video track associated with the EDRAP sample group.

[0117] 8.20 Transformed Sample Package Tracks

[0118] 8.20.1 Introduction

[0119] When a restricted scheme with SchemeType 'tspt' is used for tracks, the samples associated with a sample entry may contain more than one sample of the original track on which a transform has been applied to generate the current track. Such current tracks are called sample-wrapped tracks.

[0120] 8.20.2 Transformed sample packaging information box.

[0121] 8.20.2.1 Definitions

[0122] Box type: 'tspi'

[0123] Container: SchemeInformationBox

[0124] Mandatory: yes (when SchemeType is 'tspt')

[0125] Quantity: One

[0126] 8.20.2.2 Syntax

[0127] aligned(8)class TransformedSamplePackingInformationBox extendsFullBox('tspi',version=0,flags=0)

[0128] {

[0129] unsigned int(8)mode;

[0130] }

[0131] 8.20.2.3 Semantics

[0132] A mode equal to 0 specifies that all samples of the original stream have been retained in the transform. A value of 1 specifies that only some samples have been retained. All other values ​​are reserved for future use.

[0133] 10.11 Extended DRAP (EDRAP) Sample Group

[0134] 10.11.1 Definitions

[0135] The EDRAP sample group records the EDRAP samples in a track. This sample group is similar to the DRAP sample group specified in subclause 10.8; however, it is able to signal additional samples, which can also be used for random access, but with more complex dependencies.

[0136] NOTE 1: Similar to DRAP samples, EDRAP samples can be used in combination with SAP samples of types 1, 2, and 3 only.

[0137] NOTE 2: A DRAP sample is always an EDRAP sample.

[0138] 10.11.2 Syntax

[0139]

[0140]

[0141] 10.11.3 Semantics

[0142] edrap_type is a non-negative integer. When edrap_type is in the range of 1 to 3, it indicates the SAP_type (as specified in Appendix I) to which the EDRAP sample would have corresponded if the EDRAP sample had not depended on the most recent previous SAP or other EDRAP samples. Other type values ​​are reserved.

[0143] num_ref_edrap_pics indicates the number of other EDRAP samples that are earlier than the EDRAP sample in decoding order and that, when decoding starts from an EDRAP sample, need to be referenced to be able to correctly decode the EDRAP sample and all samples that follow the EDRAP sample in both decoding and output order.

[0144] NOTE: EDRAP samples that are also DRAP samples will have num_ref_edrap_pics set to 0.

[0145] reserved SHALL be equal to 0. The semantics of this subclause applies only to sample group description entries with reserved equal to 0. When parsing this sample group, the parser shall allow and ignore sample group description entries with reserved greater than 0.

[0146] ref_edrap_idx_delta[i] indicates the i-th required previous EDRAP sample of the current EDRAP sample. It is assumed that the EDRAP sample list associated with a SAP sample of type 1, 2, or 3 is all EDRAP samples after the SAP sample and before the next SAP sample (if any). The EDRAP sample index is defined as the index of the EDRAP sample list. The value of ref_edrap_idx_delta[i] is equal to the EDRAP sample index of the current EDRAP sample and the EDRAP sample index of the i-th required previous EDRAP sample. A value of 1 indicates that the i-th EDRAP sample is the last EDRAP sample before the EDRAP sample in decoding order, a value of 2 indicates that the i-th EDRAP sample is the second to last EDRAP sample before the EDRAP sample in decoding order, and so on.

[0147] 3. Problem.

[0148] The existing designs of the DRAP sample group and the EDRAP sample group have the following problems:

[0149] 1) For a DRAP sample group, when randomly accessing from a DRAP sample sampleA, the parameter set required for decoding sampleA or a sample that follows sampleA in both decoding and output order, such as a picture parameter set (PPS) for AVC, HEVC or VVC or an adaptation parameter set (APS) for VVC, may be present in sample sampleB, which is between the nearest previous SAP sample of type 1, 2 or 3 and sampleA. In this case, in order to be able to decode sampleA, not only the SAP sample but also sampleB (at least part of it) is required.

[0150] 2) For an EDRAP sample group, when randomly accessing from an EDRAP sample sampleA, a parameter set required for decoding sampleA or a sample that follows sampleA in both decoding and output order, such as a picture parameter set (PPS) for AVC, HEVC or VVC or an adaptation parameter set (APS) for VVC, may be present in a sample sampleB that is between the nearest previous SAP sample of type 1, 2 or 3 and sampleA and is not one of the required previous EDRAP samples. In this case, in order to be able to decode sampleA, not only the required SAP and EDRAP samples are required, but also sampleB (at least part of it).

[0151] 4. Solution to the problem

[0152] In order to solve the above problems, the methods summarized as follows are disclosed. The present invention should be considered as an example to explain the general concept and should not be interpreted in a narrow way. In addition, these inventions can be applied alone or in combination in any way.

[0153] 1) To solve the first problem, the following constraints are specified:

[0154] The following applies to any DRAP sample sampleA that is mapped to a DRAP sample group:

[0155] Assume sampleSeq is a sample sequence consisting of the following samples, which are arranged in the order of the following bullet points:

[0156] The most recent previous SAP sample of type 1, 2, or 3,

[0157] sampleA, and

[0158] All samples in the track that come after sampleA in both decoding and output order.

[0159] For each sample sampleB in sampleSeq, all data needed to process sampleB shall be accessible in the referenced sample entry, sampleB itself, or in any sample that precedes sampleB in decoding order and is present in sampleSeq.

[0160] NOTE: For some video codecs, all the data needed to process sample sampleB includes the parameter set needed to decode sampleB.

[0161] 2) To solve the second problem, the following constraints are specified:

[0162] The following applies to any EDRAP sample sampleA that is mapped to the EDRAP sample group:

[0163] Assume sampleSeq is a sample sequence consisting of the following samples, which are arranged in the order of the following bullet points:

[0164] The most recent previous SAP sample of type 1, 2, or 3,

[0165] the EDRAP sample identified by ref_edrap_idx_delta[i] in decoding order, where i is in the range of 0 to num_ref_edrap_samples-1 (inclusive),

[0166] sampleA, and

[0167] All samples in the track that come after sampleA in both decoding and output order.

[0168] For each sample sampleB in sampleSeq, all data required to process sampleB shall be accessible in the referenced sample entry, sampleB itself, or in any sample that precedes sampleB in decoding order and is present in sampleSeq.

[0169] NOTE: For some video codecs, all the data needed to process sample sampleB includes the parameter set needed to decode sampleB.

[0170] Example

[0171] The following are some exemplary embodiments of some aspects of the invention summarized in Section 5 above. Most relevant parts that have been added or modified are indicated in italics. There may be other changes of an editorial nature, which are therefore not highlighted.

[0172] 5.1. First embodiment

[0173] This embodiment is used for the solutions in items 1 and 2.

[0174] 3.1 Definition

[0175] EDRAP Sample

[0176] A sample in which all subsequent samples in decoding and output order can be correctly decoded, provided that the most recent previous SAP sample of type 1, 2, or 3 and zero or more previous EDRAP samples are available when decoding the sample and subsequent samples

[0177] Item Note 1 The most recent previous SAP sample and zero or more previous EDRAP samples of type 1, 2, or 3 as described above are referred to as the required previous SAP and EDRAP samples of an EDRAP sample. ...

[0179] 10.8 Dependent Random Access Point (DRAP) Sample Group

[0180] 10.8.1 Definitions

[0181] A dependent random access point (DRAP) sample is a sample such that all samples following it in decoding order and output order can be correctly decoded if the most recent previous SAP sample of type 1, 2, or 3 is available for reference.

[0182] NOTE 1 The most recent SAP sample can be a synchronized sample or tagged by a SAP sample group.

[0183] NOTE 2 DRAP samples can be used only in combination with SAP samples of type 1, 2 and 3. This is to enable the functionality of creating a decodable sequence of samples by concatenating the preceding SAP sample and the DRAP sample and the samples following the DRAP sample in decoding order and output order.

[0184] The following applies to any DRAP sample sampleA that is mapped to a DRAP sample group:

[0185] Assume sampleSeq is a sample sequence consisting of the following samples, which are arranged in the order of the following bullet points:

[0186] The most recent previous SAP sample of type 1, 2, or 3,

[0187] sampleA, and

[0188] All samples in the track that come after sampleA in both decoding and output order.

[0189] For each sample sampleB in sampleSeq, all data required to process sampleB shall be accessible in the referenced sample entry, sampleB itself, or in any sample that precedes sampleB in decoding order and is present in sampleSeq.

[0190] NOTE 3 For some video codecs, all the data needed to process sample sampleB includes the parameter set needed to decode sampleB.

[0191] 10.8.2 Syntax.

[0192]

[0193] 10.8.3 Semantics.

[0194] DRAP_type is a non-negative integer. When DRAP_type is in the range of 1 to 3, it indicates the SAP_type (as specified in Appendix I) to which the DRAP sample should have corresponded if the DRAP sample did not depend on the most recent previous SAP. Other type values ​​are reserved.

[0195] reserved SHALL be equal to 0. The semantics of this subclause applies only to sample group description entries with reserved equal to 0. When parsing this sample group, the parser shall allow and ignore sample group description entries with reserved greater than 0.

[0196] 10.11 Extended DRAP (EDRAP) Sample Group

[0197] 10.11.1 Definitions

[0198] An EDRAP sample group records the EDRAP samples in a track. This sample group is similar to the DRAP sample group specified in subclause 10.8; however, it is able to signal additional samples, which can also be used for random access, but with more flexible dependencies. When a track has a SampleToGroupBox with grouping_type equal to 'edrp', each EDRAP sample in the track shall be a member of an EDRAP sample group.

[0199] NOTE 1: Similar to DRAP samples, EDRAP samples can be used in combination with SAP samples of types 1, 2, and 3 only.

[0200] NOTE 2: A DRAP sample is always an EDRAP sample.

[0201] 10.11.2 Syntax

[0202]

[0203] 10.11.3 Semantics

[0204] edrap_type is a non-negative integer. When edrap_type is in the range of 1 to 3, it indicates the SAP_type (as specified in Appendix I) to which the EDRAP sample would have corresponded if the EDRAP sample had not depended on the most recent previous SAP or other EDRAP samples. Other type values ​​are reserved.

[0205] NOTE 1 An EDRAP sample and all subsequent samples in the same track may depend on the nearest previous SAP and / or some previous EDRAP samples without depending on any other samples before the EDRAP sample. Thus, if the encoder chooses to encode an EDRAP sample such that the EDRAP sample does not depend on the nearest previous SAP or any previous EDRAP samples, then the EDRAP sample becomes a SAP.

[0206] num_ref_edrap_samples indicates the number of other EDRAP samples that are earlier than the EDRAP sample in decoding order and that need to be used for reference when decoding starts from the EDRAP sample to be able to correctly decode the EDRAP sample and all samples that follow the EDRAP sample in decoding and output order.

[0207] NOTE 2 An EDRAP sample that is also a DRAP sample will have num_ref_edrap_samples equal to 0.

[0208] reserved SHALL be equal to 0. The semantics of this subclause applies only to sample group description entries with reserved equal to 0. When parsing this sample group, the parser shall allow and ignore sample group description entries with reserved greater than 0.

[0209] ref_edrap_idx_delta[i] indicates the i-th required previous EDRAP sample of the current EDRAP sample. It is assumed that the EDRAP sample list associated with a SAP sample of type 1, 2, or 3 is all EDRAP samples after the SAP sample and before the next SAP sample (if any). The EDRAP sample index is defined as the index of the EDRAP sample list. The value of ref_edrap_idx_delta[i] is equal to the EDRAP sample index of the current EDRAP sample and the EDRAP sample index of the i-th required previous EDRAP sample. A value of 1 indicates that the i-th EDRAP sample is the last EDRAP sample before the EDRAP sample in decoding order, a value of 2 indicates that the i-th EDRAP sample is the second to last EDRAP sample before the EDRAP sample in decoding order, and so on.

[0210] The following applies to any EDRAP sample sampleA that is mapped to the EDRAP sample group:

[0211] Assume sampleSeq is a sample sequence consisting of the following samples, which are arranged in the order of the following bullet points:

[0212] The most recent previous SAP sample of type 1, 2, or 3,

[0213] the EDRAP sample identified by ref_edrap_idx_delta[i] in decoding order, where i is in the range of 0 to num_ref_edrap_samples-1 (inclusive),

[0214] sampleA, and

[0215] All samples in the track that come after sampleA in both decoding and output order.

[0216] For each sample sampleB in sampleSeq, all data required to process sampleB shall be accessible in the referenced sample entry, sampleB itself, or in any sample that precedes sampleB in decoding order and is present in sampleSeq.

[0217] NOTE 3 For some video codecs, all the data needed to process sample sampleB includes the parameter set needed to decode sampleB.

[0218] Fig. 9 4000 . 4002 . 4004 . 4005 . 4006 . 4007 . 4008 . 4009 . 4010 . 4011 . 4012 . 4013 . 4014 . 4015 . 4016 . 4017 . 4018 . 4019 . 4020 . 4021 . 4022 . 4023 . 4024 . 4025 . 4026 . 4027 . 4028 . 4029 . 4030 . 4031 . 4032 . 4033 . 4034 . 4035 . 4036 . 4037 . 4038 . 4039 . 4100 . 4101 . 4102 . 4103 . 4104 . 4105 . 4106 . 4107 . 4108 . 4109 . 4110

[0219] System 4000 may include a codec component 4004, which may implement various codecs or encoding methods described in this document. Codec component 4004 may reduce the average bit rate of the video from input 4002 to the output of codec component 4004 to generate a codec representation of the video. Therefore, codec technology is sometimes referred to as video compression or video code conversion technology. The output of codec component 4004 may be stored, or sent via a connected communication (as represented by component 4006). Component 4008 may use a bitstream (or codec) representation of the storage or communication of the video received at input 4002 to generate pixel values ​​or displayable video sent to display interface 4010. The process of generating user-visible video from a bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "codec" operations or tools, it should be understood that codec tools or operations are used at encoders, and corresponding decoding tools or operations that are opposite to the codec results will be performed by decoders.

[0220] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB) or High Definition Multimedia Interface (HDMI) or Display Port, etc. Examples of storage interfaces include Serial Advanced Technology Attachment (SATA), Peripheral Component Interconnect (PCI), Integrated Drive Electronics (IDE) interface, etc. The techniques described in this document may be embodied in various electronic devices, such as mobile phones, laptop computers, smart phones, or other devices capable of performing digital data processing and / or video display.

[0221] Fig.10 4106 is a block diagram of an exemplary video processing device 4100. Device 4100 can be used to implement one or more of the methods described herein. Device 4100 can be embodied in a smart phone, a tablet computer, a computer, an Internet of Things (IoT) receiver, etc. Device 4100 may include one or more processors 4102, one or more memories 4104, and video processing circuitry 4106. (Multiple) processors 4102 can be configured to implement one or more methods described in this document. (Multiple) memories 4104 can be used to store data and code for implementing the methods and techniques described herein. Video processing circuitry 4106 can be used to implement some of the techniques described in this document in hardware circuits. In some embodiments, video processing circuitry 4106 can be at least partially included in processor 4102, such as a graphics coprocessor.

[0222] Fig.114200 is a flow chart of an exemplary method 4200 for video processing. The method 4200 includes determining, at step 4202, to perform a conversion, such as a conversion between visual media data and a bitstream. In one embodiment, determining, at step 4202, to perform a conversion may include generating a sample sequence (sampleSeq) that includes a most recent previous SAP of type 1, type 2, or type 3, a first DRAP sample (sampleA), and all samples that follow the first DRAP sample in both the decoding order and the output order in a track that includes at least one second DRAP sample (sampleB). Determining, at step 4202, to perform a conversion may also include, for each second DRAP sample in the sample sequence, ensuring that all data for processing the second DRAP sample is accessible in one of a referenced sample entry, in the second DRAP sample itself, or in a sample that precedes the second DRAP sample in the decoding order and is present in the sample sequence.

[0223] In another embodiment, determining in step 4202 to perform a conversion may include generating a sample sequence (sampleSeq) that includes: the most recent previous SAP of type 1, type 2, or type 3; a first EDRAP sample (sampleA) identified by ref_edrap_idx_delta[i] in decoding order, where i is in the range of 0 to num_ref_edrap_samples-1, inclusive; and all samples that follow the first EDRAP sample in both decoding order and output order in a track that includes at least one second EDRAP sample (sampleB). In this embodiment, determining in step 4202 to perform a conversion may also include, for each second EDRAP sample in the sample sequence, ensuring that all data for processing the second EDRAP sample is accessible in one of a referenced sample entry, in the second EDRAP sample itself, or in a sample that precedes the second EDRAP sample in decoding order and is present in the sample sequence.

[0224] Regardless of the particular embodiment implemented by step 4202, the method continues to step 4204, where conversion between the visual media data and a bitstream is performed based on the determination of step 4202. Depending on the example, the conversion of step 4204 may include encoding at an encoder or decoding at a decoder.

[0225] It should be noted that method 4200 may be implemented in an apparatus for processing video data, the apparatus comprising a processor and a non-transitory memory having instructions thereon, such as video encoder 4400, video decoder 4500, and / or encoder 4600. In such cases, the instructions, after being executed by the processor, cause the processor to perform method 4200. In addition, method 4200 may be performed by a non-transitory computer-readable medium including a computer program product for use by a video codec device. The computer program product includes computer-executable instructions stored on a non-transitory computer-readable medium, such that when executed by a processor, the video codec device performs method 4200.

[0226] Fig.12 43 is a block diagram illustrating an exemplary video codec system 4300 that can use the techniques of the present disclosure. The video codec system 4300 may include a source device 4310 and a target device 4320. The source device 4310 generates encoded video data and may be referred to as a video encoding device. The target device 4320 may decode the encoded video data generated by the source device 4310 and may be referred to as a video decoding device.

[0227] Source device 4310 may include video source 4312, video encoder 4314 and input / output (I / O) interface 4316. Video source 4312 may include sources such as video capture devices, interfaces for receiving video data from video content providers and / or computer graphics systems for generating video data, or a combination of such sources. Video data may include one or more pictures. Video encoder 4314 encodes video data from video source 4312 to generate a bitstream. The bitstream may include a bit sequence forming a coded representation of video data. The bitstream may include coded pictures and associated data. The coded pictures are coded representations of pictures. Associated data may include sequence parameter sets, image parameter sets and other syntax structures. I / O interface 4316 may include a modulator / demodulator (modem) and / or a transmitter. The coded video data may be sent directly to target device 4320 via network 4330 via I / O interface 4316. The coded video data may also be stored on storage medium / server 4340 for access by target device 4320.

[0228] Target device 4320 may include an I / O interface 4326, a video decoder 4324, and a display device 4322. I / O interface 4326 may include a receiver and / or a modem. I / O interface 4326 may obtain encoded video data from source device 4310 or storage medium / server 4340. Video decoder 4324 may decode the encoded video data. Display device 4322 may display the decoded video data to a user. Display device 4322 may be integrated with target device 4320, or may be external to target device 4320, and target device 4320 may be configured to interface with an external display device.

[0229] The video encoder 4314 and the video decoder 4324 may operate according to a video compression standard, such as the High Efficiency Video Codec (HEVC) standard, the Versatile Video Codec (VVC) standard, and other current and / or further standards.

[0230] Fig.13 is a block diagram showing an example of a video encoder 4400, which may be Fig.12 Video encoder 4314 in system 4300 shown. Video encoder 4400 can be configured to perform any or all of the techniques of the present disclosure. Video encoder 4400 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video encoder 4400. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0231] The functional components of the video encoder 4400 may include a segmentation unit 4401; a prediction unit 4402, which may include a mode selection unit 4403, a motion estimation unit 4404, a motion compensation unit 4405 and an intra-frame prediction unit 4406; a residual generation unit 4407; a transform processing unit 4408; a quantization unit 4409; an inverse quantization unit 4410; an inverse transform unit 4411; a reconstruction unit 4412; a buffer 4413; and an entropy coding unit 4414.

[0232] In other examples, the video encoder 4400 may include more, fewer, or different functional components. In an example, the prediction unit 4402 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode in which at least one reference picture is a picture in which the current video block is located.

[0233] Furthermore, some components, such as the motion estimation unit 4404 and the motion compensation unit 4405 , may be highly integrated but are represented separately in the example of the video encoder 4400 for purposes of explanation.

[0234] The segmentation unit 4401 may segment the picture into one or more video blocks. The video encoder 4400 and the video decoder 4500 may support various video block sizes.

[0235] The mode selection unit 4403 may select one of the coding modes (intra or inter) based on the error result, for example, and provide the resulting intra or inter coded block to the residual generation unit 4407 to generate residual block data, and to the reconstruction unit 4412 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 4403 may select a combined intra and inter prediction (CIIP) mode, where the prediction is based on an inter prediction signal and an intra prediction signal. In the case of inter prediction, the mode selection unit 4403 may also select a resolution of the motion vector for the block (e.g., sub-pixel or integer pixel precision).

[0236] To perform inter-frame prediction on the current video block, the motion estimation unit 4404 may generate motion information of the current video block by comparing the current video block with one or more reference frames from the buffer 4413. The motion compensation unit 4405 may determine a predicted video block of the current video block based on the motion information of pictures other than the picture associated with the current video block from the buffer 4413 and decoded samples.

[0237] The motion estimation unit 4404 and the motion compensation unit 4405 may perform different operations on the current video block, for example, depending on whether the current video block is in an I slice, a P slice, or a B slice.

[0238] In some examples, the motion estimation unit 4404 may perform unidirectional prediction on the current video block, and the motion estimation unit 4404 may search for a reference video block of the current video block in the reference picture of list 0 or list 1. The motion estimation unit 4404 may then generate a reference index and a motion vector, the reference index indicating the reference picture in list 0 or list 1 including the reference video block, and the motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 4404 may output the reference index, the prediction direction indicator, and the motion vector as the motion information of the current video block. The motion compensation unit 4405 may generate a predicted video block of the current block based on the reference video block indicated by the motion information of the current video block.

[0239] In other examples, the motion estimation unit 4404 may perform bidirectional prediction on the current video block, and the motion estimation unit 4404 may search for a reference video block of the current video block in the reference pictures in list 0, and may also search for another reference video block of the current video block in the reference pictures in list 1. The motion estimation unit 4404 may then generate a reference index and a motion vector, the reference index indicating the reference pictures in list 0 and list 1 including the reference video block, and the motion vector indicating the spatial displacement between the reference video block and the current video block. The motion estimation unit 4404 may output the reference index and the motion vector of the current video block as the motion information of the current video block. The motion compensation unit 4405 may generate a predicted video block of the current video block based on the reference video block indicated by the motion information of the current video block.

[0240] In some examples, motion estimation unit 4404 may output a full set of motion information for a decoding process of a decoder. In some examples, motion estimation unit 4404 may not output a full set of motion information for the current video. Instead, motion estimation unit 4404 may signal the motion information of the current video block with reference to the motion information of another video block. For example, motion estimation unit 4404 may determine that the motion information of the current video block is sufficiently similar to the motion information of a neighboring video block.

[0241] In one example, the motion estimation unit 4404 may indicate a value in a syntax structure associated with the current video block that indicates to the video decoder 4500 that the current video block has the same motion information as another video block.

[0242] In another example, the motion estimation unit 4404 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 4500 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0243] As described above, the video encoder 4400 can predictively signal motion vectors. Two examples of predictive signaling techniques that can be implemented by the video encoder 4400 include advanced motion vector prediction (AMVP) and merge mode signaling.

[0244] The intra prediction unit 4406 may perform intra prediction on the current video block. When the intra prediction unit 4406 performs intra prediction on the current video block, the intra prediction unit 4406 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a predicted video block and various syntax elements.

[0245] The residual generation unit 4407 may generate residual data of the current video block by subtracting the predicted video block of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample components of samples in the current video block.

[0246] In other examples, the current video block may not have residual data for the current video block, such as in skip mode, and the residual generation unit 4407 may not perform a subtraction operation.

[0247] The transform processing unit 4408 may generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0248] After the transform processing unit 4408 generates a transform coefficient video block associated with the current video block, the quantization unit 4409 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0249] The inverse quantization unit 4410 and the inverse transform unit 4411 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct a residual video block from the transform coefficient video block. The reconstruction unit 4412 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by the prediction unit 4402 to generate a reconstructed video block associated with the current block for storage in the buffer 4413.

[0250] After the reconstruction unit 4412 reconstructs the video block, a loop filtering operation may be performed to reduce video block artifacts in the video block.

[0251] The entropy coding unit 4414 may receive data from other functional components of the video encoder 4400. When the entropy coding unit 4414 receives the data, the entropy coding unit 4414 may perform one or more entropy coding operations to generate entropy coded data, and output a bitstream including the entropy coded data.

[0252] Fig.14 is a block diagram showing an example of a video decoder 4500, which may be Fig.12 Video decoder 4324 in the system 4300 shown. Video decoder 4500 can be configured to perform any or all of the techniques of the present disclosure. In the example shown, video decoder 4500 includes multiple functional components. The techniques described in this disclosure can be shared between various components of video decoder 4500. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0253] In the illustrated example, the video decoder 4500 includes an entropy decoding unit 4501, a motion compensation unit 4502, an intra prediction unit 4503, an inverse quantization unit 4504, an inverse transform unit 4505, a reconstruction unit 4506, and a buffer 4507. In some examples, the video decoder 4500 may perform a decoding process that is generally reciprocal to the encoding process described with reference to the video encoder 4400.

[0254] The entropy decoding unit 4501 may retrieve an encoded bitstream. The encoded bitstream may include entropy-encoded video data (e.g., encoded video data blocks). The entropy decoding unit 4501 may decode the entropy-encoded video data, and based on the entropy-decoded video data, the motion compensation unit 4502 may determine motion information including motion vectors, motion vector precision, reference picture list index, and other motion information. For example, the motion compensation unit 4502 may determine such information by performing AMVP and merge modes.

[0255] The motion compensation unit 4502 may generate a motion compensated block, possibly performing interpolation based on an interpolation filter. An identifier of an interpolation filter to be used with sub-pixel precision may be included in the syntax element.

[0256] The motion compensation unit 4502 may calculate interpolated values ​​of sub-integer pixels of the reference block using interpolation filters as used by the video encoder 4400 during encoding of the video block. The motion compensation unit 4502 may determine the interpolation filters used by the video encoder 4400 based on received syntax information and use the interpolation filters to generate a prediction block.

[0257] The motion compensation unit 4502 can use some syntax information to determine the size of the blocks used to encode (multiple) frames and / or (multiple) slices of the encoded video sequence, partitioning information describing how each macroblock of the pictures of the encoded video sequence is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-frame codec block, and other information used to decode the encoded video sequence.

[0258] The intra prediction unit 4503 may form a prediction block from spatially neighboring blocks using, for example, an intra prediction mode received in the bitstream. The inverse quantization unit 4504 inversely quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 4501. The inverse transform unit 4505 applies an inverse transform.

[0259] The reconstruction unit 4506 may add the residual block to the corresponding prediction block generated by the motion compensation unit 4502 or the intra prediction unit 4503 to form a decoded block. If necessary, a deblocking filter may also be applied to filter the decoded block to remove blocking artifacts. The decoded video block is then stored in a buffer 4507, which provides reference blocks for subsequent motion compensation / intra prediction and also generates a decoded video for presentation on a display device.

[0260] Fig.15 4600 is a schematic diagram of an exemplary encoder 4600. Encoder 4600 is suitable for implementing techniques for VVC. Encoder 4600 includes three loop filters, namely, a deblocking filter (DF) 4602, a sample adaptive offset (SAO) 4604, and an adaptive loop filter (ALF) 4606. Unlike DF 4602, which uses a predefined filter, SAO 4604 and ALF 4606 use the original samples of the current picture to reduce the mean square error between the original samples and the reconstructed samples by adding an offset and applying a finite impulse response (FIR) filter, respectively, where information on the codec side signals the offset and filter coefficients. ALF 4606 is located at the last processing stage of each picture and can be seen as a tool that attempts to capture and repair artifacts generated by previous stages.

[0261] The encoder 4600 also includes an intra prediction component 4608 and a motion estimation / compensation (ME / MC) component 4610 configured to receive an input video. The intra prediction component 4608 is configured to perform intra prediction, while the ME / MC component 4610 is configured to perform inter prediction using a reference picture obtained from a reference picture buffer 4612. The residual block from the inter prediction or intra prediction is fed to a transform (T) component 4614 and a quantization (Q) component 4616 to generate quantized residual transform coefficients, which are fed to an entropy coding component 4618. The entropy coding component 4618 entropy codes and decodes the prediction result and the quantized transform coefficients and sends them to a video decoder (not shown). The quantization component output from the quantization component 4616 can be fed to an inverse quantization (IQ) component 4620, an inverse transform component 4622, and a reconstruction (REC) component 4624. The REC component 4624 is capable of outputting images to the DF 4602, SAO 4604, and ALF 4606 for filtering before the images are stored in the reference picture buffer 4612.

[0262] A list of some example preferred solutions is provided next.

[0263] The following solutions show examples of the techniques discussed in this article.

[0264] 1. A method for processing video data, comprising: generating a sample sequence (sampleSeq), the sample sequence comprising a most recent previous stream access point (SAP) of type 1, type 2 or type 3, a first dependent random access point (DRAP) sample (sampleA), and all samples that follow the first DRAP sample in both decoding order and output order in a track including at least one second DRAP sample (sampleB); for each second DRAP sample in the sample sequence, ensuring that all data for processing the second DRAP sample is accessible; and performing conversion between a video comprising the video data and a bitstream of the video data based on the data.

[0265] 2. The method of claim 1, wherein the first DRAP sample is mapped to a DRAP sample group.

[0266] 3. The method according to any one of claims 1-2, wherein all data used to process the second DRAP sample is accessible in the reference sample entry.

[0267] 4. The method according to any one of claims 1-2, wherein all data used to process the second DRAP sample is accessible in the second DRAP sample itself.

[0268] 5. The method of any of claims 1-2, wherein all data for processing the second DRAP sample is accessible in one of the samples that precedes the second DRAP sample in decoding order and is present in the sample sequence.

[0269] 6. The method of any one of claims 1-5, wherein all data used to process the second DRAP sample includes a parameter set for decoding the second DRAP sample.

[0270] 7. A method for processing video data, comprising: generating a sample sequence (sampleSeq), the sample sequence comprising the nearest previous stream access point (SAP) of type 1, type 2 or type 3, a first extended dependent random access point (EDRAP) sample (sampleA) identified by ref_edrap_idx_delta[i] in decoding order, where i is in the range of 0 to num_ref_edrap_samples-1, including 0 and num_ref_edrap_samples-1, and all samples that follow the first EDRAP sample in both decoding order and output order in a track including at least one second EDRAP sample (sampleB); for each second EDRAP sample in the sample sequence, ensuring that all data for processing the second EDRAP sample is accessible; and performing conversion between a video comprising the video data and a bitstream of the video data based on the data.

[0271] The method of claim 7 , wherein the first EDRAP sample is mapped to an EDRAP sample group.

[0272] 9. The method according to any of claims 7-8, wherein all data used to process the second EDRAP sample is accessible in a reference sample entry.

[0273] 10. The method according to any of claims 7-8, wherein all data used for processing the second EDRAP sample is accessible in the second EDRAP sample itself.

[0274] 11. The method of any of claims 7-8, wherein all data for processing the second EDRAP sample is accessible in one of the samples that precedes the second EDRAP sample in decoding order and is present in the sample sequence.

[0275] 12. The method of any one of claims 7-11, wherein all data used to process the second EDRAP sample comprises a parameter set for decoding the second EDRAP sample.

[0276] 13. An apparatus for processing video data comprising: a processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1-12.

[0277] 14. A non-transitory computer-readable medium, comprising a computer program product for use by a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, so that when executed by a processor, the video codec device performs the method according to any one of claims 1-12.

[0278] 15. A non-transitory computer-readable recording medium storing a bit stream of a video generated by a method performed by a video processing device, wherein the method includes the method according to any one of claims 1-12.

[0279] 16. A method for storing a bitstream of a video comprising the method according to any one of claims 1-12, wherein performing the conversion between a video comprising the video data and the bitstream of the video data comprises generating the bitstream, and wherein the method further comprises storing the bitstream in a non-transitory computer-readable recording medium.

[0280] 17. A method, apparatus or system is described in this document.

[0281] In the solution described herein, an encoder can comply with the format rules by generating a codec representation according to the format rules. In the solution described herein, a decoder can parse syntax elements in the codec representation using the format rules, understand the presence and absence of syntax elements according to the format rules, and generate decoded video.

[0282] In this document, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, during the conversion from a pixel representation of a video to a corresponding bitstream representation, a video compression algorithm may be applied, or vice versa. As defined by the syntax, the bitstream representation of the current video block may correspond, for example, to bits that are co-located or dispersed at different locations within the bitstream. For example, a macroblock may be encoded based on a transformed and encoded error residual value, and also using bits in a header and other fields in the bitstream. In addition, during the conversion, the decoder may parse the bitstream based on this determination, knowing that some fields may or may not be present, as described in the above solution. Similarly, the encoder may determine whether to include or not include certain syntax fields, and generate the codec representation accordingly by including or excluding the syntax fields in the codec representation.

[0283] The disclosed and other solutions, examples, embodiments, modules and functional operations described in this document may be implemented in digital electronic circuits, or in computer software, firmware or hardware, including the structures disclosed in this document and their structural equivalents, or in a combination of one or more of them. The disclosed and other embodiments may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium, for execution by a data processing device or for controlling the operation of the data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material combination that implements a machine-readable propagation signal, or a combination of one or more of them. The term "data processing device" encompasses all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, a device may include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagation signal is an artificially generated signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0284] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing portions of one or more modules, subroutines, or code). A computer program may be deployed to execute on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0285] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, and the apparatus can also be implemented as special purpose logic circuits, such as field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs).

[0286] Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or be coupled in an operational manner to receive data from one or more mass storage devices or to transmit data to one or more mass storage devices or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and compact disc read-only memory (CD ROM) and digital versatile disc read-only memory (DVD-ROM) disks. The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.

[0287] Although the present disclosure contains many details, these should not be understood as limitations on any subject matter or the scope that may be claimed, but rather as descriptions of features peculiar to specific embodiments of specific technologies. Certain features described in the context of separate embodiments of the present disclosure may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may involve a sub-combination or a variant of a sub-combination.

[0288] Similarly, although operations are depicted in a particular order in the figures, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, in order to achieve the desired results. Furthermore, the separation of various system components in the embodiments described in this disclosure should not be understood as requiring such separation in all embodiments.

[0289] Only a few implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and illustrated in this disclosure.

[0290] A first component is directly coupled to a second component when there are no intermediate components other than a line, trace, or another medium between the first and second components. A first component is indirectly coupled to a second component when there are intermediate components other than a line, trace, or another medium between the first and second components. The term "coupled" and variations thereof include both direct and indirect couplings. Unless otherwise indicated, use of the term "about" is intended to include a range of ±10% of the subsequent numerical value.

[0291] Although several embodiments have been provided in the present disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are considered to be illustrative rather than restrictive, and are not intended to be limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0292] In addition, without departing from the scope of the present disclosure, the techniques, systems, subsystems, and methods described and shown as discrete or independent in various embodiments may be combined or integrated with other systems, modules, techniques, or methods. Other items shown or discussed as coupled may be directly connected, or may be indirectly coupled or communicated electrically, mechanically, or otherwise through some interface, device, or intermediate component. Other examples of changes, substitutions, and alterations may be determined by those skilled in the art, and these changes, substitutions, and alterations may be made without departing from the spirit and scope disclosed herein.

[0293] The solutions listed in this disclosure may be used to compress an image, compress a video, compress a portion of an image, or compress a portion of a video.

[0294] In addition, the techniques, systems, subsystems, and methods described and illustrated in the various embodiments may be used to compress an image, compress a video, compress a portion of an image, or compress a portion of a video.

Claims

1. A method for processing video data, comprising: generating a sample sequence (sampleSeq) comprising a most recent previous stream access point (SAP) of type 1, type 2, or type 3, a first dependent random access point (DRAP) sample (sampleA), and all samples following the first DRAP sample in both decoding order and output order in a track including at least one second DRAP sample (sampleB); For each second DRAP sample in the sequence of samples, ensuring that all data used to process the second DRAP sample is accessible; and Based on the data, conversion between a video including the video data and a bitstream of the video data is performed.

2. The method according to claim 1, wherein: The first DRAP sample is mapped to a DRAP sample group.

3. The method according to any one of claims 1 to 2, wherein: All data used to process the second DRAP sample is accessible in the reference sample entry.

4. The method according to any one of claims 1 to 2, wherein: All data used to process the second DRAP sample is accessible in the second DRAP sample itself.

5. The method according to any one of claims 1 to 2, wherein: All data for processing the second DRAP sample is accessible in one of the samples that precedes the second DRAP sample in the decoding order and that is present in the sequence of samples.

6. The method according to any one of claims 1 to 5, wherein: All data for processing the second DRAP sample includes a parameter set for decoding the second DRAP sample.

7. A method for processing video data, comprising: Generate a sample sequence (sampleSeq), the sample sequence comprising: the most recent previous stream access point (SAP) of type 1, type 2, or type 3; a first extended dependent random access point (EDRAP) sample (sampleA) identified by ref_edrap_idx_delta[i] in decoding order, where i is in the range of 0 to num_ref_edrap_samples-1, inclusive; and all samples that follow the first EDRAP sample in both the decoding order and the output order in a track including at least one second EDRAP sample (sampleB); For each second EDRAP sample in the sequence of samples, ensuring that all data used to process the second EDRAP sample is accessible; and Based on the data, conversion between a video including the video data and a bitstream of the video data is performed.

8. The method according to claim 7, wherein: The first EDRAP sample is mapped to an EDRAP sample group.

9. The method according to any one of claims 7 to 8, wherein: All data used to process the second EDRAP sample are accessible in the reference sample entry.

10. The method according to any one of claims 7 to 8, wherein: All data used to process the second EDRAP sample is accessible in the second EDRAP sample itself.

11. The method according to any one of claims 7 to 8, wherein: All data for processing the second EDRAP sample is accessible in one of the samples that precedes the second EDRAP sample in the decoding order and that is present in the sequence of samples.

12. The method according to any one of claims 7 to 11, wherein: All data for processing the second EDRAP sample includes a parameter set for decoding the second EDRAP sample.

13. An apparatus for processing video data, comprising: processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1-12.

14. A non-transitory computer-readable medium, comprising a computer program product for use by a video codec device, the computer program product comprising computer executable instructions stored on the non-transitory computer-readable medium, so that when executed by a processor, the video codec device performs the method according to any one of claims 1-12.

15. A non-transitory computer-readable recording medium storing a bit stream of a video generated by a video processing device executing a method, wherein: The method comprises the method according to any one of claims 1-12.

16. A method for storing a bit stream of a video comprising the method of any one of claims 1 to 12, wherein: Performing the conversion between the video including the video data and the bitstream of the video data includes generating the bitstream, and wherein the method further includes storing the bitstream in a non-transitory computer-readable recording medium.

Images