Decoder configuration information for VVC video coding

Abstract

A mechanism for processing video data is disclosed, which converts between visual media data and a visual media data file, the visual media data file including a Generic Video Coding (VVC) decoder configuration record and a plurality of pictures coded into one or more sub-layers, the VVC decoder configuration record including a number of one or more sub-layers and a VVC Profile Tier Level (PTL) record for the one or more sub-layers based on the number of one or more sub-layers.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on International Patent Application PCT / CN2021 / 126297, filed October 26, 2021, and claims priority to and the benefit of International Patent Application PCT / CN2020 / 123540, filed October 26, 2020. The aforementioned patent applications are incorporated herein by reference in their entireties.

[0002] This patent specification relates to the creation, storage and consumption of digital audiovisual media information in file formats. [Background technology]

[0003] Digital video accounts for the largest bandwidth usage on the Internet and other digital communications networks, and the bandwidth demands for digital video use are expected to continue to grow as the number of connected user devices capable of receiving and displaying video increases. Summary of the Invention

[0004] A first aspect relates to a video data processing method comprising: converting between a visual media data file and a Generic Video Coding (VVC) decoder configuration record, the visual media data including a plurality of pictures coded into one or more sub-layers; the VVC decoder configuration record including a number of one or more sub-layers and a VVC Profile Tier Level (PTL) record for the sub-layer based on the number of the one or more sub-layers.

[0005] Optionally, in any of the aforementioned aspects, another embodiment specifies that the converting comprises receiving a media file including a generic video coding (VVC) decoder configuration record and a plurality of pictures coded into one or more sub-layers, parsing the VVC decoder configuration record to obtain a number of one or more sub-layers and a VVC PTL record for the one or more sub-layers based on the number of the one or more sub-layers, and decoding the one or more sub-layers based on the VVC PTL record.

[0006] Optionally, in any of the aforementioned aspects, another embodiment specifies that the conversion comprises encoding the plurality of pictures into one or more sub-layers of the visual media file, determining a number of the one or more sub-layers, and encoding a VVC decoder configuration record into the media file, the VVC decoder configuration consisting of the number of the one or more sub-layers and a VVC PTL record for the one or more sub-layers, and storing in the memory of the visual file.

[0007] Optionally, in any of the foregoing aspects, another embodiment provides that the number of one or more sub-layers is signaled in a VVC decoder configuration record before the VVC PTL record.

[0008] Optionally, in any of the aforementioned aspects, another embodiment specifies that the VVC decoder configuration record comprises a constant frame rate syntax element, a chroma format identification code syntax element, and a bit depth minus 8 syntax element, and the VVC PTL record is positioned in the VVC decoder configuration record after the constant frame rate syntax element, the chroma format identification code syntax element, and the bit depth minus 8 syntax element.

[0009] Optionally, in any of the aforementioned aspects, another embodiment specifies that the number of one or more sub-layers is positioned in the VVC decoder configuration record before a constant frame rate syntax element, the chroma format identification code syntax element, and the bit depth minus 8 syntax element.

[0010] Optionally, in any of the aforementioned aspects, another embodiment specifies that the VVC decoder configuration record further includes reserved bits positioned after the bit depth minus 8 syntax elements, and the VVC PTL record is disposed after the reserved bits.

[0011] Optionally, in any of the foregoing aspects, another embodiment provides that the VVC decoder configuration record further includes a reserved bit located after the VVC PTL record.

[0012] Optionally, in any of the aforementioned aspects, another embodiment specifies that the VVC decoder configuration record includes a maximum required size of the decoded picture buffer, a maximum picture output reordering, a maximum delay, a Gradual Decoding Refresh (GDR) picture enable flag, a Clean Random Access (CRA) picture enable flag, a reference image resample enable flag, a spatial resolution change using coded layer video sequence (CLVS) enable flag, a sub-picture partitioning enable flag, a maximum number of sub-pictures per picture, a wavefront parallel processing (WPP) enable flag, a tile partitioning enable flag, a maximum number of slices per picture, a rectangular slice enable flag, a raster scan slice enable flag, a maximum number of slices per picture, or a combination thereof.

[0013] Optionally, in any of the aforementioned aspects, another embodiment specifies that the VVC decoder configuration record further includes one or more of: a maximum required size of the decoded picture buffer, a maximum picture output order change, a maximum latency, a GDR picture enable flag, a CRA picture enable flag, a reference image resampling enable flag, a spatial resolution change due to CLVS enable flag, a sub-picture partitioning enable flag, a maximum number of sub-pictures for each picture, a WPP enable flag, a tile partitioning enable flag, a maximum number of tiles per picture, a slice partitioning enable flag, a number of rectangular slices, a raster scan slice enable flag, and a maximum number of slices per picture if the VVC decoder configuration record constitutes a VVC PTL record.

[0014] Optionally, in any of the foregoing aspects, another embodiment specifies that byte alignment is required for all syntax elements located after a VVC PTL record in a VVC decoder configuration record.

[0015] A second aspect relates to a video data processing apparatus comprising a processor and a non-transitory memory comprising instructions, which upon execution of the instructions by the processor cause the processor to perform a conversion between visual media data including a Generic Video Coding (VVC) decoder configuration record and a plurality of pictures coded into one or more sub-layers and a visual media data file, the VVC decoder configuration record including a number of the one or more sub-layers and one or more VVC Profile Tier Level (PTL) records for the one or more sub-layers based on the number of the one or more sub-layers.

[0016] Optionally, in any of the aforementioned aspects, another embodiment specifies that the conversion comprises receiving a media file including a VVC decoder configuration record and a plurality of pictures coded into one or more sub-layers, parsing the VVC decoder configuration record to obtain a number of the one or more sub-layers and a VVC PTL record for the one or more sub-layers based on the number of the one or more sub-layers, and decoding the one or more sub-layers based on the VVC PTL record.

[0017] Optionally, in any of the aforementioned aspects, another embodiment specifies that the conversion comprises encoding the plurality of pictures into one or more sub-layers of a visual media file, determining a number of the one or more sub-layers, encoding a VVC decoder configuration record into the media file, the VVC decoder configuration consisting of the number of the one or more sub-layers and a VVC PTL record for the one or more sub-layers, and saving the visual media file in memory.

[0018] Optionally, in any of the foregoing aspects, another embodiment provides that the number of one or more sub-layers is signaled in a VVC decoder configuration record before the VVC PTL record.

[0019] Optionally, in any of the aforementioned aspects, another embodiment specifies that the VVC decoder configuration record further comprises a constant frame rate syntax element, a chroma format identification code syntax element, and a bit depth minus 8 syntax element, and that the VVC PTL record is positioned in the VVC decoder configuration record after the constant frame rate syntax element, the chroma format identification code syntax element, and the bit depth minus 8 syntax element.

[0020] Optionally, in any of the aforementioned aspects, another embodiment specifies that the number of one or more sub-layers is positioned in the VVC decoder configuration record before a constant frame rate syntax element, a chroma format identification code syntax element, and a bit depth minus 8 syntax element.

[0021] Optionally, in any of the aforementioned aspects, another embodiment specifies that the VVC decoder configuration record further includes reserved bits located after the bit depth minus 8 syntax element, and the VVC PTL record is located after the reserved bits.

[0022] Optionally, in any of the foregoing aspects, another embodiment provides that the VVC decoder configuration record further includes a reserved bit located after the VVC PTL record.

[0023] Optionally, in any of the aforementioned aspects, another embodiment specifies that the VVC decoder configuration record further includes a maximum required size of the decoded picture buffer, a maximum picture output order change, a maximum delay, a GDR picture enable flag, a CRA picture enable flag, a reference image resample enable flag, a CLVS enable flag, a sub-picture split enable flag, a maximum number of sub-pictures per picture, a WPP enable flag, a tile split enable flag, a maximum number of slices per picture, a slice split enable flag, a rectangular slice enable flag, a raster scan slice enable flag, or a combination thereof.

[0024] Optionally, in any of the aforementioned aspects, another embodiment specifies that the VVC decoder configuration record further includes one or more of: a maximum required size of the decoded picture buffer, a maximum picture output reordering, a maximum latency, a GDR picture enable flag, a CRA picture enable flag, a reference image resampling enable flag, a spatial resolution change due to CLVS enable flag, a sub-picture partitioning enable flag, a maximum number of sub-pictures for each picture, a WPP enable flag, a tile partitioning enable flag, a maximum number of tiles per picture, a slice partitioning enable flag, a number of rectangular slices, a raster scan slice enable flag, and, if the VVC decoder configuration record consists of a VVC PTL record, a number of slices per picture.

[0025] Optionally, in any of the foregoing aspects, another embodiment provides that byte alignment is required for all syntax elements located after a VVC PTL record in a VVC decoder configuration record.

[0026] A third aspect relates to a non-transitory computer-readable medium including a computer program product for use by a video coding device, the computer program product including computer-executable instructions stored on the non-transitory computer-readable medium that, when executed by a processor, cause the video coding device to perform a method of any of the preceding aspects.

[0027] For clarity, any one of the above-described embodiments may be combined with any one or more of the other above-described embodiments to create new embodiments within the scope of the present disclosure.

[0028] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.

[0029] For a complete understanding of the present disclosure, reference should be made to the following brief description taken in conjunction with the accompanying drawings and detailed description, where like reference numerals represent like parts. [Brief description of the drawings]

[0030] [Figure 1] FIG. 1 is a schematic diagram of an example media file that includes a Versatile Video Coding (VVC) bitstream of video data. [Diagram 2] FIG. 2 is a flow chart illustrating an example method for encoding a VVC decoder configuration record. [Diagram 3] FIG. 3 is a flow chart illustrating an example of a method for decoding a VVC decoder configuration record. [Figure 4] FIG. 4 is a block diagram showing an example of a video processing system. [Diagram 5] FIG. 5 is a block diagram illustrating an example of a video processing device. [Figure 6] FIG. 6 is a flow chart illustrating an exemplary method of video processing. [Figure 7]FIG. 7 is a block diagram illustrating an example video coding system. [Figure 8] FIG. 8 is a block diagram illustrating an example of an encoder. [Figure 9] FIG. 9 is a block diagram illustrating an example of a decoder. [Figure 10] FIG. 10 is a schematic diagram illustrating an example of an encoder. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] Although exemplary implementations of one or more embodiments are provided below, it should be understood at the outset that the disclosed systems and / or methods may be implemented using any number of technologies, whether now known or later developed. The present disclosure should in no way be limited to the exemplary implementations, drawings, and technologies described herein, but may be amended within the scope of the appended claims, along with their full scope of equivalents.

[0032] The term VVC (also known as H.266) is used in some descriptions for ease of understanding and is not intended to limit the scope of the disclosed technology. As such, the technology described herein is applicable to other video codec protocols and designs. In this specification, editorial changes are indicated by strikethrough to indicate struck out text and italics to indicate added text with respect to the current draft of the VVC specification or the International Organization for Standardization (ISO) Base Media File Format (ISOBMFF) file format.

[0033] Examples of implementations of the above-mentioned aspects are described below.

[0034] This specification relates to video file formats. In particular, this document relates to signaling decoder configuration information and "role" sample groups in media files carrying Versatile Video Coding (VVC) video bitstreams based on the ISO Base Media File Format (ISOBMFF). The ideas can be applied individually or in various combinations to video bitstreams coded by any codec, e.g., the VVC standard, and any video file format, e.g., the VVC video file format under development.

[0035] Adaptive Color Transformation (ACT), Adaptive Loop Filter (ALF), Adaptive Motion Vector Resolution (AMVR), Adaptive Parameter Set (APS), Access Unit (AU), Access Unit Delimiter (AUD), Advanced Video Coding (Rec. ITU-T H.264 | ISO / IEC 14496-10) (AVC), Bi-Prediction (B), Bi-Prediction with CU Level Weights (BCW), Bidirectional Optical Flow (BDOF), Block-Based Delta Pulse Code Modulation (BDPCM), Buffering Period (BP), Bi-Directional Optical Coding (CABAC), Coding Block (CB), Constant Bitrate (CBR), Cross Component Adaptive Loop Filter (CCALF), Coded Layered Video Stream (CLVS), Coded Picture Buffer (CPB), Clean Random Access (CRA), Cyclic Redundancy Check (CRC), Coding Tree Block (CTB), Coding Tree Unicast (CTB), Coding Unit (CTU), Coding Unit (CU), Coded Picture Sequence (CVS), Decoded Picture Buffer (DPB), Decoding Capability Information (DCI), Dependent Random Access Point (DRAP), Decoding Unit (DU), Decoding Unit Information (DUI), Exponential Golon (EG), kth Exponential Golon (EGk), End of Bitstream (EOB), End of Sequence (EOS), Filler Data (FD), First-In-First-Out (FIFO), Fixed Length (FL), Green-Blue-Red (GBR), General Constraint Information (GCI), Gradual Decoding Refresh (GDR), Geometry Partitioning Mode (GPM), Rec.ITU-T H.265|High Efficiency Video Coding (HEVC), also known as ISO / IEC 23008-2, Virtual Reference Decoder (HRD), Virtual Stream Scheduler (HSS), Intra (I), Intra Block Copy (IBC), Instantaneous Decoding Update (IDR), Inter-layer Reference Picture (ILRP), Intra Random Access Point (IRAP), Low Frequency Nonlinear Transform (LFNST), Least Probable Symbol (LPS), Least Significant Bit (LSB), Long Term Reference Picture (LTRP), Luminance Mapping with Chroma Scaling (LMCS), Matrix-Based Intra Prediction (MIP), Maximum Probable Symbol (MPS), Most Significant Bit (MSB), Multiple Transform Selection (MTS), Motion Vector Prediction (MVP), Network Abstraction Layer (NAL), Output Layer Set (OLS), Operation Point (OP), Operation Point Information (OPI), Prediction (P), Picture Header (PH), Picture Order Count (POC), Picture Parameter Set (PPS), Prediction Refinement by Optical Flow (PROF), Picture Timing (PT), Picture Unit (PU), Quantization Parameter (QP), Random Access Decodable Leading Picture (RADL), Random Access Skip-Type Leading Picture (RASL), Raw Byte Sequence Payload (RBSP), Red Green Blue (RGB), Reference Picture List (RPL), Sample Adapt Offset (SAO), Sample Assessment Ratio (SAR), Supplemental Enhancement Information (SEI), Slice Header (SH), Sub-Picture Level Information (SLI), Data Bitstream (SODB), Sequence Parameter Set (SPS), Short-Term Reference Picture (STRP), Step-Wise Time Sublayer Access (STSA), Truncation Rice (TR), Variable Bitrate (VBR), Video Coding Layer (VCL), Video Parameter Set (VPS), Versatile Supplemental Enhancement Information (VSEI), also known as Rec. ITU-T H.274|ISO / IEC 23002-7, Video Usability Information (VUI), Generic Video Coding (VVC), also known as Rec. ITU-T H.266|ISO / IEC 23090-3, Wavefront Parallel Processing (WPP).

[0036] Video coding standards have evolved mainly through standard development by ITU-T and ISO / IEC. ITU-T has H.261 and H.263, ISO / IEC has MPEG-1 and MPEG-4 Visual, and together they have H.262 / MPEG-2 Video, H.264 / MPEG-4 Advanced Video Coding (AVC), and H.265 / HEVC. Since H.262, video coding standards have been based on hybrid video coding structures that utilize temporal prediction and transform coding. To explore video coding technologies beyond HEVC, the Video Coding Experts Group (VCEG) and MPEG jointly established the Joint Video Exploration Team (JVET). Many methods were adopted by JVET and summarized in a reference software called the Joint Exploration Model (JEM). Later, when the Generalized Video Coding (VVC) project was officially launched, JVET was renamed the Joint Video Experts Team (JVET). VVC is a coding standard that aims to reduce the bit rate by 50% compared to HEVC. VVC was finalized by JVET.

[0037] The Generic Video Coding (VVC) standard (ITU-T H.266|ISO / IEC 23090-3) and the associated Generic Supplementary Enhancement Information (VSEI) standard (ITU-T H.274|ISO / IEC 23002-7) have been designed for use in a wide range of applications, from television broadcast, videoconferencing or playback from storage media, to advanced uses such as adaptive bitrate streaming, video region extraction, compositing, combining content from multiple coded video bitstreams, multi-view video, scalable layer coding, and viewport-adaptive 360° immersive media.

[0038] Media streaming applications are generally based on Internet Protocol (IP), Transmission Control Protocol (TCP), and Hypertext Transfer Protocol (HTTP) transport methods and generally rely on file formats such as ISO Base Media File Format (ISOBMFF). One such streaming system is dynamic adaptive streaming over HTTP (DASH). When video formats are used in ISOBMFF and DASH, video format specific file format specifications such as AVC file format and HEVC file format will be used for encapsulation of video content in ISOBMFF tracks and DASH representations and segments. Information about the video bitstream, such as profile, tier, and level, as well as many other pieces of information, will be exposed as file format level metadata and / or DASH Media Presentation Description (MPD) for content selection purposes, such as for initialization at the beginning of a streaming session and for selection of appropriate media segments for stream adaptation during a streaming session.

[0039] Similarly, when an image format is used with ISOBMFF, it will adopt a file format specification specific to the image format, such as the AVC image file format or the HEVC image file format. A file format for storage of VVC video content based on ISOBMFF, the VVC video file format, is under development by MPEG. A file format for storage of image content coded using VVC, based on ISOBMFF, the VVC image file format, is under development by MPEG.

[0040] The following describes the design of the VVC image file format and some VVC file format features based on MPEG. This subclause specifies decoder configuration information for ISO / IEC 23090-3 video content. This record, if stored in a sample entry, contains the size of the parameter set, DCI, OPI, and SEI NAL units, as well as the length field used in each sample to indicate the length of its containing NAL units. This record is outer frame (its size is provided by the structure that contains it). This record contains a version field. This version of the specification defines version 1 of this record. Incompatible changes to this record are indicated by a change in the version number. If the version number is unrecognized, a reading device MUST NOT attempt to decode this record or the stream to which this record applies. Compatible extensions of this record extend this record and do not change the configuration version code. Reading devices MUST be prepared to ignore unrecognized data beyond the definition of understandable data.

[0041] The VVC Profile Tier Level Record (VvcPTLRecord) MUST be present in a decoder configuration record if the track natively contains a VVC bitstream or resolves a "subp" track reference, in which case the specific output layer set in the VVC bitstream is indicated by the field output_layer_set_idx. If ptl_present_flag is equal to 0 in the track's decoder configuration record, the track MUST have an "oref" track reference whose ID can refer to either a VVC track or an "opeg" entity group. The values ​​of the syntax elements chroma_format_idc and bit_depth in minus 8 format (bit_depth_minus8) of the VvcPTLRecord MUST be valid for all parameter sets (referred to as "all parameter sets" in the following text of this paragraph) that will be referenced when the stream described by this record is decoded. In particular, the following restrictions MAY apply:

[0042] The profile directive general_profile_idc shall indicate the profile to which the output layer set indicated by the output layer set index (output_layer_set_idx) of this configuration record conforms. If different profiles are marked for different CVS of the output layer set identified by the output_layer_set_idx of this configuration record, it may be necessary to inspect the stream to determine which profile the entire stream conforms to. If the entire stream is not inspected, or inspection reveals that there is no profile to which the entire stream conforms, the entire stream is expected to be split into two or more substreams with separate configuration records capable of satisfying these rules. The tier directive general_tier_flag shall indicate the tier corresponding to the highest tier or higher indicated by all profile_tier_level() syntax structures (in all parameter sets) to which the output layer set identified by the output_layer_set_idx of this configuration record conforms.

[0043] Each bit in general_constraint_info may be set only if the bit is set in all general_constraints_info() syntax structures of all profile_tier_level() syntax structures (in all parameter sets) to which the output layer set identified by output_layer_set_idx of this configuration record conforms. The level indicator general_level_idc shall indicate a capability level equivalent to or greater than the highest level of all profile_tier_level() syntax structures (in all parameter sets) to which the output layer set indicated by output_layer_set_idx of this configuration record conforms.

[0044] The following constraints apply to the chroma format identification code (chroma_format_idc): If the VVC stream to which the configuration record applies is a single-layer bitstream, the value of sps_chroma_format_idc defined in ISO / IEC 23090-3 must be the same for all SPSs referenced by the VCL NAL units of the sample to which the current sample entry description applies, and the value of chroma_format_idc must be equal to the value of sps_chroma_format_idc. Otherwise (the VVC stream to which the configuration record applies is a multi-layer bitstream), the value of vps_ols_dpb_chroma_format[MultiLayerOlsIdx[output_layer_set_idx]] must be the same for all SPSs to which the current sample entry description applies, and the value of chroma_format_idc must be equal to the value of vps_ols_dpb_chroma_format[MultiLayerOlsIdx[output_layer_set_idx]].

[0045] The following constraints apply to bit_depth_minus8: If the VVC stream to which the configuration record applies is a single-layer bitstream, then the value of sps_bitdepth_minus8 MUST be the same in all SPSs referenced by the VCL NAL units of the sample to which the current sample entry description applies, and the value of sps_bitdepth_minus8 MUST be equal to that value of sps_bitdepth_minus8. Otherwise (the VVC stream to which the configuration record applies is a multi-layer bitstream), then the value of vps_ols_dpb_bitdepth_minus8[MultiLayerOlsIdx[output_layer_set_idx], ] MUST be the same for all CVSs to which the current sample entry description applies, and the value of bit_depth_minus8 MUST be equal to that value of vps_ols_dpb_bitdepth_minus8[MultiLayerOlsIdx[output_layer_set_idx]].

[0046] The following constraints apply to picture_width: If the VVC stream to which the configuration record applies is a single-layer bitstream, the value of sps_pic_width_max_in_luma_samples defined in ISO / IEC23090-3 shall be the same for all SPSs referenced by the VCL NAL units of the sample to which the current sample entry description applies, and the value of picture_width shall be equal to the value of sps_pic_width_max_in_luma_samples. Otherwise (if the VVC stream to which the configuration record applies is a multi-layer bitstream), the value of vps_ols_dpb_pic_width[MultiLayerOlsIdx[output_layer_set_idx]] shall be the same for all SPSs to which the current sample entry description applies, and the value of vps_ols_dpb_pic_width[MultiLayerOlsIdx[output_layer_set_idx]] shall be equal to that value.

[0047] The following constraints apply to picture_height: If the VVC stream to which the configuration record applies is a single-layer bitstream, the value of sps_pic_height_max_in_luma_samples must be the same for all SPSs referenced by the VCL NAL units of the sample to which the current sample entry description applies, and the value of picture_height must be equal to that value of sps_pic_height_max_in_luma_samples. Otherwise (if the VVC stream to which the configuration record applies is a multi-layer bitstream), the value of vps_ols_dpb_pic_height[MultiLayerOlsIdx[output_layer_set_idx]] must be the same for all SPSs to which the current sample entry description applies, and the value of vps_ols_dpb_pic_height[MultiLayerOlsIdx[output_layer_set_idx]] must be equal to that value.

[0048] The VVC decoder configuration record explicitly indicates the chroma format, bit depth, and other format information used by the VVC video elementary stream. Two sequences with different color space and bit depth indications in the VUI information will also employ two different VVC sample entries.

[0049] There is a sequence for carrying initialization non-VCL NAL units. The NAL unit type is restricted to only indicate DCI, OPI, VPS, SPS, PPS, prefix APS, and prefix SEI NAL units. Reserved NAL unit types may be further defined, and readers should ignore sequences with reserved or unauthorized values ​​of NAL unit type. This tolerant behavior is designed to be error-free and allows for the possibility of backward-compatible extensions to these sequences herein. The NAL units included in a sample entry are included immediately after the AUD and OPI NAL units (any one of them) or at the beginning of the access unit reconstructed from the first sample that references the sample entry.

[0050] The recommended ordering is DCI, OPI, VPS, SPS, PPS, prefix APS, prefix SEI.

[0051] Example syntax for VVCPTLRecord and VvcDecoderConfigurationRecord is as follows: aligned(8) class VvcPTLRecord(num_sublayers) { bit(2) reserved = 0; unsigned int(6) num_bytes_constraint_info; unsigned int(7) general_profile_idc; unsigned int(1) general_tier_flag; unsigned int(8) general_level_idc; unsigned int(1) ptl_frame_only_constraint_flag; unsigned int(1) ptl_multilayer_enabled_flag; unsigned int(8*num_bytes_constraint_info - 2) general_constraint_info; for (i=num_sublayers - 2; i >= 0; i--) unsigned int(1) ptl_sublayer_level_present_flag[i]; for (j=num_sublayers; j<=8 && num_sublayers > 1; j++) bit(1) ptl_reserved_zero_bit = 0; for (i=num_sublayers-2; i >= 0; i--) if (ptl_sublayer_level_present[i]) unsigned int(8) sublayer_level_idc[i]; unsigned int(8) num_sub_profiles; for (j=0; j < num_sub_profiles; j++) unsigned int(32) general_sub_profile_idc[j]; }. aligned(8) class VvcDecoderConfigurationRecord { unsigned int(8) configurationVersion = 1; bit(5) reserved = ‘11111’b; unsigned int(2) lengthSizeMinusOne; unsigned int(1) ptl_present_flag; if (ptl_present_flag){。 VvcPTLRecord(numTemporalLayers) track_ptl; unsigned int(16) output_layer_set_idx; unsigned int(16) avgFrameRate; unsigned int(2) constantFrameRate; unsigned int(3) numTemporalLayers; unsigned int(2) chroma_format_idc; unsigned int(3) bit_depth_minus8; bit(6) reserved = '111111'b; unsigned_int(16) picture_width; unsigned_int(16) picture_height; } unsigned int(8) numOfArrays; for (j=0; j < numOfArrays; j++) { unsigned int(1) array_completeness; bit(2) reserved = 0; unsigned int(5) NAL_unit_type; unsigned int(16) numNalus; for (i=0; i< numNalus; i++) { unsigned int(16) nalUnitLength; bit(8*nalUnitLength) nalUnit; } } }

[0052] Example semantics for the above syntax elements are as follows:

[0053] num_bytes_constraint_info is used to specify the length of the general_constraint_info field. The length of the general_constraint_info field is num_bytes_constraint_info*8-2 bits. The value must be greater than 0. A value equal to 1 indicates that the gci_present_flag of the general_constraint_info() syntax structure pointed to by the general_constraint_info field is 0.

[0054] general_profile_idc, general_tier_flag, general_level_idc, ptl_frame_only_constraint_flag, ptl_multilayer_enabled_flag, general_constraint_info, sublayer_level_present[j], sublayer_level_idc[i], num_sub_profiles, and general_sub_profile_idc[j] contain the matching values ​​of the fields or syntax for which general_tier_flag, general_level_idc, ptl_frame_only_constraint_flag, ptl_multilayer_enabled_flag, general_constraint_info(), ptl_sublayer_level_present[i], sublayer_level_idc[i], ptl_num_sub_profiles, and general_sub_profile_idc[j] describe the streams to which this configuration record applies.

[0055] lengthSizeMinusOne plus 1 is the length in bytes of the NALUnitLength field of the VVC video stream sample in the stream to which this configuration record applies. For example, a size of 1 byte is indicated by 0, and the value of this field MUST be 0, 1, or 3, corresponding to lengths encoded in 1, 2, and 4 bytes, respectively.

[0056] ptl_present_flag equal to 1 specifies that the track contains a VVC bitstream corresponding to the operation point specified by output_layer_set_idx and numTemporalLayers, and all NAL units in the track belong to that operation point. ptl_present_flag equal to 0 specifies that the track does not contain a VVC bitstream corresponding to a particular operation point, may contain VVC bitstreams corresponding to multiple output layer sets, or may contain one or more individual layers that do not form an output layer set, or individual sublayers except for those with TemporalId equal to 0.

[0057] track_ptl specifies the profile, tier, and level of the output layer set represented by the VVC bitstream included in the track.

[0058] output_layer_set_idx specifies the index of the output layer set represented by the VVC bitstream contained in the track. The value of output_layer_set_idx can be used as the value of the variable TargetOlsIdx that an external means or an OPI NAL unit provides to a VVC decoder to decode the bitstream contained in the track.

[0059] avgFrameRate gives the average frame rate of the stream to which this configuration record applies, in units of frames / (256 seconds). A value of 0 indicates an unspecified average frame rate. If the track contains multiple layers and samples are reconstructed for an operation point specified by output_layer_set_idx and numTemporalLayers, it gives the average access unit rate of the bitstream at the operation point. Raster scan slice enable flag

[0060] constantFrameRate equal to 1 indicates that the stream to which this configuration record applies is constant frame rate. A value of 2 indicates that the representation of each temporal layer of the stream is constant frame rate. A value of 0 indicates that the stream may or may not be constant frame rate. If the track contains multiple layers and samples are reconstructed for an operation point specified by output_layer_set_idx and numTemporalLayers, this indicates an indication of whether the bitstream for that operation point is constant access unit rate or not.

[0061] numTemporalLayers equal to or greater than 1 indicates that the track to which this configuration record is applied is temporally scalable and contains a number of temporal layers (also called temporal sublayers or sublayers) equal to numTemporalLayers. A value of 1 indicates that the track to which this configuration record is applied is not temporally scalable. A value of 0 indicates that it is unknown whether the track to which this configuration record is applied is temporally scalable.

[0062] chroma_format_idc indicates the chroma format applied to this track.

[0063] picture_width indicates the maximum picture width in luma samples that will be applied to this track.

[0064] picture_height indicates the maximum picture height, in luma samples, that will be applied to this track.

[0065] bit_depth_minus8 indicates the bit depth applied to this track.

[0066] numArrays indicates the number of NAL unit arrays of the indicated type.

[0067] When array_completeness is equal to 1, it indicates that all NAL units of the given type are in the next array and not in the stream. When array_completeness is equal to 0, it indicates that there may be more NAL units of the indicated type in the stream. Allowed values ​​are constrained by the sample entry name.

[0068] NAL_unit_type indicates the type of the NAL units in the next array (that is, they are all of that type) and is restricted to take one of the values ​​indicating DCI, OPI, VPS, SPS, PPS, prefix APS, or prefix SEI NAL units.

[0069] numNalus indicates the number of NAL units of the indicated type contained in the configuration record of the stream to which this configuration record applies. The SEI array shall only contain SEI messages that are of a declarative nature, i.e., SEI messages that provide information about the entire stream. An example of such an SEI could be a user data SEI.

[0070] nalUnitLength indicates the length of the NAL unit in bytes.

[0071] A nalUnit includes a DCI, OPI, VPS, SPS, PPS, APS or declarative SEI NAL unit.

[0072] The random access recovery point sample group, also known as the "roll" sample group, is used to provide recovery point information for incremental decoding. When the "roll" sample group is used in a VVC track, the syntax and semantics of the grouping_type_parameter are specified identically to the "sap" sample group.

[0073] layer_id_method_idc equal to 0 and 1 is used when the picture of the target layer of the sample mapped to the "roll" sample group is a GDR picture. When layer_id_method_idc equal to 0, the "roll" sample group specifies the behavior of all layers present in the track.

[0074] The semantics when layer_id_method_idc is equal to 1 are specified here.

[0075] layer_id_method_idc equal to 2 and 3 is used when all pictures of the target layer of samples mapped to the "roll" sample group are not GDR pictures, and for pictures of the target layer that are not GDR pictures the following applies: for a subpicture index i in the range of 0 to sps_num_subpics_minus1, whose reference PPS has pps_mixed_naru_types_in_pic_flag equal to 1, both of the following are true: there is at least one IRAP subpicture with sps_subpic_treated_as_pic_flag[i] equal to 1 and identical to the next subpicture index i of the current sample in the same CLVS. When layer_id_method_idc is 2, the "roll" sample group specifies the behavior of all layers present in the track. The semantics when layer_id_method_idc is equal to 3 are specified in this specification. If a sample with layer_id_methoc_idc equal to 2 or 3 is used to start decoding, the reader MUST further modify the SPS, PPS, and PH NAL units of the bitstream such that a bitstream starting with a sample marked as belonging to a sample group with layer_id_methoc_idc equal to 2 and 3 is a conforming bitstream when the SPS referenced by that sample has sps_gdr_enabled_flag equal to 1, and any PPS referenced by such a sample has pps_mixed_naru_types_in_pic_flag equal to 0, all VCL NAL units of the AU have nal_unit_type equal to GDR_NUT, ph_gdr_pic_flag equal to 1 in one of the picture headers of the AU, and the value of ph_recovery_poc_cnt corresponds to the roll_distance of the sample group to which this AU belongs. When a "role" sample group pertains to a dependent layer but not to its reference layer(s), the sample group indicates characteristics that will be applied if all reference layers of the dependent layer are available and decoded.The sample group can be used to start the decoding of the prediction layer.

[0076] When layer_id_method_idc is equal to 1, each bit in the field target_layers represents a layer carried in the track. Since this field is only 28 bits long, the indication of SAPs in a track is constrained to a maximum of 28 layers. Each bit in this field, starting from the least significant bit (LSB), shall be mapped to the list of layer ID values ​​signaled in the layer information sample group ('linf') associated with that sample, in ascending order of layer ID value.

[0077] The following are examples of technical problems solved by the disclosed technical solution. The current design of the VVC video file format for signaling decoder configuration information and "role" sample group has the following problems. The first problem is that picture format parameters such as color format, bit depth, picture width, picture height, etc. are signaled when profile, tier, level information (PTL) is signaled in the VvcDecoderConfigurationRecord. These information can be used for content selection. However, other parameters useful for content selection purposes, such as required decoded picture buffer size, maximum picture output reordering, maximum latency, GDR picture enable flag, CRA picture enable flag, reference picture resampling enable flag, spatial resolution change by CLVS enable flag, sub-picture split enable flag, maximum number of sub-pictures per picture, WPP enable flag, tile split enable flag, maximum number of tiles in a picture, slice split enable flag, rectangular slice enable flag, raster scan slice enable flag, maximum number of slices in a picture, etc. may not be signaled in the decoder configuration record.

[0078] Next, in the VvcDecoderConfigurationRecord, when PTL information is signaled, the numTemporalLayers field is also signaled after the PTL information signaling, although the syntax structure of the PTL information signaling depends on the numTemporalLayers field.

[0079] Third, in the description of the Random Access Recovery Point sample group, i.e., the "ROLL" sample group, the semantics of the fields with layer_id_method_idc equal to 1 or 3 are not adequately specified. In particular, the signaling of the applicable layer may be specified when layer_id_method_idc is equal to 1, but may not be specified when layer_id_method_idc is equal to 3.

[0080] Disclosed herein is a mechanism that addresses one or more of the above problems. In one example, a VVC decoder configuration record is modified to position the number of sublayers to precede the PTL record. In this way, the decoder can first obtain the number of sublayers and use the number to obtain a PTL record for each of the sublayers. In another example, a grouping type parameter of a roll sample group is modified to more clearly describe the correlation between the access points in the roll sample group and the layers to which they apply. For example, a target layer can indicate the layer that correlates to the access point. Additionally, a layer identifier method identification code can be set to indicate whether the access point applies to all layers or only to layers in the target layer parameter. Additionally, a layer identifier method identification code can be set to indicate whether the access point consists of only GDR pictures or a combination of GDR pictures and mixed NAL unit pictures.

[0081] In order to solve the above problems and others, the methods summarized below are disclosed. Note that these items should be considered as examples to explain the general concept and should not be interpreted in a narrow sense. Furthermore, these items can be applied individually or combined.

[0082] Example 1 To solve the first problem, any one or more of the following parameters may be signaled in the VvcDecoderConfigurationRecord: maximum required size of the decoded picture buffer, maximum picture output reordering (e.g., the maximum allowed number of pictures that can precede any picture in decoding order and follow that picture in output order), maximum delay (e.g., the maximum number of pictures that can precede any picture in output order and follow that picture in decoding order), GDR picture enable flag, CRA picture enable flag, reference image resampling enable flag, CLVS spatial resolution change enable flag, sub-picture partitioning enable flag, maximum number of sub-pictures per picture, WPP enable flag, maximum number of tile partitioning, slice partitioning, maximum number of rectangular slices, number of raster scan slices allowed, and maximum number of tiles per picture. (a) In one example, one or more of the above parameters are signaled in the VvcDecoderConfigurationRecord only if PTL information is signaled therein. (b) In one example, one or more parameters may be present before the signaling of the PTL information. Furthermore, byte alignment may be a requirement for all parameters that are signaled before the PTL information. In one example, a reserved bit may also be signaled. (c) In one example, one or more parameters may be present after the signaling of the PTL information. Furthermore, byte alignment may be required for all parameters that may be signaled after the PTL information. In one example, a reserved bit may also be signaled. (d) In one example, a subset of one or more parameters may be present before the signaling of the PTL information and the remaining parameters may be present after the signaling of the PTL information. Further, byte alignment may be required for all parameters that may be signaled before the PTL information. In one example, a reserved bit may also be signaled. (e) Additionally, byte alignment may be required for all parameters that may be signaled after the PTL information. In one example, reserved bits may also be signaled.

[0083] Example 2 To solve the second problem, the VvcDecoderConfigurationRecord is modified such that when PTL information is signaled, the numTemporalLayers field is also signaled before the signaling of the PTL information. (a) In one example, when PTL information is signaled in the VvcDecoderConfigurationRecord, it is signaled after the fields chroma_format_idc, bit_depth_minus8, numTemporalLayers, and constantFrameRate. In one example, the PTL information is signaled directly after all the above mentioned fields and some reserved bits. (b) In one example, when PTL information is signaled in the VvcDecoderConfigurationRecord, it is signaled after the numTemporalLayers and constantFrameRate fields. In one example, the PTL information is signaled directly after all the above mentioned fields and some reserved bits. Furthermore, the reserved bits are further signaled after the PTL information. (c) In another example, when PTL information is signaled in the VvcDecoderConfigurationRecord, it is signaled as the last field among all fields subject to "if(ptl_present_flag)". (d) In one example, the reserved bit is signaled before the signaling of the PTL information.

[0084] Example 3 To solve problem 3, make one or more of the following changes to the following sentence: (a) Change "When layer_id_method_idc is equal to 1, the semantics are specified in clause 9.5.7." to: "When layer_id_method_idc is equal to 1, the layers whose behavior is specified by the 'role' sample group are specified in clause 9.5.7." As used in this specification, clause 9.5.7 refers to the correspondingly numbered clause of document ISO / IEC 14496-15:2021 (E) titled Information technology - Coding of audio-visual objects - Part 15: Transport of network abstraction layer (NAL) units structured video in the ISO base media file format. (b) Change "The semantics of when layer_id_method_idc is equal to 3 are specified in Section 9.5.7." to read as follows: "When layer_id_method_idc is equal to 3, the layers whose behavior is specified by the "role" sample group are specified by Section 9.5.7 as when layer_id_method_idc is equal to 1."

[0085] Example 4 To resolve issue 3, or make one or more of the alternative changes: (a) In Section 9.5.7, change the following sentence: “When layer_id_method_idc is equal to 1, each bit of the field target_layers represents a layer carried on the truck” to read: “When layer_id_method_idc is equal to 1 or 3, each bit of the field target_layers represents a layer carried on the truck.” (b) The following sentence: “When layer_id_method_idc is equal to 1, the semantics are specified in Section 9.5.7.” should be changed to: “When layer_id_method_idc is equal to 1, the layers whose behavior is specified by the ‘role’ sample group are specified in Section 9.5.7.” (c) The following sentence: “When layer_id_method_idc is equal to 3, the semantics are specified in Section 9.5.7.” should be changed to: “When layer_id_method_idc is equal to 3, the layers whose behavior is specified by the ‘role’ sample group are specified in Section 9.5.7.”

[0086] Below are some exemplary embodiments applicable to the VVC video file format standard specification in some aspects summarized above. The changed text is based on the latest draft specification of the relevant features as above. The relevant parts that have been added or modified are underlined and bold, and the parts that have been deleted are bold italics.

[0087] In one example, the syntax of the VvcDecoderConfigurationRecord is modified as follows: [ka]

[0088] In one example, the semantics of the VvcDecoderConfigurationRecord are modified as follows:

[0089] ptl_present_flag equal to 1 specifies that the track contains a VVC bitstream corresponding to the operation point specified by output_layer_set_idx and numTemporalLayers, and all NAL units in the track belong to that operation point. ptl_present_flag equal to 0 specifies that the track does not contain a VVC bitstream corresponding to a particular operation point, may contain VVC bitstreams corresponding to multiple output layer sets, or may contain one or more individual layers that do not form an output layer set, or individual sublayers except for those with TemporalId equal to 0.

[0090] [ka]

[0091] [ka]

[0092] [ka]

[0093] [ka]

[0094] track_ptl specifies the profile, tier, and level of the output layer set represented by the VVC bitstream included in the track.

[0095] The output_layer_set_idx specifies the index of the output layer set represented by the VVC bitstream included in the track. The value of the output_layer_set_idx can be used as the value of the TargetOlsIdx variable provided by an external means or an OPI NAL unit to a VVC decoder specified in ISO / IEC23090-3 for decoding the bitstream included in the track.

[0096] [ka]

[0097] [ka]

[0098] [ka]

[0099] [ka]

[0100] picture_width indicates the maximum picture width in luma samples that will be applied to this track.

[0101] picture_height indicates the maximum picture height, in luma samples, that will be applied to this track.

[0102] [ka]

[0103] [ka]

[0104] numArrays indicates the number of NAL unit arrays of the indicated type.

[0105] In one example, the description of the random access recovery point sample group is modified as follows: The random access recovery point sample group "ROLL" is used to provide information about recovery points for gradual decoding refresh. When the "ROLL" sample group is used in a VVC track, the syntax and semantics of the grouping_type_parameter are specified equal to the syntax of the "SAP" sample group. When the target layer picture of the sample mapped to the "ROLL" sample group is a GDR picture, layer_id_method_idc equal to 0 and 1 is used. When layer_id_method_idc equal to 0, the "ROLL" sample group specifies the behavior of all layers present in the track.

[0106] [ka]

[0107] layer_id_method_idc equal to 2 and 3 is used when all target layer pictures of samples mapped to the "roll" sample group are not GDR pictures, and for target layer pictures that are not GDR pictures the following applies: for subpicture index i in the range 0 to sps_num_subpics_minus1, whose reference PPS has pps_mixed_naru_types_in_pic_flag equal to 1, both of the following are true: there is at least one IRAP subpicture with sps_subpic_treated_as_pic_flag[i] equal to 1 and identical to the next subpicture index i of the current sample in the same CLVS. If layer_id_method_idc is 2, the "roll" sample group specifies the behavior of all layers present in the track.

[0108] [ka]

[0109] If the reader starts decoding using a sample with layer_id_methoc_idc equal to 2 or 3, the reader MUST further modify the SPS, PPS, and PH NAL units of the reconstructed bitstream according to clause 11.6 (document ISO / IEC 14496-15:2021(E)). A bitstream with layer_id_method_idc equal to 2 and 3 and starting with a sample marked as belonging to this sample group is a conforming bitstream when the SPS to which such sample refers has sps_gdr_enabled_flag equal to 1 and any PPS has pps_mixed_nalu_types_in_pic_flag equal to 0, all VCL NAL units of the AU have nal_unit_type equal to GDR_NUT, and any picture header of the AU has ph_gdr_pic_flag equal to 1, and the picture header of the AU has ph_gdr_pic_flag equal to 1 and a value of ph_recovery_poc_cnt corresponding to the roll_distance of this sample group to which the AU belongs.

[0110] If a "roll" sample group pertains to a dependent layer but not to its reference layer(s), the sample group indicates characteristics that are applied when all reference layers of the dependent layer are available and decoded. The sample group can be used to initiate decoding of the prediction layer.

[0111] In one example, the syntax of the VvcDecoderConfigurationRecord is modified as follows: [ka]

[0112] In one example, the semantics of the VvcDecoderConfigurationRecord are modified as follows:

[0113] ptl_present_flag equal to 1 specifies that the track contains a VVC bitstream corresponding to the operation point specified by output_layer_set_idx and numTemporalLayers, and all NAL units in the track belong to that operation point. ptl_present_flag equal to 0 specifies that the track does not contain a VVC bitstream corresponding to a particular operation point, may contain VVC bitstreams corresponding to multiple output layer sets, or may contain one or more individual layers that do not form an output layer set, or individual sublayers except for those with TemporalId equal to 0.

[0114] [ka]

[0115] The output_layer_set_idx specifies the index of the output layer set represented by the VVC bitstream included in the track. The value of the output_layer_set_idx can be used as the value of the TargetOlsIdx variable provided by an external means or an OPI NAL unit to a VVC decoder specified in ISO / IEC23090-3 for decoding the bitstream included in the track.

[0116] avgFrameRate indicates the average frame rate of the stream to which this configuration record applies, in units of frames / (256 seconds). A value of 0 indicates an unspecified average frame rate. If the track contains multiple layers and samples are reconstructed for an operation point specified by output_layer_set_idx and numTemporalLayers, this gives the average access unit rate of the bitstream at the operation point.

[0117] constantFrameRate equal to 1 indicates that the stream to which this configuration record applies is constant frame rate. A value of 2 indicates that the representation of each temporal layer of the stream is constant frame rate. A value of 0 indicates that the stream may or may not be constant frame rate. If the track contains multiple layers and samples are reconstructed for an operation point specified by output_layer_set_idx and numTemporalLayers, this indicates an indication of whether the bitstream for that operation point is constant access unit rate or not.

[0118] numTemporalLayers equal to or greater than 1 indicates that the track to which this configuration record applies is temporally scalable and contains a number of temporal layers (also called temporal sublayers or sublayers in ISO / IEC23090-3) equal to numTemporalLayers. A value of 1 indicates that the track to which this configuration record applies is not temporally scalable. A value of 0 indicates that it is unknown whether the track to which this configuration record applies is temporally scalable.

[0119] chroma_format_idc indicates the chroma format applied to this track.

[0120] [ka]

[0121] picture_width indicates the maximum picture width in luma samples that will be applied to this track.

[0122] picture_height indicates the maximum picture height, in luma samples, that will be applied to this track.

[0123] [ka]

[0124] [ka]

[0125] numArrays indicates the number of NAL unit arrays of the indicated type.

[0126] 1 is a schematic diagram of an example media file 100 that includes a VVC bitstream 127 of video data. The media file includes pictures 125 that may be displayed to create a video sequence. The pictures 125 are compressed in the VVC bitstream 127. The bitstream 127 is also composed of various parameter sets 123 that indicate to the decoded pictures the parameters used to compress the pictures 125. The parameter sets 123 may be a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), or an adaptation parameter set (APS), which respectively include parameters for an entire video, parameters for a sequence of a video, parameters for one or more pictures, and parameters for a region of one or more pictures.

[0127] Compression can include intra-prediction and inter-prediction. In intra-prediction, a picture 125 is divided into blocks, and each block is coded relative to other blocks in the same picture 125. In inter-prediction, a picture 125 is divided into blocks, and each block is coded relative to other blocks in other pictures 125. A coded picture 125 according to inter-prediction or intra-prediction may be referred to as an inter-coded picture or an intra-coded picture, respectively. One advantage of an inter-coded picture is that such a picture 125 is substantially more compressed than an intra-coded picture. However, because an inter-coded picture is coded relative to other pictures 125, a video decoder cannot start decoding a video sequence at an inter-coded picture. Instead, a video decoder can start decoding video at any intra-coded picture. An intra-coded picture may also be referred to as an IRAP picture, because an intra-coded picture can serve as an access point 135 to a video stream. An access point 135 is any location within a video stream where a decoder can generally begin decoding the video stream without experiencing decoding errors due to lost information, for example, except for GDR pictures as described below.

[0128] In some embodiments, picture 125 may be divided into sub-pictures. A sub-picture is a rectangular region of picture 125. The advantage of sub-pictures is that they can be treated separately during decoding and display processes. Sub-pictures can be displayed instead of displaying the entire picture 125, for example, in picture-in-picture applications or virtual reality applications. Sub-pictures can also be rearranged and stitched together in different configurations, for example, in video calling applications. In some embodiments, the set of access points 135 may be different for different sub-pictures in the same picture 125. For example, sub-pictures with less important images may have fewer access points 135 to increase compression. When this occurs, picture 125 may include intra-coded and inter-coded sub-pictures, also referred to as IRAP and non-IRAP sub-pictures, such sub-pictures also referred to as IRAP and non-IRAP sub-pictures. Bitstream 127 is a set of Network Abstraction Layer (NAL) units, which are video data elements sized to fit into communication network packets. Thus, parameter sets 123 and pictures 125 are carried in NAL units of bitstream 127. Therefore, picture 125, which has both IRAP and non-IRAP subpictures, can be referred to as a mixed NAL unit picture.

[0129] Another access point 135 scheme involves the use of GDR pictures. A GDR picture includes an intra-coded section and one or more inter-coded sections. GDR pictures are used in groups to create the access points 135. Specifically, a first GDR picture includes an intra-coded region in the leftmost portion of the picture 125, and the remaining part of the picture is coded according to inter-coding. A second GDR picture includes an intra-coded region shifted right to a position that touches but does not overlap the intra-coded region of the first GDR picture. The remaining part of the second GDR picture is inter-coded. In this way, the intra-coded pictures sweep from left to right across multiple pictures. One constraint of GDR pictures is that the inter-coded region to the left of the intra-coded region can only reference the previous GDR picture in the group of the current GDR picture. The decoder may start decoding with the first GDR picture of the group. In this case, the decoder can decode the intra-coded region but not the inter-coded region. The decoder can then proceed to the second GDR picture, where it can decode both the intra-coded region and the inter-coded region to the left of the intra-coded region. When the decoder reaches the last GDR picture, all regions are decoded and the video can be displayed. Although GDR pictures will cause errors when used as access points 135, such errors will not persist beyond the last GDR picture in the group. Thus, GDR pictures are generally not displayed when a group is used as an access point 135. The advantage of GDR pictures is that each GDR picture is smaller than an entire IRAP picture, reducing the data burst associated with each access point 135.If the decoder is not using a GDR picture as an access point 135, the video prior to the group of GDR pictures is available, and therefore the decoder can decode all GDR pictures in the group without errors in the inter-coded region. Note that GDR pictures are generally prohibited from being used in combination with mixed NAL unit pictures.

[0130] Pictures 125 and parameter sets 123 may be organized into layers 120 and / or sublayers. A layer 120 is a grouping of pictures 125 and parameter sets 123 that may be decoded and output as part of an output layer set. For example, different layers 120 may be coded at different resolutions. In another example, an output layer set may include a base layer and an enhancement layer. This allows a decoder to decode a base layer to obtain a first resolution picture, and then decode a desired number of enhancement layers to increase the resolution based on device and network capabilities. A sublayer 121 is a type of layer 120 that allows temporal scaling. For example, a picture 125 may be assigned to different sublayers 121 based on a temporal identifier (Id). In this way, each sublayer 121 includes a subset of pictures 125. This allows a decoder to decode and display a selected sublayer 121 to achieve a desired frame rate.

[0131] The layers 120 and / or sublayers 121 of the bitstream 127 may be arranged in tracks 110. The tracks 110 include a sequence of timed samples of a particular type that may be decoded and displayed by a decoder. In this context, a sample is a unit of media data. For example, the tracks 110 may include a timed set of compressed video samples (e.g., pictures 125 over time), compressed audio samples, hint data samples, parameter samples, etc. It should be noted that although the term sample may refer to pixel values ​​of pixels, that is not the intended definition in this context. The tracks 110 may include any one of the layers 120 and / or any one of the sublayers 121 that include such samples.

[0132] As can be appreciated by the preceding description, the data in the media file 100 can be arranged in many ways. Thus, the media file 100 also includes a sample table box 130 that contains parameters that describe the samples (e.g., media data) contained in the tracks 110. For example, a decoder can read the sample table box 130 to determine how to begin processing the data contained in the various tracks 110. Among many other parameters, the sample table box 130 can include roll sample groups 131 and VVC decoder configuration records 141.

[0133] The roll sample group 131 is also referred to as a random access recovery sample group. The roll sample group 131 is a data unit used for signaling access points 135 to the layer 120 of the VVC bitstream 127, and is mainly used for signaling access points 135 occurring in GDR pictures. It is also possible to adopt a random access point (RAP) sample group for signaling access points occurring in other IRAP pictures, such as IDR, CRA, and broken link access (BLA). Thus, the roll sample group 131 includes a list of access points 135 that point to GDR pictures included in the VVC bitstream 127. The access points 135 are considered to be samples of the roll sample group 131. In some implementations, the operation of the roll sample group 131 is unclear. The present disclosure addresses such a problem by providing a parameter that clearly describes the relationship between the access points 135 of the roll sample group 131 and the layer 120.

[0134] The roll sample group 131 includes a grouping type parameter 137, which may be represented as group_type_parameter. The grouping type parameter 137 is a parameter that specifies the correlation / correspondence between the access point 135 and the layers 120. It should be noted that when the access point 135 is applied to the layer 120, the layer 120 may be referred to as an associated layer. Thus, the layer 120 includes a group of associated layers, which may be similar to the set of all layers 120 or a subset of the layers 120. The grouping type parameter 137 further includes a target layer parameter 136 and a layer identifier method specific code 138, which may be represented as target_layers and layer_id_method_idc, respectively. In one example implementation, the target layer parameter 136 includes a number of bits, each bit specifying one of the associated layers. In one example, the target layer parameter 136 may be 24 bits long, and thus may be able to specify up to 24 associated layers.

[0135] The layer identifier method identification code 138 defines any nature of the access point 135 and clarifies the correlation between the access point 135 and the layer. In one example, the layer identifier method identification code 138 may include a 4-bit value. In a specific implementation, the layer identifier method identification code 138 may be set to zero or 2 to indicate that the access point 135 applies to all layers 120. In this case, all layers are relevant layers, and the target layer parameters 136 may be omitted from the media file 100 and / or ignored by the decoder. Additionally, the layer identifier method identification code 138 may be set to 1 or 4 to indicate that the access point 135 applies only to relevant layers indicated by the target layer parameters 136. Additionally, the layer identifier method identification code 138 may indicate the nature of the picture 125 present in the access point 135. For example, the layer identifier method identification code 138 may be set to zero or 1 to indicate that the access points 135 are all GDR pictures. Additionally, the layer identifier method identification code 138 can be set to 2 or 3 to indicate that the access point 135 can be either a GDR picture or a mixed NAL unit picture using both IRAP and non-IRAP subpictures.

[0136] In a particular implementation, layer_id_method_idc may be set to zero if all access points of the layer are GDR pictures and specifies that the access points apply to all layers. Furthermore, layer_id_method_idc is set to 1 if all access points of the associated layer are GDR pictures and specifies that the access points apply only to the associated layer. Also, layer_id_method_idc is set to 2 if the access points of the layer are GDR pictures, combined NAL unit pictures, and combinations thereof, and specifies that the access points apply only to the associated layer. Finally, layer_id_method_idc is set to 3 if the access points of the layer are GDR pictures, mixed NAL unit pictures, and combinations thereof, and specifies that the access points apply only to the associated layer. In this way, the decoder can analyze the access points 135, the grouping type parameter 137, the target layer 136, and the layer identifier method identification code 138 to determine the correlation between the access points 135 of the roll sample group 131 and the layers 120. The decoder can then use the access point 135 to start decoding the picture 125 in the associated layer.

[0137] Additionally, the sample table box 130 may include a VVC decoder configuration record 141, which may be denoted as a VVCDecoderConfigurationRecord. The VVC decoder configuration record 141 includes data that a decoder can use to select content. For example, the VVC decoder configuration record 141 may include data describing the output layer set and corresponding layers 120 of a track 110. The decoder can then use such data to select the track 110 to be decoded and displayed. For example, the VVC decoder configuration record 141 may include data describing a VVC profile tier level (PTL) record 143, an output layer set index, a frame rate, a number of sub-layers 121, a bit depth, a chroma format, a picture sizing, etc.

[0138] The VVC PTL record 143 indicates profile, tier, and level information for the layer 120 and / or sublayer 121. The profile, tier, and level specify the limitations of the bitstream and therefore the capabilities required to decode the bitstream. The profile, tier, and level may also be used to indicate interoperability points between individual decoder implementations. A profile is a defined set of coding tools used to create a compliant bitstream. Each profile specifies a subset of algorithmic capabilities and limitations that all decoders conforming to that profile must support. A level is a set of constraints for the bitstream (e.g., maximum luminance sample rate, maximum bitrate of resolution, etc.). For example, a level may be a set of constraints indicating the decoder capabilities required to play a bitstream of a specified profile (e.g., hardware constraints). The levels are divided into two tiers: main and high. The main tier is a lower tier than the high tier. The tiers are adopted to address applicability with different maximum bitrates. The main tier is designed for most applications and the high tier is designed for very demanding applications. Within any profile, the level of the tiers generally corresponds to the processing load and memory capabilities of a particular decoder, and thus, it is desirable for the decoder to select layers 120 and / or sub-layers 121 for playback by determining layers 120 and / or sub-layers 121 that have PTL information that matches the capabilities of the decoder.

[0139] In some example implementations, the VVC decoder configuration record 141 is ambiguous because the number of sub-layers 145 is signaled in the VVC decoder configuration record 141 after the VVC PTL record 143. This is problematic because the decoder needs the number of sub-layers 145 before the decoder can interpret the VVC PTL record 143. In this disclosure, the number of sub-layers 145 is signaled in the VVC decoder configuration record 141 before the VVC PTL record 143. The decoder can then parse the VVC decoder configuration record to obtain the number of sub-layers 145 and use the number of sub-layers 145 to determine the number of VVC PTL records for the sub-layer 121. In one example, the VVC decoder configuration record 141 is composed of a constant frame rate syntax element, a chroma format identification code syntax element, and a bit depth minus 8 syntax element. The VVC PTL record 143 can be located in the VVC decoder configuration record 141 after the constant frame rate syntax element, the chroma format identification code syntax element, and the bit depth minus 8 syntax element. Additionally, the number of sub-layers 145 may be placed in the VVC decoder configuration record 141 before the constant frame rate syntax element, the chroma format identification code syntax element, and the bit depth minus 8 syntax element.

[0140] Specifically, to place the number of sub-layers 145 in front of the VVC PTL record 143 used to determine the PTL information of the track 110, layer 120, and / or sub-layer 121, the VVC decoder configuration record 141 may be implemented as follows: aligned(8) class VvcDecoderConfigurationRecord { unsigned int(8) configurationVersion = 1; bit(5) reserved = '11111'b; unsigned int(2) lengthSizeMinusOne; unsigned int(1) ptl_present_flag; if (ptl_present_flag){ unsigned int(2) chroma_format_idc; unsigned int(3) bit_depth_minus8; unsigned int(3) numTemporalLayers; unsigned int(2) constantFrameRate; bit(6) reserved = ‘111111’b; VvcPTLRecord(numTemporalLayers) track_ptl; unsigned int(16) output_layer_set_idx; unsigned_int(16) picture_width; unsigned_int(16) picture_height; unsigned int(16) avgFrameRate; } unsigned int(8) numOfArrays; for (j=0; j < numOfArrays; j++) { unsigned int(1) array_completeness; bit(2) reserved = 0; unsigned int(5) NAL_unit_type; unsigned int(16) numNalus; for (i=0; i< numNalus; i++) { unsigned int(16) nalUnitLength; bit(8*nalUnitLength) nalUnit; } } }

[0141] In another example, various additional information may be included in the VVC decoder configuration record 141 to support the selection of tracks 110, layers 120, and / or sub-layers 121 at the decoder. Such information may include a maximum required size of the decoded picture buffer, a maximum picture output reordering, a maximum latency, a GDR picture enable flag, a CRA picture enable flag, a reference picture resampling enable flag, a spatial resolution change due to CLVS enable, a sub-picture partitioning enable flag, a maximum number of sub-pictures in each picture, a WPP enable flag, a tile split enable flag, a maximum number of tiles per picture, a slice split enable flag, a raster scan slice enable flag, a number of slices per picture, or a combination thereof. In some examples, such information may be included only if the VVC decoder configuration record 141 configures the VVC PTL record 143.

[0142] By including such information and / or by rearranging the order of the data, the VVC decoder configuration record 141 is improved to enable additional functionality and / or more efficient selection of tracks 110, layers 120, and / or sub-layers 121 by the decoder.

[0143] FIG. 2 is a flow chart of an exemplary method 200 of encoding a VVC decoder configuration record, for example, by encoding the VVC decoder configuration record into the media file 100. In step 201, an encoder encodes a plurality of pictures into one or more sub-layers in a media file. In step 203, the encoder determines the number of sub-layers. In step 205, the encoder encodes a VVC decoder configuration record into the media file, the VVC decoder configuration including the number of sub-layers and one or more VVC PTL records for the sub-layers. In one example, the number of sub-layers can be signaled in the VVC decoder configuration record before the VVC PTL record. In one example, the VVC decoder configuration record further includes a constant frame rate syntax element, a chroma format identification code syntax element, and a bit depth minus 8 syntax element. The VVC PTL record can be located in the VVC decoder configuration record after the constant frame rate syntax element, the chroma format identification code syntax element, and the bit depth minus 8 syntax element. Further, the number of sub-layers may be located in the VVC decoder configuration record before the constant frame rate syntax element, the chroma format identification code syntax element, and the bit depth minus 8 syntax element. In one example, the VVC decoder configuration record further includes reserved bits located after the bit depth minus 8 syntax element, and the VVC PTL record is located after the reserved bits. In one example, the VVC decoder configuration record further includes reserved bits located after the VVC PTL record.

[0144] In one example, the VVC decoder configuration record further includes a maximum required size of the decoded picture buffer, a maximum picture output reorder, a maximum delay, a GDR picture enable flag, a CRA picture enable flag, a reference picture resampling enable flag, a spatial resolution change with CLVS enable flag, a sub-picture split enable flag, a maximum number of sub-pictures per picture, a WPP enable flag, a tile split enable flag, a maximum number of tiles per picture, a slice split enable flag, a rectangular slice enable flag, a raster scan slice enable flag, a maximum number of slices per picture, or a combination thereof. In one example, the VVC decoder configuration record only discloses one or more of the preceding parameters if the VVC decoder configuration record configures a VVC PTL record.

[0145] In one example, byte alignment is required for all syntax elements located after the VVC PTL record in the VVC decoder configuration record.

[0146] In step 207, the encoder stores the media file in a memory, for example for later transmission to a decoder. In an embodiment, the media file is transmitted to a decoder.

[0147] FIG. 3 is a flow chart illustrating an example method 300 of decoding a VVC decoder configuration record, employing, for example, a media file 100 received as a result of method 200. In step 301, a decoder receives a media file including a VVC decoder configuration record and a plurality of pictures coded into one or more sub-layers. In step 303, the decoder parses the VVC decoder configuration record to obtain a number of sub-layers and one or more VVC PTL records for the sub-layers based on the number of sub-layers. In one example, the number of sub-layers may be signaled in the VVC decoder configuration record before the VVC PTL record. In one example, the VVC decoder configuration record further includes a constant frame rate syntax element, a chroma format identification code syntax element, and a bit depth minus 8 syntax element. The VVC PTL record may be located in the VVC decoder configuration record after the constant frame rate syntax element, the chroma format identification code syntax element, and the bit depth minus 8 syntax element. Further, the number of sub-layers may be located in the VVC decoder configuration record before the constant frame rate syntax element, the chroma format identification code syntax element, and the bit depth minus 8 syntax element. In one example, the VVC decoder configuration record further includes reserved bits located after the bit depth minus 8 syntax element, and the VVC PTL record is located after the reserved bits. In one example, the VVC decoder configuration record further includes reserved bits located after the VVC PTL record.

[0148] In one example, the VVC decoder configuration record further includes a maximum required size of the decoded picture buffer, a maximum picture output reorder, a maximum delay, a GDR picture enable flag, a CRA picture enable flag, a reference picture resampling enable flag, a spatial resolution change with CLVS enable flag, a sub-picture split enable flag, a maximum number of sub-pictures per picture, a WPP enable flag, a tile split enable flag, a maximum number of tiles per picture, a slice split enable flag, a rectangular slice enable flag, a raster scan slice enable flag, a maximum number of slices per picture, or a combination thereof. In one example, the VVC decoder configuration record only discloses one or more of the preceding parameters if the VVC decoder configuration record configures a VVC PTL record.

[0149] In one example, byte alignment is required for all syntax elements located after the VVC PTL record in the VVC decoder configuration record.

[0150] In step 305, the decoder decodes one or more sub-layers based on the VVC PTL record. The decoder can then forward the decoded media file, or portions thereof (e.g., particular layers and / or sub-layers), to a display for viewing by a user.

[0151] FIG. 4 is a block diagram illustrating a video processing system 400 in which various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of system 400. System 400 may include an input 402 for receiving video content. The video content may be received in raw or uncompressed format, e.g., 8-bit or 10-bit multi-component pixel values, or may be in a compressed or encoded format. Input 402 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces, such as Ethernet, passive optical network (PON), and wireless interfaces, such as Wi-Fi or cellular interfaces.

[0152] The system 400 may include a coding component 404 that may implement various coding or encoding methods described herein. The coding component 404 may reduce the average bit rate of the video from the input 402 to the output of the coding component 404 to generate a coded representation of the video. Thus, coding techniques may be referred to as video compression techniques or video conversion techniques. The output of the coding component 404 may be stored or transmitted over a communication connection, as represented by component 406. The stored or communicated bitstream (or coded) representation of the video received at the input 402 may be used by component 408 to generate pixel values ​​or displayable video that are sent to the display interface 410. The process of generating a user-displayable video from the bitstream representation may be referred to as decompressing the video. Additionally, although certain video processing operations are referred to as "coding" operations or tools, it will be understood that the coding tools or operations are used at the encoder and the corresponding decoding tools or operations that reverse the results of the coding are performed at the decoder.

[0153] Examples of peripheral bus interfaces or display interfaces include Universal Serial Bus (USB) or High Definition Multimedia Interface (HDMI) or Displayport. Examples of storage interfaces include serial advanced technology attachment (SATA), PCI, IDE interfaces, etc. The techniques described herein may be implemented in various electronic devices capable of digital data processing and / or video display, such as mobile phones, laptops, smartphones, etc.

[0154] FIG. 5 is a block diagram of an exemplary video processing device 500. The device 500 can be used to implement one or more of the methods described herein. The device 500 can be implemented in a smartphone, a tablet, a computer, an Internet of Things (IoT) receiver, and the like. The device 500 may include one or more processors 502, one or more memories 504, and video processing hardware 506. The processor(s) 502 are configurable to implement one or more of the methods described herein. The memory (storage) 504 can be used to store data and code used to implement the methods and techniques described herein. The video processing hardware 506 may be used to implement some of the techniques described herein in hardware circuitry. In some embodiments, the video processing hardware 506 may be at least partially included in the processor 502, e.g., a graphics co-processor.

[0155] 6 is a flow chart of an example method 600 of video processing. Method 600 includes converting between visual media data and a file that stores information corresponding to the visual media data according to a video file format. In the context of an encoder, this conversion may be performed by encoding the visual media data into a visual media data file in a video file format. In the context of a decoder, this conversion may be performed by decoding the visual media data file in the video file format to obtain the visual media data for display.

[0156] 7 is a block diagram illustrating an example video coding system 700 that may utilize techniques of this disclosure. As shown in FIG. 7, the video coding system 700 may include a source device 710 and a destination device 720. The source device 710 generates video data to be encoded, which may be referred to as a video encoding device. The destination device 720 may decode the encoded video data generated by the source device 710, which may be referred to as a video decoding device.

[0157] The source device 710 may include a video source 712, a video encoding target 714, and an input / output (I / O) interface 716. The video source 712 may include sources such as a video capture device, an interface for receiving video data from a video content provider and / or a computer graphics system for generating video data, or a combination of such sources. The video data may consist of one or more pictures. The video encoder 714 encodes the video data from the video source 712 to generate a bitstream. The bitstream may include a sequence of bits that form a coded representation of the video data. The bitstream may include a coded picture and associated data. A coded picture is a coded representation of a picture. The associated data may include sequence parameter sets, picture parameter sets, and other syntax structures. The I / O interface 716 may include a modulator / demodulator (modem) and / or a transmitter. The coded video data may be transmitted directly to a destination device 720 via the I / O interface 716 over a network 730. The encoded video data may also be stored on a storage medium / server 740 for access by the destination device 720 .

[0158] The destination device 720 may include an I / O interface 726, a video decoding device 724, and a display device 722. The I / O interface 726 may include a receiver and / or a modem. The I / O interface 726 may obtain encoded video data from the source device 710 or a storage medium / server 740. The video decoder 724 may decode the encoded video data. The display device 722 may display the decoded video data to a user. The display device 722 may be integrated with the destination device 720 or may be external to the destination device 720 and configurable to interface with an external display device.

[0159] The video encoder 714 and the video decoder 724 may operate in accordance with a video compression standard, such as the High Efficiency Video Coding (HEVC) standard, the Versatile Video Coding (VVM) standard, and other current and / or future standards.

[0160] 8 is a block diagram illustrating an example of a video encoder 800, which may be the video encoder 714 in the system 700 shown in FIG. 7. The video encoder 800 is configurable to perform any one or all of the techniques of this disclosure. In the example of FIG. 8, the video encoder 800 contains multiple functional modules. The techniques described in this disclosure may be shared among various components of the video encoder 800. In some examples, a processor could be configurable to perform any or all of the techniques described in this disclosure.

[0161] The functional modules of the video encoder 800 include a division unit 801, a prediction unit 802, which may include a mode selection unit 803, a motion estimation unit 804, a motion compensation unit 805, an intra prediction unit 806, a residual generation unit 807, a transform processing unit 808, a quantization unit 809, an inverse quantization unit 811, a reconstruction unit 812, a buffer 813, and an entropy coding unit 814.

[0162] In other examples, the video encoder 800 may include more, fewer, or different functionality modules. In one example, the prediction unit 802 may include an intra block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode in which at least one reference picture is the picture in which the current video block is located.

[0163] Furthermore, some components, such as the motion estimation unit 804 and the motion compensation unit 805, may be highly integrated, but are depicted separately in the example of FIG. 8 for illustrative purposes.

[0164] The division unit 801 may divide a picture into one or more video blocks. The video encoder 800 and the video decoder 900 may support a variety of video block sizes.

[0165] The mode selection unit 803 may select either intra or inter coding mode, for example based on an error result, and provide the resulting intra or inter coded block to a residual generation unit 807 to generate residual block data and to a reconstruction unit 812 to reconstruct a coded block for use as a reference picture. In some examples, the mode selection unit 803 may select a combined intra and inter prediction (CIIP) mode, where prediction is based on an inter prediction signal and an intra prediction signal. The mode selection unit 803 may also select a resolution (e.g., sub-pixel accuracy or integer pixel accuracy) of a motion vector of a block in case of inter prediction.

[0166] To perform inter prediction on a current video block, motion estimation unit 804 may generate motion information for the current video block by comparing one or more reference frames from buffer 813 with the current video block. Motion compensation unit 805 may determine a prediction video block for the current video block based on the motion information and decoded samples of pictures from buffer 813 other than the picture associated with the current video block.

[0167] Motion estimation unit 804 and motion compensation unit 805 may perform different operations on a current video block depending on whether the current video block is an I slice, a P slice, or a B slice, for example.

[0168] In some examples, motion estimation unit 804 may perform unidirectional prediction on the current video block, and motion estimation unit 804 may search reference pictures in list 0 or list 1 for reference video blocks for the current video block. Motion estimation unit 804 may then generate a reference index indicating a reference picture in list 0 or list 1 that contains the reference video block, and a motion vector indicating a spatial displacement between the current video block and the reference video block. Motion estimation unit 804 may output the reference index, prediction direction index, and motion vector as motion information for the current video block. Motion compensation unit 805 may generate a prediction video block for the current block based on the reference video block indicated by the motion information of the current video block.

[0169] In another example, motion estimation unit 804 may perform bidirectional prediction on the current video block, and motion estimation unit 804 may search reference pictures in list 0 for a reference video block for the current video block and may search reference pictures in list 1 for another video block for the current video block. Motion estimation unit 804 may then generate reference indexes indicating the reference pictures in list 0 and list 1 that contain the reference video blocks, and motion vectors indicating the spatial displacement between the reference video blocks and the current video block. Motion estimation unit 804 may output the reference indexes and the motion vector of the current video block as motion information of the current video block. Motion compensation unit 805 may generate a prediction video block for the current video block based on the reference video blocks indicated by the motion information of the current video block.

[0170] In some examples, the motion estimation unit 804 may output a full set of motion information for the decoder's decoding process. In some examples, the motion estimation unit 804 may not output a full set of motion information for the current picture. Rather, the motion estimation unit 804 may signal the motion information of the current video block with reference to the motion information of another video block. For example, the motion estimation unit 804 may determine that the motion information of the current video block is sufficiently similar to the motion information of a nearby video block.

[0171] In one example, motion estimation unit 804 may indicate a value in a syntax structure associated with the current video block that indicates to video decoder 900 that the current video block has the same motion information as another video block.

[0172] In another example, motion estimation unit 804 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. Video decoder 900 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0173] As mentioned above, the video encoder 800 can predictively signal motion vectors. Two examples of predictive signaling techniques that can be implemented by the video encoder 800 include Advanced Motion Vector Prediction (AMVP) and merge mode signaling.

[0174] The intra prediction unit 806 may perform intra prediction on the current video block. When the intra prediction unit 806 performs intra prediction on the current video block, the intra prediction unit 806 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a prediction video block and various syntax elements.

[0175] Residual generation unit 807 may generate residual data for the current video block by subtracting (e.g., as indicated by a minus sign) the prediction video block(s) of the current video block from the current video block. The residual data for the current video block may include residual video blocks that correspond to different sample components of the samples of the current video block.

[0176] In other examples, such as in skip mode, residual data for the current video block may not exist, and residual generation unit 807 may not perform the subtraction operation.

[0177] Transform processing unit 808 may generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to a residual video block associated with the current video block.

[0178] After the transform processing unit 808 generates a transform coefficient image block associated with the current video block, the quantization unit 809 may quantize the transform coefficient image block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0179] Inverse quantization unit 810 and inverse transform unit 811 may apply inverse quantization and inverse transform, respectively, to the transform coefficient video block to reconstruct a residual video block from the transform coefficient video block. Reconstruction unit 812 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by prediction unit 802 to generate a reconstructed video block associated with the current block for storage in buffer 813.

[0180] After reconstruction unit 812 reconstructs the video blocks, a loop filtering operation may be performed to reduce video blocking artifacts in the video blocks.

[0181] The entropy encoding unit 814 may receive data from other functional modules of the video encoder 800. Once the entropy encoding unit 814 receives the data, the entropy encoding unit 814 may perform one or more entropy encoding operations to generate entropy coded data and output a bitstream including the entropy coded data.

[0182] FIG. 9 is a block diagram illustrating an example of a video decoder 900, which may be video decoder 724 in system 700 shown in FIG.

[0183] The video decoder 900 can be configured to perform any one or all of the techniques of this disclosure. In the example of FIG. 9, the video decoder 900 includes multiple functional modules. The techniques described in this disclosure may be shared among various components of the video decoder 900. In some examples, a processor can be configured to perform any one or all of the techniques described in this disclosure.

[0184] 9, the video decoder 900 includes an entropy decoding unit 901, a motion compensation unit 902, an intra prediction unit 903, an inverse quantization unit 904, an inverse transform unit 905, and a reconstruction unit 906 and a buffer 907. The video decoder 900 may, in some examples, perform a decoding path that is generally the reverse of the encoding path described with respect to the video encoder 800 (FIG. 8).

[0185] The entropy decoding unit 901 may obtain an encoded bitstream. The encoded bitstream may include entropy coded video data (e.g., coded blocks of video data). The entropy decoding unit 901 may decode the entropy coded video data, and from the entropy decoded video data, the motion compensation unit 902 may determine motion information including motion vectors, motion vector precision, reference picture list index, and other motion information. The motion compensation unit 902 may determine such information, for example, by performing AMVP and merge mode.

[0186] The motion compensation unit 902 may generate motion compensated blocks, possibly performing interpolation based on an interpolation filter. An identifier for the interpolation filter to be used with sub-pixel accuracy may be included in the syntax element.

[0187] The motion compensation unit 902 may calculate interpolated values ​​for sub-integer pixels of the reference block using an interpolation filter as used by the video encoder 800 during encoding of the video block. The motion compensation unit 902 may determine the interpolation filter used by the video encoder 800 according to the received syntax information and generate a predictive block using the interpolation filter.

[0188] The motion compensation unit 902 may use a portion of the syntax information to determine the size of the blocks used to encode the frame(s) and / or slice(s) of the encoded video sequence, partition information describing how each macroblock of a picture of the encoded video sequence is divided, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-encoded block, and other information for decoding the encoded video sequence.

[0189] The intra prediction unit 903 may form a prediction block from spatially neighboring blocks, e.g., using an intra prediction mode received in the bitstream. The inverse quantization unit 904 inverse quantizes, i.e., dequantizes, the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 901. The inverse transform unit 905 applies an inverse transform.

[0190] The reconstruction unit 906 may sum the residual blocks with corresponding prediction blocks generated by the motion compensation unit 902 or intra prediction unit 903 to form decoded blocks. If desired, a deblocking filter may be applied to filter the decoded blocks to remove blockiness artifacts. The decoded video blocks are then stored in a buffer 907, which provides reference blocks for subsequent motion compensation / intra prediction and also generates decoded video for presentation on a display device.

[0191] FIG. 10 is a schematic diagram showing an example of a target encoder 1000. The encoder 1000 is suitable for implementing the technique of VVC. The encoder 1000 includes three in-loop filters, namely, a deblocking filter (DF) 1002, a sample adaptive offset (SAO) 1004, and an adaptive loop filter (ALF) 1006. Unlike the DF 1002, which uses a predefined filter, the SAO 1004 and the ALF 1006 utilize the original samples of the current picture to reduce the mean square error between the original samples and the reconstructed samples by adding an offset and by applying a Finite Impulse Response (FIR) filter with coded side information signaling the offset and the filter coefficients, respectively. The ALF 1006 is located at the last processing stage of each picture and can be considered as a tool that tries to catch and correct the artifacts created by the previous stage.

[0192] The encoder 1000 further includes an intra prediction component 1008 and a motion estimation / compensation (ME / MC) component 1010 configurable to receive an input video. The intra prediction component 1008 is configured to perform intra prediction, while the ME / MC component 1010 is configured to utilize reference pictures from a reference picture buffer 1012 to perform inter prediction. Residual blocks from the inter or intra prediction are provided to a transform component 1014 and a quantization component 1016 to generate quantized residual transform coefficients, which are provided to an entropy coding component 1018. The entropy coding component 1018 entropy codes the prediction results and the quantized transform coefficients and transmits the same towards a video decoder (not coded). The quantization component output from the quantization component 1016 can be provided to an inverse quantization component 1020, an inverse transform component 1022, and a reconstruction (REC) component 1024. The REC component 1024 can output images to the DF 1002 , SAO 1004 , and ALF 1006 to perform filtering before the images are stored in the reference picture buffer 1012 .

[0193] Next, in some instances, a list of preferred solutions is provided.

[0194] The following solutions are illustrative of the techniques discussed herein.

[0195] 1. A visual media processing method (e.g., method 600 shown in FIG. 6 ) performing (602) a conversion between visual media data and a file storing information corresponding to the visual media data in accordance with a video file format; where the video file format includes a decoder configuration record capable of configuring information for content selection, where the decoder configuration record includes one or more fields: requested decoded picture buffer size, maximum picture output reordering, maximum delay, gradual decoding update picture enable flag, clean random access picture enable flag, reference picture resampling enable flag, spatial resolution change according to coded video sequence, sub-picture partition enable flag, maximum number of sub-pictures per picture, wavefront parallelism enable flag, tile partition enable flag, maximum number of slices per picture, slice partition enable flag, rectangular slice enable flag, raster scan slice enable flag, maximum number of slices per picture.

[0196] 2. A visual media processing method comprising: converting between visual media data and a file storing information corresponding to the visual media data according to a video file format that complies with rules; the rules specifying that a field indicating the number of temporal layers is included in the decoder configuration record depending on whether profile level tier information for the visual media data is included in the file; and the rules further specifying that the field is included prior to the profile level tier information.

[0197] 3. The method according to solution 2, wherein the rules further define an order in which the profile level tier information occurs within the video file format with respect to one or more additional information fields.

[0198] 4. In the method described in Solution 2, the one or more additional information fields include a chroma format indication field, a bit depth field, a field indicating the number of temporal layers or a field indicating whether a constant frame rate is used for the visual media data.

[0199] 5. In the method described in solution 2, the one or more additional information fields include a reserved bit field.

[0200] 6. In the method according to any one of solutions 2 to 5, the rule specifies that the profile tier level information is included as the last field of the decoder configuration record.

[0201] 7. A method according to any one of solutions 1 to 6, wherein the converting includes generating a bitstream representation of the visual media data and storing the bitstream representation in a file according to a format rule.

[0202] 8. The method according to any one of solutions 1 to 6, wherein the conversion includes parsing the file according to formatting rules to recover the visual media data.

[0203] 9. A video decoding device including a processor capable of implementing the method described in one or more of solutions 1 to 8.

[0204] 10. A video encoding device including a processor configurable to implement the methods described in one or more of solutions 1 to 8.

[0205] 11. A computer program product having computer code stored thereon, which code, when executed by a processor, causes the processor to implement a method according to any of solutions 1 to 8.

[0206] 12. A computer-readable medium having recorded thereon a bitstream representation conforming to a file format generated according to any one of solutions 1 to 8.

[0207] 13. A method, apparatus or system as described herein. In the solutions described herein, an encoder can comply with the format rules by generating a coded representation that complies with the format rules. In the solutions described herein, a decoder can use the format rules to parse syntax elements in the coded representation with knowledge of the presence or absence of syntax elements that comply with the format rules to generate decoded video.

[0208] In this specification, the term "video processing" may refer to video encoding, video decoding, video compression or video decompression. For example, a video compression algorithm may be applied during conversion from a pixel representation of a video to a corresponding bitstream representation or vice versa. The bitstream representation of a current video block may correspond to bits that are at the same position in the bitstream or spread across different locations, for example, as defined by the syntax. For example, a macroblock may be coded in terms of transformed and coded error residual values, and using bits of headers and other fields in the bitstream. Furthermore, during conversion, the decoder may parse the bitstream with the knowledge that some fields may or may not be present based on the decision, as described in the above solution. Similarly, the encoder may determine that a particular syntax field is or is not included, and generate the coded representation accordingly by including or excluding the syntax field in the coded representation.

[0209] The disclosed and other solutions, examples, embodiments, modules, and functional operations described herein can be implemented in digital electronic circuitry, including structures disclosed herein and their structural equivalents, or in computer software, firmware, or hardware, or in one or more combinations thereof. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by or for controlling the operation of a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition that provides a machine-readable propagated signal, or one or more combinations thereof. The term "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, an apparatus can include code that establishes an execution environment for the computer program, such as processor firmware, a protocol stack, a database management system, code that constitutes an operating system, or one or more combinations thereof. A propagated signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to an appropriate receiving device.

[0210] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including computational or interpretive languages, and can be deployed in any form, such as as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in several coordinated files (modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer, or on several computers located at one site or distributed across several sites and interconnected by a communications network.

[0211] The processes and logic flows described herein may be computed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and an apparatus may be implemented as, an application specific integrated circuit, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0212] Processors suitable for executing computer programs include, by way of example, both general and specialized microprocessors, and any one or more processors of any kind of digital computer. Typically, a processor receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, transferring data, or both, or for operationally computing. However, a computer need not have such devices. Computer-readable media suitable for storing computer-readable program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magnetic optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0213] Although this patent document contains many specificities, these should not be construed as limitations on the scope of any subject matter or claims, but rather as descriptions of features that may be inherent to certain embodiments of a particular technology. Certain features described in this patent document in the context of separate embodiments may also be implemented in combination in one embodiment. Conversely, various features described in the context of one embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a particular combination, and may even be initialized as such, one or more features from a claimed combination may in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0214] Similarly, although operations are depicted in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown, or in the order in which they are performed, or that all of the operations shown be performed, to achieve desirable results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0215] Only a few implementations and examples have been described; other implementations, extensions and variations can be made based on what is described and illustrated in this patent document.

[0216] A first component is directly bonded to a second component when there are no intervening components, other than lines, traces, and other media, between the first and second components. A first component is indirectly bonded to a second component when there are intervening components, other than lines, traces, or other media, between the first and second components. The term "bonded" and variations thereof include both direct and indirect bonding. Use of the term "about" refers to a range that includes ±10% of the number that follows unless otherwise specified.

[0217] Although several embodiments are provided in this disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the disclosure. The examples are considered to be illustrative and not restrictive, and the intention is not to be limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not present in the implementation.

[0218] Additionally, techniques, systems, subsystems, and methods described and illustrated as discrete or separate in various embodiments may be combined or integrated with other systems, modules, technologies, or methods without departing from the scope of the present disclosure. If not shown or discussed as combined, they may be directly connected or indirectly coupled or in communication through some interface, device, or intermediate component, whether electrical, mechanical, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by those skilled in the art and may be made without departing from the spirit and scope disclosed herein.

Claims

1. performing a conversion between a visual media data file and a generic video coding (VVC) decoder configuration record, the visual media data including a plurality of pictures coded into one or more sub-layers, the VVC decoder configuration record including a number of the one or more sub-layers and one or more VVC profile tier level (PTL) records for the one or more sub-layers based on the number of the one or more sub-layers; the VVC decoder configuration record includes a constant frame rate syntax element, a chroma format identification code syntax element, and a bit depth minus 8 syntax element, and the VVC PTL record is positioned in the VVC decoder configuration record after the constant frame rate syntax element, the chroma format identification code syntax element, and the bit depth minus 8 syntax element. Video data processing method.

2. The conversion is receiving the VVC decoder configuration record and a media file including a plurality of pictures coded in one or more sub-layers; parsing the VVC decoder configuration record to obtain a number of the one or more sub-layers and the one or more VVC PTL records for the one or more sub-layers based on the number of the one or more sub-layers; and decoding the one or more sub-layers based on the VVC PTL record. The method of claim 1.

3. The conversion is encoding the plurality of pictures into the one or more sub-layers in a visual media file; determining a number of the one or more sub-layers; and encoding the VVC decoder configuration record into a media file, the VVC decoder configuration record including a number of the one or more sub-layers and the one or more VVC PTL records for the one or more sub-layers; storing the visual media file in a memory. The method of claim 1.

4. the number of the one or more sub-layers is signaled in the VVC Decoder Configuration Record prior to the VVC PTL Record; The method according to any one of claims 1 to 3.

5. the number of the one or more sub-layers is located in the VVC decoder configuration record before a constant frame rate syntax element, a chroma format identification code syntax element, and a bit depth minus 8 syntax element; 5. The method according to any one of claims 1 to 4.

6. The VVC Decoder Configuration Record further includes a reserved bit located after a bit depth minus 8 syntax element, and the VVC PTL Record is located after the reserved bit.

6. The method according to any one of claims 1 to 5.

7. 1. A video data processing apparatus comprising a processor and a non-transitory memory having instructions thereon, the instructions, when executed by the processor, causing the processor to: converting between a visual media data file and a generic video coding (VVC) decoder configuration record, the visual media data including a plurality of pictures coded into one or more sub-layers, the VVC decoder configuration record comprising one or more VVC profile tier level (PTL) records for the one or more sub-layers based on a number of the one or more sub-layers; the VVC decoder configuration record includes a constant frame rate syntax element, a chroma format identification code syntax element, and a bit depth minus 8 syntax element, and the VVC PTL record is positioned in the VVC decoder configuration record after the constant frame rate syntax element, the chroma format identification code syntax element, and the bit depth minus 8 syntax element. Video data processing device.

8. 1. A non-transitory computer-readable medium containing a computer program product for use by a video coding device, the computer program product comprising computer-executable instructions stored on the non-transitory computer-readable medium that, when executed by a processor, causes the video coding device to: the video coding device performs conversion between a generic video coding (VVC) decoder configuration record and visual media data including a plurality of pictures coded into one or more sub-layers and a visual media data file, the VVC decoder configuration record including a number of the one or more sub-layers and one or more VVC profile tier level (PTL) records for the one or more sub-layers based on the number of the one or more sub-layers; the VVC decoder configuration record includes a constant frame rate syntax element, a chroma format identification code syntax element, and a bit depth minus 8 syntax element, and the VVC PTL record is positioned in the VVC decoder configuration record after the constant frame rate syntax element, the chroma format identification code syntax element, and the bit depth minus 8 syntax element. Non-transitory computer-readable medium.

9. 1. A method for storing visual media data files, the method comprising: generating a visual media data file including a generic video coding (VVC) decoder configuration record and a plurality of pictures coding one or more sub-layers; storing the visual media data file on a non-transitory computer readable recording medium, wherein the VVC decoder configuration record includes a number of the one or more sub-layers and one or more VVC Profile Tier Level (PTL) records for the one or more sub-layers based on the number of the one or more sub-layers; the VVC decoder configuration record includes a constant frame rate syntax element, a chroma format identification code syntax element, and a bit depth minus 8 syntax element, and the VVC PTL record is positioned in the VVC decoder configuration record after the constant frame rate syntax element, the chroma format identification code syntax element, and the bit depth minus 8 syntax element. method.

Images