Profile, tier, level and general constraint indication in coded video

Abstract

Systems, methods, and apparatuses for encoding or decoding a file format that stores one or more images are described. An example method includes performing a conversion between a visual media file and a bitstream of visual media data according to a format rule, wherein the bitstream includes one or more output layer sets and one or more parameter sets, the one or more parameter sets including one or more tier level syntax structures, wherein at least one of the tier level syntax structures includes a general constraint information syntax structure, wherein the format rule specifies that a syntax element is included in a configuration record in the visual media file, and wherein the syntax element indicates a tier, layer, or level to which an output layer set identified by an output layer set index indicated in the configuration record conforms.

Description

[0001] Cross-references to related applications

[0002] This application is made to promptly claim priority and interest in U.S. Provisional Patent Application No. 63 / 079,892, filed September 17, 2020, in accordance with applicable patent law and / or the rules of the Paris Convention. For all purposes provided by law, the entire disclosure of the foregoing application is incorporated herein by reference as part of the disclosure of this application. Technical Field

[0003] This patent document relates to the generation, storage, and consumption of digital audio and video media information in file formats. Background Technology

[0004] Digital video accounts for the largest share of bandwidth usage in the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth demand for digital video is expected to continue to grow. Summary of the Invention

[0005] This document discloses techniques for encoding and decoding representations of video or images that can be used by video encoders and decoders to process video or images according to file formats.

[0006] In one example aspect, a video processing method is disclosed. The method includes: performing a conversion between a visual media file and a bitstream of visual media data according to format rules, wherein the bitstream includes one or more output layer sets and one or more parameter sets, the one or more parameter sets including one or more level-level syntax structures, wherein at least one of the level-level syntax structures includes a general constraint information syntax structure, wherein the format rules specify syntax elements included in a configuration record in the visual media file, and wherein the syntax elements indicate the level, layer, or grade to which the output layer set identified by the output layer set index indicated in the configuration record conforms.

[0007] In another example, a different video processing method is disclosed. This method includes performing a conversion between a visual media file and a bitstream of visual media data according to a format rule, wherein the format rule specifies the characteristics of syntax elements in the visual media file, wherein the syntax elements have values ​​indicating the number of bytes used to indicate constraint information associated with the bitstream.

[0008] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a visual media file and a bitstream of visual media data according to a format rule, wherein the format rule specifies a property of a syntax element in the visual media file, and wherein the format rule specifies that a syntax element having a value indicative of a level identification is coded using octets in either or both of a subpicture common group box or a subpicture multiple group box.

[0009] In another example aspect, another video processing method is disclosed. The method includes performing a conversion between a visual media file and a bitstream of visual media data according to a format rule, wherein the format rule specifies a property of a syntax element in the visual media file, and wherein the format rule specifies that a syntax element having a value indicative of a level identification is coded using octets in either or both of a subpicture common group box or a subpicture multiple group box.

[0010] In yet another example aspect, a video encoder apparatus is disclosed. The video encoder includes a processor configured to implement a method described above.

[0011] In yet another example aspect, a video decoder apparatus is disclosed. The video decoder includes a processor configured to implement a method described above.

[0012] In yet another example aspect, a computer readable medium having code stored thereon is disclosed. The code embodies one of the methods described herein in the form of processor-executable code.

[0013] In yet another example aspect, a computer readable medium having a bitstream stored thereon is disclosed. The bitstream is generated or processed using the methods presented in this document.

[0014] These and other features are presented throughout this document. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a block diagram of an example video processing system.

[0016] Figure 2 is a block diagram of a video processing apparatus.

[0017] Figure 3 is a flowchart of an example method of video processing.

[0018] Figure 4 is a block diagram illustrating a video coding system according to some embodiments of the present disclosure.

[0019] Figure 5 is a block diagram illustrating an encoder according to some embodiments of the present disclosure.

[0020] Figure 6is a block diagram illustrating a decoder according to some embodiments of the disclosure.

[0021] Figure 7 An example of an encoder block diagram is shown.

[0022] Figures 8 to 10 is a flowchart of, for example, a video processing method. DETAILED DESCRIPTION

[0023] For ease of understanding, section headings have been used in this document, and the teachings and embodiments disclosed in each section are not intended to be limited only to that section. Furthermore, the use of H.266 terminology in some descriptions is only for ease of understanding and is not intended to limit the scope of the disclosed technology. Likewise, the technology described herein is also applicable to other video codec protocols and designs. In this document, editorial changes to text are shown by opening and closing double brackets (e.g., [[]]) indicating that the text between the double brackets is cancelled, and by bold italic text indicating added text, with respect to the current draft of the VVC specification or the ISOBMFF file format specification.

[0024] 1. Brief discussion

[0025] This document is related to video file formats. In particular, it is concerned with the signaling of decoder configuration information and subpicture entity groups in media files carrying Versatile Video Coding (VVC) video bitstreams based on the ISO Base Media File Format (ISOBMFF). These ideas can be applied individually or in various combinations for video bitstreams coded by any codec (e.g., the VVC standard), and for any video file format (e.g., the VVC video file format that is being developed).

[0026] 2. Abbreviations

[0027] ACT Adaptive Color Transform

[0028] ALF Adaptive Loop Filter

[0029] AMVR Adaptive Motion Vector Resolution

[0030] APS Adaptation Parameter Set

[0031] AU Access Unit

[0032] AUD Access Unit Delimiter

[0033] AVC Advanced Video Coding (Rec. ITU-T H.264 | ISO / IEC 14496-10)

[0034] B Bi-prediction

[0035] BCW bi-prediction with cu-level weights

[0036] BDOF bi-directional optical flow

[0037] BDPCM block-based delta pulse code modulation

[0038] BP buffering period

[0039] CABAC context-based adaptive binary arithmetic coding

[0040] CB coded block

[0041] CBR constant bit rate

[0042] CCALF cross-component adaptive loop filter

[0043] CPB coded picture buffer

[0044] CRA clean random access

[0045] CRC cyclic redundancy check

[0046] CTB coded tree block

[0047] CTU coded tree unit

[0048] CU coding unit

[0049] CVS coded video sequence

[0050] DPB decoded picture buffer

[0051] DCI decoding capability information

[0052] DRAP dependent random access point

[0053] DU decoding unit

[0054] DUI decoding unit information

[0055] EG exponential golomb

[0056] Egk k-th order exponential golomb

[0057] EOB end of bitstream

[0058] EOS end of sequence

[0059] FD filler data

[0060] FIFO first in, first out

[0061] FL fixed length

[0062] GBR green, blue, red

[0063] GCI general constraint information

[0064] CGR clean gradual refresh

[0065] GPM geometric partition mode

[0066] HEVC high efficiency video coding (Rec. ITU-T H.265 | ISO / IEC 23008-2)

[0067] HRD hypothetical reference decoder

[0068] HSS hypothetical stream scheduler

[0069] I intra

[0070] IBC intra block copy

[0071] IDR instantaneous decoding refresh

[0072] ILRP inter-layer reference picture

[0073] IRAP intra random access point

[0074] LFNST low-frequency non-separable transform

[0075] LPS least probable symbol

[0076] LSB least significant bit

[0077] LTRP long-term reference picture

[0078] MCS luma mapping with chroma scaling

[0079] MIP matrix-based intra prediction

[0080] MPS most probable symbol

[0081] MSB most significant bit

[0082] MTS multiple transform selection

[0083] MVP motion vector prediction

[0084] NAL network abstraction layer

[0085] OLS output layer set

[0086] OP operation point

[0087] OPI operation point information

[0088] P predictive

[0089] PH picture header

[0090] POC picture order count

[0091] PPS picture parameter set

[0092] PROF prediction refinement with optical flow

[0093] PT picture timing

[0094] PU picture unit

[0095] QP quantization parameter

[0096] RADL random access decodable leading (picture)

[0097] RASL random access skipped leading (picture)

[0098] RBSP raw byte sequence payload

[0099] RGB red, green, blue

[0100] RPL reference picture list

[0101] SAO sample adaptive offset

[0102] SAR sample aspect ratio

[0103] SEI supplemental enhancement information

[0104] SH slice header

[0105] SLI subpicture level information

[0106] SODB string of data bits

[0107] SPS sequence parameter set

[0108] STRP short-term reference picture

[0109] STSA stepping time duration sublayer access

[0110] TR truncated rice code

[0111] VBR variable bit rate

[0112] VCL video coding layer

[0113] VPS video parameter set

[0114] VSEI versatile supplemental enhancement information (Rec. ITU-T H.274 ISO / IEC 23002-7)

[0115] VUI video usability information

[0116] VVC Versatile Video Coding (Rec. ITU-T H.266 | ISO / IEC 23090-3)

[0117] 3. Introduction to Video Coding

[0118] 3.1. Video Coding Standards

[0119] Video coding standards have evolved mainly through the development of the well-known ITU-T and ISO / IEC standards. ITU-T produced H.261 and H.263, ISO / IEC produced MPEG-1 and MPEG-4 Visual, and the two organizations jointly produced the H.262 / MPEG-2 Video and H.264 / MPEG-4 Advanced Video Coding (AVC) standards and the H.265 / HEVC standard. Since H.262, the video coding standards are based on the hybrid video coding structure wherein temporal prediction plus transform coding are used. To explore future video coding technologies beyond HEVC, the Joint Video Exploration Team (JVET) was founded by VCEG and MPEG jointly in 2015. Since then, many new methods have been adopted by the JVET and brought into the reference software named Joint Exploration Model (JEM). The JVET was later renamed to Joint Video Team (JVT) when the project of Versatile Video Coding (VVC) was officially started. VVC is the new coding standard targeting at 50% bitrate reduction compared to HEVC, which has been finalized by the JVET at its 19th meeting, which ended on July 1, 2020.

[0120] The Versatile Video Coding (VVC) standard (ITU-T H.266 | ISO / IEC 23090-3) and the associated Versatile Supplemental Enhancement Information (VSEI) standard (ITU-T H.274 | ISO / IEC 23002-7) are designed for the broadest range of applications, including traditional uses such as television broadcast, video conferencing or playback from storage media, as well as newer and more advanced usage scenarios such as adaptive bitrate streaming, video region extraction, composition and merging of content from multiple coded video bitstreams, multi-view video, scalable layered coding and viewport-adaptive 360° immersive media.

[0121] 3.2. File Format Standards

[0122] Media streaming applications are typically based on IP, TCP and HTTP transport methods and typically rely on file formats such as ISO Base Media File Format (ISOBMFF). One such streaming system is dynamic adaptive streaming over HTTP (DASH). For video formats using ISOBMFF and DASH, specific to video format file format specifications such as AVC file format and HEVC file format are needed to encapsulate video content in ISOBMFF tracks and in DASH representations and segments. Important information about the video bitstream, e.g. profiles, tiers and levels and many other information, will need to be exposed as file format level metadata and / or DASH media presentation description (MPD) for content selection purposes, e.g. selection of appropriate media segments both for initialization at the beginning of a streaming session and for stream adaptation during a streaming session.

[0123] Similarly, for image formats using ISOBMFF, specific to image format file format specifications such as AVC image file format and HEVC image file format are needed.

[0124] VVC video file format, a file format for storing VVC video content based on ISOBMFF, is currently being developed by MPEG.

[0125] VVC image file format, a file format for storing image content using VVC codec based on ISOBMFF, is currently being developed by MPEG.

[0126] 3.3. Some details of VVC video file format

[0127] 3.3.1. Decoder configuration information

[0128] 3.3.1.1. VVC decoder configuration record

[0129] 3.3.1.1.1. Definition

[0130] This subclause specifies decoder configuration information for ISO / IEC 23090-3 video content.

[0131] This record contains the size of the length field used in each sample to indicate the length of the NAL units it contains and parameter sets if stored in that sample entry. This record is externally prescribed (its size is provided by the structure containing it).

[0132] This record contains a version field. The version of this specification defines version 1 of this record. Incompatible changes to the record will be indicated by a change in the version number. If the version number cannot be recognized, the reader shall not attempt to decode this record or the stream to which it applies.

[0133] Compatible extensions to this record shall extend it and shall not change the configuration version code. The reader shall be prepared to ignore unrecognized data that extends beyond the data definitions they understand.

[0134] A VvcPtlRecord shall be present in the decoder configuration record when the track itself contains a VVC bitstream or contains a VVC bitstream by resolving the'subp' track reference. If the ptl_present_flag in the decoder configuration record of the track is equal to zero, the track shall have an 'oref' track reference.

[0135] The values of the VvcPTLRecord, chroma_format_idc, and bit_depth_minus8 syntax elements shall be valid for all parameter sets (referred to as "all parameter sets" in the sentence below this paragraph) that are activated when decoding the stream described by this record. Specifically, the following restrictions apply:

[0136] The tier indication general_profile_idc shall indicate the tier to which the stream associated with this configuration record conforms.

[0137] NOTE 1: If SPS are marked with different tiers, the stream can need to be inspected to determine which tier, if any, the entire stream conforms to. If no inspection of the entire stream is done, or the inspection shows that no tier is conformed to by the entire stream, the entire stream shall be split into two or more sub-streams with separate configuration records in which these rules can be satisfied.

[0138] The level indication general_tier_flag shall indicate the level that is equal to or greater than the highest layer indicated in all parameter sets.

[0139] Each bit in general_constraint_info can only be set if all parameter sets have that bit set.

[0140] The level indication general_level_idc shall indicate the capability level that is equal to or greater than the highest level indicated for the highest layer in all parameter sets.

[0141] The following constraints apply to chroma_format_idc:

[0142] - If the value of sps_chroma_format_idc as defined in ISO / IEC 23090-3 is the same in all SPS referred to by the NAL units of the track, chroma_format_idc shall be equal to sps_chroma_format_idc.

[0143] - Otherwise, if ptl_present_flag is equal to 1, chroma_format_idc shall be equal to vps_ols_dpb_chroma_format[ output_layer_set_idx ], as defined in ISO / IEC 23090-3.

[0144] - Otherwise, chroma_format_idc shall not be present.

[0145] The following constraint applies to bit_depth_minus8:

[0146] - If the value of sps_bitdepth_minus8 as defined in ISO / IEC 23090-3 is the same in all SPS referred to by the NAL units of the track, bit_depth_minus8 shall be equal to sps_bitdepth_minus8.

[0147] - Otherwise, if ptl_present_flag is equal to 1, bit_depth_minus8 shall be equal to vps_ols_dpb_bitdepth_minus8[ output_layer_set_idx ], as defined in ISO / IEC 23090-3.

[0148] - Otherwise, bit_depth_minus8 shall not be present.

[0149] Explicit indication about chroma format and bit depth is provided in the VVC decoder configuration record, along with other important format information used by VVC video elementary streams. Two different VVC sample entries are also needed if two sequences differ in their color space indication in the VUI information.

[0150] There are array groups to carry initialization NAL units. NAL unit types are restricted to indicate DCI, VPS, SPS, PPS, prefix APS, and prefix SEI NAL units. NAL unit types reserved in ISO / IEC 23090-3 and this specification can be defined in the future, and the reader should ignore arrays with reserved or disallowed NAL unit type values.

[0151] NOTE 2: The design of this 'tolerance' behavior is to not provoke errors, allowing backward compatible extensions of these arrays in future specifications.

[0152] NOTE 3: The NAL units carried in the sample entry are included immediately after the AUD and OPIN NAL units (if any) in the access unit reconstructed from the first sample referring to this sample entry or at the beginning of the access unit.

[0153] The order of the suggested arrays is DCI, VPS, SPS, PPS, prefix APS, prefix SEI.

[0154] 3.3.1.1.2. Syntax

[0155]

[0156]

[0157]

[0158] 3.3.1.1.3. Semantics

[0159] general_profile_idc, general_tier_flag, general_sub_profile_idc, general_constraint_info(), general_level_idc, ptl_multilayer_enabled_flag, ptl_frame_only_constraint_flag, sublayer_level_present, and sublayer_level_idc[i] contain matching values for the bits in the fields general_profile_idc, general_tier_flag, general_sub_profile_idc, general_constraint_info(), general_level_idc, ptl_multilayer_enabled_flag, ptl_frame_only_constraint_flag, sublayer_level_present, and sublayer_level_idc[i] as defined for the stream to which this configuration record applies in ISO / IEC 23090-3.

[0160] avgFrameRate gives the average frame rate of the stream to which this configuration record applies in frames per (256 seconds). The value 0 indicates that the average frame rate is not specified.

[0161] constantFrameRate equal to 1 indicates that the stream to which this configuration record applies has a constant frame rate. Value 2 indicates that the representations of each temporal layer in that stream are constant frame rate. Value 0 indicates that the stream can or can not be constant frame rate.

[0162] numTemporalLayers greater than 1 indicates that the track to which this configuration record applies is scalable in the temporal domain and contains a number of temporal layers (also referred to as temporal sub-layers or sub-layers in ISO / IEC 23090-3) equal to numTemporalLayers. Value 1 indicates that the track to which this configuration record applies is not scalable in the temporal domain. Value 0 indicates that it is unknown whether the track to which this configuration record applies is scalable in the temporal domain.

[0163] lengthSizeMinusOne plus 1 indicates the length (in bytes) of the NALUnitLength field in VVC video stream samples in the stream to which this configuration record applies. For example, a size of one byte is indicated with a value of 0. The value of this field shall be one of 0, 1, or 3, which correspond to lengths coded with 1, 2, or 4 bytes, respectively.

[0164] ptl_present_flag equal to 1 specifies that the track contains a VVC bitstream corresponding to a particular output layer set. ptl_present_flag equal to 0 specifies that the track can not contain a VVC bitstream corresponding to a particular output layer set, but can contain one or more individual layers that do not form an output layer set or individual sub-layers that do not include a sub-layer with Temporalld equal to 0.

[0165] num_sub_profiles defines the number of sub-profiles indicated in the decoder configuration record.

[0166] track_ptl specifies the profile, tier, and level of the output layer set represented by the VVC bitstream contained in the track.

[0167] output_layer_set_idx specifies the output layer set index of the output layer set represented by the VVC bitstream contained in the track. The value of output_layer_set_idx can be used as the value of the TargetOlsIdx variable provided by an external component to a VVC decoder as specified in ISO / IEC 23090-3 for decoding the bitstream contained in the track.

[0168] chroma_format_present_flag equal to 0 specifies that chroma_format_idc is not present. chroma_format_present_flag equal to 1 specifies that chroma_format_idc is present.

[0169] chroma_format_idc indicates the chroma format applied to this track. The following constraint applies to chroma_format_idc:

[0170] - If the value of sps_chroma_format_idc as defined in ISO / IEC 23090-3 is the same in all SPS referred to by the NAL units of the track, chroma_format_idc shall be equal to sps_chroma_format_idc.

[0171] - Otherwise, if ptl_present_flag is equal to 1, chroma_format_idc shall be equal to vps_ols_dpb_chroma_format[ output_layer_set_idx ], as defined in ISO / IEC 23090-3.

[0172] - Otherwise, chroma_format_idc shall not be present.

[0173] bit_depth_present_flag equal to 0 specifies that bit_depth_minus8 is not present. bit_depth_present_flag equal to 1 specifies that bit_depth_minus8 is present.

[0174] bit_depth_minus8 indicates the bit depth applied to this track. The following constraint applies to bit_depth_minus8:

[0175] - If the value of sps_bitdepth_minus8 as defined in ISO / IEC 23090-3 is the same in all SPS referred to by the NAL units of the track, bit_depth_minus8 shall be equal to sps_bitdepth_minus8.

[0176] - Otherwise, if ptl_present_flag is equal to 1, bit_depth_minus8 shall be equal to vps_ols_dpb_bitdepth_minus8[ output_layer_set_idx ], as defined in ISO / IEC 23090-3.

[0177] - Otherwise, bit_depth_minus8 shall not be present.

[0178] numArrays indicates the number of arrays of NAL units of the indicated type.

[0179] When array_completeness is equal to 1, it indicates that all NAL units of the given type are in the following array, none in the stream; when array_completeness is equal to 0, it indicates that additional NAL units of the indicated type can be in the stream; the default value and allowed values are constrained by the example entry name.

[0180] NAL_unit_type indicates the type of NAL units in the following array (which shall all be of this type); it takes values defined in ISO / IEC 23090-2; it is constrained to take one of the values indicating a DCI, VPS, SPS, PPS, APS, prefix SEI or suffix SEI NAL unit.

[0181] numNalus indicates the number of NAL units of the indicated type included in the configuration record of the stream for which this configuration record applies. The SEI array shall only contain 'declarative' SEI messages, i.e. those providing information about the whole stream. An example of such SEI is the user data SEI.

[0182] nalUnitLength indicates the length of the NAL unit in bytes.

[0183] nalUnit contains a DCI, VPS, SPS, PPS, APS or declarative SEI NAL unit as specified in ISO / IEC 23090-3.

[0184] 3.3.2. Subpicture entity group

[0185] 3.3.2.1. Overview

[0186] The subpicture entity group is defined to provide level information indicating the level of conformance of a bitstream merged from multiple VVC subpicture tracks.

[0187] NOTE: The VVC base track provides another mechanism for merging VVC subpicture tracks.

[0188] The implicit reconstruction process requires modification of parameter sets. The subpicture entity group gives guidance to simplify the generation of parameter sets for reconstructing a bitstream.

[0189] When the coded sub-pictures within the group to be jointly decoded are interchangeable, i.e. the player selects several active tracks from one group of sub-pictures with the same level contribution in terms of samples, SubpicCommonGroupBox indicates the combination rule when jointly decoded and the level_idc of the resulting combination.

[0190] When there are coded sub-pictures with different properties (e.g. different resolutions) selected to be jointly decoded, SubpicMultipleGroupsBox indicates the combination rule when jointly decoded and the level_idc of the resulting combination.

[0191] All entity_id values included in a sub-picture entity group shall identify VVC sub-picture tracks. When present, SubpicCommonGroupBox and SubpicMultipleGroupsBox shall be included in the GroupsListBox in the movie-level MetaBox and shall not be included in the file-level or track-level MetaBox.

[0192] 3.3.2.2. Syntax of sub-picture common group box

[0193]

[0194] 3.3.2.3. Semantics of sub-picture common group box

[0195] level_idc specifies the level to which any selection of num_active_tracks entities in the entity group conforms.

[0196] num_active_tracks specifies the number of tracks for which the level_idc value is provided.

[0197] 3.3.2.4. Syntax of sub-picture multiple groups box

[0198]

[0199] 3.3.2.5. Semantics

[0200] level_idc specifies the level to which the combination of any num_active_tracks[i] tracks in the sub-group with ID equal to i for all i in the range 0 to num_subgroup_ids - 1, inclusive, conforms.

[0201] num_subgroup_ids specifies the number of separate subgroups, each identified by the same value of track_subgroup_id[i]. Different subgroups are identified by different values of track_subgroup_id[i].

[0202] track_subgroup_id[i] specifies the subgroup ID of the i-th track in this entity group. The subgroup ID value ranges from 0 to num_subgroup_ids - 1, inclusive.

[0203] num_active_tracks[i] specifies the number of tracks in the subgroup with ID equal to i as recorded in level_idc.

[0204] 4. Examples of technical problems solved by the disclosed technical solutions

[0205] The latest design of VVC video file format on signaling of decoder configuration information and subpicture entity group information has the following problems:

[0206] 1) It is specified that general_profile_idc shall indicate the profile to which the stream associated with this configuration record conforms. However, the stream can correspond to multiple output layer sets, so this semantics can allow signaling of an erroneous value of general_profile_idc in the configuration record.

[0207] 2) It is specified that general_tier_flag shall indicate the tier equal to or greater than the highest layer indicated in all parameter sets. However, there can be profile_tier_level() structures signaled in parameter sets and applied to OLSs not within the scope of the current configuration record, so this semantics can allow signaling of an erroneous value of this field in the configuration record. In addition, there can be profile_tier_level() structures signaled in parameter sets and not referenced, and the VPS can include PTL structures applied to OLSs not within the scope of the current configuration record.

[0208] 3) It is specified that each bit in general_constraint_info can only be set if all parameter sets have the bit set. However, there can be profile_tier_level() structures signaled in parameter sets and applied to OLSs not within the scope of the current configuration record, so this semantics can allow signaling of an erroneous value of this field in the configuration record.

[0209] 4) The general_level_idc shall indicate a capability level equal to or greater than the highest level indicated for the highest layer in all parameter sets. However, there can be profile_tier_level() structures signaled in parameter sets and applied to OLSs not within the scope of the current configuration record. In addition, the highest level of the highest layer can be lower than the highest level of the lowest layer, and the level determines the maximum picture width, height, etc., which are essential to determine the required decoding capability. Therefore, this semantics can allow signaling of an erroneous value for this field in the configuration record.

[0210] 5) In the syntax and semantics of the VvcPTLRecord() syntax structure, the issues related to the fields num_bytes_constraint_info and general_constraint_info are as follows:

[0211] a. The field num_bytes_constraint_info is coded using 8 bits. However, the maximum number of bits in the general_constraint_info() syntax structure defined in the VVC specification is 336 bits, i.e., 42 bytes, so 6 bits are sufficient.

[0212] b. In addition, the semantics of the field num_bytes_constraint_info are missing.

[0213] c. The condition for the field general_constraint_info is “if(num_bytes_constraint_info > 0)”. However, in the profile_tier_level() syntax structure defined in the VVC specification, the general_constraint_info() syntax structure exists whenever the tier, the layer, and the level exist, and the length of the general_constraint_info() syntax structure is one byte, not zero bytes, even if the first syntax element gci_present_flag in the general_constraint_info() syntax structure is equal to 0. Therefore, this condition should be changed to “if(num_bytes_constraint_info > 1)”, i.e., when the gci_present_flag of the general_constraint_info() syntax structure is equal to 0, the field general_constraint_info should not be signaled.

[0214] d.The field general_constraint_info is coded using (8*num_bytes_constraint_info - 2) bits. However, the length of general_constraint_info(), i.e., the general_constraint_info() syntax structure defined in the VVC specification, is an integer number of bytes.

[0215] 6) In VvcDecoderConfigurationRecord, the field output_layer_set_idx is always signaled when ptl_present_flag is equal to 1, i.e., when the track_ptl field is signaled. However, if the VVC bitstream carried by the VVC track (after parsing the referenced VVC track or VVC subpicture track, if any) is a single-layer bitstream, the value of the OLS index is not typically needed to know, and it can be easily derived that it is the OLS index of the OLS containing only that layer, even if it is useful to know it.

[0216] 7) The NAL_unit_type field in VvcDecoderConfigurationRecord uses 6 bits. However, 5 bits are enough.

[0217] 8) The semantics of ptl_present_flag is specified as follows: ptl_present_flag equal to 1 specifies that the track contains a VVC bitstream corresponding to a particular output layer set. ptl_present_flag equal to 0 specifies that the track can not contain a VVC bitstream corresponding to a particular output layer set, but can contain one or more individual layers that do not form an output layer set or individual sub-layers that do not include a sub-layer with Temporalld equal to 0.

[0218] However, the case where the track contains VVC bitstreams corresponding to multiple output layer sets is not covered.

[0219] 9) The levl_idc field in SubpicCommonGroupBox and SubpicMultipleGroupsBox is coded using 32 bits. However, 8 bits are enough.

[0220] 10) The num_active_tracks field in SubpicCommonGroupBox and the num_subgroup_ids field and num_active_tracks[i] fields in SubpicMultipleGroupsBox are all coded using 32 bits. However, 16 bits are enough for all of them.

[0221] 5. Solution list

[0222] To solve the above problems and other problems, the following method is disclosed. These items should be considered as examples to explain general concepts and should not be interpreted in a narrow way. In addition, these items can be applied individually or combined in any way.

[0223] 1) To solve problem 1, it is specified that the tier indication general_profile_idc shall indicate the tier to which the output layer set identified by output_layer_set_idx in this configuration record conforms.

[0224] 2) To solve problem 2, it is specified that the tier indication general_tier_flag shall indicate one tier that is equal to or greater than the highest tier indicated in all profile_tier_level() syntax structures (in all parameter sets) indicated by the output layer set identified by output_layer_set_idx in this configuration record.

[0225] a. Alternatively, it is specified that the tier indication general_tier_flag shall indicate the highest tier indicated in all profile_tier_level() syntax structures (in all parameter sets) to which the output layer set identified by output_layer_set_idx in this configuration record conforms.

[0226] b. Alternatively, it is specified that the tier indication general_tier_flag shall indicate the highest tier to which the stream associated with this configuration record conforms.

[0227] c. Alternatively, it is specified that the tier indication general_tier_flag shall indicate one tier to which the stream associated with this configuration record conforms.

[0228] 3) To solve problem 3, it is specified that each bit in general_constraint_info shall only be set if it is set in all general_constraints_info() syntax structures (in all parameter sets) of all profile_tier_level() syntax structures to which the output layer set identified by output_layer_set_idx in this configuration record conforms.

[0229] 4) To address issue 4, specify that the level indication general_level_idc shall indicate the highest level of capability equal to or greater than the highest level in all profile_tier_level() syntax structures (in all parameter sets) that the output layer set identified by output_layer_set_idx in this configuration record conforms to.

[0230] 5) To address issue 5, propose one or more of the following:

[0231] a. The field num_bytes_constraint_info is coded using 6 bits.

[0232] b. The field num_bytes_constraint_info is coded immediately after the ptl_multilayer_enabled_flag field.

[0233] c. The semantics of the field num_bytes_constraint_info are specified as follows: num_bytes_constraint_info specifies the number of bytes in the general_constraint_info() syntax structure defined in ISO / IEC 23090-3. A value equal to 1 indicates that gci_present_flag in the general_constraint_info() syntax structure is equal to 0 and that no general_constraint_info field is signaled in this VvcPTLRecord.

[0234] d. The signaling condition for the general_constraint_info field is changed from "if(num_bytes_constraint_info > 0)" to "if(num_bytes_constraint_info > 1)".

[0235] e. The general_constraint_info field is coded using 8*num_bytes_constraint_info bits instead of (8*num_bytes_constraint_info - 2) bits.

[0236] 6) To solve problem 6, the signaling of the field output_layer_set_idx in VvcDecoderConfigurationRecord is optional even when ptl_present_flag is equal to 1, e.g. conditioned on "if (track_ptl.ptl_multilayer_enabled_flag)", which indicates that the VVC bitstream contains only one layer carried in the VVC track (after resolving the referenced VVC track or VVC subpicture track, if any).

[0237] a. Alternatively, when ptl_present_flag is equal to 1 and output_layer_set_idx is not present, its value is inferred to be equal to the OLS index of the OLS containing only the layer carried in the VVC track (after resolving the referenced VVC track or VVC subpicture track, if any).

[0238] 7) To solve problem 7, the NAL_unit_type field in VvcDecoderConfigurationRecord uses 5 bits instead of 6 bits.

[0239] 8) To solve problem 8, the semantics of ptl_present_flag is specified as follows: ptl_present_flag equal to 1 specifies that the track contains a VVC bitstream corresponding to a particular output layer set. ptl_present_flag equal to 0 specifies that the track can not contain a VVC bitstream corresponding to a particular output layer set, but can contain VVC bitstreams corresponding to multiple output layer sets or contain one or more individual layers that do not form an output layer set or contain individual sub-layers that do not include sub-layers with Temporalld equal to 0.

[0240] 9) To solve problem 9, the coding of the level_idc field in one or both of SubpicCommonGroupBox and SubpicMultipleGroupsBox is specified to use 8 bits instead.

[0241] a. Alternatively, the subsequent 24 bits after the level_idc field can also be specified as reserved bits.

[0242] b. Alternatively, the subsequent 8 bits after the level_idc field can also be specified as reserved bits.

[0243] c. Alternatively, the subsequent zero bits after the level_idc field can also be specified as reserved bits.

[0244] 10) To address issue 10, the coding of one or more of the num_active_tracks field in SubpicCommonGroupBox and the num_subgroup_ids field and num_active_tracks[i] fields in SubpicMultipleGroupsBox is changed to use 16 bits.

[0245] a. Alternatively, the following 16 bits after one or more of the above-mentioned fields can also be designated as reserved bits.

[0246] b. Alternatively, the following 0 bits after one or more of the above-mentioned fields can also be designated as reserved bits.

[0247] 6. Embodiments

[0248] Below are some example embodiments for some aspects of the invention summarized above in Section 5, which can be applied to the standard specification of VVC video file format. The modified text is based on the latest draft specification. Most of the relevant parts that have been added or modified are indicated in bold italic text, and some deleted parts are indicated with open and close brackets (e.g. [[]]). The text between the two brackets indicates the deleted or cancelled text. There can be other editorial nature changes as well, so nothing is highlighted.

[0249] 6.1. First embodiment

[0250] This embodiment is for items 1, 2, 3, 4, 5a, 5b, 5c, 5d, 5e, 6, 6a, 7 and 8.

[0251] 6.1.1. Decoder configuration information

[0252] 6.1.1.1. VVC decoder configuration record

[0253] 6.1.1.1.1. Definition

[0254] This subclause specifies the decoder configuration information for ISO / IEC 23090-3 video content.

[0255] This record contains the size of the length field used in each sample to indicate the length of the NAL unit it contains and parameter sets, (if stored in that sample entry). This record is externally specified (its size is provided by the structure that contains it).

[0256] This record contains the version field. Version 1 of this record is defined by this specification. Incompatible changes to the record will be indicated by a change in the version number. If the version number cannot be recognized, the reader shall not attempt to decode the stream to which this record applies or it.

[0257] Compatible extensions to this record will extend it and will not change the configuration version code. The reader should be prepared to ignore unrecognized data definitions beyond their understanding.

[0258] When the track itself contains a VVC bitstream or contains a VVC bitstream by resolving the'subp' track reference, the VvcPtlRecord shall be present in the decoder configuration record, If ptl_present_flag in the decoder configuration record for the track is equal to zero, the track shall have an 'oref' track reference.

[0259] The values of the VvcPTLRecord, chroma_format_idc, and bit_depth_minus8 syntax elements are valid for all parameter sets [[activated]] in the stream described by this record. Specifically, the following restrictions apply:

[0260] The level indication general_profile_idc shall indicate the level to which the [[associated stream]] of this configuration record conforms. The level indication general_profile_idc shall indicate the level to which the [[associated stream]] of this configuration record conforms.

[0261] NOTE 1 - If [[SPS is with]] a different level The stream can need to be examined to determine which level, if any, the entire stream conforms to. If no examination of the entire stream is made, or if the examination shows that no level is conformed to by the entire stream, the entire stream needs to be split into two or more sub-streams with separate configuration records in which these rules can be met.

[0262] The tier indication general_tier_flag shall indicate one tier that is equal to or greater than the highest tier indicated in all [[parameter sets]] of the [[parameter sets]].

[0263] Each bit in general_constraints_info can only be set if [[all parameter sets set that bit]]

[0264] The level indication general_level_idc shall indicate one level that is equal to or greater than ​all [[parameter sets]] The highest level of capability level of [[for the highest layer indicated]] in all [[parameter sets]].

[0265] The following constraint applies to chroma_format_idc:

[0266] - If the value of sps_chroma_format_idc as defined in ISO / IEC 23090-3 is the same in all SPS referred to by the NAL units of the track, chroma_format_idc shall be equal to sps_chroma_format_idc.

[0267] - Otherwise, if ptl_present_flag is equal to 1, chroma_format_idc shall be equal to vps_ols_dpb_chroma_format[ output_layer_set_idx ], as defined in ISO / IEC 23090-3.

[0268] - Otherwise, chroma_format_idc shall not be present.

[0269] The following constraint applies to bit_depth_minus8:

[0270] - If the value of sps_bitdepth_minus8 as defined in ISO / IEC 23090-3 is the same in all SPS referred to by the NAL units of the track, bit_depth_minus8 shall be equal to sps_bitdepth_minus8.

[0271] - Otherwise, if ptl_present_flag is equal to 1, bit_depth_minus8 shall be equal to vps_ols_dpb_bitdepth_minus8[ output_layer_set_idx ], as defined in ISO / IEC 23090-3.

[0272] - Otherwise, bit_depth_minus8 shall not be present.

[0273] Display indications about chroma format and bit depth are provided in the VVC decoder configuration record, along with other important format information used by VVC video elementary streams. If two sequences have different color spaces indicated in their VUI information, then two different VVC sample entries are also needed.

[0274] There is an array group to carry initialization NAL units. NAL unit types are restricted to indicate DCI, VPS, SPS, PPS, prefix APS, and prefix SEI NAL units. NAL unit types reserved in ISO / IEC 23090-3 and this specification can be defined in the future, and the reader should ignore the array with NAL unit type values that are reserved or not allowed.

[0275] NOTE 2: This ‘tolerant’ design is to not raise errors, allowing backward-compatible extensions to these arrays in future specifications.

[0276] NOTE 3: The NAL units carried in the sample entry are included in or at the start of the access unit that is reconstructed from the first sample that references the sample entry, immediately followed by AUD and OPI NAL units, if any.

[0277] The order of the suggested arrays is DCI, VPS, SPS, PPS, prefix APS, prefix SEI.

[0278] 6.1.1.1.2. Syntax

[0279]

[0280]

[0281]

[0282] 6.1.1.1.3. Semantics

[0283] general_profile_idc, general_tier_flag, general_level_idc, ptl_frame_only_constraint_flag, ptl_multilayer_enabled_flag, general_constraint_info, ptl_sublayer_level_present[i], sublayer_level_idc[i], num_sub_profiles, and general_sub_profile_idc[j] contain matching values for the fields or syntax structures general_profile_idc, general_tier_flag, general_level_idc, ptl_frame_only_constraint_flag, ptl_multilayer_enabled_flag, general_constraint_info(), ptl_sublayer_level_present[i], sublayer_level_idc[i], tl_num_sub_profiles, and general_sub_profile_idc[j] as defined in ISO / IEC 23090-3 for the stream to which this configuration record applies.

[0284]

[0285] avgFrameRate gives the average frame rate of the stream to which this configuration record applies in frames per (256 seconds). A value of 0 indicates that the average frame rate is not specified.

[0286] A constantFrameRate equal to 1 indicates that the stream to which this configuration record applies has a constant frame rate. A value of 2 indicates that the representations of each temporal layer in the stream are constant frame rate. A value of 0 indicates that the stream can or can not be constant frame rate.

[0287] A numTemporalLayers greater than 1 indicates that the track to which this configuration record applies is scalable in the temporal domain and contains a number of temporal layers (also referred to as temporal sub-layers or sub-layers in ISO / IEC 23090-3) equal to numTemporalLayers. A value of 1 indicates that the track to which this configuration record applies is not scalable in the temporal domain. A value of 0 indicates that it is unknown whether the track to which this configuration record applies is scalable in the temporal domain.

[0288] lengthSizeMinusOne plus 1 specifies the length (in bytes) of the NALUnitLength field in VVC video stream samples in the stream to which this configuration record applies. For example, a size of one byte is indicated with the value 0. The value of this field shall be one of 0, 1, or 3, which correspond to lengths coded with 1, 2, or 4 bytes, respectively.

[0289] ptl_present_flag equal to 1 specifies that the track contains a VVC bitstream corresponding to a particular output layer set. ptl_present_flag equal to 0 specifies that the track can not contain a VVC bitstream corresponding to a particular output layer set, while or contains one or more individual layers that do not form an output layer set or individual sub-layers that do not include a sub-layer with Temporalld equal to 0.

[0290] track_ptl specifies the profile, tier, and level of the output layer set represented by the VVC bitstream contained in the track.

[0291] output_layer_set_idx specifies the output layer set index of the output layer set represented by the VVC bitstream contained in the track. The value of output_layer_set_idx can be used as the value of the TargetOlsIdx variable provided by an external component to a VVC decoder, as specified in ISO / IEC 23090-3 for decoding the bitstream contained in the track.

[0292] chroma_format_present_flag equal to 0 specifies that chroma_format_idc is not present. chroma_format_present_flag equal to 1 specifies that chroma_format_idc is present.

[0293] chroma_format_idc specifies the chroma format applied to this track.

[0294] bit_depth_present_flag equal to 0 specifies that bit_depth_minus8 is not present. bit_depth_present_flag equal to 1 specifies that bit_depth_minus8 is present.

[0295] bit_depth_minus8 specifies the bit depth applied to this track.

[0296] numArrays specifies the number of NAL unit arrays of the indicated type.

[0297] array_completeness indicates that all NAL units of the indicated type are in the following array, none in the stream, when equal to 1; when equal to 0, it indicates that additional NAL units of the indicated type can be in the stream; the allowed values are constrained by the example entry name.

[0298] NAL_unit_type indicates the type of the NAL units in the following array (which shall all be of this type); it takes values defined in ISO / IEC 23090-3; it is constrained to take one of the values indicating a DCI, VPS, SPS, PPS, prefix APS, or suffix SEI NAL unit.

[0299] numNalus indicates the number of NAL units of the indicated type included in the configuration record for the stream to which this configuration record applies. The SEI array shall contain only ‘declarative’ SEI messages, i.e. those providing information about the entire stream. One example of such SEI is the user data SEI.

[0300] nalUnitLength indicates the length of the NAL unit in bytes.

[0301] nalUnit contains a DCI, VPS, SPS, PPS, APS, or declarative SEI NAL unit as specified in ISO / IEC 23090-3.

[0302] Figure 1 is a block diagram illustrating an example video processing system 1900 in which various techniques disclosed herein can be implemented. Various implementations can include some or all of the components of the system 1900. The system 1900 can include an input 1902 for receiving video content. The video content can be received in a raw or uncompressed format, e.g., 8-bit or 10-bit multi-component pixel values, or can be received in a compressed or encoded format. The input 1902 can represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, passive optical networks (PONs), etc., and wireless interfaces such as Wi-Fi or cellular interfaces.

[0303] The system 1900 can include a codec 1904, which can implement various coding or encoding methods described in this document. The codec 1904 can reduce the average bitrate of a video from the input 1902 to the output of the codec 1904 to produce a coded representation of the video. Thus, the codec techniques are sometimes referred to as video compression or video transcoding techniques. The output of the codec 1904 can be stored, or transmitted via a communication connected, as represented by component 1906. The stored or communicated bitstream (or encoded) representation of the video received at the input 1902 can be used by component 1908 to generate pixel values or displayable video that is sent to a display interface 1910. The process of generating user-viewable video from a bitstream representation is sometimes referred to as video decompression. Also, although certain video processing operations are referred to as “encoding” operations or tools, it should be understood that the encoding tools or operations are used at an encoder, and corresponding decoding tools or operations that reverse the encoding results would be performed by a decoder.

[0304] Examples of peripheral bus interfaces or display interfaces can include Universal Serial Bus (USB) or High Definition Multimedia Interface (HDMI) or DisplayPort, among others. Examples of storage interfaces include SATA (Serial Advanced Technology Attachment), PCI, IDE interfaces, among others. The techniques described in this document can be embodied in various electronic devices such as mobile telephones, laptop computers, smart phones, or other devices capable of performing digital data processing and / or video display.

[0305] Figure 2 is a block diagram of a video processing apparatus 3600. The apparatus 3600 can be used to implement one or more methods described herein. The apparatus 3600 can be embodied in a smart phone, a tablet computer, a computer, an Internet of Things (IoT) receiver, or the like. The apparatus 3600 can include one or more processors 3602, one or more memories 3604, and video processing hardware 3606. The processor(s) 3602 can be configured to implement one or more methods described in this document. The memory(ies) 3604 can be used for storing data and code used during operation of the present techniques described herein. The video processing hardware 3606 can be used to implement, in hardware circuitry, some of the techniques described in this document. In some embodiments, the video processing hardware 3606 can be included at least in part in the processor 3602, for example, a graphics co-processor.

[0306] Figure 4 is a block diagram illustrating an example video coding system 100 that can utilize the techniques of this disclosure.

[0307] As Figure 4As shown, video coding system 100 can include a source device 110 and a destination device 120. Source device 110 generates encoded video data, which can be referred to as a video encoding device. Destination device 120 can decode the encoded video data generated by source device 110, and destination device 120 can be referred to as a video decoding device.

[0308] Source device 110 can include a video source 112, a video encoder 114, and an input / output (VO) interface 116.

[0309] Video source 112 can include a source such as a video capture device, an interface to receive video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of such sources. Video data can comprise one or more pictures. Video encoder 114 encodes video data from video source 112 to generate a bitstream. The bitstream can include a sequence of bits that form a coded representation of the video data. The bitstream can include encoded pictures and associated data. An encoded picture is a coded representation of a picture. The associated data can include sequence parameter sets, picture parameter sets, and other syntax structures. VO interface 116 can include a modulator / demodulator (modem) and / or a transmitter. The encoded video data can be transmitted directly to destination device 120 by VO interface 116 via network 130a. The encoded video data can also be stored onto a storage medium / server 130b for access by destination device 120.

[0310] Destination device 120 can include VO interface 126, video decoder 124, and display device 122.

[0311] VO interface 126 can include a receiver and / or a modem. VO interface 126 can obtain encoded video data from source device 110 or storage medium / server 130b. Video decoder 124 can decode the encoded video data. Display device 122 can display the decoded video data to a user. Display device 122 can be integrated with destination device 120, or can be external to destination device 120 which is configured to interface with an external display device.

[0312] Video encoder 114 and video decoder 124 can operate according to a video compression standard, such as the High Efficiency Video Coding (HEVC) standard, Versatile Video Coding (VVM) standard, and other current and / or further standards.

[0313] Figure 5 is a block diagram illustrating an example of a video encoder 200, which can be Figure 4 the video encoder 114 in the system 100 as shown.

[0314] Video encoder 200 can be configured to perform any or all of the techniques of this disclosure. In Figure 5 In examples, video encoder 200 includes a number of functional components. The techniques described in this disclosure can be shared among the various components of video encoder 200. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0315] The functional components of video encoder 200 can include partition unit 201, prediction unit 202, which can include mode select unit 203, motion estimation unit 204, motion compensation unit 205, and intra-prediction unit 206, residual generation unit 207, transform unit 208, quantization unit 209, inverse quantization unit 210, inverse transform unit 211, reconstruction unit 212, buffer 213, and entropy encoding unit 214.

[0316] In other examples, video encoder 200 can include more, less, or different functional components. In one example, prediction unit 202 can include an intra-block copy (IBC) unit. The IBC unit can perform prediction in an IBC mode in which at least one reference picture is the picture in which the current video block is located.

[0317] Furthermore, some components, such as motion estimation unit 204 and motion compensation unit 205, can be highly integrated, but are represented separately in Figure 5 examples for illustrative purposes.

[0318] Partition unit 201 can partition a picture into one or more video blocks. Video encoder 200 and video decoder 300 can support various video block sizes.

[0319] Mode select unit 203 can select one of the intra or inter coding modes, e.g., based on error results, and provide the resulting intra or inter coded block to residual generation unit 207 to generate residual block data and to reconstruction unit 212 to reconstruct the coded block for use as a reference picture. In some examples, mode select unit 203 can select a combination of intra and inter prediction (CIIP) mode, where the prediction is based on both inter prediction signals and intra prediction signals. Mode select unit 203 can also select a resolution for motion vectors for the block (e.g., sub-pixel or integer pixel precision) in the case of inter prediction.

[0320] To perform inter prediction on a current video block, motion estimation unit 204 can generate motion information for the current video block by comparing one or more reference frames from buffer 213 to the current video block. Motion compensation unit 205 can determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from buffer 213 other than the picture associated with the current video block.

[0321] Motion estimation unit 204 and motion compensation unit 205 can perform different operations on a current video block, for example, depending on whether the current video block is in an I slice, a P slice, or a B slice.

[0322] In some examples, motion estimation unit 204 can perform uni-prediction on a current video block, and motion estimation unit 204 can search a reference picture in List 0 or List 1 for a reference video block for the current video block. Motion estimation unit 204 can then generate a reference index indicating the reference picture in List 0 or List 1 that contains the reference video block, and a motion vector indicating a spatial displacement between the current video block and the reference video block. Motion estimation unit 204 can output the reference index, the prediction direction indicator, and the motion vector as motion information for the current video block. Motion compensation unit 205 can generate a predicted video block for the current block based on the reference video block indicated by the motion information for the current video block.

[0323] In other examples, motion estimation unit 204 can perform bi-prediction on a current video block, motion estimation unit 204 can search a reference picture in List 0 for a reference video block for the current video block, and can also search a reference picture in List 1 for another reference video block for the current video block. Motion estimation unit 204 can then generate a reference index indicating the reference pictures in List 0 and List 1 that contain the reference video blocks, and a motion vector indicating a spatial displacement between the reference video blocks and the current video block. Motion estimation unit 204 can output the reference index and the motion vector as motion information for the current video block. Motion compensation unit 205 can generate a predicted video block for the current video block based on the reference video blocks indicated by the motion information for the current video block.

[0324] In some examples, motion estimation unit 204 can output a full set of motion information for a decoding process of a decoder.

[0325] In some examples, motion estimation unit 204 can not output a full set of motion information for a current video. Instead, motion estimation unit 204 can signal motion information for a current video block with reference to motion information of another video block. For example, motion estimation unit 204 can determine that the motion information for the current video block is sufficiently similar to the motion information of a neighboring video block.

[0326] In one example, the motion estimation unit 204 can indicate a value in a syntax structure associated with the current video block that indicates to the video decoder 300 that the current video block has the same motion information as another video block.

[0327] In another example, the motion estimation unit 204 can identify, in a syntax structure associated with the current video block, another video block and a motion vector difference (MVD). The motion vector difference indicates a difference between a motion vector of the current video block and a motion vector of the indicated video block. The video decoder 300 can use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0328] As described above, the video encoder 200 can predictively signal the motion vector. Two examples of predictive signaling techniques that can be implemented by the video encoder 200 include advanced motion vector prediction (AMVP) and merge mode signaling.

[0329] The intra prediction unit 206 can perform intra prediction on the current video block. When the intra prediction unit 206 performs intra prediction on the current video block, the intra prediction unit 206 can generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block can include a predicted video block and various syntax elements.

[0330] The residual generation unit 207 can generate residual data for the current video block by subtracting (e.g., indicated by the minus sign) the predicted video block of the current video block from the current video block. The residual data for the current video block can include residual video blocks corresponding to different sample components in the current video block.

[0331] In other examples, such as in skip mode, there can be no residual data for the current video block for the current video block, and the residual generation unit 207 can not perform the subtraction operation.

[0332] The transform processing unit 208 can generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video blocks associated with the current video block.

[0333] After the transform processing unit 208 generates the transform coefficient video blocks associated with the current video block, the quantization unit 209 can quantize the transform coefficient video blocks associated with the current video block based on one or more quantization parameter (QP) values associated with the current video block.

[0334] Inverse quantization unit 210 and inverse transform unit 211 can apply inverse quantization and inverse transform, respectively, to the transform coefficient video block to reconstruct a residual video block from the transform coefficient video block. Reconstruction unit 212 can add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by prediction unit 202 to produce a reconstructed video block associated with the current block for storage in buffer 213.

[0335] After reconstruction unit 212 reconstructs a video block, in-loop filtering operations can be performed to reduce video block artifacts in the video block.

[0336] Entropy coding unit 214 can receive data from other functional components of video encoder 200. When entropy coding unit 214 receives data, entropy coding unit 214 can perform one or more entropy encoding operations to generate entropy encoded data and output a bitstream that includes the entropy encoded data.

[0337] Figure 6 is a block diagram illustrating an example of a video decoder 300 that can be Figure 4 the video decoder 114 in the system 100 shown.

[0338] Video decoder 300 can be configured to perform any or all of the techniques of this disclosure. In Figure 6 examples, video decoder 300 includes a number of functional components. The techniques described in this disclosure can be shared among the various components of video decoder 300. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0339] In Figure 6 examples, video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, and a reconstruction unit 306 and a buffer 307. In some examples, video decoder 300 can perform decoding passes that are substantially the inverse of encoding passes described with respect to video encoder 200 Figure 5 ).

[0340] Entropy decoding unit 301 can retrieve an encoded bitstream. The encoded bitstream can include entropy encoded video data (e.g., coded blocks of video data). Entropy decoding unit 301 can decode the entropy encoded video data and motion compensation unit 302 can determine motion information, including motion vectors, motion vector precision, reference picture list indices, and other motion information, from the entropy decoded video data. Motion compensation unit 302 may, for example, determine such information by performing AMVP and merge modes.

[0341] Motion compensation unit 302 can generate a motion compensated block, possibly performing interpolation based on an interpolation filter. An identifier of the interpolation filter used at sub-pixel precision can be included in the syntax elements.

[0342] Motion compensation unit 302 can calculate interpolated values for sub-integer pixels of the reference block using the interpolation filter used by video encoder 200 during encoding of the video block. Motion compensation unit 302 can determine the interpolation filter used by video encoder 200 from the received syntax information and use the interpolation filter to generate the prediction block.

[0343] Motion compensation unit 302 can use some of the syntax information to determine the size of blocks used to encode frames and / or slices of the coded video sequence, partition information describing how each macroblock of a picture of the coded video sequence is partitioned, modes indicating how each partition is encoded, one or more reference frames (and lists of reference frames) for each inter-coded block, and other information used to decode the coded video sequence.

[0344] Intra prediction unit 303 can use, for example, intra prediction modes received in the bitstream to form a prediction block from spatially neighboring blocks. Inverse quantization unit 303 inverse quantizes, i.e., de-quantizes, quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 301. Inverse transform unit 303 applies an inverse transform.

[0345] Reconstruction unit 306 can add the residual block to the corresponding prediction block generated by motion compensation unit 202 or intra prediction unit 303 to form a decoded block. If desired, a deblocking filter can also be applied to filter the decoded block in order to remove blockiness artifacts. The decoded video block is then stored in buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction, and also produces decoded video for presentation on a display device.

[0346] Next, a list of preferred solutions of some embodiments is provided.

[0347] The following solutions show example embodiments of the techniques discussed in the previous section (e.g., items 1-4).

[0348] 1. A method of visual media processing (e.g., method 3000 described in Figure 3 The method comprises performing (3002) a conversion between visual media data and a file storing a bitstream representation of the visual media data according to a format rule; wherein the format rule specifies a constraint on information included in the file in relation to a profile, level, constraint, or tier associated with the bitstream representation identified in the file.

[0349] 2. The method according to solution 1, wherein the format rule specifies that the file includes an identification of a level to which the output layer set identified in the file conforms.

[0350] 3. The method according to any of solutions 1 to 2, wherein the format rule specifies that the layer identified in the file is equal to or higher than a highest layer indicated in all syntax structures to which the output layer set included in the file conforms.

[0351] 4. The method according to any of solutions 1 to 3, wherein the format rule specifies that the constraint identified in the file aligns with corresponding values indicated by one or more constraint fields of a syntax structure indicating constraints to which the output layer set in the file conforms.

[0352] 5. The method according to any of solutions 1 to 4, wherein the format rule specifies that the level identified in the file aligns with corresponding values indicated by one or more constraint fields of a syntax structure indicating levels to which the output layer set in the file conforms.

[0353] 6. The method according to any of solutions 1 to 5, wherein the conversion comprises generating a bitstream representation of the visual media data and storing the bitstream representation into the file according to the format rule.

[0354] 7. The method according to any of solutions 1 to 5, wherein the conversion comprises parsing the file according to the format rule to recover the visual media data.

[0355] 8. A video decoding apparatus comprising a processor configured to implement the method recited in one or more of solutions 1 to 7.

[0356] 9. A video coding apparatus comprising a processor configured to implement the method recited in one or more of solutions 1 to 7.

[0357] 10. A computer program product having computer code stored thereon, the code, when executed by a processor, causing the processor to implement the method recited in any of solutions 1 to 7.

[0358] 11. A computer readable medium having a bitstream representation conforming to a file format generated according to any of solutions 1 to 7.

[0359] 12. The method, apparatus or system described in the present document.

[0360] In the solutions described herein, an encoder can conform to the format rule by generating a coded representation according to the format rule. In the solutions described herein, a decoder can use the format rule to parse syntax elements in a coded representation to generate a decoded video, as the format rule knows the presence and absence of syntax elements according to the format rule.

[0361] Technology 1. A method of processing visual media data (e.g., Figure 8 The illustrated method 8000 includes performing (8002) a conversion between a visual media file and a bitstream of visual media data according to a format rule, where the bitstream includes one or more output layer sets and one or more parameter sets, the one or more parameter sets including one or more profile tier level syntax structures, where at least one of the profile tier level syntax structures includes a general constraint information syntax structure, where the format rule specifies that a syntax element is included in a configuration record in the visual media file, and where the syntax element indicates a profile, tier, or level to which an output layer set identified by an output layer set index indicated in the configuration record conforms.

[0362] Technology 2. The method of technology 1, where the syntax element is a general profile indicator syntax element that indicates a profile to which an output layer set identified by an output layer set index conforms.

[0363] Technology 3. The method of technology 1, where the syntax element is a general tier syntax element that indicates a tier that is greater than or equal to a highest tier indicated in all profile tier level syntax structures to which the output layer set identified by the output layer set index conforms.

[0364] Technology 4. The method of technology 1, where the syntax element is a general tier syntax element that indicates a highest tier indicated in all profile tier level syntax structures to which the output layer set identified by the output layer set index conforms.

[0365] Technology 5. The method of technology 1, where the syntax element is a general tier syntax element that indicates a highest tier to which a stream associated with the configuration record conforms.

[0366] Technology 6. The method of technology 1, where the syntax element is a general tier syntax element that indicates a tier to which a stream associated with the configuration record conforms.

[0367] TECHNIQUE 7. The method of technique 1, wherein the configuration record includes a general constraint information syntax element, wherein the format rule specifies that a first bit in the general constraint information syntax element corresponds to a second bit in all general constraint information syntax structures in all tier level syntax structures to which the output layer set identified by the output layer set index conforms, and wherein the format rule specifies that the first bit is set to one only when the second bit in all general constraint information syntax structures is set to one.

[0368] TECHNIQUE 8. The method of technique 1, wherein the syntax element is a general level syntax element whose value indicates a capability level that is greater than or equal to a highest level indicated by all tier level syntax elements in the output layer set identified by the output layer set index conforms to.

[0369] TECHNIQUE 9. The method of technique 1, wherein the format rule specifies that the syntax element is not allowed to be associated with one or more other output layer sets included in a stream stored in the visual media file.

[0370] TECHNIQUE 10. The method of any of techniques 1 to 9, wherein the converting includes generating the visual media file and storing the bitstream into the visual media file according to the format rule.

[0371] TECHNIQUE 11. The method of any of techniques 1 to 9, wherein the converting includes generating the visual media file, and the method further includes storing the visual media file in a non-transitory computer readable recording medium.

[0372] TECHNIQUE 12. The method of any of techniques 1 to 9, wherein the converting includes parsing the visual media file to reconstruct the bitstream according to the format rule.

[0373] TECHNIQUE 13. The method of any of techniques 1 to 12, wherein the visual media file is processed by Versatile Video Coding (VVC).

[0374] TECHNIQUE 14. An apparatus for processing visual media data, the apparatus comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method in one or more of techniques 1 to 13.

[0375] TECHNIQUE 15. A non-transitory computer readable storage medium storing instructions that cause a processor to implement the method of any of techniques 1 to 13.

[0376] TECHNIQUE 16. A video decoding apparatus comprising a processor configured to implement the method recited in one or more of techniques 1 to 13.

[0377] Technology 17. A video encoding apparatus comprising a processor configured to implement a method recited in one or more of technologies 1 to 13.

[0378] Technology 18. A computer program product having stored thereon computer code, which, when executed by a processor, causes the processor to implement a method recited in any of technologies 1 to 13.

[0379] Technology 19. A computer readable medium having stored thereon a visual media file conforming to a file format generated according to any of technologies 1 to 13.

[0380] Technology 20. A visual media file generation method comprising generating a visual media file according to the method of any of technologies 1 to 13, and storing the visual media file on a computer readable program medium.

[0381] Technology 21. A non-transitory computer readable recording medium storing a bitstream of a visual media file generated by a method performed by a video processing apparatus, wherein the method is recited in any of technologies 1 to 13. In some embodiments, a non-transitory computer readable recording medium storing a bitstream of a visual media file generated by a method performed by a video processing apparatus, wherein the method comprises generating a visual media file based on visual media data according to a format rule, wherein the bitstream comprises one or more output layer sets and one or more parameter sets, the one or more parameter sets comprising one or more tier level syntax structures, wherein at least one of the tier level syntax structures comprises a general constraint information syntax structure, wherein the format rule specifies that a syntax element is included in a configuration record in the visual media file, and wherein the syntax element indicates a tier, a layer, or a level to which an output layer set identified by an output layer set index indicated in the configuration record conforms.

[0382] Implementation 1. A method of processing visual media data (e.g., the method 9000 recited in technology 17), the method comprising performing (9002) a conversion between a visual media file and a bitstream of visual media data according to a format rule, wherein the format rule specifies a characteristic of a syntax element in the visual media file, wherein the syntax element has a value indicating a number of bytes used to indicate constraint information associated with the bitstream. Figure 9

[0383] Implementation 2. The method of claim 1, wherein the format rule specifies that the syntax element is coded in the visual media file using six bits.

[0384] Implementation 3. The method of implementation 1, wherein the format rule specifies that the syntax element is coded in the visual media file immediately after a tier level multi-layer enabled flag syntax element in the visual media file.​

[0385] Implementation 4. The method according to implementation 1, wherein the format rule specifies that the syntax element is coded in the visual media file, specifies a number of bytes in a general constraint information syntax element in the visual media file, and wherein the format rule specifies that a value of the syntax element equal to one indicates that a general constraint information flag in the general constraint information syntax element is equal to zero and the general constraint information syntax element is prohibited from being included in a level tier level record in the visual media file.

[0386] Implementation 5. The method according to implementation 1, wherein the format rule specifies that a condition for including a general constraint information syntax element in the visual media file depends on whether a value indicated by a syntax element is greater than one.

[0387] Implementation 6. The method according to implementation 1, wherein the format rule specifies that a number of bits used for coding a general constraint information syntax element into a visual media file is a result of eight multiplied by a value indicating a number of bytes used for indicating the constraint information, and wherein the format rule specifies that the result of eight multiplied by the value indicating the number of bytes used for indicating the constraint information is not subtracted by two.

[0388] Implementation 7. A method of processing visual media data, the method comprising: performing a conversion between a visual media file and a bitstream of visual media data according to a format rule, wherein the format rule specifies that five bits are used for a syntax element in the visual media file, wherein the syntax element has a value indicating a network abstraction layer unit type in a decoder configuration record in the visual media file. In some embodiments, wherein the format rule specifies that five bits are used for another syntax element in the visual media file, and wherein the another syntax element has another value indicating a network abstraction layer unit type in a decoder configuration record in the visual media file.

[0389] Implementation 8. A method of processing visual media data, the method comprising: performing a conversion between a visual media file and a bitstream of visual media data according to a format rule, wherein a track in the visual media file comprises a video bitstream, the video bitstream comprising one or more output layer sets; and wherein the format rule specifies indicating a syntax element for the track, wherein the syntax element indicates whether the track comprises a video bitstream corresponding to a particular output layer set from the one or more output layer sets. In some embodiments, wherein a track in the visual media file comprises a video bitstream, the video bitstream comprising one or more output layer sets, wherein the format rule specifies indicating another syntax element for the track, and wherein the another syntax element indicates whether the track comprises a video bitstream corresponding to a particular output layer set from the one or more output layer sets.

[0390] Implementation 9. The method of implementation 8, wherein the syntax element indicates that the track includes video bitstreams corresponding to a plurality of output layer sets. In some embodiments, the other syntax element indicates that the track includes video bitstreams corresponding to a plurality of output layer sets.

[0391] Implementation 10. The method of implementation 8, wherein the syntax element indicates that the track includes video bitstreams that do not correspond to a particular output layer set from the one or more output layer sets. In some embodiments, the other syntax element indicates that the track includes video bitstreams that do not correspond to a particular output layer set from the one or more output layer sets.

[0392] Implementation 11. A method of processing visual media data, the method comprising: performing a conversion between a visual media file and a bitstream of visual media data according to a format rule, wherein the format rule specifies whether the visual media file includes a syntax element whose value indicates an output layer set index used to indicate an output layer set. In some embodiments, the format rule specifies whether the visual media file includes another syntax element whose value indicates an output layer set index used to indicate an output layer set.

[0393] Implementation 12. The method of implementation 11, wherein the format rule specifies that, in response to another value of a tier level present flag syntax element in the visual media file being equal to one, or in response to a tier level level multi-layer enabled flag being equal to one, the visual media file selectively indicates the syntax element whose value indicates an output layer set index in a decoder configuration record. In some embodiments, the format rule specifies that, in response to another value of a tier level present flag syntax element in the visual media file being equal to one, or in response to a tier level level multi-layer enabled flag being equal to one, the visual media file selectively indicates the other syntax element whose value indicates an output layer set index in a decoder configuration record.

[0394] Implementation 13. The method of implementation 11, wherein the format rule specifies that the visual media file is not allowed to include the syntax element whose value indicates the output layer set index, and wherein the format rule specifies that, in response to a tier level present flag syntax element in the visual media file being equal to one, a value of the output layer set index is inferred to be equal to a second value of a second output layer index of a second output layer set that includes only layers carried in the track. In some embodiments, the format rule specifies that the visual media file is not allowed to include the other syntax element whose value indicates the output layer set index, and wherein the format rule specifies that, in response to a tier level present flag syntax element in the visual media file being equal to one, a value of the output layer set index is inferred to be equal to a second value of a second output layer index of a second output layer set that includes only layers carried in the track.

[0395] Implementation 14. The method according to any of the implementations 1 to 13, wherein the converting comprises generating the visual media file and storing the bitstream into the visual media file according to the format rule.

[0396] Implementation 15. The method according to any of the implementations 1 to 13, wherein the converting comprises generating the visual media file, and the method further comprises storing the visual media file in a non-transitory computer readable recording medium.

[0397] Implementation 16. The method according to any of the implementations 1 to 13, wherein the converting comprises parsing the visual media file according to the format rule to reconstruct the bitstream.

[0398] Implementation 17. The method according to any of the implementations 1 to 16, wherein the visual media file is processed by Versatile Video Coding (VVC).

[0399] Implementation 18. An apparatus for processing visual media data, the apparatus comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method in one or more of the implementations 1 to 17.

[0400] Implementation 19. A non-transitory computer readable storage medium storing instructions, which cause a processor to implement the method of any of the implementations 1 to 17.

[0401] Implementation 20. A video decoding apparatus comprising a processor configured to implement the method recited in one or more of the implementations 1 to 17.

[0402] Implementation 21. A video coding apparatus comprising a processor configured to implement the method recited in one or more of the implementations 1 to 17.

[0403] Implementation 22. A computer program product having computer code stored thereon, the code, when executed by a processor, causing the processor to implement the method recited in any of the implementations 1 to 17.

[0404] Implementation 23. A computer readable medium having a visual media file in a file format generated according to any of the implementations 1 to 17.

[0405] Implementation 24. A visual media file generation method comprising generating a visual media file according to the method of any of the implementations 1 to 17, and storing the visual media file on a computer readable program medium.

[0406] Implementation 25. A non-transitory computer-readable recording medium storing a bitstream of a visual media file generated by a method performed by a video processing apparatus, wherein the method is in accordance with any of implementations 1 to 17. A non-transitory computer-readable recording medium storing a bitstream of a visual media file generated by a method performed by a video processing apparatus, wherein the method comprises generating the visual media file based on visual media data according to a format rule, wherein the format rule specifies a property of a syntax element in the visual media file, wherein the syntax element has a value indicative of a number of bytes used to indicate constraint information associated with the bitstream.

[0407] Operation 1. A method of processing visual media data (e.g., a method of generating a bitstream of a visual media file), comprising performing (10002) a conversion between a visual media file and the bitstream of visual media data according to a format rule, wherein the format rule specifies a property of a syntax element in the visual media file, and wherein the format rule specifies that a syntax element having a value indicative of a level identification is signaled using an octet coding in either or both of a subpicture common group box or a subpicture multiple group box. Figure 9

[0408] Operation 2. The method of operation 1, wherein the format rule specifies an absence of reserved bits immediately following the syntax element having the value indicative of the level identification.

[0409] Operation 3. The method of operation 1, wherein the format rule specifies that 24 bits immediately following the syntax element having the value indicative of the level identification are reserved bits.

[0410] Operation 4. The method of operation 1, wherein the format rule specifies that 8 bits immediately following the syntax element having the value indicative of the level identification are reserved bits.

[0411] ​Operation 5. A method of processing visual media data, comprising: performing a conversion between a visual media file and a bitstream of visual media data according to a format rule, wherein the format rule specifies a characteristic related to a first syntax element, a second syntax element, or a set of third syntax elements in the visual media file, wherein the first syntax element has a first value indicating a number of active tracks in the visual media file, wherein the second syntax element has a second value indicating a number of subgroup identifiers in the visual media file, and wherein each syntax element in the set of third syntax elements has a third value indicating a number of active tracks in the visual media file. In some embodiments, wherein the format rule specifies a characteristic related to a first syntax element, a second syntax element, or a set of third syntax elements in the visual media file, wherein the first syntax element has a first value indicating a number of active tracks in the visual media file, wherein the second syntax element has a second value indicating a number of subgroup identifiers in the visual media file, and wherein each syntax element in the set of third syntax elements has a third value indicating a number of active tracks in the visual media file.

[0412] Operation 6. The method of operation 5, wherein the format rule specifies 16 bits for indicating the first syntax element having the first value indicating the number of active tracks in the subpicture common group box in the visual media file.

[0413] Operation 7. The method of operation 5, wherein the format rule specifies 16 bits for indicating the second syntax element having the second value indicating the number of subgroup identifiers in the subpicture multiple group box in the visual media file, and wherein the format rule specifies 16 bits for indicating each syntax element in the set of third syntax elements having the third value indicating the number of active tracks in the subpicture multiple group box in the visual media file.

[0414] Operation 8. The method of operation 5, wherein the format rule specifies that 16 bits immediately following the first syntax element having the first value indicating the number of active tracks, the second syntax element having the second value indicating the number of subgroup identifiers, or each syntax element in the set of third syntax elements having the third value indicating the number of active tracks are reserved.

[0415] Operation 9. The method of operation 5, wherein the format rule specifies a lack of reserved bits immediately following the first syntax element having the first value indicating the number of active tracks, the second syntax element having the second value indicating the number of subgroup identifiers, or each syntax element in the set of third syntax elements having the third value indicating the number of active tracks.

[0416] Operation 10. The method according to any one of operations 1 to 9, wherein the converting comprises generating the visual media file and storing the bitstream into the visual media file according to the format rule.

[0417] Operation 11. The method according to any one of operations 1 to 9, wherein the converting comprises generating the visual media file, and the method further comprises storing the visual media file in a non-transitory computer readable recording medium.

[0418] Operation 12. The method according to any one of operations 1 to 9, wherein the converting comprises parsing the visual media file according to the format rule to reconstruct the bitstream.

[0419] Operation 13. The method according to any one of operations 1 to 12, wherein the visual media file is processed by Versatile Video Coding (VVC).

[0420] Operation 14. An apparatus for processing visual media data, the apparatus comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to implement the method in one or more of operations 1 to 13.

[0421] Operation 15. A non-transitory computer readable storage medium storing instructions, which cause a processor to implement the method of any one of operations 1 to 13.

[0422] Operation 16. A video decoding apparatus comprising a processor configured to implement the method recited in one or more of operations 1 to 13.

[0423] Operation 17. A video encoding apparatus comprising a processor configured to implement the method recited in one or more of operations 1 to 13.

[0424] Operation 18. A computer program product having computer code stored thereon, the code, when executed by a processor, causing the processor to implement the method recited in any one of operations 1 to 13.

[0425] Operation 19. A computer readable medium having a visual media file in a file format generated according to any one of operations 1 to 13.

[0426] Operation 20. A visual media file generation method comprising: generating a visual media file according to the method of any one of operations 1 to 13, and storing the visual media file on a computer readable program medium.

[0427] Operation 21. A non-transitory computer-readable recording medium storing a bitstream of a visual media file generated by a method performed by a video processing apparatus, wherein the method is described in any of operations 1 to 13. In some embodiments, a non-transitory computer-readable recording medium storing a bitstream of a visual media file generated by a method performed by a video processing apparatus, wherein the method comprises: generating the visual media file based on visual media data according to a format rule, wherein the format rule specifies a property of a syntax element in the visual media file, and wherein the format rule specifies that a syntax element having a value indicative of a level identification is signaled using octets in either or both of a subpicture common group box or a subpicture multiple group box.

[0428] In this document, the term "video processing" can refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm can be applied during a conversion from a pixel representation of a video to a corresponding bitstream representation, or vice versa. For example, a bitstream representation of a current video block can correspond to bits that are collocated or spread out in different locations within a bitstream as defined by syntax. For example, a macroblock can be encoded according to transformed and coded error residual values, and also using bits in a header and other fields in the bitstream. Furthermore, during conversion, a decoder can parse a bitstream based on a decision as described in the above-described solutions, with knowledge that some fields can or can not be present. Similarly, an encoder can determine that certain syntax fields are included or not included, and generate a coded representation accordingly by including or excluding syntax fields from the coded representation.

[0429] The disclosed and other solutions, examples, embodiments, modules and functional operations described herein can be realized in digital electronic circuitry, or in a computer software, firmware, or hardware, including the structural equivalents of such, or in combinations of one or more thereof. The disclosed and other embodiments can be realized as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more thereof. The term "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for said program, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. The propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus.

[0430] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and are interconnected by a communication network.

[0431] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0432] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0433] Although this patent document contains many details, these should not be construed as limiting the scope of any subject matter or of any embodiment, but as merely describing features that can be present and combinations thereof. Certain features that are described in the context of separate embodiments can also be implemented in combination with each other. Conversely, various features that are described in the context of a single embodiment can also be implemented separately or in any appropriate subcombination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0434] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such order, nor that all illustrated operations be performed, to achieve desirable results. Moreover, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0435] Only a few implementations and examples are described and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A method of processing visual media data, comprising: performing a conversion between a visual media file and a bitstream of visual media data according to a format rule, wherein the bitstream comprises one or more output layer sets and one or more parameter sets, the one or more parameter sets comprising one or more tier level syntax structures, wherein at least one of the tier level syntax structures comprises a general constraint information syntax structure, wherein the format rule specifies that a syntax element is included in a configuration record in the visual media file, wherein the syntax element indicates a tier, a layer, or a level to which an output layer set identified by an output layer set index indicated in the configuration record conforms, wherein the syntax element is a general tier indicator syntax element that indicates the tier to which the output layer set identified by the output layer set index conforms, wherein the syntax element is a general layer syntax element that indicates a layer that is greater than or equal to a highest layer indicated in all tier level syntax structures to which the output layer set identified by the output layer set index conforms.

2. The method of claim 1, wherein wherein the syntax element is a general layer syntax element that indicates a highest layer indicated in all tier level syntax structures to which the output layer set identified by the output layer set index conforms.

3. The method of claim 1, wherein wherein the syntax element is a general layer syntax element that indicates a highest layer to which a stream associated with the configuration record conforms.

4. The method of claim 1, wherein wherein the syntax element is a general layer syntax element that indicates a layer to which a stream associated with the configuration record conforms.

5. The method of claim 1, wherein wherein the configuration record comprises a general constraint information syntax element, wherein the format rule specifies that a first bit in the general constraint information syntax element corresponds to a second bit in all general constraint information syntax structures in all tier level syntax structures to which the output layer set identified by the output layer set index conforms, and wherein the format rule specifies that the first bit is set to one only if the second bit in all general constraint information syntax structures is set to equal one.

6. The method of claim 1, wherein wherein the syntax element is a general level syntax element whose value indicates a capability level that is greater than or equal to a highest level indicated in all tier level syntax elements to which the output layer set identified by the output layer set index conforms.

7. The method of claim 1, wherein, the format rule specifies that the syntax element is not allowed to be associated with one or more other output layer sets included in a stream stored in the visual media file.

8. The method according to any one of claims 1 to 7, wherein, the conversion comprises generating the visual media file and storing the bitstream into the visual media file according to the format rule.

9. The method according to any one of claims 1 to 7, wherein, the conversion comprises generating the visual media file, and the method further comprises storing the visual media file in a non-transitory computer readable recording medium.

10. The method according to any one of claims 1 to 7, wherein, the conversion comprises parsing the visual media file according to the format rule to reconstruct the bitstream.

11. The method according to any one of claims 1 to 7, wherein, processing the visual media file through versatile video coding (VVC).

12. An apparatus for processing visual media data, the apparatus comprising a processor and a non-transitory memory having instructions thereon, wherein, The instructions, when executed by the processor, cause the processor to implement the method of any one of claims 1-11.

13. A non-transitory computer-readable storage medium storing instructions that cause a processor to implement the method of any one of claims 1-11.

Images