Signaling auxiliary information

Abstract

Systems, methods, and apparatuses for encoding, decoding, or transcoding digital video are described. An example method for processing video data includes performing conversion between video and a video bitstream according to a format rule, where the format rule specifies that a supplemental enhancement information field included in the bitstream indicates whether the bitstream has one or more video layers representing auxiliary information.

Description

[Technical field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] Books This application claims priority to and the benefit of International Patent Application No. PCT / CN2020 / 118711, filed September 29, 2020. Based on International Patent Application No. PCT / CN2021 / 121513, filed September 29, 2021, It is something . above The All patent applications teeth, The full text of these By reference To the original wish It is incorporated herein by reference.

[0002] [Technical field] This patent specification relates to digital video coding techniques, including video encoding, transcoding, or decoding. [Background technology]

[0003] Digital video accounts for the largest bandwidth usage on the Internet and other digital communications networks, and the bandwidth demands for digital video usage are expected to continue to increase as the number of user devices capable of receiving and displaying video increases. Summary of the Invention

[0004] This specification discloses techniques that may be used by video encoders and decoders to process coded representations of videos or images according to a file format.

[0005] In one example aspect, a method of processing video data is disclosed, the method including performing a conversion between a video and a bitstream of video according to a format rule, the format rule specifying that a supplemental enhancement information field or a video usability information syntax structure included in the bitstream indicates whether the bitstream comprises a multiview bitstream in which multiple videos are coded in multiple video layers.

[0006] In another example aspect, a method of processing video data is disclosed, the method including performing a conversion between a video and a video bitstream according to a format rule, the format rule defining that a supplemental enhancement information field included in the bitstream indicates whether the bitstream has one or more video layers representing auxiliary information.

[0007] In another example aspect, a video processing method is disclosed that includes performing a conversion between a video including video pictures and a coded representation of the video, the coded representation conforming to a format rule, the format rule defining that a field included in the coded representation indicates that the video is a multi-view video.

[0008] In another example aspect, another video processing method is disclosed that includes performing a conversion between a video including video pictures and a coded representation of the video, the coded representation conforming to a format rule, the format rule defining that a field included in the coded representation indicates that the video is coded in a plurality of video layers with the coded representation.

[0009] In yet another exemplary aspect, a video encoder apparatus is disclosed, the video encoder having a processor configured to implement the above method.

[0010] In yet another exemplary aspect, a video decoder apparatus is disclosed, the video decoder having a processor configured to implement the above method.

[0011] In yet another exemplary aspect, a computer readable medium is disclosed having code stored thereon, the code being in the form of processor executable code that embodies one of the methods described herein.

[0012] In yet another exemplary aspect, a computer-readable medium having a bitstream stored thereon is disclosed, the bitstream being generated or processed using the methods described herein.

[0013] These and other features are described throughout this specification. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram of an example video processing system. [Diagram 2] FIG. 2 is a block diagram of a video processing device. [Diagram 3] 1 is a flowchart of an exemplary method of video processing. [Figure 4] FIG. 1 is a block diagram illustrating a video coding system according to some embodiments of the present disclosure. [Diagram 5] FIG. 2 is a block diagram illustrating an encoder according to some embodiments of the present disclosure. [Figure 6] FIG. 2 is a block diagram illustrating a decoder according to some embodiments of the present disclosure. [Figure 7] This is an example of a bitstream that contains two OLSs and has vps_max_tid_il_ref_pics_plus1[1][0] where OLS2 is equal to 0. [Figure 8] 1 is a flowchart of an exemplary method of video processing. [Figure 9] 1 is a flowchart of an exemplary method of video processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Section headings are used herein for ease of understanding and not to limit the applicability of the techniques and embodiments disclosed in each section to only that section. Additionally, the term H.266 is used in some descriptions for ease of understanding only, and not to limit the scope of the disclosed techniques. As such, the techniques described herein are applicable to other video codec protocols and designs. Editorial changes herein are indicated to the text with strikethrough indicating the deletion of text and highlighting indicating the addition of text with respect to the current draft of the VVC specification.

[0016] [1. Introduction] This specification relates to video coding technology. In particular, it relates to signaling scalability dimension information of Versatile Video Coding (VVC) video bitstreams. The ideas may be applied individually or in various combinations to any video coding standard or non-standard video codec, such as the recently finalized VVC.

[0017] [2.Acronym] ACT Adaptive Colour Transform ALF Adaptive Loop Filter AMVR Adaptive Motion Vector Resolution APS Adaptation Parameter Set AU Access Unit AUD Access Unit Delimiter AVC Advanced Video Coding(Rec.ITU-T H.265|ISO / IEC 14496-10) B Bi-predictive BCW Bi-prediction with CU-level Weights BDOF Bi-Directional Optical Flow BDPCM Block-Based Delta Pulse Code Modulation BP Buffering Period CABAC Context-based Adaptive Binary Arithmetic Coding CB Coding Block CBR Constant Bit Rate CCALF Cross-Component Adaptive Loop Filter CLVS Coded Layer Video Sequence CLVSS Coded Layer Video Sequence Start CPB Coded Picture Buffer CRA Clean Random Access CRC Cyclic Redundancy Check CTB Coding Tree Block CTU Coding Tree Unit CU Coding Unit CVS Coded Video Sequence CVSS Coded Video Sequence Start DCI Decoding Capability Information DPB Decoded Picture Buffer DRAP Dependent Random Access Point DU Decoding Unit DUI Decoding Unit Information EG Exponential-Golomb EGk k-th order Exponential-Golomb EOB End Of Bitstream EOS End Of Sequence FD Filler Data FIFO First-In, First-Out FL Fixed-Length GBR Green, Blue, and Red GCI General Constraints Information GDR Gradual Decoding Refresh GPM Geometric Partitioning Mode HEVC High Efficiency Video Coding(Rec.ITU-T H.265|ISO / IEC 23008-2) HRD Hypothetical Reference Decoder HSS Hypothetical Stream Scheduler I Intra IBC Intra Block Copy IDR Instantaneous Decoding Refresh ILRP Inter-Layer Reference Picture IRAP Intra Random Access Point LFNST Low Frequency Non-Separable Transform LPS Least Probable Symbol LSB Least Significant Bit LTRP Long-Term Reference Picture LMCS Luma Mapping with Chroma Scaling MIP Matrix-based Intra Prediction MPS Most Probable Symbol MSB Most Significant Bit MTS Multiple Transform Selection MVP Motion Vector Prediction NAL Network Abstraction Layer OLS Output Layer Set OP Operation Point OPI Operating Point Information P Predictive PH Picture Header POC Picture Order Count PPS Picture Parameter Set PROF Prediction Refinement with Optical Flow PT Picture Timing PU Picture Unit QP Quantization Parameter RADL Random Access Decodable Leading (picture) RASL Random Access Skipped Leading (picture) RBSP Raw Byte Sequence Payload RGB Red, Green, and Blue RPL Reference Picture List SAO Sample Adaptive Offset SAR Sample Aspect Ratio SEI Supplemental Enhancement Information SH Slice Header SLI Subpicture Level Information SODB String Of Data Bits SPS Sequence Parameter Set STRP Short-Term Reference Picture STSA Step-wise Temporal Sublayer Access TR Truncated Rice TU Transform Unit VBR Variable Bit Rate VCL Video Coding Layer VPS Video Parameter Set VSEI Versatile Supplemental Enhancement Information(Rec.ITU-T H.274|ISO / IEC 23002-7) VUI Video Usability Information VVC Versatile Video Coding(Rec.ITU-T H.266|ISO / IEC 23090-3)

[0018] [3. Initial discussion] [3.1. Video coding standards] Video coding standards have evolved primarily through the development of well-known ITU-T and ISO / IEC standards. ITU-T produced H.261 and H.263, ISO / IEC produced MPEG-1 and MPEG-4 Visual, and the two organizations jointly produced H.262 / MPEG-2 Video, H264 / MPEG-4 AVC (Advanced Video Coding), and H.265 / HEVC standards. Since H.262, video coding standards have been based on a hybrid video coding structure where temporal prediction plus transform coding is used. In order to explore future video coding technologies beyond HEVC, the Joint Video Exploration Team (JVET) was jointly established by VCEG and MPEG in 2015. Since then, many new methods have been adopted by JVET and put into reference software named the Joint Exploration Model (JEM). Later, when the Versatile Video Coding (VVC) project was officially launched, JVET was renamed JVET (Joint Video Experts Team). VVC is a new coding standard finalized at the 19th JVET, which ended on June 1, 2020, and aims to reduce the bitrate by 50% compared to HEVC.

[0019] The Versatile Video Coding (VVC) standard (ITU-T H.266 | ISO / IEC 23090-3) and the related Versatile Supplemental Enhancement Information (VSEI) standard (ITU-T H.274 | ISO / IEC 23002-7) are designed for use in the widest range of applications, including both traditional uses such as television broadcast, videoconferencing, or playback from storage media, and newer, more advanced use cases such as adaptive bitrate streaming, video region extraction, composition and merging of content from multiple coded video bitstreams, multi-view video, scalable layered coding, and viewport-adaptive 360° immersive video.

[0020] [3.2. Video-based Point Cloud Compression (V-PCC)] ISO / IEC 23090-5, Information technology - Coded Representation of Immersive Media - Part 5: Visual Volumetric Video-based Coding (V3C) and Video-based Point Cloud Compression (V-PCC), also known as V-PCC for short, is a standard that specifies the coded representation of point cloud signals. The V-PCC standard is another standard that was recently finalized.

[0021] The V-PCC specifies data types such as occupancy, geometry, texture attributes, material attributes, transparency attributes, reflectance attributes, and standard attributes that can be coded using a particular video codec such as VVC, HEVC, AVC, etc.

[0022] [3.3. Support for temporal scalability in VVC] VVC includes similar support for temporal scalability as found in HEVC. Such support supports signaling of temporal IDs in the NAL unit header, the restriction that pictures of a particular temporal sublayer cannot be used for inter-prediction reference with pictures of lower temporal sublayers, the sub-bitstream extraction process, and the requirement that each sub-bitstream extraction output of the appropriate input must be a conforming bitstream. Media-Aware Network Element(s) (MANE) can utilize the temporal IDs in the NAL unit header for stream adaptation based on temporal scalability.

[0023] [3.4. Changing picture resolution within a sequence in VVC] In AVC and HEVC, the spatial resolution of a picture cannot be changed unless a new sequence using the new SPS starts with an IRAP picture. VVC allows changing picture resolution within a sequence at some position without encoding an IRAP picture, which is always intra-coded. This feature is sometimes called Reference Picture Resampling (RPR) since it requires resampling of a reference picture used for inter prediction if it has a different resolution than the current picture being decoded.

[0024] To allow for the reuse of motion compensation modules of existing implementations, the scaling ratio is restricted to be greater than or equal to 1 / 2 (2x downsampling from reference picture to current picture) and less than or equal to 8 (8x upsampling). The horizontal and vertical scaling ratios are derived based on the picture width and height specified for the reference and current pictures, and the left, right, top, and bottom scaling offsets.

[0025] RPR allows for resolution changes without the need to code IRAP pictures, which can cause momentary bitrate spikes in streaming or video conferencing scenarios, for example, to react to changing network conditions. RPR can also be used in adaptation scenarios where zooming of the entire video area or some area of ​​interest is required. The scaling window offset can be negative to support larger range zoom based adaptations. A negative scaling window offset also allows for extraction of sub-picture sequences from a multi-layer bitstream, while keeping the same scaling window for the extracted sub-bitstream as found in the original bitstream.

[0026] Unlike spatial scalability in the scalable extensions of HEVC, where picture resampling and motion compensation are applied in two different stages, RPR in VVC is performed as part of the same process at the block level, where derivation of sample positions and motion vector scaling are performed during motion compensation.

[0027] Aiming to limit implementation complexity, picture resolution changes in CLVS are not allowed when a picture in CLVS has multiple sub-pictures per picture. Furthermore, decoder side motion vector refinement, bi-directional optical flow, and prediction refinement with optical flow are not applied when RPR is used between the current picture and the reference picture. Co-located pictures for derivation of temporal motion vector candidates are also restricted to have the same picture size, scaling window offset, and CTU size as the current picture.

[0028] For the support of RPR, other aspects of the VVC design are made different from HEVC. First, the picture resolution and the corresponding conformance and scaling window are signaled in PPS, not in SPS, while the maximum picture resolution and the corresponding conformance window are signaled in SPS. In the application, the maximum picture resolution with the corresponding conformance window offset in SPS can be used as the intended or desired picture output size after cropping. Second, in the case of a single-layer bitstream, each picture store (a slot in the DPB for storage of one decoded picture) occupies the buffer size required to store a decoded picture having the maximum picture resolution.

[0029] [3.5. Support for multi-layer scalability in VVC] By having the ability to inter-predict from a reference picture of a different size than the current picture using RPR in the VVC core design, VVC can easily support bitstreams containing multiple layers of different resolutions, for example, two layers with standard definition resolution and high definition resolution, respectively. In a VVC decoder, such functionality can be incorporated without requiring any additional signal processing level coding tools, since the upsampling function required to support spatial scalability can be provided by reusing the RPR upsampling filter. Nevertheless, additional high-level syntax design is required to enable bitstream scalability support.

[0030] Scalability is supported in VVC, but only in the multi-layer profile. Unlike scalability support in any previous video coding standard, including AVC and HEVC extensions, the VVC scalability design is as suitable as possible for single-layer decoder designs. Decoding capabilities for multi-layer bitstreams are specified as if there was only a single layer in the bitstream. For example, decoding capabilities, such as DPB size, are specified in a way that is independent of the number of layers in the bitstream to be decoded. Essentially, decoders designed for single-layer bitstreams do not require significant modifications to be able to decode multi-layer bitstreams.

[0031] Compared to the designs of AVC and HEVC multi-layer extensions, the HLS aspects are significantly simplified at the expense of some flexibility, e.g. 1) an IRAP AU is required to contain a picture for each of the layers present in the CVS, which removes the need to specify a layer-based initiated decoding process, and 2) instead of a complex POC reconfiguration mechanism, VVC includes a much simpler design for POC signaling, making sure that the derived POC value is the same for all pictures in an AU.

[0032] Similar to HEVC, information about layers and layer dependencies is included in the VPS. Information on OLS is provided for signaling which layers are included in the OLS, which layers are to be output, and other information such as PTL and HRD parameters associated with each OLS. Similar to HEVC, there are three modes of operation that output either all layers, only the highest layer, or a specific indicated layer in custom output mode.

[0033] There are some differences between the OLS design in VVC and the OLS design in HEVC. First, in HEVC, a layer set is signaled, then an OLS is signaled based on the layer set, and for each OLS, an output layer is signaled. The HEVC design allows a layer to belong to an OLS that is neither an output layer nor a layer required to decode the output layer. In VVC, the design requires that any layer in an OLS is either an output layer or a layer required to decode the output layer. Thus, in VVC, an OLS is signaled by indicating the output layer of the OLS, and then other layers that belong to the OLS are easily derived by the layer dependencies indicated in the VPS. Furthermore, VVC requires that each layer is included in at least one OLS.

[0034] Another difference in the VVC OLS design is that VVC may exclude some NAL units belonging to non-output layers mapped to the OLS, in contrast to HEVC, where the OLS is composed of all NAL units belonging to the set of identified layers mapped to the OLS. More specifically, the OLS of VVC consists of the set of layers mapped to the OLS, including non-output layers that contain only IRAP or GDR pictures with ph_recovery_poc_cnt equal to 0 or pictures from sublayers used for inter-layer prediction. This allows to indicate the optimal level value of the multi-layer bitstream considering only all "necessary" pictures of all sublayers in the layers forming the OLS, where "necessary" means necessary for output or decoding. Figure 7 shows an example of a two-layer bitstream with vps_max_tid_il_ref_pics_plus1[1][0] equal to 0, i.e., a sub-bitstream where only IRAP pictures from layer L0 are kept when OLS2 is extracted.

[0035] Considering some scenarios where it is beneficial to allow different RAP periodicity in different layers, an AU is allowed to have layers that contain unaligned RAPs, similar to AVC and HEVC. For faster identification of RAPs in multi-layer bitstreams, i.e., AUs with RAPs in all layers, the Access Unit Delimiter (AUD) has been extended compared to HEVC with a flag indicating whether the AU is an IRAP AU or a GDR AU. Furthermore, an AUD is mandatory to be present in such an IRAP or GDR AU if the VPS indicates multiple layers. However, in the case of single-layer bitstreams indicated by a VPS or bitstreams that do not reference a VPS, the AUD is completely optional, as seen in HEVC. This is because in this case, the RAP can be easily detected from the NAL unit type of the first slice in the AU and the respective parameter set.

[0036] To allow sharing of SPS, PPS, and APS by multiple layers, while at the same time ensuring that the bitstream extraction process does not waste parameter sets required for the decoding process, a VCL NAL unit of a first layer may reference an SPS, PPS, or APS with the same or a lower layer ID value, as long as all OLSs that contain that first layer also contain a layer identified by a lower layer ID value.

[0037] [3.6. VUI and SEI Messages] The VUI is a syntax structure transmitted as part of the SPS (and possibly also in the VPS for HEVC). The VUI carries information that does not affect the normative decoding process, but that may be important for proper rendering of the coded video.

[0038] The SEI supports processes related to decoding, display, or other purposes. Like the VUI, the SEI does not affect the normative decoding process. The SEI is carried in SEI messages. Decoder support of SEI messages is optional. However, SEI messages do affect bitstream conformance (e.g., if the syntax of an SEI message in a bitstream does not follow the specification, the bitstream is non-conforming), and some SEI messages are required by the HRD specification.

[0039] The VUI syntax structure and most of the SEI messages used in VVC are not specified in the VVC specification, but rather in the VSEI specification. The SEI messages required for HRD conformance testing are specified in the VVC specification. VVC v1 defines five SEI messages related to HRD conformance testing, and VSEI v1 specifies 20 additional SEI messages. The SEI messages carried in the VSEI specification do not directly affect conforming decoder behavior and are defined such that they can be used in a coding format agnostic manner, allowing the VSEI to be used in the future by other video coding standards in addition to VVC. Rather than specifically referencing VVC syntax element names, the VSEI specification refers to variables whose values ​​are set within the VVC specification.

[0040] Compared to HEVC, the VUI syntax structure of VVC focuses only on information related to the proper rendering of pictures and does not include any timing information or bitstream restriction indication. In VVC, the VUI is signaled in the SPS, and the SPS includes a length field before the VUI syntax structure to inform the length of the VUI payload in bytes. This allows the decoder to easily jump over information, and more importantly, allows convenient future VUI syntax structures by adding new syntax elements directly to the end of the VUI syntax structure in a manner similar to the SEI message syntax extension.

[0041] The VUI syntax structure includes the following information: ● Content that is interlaced or progressive; ● Whether the content contains frame-packed stereoscopic video or projected omnidirectional video; ●Sample aspect ratio: ● Whether the content is suitable for overscan display; ● Color description, including primaries, matrices, and transfer characteristics, which are particularly important to be able to signal Ultra High Definition (UHD) vs. High Definition (HD) color spaces and High Dynamic Range (HDR); Position of chroma compared to luma (signaling clarified for progressive content compared to HEVC).

[0042] When an SPS does not contain any VUI, the information is considered unspecified and must be specified by the application or conveyed by external means if the bitstream is intended for rendering on a display.

[0043] Table 1 lists all SEI messages defined for VVC v1 and their specifications including syntax and semantics. Of the 20 SEI messages defined in the VVC v1 specification, many are inherited from HEVC (e.g., filler payload and both user data SEI messages). Some SEI messages are essential for correct processing or rendering of the coded video content. This is true, for example, for the mastering display color volume, content light level information, or alternative transfer characteristic SEI messages that are particularly relevant for HDR content. Other examples include the equirectangular projection, spherical rotation, packing by region, and omnidirectional viewport SEI messages, which are relevant for signaling and processing of 360° video content. [Table 1] TIFF0007632805000002.tif160160

[0044] New SEI messages defined for VVC v1 include a frame field SEI message, a sample aspect ratio information SEI message, and a sub-picture level information SEI message.

[0045] The frame field SEI message contains information indicating how the associated picture should be displayed (field parity or frame repetition period), the scan type of the associated picture, and whether the associated picture is a duplicate of a previous picture. This information, together with the timing information of the associated picture, was signaled in the picture timing SEI message in previous video coding standards. However, it was observed that frame field information and timing information are two different kinds of information that are not necessarily signaled together. A typical example consists of signaling timing information at the system level but signaling frame field information in the bitstream. Therefore, it was decided to remove the frame field information from the picture timing SEI message and signal it instead in a dedicated SEI message. This change also made it possible to modify the syntax of the frame field information to convey additional, more explicit instructions to the display, such as more values ​​for field pairing and frame repetition.

[0046] The sample aspect ratio SEI message allows signaling different sample aspect ratios for different pictures in the same sequence, while the corresponding information contained in the VUI applies to the entire sequence, which may be relevant when using reference picture resampling functions with scaling factors that cause different pictures of the same sequence to have different sample aspect ratios.

[0047] The sub-picture level information SEI message provides level information of a sub-picture sequence.

[0048] [4. Technical problem solved by the disclosed technical solution] VVC supports multi-layer scalability. However, given a VVC multi-layer bitstream, it is unknown whether the OLS bitstream is a multi-view bitstream or simply a bitstream consisting of multiple layers with SNR and / or spatial scalability. Furthermore, given a VVC multi-layer bitstream, it is unknown whether there are one or more layers representing auxiliary information such as alpha, depth, etc., and if so, which layer represents what.

[0049] [5. List of technical solutions] In order to solve the above problems, the methods summarized below are disclosed. The inventions should be regarded as examples for illustrating the outline and should not be interpreted in a narrow sense. Moreover, these inventions may be applied individually or combined in any way. 1) Information indicating whether a VVC video bitstream is a multiview bitstream is signaled in the VVC video bitstream. In one example, the information is signaled in an SEI message (eg, called a Scalability Dimension SEI message). i. In one example, the scalability dimension SEI message provides information for a bitstream bitstreamInScope, which is defined as the sequence of AUs that includes, in decoding order, the AU that contains the current scalability dimension SEI message, followed by zero or more AUs including all subsequent AUs up to but not including any subsequent AU that contains a scalability dimension SEI message. ii. In one example, the SEI message includes a flag indicating whether the bitstream may be a multiview bitstream. iii. In one example, the SEI message indicates the view ID for each layer. 1. In one example, the SEI message includes a flag indicating whether the view ID is signaled per layer. 2. In one example, the length in bits of the view ID per layer is signaled in the SEI message. b. In one example, the information is signaled as part of the VUI. 2) Information indicating whether a VVC video bitstream includes one or more layers representing auxiliary information is signaled in the VVC video bitstream. In one example, the information is signaled in an SEI message (eg, a scalability dimensions SEI message). i. In one example, the scalability dimension SEI message provides information for a bitstream bitstreamInScope, which is defined as the sequence of AUs that includes, in decoding order, the AU that contains the current scalability dimension SEI message, followed by zero or more AUs including all subsequent AUs up to but not including any subsequent AU that contains a scalability dimension SEI message. ii. In one example, the SEI message includes a flag that indicates whether the bitstream may include auxiliary information carried by one or more layers. iii. In one example, the SEI message indicates an auxiliary ID for each layer. 1. In one example, the SEI message includes a flag indicating whether the auxiliary ID is signaled per layer. 2. In one example, a value of Auxiliary ID (eg, 0) indicates that the layer does not contain any auxiliary pictures. 3. In one example, a value of Auxiliary ID (eg, 1) indicates that the type of auxiliary information is alpha. 4. In one example, a value of Auxiliary ID (eg, 2) indicates that the type of auxiliary information is depth. 5. In one example, a value of Auxiliary ID (eg, 3) indicates that the type of auxiliary information is proprietary (eg, defined in the V-PCC). 6. In one example, a value of Auxiliary ID (eg, 4) indicates that the type of auxiliary information is geometry (eg, as defined in the V-PCC). 7. In one example, a value of Auxiliary ID (eg, 5) indicates that the type of auxiliary information is an attribute (eg, as defined in the V-PCC). 8. In one example, a value of Auxiliary ID (eg, 6) indicates that the type of auxiliary information is a texture attribute (eg, as defined in the V-PCC). 9. In one example, a value of Auxiliary ID (eg, 7) indicates that the type of auxiliary information is a material attribute (eg, as defined in the V-PCC). 10. In one example, a value of Auxiliary ID (eg, 8) indicates that the type of auxiliary information is a transparency attribute (eg, as defined in the V-PCC). 11. In one example, a value of Auxiliary ID (eg, 9) indicates that the type of auxiliary information is a reflectance attribute (eg, as defined in the V-PCC). 12. In one example, a value of Auxiliary ID (eg, 10) indicates that the type of auxiliary information is a standard attribute (eg, as defined in the V-PCC). b. In one example, the information is signaled as part of the VUI.

[0050] [6. Embodiment] Below are some example embodiments of some of the aspects of the invention summarized in Section 5 above that can be applied to the VVC and VSEI specifications.

[0051] [6.1. First embodiment] This embodiment relates to items 1, 1.a and all subitems thereof, 2, 2.a, 2.ai, 2.a.ii, 2.a.iii, 2.a.iii.1, 2.a.iii.2, 2.a.iii.3, and 2.a.iii.4.

[0052] [6.1.1. Scalability Dimension SEI Message Syntax] [Table 2]

[0053] [6.1.2. Semantics of Scalability Dimension SEI Messages] The scalability dimension SEI message provides scalability dimension information for each layer in bitstreamInScope (defined below): 1) view IDs for each layer, if bitstreamInScope may be a multiview bitstream, and 2) auxiliary IDs for each layer, if there may be auxiliary information (e.g., depth or alpha) carried by one or more layers in bitstreamInScope. bitstreamInScope is a sequence of AUs that includes, in decoding order, the AU that contains the current scalability dimension SEI message, followed by zero or more AUs, including all subsequent AUs up to but not including any subsequent AU that contains a scalability dimension SEI message. sd_max_layers_minus1 plus 1 indicates the maximum number of layers in bitstreamInScope. sd_multiview_info_flag equal to 1 indicates that bitstreamInScope may be a multiview bitstream and the sd_view_id_val[] syntax element is present in the scalability dimension SEI message. sd_multiview_info_flag equal to 0 indicates that bitstreamInScope is not a multiview bitstream and the sd_view_id_val[] syntax element is not present in the scalability dimension SEI message. sd_auxilary_info_flag equal to 1 indicates that there may be auxiliary information carried by one or more layers in bitstreamInScope, and the sd_aux_id[] syntax element is present in the scalability dimension SEI message. sd_auxilary_info_flag equal to 0 indicates that there is no auxiliary information carried by one or more layers in bitstreamInScope, and the sd_aux_id[] syntax element is not present in the scalability dimension SEI message. sd_view_id_len specifies the length in bits of the sd_view_id_val[i] syntax element. sd_view_id_val[i] specifies the view ID of the ith layer in bitstreamInScope. The length of the sd_view_id_val[i] syntax element is sd_view_id_len bits. If not present, the value of sd_view_id_val[i] is inferred to be equal to 0. An sd_aux_id[i] equal to 0 indicates that the i-th layer in bitstreamInScope does not contain an auxiliary picture. An sd_aux_id[i] greater than 0 indicates the type of auxiliary picture at the i-th layer in bitstreamInScope, as specified in Table 2 below. [Table 3] NOTE 1: The interpretation of auxiliary pictures associated with sd_aux_id in the range 128 to 159, inclusive, is specified by means other than the value of sd_aux_id. sd_aux_id[i] should be in the range of 0 to 2, inclusive, or 128 to 159, inclusive, for bitstreams conforming to this version of this specification. Values ​​of sd_aux_id[i] should be in the range of 0 to 2, inclusive, or 128 to 159, inclusive, but in this version of this specification, decoders should allow values ​​of sd_aux_id[i] in the range of 0 to 255, inclusive.

[0054] FIG. 1 is a block diagram illustrating an example video processing system 1900 in which various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of system 1900. System 1900 may include an input 1902 that receives video content. The video content may be received in raw or uncompressed format, e.g., 8 or 10 bit multi-component pixel values, or may be in a compressed or encoded format. Input 1902 may correspond to a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, Passive Optical Network (PON), etc., and wireless interfaces such as Wi-Fi or cellular networks.

[0055] The system 1900 may include a coding component 1904 that may implement various coding or encoding methods described herein. The coding component 1904 may reduce the average bit rate of the video from the input 1902 to an output of the coding component 1904 to generate a coded representation of the video. The coding techniques are therefore sometimes referred to as video compression or video transcoding techniques. The output of the coding component 1904 may be stored or transmitted via a connected communication, as represented by component 1906. The stored or communicated bitstream (or coded) representation of the video received at the input 1902 may be used by component 1908 to generate pixel values ​​or displayable video that is sent to a display interface 1910. The process of generating a video that can be viewed by a user from the bitstream representation is sometimes referred to as video decompression. Furthermore, it will be understood that while certain video processing operations are referred to as "coding" operations or tools, such coding tools or operations are used in an encoder and corresponding decoding tools or operations that transpose the results of the coding will be performed by a decoder.

[0056] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB) or High Definition Multimedia Interface (HDMI) or Displayport, etc. Examples of storage interfaces include Serial Advanced Technology Attachment (SATA), Peripheral Component Interconnect (PCI), Integrated Drive Electronics (IDE) interfaces, etc. The techniques described herein may be embodied in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of performing digital data processing and / or video display.

[0057] FIG. 2 is a block diagram of a video processing device 3600. The device 3600 may be used to implement one or more of the methods described herein. The device 3600 may be embodied in a smartphone, a tablet, a computer, an Internet of Things (IoT) receiver, etc. The device 3600 may include one or more processors 3602, one or more memories 3604, and video processing hardware 3606. The processor 3602 may be configured to implement one or more of the methods described herein. The memory(s) 3604 may be used to store data and code used to implement the methods and techniques described herein. The video processing hardware 3606 may be used to implement some of the techniques described herein in a hardware circuit. In some embodiments, the video processing hardware 3606 may be at least partially included in the processor 3602, e.g., a graphics co-processor.

[0058] FIG. 4 is a block diagram illustrating an example video coding system 100 that may utilize the techniques of this disclosure.

[0059] 4, video coding system 100 may include source device 110 and destination device 120. Source device 110 generates encoded video data and may be referred to as a video encoding device. Destination device 120 may decode the encoded video data generated by source device 110 and may be referred to as a video decoding device.

[0060] The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0061] The video source 112 may include a source such as a video capture device, an interface that receives video data from a video content provider, and / or a computer graphics system that generates video data, or a combination of such sources. The video data may have one or more pictures. The video encoder 114 encodes the video data from the video source 112 to generate a bitstream. The bitstream may include a sequence of bits that form a coded representation of the video data. The bitstream may include a coded picture and associated data. A coded picture is a coded representation of a picture. The associated data may include a sequence parameter set, a picture parameter set, and other syntax structures. The I / O interface 116 may include a modulator / demodulator (modem) and / or a transmitter. The coded video data may be transmitted directly over the network 130a to the destination device 120 via the I / O interface 116. The coded video data may also be stored in a storage medium / server 130b for access by the destination device 120.

[0062] The destination device 120 may include an I / O interface 126 , a video decoder 124 , and a display device 122 .

[0063] The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may obtain the encoded video data from the source device 110 or the storage medium / server 130b. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to a user. The display device 122 may be integrated with the destination device 120 or may be external to the destination device 120 and configured to interface with an external display device.

[0064] The video encoder 114 and the video decoder 124 may operate according to a video compression standard, such as the High Efficiency Video Coding (HEVC) standard, the Versatile Video Coding (VVC) standard, and other current and / or future standards.

[0065] FIG. 5 is a block diagram illustrating an example of a video encoder 200, which may be the video encoder 114 of the system 100 illustrated in FIG.

[0066] Video encoder 200 may be configured to perform any or all of the techniques of this disclosure. In the example of FIG. 5, video encoder 200 includes multiple functional components. The techniques described in this disclosure may be shared among various components of video encoder 200. In some examples, a processor may be configured to perform any or all of the techniques described in this disclosure.

[0067] Functional components of the video encoder 200 may include a partition unit 201, a prediction unit 202, which may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205, and an intra prediction unit 206, a residual generation unit 207, a transform unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transform unit 211, a reconstruction unit 212, a buffer 213, and an entropy encoding unit 214.

[0068] In other examples, video encoder 200 may include more, fewer, or different functional components. In an example, prediction unit 202 may include an Intra Block Copy (IBC) unit. The IBC unit may perform prediction in an IBC mode, where at least one reference picture is the picture in which the current video block is located.

[0069] Furthermore, some components, such as the motion estimation unit 204 and the motion compensation unit 205, may be highly integrated, but are depicted separately in the example of FIG. 5 for purposes of illustration.

[0070] Partition unit 201 may partition a picture into one or more video blocks. Video encoder 200 and video decoder 300 may support a variety of video block sizes.

[0071] The mode selection unit 203 may select one of intra or inter coding modes, for example based on an error result, and provide the resulting intra or inter coded block to a residual generation unit 207 that generates residual block data, and to a reconstruction unit 212 that reconstructs the coded block for use as a reference picture. In some examples, the mode selection unit 203 may select a Combination of Intra and Inter Prediction (CIIP) mode, in which prediction is based on an inter prediction signal and an intra prediction signal. The mode selection unit 203 may also select a resolution (e.g., sub-pixel or integer pixel precision) for the motion vector of the block in the case of inter prediction.

[0072] To perform inter prediction on the current video block, motion estimation unit 204 may generate motion information for the current video block by comparing one or more reference frames from buffer 213 with the current video block. Motion compensation unit 205 may determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from buffer 213 other than the picture associated with the current video block.

[0073] Motion estimation unit 204 and motion compensation unit 205 may perform different operations for the current video block depending on whether the current video block is an I slice, a P slice, or a B slice, for example.

[0074] In some examples, motion estimation unit 204 may perform unidirectional prediction for the current video block, and motion estimation unit 204 may look for a reference video block for the current video block from a reference picture in list 0 or list 1. Motion estimation unit 204 may then generate a reference index indicating a reference picture in list 0 or list 1 that contains the reference video block, and a motion vector indicating a spatial displacement between the current video block and the reference video block. Motion estimation unit 204 may output the reference index, the prediction direction indicator, and the motion vector as motion information for the current video block. Motion compensation unit 205 may generate a predicted video block for the current block based on the reference video block indicated by the motion information of the current video block.

[0075] In another example, motion estimation unit 204 may perform bidirectional prediction for the current video block, and motion estimation unit 204 may look for a reference video block for the current video block from a reference picture in list 0 and may look for another reference video block for the current video block from a reference picture in list 1. Motion estimation unit 204 may then generate reference indexes indicating the reference pictures in list 0 and list 1 that contain reference video blocks, and motion vectors indicating the spatial displacement between the reference video blocks and the current video block. Motion estimation unit 204 may output the reference index and the motion vector of the current video block as motion information of the current video block. Motion compensation unit 205 may generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0076] In some examples, the motion estimation unit 204 may output a full set of motion information for the decoding process of the decoder.

[0077] In some examples, motion estimation unit 204 may not output a full set of motion information for the current video. Rather, motion estimation unit 204 may signal the motion information of the current video block with reference to motion information of other video blocks. For example, motion estimation unit 204 may determine that the motion information of the current video block is sufficiently similar to the motion information of a neighboring video block.

[0078] In one example, motion estimation unit 204 may indicate in a syntax structure associated with the current video block a value that indicates to video decoder 300 that the current video block has the same motion information as other video blocks.

[0079] In other examples, motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. Video decoder 300 may use the motion vector and the motion vector difference of the indicated video block to determine the motion vector of the current video block.

[0080] As described above, video encoder 200 may predictively signal motion vectors. Two examples of predictive signaling techniques that may be implemented by video encoder 200 include Advanced Motion Vector Prediction (AMVP) and merge mode signaling.

[0081] Intra prediction unit 206 may perform intra prediction on the current video block. When intra prediction unit 206 performs intra prediction on the current video block, intra prediction unit 206 may generate predictive data for the current video block based on decoded samples of other video blocks in the same picture. The predictive data for the current video block may include a predicted video block and various syntax elements.

[0082] Residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., as indicated by a minus sign) a predicted video block of the current video block from the current video block. The residual data for the current video block may include residual video blocks that correspond to different sample components of the samples in the current video block.

[0083] In other examples, for a current video block, eg, in skip mode, residual data for the current video block may not exist and residual generation unit 207 may not perform a subtraction operation.

[0084] Transform processing unit 208 may generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to a residual video block associated with the current video block.

[0085] After transform processing unit 208 generates a transform coefficient video block associated with the current video block, quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0086] Inverse quantization unit 210 and inverse transform unit 211 may apply inverse quantization and inverse transform, respectively, to the transform coefficient video block to reconstruct a residual video block from the transform coefficient video block. Reconstruction unit 212 may add the reconstructed residual video block to corresponding samples from one or more predicted video blocks generated by prediction unit 202 to generate a reconstructed video block related to the current block for storage in buffer 213.

[0087] After reconstruction unit 212 reconstructs the video blocks, a loop filtering operation may be performed to reduce video blocking artifacts in the video blocks.

[0088] Entropy encoding unit 214 may receive data from other functional components of video encoder 200. Once entropy encoding unit 214 receives the data, it may perform one or more entropy encoding operations to generate entropy encoded data and output a bitstream including the entropy encoded data.

[0089] FIG. 6 is a block diagram illustrating an example of a video decoder 300, which may be the video decoder 124 of the system 100 illustrated in FIG.

[0090] The video decoder 300 may be configured to perform any or all of the techniques of this disclosure. In the example of FIG. 6, the video decoder 300 includes multiple functional components. The techniques described in this disclosure may be shared among various components of the video decoder 300. In some examples, a processor may be configured to perform any or all of the techniques described in this disclosure.

[0091] In the example of Figure 6, the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, a reconstruction unit 306, and a buffer 307. The video decoder 300 may, in some examples, perform a decoding pass that is generally inverse to the encoding pass described with respect to the video encoder 200 (Figure 5).

[0092] The entropy decoding unit 301 may retrieve an encoded bitstream. The encoded bitstream may include entropy coded video data (e.g., coded blocks of video data). The entropy decoding unit 301 may decode the entropy coded video data, and from the entropy decoded video data, the motion compensation unit 302 may determine motion information including motion vectors, motion vector precision, reference picture list index, and other motion information. The motion compensation unit 302 may determine such information, for example, by implementing AMVP and merge mode.

[0093] The motion compensation unit 302 may optionally perform interpolation based on an interpolation filter to generate the motion compensated block. An identifier for the interpolation filter used with sub-pixel precision may be included in the syntax element.

[0094] Motion compensation unit 302 may use the interpolation filters used by video encoder 200 during encoding of the video block to calculate interpolated values ​​for sub-integer pixels of the reference block. Motion compensation unit 302 may determine the interpolation filters used by video encoder 200 according to received syntax information and use the interpolation filters to generate the predictive block.

[0095] The motion compensation unit 302 may use some of the syntax information to determine the size of the blocks used to code the frames and / or slices of the coded video sequence, partition information describing how each macroblock of a picture of the coded video sequence is partitioned, a mode indicating how each partition is coded, one or more reference frames (and reference frame lists) for each inter-coded block, and other information for decoding the coded video sequence.

[0096] The intra prediction unit 303 may use, for example, an intra prediction mode received in the bitstream to form a prediction block from spatially neighboring blocks. The inverse quantization unit 304 inverse quantizes, i.e., dequantizes, the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 305 applies an inverse transform.

[0097] Reconstruction unit 306 may add a corresponding prediction block generated by motion compensation unit 302 or intra prediction unit 303 to the residual block to form a decoded block. Optionally, a deblocking filter may also be applied to filter the decoded block to remove blockiness artifacts. The decoded video block is then stored in buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction, as well as generating the decoded video for presentation on a display device.

[0098] A list of solutions preferred by some embodiments is given below.

[0099] The following illustrates an example embodiment of the techniques discussed in the previous section (eg, item 1).

[0100] Solution 1. 1. A method of video processing (e.g., method 700 depicted in FIG. 3), comprising: performing (702) a conversion between a video including video pictures and a coded representation of said video, the coded representation follows formatting rules; the format rules specify that a field indicating that the video is multi-view video is included in the coded representation. method.

[0101] Solution 2. the field being included in a supplemental enhancement information portion of the coded representation; Solution 1 method.

[0102] Solution 3. the field being included in a video usability information portion of the coded representation; Solution 1 method.

[0103] The following illustrates an example embodiment of the techniques discussed in the previous section (eg, item 2).

[0104] Solution 4. 1. A method of video processing, comprising: performing a conversion between a video including video pictures and said video coded representation; the coded representation follows formatting rules; the format rules specify that a field is included in the coded representation indicating that the video is coded in multiple video layers. method.

[0105] Solution 5. the field being included in a supplemental enhancement information portion of the coded representation; Solution 4 method.

[0106] Solution 6. the field being included in a video usability information portion of the coded representation; Solution 4 method.

[0107] Solution 7. The converting comprises generating the coded representation from the video. Any of solutions 1 to 6.

[0108] Solution 8. said converting comprising decoding said coded representation to generate said video. Any of solutions 1 to 6.

[0109] Solution 9. A video decoding device comprising a processor configured to implement the method according to one or more of solutions 1 to 8.

[0110] Solution 10. A video encoding device comprising a processor configured to implement the method according to one or more of solutions 1 to 8.

[0111] Solution 11. A computer program product having stored thereon computer code which, when executed by a processor, causes the processor to implement a method according to any of solutions 1 to 8.

[0112] Solution 12. A computer readable medium storing a coded representation generated according to any of solutions 1 to 8.

[0113] A method, apparatus, or system as described herein.

[0114] In the solutions described herein, an encoder may comply with the format rules by generating a coded representation in accordance with the format rules, and a decoder may use the format rules to parse syntax elements in the coded representation, knowing whether or not the syntax elements comply with the format rules, to generate decoded video.

[0115] 8 is a flow chart of an example method of video processing. Operation 802 includes performing a conversion between video and a video bitstream according to a format rule, where the format rule specifies that a supplemental enhancement information field or a video usability information syntax structure included in the bitstream indicates whether the bitstream has a multiview bitstream in which multiple views are coded in multiple video layers.

[0116] In some embodiments, the formatting rules define that the supplemental enhancement information field is included in the scalability dimension information in the supplemental enhancement information message in the bitstream. In some embodiments, the formatting rules define that the supplemental enhancement information message includes a first flag indicating whether the bitstream is a multiview bitstream. In some embodiments, the formatting rules define that the supplemental enhancement information message includes a view identifier for each video layer of a plurality of video layers of the bitstream. In some embodiments, the formatting rules define that the supplemental enhancement information message includes a length in bits of the view identifier for each video layer.

[0117] In some embodiments, the formatting rules define that the supplemental enhancement information message includes a second flag indicating whether a view identifier is included in the bitstream for each video layer. In some embodiments, the formatting rules define that the scalability dimension information in the supplemental enhancement information message provides information about a series of access units that includes, in decoding order, an access unit that includes the second scalability dimension information in the second supplemental enhancement information message followed by zero or more access units including all subsequent access units up to but not including the subsequent access unit that includes the third scalability dimension information in the third supplemental enhancement information message. In some embodiments, the formatting rules define that the supplemental enhancement information field is included in a video usability information syntax structure in the bitstream. In some embodiments, the bitstream is a versatile video coding bitstream. In some embodiments, the step of performing the conversion includes encoding the video into the bitstream. In some embodiments, the step of performing the conversion includes generating the bitstream from the video, the method further including storing the bitstream in a non-transitory computer-readable recording medium. In some embodiments, the step of performing the conversion includes decoding the video from the bitstream.

[0118] 9 is a flowchart of an exemplary method of video processing. Operation 902 includes performing a conversion between a video and a video bitstream according to a format rule, where the format rule specifies that a supplemental enhancement information field included in the bitstream indicates whether the bitstream has one or more video layers representing auxiliary information.

[0119] In some embodiments, the formatting rules define that the supplemental enhancement information field is included in the scalability dimension information in the supplemental enhancement information message in the bitstream. In some embodiments, the formatting rules define that the supplemental enhancement information message includes a first flag indicating whether the bitstream includes auxiliary information for one or more video layers. In some embodiments, the formatting rules define that the supplemental enhancement information message includes an auxiliary identifier for each video layer of a plurality of video layers of the bitstream. In some embodiments, the formatting rules define that a first value of the auxiliary identifier for a video layer indicates that the video layer does not include auxiliary pictures.

[0120] In some embodiments, the formatting rules define that a second value of the auxiliary identifier for a video layer indicates that the type of the auxiliary information for that video layer is alpha. In some embodiments, the formatting rules define that a third value of the auxiliary identifier for a video layer indicates that the type of the auxiliary information for that video layer is depth. In some embodiments, the formatting rules define that the supplemental enhancement information message includes, for each video layer, a second flag indicating whether the auxiliary identifier is included in the bitstream. In some embodiments, the formatting rules define that a fourth value of the auxiliary identifier for a video layer indicates that the type of the auxiliary information for that video layer is occupancy.

[0121] In some embodiments, the formatting rules define that a third value of the auxiliary identifier for a video layer indicates that a type of the auxiliary information for the video layer is a geometry. In some embodiments, the formatting rules define that a third value of the auxiliary identifier for a video layer indicates that a type of the auxiliary information for the video layer is an attribute. In some embodiments, the formatting rules define that a third value of the auxiliary identifier for a video layer indicates that a type of the auxiliary information for the video layer is a texture attribute. In some embodiments, the formatting rules define that a third value of the auxiliary identifier for a video layer indicates that a type of the auxiliary information for the video layer is a material attribute. In some embodiments, the formatting rules define that a third value of the auxiliary identifier for a video layer indicates that a type of the auxiliary information for the video layer is a transparent attribute.

[0122] In some embodiments, the formatting rules define that a third value of the auxiliary identifier for a video layer indicates that the type of the auxiliary information for that video layer is a reflectance attribute. In some embodiments, the formatting rules define that a third value of the auxiliary identifier for a video layer indicates that the type of the auxiliary information for that video layer is a normal attribute. In some embodiments, the formatting rules define that the scalability dimension information in the supplemental enhancement information message provides information on a series of access units that includes, in decoding order, an access unit that includes second scalability dimension information in a second supplemental enhancement information message followed by zero or more access units including all subsequent access units up to but not including the subsequent access unit that includes the third scalability dimension information in the third supplemental enhancement information message.

[0123] In some embodiments, the format rules dictate that the supplemental enhancement information field is included in the video usability information in the bitstream. In some embodiments, the video is a Versatile Video Coding video. In some embodiments, the step of performing the conversion includes encoding the video into a bitstream. In some embodiments, the step of performing the conversion includes generating a bitstream from the video, the method further including storing the bitstream in a non-transitory computer-readable recording medium. In some embodiments, the step of performing the conversion includes decoding the video from the bitstream.

[0124] In some embodiments, a video decoding device has a processor configured to implement the methods set forth in one or more of the techniques described in this patent document. In some embodiments, a video encoding device has a processor configured to implement the methods set forth in one or more of the techniques described in this patent document. In some embodiments, a computer program product has stored thereon computer instructions that, when executed by the processor, cause the processor to perform the techniques described in this patent document. In some embodiments, a non-transitory computer-readable storage medium stores a bitstream generated according to the methods of any one of the techniques described in this patent document.

[0125] In some embodiments, a non-transitory computer readable storage medium stores instructions for causing a processor to perform a method as set forth in any of the techniques described in this patent document. In some embodiments, a method of bitstream generation includes generating a bitstream of video according to any of the techniques described in this patent document and storing the bitstream in a computer readable program medium. In some embodiments, a method, apparatus, bitstream generated according to the disclosed methods, or systems described in this patent document.

[0126] In this patent document, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during the conversion from a pixel representation of a video to a corresponding bitstream representation, or vice versa. The bitstream representation of a current video block may correspond to bits that are either at the same position or spread across different locations in the bitstream, for example as defined by a syntax. For example, a macroblock may be coded with respect to a transformed and coded error residual value, as well as with bits in a header and other fields in the bitstream. Furthermore, during the conversion, a decoder may parse the bitstream with the knowledge that some fields may be present or absent based on a decision, as described in the above solution. Similarly, an encoder may determine that a particular syntax field should be included or not included, and generate a coded representation by including or excluding the syntax field in the coded representation accordingly.

[0127] Some embodiments of the disclosed techniques include making a decision or determination to enable a video processing tool or mode. In an example, if a video processing tool or mode is enabled, the encoder will use or implement that tool or mode in processing blocks of video, but may not necessarily modify the resulting bitstream based on the use of the tool or mode. That is, conversion of blocks of video to a bitstream representation of video will use the video processing tool or mode if the tool or mode is enabled based on the decision or determination. In another example, if a video processing tool or mode is enabled, the decoder will process the bitstream knowing that the bitstream has been modified based on the video processing tool or mode. That is, conversion of a bitstream representation of video to blocks of video will be performed using the video processing tool or mode enabled based on the decision or determination.

[0128] Some embodiments of the disclosed techniques include making a decision or determination to disable a video processing tool or mode. In an example, when a video processing tool or mode is disabled, an encoder will not use that tool or mode in converting blocks of video to a bitstream representation of the video. In another example, when a video processing tool or mode is disabled, a decoder will process the bitstream with the video processing tool or mode that was disabled based on the decision or determination, knowing that the bitstream has not been altered.

[0129] The disclosed and other solutions, examples, embodiments, modules and functional operations described herein can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in a combination of one or more of them. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded in a computer-readable medium for execution by or control of the operation of a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter affecting a machine-readable propagated signal, or one or more combinations thereof. The term "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, such as code constituting a processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof. A propagated signal is an artificially generated signal, for example a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to an appropriate receiver device.

[0130] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a single file dedicated to the program in question, or in multiple coordinated files (e.g., a file storing one or more modules, subprograms, or portions of code), in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0131] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry, for example a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC).

[0132] Processors suitable for executing a computer program include, by way of example, both general purpose and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. Typically, a processor will read instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, e.g., magnetic, optical, magnetic disks, or optical disks, or will be operatively coupled to receive data from and / or transfer data to such one or more mass storage devices. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include, by way of example, semiconductor memory devices, such as erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), and flash memory devices; magnetic disks, such as internal hard disks or removable disks; optical magnetic disks; and all forms of non-volatile memory, media, and memory devices, including compact disk-type read-only memories (CD ROMs) and digital versatile disk-type read-only memories (DVD-ROMs). The processor and memory may be enhanced by, or incorporated in, dedicated logic circuitry.

[0133] Although the specification contains numerous details, they should not be construed as limitations on the scope of any subject matter or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular technology. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as operating in a particular combination, and even initially claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0134] Similarly, although operations may be depicted in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown, or in a sequential order, or that all of the depicted operations be performed, to achieve desired results. Further, the separation of various system components in the embodiments described herein should not be understood as requiring such separation in all embodiments.

[0135] Only a few implementations and examples have been described; other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.

Claims

1. 1. A method for processing video data, comprising the steps of: performing a conversion between the video and a bitstream of the video according to a format rule; The format rules provide that a supplemental enhancement information (SEI) field included in the bitstream indicates whether the bitstream is allowed to have one or more video layers representing auxiliary information; The format rule defines that the supplemental enhancement information (SEI) field is included in scalability dimension information in a supplemental enhancement information (SEI) message in the bitstream. method.

2. The format rules define that the supplemental enhancement information (SEI) message includes an auxiliary identifier for each video layer of one or more video layers of the bitstream. The method of claim 1.

3. The format rule specifies that a first value of an auxiliary identifier for a video layer indicates that the video layer does not include auxiliary pictures. The method of claim 2.

4. the first value is equal to 0; The method of claim 3.

5. The format rule specifies that a second value of the auxiliary identifier of a video layer indicates that the auxiliary information type of the video layer is an alpha plane.

5. The method according to any one of claims 2 to 4.

6. the second value is equal to 1; The method according to claim 5.

7. The format rule specifies that a third value of the auxiliary identifier of a video layer indicates that the type of the auxiliary information of the video layer is a depth picture.

7. The method according to any one of claims 2 to 6.

8. the third value is equal to 2; The method of claim 7.

9. the video is a Versatile Video Coding video; 9. The method according to any one of claims 1 to 8.

10. said step of performing said conversion includes encoding said video into said bitstream; 10. The method according to any one of claims 1 to 9.

11. and performing the conversion step includes decoding the video from the bitstream.

10. The method according to any one of claims 1 to 9.

12. a processor and a non-transitory memory containing instructions; The instructions, when executed by the processor, cause the processor to: performing a conversion between the video and a bitstream of the video according to a format rule; The format rules provide that a supplemental enhancement information (SEI) field included in the bitstream indicates whether the bitstream is allowed to have one or more video layers representing auxiliary information; The format rule defines that the supplemental enhancement information (SEI) field is included in scalability dimension information in a supplemental enhancement information (SEI) message in the bitstream. Device.

13. storing instructions for causing a processor to convert between the video and a bitstream of the video in accordance with a format rule; The format rules provide that a supplemental enhancement information (SEI) field included in the bitstream indicates whether the bitstream is allowed to have one or more video layers representing auxiliary information; The format rule defines that the supplemental enhancement information (SEI) field is included in scalability dimension information in a supplemental enhancement information (SEI) message in the bitstream. A non-transitory computer-readable storage medium.

14. 1. A method for storing a video bitstream, comprising: generating the bitstream of the video according to a format rule; storing the bitstream on a computer-readable recording medium; The format rules provide that a supplemental enhancement information (SEI) field included in the bitstream indicates whether the bitstream is allowed to have one or more video layers representing auxiliary information; The format rule defines that the supplemental enhancement information (SEI) field is included in scalability dimension information in a supplemental enhancement information (SEI) message in the bitstream. method.

Images