Scalability dimension information in video encoding and decoding
By introducing the SDI view identifier length minus L syntax element, the problem of view identifier length being zero is solved, ensuring the effective application of SDI messages and the reasonable nesting of SEI messages for multi-view information retrieval, thereby improving the efficiency and accuracy of video encoding and decoding.
Patent Information
- Application Number
- CN202280026709.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-04-02
- Filing Date
- 2022-04-02
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2042-04-02
AI Technical Summary
In existing video codec standards, the zero-length view identifier renders the existence of the multi-view acquisition information (SEI) message meaningless, and when the SDI message is absent, the nesting of the multi-view acquisition information (SEI) message and the auxiliary information (SEI) message is invalid.
The Scalable Dimension Information (SDI) view identifier length minus L syntax element is used to prevent the view identifier length from being zero, and to prevent nesting of multiple view acquisition information SEI messages and auxiliary information SEI messages when the SDI message is not present.
This effectively prevents the error of the view identifier having a length of zero, ensures the proper application of SDI messages, and avoids invalid nesting of SEI messages for multiple views to obtain information, thereby improving the efficiency and accuracy of video encoding and decoding.
Smart Images

Figure CN117157980B_ABST
Abstract
Description
[0001] Cross-reference to related applications
[0002] This patent application claims priority to International Application No. PCT / CN2021 / 085292, filed on April 2, 2021, entitled "Improvement of Scalability Dimensional Information", by Beijing ByteDance Network Technology Co., Ltd., which is incorporated herein by reference. Technical Field
[0003] This disclosure relates generally to video encoding and decoding, and more particularly to Supplemental Enhancement Information (SEI) messages used in image / video encoding and decoding. Background Technology
[0004] Digital video consumes the largest share of bandwidth on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth demand for digital video is expected to continue to grow. Summary of the Invention
[0005] The disclosed aspects / embodiments provide techniques for using a Scalable Dimension Information (SDI) view identifier length minus L syntax element to prevent the length of the SDI view ID value syntax element specifying the view identifier of the i-th layer in the bitstream from being zero. The disclosed aspects / embodiments further provide techniques for preventing the bitstream from having Multi-View Acquisition Information Supplemental Enhancement (SEI) messages or auxiliary information SEI messages when SDI messages are not present in the bitstream. The disclosed aspects / embodiments also provide techniques for preventing Multi-View Acquisition Information SEI messages from being scalably nested.
[0006] The first aspect relates to a method for processing video data. The method includes: a method for processing video data comprising: using Scalability Dimension Information (SDI) Supplemental Enhancement Information (SEI) messages to indicate the length of the SDI view identifier minus the L syntax element; and performing a conversion between a video media file and a bitstream based on the SDI SEI messages.
[0007] Optionally, in any of the foregoing aspects, another implementation of the aspect specifies that the SDI view identifier length minus L syntax element is configured to prevent the length of the SDI view identifier value syntax element specifying the view identifier of the i-th layer in the bitstream from being zero.
[0008] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that L equals 1.
[0009] Alternatively, in any of the foregoing aspects, another implementation of the aspect specifies that the SDI view identifier length minus L syntax element is designated as sdi_view_id_len_minus1.
[0010] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that the SDI view identifier value syntax element is designated as sdi_view_id_val[i].
[0011] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that the length of the SDI view identifier value syntax element is specified by subtracting L from the SDI view identifier length syntax element and adding 1.
[0012] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that the SDI view identifier length minus L syntax element is encoded and decoded as an N-bit unsigned integer.
[0013] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that N equals 4.
[0014] Optionally, in any of the foregoing aspects, another implementation of the aspect specifies that the SDI view identifier length minus L syntax element is encoded or decoded as a syntax element using an N-bit fixed-pattern bit string, an N-bit signed integer, a truncated binary, a signed integer K-order exponent Golombian-encoded syntax element, where K equals 0, or an unsigned integer M-order exponent Golombian-encoded syntax element, where M equals 0.
[0015] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that the bitstream is a bitstream within a range.
[0016] Optionally, in any of the foregoing aspects, another implementation of the aspect specifies that the multi-view information SEI message and the auxiliary information SEI message are not present in the codec video sequence (CVS) unless the SDI SEI message is present in the CVS.
[0017] Optionally, in any of the foregoing aspects, another implementation of that aspect specifies that the multi-view information SEI message includes a multi-view acquisition information SEI message.
[0018] Optionally, in any of the foregoing aspects, another implementation of that aspect specifies that the auxiliary information SEI message includes the depth representation information SEI message.
[0019] Optionally, in any of the foregoing aspects, another implementation of that aspect specifies that the auxiliary information SEI message includes the α channel information SEI message.
[0020] Optionally, in any of the foregoing aspects, another implementation of the aspect specifies that when a multi-view information SEI message or an auxiliary information SEI message is present in the bitstream, one or more of the SDI multi-view information flag and the SDI auxiliary information flag are equal to 1.
[0021] Optionally, in any of the foregoing aspects, another implementation of the aspect specifies that the multi-view information SEI message includes a multi-view acquisition information SEI message, and wherein the multi-view acquisition information SEI message is not scalable and nested.
[0022] Alternatively, in any of the foregoing aspects, another implementation of that aspect stipulates that SEI messages in the bitstream with a payload type equal to 179 are constrained not to be included in scalable nested SEI messages.
[0023] Optionally, in any of the foregoing aspects, another implementation of that aspect stipulates that SEI messages in the bitstream having a payload type equal to 3, 133, 179, 180, or 205 are constrained not to be included in scalable nested SEI messages.
[0024] The second aspect relates to an apparatus for processing video data, including a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: use Scalable Dimension Information (SDI) Supplemental Enhancement Information (SEI) messages to indicate the length of the SDI view identifier minus the L syntax element; and perform conversion between a video media file and a bitstream based on the SDI SEI messages.
[0025] The third aspect relates to a non-transitory computer-readable medium, including a computer program product for use by an encoding / decoding apparatus, the computer program product including computer-executable instructions stored on the non-transitory computer-readable medium, the computer-executable instructions causing the encoding / decoding apparatus, when executed by one or more processors, to: indicate the length of the SDI view identifier minus the L syntax element using Scalable Dimension Information (SDI) Supplemental Enhancement Information (SEI) messages; and to perform conversion between video media files and bitstreams based on the SDI SEI messages.
[0026] The fourth aspect relates to a non-transitory computer-readable storage medium that stores instructions that cause a processor to: use Scalable Dimension Information (SDI) Supplemental Enhancement Information (SEI) messages to indicate the length of an SDI view identifier minus an L syntax element; and perform conversion between a video media file and a bitstream based on the SDI SEI messages.
[0027] The fifth aspect relates to a non-transitory computer-readable recording medium storing a bitstream of video generated by a method performed by a video processing apparatus, wherein the method includes: using Scalable Dimension Information (SDI) Supplemental Enhancement Information (SEI) messages to indicate the length of an SDI view identifier minus an L syntax element; and converting between a video media file and a bitstream based on the SDI SEI messages.
[0028] The sixth aspect relates to a method for storing a bitstream of video, comprising: using Scalable Dimension Information (SDI) Supplemental Enhancement Information (SEI) messages to indicate the length of an SDI view identifier minus an L syntax element; generating a bitstream based on the SDI SEI messages; and storing the bitstream in a non-transitory computer-readable recording medium.
[0029] For clarity, any of the foregoing embodiments may be combined with any one or more other foregoing embodiments to form new embodiments within the scope of this disclosure.
[0030] These and other features will become clearer from the following detailed description taken in conjunction with the accompanying drawings and claims. Attached Figure Description
[0031] To gain a more complete understanding of this disclosure, reference is now made to the following brief description in conjunction with the accompanying drawings and detailed specifications, wherein the same reference numerals represent the same parts.
[0032] Figure 1 An example of multi-layer encoding and decoding for spatial scalability is illustrated.
[0033] Figure 2 The illustration shows an example of multi-layer codec using Output Layer Set (OLS).
[0034] Figure 3 An example of a video bitstream is illustrated.
[0035] Figure 4 This is a block diagram showing an example video processing system.
[0036] Figure 5 This is a block diagram of a video processing device.
[0037] Figure 6 This is a block diagram illustrating an example video encoding / decoding system.
[0038] Figure 7 This is a block diagram illustrating an example of a video encoder.
[0039] Figure 8 This is a block diagram illustrating an example of a video decoder.
[0040] Figure 9 This is a method for encoding and decoding video data according to embodiments of the present disclosure. Detailed Implementation
[0041] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods can be implemented using any number of techniques, whether currently known or existing. This disclosure should not be limited in any way to the exemplary implementations, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but modifications can be made within the full scope of the appended claims and their equivalents.
[0042] Video codec standards have primarily evolved through the development of standards from the renowned International Telecommunication Union-Telecommunication (ITU-T) and the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). ITU-T developed H.261 and H.263, while ISO / IEC developed Moving Picture Experts Group (MPEG)-1 and MPEG-4 video. These two organizations jointly developed the H.262 / MPEG-2 video standard and the H.264 / MPEG-4 Advanced Video Codec (AVC) and H.265 / High-Efficiency Video Codec (HEVC) standards. See ITU-T and ISO / IEC, “High-Efficiency Video Codec,” Rec. ITU-T H.265|ISO / IEC 23008-2 (current version). Since H.262, video codec standards have been based on a hybrid video codec architecture, utilizing temporal prediction plus transform coding. To explore future video codec technologies beyond HEVC, the Video Codec Experts Group (VCEG) and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Subsequently, JVET adopted several new methods and incorporated them into reference software called the Joint Exploratory Model (JEM). See J. Chen, E. Alshina, G.S. Sullivan, J.-R. Ohm, J. Boyce, “Algorithmic Description of Joint Exploratory Test Model 7 (JEM7)”, JVET-G1001, August 2017. When the Multi-Functional Video Coding (VVC) project was officially launched, JVET was later renamed the Joint Video Experts Team (JVET). VVC is a new codec standard aiming to reduce the bit rate by 50% compared to HEVC, and it was finalized by JVET at its 19th meeting, which concluded on July 1, 2020. See Rec. ITU-T H.266|ISO / IEC 23090-3, “Multi-Functional Video Coding”, 2020.
[0043] The VVC standard (ITU-T H.266|ISO / IEC 23090-3) and the related Multifunctional Supplemental Enhancement Information (VSEI) standard (ITU-T H.274|ISO / IEC 23002-7) have been designed for the widest range of applications, including traditional uses (such as television broadcasting, video conferencing, or playback from storage media), as well as newer and more advanced use cases (such as adaptive bitrate streaming, video region extraction, compositing and merging of content from multiple codec video bitstreams, multi-view video, scalable layered codecs, and viewport-adaptive 360° immersive media). See B. Bross, J. Chen, S. Liu, Y.-K. Wang (ed.), “Multifunctional Video Coding (Draft 10)”, JVET-S2001, Rec.ITU-T Rec.H.274|ISO / IEC 23002-7, “Multifunctional Supplemental Enhancement Information Messages for Encoding and Decoding Video Bitstreams”, 2020, and J. Boyce, V. Drugeon, G. Sullivan, Y.-K. Wang (ed.), “Multifunctional Supplemental Enhancement Information Messages for Encoding and Decoding Video Bitstreams (Draft 5)”, JVET-S2007.
[0044] The Basic Video Codec (EVC) standard (ISO / IEC 23094-1) is another video codec standard recently developed by MPEG.
[0045] Figure 1 This is a schematic diagram illustrating an example of layer-based prediction 100. Layer-based prediction 100 is compatible with unidirectional inter-frame prediction and / or bidirectional inter-frame prediction, but it can also be performed between images in different layers.
[0046] Layer-based prediction 100 is applied between images 111, 112, 113, and 114 and images 115, 116, 117, and 118 in different layers. In the example shown, images 111, 112, 113, and 114 are part of layer N+1 132, and images 115, 116, 117, and 118 are part of layer N 131. Layers such as layer N 131 and / or layer N+1 132 are a set of images that are associated with similar characteristic values, such as similar size, quality, resolution, signal-to-noise ratio, capacity, etc. In the example shown, layer N+1 132 is associated with a larger image size than layer N 131. Therefore, in this example, images 111, 112, 113, and 114 in layer N+1 132 have a larger image size (e.g., larger height and width, and therefore more samples) than images 115, 116, 117, and 118 in layer N 131. However, such images can be separated between layers N+1 132 and N 131 using other features. Although only two layers, N+1 132 and N 131, are shown, a set of images can be divided into any number of layers based on associated characteristics. Layers N+1 132 and N 131 can also be represented by layer IDs. A layer ID is a data item associated with an image and indicates that the image is part of the indicated layer. Therefore, each image 111-118 can be associated with a corresponding layer ID to indicate which layer N+1 132 or layer N 131 contains the corresponding image.
[0047] Images 111-118 in different layers 131-132 are configured to be displayed alternately. This allows images 111-118 in different layers 131-132 to share the same temporal identifier (ID) and to be included in the same access unit (AU) 106. As used herein, an AU is a set of one or more codec images associated with the same display time for output from the decoded image buffer (DPB). For example, if a smaller image is needed, the decoder can decode and display image 115 at the current display time, or if a larger image is needed, the decoder can decode and display image 111 at the current display time. Thus, images 111-114 at the higher layer N+1 132 contain substantially the same image data as their corresponding images 115-118 at the lower layer N 131 (although the image sizes are different). Specifically, image 111 contains substantially the same image data as image 115, image 112 contains substantially the same image data as image 116, and so on.
[0048] Images 111-118 can be encoded / decoded by referencing other images 111-118 in the same layer N 131 or N+1 132. Encoding / decoding an image by referencing another image in the same layer results in inter-frame prediction 123, which is compatible with unidirectional and / or bidirectional inter-frame prediction. Inter-frame prediction 123 is depicted by a solid arrow. For example, image 113 can be encoded / decoded using inter-frame prediction 123 with one or both of images 111, 112, and / or 114 in layer N+1 132 as references, where one image is referenced for unidirectional inter-frame prediction and / or both images are referenced for bidirectional inter-frame prediction. Similarly, image 117 can be encoded / decoded using inter-frame prediction 123 with one or both of images 115, 116, and / or 118 in layer N 131 as references, where one image is referenced for unidirectional inter-frame prediction and / or both images are referenced for bidirectional inter-frame prediction. When performing inter-frame prediction 123, an image may be referred to as a reference image when it is used as a reference for another image in the same layer. For example, image 112 may be a reference image used for encoding and decoding image 113 according to inter-frame prediction 123. In a multi-layer context, inter-frame prediction 123 may also be referred to as intra-layer prediction. Thus, inter-frame prediction 123 is a mechanism for encoding and decoding samples of the current image by referencing indicator samples in a reference image that is different from the current image, where the reference image and the current image are in the same layer.
[0049] Images 111-118 can also be encoded / decoded by referencing other images 111-118 in different layers. This process is called inter-layer prediction 121 and is indicated by a dashed arrow. Inter-layer prediction 121 is a mechanism for encoding / decoding samples of the current image by referencing indicator samples in a reference image, where the current image and the reference image are in different layers and therefore have different layer IDs. For example, an image in a lower layer N 131 can be used as a reference image to encode / decode the corresponding image in a higher layer N+1 132. As a concrete example, image 111 can be encoded / decoded based on inter-layer prediction 121 with reference to image 115. In this case, image 115 is used as the inter-layer reference image. The inter-layer reference image is the reference image used for inter-layer prediction 121. In most cases, inter-layer prediction 121 is constrained such that the current image (e.g., image 111) can only use inter-layer reference images (e.g., image 115) included in the same AU 106 and located in a lower layer. When multiple layers (e.g., more than two) are available, inter-layer prediction 121 can encode / decode the current image based on multiple inter-layer reference images at a lower level than the current image.
[0050] The video encoder can employ layer-based prediction 100 to encode images 111-118 via various combinations and / or permutations of inter-frame prediction 123 and inter-layer prediction 121. For example, image 115 can be encoded / decoded based on intra-frame prediction. Then, using image 115 as a reference image, images 116-118 can be encoded / decoded based on inter-frame prediction 123. Furthermore, image 111 can be encoded / decoded based on inter-layer prediction 121 using image 115 as an inter-layer reference image. Then, using image 111 as a reference image, images 112-114 can be encoded / decoded based on inter-frame prediction 123. Thus, the reference image can be used as both a single-layer reference image and an inter-layer reference image for different encoding / decoding mechanisms. By encoding / decoding higher-layer N+1 132 images based on lower-layer N 131 images, higher-layer N+1 132 can avoid intra-frame prediction, which has significantly lower encoding / decoding efficiency than inter-frame prediction 123 and inter-layer prediction 121. In this way, the low encoding / decoding efficiency of intra-frame prediction can be limited to the smallest / lowest quality picture, and therefore limited to encoding / decoding the minimum amount of video data. Pictures used as reference pictures and / or inter-layer reference pictures can be indicated in entries of the reference picture list(s) included in the reference picture list structure.
[0051] Figure 1 Each AU 106 in the array may contain several pictures. For example, one AU 106 may contain pictures 111 and 115. Another AU 106 may contain pictures 112 and 116. In practice, each AU 106 is a collection of one or more codec pictures associated with the same display time (e.g., the same time-domain ID) for output from the decoded picture buffer (DPB) (e.g., for display to the user). Each Access Unit Delimiter (AUD) 108 is an indicator or data structure used to indicate the start of an AU (e.g., AU 108) or the boundary between AUs.
[0052] Previous H.26x video codec families have provided support for scalability in multiple tiers separate from single-layer codecs. Scalable Video Codec (SVC) is a scalable extension of AVC / H.264, providing support for spatial, temporal, and quality scalability. For SVC, a signaling notification flag is included in each macroblock (MB) in the Enhancement Layer (EL) picture to indicate whether the EL MB is predicted using juxtaposed blocks from lower layers. Predictions from juxtaposed blocks can include texture, motion vectors, and / or codec modes. SVC implementations cannot directly reuse unmodified H.264 / AVC implementations in their design. The SVC EL macroblock syntax and decoding process differ from the H.264 / AVC syntax and decoding process.
[0053] Scalable HEVC (SHVC) is an extension of the HEVC / H.265 standard, providing support for spatial and quality scalability. Multi-View HEVC (MV-HEVC) is an extension of HEVC / H.265, providing support for multi-view scalability, and 3DHEVC (3D-HEVC) is an extension of HEVC / H.264, providing support for three-dimensional (3D) video encoding and decoding, which is more advanced and efficient than MV-HEVC. Note that temporal scalability is included as part of a single-layer HEVC codec. The design of multi-layer extensions of HEVC employs the idea that decoded pictures used for inter-layer prediction come only from the same AU and are treated as Long-Term Reference Pictures (LTRPs), and are assigned reference indices in a list of reference pictures (multiple) along with other temporal reference pictures in the current layer. Inter-layer prediction (ILP) is implemented at the prediction unit (PU) level by setting the value of the reference index to reference the inter-layer reference pictures (multiple) in the list of reference pictures (multiple).
[0054] It is worth noting that both reference image resampling and spatial scalability features require resampling of the reference image or a portion thereof. Reference image resampling (RPR) can be implemented at the image level or the codec block level. However, when RPR is referred to as a codec feature, it is a feature used for single-layer codecs. Even so, from a codec design perspective, it is possible, or even preferred, to use the same resampling filter for both single-layer codec RPR features and multi-layer codec spatial scalability features.
[0055] Figure 2 An example of layer-based prediction 200 using Output Layer Set (OLS) is illustrated. Layer-based prediction 100 is compatible with unidirectional inter-frame prediction and / or bidirectional inter-frame prediction, but can also be performed between images in different layers. Figure 2 Layer-based prediction is similar to Figure 1 Therefore, for the sake of brevity, the full description of layer-based predictions will not be repeated.
[0056] Figure 2 Some layers in the codec video sequence (CVS) 290 are contained in an OLS. An OLS is a set of layers in which one or more layers are specified as output layers. Output layers are the layers of the output OLS. Figure 2Three different OLSs are depicted: OLS 1, OLS 2, and OLS 3. As shown in the figure, OLS 1 includes layers N 231 and N+1 232. Layer N 231 includes images 215, 216, 217, and 218, and layer N+1 232 includes images 211, 212, 213, and 214. OLS 2 includes layers N 231, N+1 232, N+2 233, and N+3 234. Layer N+2 233 includes images 241, 242, 243, and 244, and layer N+3 234 includes images 251, 252, 253, and 254. OLS 3 includes layers N 231, N+1 232, and N+2 233. Although three OLSs are shown, different numbers of OLSs can be used in practical applications. In the illustrated embodiment, none of the OLSs includes layer N+4 235, which contains images 261, 262, 263, and 264.
[0057] Each different OLS can contain any number of layers. Different OLSs are generated to accommodate the different encoding and decoding capabilities of various devices. For example, OLS 1, containing only two layers, can be generated to accommodate mobile phones with relatively limited encoding and decoding capabilities. On the other hand, OLS 2, containing four layers, can be generated to accommodate large-screen televisions, which can decode higher layers than mobile phones. OLS 3, containing three layers, can be generated to accommodate personal computers, laptops, or tablets, which can decode higher layers than mobile phones, but cannot decode the highest layers like those on large-screen televisions.
[0058] Figure 2 The layers in a network can be completely independent of each other. That is, each layer can be encoded and decoded without using inter-layer prediction (ILP). In this case, these layers are called syndicated layers. Figure 2 One or more layers in a network can also use ILP for encoding and decoding. These layers can be signaled as syndicated layers or as layers using ILP for encoding and decoding via flags in the Video Parameter Set (VPS). When some layers use ILP, the layer dependencies between them are also signaled in the VPS.
[0059] In an embodiment, when the layer is a cascade layer, only one layer is selected for decoding and output. In an embodiment, when some layers use ILP, all layers (e.g., the entire bitstream) are specified to be decoded, and some of these layers are specified as output layers. For example, the output layers (multiple) can be 1) only the highest layer, 2) all layers, or 3) the highest layer plus a set of indicated lower layers. For example, when the set of the highest layer plus the indicated lower layers is specified for output by a flag in the VPS, layer N+3 234 (the highest layer) and layers N 231 and N+1 232 (lower layers) from OLS 2 are output.
[0060] Figure 2 Some layers in the bitstream can be called primary layers, while others can be called secondary layers. For example, layers N231 and N+1232 can be called primary layers, and layers N+2233 and N+3234 can be called secondary layers. Secondary layers can be called alpha secondary layers or deep secondary layers. When secondary information is present in the bitstream, primary layers can be associated with secondary layers.
[0061] Unfortunately, the existing standard has flaws. 1. Currently, the syntax element sdi_view_id_len is encoded as u(4), and this value is required to be in the range of 0 to 15 (inclusive). This value specifies the bit length of the sdi_view_id_val[i] syntax element, which specifies the view ID of the i-th level in the bitstream. However, the length of sdi_view_id_val[i] should not be equal to 0, although this is currently allowed.
[0062] 2. When there is some auxiliary information in the bitstream, such as SDI SEI messages (also known as scalability dimension SEI messages) and depth representation information SEI messages or alpha channel information SEI messages, it is unknown which non-auxiliary layer or main layer the auxiliary information is applied to.
[0063] 3. If the Scalability Dimension Information (SEI) message does not exist in the bitstream, then the presence of Multi-View Acquisition Information (SEI) messages, Depth Representation Information (SEI) messages, or Alpha Channel Information (SEI) messages in the bitstream is meaningless.
[0064] 4. Multiple View Information Acquisition: The SEI message contains information about all views present in the bitstream. Therefore, it makes no sense to scalable nesting of it where currently permissible.
[0065] This document discloses techniques for addressing one or more of the aforementioned problems. For example, this disclosure provides a technique for preventing the length of the SDI view ID value syntax element specifying the view identifier of the i-th layer in the bitstream from being zero by using a Scalable Dimension Information (SDI) view identifier length minus L syntax element. The disclosed aspects / embodiments further provide techniques for preventing the bitstream from having Multi-View Acquisition Information Supplemental Enhancement (SEI) messages or auxiliary information SEI messages when SDI messages are not present in the bitstream. The disclosed aspects / embodiments also provide techniques for preventing Multi-View Acquisition Information SEI messages from being scalably nested.
[0066] Figure 3 An embodiment of video bitstream 300 is illustrated. As used herein, video bitstream 300 may also be referred to as a codec video bitstream, a bitstream, or a variant thereof. Figure 3 As shown, bitstream 300 includes one or more of the following: Decoding Capability Information (DCI) 302, Video Parameter Set (VPS) 304, Sequence Parameter Set (SPS) 306, Picture Parameter Set (PPS) 308, Picture Header (PH) 312, Picture 314, and SEI message 322. Each of DCI 302, VPS 304, SPS 306, and PPS 308 can be collectively referred to as a parameter set. In this embodiment, Figure 3 Other parameter sets, not shown in the figure, may also be included in bitstream 300, such as the Adaptive Parameter Set (APS), which is a syntax structure containing syntax elements applied to zero or more stripes determined by zero or more syntax elements found in the strip header.
[0067] DCI 302, also known as the Decoder Parameter Set (DPS) or Decoder Parameter Set, is a syntactic structure containing syntax elements applied throughout the bitstream. DCI 302 includes parameters that remain constant throughout the lifetime of the video bitstream (e.g., bitstream 300), which can be translated to the lifetime of a session. DCI 302 can include grade, level, and subgrade information to determine the maximum complexity interoperability point guaranteed never to be exceeded, even if video sequence splicing occurs during the session. It may also optionally include constraint flags indicating that the video bitstream will be constrained by the use of certain features indicated by the values of those flags. This allows the bitstream to be marked as not using certain tools, which in particular allows for resource allocation in decoder implementations. Like all parameter sets, DCI 302 exists when it is first referenced and is referenced by the first picture in the video sequence, meaning it must be transmitted between the first Network Abstraction Layer (NAL) units in the bitstream. While there can be multiple DCI 302s in a bitstream, the values of the syntax elements within them cannot be inconsistent when referenced.
[0068] VPS 304 includes decoding dependencies or information for building the reference picture set used for the enhancement layer. VPS 304 provides a holistic view or overview of the scalable sequence, including what types of action points are provided, the tiers, levels, and grades of action points, and other advanced attributes of the bitstream that can be used as the basis for session negotiation and content selection.
[0069] In an embodiment, when instructing certain layers to use ILP, VPS 304 indicates that the total number of OLS specified by the VPS is equal to the number of layers, indicates that the i-th OLS includes layers with layer indices from 0 to i (inclusive of end values), and indicates that for each OLS, only the highest layer in the OLS is output.
[0070] SPS 306 contains data common to all pictures in a picture sequence (SOP). SPS 306 is a syntax structure containing syntax elements applied to zero or more complete CLVSs, as determined by the content of the syntax elements found in the PPS, which is referenced by the syntax elements found in each picture header. In contrast, PPS 308 contains data common to the entire picture sequence. PPS 308 is a syntax structure containing syntax elements applied to zero or more complete encode / decode pictures, as determined by the syntax elements found in each picture header (e.g., PH 312).
[0071] DCI 302, VPS 304, SPS 306, and PPS 308 are contained within different types of Network Abstraction Layer (NAL) units. An NAL unit is a syntactic structure containing an indication of the type of data to be followed (e.g., encoded or decoded video data). NAL units are categorized into Video Codec Layer (VCL) and non-VCL NAL units. VCL NAL units contain data representing the values of samples in a video frame, while non-VCL NAL units contain any associated additional information, such as parameter sets (important data that can be applied to multiple VCL NAL units) and supplementary enhancement information (timing information and other supplementary data that can enhance the usability of decoding video signaling but are not necessary for decoding sample values in a video frame).
[0072] In an embodiment, DCI 302 is contained within a non-VCL NAL cell designated as a DCI NAL cell or a DPS NAL cell. That is, a DCI NAL cell has a DCI NAL cell type (NUT), while a DPS NAL cell has a DPS NUT. In an embodiment, VPS 304 is contained within a non-VCL NAL cell designated as a VPS NAL cell. Therefore, a VPS NAL cell has a VPS NUT. In an embodiment, SPS 306 is a non-VCL NAL cell designated as an SPS NAL cell. Therefore, an SPS NAL cell has an SPS NUT. In an embodiment, PPS 308 is contained within a non-VCL NAL cell designated as a PPS NAL cell. Therefore, a PPS NAL cell has a PPS NUT.
[0073] PH 312 is a syntactic structure containing syntactic elements applied to all stripes (e.g., stripe 318) of the encoded / decoded picture (e.g., picture 314). In an embodiment, PH 312 is a non-VCL NAL unit type designated as a PH NAL unit. Therefore, a PH NAL unit has a PH NUT (e.g., PH_NUT).
[0074] In an embodiment, the PH NAL unit associated with PH 312 has a time-domain ID and a layer ID. The time-domain ID indicates the temporal position of the PH NAL unit relative to other PH NAL units in the bitstream (e.g., bitstream 300). The layer ID indicates the layer containing the PH NAL unit (e.g., layer 131 or layer 132). In an embodiment, the time-domain ID is similar to but different from the Picture Order Count (POC). The POC uniquely identifies each picture sequentially. In a single-layer bitstream, the time-domain ID and POC will be the same. In a multi-layer bitstream (e.g., see...), the time-domain ID and POC will be different. Figure 1 Images within the same AU will have different POCs, but will have the same temporal ID.
[0075] In this embodiment, the PH NAL unit precedes the VCL NAL unit that contains the first strip 318 of the associated image 314. This establishes an association between PH 312 and the strip 318 of the image 314 associated with PH 312, without requiring an image header ID signaled in PH 312 and referenced from the strip header 320. Therefore, it can be inferred that all VCL NAL units between two PH 312 belong to the same image 314, and that image 314 is associated with the first PH 312 between the two PH 312. In this embodiment, the first VCL NAL unit following PH 312 contains the first strip 318 of the image 314 associated with PH 312.
[0076] In this embodiment, the PH NAL unit follows a picture-level parameter set (e.g., PPS) or a higher-level parameter set, such as DCI (aka, DPS), VPS, SPS, PPS, etc., with both a temporal ID and a layer ID that are smaller than the PH NAL unit, respectively. Therefore, these parameter sets are not repeated within a picture or access unit. Due to this ordering, PH 312 can be resolved immediately. That is, the parameter set containing parameters related to the entire picture is located before the PH NAL unit in the bitstream. Any content containing parameters specific to a picture is located after the PH NAL unit.
[0077] In one alternative, the PH NAL unit follows a picture-level parameter set and a Prefix Supplement Enhancement Information (SEI) message, or a higher-level parameter set such as DCI (also known as DPS), VPS, SPS, PPS, APS, SEI messages, etc.
[0078] Image 314 is an array of luminance samples in monochrome format or an array of luminance samples in 4:2:0, 4:2:2, and 4:4:4 color formats, along with two corresponding arrays of chrominance samples.
[0079] Image 314 can be a frame or a field. However, in a CVS 316, either all images 314 are frames, or all images 314 are fields. CVS 316 is a codec video sequence for each codec layer video sequence (CLVS) in the video bitstream 300. It is worth noting that CVS 316 and CLVS are the same when the video bitstream 300 comprises a single layer. CVS 316 and CLVS differ only when the video bitstream 300 comprises multiple layers (e.g., as...). Figure 1 and 2 (As shown).
[0080] Each image 314 contains one or more stripes 318. A stripe 318 is an integer number of complete slices or an integer number of consecutive complete code-decoder tree unit (CTU) rows within a slice of an image (e.g., image 314). Each stripe 318 is exclusively contained within a single NAL unit (e.g., a VCL·NAL unit). A slice (not shown) is a rectangular region of CTUs within a specific slice column and a specific slice row in an image (e.g., image 314). A CTU (not shown) is a code-decoder tree block (CTB) of luminance samples, two corresponding CTBs of chrominance samples of an image with three sample arrays, or a CTB of samples of a monochrome image or an image encoded using three separate color planes and a syntax structure for encoding and decoding samples. A CTB (not shown) is an N×N sample block for some value of N, such that dividing the components into CTBs is a partition. A block (not shown) is an M×N (M columns by N rows) array of samples (e.g., pixels), or an M×N array of transform coefficients.
[0081] In this embodiment, each stripe 318 includes a stripe header 320. The stripe header 320 is part of the encoded stripe 318 and contains data elements related to all stripes or CTU lines within the chip represented in the stripe 318. That is, the stripe header 320 contains information about the stripe 318, such as the stripe type, which reference image will be used, etc.
[0082] Image 314 and its strip 318 contain data related to the image or video being encoded or decoded. Therefore, image 314 and its strip 318 can simply be referred to as the payload or data carried in bitstream 300.
[0083] Bitstream 300 also contains one or more SEI messages, such as SEI message 322, SEI message 326, and SEI message 328. SEI messages contain supplemental enhancement information. SEI messages can contain various types of data that indicate the timing of video frames, describe various properties of the encoded video, or how the encoded video is used or enhanced. SEI messages are also defined to contain arbitrary user-defined data. SEI messages do not affect the core decoding process but can indicate suggestions on how to post-process or display the video. Some other advanced properties of the video content are conveyed in Video Usability Information (VUI), such as indications of the color space used to interpret the video content. With the development of new color spaces, such as high dynamic range and wide color gamut video, additional VUI identifiers have been added to indicate them.
[0084] In an embodiment, SEI message 322 may be an SDI SEI message. When auxiliary information is present in the bitstream, the SDI SEI message can be used to indicate which primary layers are associated with auxiliary layers. For example, an SDI SEI message may include one or more syntax elements 324 to indicate which primary layers are associated with auxiliary layers when auxiliary information is present in the bitstream. A discussion of various SEI messages and the syntax elements contained within them is provided below.
[0085] In this embodiment, SEI message 326 is a multi-view information SEI message, such as a multi-view acquisition information SEI message. When present in bitstream 300, the multi-view information SEI message includes one or more syntax elements 324 that specify various parameters of the acquisition environment, such as internal and external camera parameters. These parameters are very useful for view distortion and interpolation.
[0086] In an embodiment, SEI message 328 may be an auxiliary information SEI message, such as a depth representation information SEI message or an alpha channel information SEI message. When present in bitstream 300, the depth representation information SEI message includes one or more syntax elements 324 that specify various depth representations of the depth view for rendering previously decoded textures and depth view components, such as view composition, on a 3D display. For random access purposes, the SEI message may be associated with an Instantaneous Decoder Refresh (IDR) access unit. When present in bitstream 300, the alpha channel information SEI message includes one or more syntax elements 324 that provide information about alpha channel sample values and post-processing of alpha plane auxiliary images applied to decoding, as well as one or more associated primary images. Blending is the process of combining two images into a single image. The image to be blended is associated with an auxiliary image identified as an alpha plane. The alpha channel information SEI message may be used to specify how the pixel values of the image to be blended are converted into another image including interpreted values.
[0087] Those skilled in the art will recognize that, in practical applications, bitstream 300 may contain other parameters and information.
[0088] To address the aforementioned issues, methods summarized below are disclosed. These techniques should be considered as examples for explaining general concepts, rather than interpreted in a narrow sense. Furthermore, these techniques can be applied individually or in combination in any way.
[0089] Example 1
[0090] 1) To solve problem 1, in one example, instead of notifying the length of the view ID syntax element via the syntax element sdi_view_id_len signaling, the length minus L (e.g., L=1) is notified via the syntax element sdi_view_id_len_minusL signaling.
[0091] a. In addition, in one example, N bits can be used to encode and decode syntax elements into unsigned integers.
[0092] i. In one example, N can be equal to 4.
[0093] ii. Alternatively, the syntax may be encoded or decoded using a fixed-pattern bit string of N bits, or using a signed integer of N bits, or truncated binary, or syntax elements of a signed integer of order K (e.g., K=0) exponent Golombian code, or syntax elements of an unsigned integer of order M (e.g., M=0) exponent Golombian code.
[0094] b. In one example, or, the length is still signaled via, for example, the syntax element sdi_view_id_len, but the value of the constraint syntax element should not be equal to 0.
[0095] Example 2
[0096] 2) To solve problem 2, it is recommended to apply the auxiliary layer (i.e., the layer corresponding to sdi_aux_id[i] equal to 1 or 2) to one or more associated layers.
[0097] a. In one example, the signaling notification in the Scalability Dimension Information (SEI) message may indicate one or more syntax elements of the associated layer for each auxiliary layer.
[0098] i. In one example, the associated layer is specified by the layer ID.
[0099] ii. In another example, the associated layer is specified by the layer index.
[0100] iii. In another example, the indication of whether an auxiliary layer is applied to one or more associated layers can be specified by one or more syntax elements of the associated layer.
[0101] 1. In one example, syntax elements can be used to indicate whether an auxiliary layer is applied to all associated layers.
[0102] 2. In one example, syntax elements can be used to indicate whether an auxiliary layer is applied to a specific associated layer.
[0103] a. In one example, one or more main layers are indicated by syntax elements.
[0104] i. In one example, all major layers can be indicated by syntax elements.
[0105] ii. In one example, a syntax element may simply indicate the primary layer whose layer index is less than that of the secondary layer.
[0106] iii. In one example, a syntax element may simply indicate the main layer whose layer index is greater than that of the auxiliary layer.
[0107] b. In one example, the syntax elements are encoded and decoded into flags.
[0108] b. Alternatively, it is suggested that one or more related layers for each auxiliary layer be exported without explicit signaling notification.
[0109] i. In one example, the associated layer of each auxiliary layer can be nuh_layer_id equal to nuh_layer_id of the auxiliary layer plus layers N1, N2 and Nk, respectively, where k is an integer, and for any i,j (i != j) in the range from 1 to k (inclusive), Ni != Nj.
[0110] 1. In one example, k equals 1, and N1 can be equal to 1, 2, -1, or -2.
[0111] a. In one example, k equals 2, and N1 = 1, N2 = 2.
[0112] ii. In one example, the associated layer of each auxiliary layer can be a layer whose layer index is equal to the layer index of the auxiliary layer plus N1, N2 and Nk respectively, where k is an integer and for any i,j (i != j) in the range from 1 to k (inclusive), Ni != Nj.
[0113] 1. In one example, k equals 1, and N1 can be equal to 1, or 2, or -1, or -2.
[0114] 2. In one example, k is greater than 1.
[0115] a. In one example, k equals 2, and N1 = 1, N2 = 2.
[0116] c. Alternatively, the indication of the associated layer for each auxiliary layer can be explicitly signaled as one or a set of syntax elements in the Scalability Dimension Information (SEI) message.
[0117] d. Alternatively, the indication of the associated layer of the auxiliary information SEI message (e.g., depth representation information or alpha channel information) may be explicitly signaled by one or more syntax elements in the auxiliary information SEI message.
[0118] i. In one example, the auxiliary information SEI message may refer to the depth representation information SEI message or the alpha channel information SEI message.
[0119] ii. In one example, one or more syntax elements may indicate the layer ID value of the associated layer.
[0120] 1. In one example, the layer ID indicated by the syntax element may be less than or equal to the maximum layer ID value, i.e., vps_layer_id[vps_max_layers_minus1] or vps_layer_id[sdi_max_layers_minus1].
[0121] iii. In one example, one or more syntax elements may indicate the layer index value of the associated layer.
[0122] 1. In one example, the layer index indicated by the syntax element may be less than the maximum number of layers in the bitstream (e.g., sdi_max_layers_minus1 plus 1 or vps_max_layers_minus1 plus 1).
[0123] iv. In one example, signaling can be used to indicate whether one or more layers are associated with an auxiliary layer.
[0124] 1. In one example, a syntax element can be used to specify whether the auxiliary information SEI message applies to all layers.
[0125] a. In one example, the auxiliary_all_layer_flag being equal to X (X being 1 or 0) specifies that auxiliary information SEI messages are applied to all relevant primary layers.
[0126] 2. In one example, one or more syntax elements can be used to specify whether the auxiliary information SEI message is applied to one or more layers.
[0127] a. In one example, N syntax elements can be used to specify whether the auxiliary information SEI message is applied to layer N, where each syntax element is used for each layer.
[0128] i. In one example, 1 bit can be used to encode or decode a syntax element into a flag.
[0129] b. In one example, a syntax element can be used to specify whether the auxiliary information SEI message is applied to one or more layers.
[0130] i. In one example, the syntax element could be the Kth (e.g., K=0) exponent of the Columbus code.
[0131] ii. In one example, the syntax element equal to 5 specifies that the auxiliary information SEI message is applied to layers 0 and 2, but not to layer 1.
[0132] 1. Alternatively, N can be represented as the layer number. A syntax element equal to 5 specifies that the SEI auxiliary information message applies to layers (N-1) and (N-3), but not to layer (N-2).
[0133] c. The above syntax elements can be conditionally signaled, for example, only when the SEI auxiliary information message is not applied to all layers.
[0134] e. In one example, signaling in the bitstream can indicate the associated layer number of an auxiliary image of a layer.
[0135] f. In one example, the above syntax elements can be signaled using an N-bit unsigned integer, or an N-bit fixed-pattern bit string, or an N-bit signed integer, or a truncated binary, or a syntax element of a signed integer of order K (e.g., K=0) exponent Golomb code, or a syntax element of an unsigned integer of order M (e.g., M=0) exponent Golomb code.
[0136] g. In one example, conditional signaling may be used to notify the number of associated layers of the auxiliary picture and / or an indication of the number of associated layers of the auxiliary picture, for example, only if the i-th layer in bitstreamInScope contains the auxiliary picture (e.g., sdi_aux_id[i]>0). bitstreamInScope (also known as the bitstream within the scope) is defined as a sequence of AUs that, in decoding order, includes an initial AU containing the SDI SEI message and zero or more subsequent AUs up to but not including any subsequent AU containing another SDI SEI message.
[0137] Example 3
[0138] 3) To address problem 3, a bitstream consistency requirement was added, namely, multi-view or auxiliary information SEI messages should not exist in CVSs that do not have scalable dimension information SEI messages.
[0139] b. In addition, the auxiliary information SEI message can refer to the depth representation information SEI message or the alpha channel information SEI message.
[0140] c. Alternatively, add a bitstream consistency requirement, that is, when multiview or auxiliary information SEI messages are present in the bitstream, at least one of sdi_multiview_info_flag and sdi_auxiliary_info_flag in the scalable dimension information SEI message must be equal to 1.
[0141] Example 4
[0142] 4) To address problem 4, in one example, a bitstream consistency requirement was added, namely that multi-view information SEI messages should not be scalable and nested.
[0143] a. Alternatively, SEI messages with payloadType equal to 179 (multi-view capture) should not be included in scalable nested SEI messages.
[0144] The following are some example implementations of the examples summarized above. Each implementation can be applied to VVC. Most of the relevant parts that have been added or modified are indicated in bold italics, while some deleted parts are indicated in italics. There may be other editorial changes, which are not highlighted.
[0145] Each scalability dimension SEI message syntax described below includes one or more syntax elements. A syntax element can be, for example, one or more values, flags, variables, phrases, indicators, indexes, maps, data elements, or combinations thereof included in the scalability dimension SEI message syntax disclosed herein. In embodiments, syntax elements may be organized into a set of values, flags, variables, phrases, indicators, indexes, maps, and / or data elements.
[0146] Example 1
[0147] Scalability Dimension SEI Message Syntax
[0148]
[0149] Scalability Dimension SEI Message Semantics
[0150] The Scalability Dimension (SEI) message provides scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) the view ID of each layer when bitstreamInScope may be a multi-view bitstream; and 2) the auxiliary ID of each layer when bitstreamInScope may have one or more layers carrying auxiliary information (such as depth or alpha).
[0151] bitstreamInScope is an AU sequence that includes, in the decoding order, an initial AU containing the SDI SEI message and zero or more subsequent AUs, up to but not including any subsequent AU containing another SDI SEI message.
[0152] The increment of sdi_max_layers_minus1 indicates the maximum number of layers in bitstreamInScope.
[0153] A value of 1 for `sdi_multiview_info_flag` indicates that `bitstreamInScope` may be a multiview bitstream, and that the `sdi_view_id_val[]` syntax element exists in the Scalability Dimension SEI message. A value of 0 for `sdi_multiview_flag` indicates that `bitstreamInScope` is not a multiview bitstream, and that the `sdi_view_id_val[]` syntax element does not exist in the Scalability Dimension SEI message.
[0154] A value of 1 for `sdi_auxiliary_info_flag` indicates that auxiliary information may exist, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element exists in the scalability dimension SEI message. A value of 0 for `sdi_auxiliary_info_flag` indicates that no auxiliary information exists, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element does not exist in the scalability dimension SEI message.
[0155] Specifies the length of the sdi_view_id_val[i] syntax element, in bits.
[0156] `sdi_view_id_val[i]` specifies the view ID of the i-th level in `bitstreamInScope`. The length of the `sdi_view_id_val[i]` syntax element is... The value of sdi_view_id_val[i] is inferred to be 0 when it does not exist.
[0157] A value of 0 for sdi_aux_id[i] indicates that the i-th layer in bitstreamInScope does not contain auxiliary images. A value greater than 0 for sdi_aux_id[i] indicates the type of auxiliary image in the i-th layer of bitstreamInScope as specified in Table 1.
[0158] Table 1 – Mapping from sdi_aux_id[i] to auxiliary image types
[0159] sdi_aux_id[i] name Types of auxiliary images 1 AUX_ALPHA α plane 2 AUX_DEPTH Depth Images 3..127 Reserved 128..159 Unspecified 160..255 Reserved
[0160] Note 1 - Explanation of the auxiliary picture associated with sdi_aux_id in the range of 128 to 159 (inclusive) by means other than the sdi_aux_id value.
[0161] For this version of the bitstream conforming to this specification, sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive). Although the value of sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive), in this version of the specification, the decoder should allow the value of sdi_aux_id[i] to be in the range of 0 to 255 (inclusive).
[0162] Example 2
[0163] Scalability Dimension SEI Message Syntax
[0164]
[0165] Scalability Dimension SEI Message Semantics
[0166] The Scalability Dimension (SEI) message provides scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) the view ID of each layer when bitstreamInScope may be a multi-view bitstream; and 2) the auxiliary ID of each layer when bitstreamInScope may have one or more layers carrying auxiliary information (such as depth or alpha).
[0167] bitstreamInScope is an AU sequence that includes, in the decoding order, an initial AU containing the SDI SEI message and zero or more subsequent AUs, up to but not including any subsequent AU containing another SDI SEI message.
[0168] Indicates the maximum number of layers in bitstreamInScope.
[0169] A value of 1 for `sdi_multiview_info_flag` indicates that `bitstreamInScope` may be a multiview bitstream, and that the `sdi_view_id_val[]` syntax element exists in the Scalability Dimension SEI message. A value of 0 for `sdi_multiview_flag` indicates that `bitstreamInScope` is not a multiview bitstream, and that the `sdi_view_id_val[]` syntax element does not exist in the Scalability Dimension SEI message.
[0170] A value of 1 for `sdi_auxiliary_info_flag` indicates that auxiliary information may exist, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element exists in the scalability dimension SEI message. A value of 0 for `sdi_auxiliary_info_flag` indicates that no auxiliary information exists, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element does not exist in the scalability dimension SEI message.
[0171] Specifies the length of the sdi_view_id_val[i] syntax element, in bits.
[0172] `sdi_view_id_val[i]` specifies the view ID of the i-th level in `bitstreamInScope`. The length of the `sdi_view_id_val[i]` syntax element is... The value of sdi_view_id_val[i] is inferred to be 0 when it does not exist.
[0173] A value of 0 for sdi_aux_id[i] indicates that the i-th layer in bitstreamInScope does not contain auxiliary images. A value greater than 0 for sdi_aux_id[i] indicates the type of auxiliary image in the i-th layer of bitstreamInScope as specified in Table 1.
[0174] Table 1 – Mapping of sdi aux id[i] to auxiliary image types
[0175]
[0176]
[0177] Note 1 - Explanation of the auxiliary picture associated with sdi_aux_id in the range of 128 to 159 (inclusive) by means other than the sdi_aux_id value.
[0178] For this version of the bitstream conforming to this specification, sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive). Although the value of sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive), in this version of the specification, the decoder should allow the value of sdi_aux_id[i] to be in the range of 0 to 255 (inclusive).
[0179] Example 3
[0180] Scalability Dimension SEI Message Syntax
[0181]
[0182] Scalability Dimension SEI Message Semantics
[0183] The Scalability Dimension (SEI) message provides scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) the view ID of each layer when bitstreamInScope may be a multi-view bitstream; and 2) the auxiliary ID of each layer when bitstreamInScope may have one or more layers carrying auxiliary information (such as depth or alpha).
[0184] bitstreamInScope is an AU sequence that includes, in the decoding order, an initial AU containing the SDI SEI message and zero or more subsequent AUs, up to but not including any subsequent AU containing another SDI SEI message.
[0185] The increment of sdi_max_layers_minus1 indicates the maximum number of layers in bitstreamInScope.
[0186] A value of 1 for `sdi_multiview_info_flag` indicates that `bitstreamInScope` may be a multiview bitstream, and that the `sdi_view_id_val[]` syntax element exists in the Scalability Dimension SEI message. A value of 0 for `sdi_multiview_flag` indicates that `bitstreamInScope` is not a multiview bitstream, and that the `sdi_view_id_val[]` syntax element does not exist in the Scalability Dimension SEI message.
[0187] A value of 1 for `sdi_auxiliary_info_flag` indicates that auxiliary information may exist, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element exists in the scalability dimension SEI message. A value of 0 for `sdi_auxiliary_info_flag` indicates that no auxiliary information exists, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element does not exist in the scalability dimension SEI message.
[0188] The length of the sdi_view_id_len syntax element is specified in bits.
[0189]
[0190] Specifies the length of the sdi_view_id_val[i] syntax element, in bits.
[0191] `sdi_view_id_val[i]` specifies the view ID of the i-th level in `bitstreamInScope`. The length of the `sdi_view_id_val[i]` syntax element is `sdi_view_id_len` bits. If it does not exist, the value of `sdi_view_id_val[i]` is inferred to be equal to 0.
[0192] A value of 0 for sdi_aux_id[i] indicates that the i-th layer in bitstreamInScope does not contain auxiliary images. A value greater than 0 for sdi_aux_id[i] indicates the type of auxiliary image in the i-th layer of bitstreamInScope as specified in Table 1.
[0193] Table 1 – Mapping from sdi_aux_id[i] to auxiliary image types
[0194]
[0195]
[0196] Note 1 - Explanation of the auxiliary picture associated with sdi_aux_id in the range of 128 to 159 (inclusive) by means other than the sdi_aux_id value.
[0197] For this version of the bitstream conforming to this specification, sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive). Although the value of sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive), in this version of the specification, the decoder should allow the value of sdi_aux_id[i] to be in the range of 0 to 255 (inclusive).
[0198] Example 4
[0199] Scalability Dimension SEI Message Syntax
[0200]
[0201] Scalability Dimension SEI Message Semantics
[0202] The Scalability Dimension (SEI) message provides scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) the view ID of each layer when bitstreamInScope may be a multi-view bitstream; and 2) the auxiliary ID of each layer when bitstreamInScope may have one or more layers carrying auxiliary information (such as depth or alpha).
[0203] bitstreamInScope is an AU sequence that includes, in the decoding order, an initial AU containing the SDI SEI message and zero or more subsequent AUs, up to but not including any subsequent AU containing another SDI SEI message.
[0204] The increment of sdi_max_layers_minus1 indicates the maximum number of layers in bitstreamInScope.
[0205] A value of 1 for `sdi_multiview_info_flag` indicates that `bitstreamInScope` may be a multiview bitstream, and that the `sdi_view_id_val[]` syntax element exists in the Scalability Dimension SEI message. A value of 0 for `sdi_multiview_flag` indicates that `bitstreamInScope` is not a multiview bitstream, and that the `sdi_view_id_val[]` syntax element does not exist in the Scalability Dimension SEI message.
[0206] A value of 1 for `sdi_auxiliary_info_flag` indicates that auxiliary information may exist, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element exists in the scalability dimension SEI message. A value of 0 for `sdi_auxiliary_info_flag` indicates that no auxiliary information exists, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element does not exist in the scalability dimension SEI message.
[0207] The length of the sdi_view_id_len syntax element is specified in bits.
[0208]
[0209] `sdi_view_id_val[i]` specifies the view ID of the i-th level in `bitstreamInScope`. The length of the `sdi_view_id_val[i]` syntax element is `sdi_view_id_len` bits. If it does not exist, the value of `sdi_view_id_val[i]` is inferred to be equal to 0.
[0210] A value of 0 for sdi_aux_id[i] indicates that the i-th layer in bitstreamInScope does not contain auxiliary images. A value greater than 0 for sdi_aux_id[i] indicates the type of auxiliary image in the i-th layer of bitstreamInScope as specified in Table 1.
[0211] Table 1 – Mapping of sdi aux id[i] to auxiliary image types
[0212]
[0213]
[0214] Note 1 - Explanation of the auxiliary picture associated with sdi_aux_id in the range of 128 to 159 (inclusive) by means other than the sdi_aux_id value.
[0215] For this version of the bitstream conforming to this specification, sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive). Although the value of sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive), in this version of the specification, the decoder should allow the value of sdi_aux_id[i] to be in the range of 0 to 255 (inclusive).
[0216]
[0217] Example 5
[0218] Scalability Dimension SEI Message Syntax
[0219]
[0220]
[0221] Scalability Dimension SEI Message Semantics
[0222] The Scalability Dimension (SEI) message provides scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) the view ID of each layer when bitstreamInScope may be a multi-view bitstream; and 2) the auxiliary ID of each layer when bitstreamInScope may have one or more layers carrying auxiliary information (such as depth or alpha).
[0223] bitstreamInScope is an AU sequence that includes, in the decoding order, an initial AU containing the SDI SEI message and zero or more subsequent AUs, up to but not including any subsequent AU containing another SDI SEI message.
[0224] The increment of sdi_max_layers_minus1 indicates the maximum number of layers in bitstreamInScope.
[0225] A value of 1 for `sdi_multiview_info_flag` indicates that `bitstreamInScope` may be a multiview bitstream, and that the `sdi_view_id_val[]` syntax element exists in the Scalability Dimension SEI message. A value of 0 for `sdi_multiview_flag` indicates that `bitstreamInScope` is not a multiview bitstream, and that the `sdi_view_id_val[]` syntax element does not exist in the Scalability Dimension SEI message.
[0226] A value of 1 for `sdi_auxiliary_info_flag` indicates that auxiliary information may exist, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element exists in the scalability dimension SEI message. A value of 0 for `sdi_auxiliary_info_flag` indicates that no auxiliary information exists, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element does not exist in the scalability dimension SEI message.
[0227] The length of the sdi_view_id_len syntax element is specified in bits.
[0228] `sdi_view_id_val[i]` specifies the view ID of the i-th level in `bitstreamInScope`. The length of the `sdi_view_id_val[i]` syntax element is `sdi_view_id_len` bits. If it does not exist, the value of `sdi_view_id_val[i]` is inferred to be equal to 0.
[0229] A value of 0 for sdi_aux_id[i] indicates that the i-th layer in bitstreamInScope does not contain auxiliary images. A value greater than 0 for sdi_aux_id[i] indicates the type of auxiliary image in the i-th layer of bitstreamInScope as specified in Table 1.
[0230] Table 1 – Mapping from sdi_aux_id[i] to auxiliary image types
[0231] sdi_aux_id[i] name Types of auxiliary images 1 AUX_ALPHA α plane 2 AUX_DEPTH Depth Images 3..127 Reserved 128..159 Unspecified 160..255 Reserved
[0232] Note 1 - Explanation of the auxiliary picture associated with sdi_aux_id in the range of 128 to 159 (inclusive) by means other than the sdi_aux_id value.
[0233] For this version of the bitstream conforming to this specification, sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive). Although the value of sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive), in this version of the specification, the decoder should allow the value of sdi_aux_id[i] to be in the range of 0 to 255 (inclusive).
[0234]
[0235] Example 6
[0236] Scalability Dimension SEI Message Syntax
[0237]
[0238]
[0239] Scalability Dimension SEI Message Semantics
[0240] The Scalability Dimension (SEI) message provides scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) the view ID of each layer when bitstreamInScope may be a multi-view bitstream; and 2) the auxiliary ID of each layer when bitstreamInScope may have one or more layers carrying auxiliary information (such as depth or alpha).
[0241] bitstreamInScope is an AU sequence that includes, in the decoding order, an initial AU containing the SDI SEI message and zero or more subsequent AUs, up to but not including any subsequent AU containing another SDI SEI message.
[0242] The increment of sdi_max_layers_minus1 indicates the maximum number of layers in bitstreamInScope.
[0243] A value of 1 for `sdi_multiview_info_flag` indicates that `bitstreamInScope` may be a multiview bitstream, and that the `sdi_view_id_val[]` syntax element exists in the Scalability Dimension SEI message. A value of 0 for `sdi_multiview_flag` indicates that `bitstreamInScope` is not a multiview bitstream, and that the `sdi_view_id_val[]` syntax element does not exist in the Scalability Dimension SEI message.
[0244] A value of 1 for `sdi_auxiliary_info_flag` indicates that auxiliary information may exist, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element exists in the scalability dimension SEI message. A value of 0 for `sdi_auxiliary_info_flag` indicates that no auxiliary information exists, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element does not exist in the scalability dimension SEI message.
[0245] The length of the sdi_view_id_len syntax element is specified in bits.
[0246] `sdi_view_id_val[i]` specifies the view ID of the i-th level in `bitstreamInScope`. The length of the `sdi_view_id_val[i]` syntax element is `sdi_view_id_len` bits. If it does not exist, the value of `sdi_view_id_val[i]` is inferred to be equal to 0.
[0247] A value of 0 for sdi_aux_id[i] indicates that the i-th layer in bitstreamInScope does not contain auxiliary images. A value greater than 0 for sdi_aux_id[i] indicates the type of auxiliary image in the i-th layer of bitstreamInScope as specified in Table 1.
[0248] Table 1 – Mapping from sdi_aux_id[i] to auxiliary image types
[0249] sdi_aux_id[i] name Types of auxiliary images 1 AUX_ALPHA α plane 2 AUX_DEPTH Depth Images 3..127 Reserved 128..159 Unspecified 160..255 Reserved
[0250] Note 1 - Explanation of the auxiliary picture associated with sdi_aux_id in the range of 128 to 159 (inclusive) by means other than the sdi_aux_id value.
[0251] For this version of the bitstream conforming to this specification, sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive). Although the value of sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive), in this version of the specification, the decoder should allow the value of sdi_aux_id[i] to be in the range of 0 to 255 (inclusive).
[0252]
[0253] Example 7
[0254] Scalability Dimension SEI Message Syntax
[0255]
[0256] Scalability Dimension SEI Message Semantics
[0257] The Scalability Dimension (SEI) message provides scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) the view ID of each layer when bitstreamInScope may be a multi-view bitstream; and 2) the auxiliary ID of each layer when bitstreamInScope may have one or more layers carrying auxiliary information (such as depth or alpha).
[0258] bitstreamInScope is an AU sequence that includes, in the decoding order, an initial AU containing the SDI SEI message and zero or more subsequent AUs, up to but not including any subsequent AU containing another SDI SEI message.
[0259] The increment of sdi_max_layers_minus1 indicates the maximum number of layers in bitstreamInScope.
[0260] A value of 1 for `sdi_multiview_info_flag` indicates that `bitstreamInScope` may be a multiview bitstream, and that the `sdi_view_id_val[]` syntax element exists in the scalability dimension SEI message. A value of 0 for `sdi_multiview_flag` indicates that `bitstreamInScope` is not a multiview bitstream, and that the `sdi_view_id_val[]` syntax element does not exist in the scalability dimension SEI message.
[0261] A value of 1 for `sdi_auxiliary_info_flag` indicates that auxiliary information may exist, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element exists in the scalability dimension SEI message. A value of 0 for `sdi_auxiliary_info_flag` indicates that no auxiliary information exists, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element does not exist in the scalability dimension SEI message.
[0262] The length of the sdi_view_id_len syntax element is specified in bits.
[0263] `sdi_view_id_val[i]` specifies the view ID of the i-th level in `bitstreamInScope`. The length of the `sdi_view_id_val[i]` syntax element is `sdi_view_id_len` bits. If it does not exist, the value of `sdi_view_id_val[i]` is inferred to be equal to 0.
[0264] A value of 0 for sdi_aux_id[i] indicates that the i-th layer in bitstreamInScope does not contain auxiliary images. A value greater than 0 for sdi_aux_id[i] indicates the type of auxiliary image in the i-th layer of bitstreamInScope as specified in Table 1.
[0265] Table 1 – Mapping from sdi_aux_id[i] to auxiliary image types
[0266] sdi_aux_id[i] name Types of auxiliary images 1 AUX_ALPHA α plane 2 AUX_DEPTH Depth Images 3..127 Reserved 128..159 Unspecified 160..255 Reserved
[0267] Note 1 - Explanation of the auxiliary picture associated with sdi_aux_id in the range of 128 to 159 (inclusive) by means other than the sdi_aux_id value.
[0268] For this version of the bitstream conforming to this specification, sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive). Although the value of sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive), in this version of the specification, the decoder should allow the value of sdi_aux_id[i] to be in the range of 0 to 255 (inclusive).
[0269]
[0270] Example 8
[0271] Scalability Dimension SEI Message Syntax
[0272]
[0273]
[0274] Scalability Dimension SEI Message Semantics
[0275] The Scalability Dimension (SEI) message provides scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) the view ID of each layer when bitstreamInScope may be a multi-view bitstream; and 2) the auxiliary ID of each layer when bitstreamInScope may have one or more layers carrying auxiliary information (such as depth or alpha).
[0276] bitstreamInScope is an AU sequence that includes, in the decoding order, an initial AU containing the SDI SEI message and zero or more subsequent AUs, up to but not including any subsequent AU containing another SDI SEI message.
[0277] The increment of sdi_max_layers_minus1 indicates the maximum number of layers in bitstreamInScope.
[0278] A value of 1 for `sdi_multiview_info_flag` indicates that `bitstreamInScope` may be a multiview bitstream, and that the `sdi_view_id_val[]` syntax element exists in the Scalability Dimension SEI message. A value of 0 for `sdi_multiview_flag` indicates that `bitstreamInScope` is not a multiview bitstream, and that the `sdi_view_id_val[]` syntax element does not exist in the Scalability Dimension SEI message.
[0279] A value of 1 for `sdi_auxiliary_info_flag` indicates that auxiliary information may exist, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element exists in the scalability dimension SEI message. A value of 0 for `sdi_auxiliary_info_flag` indicates that no auxiliary information exists, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element does not exist in the scalability dimension SEI message.
[0280] The length of the sdi_view_id_len syntax element is specified in bits.
[0281] `sdi_view_id_val[i]` specifies the view ID of the i-th level in `bitstreamInScope`. The length of the `sdi_view_id_val[i]` syntax element is `sdi_view_id_len` bits. If it does not exist, the value of `sdi_view_id_val[i]` is inferred to be equal to 0.
[0282] A value of 0 for sdi_aux_id[i] indicates that the i-th layer in bitstreamInScope does not contain auxiliary images. A value greater than 0 for sdi_aux_id[i] indicates the type of auxiliary image in the i-th layer of bitstreamInScope as specified in Table 1.
[0283] Table 1 – Mapping from sdi_aux_id[i] to auxiliary image types
[0284] sdi_aux_id[i] name Types of auxiliary images 1 AUX_ALPHA α plane 2 AUX_DEPTH Depth Images 3..127 Reserved 128..159 Unspecified 160..255 Reserved
[0285] Note 1 - Explanation of the auxiliary picture associated with sdi_aux_id in the range of 128 to 159 (inclusive) by means other than the sdi_aux_id value.
[0286] For this version of the bitstream conforming to this specification, sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive). Although the value of sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive), in this version of the specification, the decoder should allow the value of sdi_aux_id[i] to be in the range of 0 to 255 (inclusive).
[0287]
[0288] Example 9
[0289] Scalability Dimension SEI Message Syntax
[0290]
[0291] Scalability Dimension SEI Message Semantics
[0292] The Scalability Dimension (SEI) message provides scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) the view ID of each layer when bitstreamInScope may be a multi-view bitstream; and 2) the auxiliary ID of each layer when bitstreamInScope may have one or more layers carrying auxiliary information (such as depth or alpha).
[0293] bitstreamInScope is an AU sequence that includes, in the decoding order, an initial AU containing the SDI SEI message and zero or more subsequent AUs, up to but not including any subsequent AU containing another SDI SEI message.
[0294] The increment of sdi_max_layers_minus1 indicates the maximum number of layers in bitstreamInScope.
[0295] A value of 1 for `sdi_multiview_info_flag` indicates that `bitstreamInScope` may be a multiview bitstream, and that the `sdi_view_id_val[]` syntax element exists in the Scalability Dimension SEI message. A value of 0 for `sdi_multiview_flag` indicates that `bitstreamInScope` is not a multiview bitstream, and that the `sdi_view_id_val[]` syntax element does not exist in the Scalability Dimension SEI message.
[0296] A value of 1 for `sdi_auxiliary_info_flag` indicates that auxiliary information may exist, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element exists in the scalability dimension SEI message. A value of 0 for `sdi_auxiliary_info_flag` indicates that no auxiliary information exists, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element does not exist in the scalability dimension SEI message.
[0297] The length of the sdi_view_id_len syntax element is specified in bits.
[0298] `sdi_view_id_val[i]` specifies the view ID of the i-th level in `bitstreamInScope`. The length of the `sdi_view_id_val[i]` syntax element is `sdi_view_id_len` bits. If it does not exist, the value of `sdi_view_id_val[i]` is inferred to be equal to 0.
[0299] A value of 0 for sdi_aux_id[i] indicates that the i-th layer in bitstreamInScope does not contain auxiliary images. A value greater than 0 for sdi_aux_id[i] indicates the type of auxiliary image in the i-th layer of bitstreamInScope as specified in Table 1.
[0300] Table 1 – Mapping from sdi_aux_id[i] to auxiliary image types
[0301]
[0302] Note 1 - Explanation of the auxiliary picture associated with sdi_aux_id in the range of 128 to 159 (inclusive) by means other than the sdi_aux_id value.
[0303] For this version of the bitstream conforming to this specification, sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive). Although the value of sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive), in this version of the specification, the decoder should allow the value of sdi_aux_id[i] to be in the range of 0 to 255 (inclusive).
[0304]
[0305] Example 10
[0306] Scalability Dimension SEI Message Syntax
[0307]
[0308]
[0309] Scalability Dimension SEI Message Semantics
[0310] The Scalability Dimension (SEI) message provides scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) the view ID of each layer when bitstreamInScope may be a multi-view bitstream; and 2) the auxiliary ID of each layer when bitstreamInScope may have one or more layers carrying auxiliary information (such as depth or alpha).
[0311] bitstreamInScope is an AU sequence that includes, in the decoding order, an initial AU containing the SDI SEI message and zero or more subsequent AUs, up to but not including any subsequent AU containing another SDI SEI message.
[0312] The increment of sdi_max_layers_minus1 indicates the maximum number of layers in bitstreamInScope.
[0313] A value of 1 for `sdi_multiview_info_flag` indicates that `bitstreamInScope` may be a multiview bitstream, and that the `sdi_view_id_val[]` syntax element exists in the Scalability Dimension SEI message. A value of 0 for `sdi_multiview_flag` indicates that `bitstreamInScope` is not a multiview bitstream, and that the `sdi_view_id_val[]` syntax element does not exist in the Scalability Dimension SEI message.
[0314] A value of 1 for `sdi_auxiliary_info_flag` indicates that auxiliary information may exist, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element exists in the scalability dimension SEI message. A value of 0 for `sdi_auxiliary_info_flag` indicates that no auxiliary information exists, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element does not exist in the scalability dimension SEI message.
[0315] The length of the sdi_view_id_len syntax element is specified in bits.
[0316] `sdi_view_id_val[i]` specifies the view ID of the i-th level in `bitstreamInScope`. The length of the `sdi_view_id_val[i]` syntax element is `sdi_view_id_len` bits. If it does not exist, the value of `sdi_view_id_val[i]` is inferred to be equal to 0.
[0317] A value of 0 for sdi_aux_id[i] indicates that the i-th layer in bitstreamInScope does not contain auxiliary images. A value greater than 0 for sdi_aux_id[i] indicates the type of auxiliary image in the i-th layer of bitstreamInScope as specified in Table 1.
[0318] Table 1 – Mapping from sdi_aux_id[i] to auxiliary image types
[0319] sdi_aux_id[i] name Types of auxiliary images 1 AUX_ALPHA α plane 2 AUX_DEPTH Depth Images 3..127 Reserved 128..159 Unspecified 160..255 Reserved
[0320] Note 1 - Explanation of the auxiliary picture associated with sdi_aux_id in the range of 128 to 159 (inclusive) by means other than the sdi_aux_id value.
[0321] For this version of the bitstream conforming to this specification, sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive). Although the value of sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive), in this version of the specification, the decoder should allow the value of sdi_aux_id[i] to be in the range of 0 to 255 (inclusive).
[0322]
[0323] Example 11
[0324] Scalability Dimension SEI Message Syntax
[0325]
[0326] Scalability Dimension SEI Message Semantics
[0327] The Scalability Dimension (SEI) message provides scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) the view ID of each layer when bitstreamInScope may be a multi-view bitstream; and 2) the auxiliary ID of each layer when bitstreamInScope may have one or more layers carrying auxiliary information (such as depth or alpha).
[0328] bitstreamInScope is an AU sequence that includes, in the decoding order, an initial AU containing the SDI SEI message and zero or more subsequent AUs, up to but not including any subsequent AU containing another SDI SEI message.
[0329] The increment of sdi_max_layers_minus1 indicates the maximum number of layers in bitstreamInScope.
[0330] A value of 1 for `sdi_multiview_info_flag` indicates that `bitstreamInScope` may be a multiview bitstream, and that the `sdi_view_id_val[]` syntax element exists in the Scalability Dimension SEI message. A value of 0 for `sdi_multiview_flag` indicates that `bitstreamInScope` is not a multiview bitstream, and that the `sdi_view_id_val[]` syntax element does not exist in the Scalability Dimension SEI message.
[0331] A value of 1 for `sdi_auxiliary_info_flag` indicates that auxiliary information may exist, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element exists in the scalability dimension SEI message. A value of 0 for `sdi_auxiliary_info_flag` indicates that no auxiliary information exists, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element does not exist in the scalability dimension SEI message.
[0332] The length of the sdi_view_id_len syntax element is specified in bits.
[0333] `sdi_view_id_val[i]` specifies the view ID of the i-th level in `bitstreamInScope`. The length of the `sdi_view_id_val[i]` syntax element is `sdi_view_id_len` bits. If it does not exist, the value of `sdi_view_id_val[i]` is inferred to be equal to 0.
[0334] A value of 0 for sdi_aux_id[i] indicates that the i-th layer in bitstreamInScope does not contain auxiliary images. A value greater than 0 for sdi_aux_id[i] indicates the type of auxiliary image in the i-th layer of bitstreamInScope as specified in Table 1.
[0335] Table 1 – Mapping from sdi_aux_id[i] to auxiliary image types
[0336] sdi_aux_id[i] name Types of auxiliary images 1 AUX_ALPHA α plane 2 AUX_DEPTH Depth Images 3..127 Reserved 128..159 Unspecified 160..255 Reserved
[0337] Note 1 - Explanation of the auxiliary picture associated with sdi_aux_id in the range of 128 to 159 (inclusive) by means other than the sdi_aux_id value.
[0338] For this version of the bitstream conforming to this specification, sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive). Although the value of sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive), in this version of the specification, the decoder should allow the value of sdi_aux_id[i] to be in the range of 0 to 255 (inclusive).
[0339] Example 12
[0340] Scalability Dimension SEI Message Syntax
[0341]
[0342]
[0343] Scalability Dimension SEI Message Semantics
[0344] The Scalability Dimension (SEI) message provides scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) the view ID of each layer when bitstreamInScope may be a multi-view bitstream; and 2) the auxiliary ID of each layer when bitstreamInScope may have one or more layers carrying auxiliary information (such as depth or alpha).
[0345] bitstreamInScope is an AU sequence that includes, in the decoding order, an initial AU containing the SDI SEI message and zero or more subsequent AUs, up to but not including any subsequent AU containing another SDI SEI message.
[0346] The increment of sdi_max_layers_minus1 indicates the maximum number of layers in bitstreamInScope.
[0347] A value of 1 for `sdi_multiview_info_flag` indicates that `bitstreamInScope` may be a multiview bitstream, and that the `sdi_view_id_val[]` syntax element exists in the Scalability Dimension SEI message. A value of 0 for `sdi_multiview_flag` indicates that `bitstreamInScope` is not a multiview bitstream, and that the `sdi_view_id_val[]` syntax element does not exist in the Scalability Dimension SEI message.
[0348] A value of 1 for `sdi_auxiliary_info_flag` indicates that auxiliary information may exist, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element exists in the scalability dimension SEI message. A value of 0 for `sdi_auxiliary_info_flag` indicates that no auxiliary information exists, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element does not exist in the scalability dimension SEI message.
[0349] The length of the sdi_view_id_len syntax element is specified in bits.
[0350] `sdi_view_id_val[i]` specifies the view ID of the i-th level in `bitstreamInScope`. The length of the `sdi_view_id_val[i]` syntax element is `sdi_view_id_len` bits. If it does not exist, the value of `sdi_view_id_val[i]` is inferred to be equal to 0.
[0351] A value of 0 for sdi_aux_id[i] indicates that the i-th layer in bitstreamInScope does not contain auxiliary images. A value greater than 0 for sdi_aux_id[i] indicates the type of auxiliary image in the i-th layer of bitstreamInScope as specified in Table 1.
[0352]
[0353] Note 1 – α-auxiliary layers or deep auxiliary layers are the main layers of their associated non-auxiliary layers.
[0354] Table 1 – Mapping from sdi_aux_id[i] to auxiliary image types
[0355] sdi_aux_id[i] name Types of auxiliary images 1 AUX_ALPHA α plane 2 AUX_DEPTH Depth Images 3..127 Reserved 128..159 Unspecified 160..255 Reserved
[0356] Note 1 - Explanation of the auxiliary picture associated with sdi_aux_id in the range of 128 to 159 (inclusive) by means other than the sdi_aux_id value.
[0357] For this version of the bitstream conforming to this specification, sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive). Although the value of sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive), in this version of the specification, the decoder should allow the value of sdi_aux_id[i] to be in the range of 0 to 255 (inclusive).
[0358] Example 13
[0359] Depth Representation Information SEI Message
[0360] Depth Representation Information SEI Message Syntax
[0361]
[0362] Depth Representation Information SEI Message Semantics
[0363] The syntax elements in the Depth Representation Information (SEI) message specify various parameters for the auxiliary image of type AUX_DEPTH, so that the primary and auxiliary images are processed and decoded before being rendered on a 3D display, such as for view composition. Specifically, the depth or parallax range of the depth image is specified.
[0364] When present, the Depth Representation Information (SEI) message should be associated with one or more layers whose sdi_aux_id value is equal to AUX_DEPTH. The following semantics apply to each nuh_layer_id targetLayerId in the nuh_layer_id value to which the Depth Representation Information (SEI) message is applied.
[0365] When present, the Depth Representation Information (SEI) message can be included in any access cell. It is recommended that, when present, the SEI message be included in an access cell for random access purposes, where the codec image with nuh_layer_id equal to targetLayerId is an Intra-Random Access Image (IRAP) image.
[0366] For an auxiliary picture where sdi_aux_id[targetLayerId] equals AUX_DEPTH, for all j values in the range of 0 to 2 (inclusive) and 4 to 15 (inclusive), the associated primary picture (if any) is the picture in the same access unit where sdi_aux_id[nuhLayerIdB] equals 0, such that ScalabilityId[LayerIdxInVps[targetLayerId]][j] equals ScalabilityId[LayerIdxInVps[nuhLayerIdB]][j].
[0367] The information indicated in the SEI message applies to all images whose nuh_layer_id is equal to targetLayerId, starting from the access unit containing the SEI message, up to but not including the next image, in the decoding order, associated with the depth representation information SEI message applicable to targetLayerId, or up to the end of the CLVS where nuh_layer_id is equal to targetLayerId, whichever is earlier in the decoding order.
[0368]
[0369] A z_near_flag value of 0 indicates that the syntax structure does not contain a syntax element specifying the nearest depth value. A z_near_flag value of 1 indicates that the syntax structure contains a syntax element specifying the nearest depth value.
[0370] A z_far_flag value of 0 indicates that the syntax structure does not contain a syntax element with a specified maximum depth value. A z_far_flag value of 1 indicates that the syntax structure contains a syntax element with a specified maximum depth value.
[0371] A value of 0 for d_min_flag indicates that the syntax structure does not contain any syntax element specifying the minimum disparity value. A value of 1 for d_min_flag indicates that the syntax structure contains a syntax element specifying the minimum disparity value.
[0372] A value of 0 for d_max_flag indicates that the syntax structure does not contain a syntax element that specifies the maximum disparity value. A value of 1 for d_max_flag indicates that the syntax structure contains a syntax element that specifies the maximum disparity value.
[0373] The `depth_representation_type` specifies the representation definition of the decoded luminance samples of the auxiliary image as defined in Table Y1. In Table Y1, parallax specifies the horizontal displacement between two texture views, while the Z value specifies the distance from the camera.
[0374] The variable maxVal is set to equal to (1<<(8+sps_bitdepth_minus8))-1, where sps_bitdepth_minus8 is the active SPS contained in the layer whose nuh_layer_id is equal to targetLayerId or its inferred value.
[0375] Definition of table Y1–depth_representation_type
[0376]
[0377] disparity_ref_view_id specifies the ViewId value on which the disparity value is derived.
[0378] Note 1 - Disparity_ref_view_id exists only when d_min_flag equals 1 or d_max_flag equals 1, and is useful for depth_representation_type values of 1 and 3.
[0379] The variables in column x of table Y2 are derived from the corresponding variables in columns s, e, n, and v of table Y2, as shown below:
[0380] --If the value of e is in the range of 0 to 127 (excluding extreme values), then x is set to equal to (-1). s *2 e-31 *(1+n÷2 v ).
[0381] Otherwise (if e equals 0), x is set to equal to (-1). s *2 -(30+v) *n.
[0382] Note 1 - The above specifications are similar to those in IEC 60559:1989.
[0383] Table Y2 – Relationship between depth parameter variables and syntax elements
[0384] x s e n v ZNear ZNearSign ZNearExp ZNearMantissa ZNearManLen ZFar ZFarSign ZFarExp ZFarMantissa ZFarManLen DMax DMaxSign DMaxExp DMaxMantissa DMaxManLen DMin DMinSign DMinExp DMinMantissa DMinManLen
[0385] The dmin and DMax values, when present, are specified in units of the luminance sample width of the encoded and decoded image, where ViewId is equal to the ViewId of the auxiliary image.
[0386] When they exist, ZNear and ZFar values have the same unit, but this is not specified.
[0387] depth_nonlinear_representation_num_minus1 plus 2 specifies the number of piecewise linear segments used to map depth values to a scale uniformly quantized according to disparity.
[0388] For i ranging from 0 to depth_nonlinear_representation_num_minus1+2 (inclusive), the depth_nonlinear_representation_model[i] is defined to map the decoded luminance sample values of the auxiliary image to piecewise linear segments of a scale uniformly quantized according to disparity. The values of depth_nonlinear_representation_model[0] and depth_nonlinear_representation_model[depth_nonlinear_representation_num_minus1+2] are both inferred to be equal to 0.
[0389] Note 2 - When depth_representation_type equals 3, the auxiliary image contains depth samples with a non-linear transformation. The variable DepthLUT[i], as specified below, is used to transform the decoded depth sample values from a non-linear representation to a linear representation, i.e., uniformly quantized disparity values. The shape of this transformation is defined by approximating a line segment in the non-linear disparity space from a two-dimensional linear disparity. The first (0, 0) and last (maxVal, maxVal) nodes of the curve are predefined. The positions of the additional nodes are sent in the form of deviations from the straight curve (depth_nonlinear_representation_model[i]). These deviations are uniformly distributed across the entire range from 0 to maxVal (inclusive), with spacing depending on the value of nonlinear_depth_representation_num_minus1.
[0390] The variable DepthLUT[i] for i in the range of 0 to maxVal (inclusive) is defined as follows:
[0391]
[0392] When depth_representation_type equals 3, DepthLUT[dS] of all decoded luminance sample values dS of the auxiliary image in the range of 0 to maxVal (inclusive) represents the disparity uniformly quantized to the range of 0 to maxVal (inclusive).
[0393] The syntax structure specifies the values of elements in the depth representation information SEI message.
[0394] The syntax structure sets the values of the variables OutSign, OutExp, OutMantissa, and OutManLen, which represent floating-point values. When this syntax structure is contained within another syntax structure, the variable names OutSign, OutExp, OutMantissa, and OutManLen will be interpreted as being replaced by the variable names used when the syntax structure was contained.
[0395] A value of 0 for da_sign_flag indicates a positive sign for the floating-point value. A value of 1 for da_sign_flag indicates a negative sign. The variable OutSign is set to equal da_sign_flag.
[0396] `da_exponent` specifies the exponent of the floating-point value. The value of `da_exponent` should be between 0 and 2. 7 The range is -2 (inclusive). Value 2 7 -1 is reserved for future use by ITU-T|ISO / IEC. The decoder should set the value to 2. 7 -1 is considered an unspecified value. The variable OutExp is set to equal da_exponent.
[0397] The increment of 1 in `da_mantissa_len_minus1` specifies the number of bits in the `da_mantissa` syntax element. The value of `da_mantissa_len_minus1` should be in the range of 0 to 31 (inclusive). The variable `OutManLen` is set to equal to `da_mantissa_len_minus1 + 1`.
[0398] `da_mantissa` specifies the mantissa of a floating-point value. The variable `OutMantissa` is set to be equal to `da_mantissa`.
[0399] Example 14
[0400] Depth Representation Information SEI Message
[0401] Depth Representation Information SEI Message Syntax
[0402]
[0403]
[0404]
[0405] Depth Representation Information SEI Message Semantics
[0406] The syntax elements in the Depth Representation Information (SEI) message specify various parameters for the auxiliary image of type AUX_DEPTH, so that the primary and auxiliary images are processed and decoded before being rendered on a 3D display, such as for view composition. Specifically, the depth or parallax range of the depth image is specified.
[0407] When present, the Depth Representation Information (SEI) message should be associated with one or more layers whose sdi_aux_id value is equal to AUX_DEPTH. The following semantics apply to each nuh_layer_id targetLayerId in the nuh_layer_id value to which the Depth Representation Information (SEI) message is applied.
[0408] When present, the depth representation information (SEI) message can be included in any access cell. It is recommended that, when present, the SEI message be included in an access cell for random access purposes, where the encoded / decoded image with nuh_layer_id equal to targetLayerId is an IRAP image.
[0409] For an auxiliary picture where sdi_aux_id[targetLayerId] equals AUX_DEPTH, for all j values in the range of 0 to 2 (inclusive) and 4 to 15 (inclusive), the associated primary picture (if any) is the picture in the same access unit where sdi_aux_id[nuhLayerIdB] equals 0, such that ScalabilityId[LayerIdxInVps[targetLayerId]][j] equals ScalabilityId[LayerIdxInVps[nuhLayerIdB]][j].
[0410] The information indicated in the SEI message applies to all images whose nuh_layer_id is equal to targetLayerId, starting from the access unit containing the SEI message, up to but not including the next image, in the decoding order, associated with the depth representation information SEI message applicable to targetLayerId, or up to the end of the CLVS where nuh_layer_id is equal to targetLayerId, whichever is earlier in the decoding order.
[0411]
[0412] A z_near_flag value of 0 indicates that the syntax structure does not contain a syntax element specifying the nearest depth value. A z_near_flag value of 1 indicates that the syntax structure contains a syntax element specifying the nearest depth value.
[0413] A z_far_flag value of 0 indicates that the syntax structure does not contain a syntax element with a specified maximum depth value. A z_far_flag value of 1 indicates that the syntax structure contains a syntax element with a specified maximum depth value.
[0414] A value of 0 for d_min_flag indicates that the syntax structure does not contain any syntax element specifying the minimum disparity value. A value of 1 for d_min_flag indicates that the syntax structure contains a syntax element specifying the minimum disparity value.
[0415] A value of 0 for d_max_flag indicates that the syntax structure does not contain a syntax element that specifies the maximum disparity value. A value of 1 for d_max_flag indicates that the syntax structure contains a syntax element that specifies the maximum disparity value.
[0416] The `depth_representation_type` specifies the representation definition of the decoded luminance samples of the auxiliary image as defined in Table Y1. In Table Y1, parallax specifies the horizontal displacement between two texture views, while the Z value specifies the distance from the camera.
[0417] The variable maxVal is set to equal to (1<<(8+sps_bitdepth_minus8))-1, where sps_bitdepth_minus8 is the active SPS contained in the layer whose nuh_layer_id is equal to targetLayerId or its inferred value.
[0418] Definition of table Y1–depth_representation_type
[0419]
[0420] disparity_ref_view_id specifies the ViewId value on which the disparity value is derived.
[0421] Note 1 - Disparity_ref_view_id exists only when d_min_flag equals 1 or d_max_flag equals 1, and is useful for depth_representation_type values of 1 and 3.
[0422] The variables in column x of table Y2 are derived from the corresponding variables in columns s, e, n, and v of table Y2, as shown below:
[0423] --If the value of e is in the range of 0 to 127 (excluding extreme values), then x is set to equal to (-1). s *2 e-31 *(1+n÷2 v ).
[0424] Otherwise (if e equals 0), x is set to equal to (-1). s *2 -(30+v) *n.
[0425] Note 1 - The above specifications are similar to those in IEC 60559:1989.
[0426] Table Y2 – Relationship between depth parameter variables and syntax elements
[0427] x s e n v ZNear ZNearSign ZNearExp ZNearMantissa ZNearManLen ZFar ZFarSign ZFarExp ZFarMantissa ZFarManLen DMax DMaxSign DMaxExp DMaxMantissa DMaxManLen DMin DMinSign DMinExp DMinMantissa DMinManLen
[0428] The dmin and DMax values, when present, are specified in units of the luminance sample width of the encoded and decoded image, where ViewId is equal to the ViewId of the auxiliary image.
[0429] When they exist, ZNear and ZFar values have the same unit, but this is not specified.
[0430] depth_nonlinear_representation_num_minus1 plus 2 specifies the number of piecewise linear segments used to map depth values to a scale uniformly quantized according to disparity.
[0431] For i ranging from 0 to depth_nonlinear_representation_num_minus1+2 (inclusive), the depth_nonlinear_representation_model[i] is defined to map the decoded luminance sample values of the auxiliary image to piecewise linear segments of a scale uniformly quantized according to disparity. The values of depth_nonlinear_representation_model[0] and depth_nonlinear_representation_model[depth_nonlinear_representation_num_minus1+2] are both inferred to be equal to 0.
[0432] Note 2 - When depth_representation_type equals 3, the auxiliary image contains depth samples with a non-linear transformation. The variable DepthLUT[i], as specified below, is used to transform the decoded depth sample values from a non-linear representation to a linear representation, i.e., uniformly quantized disparity values. The shape of this transformation is defined by approximating a line segment in the non-linear disparity space from a two-dimensional linear disparity. The first (0, 0) and last (maxVal, maxVal) nodes of the curve are predefined. The positions of the additional nodes are sent in the form of deviations from the straight curve (depth_nonlinear_representation_model[i]). These deviations are uniformly distributed across the entire range from 0 to maxVal (inclusive), with spacing depending on the value of nonlinear_depth_representation_num_minus1.
[0433] The variable DepthLUT[i] for i in the range of 0 to maxVal (inclusive) is defined as follows:
[0434]
[0435] When depth_representation_type equals 3, DepthLUT[dS] of all decoded luminance sample values dS of the auxiliary image in the range of 0 to maxVal (inclusive) represents the disparity uniformly quantized to the range of 0 to maxVal (inclusive).
[0436] The syntax structure specifies the values of elements in the depth representation information SEI message.
[0437] The syntax structure sets the values of the variables OutSign, OutExp, OutMantissa, and OutManLen, which represent floating-point values. When this syntax structure is contained within another syntax structure, the variable names OutSign, OutExp, OutMantissa, and OutManLen will be interpreted as being replaced by the variable names used when the syntax structure was contained.
[0438] A value of 0 for da_sign_flag indicates a positive sign for the floating-point value. A value of 1 for da_sign_flag indicates a negative sign. The variable OutSign is set to equal da_sign_flag.
[0439] `da_exponent` specifies the exponent of the floating-point value. The value of `da_exponent` should be between 0 and 2. 7 The range is -2 (inclusive). Value 2 7-1 is reserved for future use by ITU-T|ISO / IEC. The decoder should set the value to 2. 7 -1 is considered an unspecified value. The variable OutExp is set to equal da_exponent.
[0440] The increment of 1 in `da_mantissa_len_minus1` specifies the number of bits in the `da_mantissa` syntax element. The value of `da_mantissa_len_minus1` should be in the range of 0 to 31 (inclusive). The variable `OutManLen` is set to equal to `da_mantissa_len_minus1 + 1`.
[0441] `da_mantissa` specifies the mantissa of a floating-point value. The variable `OutMantissa` is set to be equal to `da_mantissa`.
[0442] Example 15
[0443] α Channel Information SEI Message
[0444] α Channel Information SEI Message Syntax
[0445]
[0446] α Channel Information SEI Message Semantics
[0447] The α channel information SEI message provides information about the α channel sample values and post-processing applied to the decoded α plane in auxiliary pictures of type AUX_ALPHA and one or more associated primary pictures.
[0448] For an auxiliary picture where nuh_layer_id equals nuhLayerIdA and sdi_aux_id[nuhLayerIdA] equals AUX_ALPHA, for all j values in the ranges of 0 to 2 (inclusive) and 4 to 15 (inclusive), the associated primary picture (if any) is the picture in the same access unit where sdi_aux_id[nuhLayerIdB] equals 0, such that scalability id[LayerIdxInVps[nuhLayerIdA]][j] equals scalability id[LayerIdxInVps[nuhLayerIdB]][j].
[0449] When the access unit contains an auxiliary image picA with nuh_layer_id equal to nuhLayerIdA and sdi_aux_id[nuhLayerIdA] equal to AUX_ALPHA, the alpha channel sample values of picA continue in output order until one or more of the following conditions are true:
[0450] -- Output the next image whose nuh_layer_id is equal to nuhLayerIdA, in the order of output.
[0451] --End of CLVS including auxiliary image picA.
[0452] --End of bitstream.
[0453] --nuh_layer_id equals nuhLayerIdA for the end of any related primary layer's CLVS for auxiliary image layers.
[0454] The following semantics apply to each nuh_layer_id targetLayerId in the nuh_layer_id value applied to the SEI message of the α channel information.
[0455]
[0456] An alpha_channel_cancel_flag value of 1 indicates that the alpha channel information SEI message cancels the persistence of any previous alpha channel information SEI messages in the order they are applied to the output of the current layer. An alpha_channel_cancel_flag value of 0 indicates that an alpha channel message follows.
[0457] Let currPic be the image associated with the alpha channel information SEI message. The semantics of the alpha channel information SEI message are continuously applied to the current layer in the order of output until one or more of the following conditions are true:
[0458] --A new CLVS for the current layer begins.
[0459] --End of bitstream.
[0460] --After the decoding process of calling the image sequence count of picB, immediately output the image picB in the access unit where numh_layer_id is equal to targetLayerId. This access unit contains an α channel information SEI message where numh_layer_id is equal to targetLayerId, and its PicOrderCnt(picB) is greater than PicOrderCnt(currPic). PicOrderCnt(picB) and PicOrderCnt(currPic) are the PicOrderCntVal values of picB and currPic, respectively.
[0461] An alpha_channel_use_idc value of 0 indicates that, for alpha mixing purposes, the decoded samples of the associated primary picture should be multiplied by the interpreted sample value of the auxiliary codec picture during display after the output from the decoding process. An alpha_channel_use_idc value of 1 indicates that, for alpha mixing purposes, the decoded samples of the associated primary picture should not be multiplied by the interpreted sample value of the auxiliary codec picture during display after the output from the decoding process. An alpha_channel_use_idc value of 2 indicates that the use of auxiliary pictures is not specified. Values of alpha_channel_use_idc greater than 2 are reserved for future use by ITU-T|ISO / IEC. When it does not exist, the value of alpha_channel_use_idc is inferred to be equal to 2.
[0462] `alpha_channel_bit_depth_minus8` plus 8 specifies the bit depth of the luminance sample array of the auxiliary image. `alpha_channel_bit_depth_minus8` should be in the range of 0 to 7 (inclusive). `alpha_channel_bit_depth_minus8` should be equal to `bit_depth_luma_minus8` of the relevant primary image.
[0463] `alpha_transparent_value` specifies the interpreted sample value for the luminance sample of the auxiliary codec image. For this auxiliary codec image luminance sample, the correlated luminance and chrominance samples of the primary codec image are considered transparent for alpha blending purposes. The number of bits used to represent the `alpha_transparent_value` syntax element is `alpha_channel_bit_depth_minus8+9`.
[0464] `alpha_opaque_value` specifies the interpreted sample value for the luminance sample of the auxiliary codec image. For this auxiliary codec image luminance sample, the correlated luminance and chrominance samples of the primary codec image are considered opaque and used for alpha blending. The number of bits used to represent the `alpha_opaque_value` syntax element is `alpha_channel_bit_depth_minus8+9`.
[0465] An `alpha_channel_incr_flag` value of 0 indicates that the interpreted sample value for each decoded auxiliary image luminance sample value is equal to the decoded auxiliary image sample value for alpha mixing purposes. An `alpha_channel_incr_flag` value of 1 indicates that, for alpha mixing purposes, after decoding the auxiliary image samples, any auxiliary image luminance sample value greater than `Min(alpha_opaque_value, alpha_transparent_value)` should be incremented by 1 to obtain the interpreted sample value for the auxiliary image sample, and any auxiliary image luminance sample value less than or equal to `Min(alpha_opaque_value, alpha_transparent_value)` should be used unchanged as the interpreted sample value for the decoded auxiliary image sample value. When this value is not present, the value of `alpha_channel_incr_flag` is inferred to be 0.
[0466] An alpha_channel_clip_flag value of 0 indicates that no clipping operation is applied to obtain the interpreted sample values for the decoding aid image. An alpha_channel_clip_flag value of 1 indicates that the interpreted sample values for the decoding aid image are changed according to the clipping process described by the alpha_channel_clip_type_flag syntax element. When it does not exist, the value of alpha_channel_clip_flag is inferred to be 0.
[0467] An `alpha_channel_clip_type_flag` value of 0 indicates that, for alpha blending purposes, after decoding the auxiliary image samples, any auxiliary image luminance sample greater than (alpha_opaque_value - alpha_transparent_value) / 2 is set to equal to `alpha_opaque_value` to obtain the interpreted sample value, and any auxiliary image luminance sample less than or equal to (alpha_opaque_value - alpha_transparent_value) / 2 is set to equal to `alpha_transparent_value` to obtain the interpreted sample value. An `alpha_channel_clip_type_flag` value of 1 indicates that, for alpha blending purposes, after decoding the auxiliary image samples, any auxiliary image luminance sample greater than `alpha_opaque_value` is set to equal to `alpha_opaque_value` to obtain the interpreted sample value, and any auxiliary image luminance sample less than or equal to `alpha_transparent_value` is set to equal to `alpha_transparent_value` to obtain the interpreted sample value.
[0468] Note - When both alpha_channel_incr_flag and alpha_channel_clip_flag are equal to 1, the clipping operation specified by alpha_channel_clip_type_flag should be applied first, and then the changes specified by alpha_channel_incr_flag should be applied to obtain the interpretation sample values of the auxiliary image brightness samples.
[0469] Example 16
[0470] α Channel Information SEI Message
[0471] α Channel Information SEI Message Syntax
[0472]
[0473]
[0474] α Channel Information SEI Message Semantics
[0475] The α channel information SEI message provides information about the α channel sample values and post-processing applied to the decoded α plane in auxiliary pictures of type AUX_ALPHA and one or more associated primary pictures.
[0476] For an auxiliary picture where nuh_layer_id equals nuhLayerIdA and sdi_aux_id[nuhLayerIdA] equals AUX_ALPHA, for all j values in the ranges of 0 to 2 (inclusive) and 4 to 15 (inclusive), the associated primary picture (if any) is the picture in the same access unit where sdi_aux_id[nuhLayerIdB] equals 0, such that scalability id[LayerIdxInVps[nuhLayerIdA]][j] equals scalability id[LayerIdxInVps[nuhLayerIdB]][j].
[0477] When the access unit contains an auxiliary image picA with nuh_layer_id equal to nuhLayerIdA and sdi_aux_id[nuhLayerIdA] equal to AUX_ALPHA, the alpha channel sample values of picA continue in output order until one or more of the following conditions are true:
[0478] -- Output the next image whose nuh_layer_id is equal to nuhLayerIdA, in the order of output.
[0479] --End of CLVS including auxiliary image picA.
[0480] --End of bitstream.
[0481] --nuh_layer_id equals nuhLayerIdA for the end of any related primary layer's CLVS for auxiliary image layers.
[0482] The following semantics apply to each nuh_layer_id targetLayerId in the nuh_layer_id value applied to the SEI message of the α channel information.
[0483] An alpha_channel_cancel_flag value of 1 indicates that the alpha channel information SEI message cancels the persistence of any previous alpha channel information SEI messages in the order they are applied to the output of the current layer. An alpha_channel_cancel_flag value of 0 indicates that an alpha channel message follows.
[0484] Let currPic be the image associated with the alpha channel information SEI message. The semantics of the alpha channel information SEI message are continuously applied to the current layer in the order of output until one or more of the following conditions are true:
[0485] --A new CLVS for the current layer begins.
[0486] --End of bitstream.
[0487] --After the decoding process of calling the image sequence count of picB, immediately output the image picB in the access unit where numh_layer_id is equal to targetLayerId. This access unit contains an α channel information SEI message where numh_layer_id is equal to targetLayerId, and its PicOrderCnt(picB) is greater than PicOrderCnt(currPic). PicOrderCnt(picB) and PicOrderCnt(currPic) are the PicOrderCntVal values of picB and currPic, respectively.
[0488]
[0489] An alpha_channel_use_idc value of 0 indicates that, for alpha mixing purposes, the decoded samples of the associated primary picture should be multiplied by the interpreted sample value of the auxiliary codec picture during display after the output from the decoding process. An alpha_channel_use_idc value of 1 indicates that, for alpha mixing purposes, the decoded samples of the associated primary picture should not be multiplied by the interpreted sample value of the auxiliary codec picture during display after the output from the decoding process. An alpha_channel_use_idc value of 2 indicates that the use of auxiliary pictures is not specified. Values of alpha_channel_use_idc greater than 2 are reserved for future use by ITU-T|ISO / IEC. When it does not exist, the value of alpha_channel_use_idc is inferred to be equal to 2.
[0490] `alpha_channel_bit_depth_minus8` plus 8 specifies the bit depth of the luminance sample array of the auxiliary image. `alpha_channel_bit_depth_minus8` should be in the range of 0 to 7 (inclusive). `alpha_channel_bit_depth_minus8` should be equal to `bit_depth_luma_minus8` of the relevant primary image.
[0491] `alpha_transparent_value` specifies the interpreted sample value for the luminance sample of the auxiliary codec image. For this auxiliary codec image luminance sample, the correlated luminance and chrominance samples of the primary codec image are considered transparent for alpha blending purposes. The number of bits used to represent the `alpha_transparent_value` syntax element is `alpha_channel_bit_depth_minus8+9`.
[0492] `alpha_opaque_value` specifies the interpreted sample value for the luminance sample of the auxiliary codec image. For this auxiliary codec image luminance sample, the correlated luminance and chrominance samples of the primary codec image are considered opaque and used for alpha blending. The number of bits used to represent the `alpha_opaque_value` syntax element is `alpha_channel_bit_depth_minus8+9`.
[0493] An `alpha_channel_incr_flag` value of 0 indicates that the interpreted sample value for each decoded auxiliary image luminance sample value is equal to the decoded auxiliary image sample value for alpha mixing purposes. An `alpha_channel_incr_flag` value of 1 indicates that, for alpha mixing purposes, after decoding the auxiliary image samples, any auxiliary image luminance sample value greater than `Min(alpha_opaque_value, alpha_transparent_value)` should be incremented by 1 to obtain the interpreted sample value for the auxiliary image sample, and any auxiliary image luminance sample value less than or equal to `Min(alpha_opaque_value, alpha_transparent_value)` should be used unchanged as the interpreted sample value for the decoded auxiliary image sample value. When this value is not present, the value of `alpha_channel_incr_flag` is inferred to be 0.
[0494] An alpha_channel_clip_flag value of 0 indicates that no clipping operation is applied to obtain the interpreted sample values for the decoding aid image. An alpha_channel_clip_flag value of 1 indicates that the interpreted sample values for the decoding aid image are changed according to the clipping process described by the alpha_channel_clip_type_flag syntax element. When it does not exist, the value of alpha_channel_clip_flag is inferred to be 0.
[0495] An `alpha_channel_clip_type_flag` value of 0 indicates that, for alpha blending purposes, after decoding the auxiliary image samples, any auxiliary image luminance sample greater than (alpha_opaque_value - alpha_transparent_value) / 2 is set to equal to `alpha_opaque_value` to obtain the interpreted sample value, and any auxiliary image luminance sample less than or equal to (alpha_opaque_value - alpha_transparent_value) / 2 is set to equal to `alpha_transparent_value` to obtain the interpreted sample value. An `alpha_channel_clip_type_flag` value of 1 indicates that, for alpha blending purposes, after decoding the auxiliary image samples, any auxiliary image luminance sample greater than `alpha_opaque_value` is set to equal to `alpha_opaque_value` to obtain the interpreted sample value, and any auxiliary image luminance sample less than or equal to `alpha_transparent_value` is set to equal to `alpha_transparent_value` to obtain the interpreted sample value.
[0496] Note - When both alpha_channel_incr_flag and alpha_channel_clip_flag are equal to 1, the clipping operation specified by alpha_channel_clip_type_flag should be applied first, and then the changes specified by alpha_channel_incr_flag should be applied to obtain the interpretation sample values of the auxiliary image brightness samples.
[0497] Example 17
[0498] Retrieving SEI messages from multiple views
[0499] SEI Message Syntax for Retrieving Information from Multiple Views
[0500]
[0501]
[0502]
[0503] Multi-view information retrieval SEI message semantics
[0504] Obtaining information from multiple views The SEI message specifies various parameters for acquiring the environment. Specifically, it specifies the parameters for both internal and external cameras. These parameters can be used to process and decode the view before rendering it on a 3D display.
[0505] The following semantics apply to each nuh_layer_id targetLayerId in the nuh_layer_id value applied to the SEI message for multi-view information retrieval.
[0506] When present, the SEI message for multi-view acquisition applied to the current layer should be included in the access unit containing the IRAP image, which is the first image of the current layer's CLVS. The signaling notification information in the SEI message is applied to the CLVS.
[0507]
[0508] When a multi-view information SEI message is contained within a scalable nested SEI message, the syntax elements sn_ols_flag and sn_all_layers_flag in the scalable nested SEI message should be equal to 0.
[0509] The variable numViewsMinus1 is exported as follows:
[0510] --If the multi-view information SEI message is not included in the scalable nested SEI message, then numViewsMinus1 is set to equal to 0.
[0511] --Otherwise (the multi-view SEI message is contained within the scalable nested SEI message), numViewsMinus1 is set to equal sn_num_layers_minus1.
[0512] The multi-view retrieval information contained in the multi-view retrieval information SEI message indicates that some views may not exist.
[0513] In the following semantics, index i refers to the syntax element and variable applied to the layer whose nuh_layer_id is equal to NestingLayerId[i].
[0514] The external camera parameters are defined according to the right-hand coordinate system, where the top-left corner of the image is the origin, i.e., the coordinates (0, 0), and the other corners of the image have non-negative coordinates. Based on these specifications, for the i-th camera, the 3D world point wP = [xyz] is mapped to the 2D camera point cP[i] = [uv 1], according to:
[0515] s*cP[i]=A[i]*R -1 [i]*(wP-T[i]) (X)
[0516] Where A[i] represents the internal camera parameter matrix, R -1 [i] denotes the inverse of the rotation matrix R[i], T[i] denotes the translation vector, and s (scalar value) is an arbitrary scale factor chosen so that the third coordinate of cP[i] is equal to 1. The elements of A[i], R[i], and T[i] are determined according to the syntax elements of the signaling notification in the SEI message, and are specified as follows.
[0517] An intrinsic_param_flag value of 1 indicates the existence of internal camera parameters. An intrinsic_param_flag value of 0 indicates the absence of internal camera parameters.
[0518] An extrinsic_param_flag value of 1 indicates the presence of external camera parameters. An extrinsic_param_flag value of 0 indicates the absence of external camera parameters.
[0519] An intrinsic_params_equal_flag value of 1 indicates that all cameras have the same intrinsic camera parameters, and there is only one set of intrinsic camera parameters. An intrinsic_params_equal_flag value of 0 indicates that each camera has different intrinsic camera parameters, and each camera has its own set of intrinsic camera parameters.
[0520] prec_focal_length is specified as 2 -prec_focal_length The exponent of the maximum permissible truncation error for focal_length_x[i] and focal_length_y[i]. The value of prec_focal_length should be in the range of 0 to 31 (inclusive).
[0521] prec_principal_point specifies that 2 -prec_principal_point The exponent of the maximum permissible truncation error for principal_point_x[i] and principal_point_y[i]. The value of prec_principal_point should be in the range of 0 to 31 (inclusive).
[0522] prec_skew_factor is specified by 2 -prec_skew_factor The exponent of the maximum permissible truncation error given the skew factor. The value of prec_skew_factor should be in the range of 0 to 31 (inclusive).
[0523] A value of 0 for sign_focal_length_x[i] indicates that the focal length of the i-th camera in the horizontal direction is positive. A value of 1 for sign_focal_length_x[i] indicates that the focal length is negative.
[0524] `exponent_focal_length_x[i]` specifies the exponential portion of the focal length of the i-th camera in the horizontal direction. The value of `exponent_focal_length_x[i]` should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should interpret value 63 as an indication of an unspecified focal length.
[0525] `mantissa_focal_length_x[i]` specifies the mantissa portion of the focal length of the i-th camera in the horizontal direction. The length of the `mantissa_focal_length_x[i]` syntax element is variable and is determined as follows:
[0526] --If exponent_focal_length_x[i] equals 0, then the length is Max(0,prec_focal_length-30).
[0527] --Otherwise (exponent_focal_length_x[i] is in the range of 0 to 63 (inclusive), the length is Max(0,exponent_focal_length_x[i]+prec_focal_length-31).
[0528] A value of 0 for sign_focal_length_y[i] indicates that the focal length in the vertical direction of the i-th camera is positive. A value of 1 for sign_focal_length_y[i] indicates that the focal length is negative.
[0529] `exponent_focal_length_y[i]` specifies the exponential portion of the focal length in the vertical direction for the i-th camera. The value of `exponent_focal_length_y[i]` should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should interpret value 63 as an indication of an unspecified focal length.
[0530] `mantissa_focal_length_y[i]` specifies the last part of the focal length of the i-th camera in the vertical direction.
[0531] The length of the mantissa_focal_length_y[i] syntax element is variable and is determined as follows:
[0532] --If exponent_focal_length_y[i] equals 0, then the length is Max(0,prec_focal_length-30).
[0533] --Otherwise (exponent_focal_length_y[i] is in the range of 0 to 63 (excluding the end value), the length is Max(0,exponent_focal_length_y[i]+prec_focal_length-31).
[0534] A value of 0 for sign_principal_point_x[i] indicates that the principal point of the i-th camera has a positive sign in the horizontal direction. A value of 1 for sign_principal_point_x[i] indicates that the sign is negative.
[0535] `exponent_principal_point_x[i]` specifies the exponent portion of the principal point of the i-th camera in the horizontal direction. The value of `exponent_principal_point_x[i]` should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should treat value 63 as indicating an unspecified principal point.
[0536] `mantissa_principal_point_x[i]` specifies the mantissa portion of the principal point of the i-th camera in the horizontal direction. The length of the `mantissa_principal_point_x[i]` syntax element, in bits, is variable and is determined as follows:
[0537] --If exponent_principal_point_x[i] equals 0, then the length is Max(0, prec_principal_point-30).
[0538] --Otherwise (exponent_principal_point_x[i] is in the range of 0 to 63 (excluding the end value), the length is Max(0,exponent_principal_point_x[i]+prec_principal_point-31).
[0539] A value of 0 for sign_principal_point_y[i] indicates that the sign of the principal point of the i-th camera is positive in the vertical direction. A value of 1 for sign_principal_point_y[i] indicates that the sign is negative.
[0540] `exponent_principal_point_y[i]` specifies the exponential portion of the principal point of the i-th camera in the vertical direction. The value of `exponent_principal_point_y[i]` should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should treat value 63 as an indication of an unspecified principal point.
[0541] `mantissa_principal_point_y[i]` specifies the mantissa portion of the principal point of the i-th camera in the vertical direction. The length of the `mantissa_principal_point_y[i]` syntax element, in bits, is variable and is determined as follows:
[0542] --If exponent_principal_point_y[i] equals 0, then the length is Max(0, prec_principal_point-30).
[0543] --Otherwise (exponent_principal_point_y[i] is in the range of 0 to 63 (excluding the end value), the length is Max(0,exponent_principal_point_y[i]+prec_principal_point-31).
[0544] The value of sign_skew_factor[i] being 0 indicates that the sign of the tilt factor of the i-th camera is positive.
[0545] The sign_skew_factor[i] being equal to 1 indicates that the sign is negative.
[0546] `exponent_skew_factor[i]` specifies the exponent portion of the tilt factor for the i-th camera. The value of `exponent_skew_factor[i]` should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should interpret value 63 as an indication of an unspecified tilt factor.
[0547] `mantissa_skew_factor[i]` specifies the mantissa portion of the tilt factor for the i-th camera. The length of the `mantissa_skew_factor[i]` syntax element is variable and is determined as follows:
[0548] --If exponent_skew_factor[i] equals 0, then the length is Max(0,prec_skew_factor-30).
[0549] --Otherwise (exponent_skew_factor[i] is in the range of 0 to 63 (excluding the end value), the length is Max(0,exponent_skew_factor[i]+prec_skew_factor-31).
[0550] The internal matrix A[i] of the i-th camera is represented by the following formula.
[0551]
[0552] prec_rotation_param specifies that 2 -prec_rotation_param The exponent of the maximum permissible truncation error for r[i][j][k]. The value of prec_rotation_param should be in the range of 0 to 31 (inclusive).
[0553] prec_translation_param specifies that it is 2 -prec_translation_param The exponent of the maximum permissible truncation error for t[i][j]. The value of prec_translation_param should be in the range of 0 to 31 (inclusive).
[0554] A sign_r[i][j][k] equal to 0 indicates that the (j, k) component of the rotation matrix of the i-th camera has a positive sign. A sign_r[i][j][k] equal to 1 indicates that the sign is negative.
[0555] The exponent_r[i][j][k] specifies the exponent portion of the (j, k) component of the rotation matrix for the i-th camera. The value of the exponent r[i][j][k] should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should interpret value 63 as indicating an unspecified rotation matrix.
[0556] `mantissa_r[i][j][k]` specifies the mantissa portion of the (j, k) component of the rotation matrix for the i-th camera. The length of the `mantissa_r[i][j][k]` syntax element, in bits, is variable and is determined as follows:
[0557] --If exponent_r[i] equals 0, then the length is Max(0,prec_rotation_param-30).
[0558] --otherwise (exponent_r[i] is in the range from 0 to 63 (exclusive)), the length is Max(0, exponent_r[i] + prec_rotation_param - 31).
[0559] The rotation matrix R[i] of the i-th camera is expressed as follows:
[0560]
[0561] sign_t[i][j] equal to 0 indicates that the sign of the j-th component of the translation vector of the i-th camera is positive. sign_t[i][j] equal to 1 indicates that the sign is negative.
[0562] exponent_t[i][j] specifies the exponent part of the j-th component of the translation vector of the i-th camera. The value of exponent_t[i][j] shall be in the range from 0 to 62 (inclusive). The value 63 is reserved for future use by ITU-T|ISO / IEC. The decoder shall treat the value 63 as indicating an unspecified translation vector.
[0563] mantissa_t[i][j] specifies the mantissa part of the j-th component of the translation vector of the i-th camera. The length v in bits of the mantissa_t[i][j] syntax element is variable and is determined as follows:
[0564] --If exponent_t[i] is equal to 0, the length v is set equal to Max(0, prec_translation_param - 30).
[0565] --otherwise (0 < exponent_t[i] < 63), the length v is set equal to Max(0, exponent_t[i] + prec_translation_param - 31).
[0566] The translation vector T[i] of the i-th camera is expressed by the following formula:
[0567]
[0568] The association between the camera parameter variables and the corresponding syntax elements is specified in Table ZZ. Each component of the intrinsic matrix and the rotation matrix as well as the translation vector are obtained from the variables specified in Table ZZ, and the variable x is calculated as follows:
[0569] --If e is in the range from 0 to 63 (exclusive), x is set equal to (-1) s *2 e-31 *(1+n÷2 v ).
[0570] Otherwise (if e equals 0), x is set to equal to (-1). s *2 -(30+v) *n.
[0571] Note - The above specifications are similar to those of IEC 60559:1989.
[0572] Table ZZ – Relationship between camera parameter variables and syntax elements.
[0573]
[0574] Example 18
[0575] Depth Representation Information SEI Message
[0576] Depth Representation Information SEI Message Syntax
[0577]
[0578]
[0579]
[0580] Depth Representation Information SEI Message Semantics
[0581] The syntax elements in the Depth Representation Information (SEI) message specify various parameters for the auxiliary image of type AUX_DEPTH, so that the primary and auxiliary images are processed and decoded before being rendered on a 3D display, such as for view composition. Specifically, the depth or parallax range of the depth image is specified.
[0582] When present, the Depth Representation Information (SEI) message should be associated with one or more layers whose sdi_aux_id value is equal to AUX_DEPTH. The following semantics apply to each nuh_layer_id targetLayerId in the nuh_layer_id value to which the Depth Representation Information (SEI) message is applied.
[0583] When present, the depth representation information (SEI) message can be included in any access cell. It is recommended that, when present, the SEI message be included in an access cell for random access purposes, where the encoded / decoded image with nuh_layer_id equal to targetLayerId is an IRAP image.
[0584]
[0585] For an auxiliary picture where sdi_aux_id[targetLayerId] equals AUX_DEPTH, for all j values in the range of 0 to 2 (inclusive) and 4 to 15 (inclusive), the associated primary picture (if any) is the picture in the same access unit where sdi_aux_id[nuhLayerIdB] equals 0, such that ScalabilityId[LayerIdxInVps[targetLayerId]][j] equals ScalabilityId[LayerIdxInVps[nuhLayerIdB]][j].
[0586] The information indicated in the SEI message applies to all images whose nuh_layer_id is equal to targetLayerId, starting from the access unit containing the SEI message, up to but not including the next image, in the decoding order, associated with the depth representation information SEI message applicable to targetLayerId, or up to the end of the CLVS where nuh_layer_id is equal to targetLayerId, whichever is earlier in the decoding order.
[0587] A z_near_flag value of 0 indicates that the syntax structure does not contain a syntax element that specifies the nearest depth value. A z_near_flag value of 1 indicates that the syntax structure contains a syntax element that specifies the nearest depth value.
[0588] A z_far_flag value of 0 indicates that the syntax structure does not contain any syntax elements that specify the farthest depth value. A z_far_flag value of 1 indicates that the syntax structure contains any syntax elements that specify the farthest depth value.
[0589] A value of 0 for d_min_flag indicates that the syntax structure does not contain any syntax element specifying the minimum disparity value. A value of 1 for d_min_flag indicates that the syntax structure contains a syntax element specifying the minimum disparity value.
[0590] A value of 0 for d_max_flag indicates that the syntax structure does not contain a syntax element that specifies the maximum disparity value. A value of 1 for d_max_flag indicates that the syntax structure contains a syntax element that specifies the maximum disparity value.
[0591] The `depth_representation_type` specifies the representation definition of the decoded luminance samples of the auxiliary image as defined in Table Y1. In Table Y1, parallax specifies the horizontal displacement between two texture views, while the Z value specifies the distance from the camera.
[0592] The variable maxVal is set to equal to (1<<(8+sps_bitdepth_minus8))-1, where sps_bitdepth_minus8 is the active SPS contained in the layer whose nuh_layer_id is equal to targetLayerId or its inferred value.
[0593] Definition of table Y1–depth_representation_type
[0594]
[0595] The disparity_ref_view_id specifies the ViewId value on which the disparity value is derived.
[0596] Note 1 - Disparity_ref_view_id exists only when d_min_flag equals 1 or d_max_flag equals 1, and is useful for depth_representation_type values of 1 and 3.
[0597] The variables in column x of table Y2 are derived from the corresponding variables in columns s, e, n, and v of table Y2, as shown below:
[0598] --If the value of e is in the range of 0 to 127 (excluding extreme values), then x is set to (-1). s *2 e-31 *(1+n÷2 v ).
[0599] Otherwise (if e equals 0), x is set to equal to (-1). s *2 -(30+v) *n.
[0600] Note 1 - The above specifications are similar to those in IEC 60559:1989.
[0601] Table Y2 – Relationship between depth parameter variables and syntax elements
[0602] x s e n v ZNear ZNearSign ZNearExp ZNearMantissa ZNearManLen ZFar ZFarSign ZFarExp ZFarMantissa ZFarManLen DMax DMaxSign DMaxExp DMaxMantissa DMaxManLen DMin DMinSign DMinExp DMinMantissa DMinManLen
[0603] The dmin and DMax values, when present, are specified in units of the luminance sample width of the encoded and decoded image, where ViewId is equal to the ViewId of the auxiliary image.
[0604] When they exist, the units of ZNear and ZFar values are the same, but this is not specified.
[0605] depth_nonlinear_representation_num_minus1 plus 2 specifies the number of piecewise linear segments used to map depth values to a scale uniformly quantized according to disparity.
[0606] For i ranging from 0 to depth_nonlinear_representation_num_minus1+2 (inclusive), the depth_nonlinear_representation_model[i] is defined to map the decoded luminance sample values of the auxiliary image to piecewise linear segments of a scale uniformly quantized according to disparity. The values of depth_nonlinear_representation_model[0] and depth_nonlinear_representation_model[depth_nonlinear_representation_num_minus1+2] are both inferred to be equal to 0.
[0607] Note 2 - When depth_representation_type equals 3, the auxiliary image contains depth samples with a non-linear transformation. The variable DepthLUT[i], as specified below, is used to transform the decoded depth sample values from a non-linear representation to a linear representation, i.e., uniformly quantized disparity values. The shape of this transformation is defined by approximating a line segment in the non-linear disparity space from a two-dimensional linear disparity. The first (0, 0) and last (maxVal, maxVal) nodes of the curve are predefined. The positions of the additional nodes are sent in the form of deviations from the straight curve (depth_nonlinear_representation_model[i]). These deviations are uniformly distributed across the entire range from 0 to maxVal (inclusive), with spacing depending on the value of nonlinear_depth_representation_num_minus1.
[0608] The variable DepthLUT[i] for i in the range of 0 to maxVal (inclusive) is defined as follows:
[0609]
[0610] When depth_representation_type equals 3, DepthLUT[dS] of all decoded luminance sample values dS of the auxiliary image in the range of 0 to maxVal (inclusive) represents the disparity uniformly quantized to the range of 0 to maxVal (inclusive).
[0611] The syntax structure specifies the values of elements in the SEI message that represent depth information.
[0612] The syntax structure sets the values of the variables OutSign, OutExp, OutMantissa, and OutManLen, which represent floating-point values. When this syntax structure is contained within another syntax structure, the variable names OutSign, OutExp, OutMantissa, and OutManLen will be interpreted as being replaced by the variable names used when the syntax structure was contained.
[0613] A value of 0 for da_sign_flag indicates a positive sign for the floating-point value. A value of 1 for da_sign_flag indicates a negative sign. The variable OutSign is set to equal da_sign_flag.
[0614] `da_exponent` specifies the exponent of the floating-point value. The value of `da_exponent` should be between 0 and 2. 7 The range is -2 (inclusive). Value 2 7 -1 is reserved for future use by ITU-T|ISO / IEC. The decoder should set the value to 2. 7 -1 is considered an unspecified value. The variable OutExp is set to equal da_exponent.
[0615] The increment of 1 in `da_mantissa_len_minus1` specifies the number of bits in the `da_mantissa` syntax element. The value of `da_mantissa_len_minus1` should be in the range of 0 to 31 (inclusive). The variable `OutManLen` is set to equal to `da_mantissa_len_minus1 + 1`.
[0616] `da_mantissa` specifies the mantissa of a floating-point value. The variable `OutMantissa` is set to be equal to `da_mantissa`.
[0617] Example 19
[0618] Depth Representation Information SEI Message
[0619] Depth Representation Information SEI Message Syntax
[0620]
[0621]
[0622]
[0623] Depth Representation Information SEI Message Semantics
[0624] The syntax elements in the Depth Representation Information (SEI) message specify various parameters for the auxiliary image of type AUX_DEPTH, so that the primary and auxiliary images are processed and decoded before being rendered on a 3D display, such as for view composition. Specifically, the depth or parallax range of the depth image is specified.
[0625]
[0626]
[0627] When present, the depth representation information (SEI) message should be associated with one or more layers whose sdi_aux_id value is equal to AUX_DEPTH. The following semantics are applied to each nuh_layer_idtargetLayerId in the nuh_layer_id value applied to the depth representation information SEI message.
[0628] When present, the depth representation information (SEI) message can be included in any access cell. It is recommended that, when present, the SEI message be included in an access cell for random access purposes, where the encoded / decoded image with nuh_layer_id equal to targetLayerId is an IRAP image.
[0629] For an auxiliary picture where sdi_aux_id[targetLayerId] equals AUX_DEPTH, for all j values in the range of 0 to 2 (inclusive) and 4 to 15 (inclusive), the associated primary picture (if any) is the picture in the same access unit where sdi_aux_id[nuhLayerIdB] equals 0, such that ScalabilityId[LayerIdxInVps[targetLayerId]][j] equals ScalabilityId[LayerIdxInVps[nuhLayerIdB]][j].
[0630] The information indicated in the SEI message applies to all images whose nuh_layer_id is equal to targetLayerId, starting from the access unit containing the SEI message, up to but not including the next image, in the decoding order, associated with the depth representation information SEI message applicable to targetLayerId, or up to the end of the CLVS where nuh_layer_id is equal to targetLayerId, whichever is earlier in the decoding order.
[0631] A z_near_flag value of 0 indicates that the syntax structure does not contain a syntax element that specifies the nearest depth value. A z_near_flag value of 1 indicates that the syntax structure contains a syntax element that specifies the nearest depth value.
[0632] A z_far_flag value of 0 indicates that the syntax structure does not contain any syntax elements that specify the farthest depth value. A z_far_flag value of 1 indicates that the syntax structure contains any syntax elements that specify the farthest depth value.
[0633] A value of 0 for d_min_flag indicates that the syntax structure does not contain any syntax element specifying the minimum disparity value. A value of 1 for d_min_flag indicates that the syntax structure contains a syntax element specifying the minimum disparity value.
[0634] A value of 0 for d_max_flag indicates that the syntax structure does not contain a syntax element that specifies the maximum disparity value. A value of 1 for d_max_flag indicates that the syntax structure contains a syntax element that specifies the maximum disparity value.
[0635] The `depth_representation_type` specifies the representation definition of the decoded luminance samples of the auxiliary image as defined in Table Y1. In Table Y1, parallax specifies the horizontal displacement between two texture views, while the Z value specifies the distance from the camera.
[0636] The variable maxVal is set to equal to (1<<(8+sps_bitdepth_minus8))-1, where sps_bitdepth_minus8 is the active SPS contained in the layer whose nuh_layer_id is equal to targetLayerId or its inferred value.
[0637] Definition of table Y1–depth_representation_type
[0638]
[0639] The disparity_ref_view_id specifies the ViewId value on which the disparity value is derived.
[0640] Note 1 - Disparity_ref_view_id exists only when d_min_flag equals 1 or d_max_flag equals 1, and is useful for depth_representation_type values of 1 and 3.
[0641] The variables in column x of table Y2 are derived from the corresponding variables in columns s, e, n, and v of table Y2, as shown below:
[0642] --If the value of e is in the range of 0 to 127 (excluding extreme values), then x is set to (-1). s *2 e-31 *(1+n÷2 v ).
[0643] Otherwise (if e equals 0), x is set to equal to (-1). s *2 -(30+v) *n.
[0644] Note 1 - The above specifications are similar to those in IEC 60559:1989.
[0645] Table Y2 – Relationship between depth parameter variables and syntax elements
[0646] x s e n v ZNear ZNearSign ZNearExp ZNearMantissa ZNearManLen ZFar ZFarSign ZFarExp ZFarMantissa ZFarManLen DMax DMaxSign DMaxExp DMaxMantissa DMaxManLen DMin DMinSign DMinExp DMinMantissa DMinManLen
[0647] The dmin and DMax values, when present, are specified in units of the luminance sample width of the encoded and decoded image, where ViewId is equal to the ViewId of the auxiliary image.
[0648] When they exist, the units of ZNear and ZFar values are the same, but this is not specified.
[0649] depth_nonlinear_representation_num_minus1 plus 2 specifies the number of piecewise linear segments used to map depth values to a scale uniformly quantized according to disparity.
[0650] For i ranging from 0 to depth_nonlinear_representation_num_minus1+2 (inclusive), the depth_nonlinear_representation_model[i] is defined to map the decoded luminance sample values of the auxiliary image to piecewise linear segments of a scale uniformly quantized according to disparity. The values of depth_nonlinear_representation_model[0] and depth_nonlinear_representation_model[depth_nonlinear_representation_num_minus1+2] are both inferred to be equal to 0.
[0651] Note 2 - When depth_representation_type equals 3, the auxiliary image contains depth samples with a non-linear transformation. The variable DepthLUT[i], as specified below, is used to transform the decoded depth sample values from a non-linear representation to a linear representation, i.e., uniformly quantized disparity values. The shape of this transformation is defined by approximating a line segment in the non-linear disparity space from a two-dimensional linear disparity. The first (0, 0) and last (maxVal, maxVal) nodes of the curve are predefined. The positions of the additional nodes are sent in the form of deviations from the straight curve (depth_nonlinear_representation_model[i]). These deviations are uniformly distributed across the entire range from 0 to maxVal (inclusive), with spacing depending on the value of nonlinear_depth_representation_num_minus1.
[0652] The variable DepthLUT[i] for i in the range of 0 to maxVal (inclusive) is defined as follows:
[0653]
[0654] When depth_representation_type equals 3, DepthLUT[dS] of all decoded luminance sample values dS of the auxiliary image in the range of 0 to maxVal (inclusive) represents the disparity uniformly quantized to the range of 0 to maxVal (inclusive).
[0655] The syntax structure specifies the values of elements in the SEI message that represent depth information.
[0656] The syntax structure sets the values of the variables OutSign, OutExp, OutMantissa, and OutManLen, which represent floating-point values. When this syntax structure is contained within another syntax structure, the variable names OutSign, OutExp, OutMantissa, and OutManLen will be interpreted as being replaced by the variable names used when the syntax structure was contained.
[0657] A value of 0 for da_sign_flag indicates a positive sign for the floating-point value. A value of 1 for da_sign_flag indicates a negative sign. The variable OutSign is set to equal da_sign_flag.
[0658] `da_exponent` specifies the exponent of the floating-point value. The value of `da_exponent` should be between 0 and 2. 7 The range is -2 (inclusive). Value 2 7-1 is reserved for future use by ITU-T|ISO / IEC. The decoder should set the value to 2. 7 -1 is considered an unspecified value. The variable OutExp is set to equal da_exponent.
[0659] The increment of 1 in `da_mantissa_len_minus1` specifies the number of bits in the `da_mantissa` syntax element. The value of `da_mantissa_len_minus1` should be in the range of 0 to 31 (inclusive). The variable `OutManLen` is set to equal to `da_mantissa_len_minus1 + 1`.
[0660] `da_mantissa` specifies the mantissa of a floating-point value. The variable `OutMantissa` is set to be equal to `da_mantissa`.
[0661] Example 20
[0662] α Channel Information SEI Message
[0663] α Channel Information SEI Message Syntax
[0664]
[0665] α Channel Information SEI Message Semantics
[0666] The α channel information SEI message provides information about the α channel sample values and post-processing applied to the decoded α plane in auxiliary pictures of type AUX_ALPHA and one or more associated primary pictures.
[0667] For an auxiliary picture where nuh_layer_id equals nuhLayerIdA and sdi_aux_id[nuhLayerIdA] equals AUX_ALPHA, for all j values in the ranges of 0 to 2 (inclusive) and 4 to 15 (inclusive), the associated primary picture (if any) is the picture in the same access unit where sdi_aux_id[nuhLayerIdB] equals 0, such that scalability id[LayerIdxInVps[nuhLayerIdA]][j] equals scalability id[LayerIdxInVps[nuhLayerIdB]][j].
[0668] When the access unit contains an auxiliary image picA with nuh_layer_id equal to nuhLayerIdA and sdi_aux_id[nuhLayerIdA] equal to AUX_ALPHA, the alpha channel sample values of picA continue in output order until one or more of the following conditions are true:
[0669] -- Output the next image whose nuh_layer_id is equal to nuhLayerIdA, in the order of output.
[0670] --End of CLVS including auxiliary image picA.
[0671] --End of bitstream.
[0672] --nuh_layer_id equals nuhLayerIdA for the end of any related primary layer's CLVS for auxiliary image layers.
[0673]
[0674] The following semantics apply to each nuh_layer_id targetLayerId in the nuh_layer_id value applied to the SEI message of the α channel information.
[0675] An alpha_channel_cancel_flag value of 1 indicates that the alpha channel information SEI message cancels the persistence of any previous alpha channel information SEI messages in the order they are applied to the output of the current layer. An alpha_channel_cancel_flag value of 0 indicates that an alpha channel message follows.
[0676] Let currPic be the image associated with the alpha channel information SEI message. The semantics of the alpha channel information SEI message are continuously applied to the current layer in the order of output until one or more of the following conditions are true:
[0677] --A new CLVS for the current layer begins.
[0678] --End of bitstream.
[0679] --After the decoding process of calling the image sequence count of picB, immediately output the image picB in the access unit where numh_layer_id is equal to targetLayerId. This access unit contains an α channel information SEI message where numh_layer_id is equal to targetLayerId, and its PicOrderCnt(picB) is greater than PicOrderCnt(currPic). PicOrderCnt(picB) and PicOrderCnt(currPic) are the PicOrderCntVal values of picB and currPic, respectively.
[0680] An alpha_channel_use_idc value of 0 indicates that, for alpha mixing purposes, the decoded samples of the associated primary picture should be multiplied by the interpreted sample value of the auxiliary codec picture during display after the output from the decoding process. An alpha_channel_use_idc value of 1 indicates that, for alpha mixing purposes, the decoded samples of the associated primary picture should not be multiplied by the interpreted sample value of the auxiliary codec picture during display after the output from the decoding process. An alpha_channel_use_idc value of 2 indicates that the use of auxiliary pictures is not specified. Values of alpha_channel_use_idc greater than 2 are reserved for future use by ITU-T|ISO / IEC. When it does not exist, the value of alpha_channel_use_idc is inferred to be equal to 2.
[0681] `alpha_channel_bit_depth_minus8` plus 8 specifies the bit depth of the luminance sample array of the auxiliary image. `alpha_channel_bit_depth_minus8` should be in the range of 0 to 7 (inclusive). `alpha_channel_bit_depth_minus8` should be equal to `bit_depth_luma_minus8` of the relevant primary image.
[0682] `alpha_transparent_value` specifies the interpreted sample value for the luminance sample of the auxiliary codec image. For this auxiliary codec image luminance sample, the correlated luminance and chrominance samples of the primary codec image are considered transparent for alpha blending purposes. The number of bits used to represent the `alpha_transparent_value` syntax element is `alpha_channel_bit_depth_minus8+9`.
[0683] `alpha_opaque_value` specifies the interpreted sample value for the luminance sample of the auxiliary codec image. For this auxiliary codec image luminance sample, the correlated luminance and chrominance samples of the primary codec image are considered opaque and used for alpha blending. The number of bits used to represent the `alpha_opaque_value` syntax element is `alpha_channel_bit_depth_minus8+9`.
[0684] An `alpha_channel_incr_flag` value of 0 indicates that the interpreted sample value for each decoded auxiliary image luminance sample value is equal to the decoded auxiliary image sample value for alpha mixing purposes. An `alpha_channel_incr_flag` value of 1 indicates that, for alpha mixing purposes, after decoding the auxiliary image samples, any auxiliary image luminance sample value greater than `Min(alpha_opaque_value, alpha_transparent_value)` should be incremented by 1 to obtain the interpreted sample value for the auxiliary image sample, and any auxiliary image luminance sample value less than or equal to `Min(alpha_opaque_value, alpha_transparent_value)` should be used unchanged as the interpreted sample value for the decoded auxiliary image sample value. When this value is not present, the value of `alpha_channel_incr_flag` is inferred to be 0.
[0685] An alpha_channel_clip_flag value of 0 indicates that no clipping operation is applied to obtain the interpreted sample values for the decoding aid image. An alpha_channel_clip_flag value of 1 indicates that the interpreted sample values for the decoding aid image are changed according to the clipping process described by the alpha_channel_clip_type_flag syntax element. When it does not exist, the value of alpha_channel_clip_flag is inferred to be 0.
[0686] An `alpha_channel_clip_type_flag` value of 0 indicates that, for alpha blending purposes, after decoding the auxiliary image samples, any auxiliary image luminance sample greater than (alpha_opaque_value - alpha_transparent_value) / 2 is set to equal to `alpha_opaque_value` to obtain the interpreted sample value, and any auxiliary image luminance sample less than or equal to (alpha_opaque_value - alpha_transparent_value) / 2 is set to equal to `alpha_transparent_value` to obtain the interpreted sample value. An `alpha_channel_clip_type_flag` value of 1 indicates that, for alpha blending purposes, after decoding the auxiliary image samples, any auxiliary image luminance sample greater than `alpha_opaque_value` is set to equal to `alpha_opaque_value` to obtain the interpreted sample value, and any auxiliary image luminance sample less than or equal to `alpha_transparent_value` is set to equal to `alpha_transparent_value` to obtain the interpreted sample value.
[0687] Note - When both alpha_channel_incr_flag and alpha_channel_clip_flag are equal to 1, the clipping operation specified by alpha_channel_clip_type_flag should be applied first, and then the changes specified by alpha_channel_incr_flag should be applied to obtain the interpretation sample values of the auxiliary image brightness samples.
[0688] Example 21
[0689] α Channel Information SEI Message
[0690] α Channel Information SEI Message Syntax
[0691]
[0692] α Channel Information SEI Message Semantics
[0693] α channel information SEI messages provide information about the α channel sample values and post-processing applied to the decoded α plane in auxiliary pictures of type AUX_ALPHA and one or more associated primary pictures.
[0694]
[0695] When access unit When an auxiliary image picA is included, where nuh_layer_id equals nuhLayerIdA and sdi_aux_id[nuhLayerIdA] equals AUX_ALPHA, the alpha channel sample values of picA continue in output order until one or more of the following conditions are true:
[0696] Output the next image whose nuh_layer_id is equal to nuhLayerIdA, in the order of output.
[0697] --End of CLVS including auxiliary image picA.
[0698] --End of bitstream.
[0699] --nuh_layer_id equals nuhLayerIdA for the end of any related primary layer's CLVS for auxiliary image layers.
[0700] The following semantics apply to each nuh_layer_id targetLayerId in the nuh_layer_id value applied to the SEI message of the α channel information.
[0701] An alpha_channel_cancel_flag value of 1 indicates that the alpha channel information SEI message cancels the persistence of any previous alpha channel information SEI messages in the order they are applied to the output of the current layer. An alpha_channel_cancel_flag value of 0 indicates that an alpha channel message follows.
[0702] Let currPic be the image associated with the alpha channel information SEI message. The semantics of the alpha channel information SEI message are continuously applied to the current layer in the order of output until one or more of the following conditions are true:
[0703] --A new CLVS for the current layer begins.
[0704] --End of bitstream.
[0705] --After calling the image sequence counting decoding process of picB, immediately output the access unit. picB where numh_layer_id equals targetLayerId picture, The access unit contains an α channel information SEI message with numh_layer_id equal to targetLayerId, whose PicOrderCnt(picB) is greater than PicOrderCnt(currPic), where PicOrderCnt(picB) and PicOrderCnt(currPic) are the PicOrderCntVal values of picB and currPic, respectively.
[0706] An alpha_channel_use_idc value of 0 indicates that, for alpha mixing purposes, the decoded samples of the associated primary picture should be multiplied by the interpreted sample value of the auxiliary codec picture during display after the output from the decoding process. An alpha_channel_use_idc value of 1 indicates that, for alpha mixing purposes, the decoded samples of the associated primary picture should not be multiplied by the interpreted sample value of the auxiliary codec picture during display after the output from the decoding process. An alpha_channel_use_idc value of 2 indicates that the use of auxiliary pictures is not specified. Values of alpha_channel_use_idc greater than 2 are reserved for future use by ITU-T|ISO / IEC. When it does not exist, the value of alpha_channel_use_idc is inferred to be equal to 2.
[0707] `alpha_channel_bit_depth_minus8` plus 8 specifies the bit depth of the luminance sample array of the auxiliary image. `alpha_channel_bit_depth_minus8` should be in the range of 0 to 7 (inclusive). `alpha_channel_bit_depth_minus8` should be equal to `bit_depth_luma_minus8` of the relevant primary image.
[0708] `alpha_transparent_value` specifies the interpreted sample value for the luminance sample of the auxiliary codec image. For this auxiliary codec image luminance sample, the correlated luminance and chrominance samples of the primary codec image are considered transparent for alpha blending purposes. The number of bits used to represent the `alpha_transparent_value` syntax element is `alpha_channel_bit_depth_minus8+9`.
[0709] `alpha_opaque_value` specifies the interpreted sample value for the luminance sample of the auxiliary codec image. For this auxiliary codec image luminance sample, the correlated luminance and chrominance samples of the primary codec image are considered opaque and used for alpha blending. The number of bits used to represent the `alpha_opaque_value` syntax element is `alpha_channel_bit_depth_minus8+9`.
[0710] An `alpha_channel_incr_flag` value of 0 indicates that the interpreted sample value for each decoded auxiliary image luminance sample value is equal to the decoded auxiliary image sample value for alpha mixing purposes. An `alpha_channel_incr_flag` value of 1 indicates that, for alpha mixing purposes, after decoding the auxiliary image samples, any auxiliary image luminance sample value greater than `Min(alpha_opaque_value, alpha_transparent_value)` should be incremented by 1 to obtain the interpreted sample value for the auxiliary image sample, and any auxiliary image luminance sample value less than or equal to `Min(alpha_opaque_value, alpha_transparent_value)` should be used unchanged as the interpreted sample value for the decoded auxiliary image sample value. When this value is not present, the value of `alpha_channel_incr_flag` is inferred to be 0.
[0711] An alpha_channel_clip_flag value of 0 indicates that no clipping operation is applied to obtain the interpreted sample values for the decoding aid image. An alpha_channel_clip_flag value of 1 indicates that the interpreted sample values for the decoding aid image are changed according to the clipping process described by the alpha_channel_clip_type_flag syntax element. When it does not exist, the value of alpha_channel_clip_flag is inferred to be 0.
[0712] An `alpha_channel_clip_type_flag` value of 0 indicates that, for alpha blending purposes, after decoding the auxiliary image samples, any auxiliary image luminance sample greater than (alpha_opaque_value - alpha_transparent_value) / 2 is set to equal to `alpha_opaque_value` to obtain the interpreted sample value, and any auxiliary image luminance sample less than or equal to (alpha_opaque_value - alpha_transparent_value) / 2 is set to equal to `alpha_transparent_value` to obtain the interpreted sample value. An `alpha_channel_clip_type_flag` value of 1 indicates that, for alpha blending purposes, after decoding the auxiliary image samples, any auxiliary image luminance sample greater than `alpha_opaque_value` is set to equal to `alpha_opaque_value` to obtain the interpreted sample value, and any auxiliary image luminance sample less than or equal to `alpha_transparent_value` is set to equal to `alpha_transparent_value` to obtain the interpreted sample value.
[0713] Note - When both alpha_channel_incr_flag and alpha_channel_clip_flag are equal to 1, the clipping operation specified by alpha_channel_clip_type_flag should be applied first, and then the changes specified by alpha_channel_incr_flag should be applied to obtain the interpretation sample values of the auxiliary image brightness samples.
[0714] Example 22
[0715] Scalability Dimension Information (SDI) SEI Message
[0716] Scalability Dimension SEI Message Syntax
[0717]
[0718]
[0719] Scalability Dimension SEI Message Semantics
[0720] The Scalability Dimension (SEI) message provides scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) the view ID of each layer when bitstreamInScope may be a multi-view bitstream; and 2) the auxiliary ID of each layer when bitstreamInScope may have one or more layers carrying auxiliary information (such as depth or alpha).
[0721] bitstreamInScope is an AU sequence that includes, in the decoding order, an initial AU containing the SDI SEI message and zero or more subsequent AUs, up to but not including any subsequent AU containing another SDI SEI message.
[0722] The increment of sdi_max_layers_minus1 indicates the maximum number of layers in bitstreamInScope.
[0723] A value of 1 for `sdi_multiview_info_flag` indicates that `bitstreamInScope` may be a multiview bitstream, and that the `sdi_view_id_val[]` syntax element exists in the scalability dimension SEI message. A value of 0 for `sdi_multiview_flag` indicates that `bitstreamInScope` is not a multiview bitstream, and that the `sdi_view_id_val[]` syntax element does not exist in the scalability dimension SEI message.
[0724] A value of 1 for `sdi_auxiliary_info_flag` indicates that auxiliary information may exist, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element exists in the scalability dimension SEI message. A value of 0 for `sdi_auxiliary_info_flag` indicates that no auxiliary information exists, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element does not exist in the scalability dimension SEI message.
[0725] The length of the sdi_view_id_len syntax element is specified in bits.
[0726] `sdi_view_id_val[i]` specifies the view ID of the i-th level in `bitstreamInScope`. The length of the `sdi_view_id_val[i]` syntax element is `sdi_view_id_len` bits. If it does not exist, the value of `sdi_view_id_val[i]` is inferred to be equal to 0.
[0727] A value of 0 for sdi_aux_id[i] indicates that the i-th layer in bitstreamInScope does not contain auxiliary images. A value greater than 0 for sdi_aux_id[i] indicates the type of auxiliary image in the i-th layer of bitstreamInScope as specified in Table 1.
[0728]
[0729] A value of 0 for sdi_aux_id[i] indicates that the i-th layer in bitstreamInScope does not contain auxiliary images. A value greater than 0 for sdi_aux_id[i] indicates the type of auxiliary image in the i-th layer of bitstreamInScope as specified in Table 1.
[0730] Table 1 – Mapping from sdi_aux_id[i] to auxiliary image types
[0731] sdi_aux_id[i] name Types of auxiliary images 1 AUX_ALPHA α plane 2 AUX_DEPTH Depth Images 3..127 Reserved 128..159 Unspecified 160..255 Reserved
[0732] Note 1 - Explanation of the auxiliary picture associated with sdi_aux_id in the range of 128 to 159 (inclusive) by means other than the sdi_aux_id value.
[0733] For this version of the bitstream conforming to this specification, sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive). Although the value of sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive), in this version of the specification, the decoder should allow the value of sdi_aux_id[i] to be in the range of 0 to 255 (inclusive).
[0734] Retrieving SEI messages from multiple views
[0735] SEI Message Syntax for Retrieving Information from Multiple Views
[0736]
[0737]
[0738] Multi-view information retrieval SEI message semantics
[0739] Obtaining information from multiple views The SEI message specifies various parameters for acquiring the environment. Specifically, it specifies the parameters for both internal and external cameras. These parameters can be used to process and decode the view before rendering it on a 3D display.
[0740]
[0741]
[0742] The following semantics apply to each nuh_layer_id targetLayerId in the nuh_layer_id value applied to the SEI message for multi-view information retrieval.
[0743] When present, the SEI message for multi-view acquisition applied to the current layer should be included in the access unit containing the IRAP image, which is the first image of the current layer's CLVS. The signaling notification information in the SEI message is applied to the CLVS.
[0744] When a multi-view information SEI message is contained within a scalable nested SEI message, the syntax elements sn_ols_flag and sn_all_layers_flag in the scalable nested SEI message should be equal to 0.
[0745] The variable numViewsMinus1 is exported as follows:
[0746] --If the multi-view information SEI message is not included in the scalable nested SEI message, then numViewsMinus1 is set to equal to 0.
[0747] --Otherwise (the multi-view SEI message is contained within the scalable nested SEI message), numViewsMinus1 is set to equal sn_num_layers_minus1.
[0748] The multi-view retrieval information contained in the multi-view retrieval information SEI message indicates that some views may not exist.
[0749] In the following semantics, index i refers to the syntax element and variable applied to the layer whose nuh_layer_id is equal to NestingLayerId[i].
[0750] The external camera parameters are defined according to the right-hand coordinate system, where the top-left corner of the image is the origin, i.e., the coordinates (0, 0), and the other corners of the image have non-negative coordinates. Based on these specifications, for the i-th camera, the 3D world point wP = [xyz] is mapped to the 2D camera point cP[i] = [uv 1], according to:
[0751] s*cP[i]=A[i]*R -1 [i]*(wP-T[i]) (X)
[0752] Where A[i] represents the internal camera parameter matrix, R -1 [i] denotes the inverse of the rotation matrix R[i], T[i] denotes the translation vector, and s (scalar value) is an arbitrary scale factor chosen so that the third coordinate of cP[i] is equal to 1. The elements of A[i], R[i], and T[i] are determined according to the syntax elements of the signaling notification in the SEI message, and are specified as follows.
[0753] An intrinsic_param_flag value of 1 indicates the existence of internal camera parameters. An intrinsic_param_flag value of 0 indicates the absence of internal camera parameters.
[0754] An extrinsic_param_flag value of 1 indicates the presence of external camera parameters. An extrinsic_param_flag value of 0 indicates the absence of external camera parameters.
[0755] An intrinsic_params_equal_flag value of 1 indicates that all cameras have the same intrinsic camera parameters, and there is only one set of intrinsic camera parameters. An intrinsic_params_equal_flag value of 0 indicates that each camera has different intrinsic camera parameters, and each camera has its own set of intrinsic camera parameters.
[0756] prec_focal_length is specified as 2 -prec_focal_length The exponent of the maximum permissible truncation error for focal_length_x[i] and focal_length_y[i]. The value of prec_focal_length should be in the range of 0 to 31 (inclusive).
[0757] prec_principal_point specifies that 2 -prec_principal_point The exponent of the maximum permissible truncation error for principal_point_x[i] and principal_point_y[i]. The value of prec_principal_point should be in the range of 0 to 31 (inclusive).
[0758] prec_skew_factor is specified by 2 -prec_skew_factor The exponent of the maximum permissible truncation error given the skew factor. The value of prec_skew_factor should be in the range of 0 to 31 (inclusive).
[0759] A value of 0 for sign_focal_length_x[i] indicates that the focal length of the i-th camera in the horizontal direction is positive. A value of 1 for sign_focal_length_x[i] indicates that the focal length is negative.
[0760] `exponent_focal_length_x[i]` specifies the exponential portion of the focal length of the i-th camera in the horizontal direction. The value of `exponent_focal_length_x[i]` should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should interpret value 63 as an indication of an unspecified focal length.
[0761] `mantissa_focal_length_x[i]` specifies the mantissa portion of the focal length of the i-th camera in the horizontal direction. The length of the `mantissa_focal_length_x[i]` syntax element is variable and is determined as follows:
[0762] --If exponent_focal_length_x[i] equals 0, then the length is Max(0,prec_focal_length-30).
[0763] --Otherwise (exponent_focal_length_x[i] is in the range of 0 to 63 (inclusive), the length is Max(0,exponent_focal_length_x[i]+prec_focal_length-31).
[0764] A value of 0 for sign_focal_length_y[i] indicates that the focal length in the vertical direction of the i-th camera is positive. A value of 1 for sign_focal_length_y[i] indicates that the focal length is negative.
[0765] `exponent_focal_length_y[i]` specifies the exponential portion of the focal length in the vertical direction for the i-th camera. The value of `exponent_focal_length_y[i]` should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should interpret value 63 as an indication of an unspecified focal length.
[0766] `mantissa_focal_length_y[i]` specifies the last part of the focal length of the i-th camera in the vertical direction.
[0767] The length of the mantissa_focal_length_y[i] syntax element is variable and is determined as follows:
[0768] --If exponent_focal_length_y[i] equals 0, then the length is Max(0,prec_focal_length-30).
[0769] --Otherwise (exponent_focal_length_y[i] is in the range of 0 to 63 (excluding the end value), the length is Max(0,exponent_focal_length_y[i]+prec_focal_length-31).
[0770] A value of 0 for sign_principal_point_x[i] indicates that the principal point of the i-th camera has a positive sign in the horizontal direction. A value of 1 for sign_principal_point_x[i] indicates that the sign is negative.
[0771] `exponent_principal_point_x[i]` specifies the exponent portion of the principal point of the i-th camera in the horizontal direction. The value of `exponent_principal_point_x[i]` should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should treat value 63 as indicating an unspecified principal point.
[0772] `mantissa_principal_point_x[i]` specifies the mantissa portion of the principal point of the i-th camera in the horizontal direction. The length of the `mantissa_principal_point_x[i]` syntax element, in bits, is variable and is determined as follows:
[0773] --If exponent_principal_point_x[i] equals 0, then the length is Max(0, prec_principal_point-30).
[0774] --Otherwise (exponent_principal_point_x[i] is in the range of 0 to 63 (excluding the end value), the length is Max(0,exponent_principal_point_x[i]+prec_principal_point-31).
[0775] A value of 0 for sign_principal_point_y[i] indicates that the sign of the principal point of the i-th camera is positive in the vertical direction. A value of 1 for sign_principal_point_y[i] indicates that the sign is negative.
[0776] `exponent_principal_point_y[i]` specifies the exponential portion of the principal point of the i-th camera in the vertical direction. The value of `exponent_principal_point_y[i]` should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should treat value 63 as an indication of an unspecified principal point.
[0777] `mantissa_principal_point_y[i]` specifies the mantissa portion of the principal point of the i-th camera in the vertical direction. The length of the `mantissa_principal_point_y[i]` syntax element, in bits, is variable and is determined as follows:
[0778] --If exponent_principal_point_y[i] equals 0, then the length is Max(0, prec_principal_point-30).
[0779] --Otherwise (exponent_principal_point_y[i] is in the range of 0 to 63 (excluding the end value), the length is Max(0,exponent_principal_point_y[i]+prec_principal_point-31).
[0780] The value of sign_skew_factor[i] being 0 indicates that the sign of the tilt factor of the i-th camera is positive.
[0781] The sign_skew_factor[i] being equal to 1 indicates that the sign is negative.
[0782] `exponent_skew_factor[i]` specifies the exponent portion of the tilt factor for the i-th camera. The value of `exponent_skew_factor[i]` should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should interpret value 63 as an indication of an unspecified tilt factor.
[0783] `mantissa_skew_factor[i]` specifies the mantissa portion of the tilt factor for the i-th camera. The length of the `mantissa_skew_factor[i]` syntax element is variable and is determined as follows:
[0784] --If exponent_skew_factor[i] equals 0, then the length is Max(0,prec_skew_factor-30).
[0785] --Otherwise (exponent_skew_factor[i] is in the range of 0 to 63 (excluding the end value), the length is Max(0,exponent_skew_factor[i]+prec_skew_factor-31).
[0786] The internal matrix A[i] of the i-th camera is represented by the following formula.
[0787]
[0788] prec_rotation_param specifies that 2 -prec_rotation_param The exponent of the maximum permissible truncation error for r[i][j][k]. The value of prec_rotation_param should be in the range of 0 to 31 (inclusive).
[0789] prec_translation_param specifies that it is 2 -prec_translation_param The exponent of the maximum permissible truncation error for t[i][j]. The value of prec_translation_param should be in the range of 0 to 31 (inclusive).
[0790] A sign_r[i][j][k] equal to 0 indicates that the (j, k) component of the rotation matrix of the i-th camera has a positive sign. A sign_r[i][j][k] equal to 1 indicates that the sign is negative.
[0791] The exponent_r[i][j][k] specifies the exponent portion of the (j, k) component of the rotation matrix for the i-th camera. The value of the exponent r[i][j][k] should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should interpret value 63 as indicating an unspecified rotation matrix.
[0792] `mantissa_r[i][j][k]` specifies the mantissa portion of the (j, k) component of the rotation matrix for the i-th camera. The length of the `mantissa_r[i][j][k]` syntax element, in bits, is variable and is determined as follows:
[0793] -- If exponent_r[i] is equal to 0, the length is Max(0, prec_rotation_param-30).
[0794] -- Otherwise (exponent_r[i] is in the range of 0 to 63, excluding the endpoints), the length is Max(0, exponent_r[i] + prec_rotation_param-31).
[0795] The rotation matrix R[i] of the i-th camera is expressed as follows:
[0796]
[0797] sign_t[i][j] being equal to 0 indicates that the sign of the j-th component of the translation vector of the i-th camera is positive. sign_t[i][j] being equal to 1 indicates that the sign is negative.
[0798] exponent_t[i][j] specifies the exponent part of the j-th component of the translation vector of the i-th camera. The value of exponent_t[i][j] shall be in the range of 0 to 62, inclusive. The value 63 is reserved for future use by ITU-T|ISO / IEC. The decoder shall treat the value 63 as indicating an unspecified translation vector.
[0799] mantissa_t[i][j] specifies the mantissa part of the j-th component of the translation vector of the i-th camera. The length v of the mantissa_t[i][j] syntax element in bits is variable and is determined as follows:
[0800] -- If exponent_t[i] is equal to 0, the length v is set equal to Max(0, prec_translation_param-30).
[0801] -- Otherwise (0 < exponent_t[i] < 63), the length v is set equal to Max(0, exponent_t[i] + prec_translation_param-31).
[0802] The translation vector T[i] of the i-th camera is expressed by the following formula:
[0803]
[0804] The association between camera parameter variables and the corresponding syntax elements is specified in Table ZZ. Each component of the intrinsic matrix and the rotation matrix, as well as the translation vector, is obtained from the variables specified in Table ZZ, and the variable x is calculated as follows:
[0805] --If e is in the range of 0 to 63 (excluding extreme values), then x is set to (-1). s *2 e-31 *(1+n÷2 v ).
[0806] Otherwise (if e equals 0), x is set to equal to (-1). s *2 -(30+v) *n.
[0807] Note - The above specifications are similar to those of IEC 60559:1989.
[0808] Table ZZ – Relationship between camera parameter variables and syntax elements.
[0809]
[0810]
[0811] Figure 4 This is a block diagram of an example video processing system 400 that can implement the various techniques disclosed herein. Various implementations may include some or all of the components of the video processing system 400. The video processing system 400 may include an input 402 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8 or 10-bit multi-component pixel values), or in a compressed or encoded format. Input 402 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces (such as Ethernet, Passive Optical Networking (PON), etc.) and wireless interfaces (such as Wi-Fi or cellular interfaces).
[0812] Video processing system 400 may include codec component 404 capable of implementing the various codec or encoding methods described herein. Codec component 404 can reduce the average bit rate of the video from input 402 to the output of codec component 404 to produce a codec representation of the video. Therefore, codec techniques are sometimes referred to as video compression or video transcoding techniques. The output of codec component 404 may be stored or transmitted via connected communication, as represented by component 406. The stored or communicated bitstream (or codec) representation of the video received at input 402 may be used by component 408 to generate pixel values or displayable video that is sent to display interface 410. The process of generating user-visible video from the bitstream representation is sometimes referred to as video decompression. Furthermore, although some video processing operations are referred to as “codec” operations or tools, it should be understood that the codec tool or operation is used at the encoder, and the corresponding decoding tool or operation will be inverted by the decoder to retrieve the result of the codec.
[0813] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or DisplayPort. Examples of storage interfaces include SATA (Serial Advanced Technology Accessory), Peripheral Component Interconnect (PCI), and Integrated Drive Electronic Devices (IDE) interfaces. The technologies described in this document can be implemented in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of digital data processing and / or video display.
[0814] Figure 5 This is a block diagram of a video processing apparatus 500. Apparatus 500 can be used to implement one or more of the methods described herein. Apparatus 500 can be implemented in smartphones, tablets, computers, Internet of Things (IoT) receivers, etc. Apparatus 500 may include one or more processors 502, one or more memories 504, and video processing hardware 506 (also referred to as video processing circuitry). The processors(multiple) 502 can be configured to implement one or more methods described herein. The memories(multiple) 504 can be used to store data and code used to implement the methods and techniques described herein. The video processing hardware 506 can be used to implement some of the techniques described herein in hardware circuitry. In some embodiments, hardware 506 may be partially or entirely located within processor 502, such as a graphics processor.
[0815] Figure 6 This is a block diagram illustrating an example video codec system 600 that can utilize the techniques disclosed herein. Figure 6 As shown, the video encoding / decoding system 600 may include a source device 610 and a destination device 620. The source device 610 generates encoded video data and may be referred to as a video encoding device. The destination device 620 can decode the encoded video data generated by the source device 610 and may be referred to as a video decoding device.
[0816] The source device 610 may include a video source 612, a video encoder 614, and an input / output (I / O) interface 616.
[0817] Video source 612 may include sources such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system that generates video data, or a combination of these sources. Video data may include one or more pictures. Video encoder 614 encodes the video data from video source 612 to generate a bitstream. The bitstream may include a sequence of bits forming a codec representation of the video data. The bitstream may include codec pictures and associated data. A codec picture is a codec representation of a picture. Associated data may include sequence parameter sets, picture parameter sets, and other syntax elements. I / O interface 616 includes a modulator / demodulator (modem) and / or a transmitter. Encoded video data may be transmitted directly to destination device 620 via network 630 through I / O interface 616. Encoded video data may also be stored on storage medium / server 640 for access by destination device 620.
[0818] Destination device 620 may include I / O interface 626, video decoder 624, and display device 622.
[0819] I / O interface 626 may include a receiver and / or a modem. I / O interface 626 may acquire encoded video data from source device 610 or storage medium / server 640. Video decoder 624 may decode the encoded video data. Display device 622 may display the decoded video data to a user. Display device 622 may be integrated with destination device 620 or may be external to destination device 620 configured to connect to an external display device.
[0820] The video encoder 614 and the video decoder 624 can operate according to video compression standards such as the High Efficiency Video Codec (HEVC) standard, the Multi-Functional Video Codec (VVC) standard, and other current and / or other standards.
[0821] Figure 7 This is a block diagram illustrating an example of a video encoder 700, which can be... Figure 6 The video encoder 614 in the video encoding and decoding system 600 shown in the figure.
[0822] The video encoder 700 can be configured to perform any or all of the techniques disclosed herein. Figure 7 In the example, the video encoder 700 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video encoder 700. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.
[0823] The functional components of the video encoder 700 may include a segmentation unit 701, a prediction unit 702 (which may include a mode selection unit 703, a motion estimation unit 704, a motion compensation unit 705, and an intra-frame prediction unit 706), a residual generation unit 707, a transform unit 708, a quantization unit 709, an inverse quantization unit 710, an inverse transform unit 711, a reconstruction unit 712, a buffer 713, and an entropy coding unit 714.
[0824] In other examples, the video encoder 700 may include more, fewer, or different functional components. In one example, the prediction unit 702 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in IBC mode, where at least one reference picture is the picture in which the current video block is located.
[0825] Furthermore, some components, such as the motion estimation unit 704 and the motion compensation unit 705, can be highly integrated, but for illustrative purposes... Figure 7 The examples are shown separately.
[0826] The segmentation unit 701 can segment an image into one or more video blocks. Figure 6 The video encoder 614 and video decoder 600 can support various video block sizes.
[0827] The mode selection unit 703 can, for example, select one of the intra-frame or inter-frame encoding / decoding modes based on the error result, and provide the obtained intra-frame or inter-frame encoded / decoded blocks to the residual generation unit 707 to generate residual block data and to the reconstruction unit 712 to reconstruct the encoded / decoded blocks for use as reference images. In some examples, the mode selection unit 703 can select a combined intra-frame and inter-frame prediction (CIIP) mode, where the prediction is based on the inter-frame prediction signal and the intra-frame prediction signal. The mode selection unit 703 can also select the resolution of the motion vector (e.g., sub-pixel or integer pixel precision) for the blocks in the inter-frame prediction case.
[0828] To perform inter-frame prediction for the current video block, motion estimation unit 704 can generate motion information for the current video block by comparing one or more reference frames from buffer 713 with the current video block. Motion compensation unit 705 can determine the predicted video block for the current video block based on the motion information of the image from buffer 713 (rather than the image associated with the current video block) and decoded samples.
[0829] The motion estimation unit 704 and motion compensation unit 705 can perform different operations on the current video block, depending on whether the current video block is in an I-band, P-band, or B-band. I-bands (or I-frames) have the lowest compression ratio but do not require other video frames for decoding. S-bands (or P-frames) can use data from previous frames for decompression and are easier to compress than I-frames. B-bands (or B-frames) can use both previous and preceding frames as data references to achieve the highest data compression.
[0830] In some examples, motion estimation unit 704 can perform unidirectional prediction of the current video block, and can search for a reference video block for the current video block in the reference images of list 0 or list 1. Motion estimation unit 704 can then generate a reference index indicating that the reference image in list 0 or list 1 contains the reference video block, and a motion vector indicating the spatial displacement between the current video block and the reference video block. Motion estimation unit 704 can output the reference index, prediction direction indicator, and motion vector as motion information for the current video block. Motion compensation unit 705 can generate a predicted video block for the current block based on the reference video block indicated by the motion information of the current video block.
[0831] In other examples, motion estimation unit 704 can perform bidirectional prediction of the current video block. Motion estimation unit 704 can search for a reference video block for the current video block in the reference images of list 0 and can also search for another reference video block for the current video block in the reference images of list 1. Motion estimation unit 704 can then generate a reference index indicating that the reference images in list 0 or list 1 contain the reference video block, and a motion vector indicating the spatial displacement between the reference video block and the current video block. Motion estimation unit 704 can output the reference index and the motion vector of the current video block as the motion information of the current video block. Motion compensation unit 705 can generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.
[0832] In some examples, the motion estimation unit 704 may output the complete set of motion information for the decoder's decoding process. In other examples, the motion estimation unit 704 may not output the complete set of motion information for the current video. Instead, the motion estimation unit 704 may refer to the motion information of another video block to signal the motion information of the current video block. For example, the motion estimation unit 704 may determine that the motion information of the current video block is sufficiently similar to the motion information of an adjacent video block.
[0833] In one example, the motion estimation unit 704 may indicate in the syntax structure associated with the current video block that the video decoder 624 indicates that the current video block has the same motion information value as another video block.
[0834] In another example, motion estimation unit 704 can identify another video block and motion vector difference (MVD) in the syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicating video block. Video decoder 624 can use the motion vector of the indicating video block and the motion vector difference to determine the motion vector of the current video block.
[0835] As discussed above, the video encoder 614 can predictively signal motion vectors. Two examples of predictive signaling notification techniques that can be implemented by the video encoder 614 include Advanced Motion Vector Prediction (AMVP) and merge pattern signaling notification.
[0836] The intra-prediction unit 706 can perform intra-prediction on the current video block. When performing intra-prediction on the current video block, the intra-prediction unit 706 can generate prediction data for the current video block based on the decoded samples of other video blocks in the same frame. The prediction data for the current video block may include the predicted video block and various syntax elements.
[0837] The residual generation unit 707 can generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) multiple predicted video blocks from the current video block. The residual data for the current video block can include residual video blocks corresponding to different sample components of the samples in the current video block.
[0838] In other examples, such as in skip mode, residual data for the current video block may not exist, and the residual generation unit 707 may not perform a subtraction operation.
[0839] Transform unit 708 can generate one or more transform coefficient video blocks of the current video block by applying one or more transforms to the residual video block associated with the current video block.
[0840] After the transform unit 708 generates a transform coefficient video block associated with the current video block, the quantization unit 709 can quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values associated with the current video block.
[0841] The inverse quantization unit 710 and the inverse transform unit 711 can apply inverse quantization and inverse transform to the transform coefficient video block respectively to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 712 can add the reconstructed residual video block to the corresponding samples of one or more predicted video blocks generated by the prediction unit 702 to produce a reconstructed video block associated with the current block for storage in the buffer 713.
[0842] After the video block is reconstructed by reconstruction unit 712, a loop filtering operation can be performed to reduce video block artifacts in the video block.
[0843] The entropy encoding unit 714 can receive data from other functional components of the video encoder 700. When the entropy encoding unit 714 receives data, it can perform one or more entropy encoding operations to generate entropy-encoded data and output a bitstream including the entropy-encoded data.
[0844] Figure 8 This is a block diagram illustrating an example of a video decoder 800, which can be... Figure 6 The video decoder 624 in the video encoding and decoding system 600 shown in the figure.
[0845] The video decoder 800 can be configured to perform any or all of the techniques disclosed herein. Figure 8 In the example, the video decoder 800 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video decoder 800. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.
[0846] exist Figure 8 In the example, the video decoder 800 includes an entropy decoding unit 801, a motion compensation unit 802, an intra-frame prediction unit 809, an inverse quantization unit 804, an inverse transform unit 805, a reconstruction unit 806, and a buffer 807. In some examples, the video decoder 800 can perform operations related to the video encoder 614 ( Figure 6 The decoding process is the overall inversion of the encoding process described.
[0847] The entropy decoding unit 801 can retrieve the encoded bitstream. The encoded bitstream may include entropy-encoded video data (e.g., encoded blocks of video data). The entropy decoding unit 801 can decode the entropy-encoded video, and based on the entropy-encoded video data, the motion compensation unit 802 can determine motion information including motion vectors, motion vector precision, reference image list index, and other motion information. The motion compensation unit 802 can determine such information, for example, by performing AMVP and merge mode signaling notifications.
[0848] The motion compensation unit 802 can generate motion compensation blocks, possibly based on interpolation filters. The identifier of the interpolation filter to be used at sub-pixel precision can be included in the syntax element.
[0849] The motion compensation unit 802 can use the interpolation filter used by the video encoder 614 during the encoding of the video block to calculate the interpolation values of a sub-integer number of pixels of the reference block. The motion compensation unit 802 can determine the interpolation filter used by the video encoder 614 based on the received syntax information and use the interpolation filter to generate the prediction block.
[0850] The motion compensation unit 802 can use some syntax information to determine: the size of the blocks used to encode (multiple) frames and / or (multiple) stripes of the encoded video sequence, segmentation information describing how each macroblock of the image of the encoded video sequence is segmented, the mode indicating how each segment is encoded, one or more reference frames (and a list of reference frames) for each inter-frame coded block, and other information for decoding the encoded video sequence.
[0851] Intra-prediction unit 803 can use, for example, an intra-prediction mode received in the bitstream to form prediction blocks from spatially adjacent blocks. Inverse quantization unit 804 inverse quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 801. Inverse transform unit 805 applies the inverse transform.
[0852] The reconstruction unit 806 can sum the residual blocks using the corresponding prediction blocks generated by the motion compensation unit 802 or the intra-frame prediction unit 803 to form a decoded block. As desired, a deblocking filter can also be applied to filter the decoded block to remove block artifacts. The decoded video block is then stored in a buffer 807, which provides a reference block for subsequent motion compensation / intra-frame prediction and also produces the decoded video for presentation on the display device.
[0853] Figure 9 This is a method 900 for encoding and decoding video data according to embodiments of the present disclosure. Method 900 can be executed by an encoding / decoding device (e.g., an encoder) having a processor and memory. Method 900 can be implemented when information is transmitted in a bitstream using SEI messages.
[0854] In block 902, the codec uses Scalability Dimension Information (SDI) Supplemental Enhancement Information (SEI) messages to indicate the SDI view identifier length minus the L syntax element. An SDI SEI message is a type of SEI message, for example, similar to... Figure 3 The SEI message in bitstream 300. The SDI view identifier length minus L syntax element is a syntax element, for example, similar to Figure 3 Syntax element 324 in bitstream 300. SEI messages, including SDI SEI messages, can carry any syntax elements disclosed here.
[0855] In block 904, the codec unit performs conversion between video media files and bitstreams based on SDI SEI messages.
[0856] When implemented in an encoder, the conversion involves receiving a media file (e.g., a video unit) and encoding the SEI messages into a bitstream. When implemented in a decoder, the conversion involves receiving a bitstream including the SEI messages and decoding the SEI messages in the bitstream to generate a video media file.
[0857] In this embodiment, the SDI view identifier length minus L syntax element is configured to prevent the length of the SDI view identifier value syntax element from being zero. The SDI view identifier value syntax element specifies the view identifier of the i-th layer in the bitstream. In this embodiment, L equals 1. In this embodiment, the SDI view identifier length minus L syntax element is specified as sdi_view_id_len_minus1. In this embodiment, the SDI view identifier value syntax element is specified as sdi_view_id_val[i]. In this embodiment, the SDI view identifier length minus L syntax element plus 1 specifies the length of the SDI view identifier value syntax element.
[0858] In this embodiment, the SDI view identifier length minus the L syntax element is encoded and decoded into an unsigned integer using N bits. For example, an unsigned integer is an integer (e.g., an integer) that does not have an associated sign (e.g., positive or negative). In this embodiment, N equals 4.
[0859] In an embodiment, the SDI view identifier length minus L syntax element is encoded as a syntax element using an N-bit fixed-pattern bit string, an N-bit signed integer, truncated binary, a signed integer K-order exponential Golomb codec, where K equals 0, or an unsigned integer M-order exponential Golomb codec, where M equals 0. A bit string is a compact array data structure storing bits. A fixed-pattern bit string is an array data structure with a fixed pattern. A signed integer is an integer (e.g., an integer) with an associated sign (e.g., positive or negative). Truncated binary or truncated binary encoding is an entropy encoding typically used for uniform probability distributions of a finite alphabet. Exponential Golomb code (Exponential Golomb code) is a general-purpose code.
[0860] In this embodiment, the bitstream is a range of bitstreams. In this embodiment, the range of bitstreams is a sequence of Access Units (AUs) in the decoding order, including an initial AU containing an SDI SEI message, followed by zero or more subsequent AUs, up to but not including any subsequent AU containing another SDI SEI message.
[0861] In this embodiment, the Multi-View Information (SEI) message and the Auxiliary Information (SEI) message are not present in the Codec Video Sequence (CVS) unless the SDI SEI message is present in the CVS.
[0862] In this embodiment, the multi-view information SEI message includes a multi-view acquisition information SEI message. In this embodiment, the auxiliary information SEI message includes a depth representation information SEI message. In this embodiment, the auxiliary information SEI message includes an alpha channel information SEI message.
[0863] In an embodiment, when a Multiview Information (SEI) message or an Auxiliary Information (SEI) message is present in the bitstream, one or more of the SDI Multiview Information flags (e.g., the sdi_multiview_info flag) and the SDI Auxiliary Information flags (e.g., the sdi_auxiliary_info_flag) are equal to 1. Flags are variables or single-bit syntax elements that can take one of two possible values: 0 and 1.
[0864] In this embodiment, the multi-view information SEI message includes a multi-view acquisition information SEI message, and the multi-view acquisition information SEI message is not scalable nested. A scalable nested SEI message is an SEI message within a scalable nested SEI message. A scalable nested SEI message is a message containing multiple scalable nested SEI messages, which correspond to one or more layers in one or more output layer sets or multi-layer bitstreams.
[0865] In this embodiment, SEI messages in the bitstream with a payload type equal to 179 are constrained from being included in scalable nested SEI messages. In this embodiment, SEI messages in the bitstream with a payload type equal to 3, 133, 179, 180, or 205 are constrained from being included in scalable nested SEI messages.
[0866] In an embodiment, method 900 may utilize or combine one or more features or processes of other methods disclosed herein.
[0867] The following provides a list of preferred solutions for some embodiments.
[0868] The following solutions illustrate example embodiments of the techniques discussed in this disclosure (e.g., Example 1).
[0869] 1. A video processing method, comprising: performing a conversion between a video and a bitstream of the video; wherein the bitstream conforms to a format rule; wherein the format rule specifies that the length of a syntax element indicating a view identifier syntax element is less than L, where L is an integer.
[0870] 2. The method described in Solution 1, wherein N bits are used to encode and decode the syntax elements into unsigned integers.
[0871] 3. The method according to any one of solutions 1-2, wherein L is a positive integer.
[0872] 4. The method according to Solution 1, wherein L = 0, and wherein the syntax element is not allowed to have a zero value.
[0873] 5. A method for video processing, comprising: performing a conversion between a video comprising multiple layers and a bitstream of the video, wherein the bitstream conforms to a format rule, wherein the format rule specifies that the bitstream includes auxiliary layers associated with one or more associated layers of the video.
[0874] 6. The method according to Solution 5, wherein the format rules further specify whether or how the bitstream includes one or more syntax elements indicating the relationship between an auxiliary layer and one or more associated layers, wherein the one or more syntax elements are included in a scalability dimension supplemental enhancement information syntax structure.
[0875] 7. The method according to Solution 6, wherein the format rules specify that one or more associated layers are indicated by corresponding layer identifiers (IDs).
[0876] 8. The method according to Solution 6, wherein the format rule specifies that one or more associated layers are indicated by the corresponding layer index.
[0877] 9. The method according to any one of solutions 5-8, wherein the format rules specify that the bitstream includes one or more syntax elements that indicate whether an auxiliary layer is applicable to one or more associated layers.
[0878] 10. The method according to Solution 9, wherein the one or more syntax elements include syntax elements indicating that the auxiliary layer applies to all one or more associated layers.
[0879] 11. The method according to Solution 9, wherein the formatting rules specify that each associated layer includes a syntax element that indicates whether the auxiliary layer is applicable to the corresponding associated layer.
[0880] 12. The method according to solution 11, wherein the syntax elements indicate all the main layers associated with the auxiliary layers.
[0881] 13. The method according to solution 11, wherein the syntax element indicates all major layers that are associated with the auxiliary layer and have a layer index that is less than the layer index of the auxiliary layer.
[0882] 14. The method according to solution 11, wherein the syntax element indicates all major layers associated with the auxiliary layer and having a layer index greater than that of the auxiliary layer.
[0883] 15. The method according to any one of solutions 11-14, wherein the syntax element is a flag.
[0884] 16. The method according to Solution 6, wherein the format rules specify that the bitstream does not include explicit syntax elements indicating the applicability of the auxiliary layer to one or more associated layers, and the applicability is derived during conversion.
[0885] 17. The method according to solution 16, wherein the format rules specify that the associated layers of the auxiliary layer have a layer ID equal to the layer ID of the auxiliary layer plus N1, N2, ... Nk, where k is an integer, and for i = 1, ... k, no two Ni are equal to each other.
[0886] 18. The method according to solution 17, wherein k = 1 and N1 is one of 1, -1, 2 or -2.
[0887] 19. The method described in solution 17, wherein k is greater than 1.
[0888] 20. The method described in solution 19, where k equals 2, and N1 = 1, N2 = 2.
[0889] 21. The method according to Solution 5, wherein the format rules further specify that the bitstream omits one or more syntax elements indicating the relationship between an auxiliary layer and one or more associated layers, and wherein the relationship is derived based on predetermined rules.
[0890] 22. The method according to Solution 5, wherein the format rules further specify that the bitstream contains one or more syntax elements indicating the relationship between an auxiliary layer and one or more associated layers, wherein the one or more syntax elements are contained in an auxiliary information supplementary enhancement information syntax structure.
[0891] 23. The method according to any one of solutions 5-22, wherein the format rules specify the inclusion of syntax elements in the bitstream that indicate the number of associated layers of the auxiliary picture of the layer.
[0892] 24. The method according to any one of solutions 5-22, wherein the format rules specify that, if certain conditions are met, a syntax element is included in the bitstream, the syntax element indicating the number of associated layers of the auxiliary picture of the layer or the number of associated layers of the auxiliary picture.
[0893] 25. The method according to solution 24, wherein the condition includes that the i-th layer in bitstreamInScope includes an auxiliary image.
[0894] 26. A video processing method, comprising: performing a conversion between a video comprising multiple video layers and a bitstream of the video, wherein the bitstream conforms to a format rule, wherein the format rule specifies whether a scalability dimension information (SEI) message is included in a encoded video sequence in response to scalability dimension information (SEI) messages, the encoded video sequence of the bitstream including multi-view supplemental enhancement information (SEI) messages or auxiliary information (SEI) messages.
[0895] 27. The method described in Solution 26, wherein the format rules specify that the Multi-View Information SEI message refers to the Multi-View Acquisition Information SEI message.
[0896] 28. The method according to any one of solutions 26-27, wherein the format rule specifies that the auxiliary information SEI message refers to the depth representation information SEI message or the alpha channel information SEI message.
[0897] 29. A method of video processing, comprising: performing a conversion between a video comprising multiple video layers and a bitstream of the video, wherein the bitstream conforms to a format rule, wherein the format rule specifies that, in response to a multi-view supplemental enhancement information (SEI) message or an auxiliary information supplemental enhancement information (SEI) message present in the bitstream, at least one of a first flag indicating the presence of multi-view information or a second flag indicating the presence of auxiliary information in a scalability dimension information (SEI) message is equal to 1.
[0898] 30. A video processing method, comprising: performing a conversion between a video comprising multiple video layers and a bitstream of the video, wherein the bitstream conforms to a format rule, wherein the format rule specifies that a multi-view acquisition information supplemental enhancement message included in the bitstream is not scalable nested, or is not included in a scalable nested supplemental enhancement message.
[0899] 31. The method according to any one of solutions 1-30, wherein the conversion includes generating video from a bitstream or generating a bitstream from a video.
[0900] 32. A method for storing a bit stream on a computer-readable medium, comprising generating the bit stream according to any one or more of the methods in solutions 1-31 and storing the bit stream on a computer-readable medium.
[0901] 33. A computer-readable medium having a bitstream of video stored thereon, the bitstream causing the video decoder to generate video when processed by a processor of a video decoder, wherein the bitstream is generated according to one or more of the methods in solutions 1-31.
[0902] 34. A video decoding apparatus, comprising a processor configured to implement the method described in one or more of solutions 1 to 31.
[0903] 35. A video encoding apparatus comprising a processor configured to implement the method described in one or more of solutions 1 to 31.
[0904] 36. A computer program product having computer code stored thereon, which, when executed by a processor, causes the processor to implement the method of any one of solutions 1 to 31.
[0905] 37. A computer-readable medium having a bit stream recorded thereon in a bit stream format conforming to any one of solutions 1 to 31.
[0906] 38. A method, an apparatus, or a bit stream generated according to the disclosed method or system described in this document.
[0907] The following documents may include additional details related to the technology disclosed herein:
[0908] [1]ITU-T and ISO / IEC, “Efficient video coding and decoding”, Rec.ITU-T H.265|ISO / IEC 23008-2 (current version).
[0909] [2] J. Chen, E. Alshina, G. J. Sullivan, J.-R. Ohm, J. Boyce, “Algorithmic description of Joint Exploration Test Model 7 (JEM7)”, JVET-G1001, August 2017.
[0910] [3]Rec.ITU-T H.266|ISO / IEC 23090-3, “Multi-functional video codec”, 2020.
[0911] [4] B. Bross, J. Chen, S. Liu, Y.-K. Wang (eds.), “Multi-functional video codec (draft 10)”, JVET-S2001.
[0912] [5] Rec.ITU-T Rec.H.274|ISO / IEC 23002-7, “Multifunctional supplementary enhancement information messages for encoding and decoding video bitstreams”, 2020.
[0913] [6] J. Boyce, V. Drugeon, G. Sullivan, Y.-K. Wang (eds.), “Multifunctional supplementary enhancement information messages for encoding and decoding video bitstreams (draft 5)”, JVET-S 2007.
[0914] The disclosures and other schemes, examples, embodiments, modules, and functional operations described in this document can be implemented in digital electronic circuits or in computer software, firmware, or hardware, containing the structures disclosed in this document and their equivalents, or combinations thereof. The disclosed and other embodiments can be implemented as one or more computer program products encoded on a computer-readable medium, i.e., one or more computer program instruction modules for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a complex influencing machine-readable propagating signals, or combinations thereof. The term "data processing apparatus" encompasses all means, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or combinations thereof. Propagating signals are artificially generated signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device.
[0915] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language, including compiled or interpreted languages, and can be deployed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on one computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a communications network.
[0916] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be performed by special-purpose logic circuitry (e.g., field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs)), and the apparatus can be implemented as special-purpose logic circuitry (e.g., FPGAs or ASICs).
[0917] Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors in any type of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magneto-optical, magneto-optical, or optical disc) for storing data, or operatively coupled to receive data from or transfer data to a mass storage device (e.g., magneto-optical, magneto-optical, or optical disc), or both. However, a computer does not necessarily need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and optical disks. Processors and memory may be supplemented by or incorporated into special-purpose logic circuitry.
[0918] While this patent document contains numerous details, these details should not be construed as limiting any subject matter or the scope of the claims, but rather as descriptions of features specific to particular embodiments of a particular art. In this patent document, certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in various suitable sub-combinations. Furthermore, although features may be described above as operating in certain combinations and even initially claimed in the same manner, in certain circumstances one or more features from the claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.
[0919] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order or sequence shown, or to perform all the operations shown, in order to achieve the desired result. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.
[0920] Only a few implementations and examples are described, and other implementations, enhancements and variations can be made based on what is described and shown in this patent document.
Claims
1. A method for processing video data, comprising: Perform the conversion between the video and the video bitstream according to the rules. The rule states that supplementary enhancement information (SEI) messages with payload type equal to 179 are not included in scalable nested SEI messages. The rule further stipulates that if the video codec sequence CVS does not contain scalability dimension information (SDI) SEI messages, then the CVS does not contain multi-view acquisition information (MAI) SEI messages.
2. The method according to claim 1, wherein, The SEI message with payload type equal to 179 is a Multi-View Acquisition Information (MAI) SEI message.
3. The method according to claim 2, wherein, The MAI SEI messages are not scalable nested.
4. The method according to claim 1, wherein, The rule stipulates that the length of the first syntax element indicating the view identifier of the current layer in the current codec video sequence CVS is based on the value of the second syntax element in the scalability dimension information (SDI) SEI message, and The second syntax element plus one specifies the length of the first syntax element in bits.
5. The method according to claim 4, wherein, The second syntax element is encoded and decoded as an unsigned integer using N bits, where N is an integer.
6. The method according to claim 5, wherein, N equals 4.
7. The method according to claim 4, wherein, The first syntax element is represented as sdi_view_id_val[], and the second syntax element is represented as sdi_view_id_len_minus1.
8. The method according to claim 4, wherein, Based on the value of a third syntax element that indicates whether the current CVS has multiple views, the first and second syntax elements are conditionally included in the bitstream.
9. The method according to claim 8, wherein, When the value of the third syntax element is equal to 1, the first syntax element and the second syntax element are included in the SDI SEI message in the bitstream, and When the value of the third syntax element is equal to 0, the first syntax element and the second syntax element are not included in the SDI SEI message in the bitstream.
10. The method according to claim 1, wherein, The bit stream is a bit stream within a range.
11. The method according to claim 1, wherein, The rule also stipulates that SEI messages with payload types equal to 3, 133, 180, or 205 are not included in the scalable nested SEI messages.
12. The method according to any one of claims 1-11, wherein, The conversion includes encoding the video into the bitstream.
13. The method according to any one of claims 1-11, wherein, The conversion includes decoding the video from the bitstream.
14. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein, When the instruction is executed by the processor, the processor: Perform the conversion between the video and the video bitstream according to the rules. The rule states that supplementary enhancement information SEI messages with payload type equal to 179 are not included in scalable nested SEI messages; The rule further stipulates that if the video codec sequence CVS does not contain scalability dimension information (SDI) SEI messages, then the CVS does not contain multi-view acquisition information (MAI) SEI messages.
15. The apparatus according to claim 14, wherein, The SEI message with payload type equal to 179 is a Multi-View Acquisition Information (MAI) SEI message. The MAI SEI messages are not scalable nested. The rule specifies that the length of the first syntax element indicating the view identifier of the current layer in the current codec video sequence (CVS) of the video is based on the value of the second syntax element in the scalability dimension information (SDI) SEI message, and wherein the second syntax element plus one specifies the length of the first syntax element in bits. The second syntax element is encoded into an unsigned integer using N bits, where N is an integer equal to 4. Wherein, the first syntax element is represented as sdi_view_id_val [], and the second syntax element is represented as sdi_view_id_len_minus1, Specifically, based on the value of a third syntax element indicating whether the current CVS has multiple views, the first and second syntax elements are conditionally included in the bitstream. Specifically, when the value of the third syntax element is equal to 1, the first syntax element and the second syntax element are included in the SDI SEI message in the bitstream; and when the value of the third syntax element is equal to 0, the first syntax element and the second syntax element are not included in the SDI SEI message in the bitstream. Wherein, the bit stream is a bit stream within a range, and The rule also stipulates that SEI messages with payload types of 3, 133, 180, or 205 are not included in the scalable nested SEI messages.
16. A non-transitory computer-readable storage medium for storing instructions, said instructions causing a processor to: Perform the conversion between the video and the video bitstream according to the rules. in, The rule stipulates that Supplemental Enhancement Information (SEI) messages with payload type equal to 179 are not included in scalable nested SEI messages; The rule further stipulates that if the video codec sequence CVS does not contain scalability dimension information (SDI) SEI messages, then the CVS does not contain multi-view acquisition information (MAI) SEI messages.
17. The non-transitory computer-readable storage medium according to claim 16, wherein, The SEI message with payload type equal to 179 is a Multi-View Acquisition Information (MAI) SEI message. The MAI SEI messages are not scalable nested. The rule specifies that the length of the first syntax element indicating the view identifier of the current layer in the current codec video sequence (CVS) of the video is based on the value of the second syntax element in the scalability dimension information (SDI) SEI message, and wherein the second syntax element plus one specifies the length of the first syntax element in bits. The second syntax element is encoded into an unsigned integer using N bits, where N is an integer equal to 4. Wherein, the first syntax element is represented as sdi_view_id_val [], and the second syntax element is represented as sdi_view_id_len_minus1, Specifically, based on the value of a third syntax element indicating whether the current CVS has multiple views, the first and second syntax elements are conditionally included in the bitstream. Specifically, when the value of the third syntax element is equal to 1, the first syntax element and the second syntax element are included in the SDI SEI message in the bitstream; and when the value of the third syntax element is equal to 0, the first syntax element and the second syntax element are not included in the SDI SEI message in the bitstream. Wherein, the bit stream is a bit stream within a range, and The rule also stipulates that SEI messages with payload types of 3, 133, 180, or 205 are not included in the scalable nested SEI messages.
18. A non-transitory computer-readable recording medium having a computer program and a bit stream stored thereon, wherein, When the computer program is executed by a processor, it implements a method for processing video data to generate the bitstream, the method comprising: The bitstream of the video is generated according to the rules. The rule states that supplementary enhancement information SEI messages with payload type equal to 179 are not included in scalable nested SEI messages; The rule further stipulates that if the video codec sequence CVS does not contain scalability dimension information (SDI) SEI messages, then the CVS does not contain multi-view acquisition information (MAI) SEI messages.
19. The non-transitory computer-readable recording medium according to claim 18, wherein, The SEI message with payload type equal to 179 is a Multi-View Acquisition Information (MAI) SEI message. The MAI SEI messages are not scalable nested. The rule specifies that the length of the first syntax element indicating the view identifier of the current layer in the current codec video sequence (CVS) of the video is based on the value of the second syntax element in the scalability dimension information (SDI) SEI message, and wherein the second syntax element plus one specifies the length of the first syntax element in bits. The second syntax element is encoded into an unsigned integer using N bits, where N is an integer equal to 4. Wherein, the first syntax element is represented as sdi_view_id_val [], and the second syntax element is represented as sdi_view_id_len_minus1, Specifically, based on the value of a third syntax element indicating whether the current CVS has multiple views, the first and second syntax elements are conditionally included in the bitstream. Specifically, when the value of the third syntax element is equal to 1, the first syntax element and the second syntax element are included in the SDI SEI message in the bitstream; and when the value of the third syntax element is equal to 0, the first syntax element and the second syntax element are not included in the SDI SEI message in the bitstream. Wherein, the bit stream is a bit stream within a range, and The rule also stipulates that SEI messages with payload types of 3, 133, 180, or 205 are not included in the scalable nested SEI messages.
20. A method for storing a video bitstream, comprising: The method for processing video data as described in claim 1 is used to generate the bitstream of the video, and The bit stream is stored in a non-transitory computer-readable recording medium.
21. The method according to claim 20, wherein, The SEI message with payload type equal to 179 is a Multi-View Acquisition Information (MAI) SEI message. The MAI SEI messages are not scalable nested. The rule specifies that the length of the first syntax element indicating the view identifier of the current layer in the current codec video sequence (CVS) of the video is based on the value of the second syntax element in the scalability dimension information (SDI) SEI message, and wherein the second syntax element plus one specifies the length of the first syntax element in bits. The second syntax element is encoded into an unsigned integer using N bits, where N is an integer equal to 4. Wherein, the first syntax element is represented as sdi_view_id_val [], and the second syntax element is represented as sdi_view_id_len_minus1, Specifically, based on the value of a third syntax element indicating whether the current CVS has multiple views, the first and second syntax elements are conditionally included in the bitstream. Specifically, when the value of the third syntax element is equal to 1, the first syntax element and the second syntax element are included in the SDI SEI message in the bitstream; and when the value of the third syntax element is equal to 0, the first syntax element and the second syntax element are not included in the SDI SEI message in the bitstream. Wherein, the bit stream is a bit stream within a range, and The rule also stipulates that SEI messages with payload types of 3, 133, 180, or 205 are not included in the scalable nested SEI messages.
