Multi-view information acquisition supplementary and enhanced information
By clearly defining the duration and location of the multi-view acquisition information (SEI) messages, the problem of low information processing efficiency in multi-view video encoding and decoding is solved, and efficient multi-view information presentation and decoding on a 3D display is achieved.
Patent Information
- Application Number
- CN202280026954.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-04-08
- Filing Date
- 2022-04-08
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-04-08
AI Technical Summary
Existing video encoding and decoding technologies lack clear specifications for the duration, location, and content of Supplemental Enhancement Information (SEI) messages when processing multiview acquisition information, resulting in inefficient video encoding and decoding processes.
By defining the duration, location, and content of Multi-View Acquisition Information (SEI) messages, uniform processing and transformation of MAI SEI messages in codec video sequences (CVS) are ensured, including the use of internal and external camera parameters, for processing on 3D displays.
It improves the efficiency and quality of the video encoding and decoding process, ensures the effective presentation and decoding of multi-view information, and is suitable for multi-view video applications on various encoding and decoding devices.
Smart Images

Figure CN117529925B_ABST
Abstract
Description
[0001] Cross-references to related applications
[0002] This application is based on International Patent Application No. PCT / CN2022 / 085744, filed on April 8, 2022, which claims priority and interest in International Patent Application No. PCT / CN2021 / 085894, filed on April 8, 2021. All of the aforementioned patent applications are incorporated herein by reference in their entirety. Technical Field
[0003] This disclosure relates generally to video encoding and decoding, and more particularly to multiview acquisition information (MAI) supplemental enhancement information (SEI) messages used in image / video encoding and decoding. Background Technology
[0004] Digital video consumes the largest share of bandwidth on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth demand for digital video is expected to continue to grow. Summary of the Invention
[0005] The disclosed aspects / embodiments provide techniques for specifying the duration of Multi-View Acquisition Information (MAI) SEI messages, specifying the position of MAI SEI messages within a coded video sequence (CVS), and specifying the content of all MAI SEI messages within the same CVS. By specifying the duration, position, and content of MAI SEI messages, the video encoding / decoding process is improved.
[0006] The first aspect relates to a method implemented by an encoding / decoding device. The method includes: determining that an MAI SEI message exists in an AU of a CVS whenever any Multi-View Acquisition Information (MAI) Supplemental Enhancement Information (SEI) message exists in any first access unit (AU) of an encoded / decoded video sequence (CVS); and performing a conversion between video and video bitstreams based on the MAI SEI message.
[0007] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that the MAI SEI message is one of any MAI SEI messages, and wherein the first AU is one of any AU.
[0008] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that the MAI SEI message specifies internal and external camera parameters.
[0009] Alternatively, in any of the foregoing aspects, another implementation of the aspect specifies that internal and external camera parameters are used to process the decoded view before it is rendered on a three-dimensional (3D) display.
[0010] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that all MAI SEI messages applicable to CVS have the same content.
[0011] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that the MAI SEI message is one of all MAI SEI messages.
[0012] Optionally, in any of the foregoing aspects, another implementation of that aspect specifies that MAI SEI messages continue from the current AU to the subsequent AU containing subsequent MAI SEI messages in the decoding order.
[0013] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that subsequent MAI SEI messages contain content different from that of the MAI SEI messages.
[0014] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that the MAI SEI messages continue from the current AU to the end of the bit stream in the decoding order.
[0015] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that the MAI SEI message follows the scalability dimension information (SDI) SEI message in the decoding order.
[0016] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that the MAI SEI message includes an internal parameter flag.
[0017] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that the MAI SEI message includes an external parameter flag.
[0018] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that the MAI SEI message applies to the entire CVS, and not just the coded layer video sequence (CLVS).
[0019] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that the MAI SEI message is included in a scalable nested SEI message.
[0020] Alternatively, in any of the foregoing aspects, another implementation of the aspect specifies that an internal parameter flag equal to 1 indicates the presence of an internal camera parameter, and wherein an internal parameter flag equal to 0 indicates the absence of an internal camera parameter.
[0021] Alternatively, in any of the foregoing aspects, another implementation of the aspect specifies that an external parameter flag equal to 1 indicates the presence of an external camera parameter, and wherein an external parameter flag equal to 0 indicates the absence of an external camera parameter.
[0022] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that the MAI SEI message is encoded into a bitstream by a video codec device.
[0023] Alternatively, in any of the foregoing aspects, another implementation of that aspect specifies that a video codec device decodes the bitstream to obtain the MAI SEI message.
[0024] The second aspect relates to an apparatus for encoding and decoding video data, including a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform any of the methods disclosed herein.
[0025] The third aspect relates to a non-transitory computer-readable medium including a computer program product for use by a codec apparatus, the computer program product including computer-executable instructions stored on the non-transitory computer-readable medium, the computer-executable instructions causing the codec apparatus to perform any of the methods disclosed herein when executed by one or more processors.
[0026] The fourth aspect relates to a non-transitory computer-readable recording medium storing a bitstream of video generated by a method performed by a video processing apparatus, wherein the method includes: determining that an AU of a CVS contains a MAISEI message whenever any Multi-View Acquisition Information (MAI) Supplemental Enhancement Information (SEI) message is present in any first access unit (AU) of a codec video sequence (CVS); and generating a bitstream based on the SDI SEI message.
[0027] The fifth aspect relates to a method for storing a bitstream of video, comprising: determining that an MAI SEI message exists in an AU of a CVS whenever any Multi-View Acquisition Information (MAI) Supplemental Enhancement Information (SEI) message exists in any first access unit (AU) of a codec video sequence (CVS); generating a bitstream including the SDI SEI message; and storing the bitstream in a non-transitory computer-readable medium.
[0028] For clarity, any of the foregoing embodiments may be combined with any one or more other foregoing embodiments to form new embodiments within the scope of this disclosure.
[0029] These and other features will become clearer from the following detailed description taken in conjunction with the accompanying drawings and claims. Attached Figure Description
[0030] To gain a more complete understanding of this disclosure, reference is now made to the following brief description in conjunction with the accompanying drawings and detailed specifications, wherein the same reference numerals represent the same parts.
[0031] Figure 1 This is a schematic diagram illustrating an example of layer-based prediction.
[0032] Figure 2 The illustration shows an example of layer-based prediction using the output layer set (OLS).
[0033] Figure 3 An example of a video bitstream is illustrated.
[0034] Figure 4 This is a block diagram showing an example video processing system.
[0035] Figure 5 This is a block diagram of a video processing device.
[0036] Figure 6 This is a block diagram illustrating an example video encoding / decoding system.
[0037] Figure 7 This is a block diagram illustrating an example of a video encoder.
[0038] Figure 8 This is a block diagram illustrating an example of a video decoder.
[0039] Figure 9 This is a method for encoding and decoding video data according to embodiments of the present disclosure. Detailed Implementation
[0040] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods can be implemented using any number of techniques, whether currently known or existing. This disclosure should not be limited in any way to the exemplary implementations, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but modifications can be made within the full scope of the appended claims and their equivalents.
[0041] Video codec standards have primarily evolved through the development of standards from the renowned International Telecommunication Union-Telecommunication (ITU-T) and the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). ITU-T developed H.261 and H.263, while ISO / IEC developed Moving Picture Experts Group (MPEG)-1 and MPEG-4 video. These two organizations jointly developed the H.262 / MPEG-2 video standard, the H.264 / MPEG-4 Advanced Video Coding (AVC) standard, and the H.265 / High Efficiency Video Coding (HEVC) standard. See ITU-T and ISO / IEC, “High Efficiency Video Coding,” Rec. ITU-T H.265|ISO / IEC 23008-2 (current version). Since H.262, video codec standards have been based on a hybrid video codec architecture, which utilizes temporal prediction plus transform coding. To explore future video coding and decoding technologies beyond HEVC, the Video Coding Experts Group (VCEG) and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted numerous new methods and incorporated them into reference software called the Joint Exploration Model (JEM). See J. Chen, E. Alshina, G.S. Sullivan, J.-R. Ohm, J. Boyce, “Algorithmic Description of Joint Exploration Test Model 7 (JEM7)”, JVET-G1001, August 2017. When the Versatile Video Coding (VVC) project was officially launched, JVET was later renamed the Joint Video Experts Team (JVET). VVC is a new codec standard aimed at reducing the bit rate by 50% compared to HEVC. This standard was finalized by the JVET at its 19th meeting, which concluded on July 1, 2020. See Rec. ITU-TH.266|ISO / IEC 23090-3, "Multi-functional Video Coding", 2020.
[0042] The VVC standard (ITU-T H.266|ISO / IEC 23090-3) and the related Versatile Supplemental Enhancement Information (VSEI) standard (ITU-T H.274|ISO / IEC 23002-7) have been designed for the widest range of applications, including traditional uses (such as television broadcasting, video conferencing, or playback from storage media) as well as newer and more advanced uses (such as adaptive bitrate streaming, video region extraction, compositing and merging of content from multiple codec video bitstreams, multi-view video, scalable layered codecs, and viewport-adaptive 360° immersive media). See B. Bross, J. Chen, S. Liu, Y.-K. Wang (ed.), “Multifunctional Video Coding (Draft 10)”, JVET-S2001, Rec.ITU-T Rec.H.274|ISO / IEC 23002-7, “Multifunctional Supplemental Enhancement Information Messages for Encoding and Decoding Video Bitstreams”, 2020, and J. Boyce, V. Drugeon, G. Sullivan, Y.-K. Wang (ed.), “Multifunctional Supplemental Enhancement Information Messages for Encoding and Decoding Video Bitstreams (Draft 5)”, JVET-S2007.
[0043] The Essential Video Coding (EVC) standard (ISO / IEC 23094-1) is another video coding standard recently developed by MPEG.
[0044] Figure 1 This is a schematic diagram illustrating an example of layer-based prediction 100. Layer-based prediction 100 is compatible with unidirectional inter-frame prediction and / or bidirectional inter-frame prediction, but it can also be performed between images in different layers.
[0045] Layer-based prediction 100 is applied between images 111, 112, 113, and 114 and images 115, 116, 117, and 118 in different layers. In the example shown, images 111, 112, 113, and 114 are part of layer N+1 132, and images 115, 116, 117, and 118 are part of layer N 131. Layers such as layer N 131 and / or layer N+1 132 are a set of images that are associated with similar characteristic values, such as similar size, quality, resolution, signal-to-noise ratio, capacity, etc. In the example shown, layer N+1 132 is associated with a larger image size than layer N 131. Therefore, in this example, images 111, 112, 113, and 114 in layer N+1 132 have a larger image size (e.g., larger height and width, and therefore more samples) than images 115, 116, 117, and 118 in layer N 131. However, such images can be separated between layers N+1 132 and N 131 using other features. Although only two layers, N+1 132 and N 131, are shown, a set of images can be divided into any number of layers based on associated characteristics. Layers N+1 132 and N 131 can also be represented by layer identifiers (IDs). A layer ID is a data item associated with an image and indicates that the image is part of the indicated layer. Therefore, each image 111-118 can be associated with a corresponding layer ID to indicate which layer N+1 132 or layer N 131 contains the corresponding image.
[0046] Images 111-118 in different layers 131-132 are configured to be displayed alternately. This allows images 111-118 in different layers 131-132 to share the same temporal identifier (ID) and to be included in the same access unit (AU) 106. As used herein, an AU is a set of one or more encoded / decoded images associated with the same display time output from the decoded picture buffer (DPB). For example, if a smaller image is needed, the decoder can decode and display image 115 at the current display time, or if a larger image is needed, the decoder can decode and display image 111 at the current display time. Thus, images 111-114 at the higher layer N+1 132 contain substantially the same image data as images 115-118 at the corresponding lower layer N 131 (although the image sizes are different). Specifically, image 111 contains substantially the same image data as image 115, image 112 contains substantially the same image data as image 116, and so on.
[0047] Images 111-118 can be encoded / decoded by referencing other images 111-118 in the same layer N 131 or N+1 132. Encoding / decoding an image by referencing another image in the same layer results in inter-frame prediction 123, which is compatible with unidirectional and / or bidirectional inter-frame prediction. Inter-frame prediction 123 is depicted by a solid arrow. For example, image 113 can be encoded / decoded using inter-frame prediction 123 with reference to one or both of images 111, 112, and / or 114 in layer N+1 132, where one image is referenced for unidirectional inter-frame prediction and / or both images are referenced for bidirectional inter-frame prediction. Similarly, image 117 can be encoded / decoded using inter-frame prediction 123 with reference to one or both of images 115, 116, and / or 118 in layer N 131, where one image is referenced for unidirectional inter-frame prediction and / or both images are referenced for bidirectional inter-frame prediction. When performing inter-frame prediction 123, an image may be referred to as a reference image when it is used as a reference for another image in the same layer. For example, image 112 may be a reference image used to encode and decode image 113 according to inter-frame prediction 123. In a multi-layer context, inter-frame prediction 123 may also be referred to as intra-layer prediction. Thus, inter-frame prediction 123 is a mechanism for encoding and decoding samples of the current image by referencing indicator samples in a reference image that is different from the current image, where the reference image and the current image are in the same layer.
[0048] Images 111-118 can also be encoded / decoded by referencing other images 111-118 in different layers. This process is called inter-layer prediction 121 and is indicated by a dashed arrow. Inter-layer prediction 121 is a mechanism for encoding / decoding samples of the current image by referencing indicator samples in a reference image, where the current image and the reference image are in different layers and therefore have different layer IDs. For example, an image in a lower layer N 131 can be used as a reference image to encode / decode the corresponding image in a higher layer N+1 132. As a concrete example, image 111 can be encoded / decoded based on inter-layer prediction 121 with reference to image 115. In this case, image 115 is used as the inter-layer reference image. The inter-layer reference image is the reference image used for inter-layer prediction 121. In most cases, inter-layer prediction 121 is constrained such that the current image (e.g., image 111) can only use inter-layer reference images (e.g., image 115) included in the same AU 106 and located in a lower layer. When multiple layers (e.g., more than two) are available, inter-layer prediction 121 can encode / decode the current image based on multiple inter-layer reference images at a lower level than the current image.
[0049] The video encoder can employ layer-based prediction 100 to encode images 111-118 via various combinations and / or permutations of inter-frame prediction 123 and inter-layer prediction 121. For example, image 115 can be encoded / decoded based on intra-frame prediction. Then, using image 115 as a reference image, images 116-118 can be encoded / decoded based on inter-frame prediction 123. Furthermore, image 111 can be encoded / decoded based on inter-layer prediction 121 using image 115 as an inter-layer reference image. Then, using image 111 as a reference image, images 112-114 can be encoded / decoded based on inter-frame prediction 123. Thus, the reference image can be used as both a single-layer reference image and an inter-layer reference image for different encoding / decoding mechanisms. By encoding / decoding higher-layer N+1 132 images based on lower-layer N 131 images, higher-layer N+1 132 can avoid intra-frame prediction, which has significantly lower encoding / decoding efficiency than inter-frame prediction 123 and inter-layer prediction 121. In this way, the low encoding / decoding efficiency of intra-frame prediction can be limited to the smallest / lowest quality picture, and therefore limited to encoding / decoding the minimum amount of video data. Pictures used as reference pictures and / or inter-layer reference pictures can be indicated in entries of the reference picture list(s) included in the reference picture list structure.
[0050] Figure 1 Each AU 106 in the array may contain several pictures. For example, one AU 106 may contain pictures 111 and 115. Another AU 106 may contain pictures 112 and 116. In practice, each AU 106 is a collection of one or more codec pictures associated with the same display time (e.g., the same time-domain ID) for output from the decoded picture buffer (DPB) (e.g., for display to the user). Each access unit delimiter (AUD) 108 is an indicator or data structure used to indicate the start of an AU (e.g., AU 106) or the boundary between AUs.
[0051] Previous H.26x video codec families have provided support for scalability in multiple layers separated from single-layer codecs. Scalable Video Coding (SVC) is a scalable extension of AVC / H.264, providing support for spatial, temporal, and quality scalability. For SVC, a signaling notification flag is included in each macroblock (MB) in the enhancement layer (EL) picture to indicate whether the ELMB is predicted using juxtaposed blocks from lower layers. Predictions from juxtaposed blocks can include textures, motion vectors, and / or codec modes. SVC implementations cannot directly reuse unmodified H.264 / AVC implementations in their design. The SVC EL macroblock syntax and decoding process differ from the H.264 / AVC syntax and decoding process.
[0052] Scalable HEVC (SHVC) is an extension of the HEVC / H.265 standard, providing support for spatial and quality scalability. Multiview HEVC (MV-HEVC) is an extension of HEVC / H.265, providing support for multiview scalability, and 3D HEVC (3D-HEVC) is an extension of HEVC / H.264, providing support for three-dimensional (3D) video encoding and decoding, which is more advanced and efficient than MV-HEVC. Note that temporal scalability is included as part of a single-layer HEVC codec. The design of multi-layer extensions of HEVC employs the idea that decoded pictures used for inter-layer prediction come only from the same AU and are treated as long-term reference pictures (LTRPs), and are assigned reference indices in a list of reference pictures (multiple) along with other temporal reference pictures in the current layer. Inter-layer prediction (ILP) is achieved at the prediction unit (PU) level by setting the value of the reference index to reference (multiple) inter-layer reference images in a list of (multiple) reference images.
[0053] It is worth noting that both reference picture resampling and spatial scalability features require resampling of a reference picture or a portion thereof. Reference picture resampling (RPR) can be implemented at the picture level or the codec block level. However, when RPR is referred to as a codec feature, it is a feature used for single-layer codecs. Even so, from a codec design perspective, it is possible, or even preferred, to use the same resampling filter for both single-layer codec RPR features and multi-layer codec spatial scalability features.
[0054] Figure 2 An example of layer-based prediction 200 using Output Layer Set (OLS) is illustrated. Layer-based prediction 100 is compatible with unidirectional inter-frame prediction and / or bidirectional inter-frame prediction, but can also be performed between images in different layers. Figure 2 Layer-based prediction 200 is similar Figure 1 Therefore, for the sake of brevity, the full description of layer-based prediction 200 will not be repeated.
[0055] Figure 2 Some layers in the codec video sequence (CVS) 290 are contained in an OLS. An OLS is a set of layers in which one or more layers are specified as output layers. Output layers are the layers of the output OLS. Figure 2 Three different OLSs are depicted: OLS1, OLS2, and OLS3. As shown in the figure, OLS1 includes layers N 231 and N+1 232. Layer N 231 includes images 215, 216, 217, and 218, and layer N+1 232 includes images 211, 212, 213, and 214. OLS2 includes layers N 231, N+1 232, N+2 233, and N+3 234. Layer N+2 233 includes images 241, 242, 243, and 244, and layer N+3 234 includes images 251, 252, 253, and 254. OLS3 includes layers N 231, N+1 232, and N+2 233. Although three OLSs are shown, different numbers of OLSs can be used in practical applications. In the illustrated embodiment, none of the OLSs includes layer N+4 235, which contains images 261, 262, 263, and 264.
[0056] Each different OLS can contain any number of layers. Different OLSs are generated to accommodate the different encoding and decoding capabilities of various devices. For example, OLS 1, containing only two layers, can be generated to accommodate mobile phones with relatively limited encoding and decoding capabilities. On the other hand, OLS 2, containing four layers, can be generated to accommodate large-screen televisions, which can decode higher layers than mobile phones. OLS 3, containing three layers, can be generated to accommodate personal computers, laptops, or tablets, which can decode higher layers than mobile phones, but cannot decode the highest layers like those on large-screen televisions.
[0057] Figure 2 The layers in a network can be completely independent of each other. That is, each layer can be encoded and decoded without using inter-layer prediction (ILP). In this case, these layers are called syndicated layers. Figure 2 One or more layers in a network can also use ILP for encoding and decoding. These layers can be signaled as syndicated layers or as layers using ILP for encoding and decoding via flags in the Video Parameter Set (VPS). When some layers use ILP, the layer dependencies between them are also signaled in the VPS.
[0058] In an embodiment, when the layer is a communal layer, only one layer is selected for decoding and output. In an embodiment, when some layers use ILP, all layers (e.g., the entire bitstream) are specified to be decoded, and some of these layers are specified as output layers. For example, the output layers (multiple) can be 1) only the highest layer, 2) all layers, or 3) the highest layer plus a set of indicated lower layers. For example, when the set of the highest layer plus the indicated lower layers is specified for output by a flag in the VPS, layer N+3 234 (the highest layer) and layers N 231 and N+1 232 (lower layers) from OLS2 are output.
[0059] Figure 2 Some layers in the bitstream can be called primary layers, while others can be called secondary layers. For example, layers N231 and N+1232 can be called primary layers (containing the primary image), and layers N+2233 and N+3234 can be called secondary layers (containing secondary images). Secondary layers can be called alpha secondary layers or depth secondary layers. When secondary information is present in the bitstream, primary layers can be associated with secondary layers.
[0060] Figure 3 An embodiment of video bitstream 300 is illustrated. As used herein, video bitstream 300 may also be referred to as a codec video bitstream, a bitstream, or a variant thereof. Figure 3As shown, bitstream 300 includes one or more of the following: decoding capability information (DCI) 302, video parameter set (VPS) 304, sequence parameter set (SPS) 306, picture parameter set (PPS) 308, picture header (PH) 312, and picture 314. Each of DCI 302, VPS 304, SPS 306, and PPS 308 can be collectively referred to as a parameter set. In this embodiment, Figure 3 Other parameter sets, not shown in the figure, may also be included in bitstream 300, such as an adaptive parameter set (APS), which is a syntax structure containing syntax elements applied to zero or more stripes determined by zero or more syntax elements found in the strip header.
[0061] DCI 302, also known as a decoding parameter set (DPS) or decoder parameter set, is a syntactic structure containing syntax elements applied throughout the bitstream. DCI 302 includes parameters that remain constant throughout the lifetime of the video bitstream (e.g., bitstream 300), which can be translated to the lifetime of a session. DCI 302 can include grade, level, and subgrade information to determine the maximum complexity interoperability point that is guaranteed never to be exceeded, even if video sequence splicing occurs during the session. It may also optionally include constraint flags that indicate that the video bitstream will be constrained by the use of certain features indicated by the values of those flags. This allows the bitstream to be marked as not using certain tools, which in particular allows for resource allocation in decoder implementations. Like all parameter sets, DCI 302 exists when it is first referenced and is referenced by the first picture in the video sequence, meaning it must be transmitted between the first network abstraction layer (NAL) units in the bitstream. While there can be multiple DCI 302s in a bitstream, the values of the syntax elements within them cannot be inconsistent when referenced.
[0062] VPS 304 includes decoding dependencies or information for building the reference picture set used for the enhancement layer. VPS 304 provides a holistic view or overview of the scalable sequence, including what type of action points are provided, the tiers, layers, and levels of action points, as well as some other high-level attributes of the bitstream that can be used as the basis for session negotiation and content selection.
[0063] In an embodiment, when instructing certain layers to use ILP, VPS 304 indicates that the total number of OLS specified by the VPS is equal to the number of layers, indicates that the i-th OLS includes layers with layer indices from 0 to i (inclusive of end values), and indicates that for each OLS, only the highest layer in the OLS is output.
[0064] SPS 306 contains data common to all pictures in the sequence of pictures (SOP). SPS 306 is a syntactic structure containing syntax elements applied to zero or more coded layer video sequences (CLVS), as determined by the content of the syntax elements found in PPS 308, which is referenced by the syntax elements found in each picture header 312. In contrast, PPS 308 contains data common to the entire picture 314. PPS 308 is a syntactic structure containing syntax elements applied to zero or more coded layer pictures, as determined by the syntax elements found in each picture header (e.g., PH 312).
[0065] DCI 302, VPS 304, SPS 306, and PPS 308 are contained within different types of Network Abstraction Layer (NAL) units. An NAL unit is a syntactic structure containing an indication of the type of data to be followed (e.g., encoded or decoded video data). NAL units are categorized into Video Coding Layer (VCL) and Non-VCL NAL units. VCL NAL units contain data representing the values of samples in a video frame, while non-VCL NAL units contain any associated additional information, such as parameter sets (important data that can be applied to multiple VCL NAL units) and supplementary enhancement information (timing information and other supplementary data that can enhance the usability of decoding video signaling but are not necessary for decoding sample values in a video frame).
[0066] In this embodiment, DCI 302 is contained within a non-VCL NAL unit designated as a DCI NAL unit or a DPS NAL unit. That is, a DCI NAL unit has a DCI NAL unit type (NUT), while a DPS NAL unit has a DPS NUT. In this embodiment, VPS 304 is contained within a non-VCL NAL unit designated as a VPS NAL unit. Therefore, a VPS NAL unit has a VPS NUT. In this embodiment, SPS 306 is a non-VCL NAL unit designated as an SPS NAL unit. Therefore, an SPS NAL unit has an SPS NUT. In this embodiment, PPS 308 is contained within a non-VCL NAL unit designated as a PPS NAL unit. Therefore, a PPS NAL unit has a PPS NUT.
[0067] PH 312 is a syntactic structure containing syntactic elements applied to all stripes (e.g., stripe 318) of the encoded / decoded picture (e.g., picture 314). In an embodiment, PH 312 is a non-VCL NAL unit type designated as a PH NAL unit. Therefore, a PH NAL unit has a PH NUT (e.g., PH_NUT).
[0068] In an embodiment, the PH NAL unit associated with PH 312 has a time-domain ID and a layer ID. The time-domain ID indicates the temporal position of the PH NAL unit relative to other PH NAL units in the bitstream (e.g., bitstream 300). The layer ID indicates the layer containing the PH NAL unit (e.g., layer 131 or layer 132). In an embodiment, the time-domain ID is similar to but different from the picture order count (POC). The POC uniquely identifies each picture sequentially. In a single-layer bitstream, the time-domain ID and POC will be the same. In a multi-layer bitstream (e.g., see...), the POC will be different. Figure 1 Images within the same AU will have different POCs, but will have the same temporal ID.
[0069] In one embodiment, the PH NAL unit precedes the VCL NAL unit containing the first strip 318 of the associated image 314. This establishes an association between PH 312 and the strip 318 of the image 314 associated with PH 312, without requiring an image header ID signaled in PH 312 and referenced from the strip header 320. Therefore, it can be inferred that all VCL NAL units between two PH 312 belong to the same image 314, and that image 314 is associated with the first PH 312 between the two PH 312. In one embodiment, the first VCL NAL unit following PH 312 contains the first strip 318 of the image 314 associated with PH 312.
[0070] In one embodiment, the PH NAL unit follows the image-level parameter set (e.g., PPS 308) or a higher-level parameter set, such as DCI 302 (aka DPS), VPS 304, SPS 306, PPS 308, etc., and has a temporal ID and layer ID that are smaller than the temporal ID and layer ID of the PH NAL unit, respectively. Therefore, these parameter sets are not repeated within an image or access unit. Due to this ordering, PH 312 can be resolved immediately. That is, the parameter set containing parameters related to the entire image is located before the PH NAL unit in the bitstream. Any content containing parameters specific to an image is located after the PH NAL unit.
[0071] In one alternative, the PH NAL unit follows the picture-level parameter set and the Prefix Supplement Enhancement Information (SEI) message, or a higher-level parameter set (such as DCI 302 (also known as DPS), VPS 304, SPS 306, PPS 308, APS, SEI message, etc.).
[0072] Image 314 is an array of luminance samples in monochrome format or an array of luminance samples in 4:2:0, 4:2:2, and 4:4:4 color formats, along with two corresponding arrays of chrominance samples.
[0073] Image 314 can be a frame or a field. However, in a CVS 316, either all images 314 are frames, or all images 314 are fields. CVS 316 is the coded layer video sequence (CLVS) of each codec layer video sequence in the video bitstream 300. It is worth noting that when the video bitstream 300 consists of a single layer, CVS 316 and CLVS are the same. Only when the video bitstream 300 consists of multiple layers do CVS 316 and CLVS differ (e.g., as...). Figure 1 and 2 (As shown).
[0074] Each image 314 contains one or more stripes 318. A stripe 318 is an integer number of complete slices or an integer number of consecutive complete coding tree unit (CTU) rows within a slice of an image (e.g., image 314). Each stripe 318 is exclusively contained within a single NAL unit (e.g., a VCL·NAL unit). A slice (not shown) is a rectangular region of CTUs within a specific slice column and a specific slice row in an image (e.g., image 314). A CTU (not shown) is a coding tree block (CTB) of luminance samples, two corresponding CTBs of chrominance samples of an image with three sample arrays, or a CTB of samples of a monochrome image or an image encoded using three separate color planes and a syntax structure for encoding and decoding samples. A CTB (not shown) is an N×N sample block for some value of N such that dividing the components into CTBs is a partition. A block (not shown) is an M×N (M columns by N rows) array of samples (e.g., pixels), or an M×N array of transform coefficients.
[0075] In this embodiment, each stripe 318 includes a stripe header 320. The stripe header 320 is part of the encoded stripe 318 and contains data elements related to all stripes or CTU lines within the chip represented in the stripe 318. That is, the stripe header 320 contains information about the stripe 318, such as the stripe type, which reference image will be used, etc.
[0076] Image 314 and its strip 318 contain data related to the image or video being encoded or decoded. Therefore, image 314 and its strip 318 can simply be referred to as the payload or data carried in bitstream 300.
[0077] Bitstream 300 also contains one or more SEI messages, such as SDI SEI message 322, multiview acquisition information (MAI) SEI message 326, depth representation information (DRI) SEI message 328, and alpha channel information (ACI) SEI message 330. As described below, SDI SEI message 322, MAI SEI message 326, DRI SEI message 328, and ACI SEI message 330 can each contain various syntax elements 324. SEI messages contain supplementary enhancement information. SEI messages can contain supplementary enhancement information. SEI messages can contain various types of data that indicate the timing of video frames, or describe various properties of the encoded video, or how to use or enhance the encoded video. SEI messages are also defined to contain arbitrary user-defined data. SEI messages do not affect the core decoding process, but can indicate suggestions on how to post-process or display the video. Other high-level attributes of video content are conveyed in videousability information (VUI), such as indications of the color space used to interpret the video content. With the development of new color spaces, such as high dynamic range and wide color gamut video, additional VUI identifiers have been added to indicate them.
[0078] Those skilled in the art will recognize that, in practical applications, bitstream 300 may contain other parameters and information.
[0079] The syntax and semantics of SDI SEI message 322 are as follows.
[0080] SDI SEI message syntax.
[0081]
[0082]
[0083] SDI SEI message semantics.
[0084] The Scalability Dimension (SEI) message provides scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) the view ID of each layer when bitstreamInScope can be a multi-view bitstream; and 2) the auxiliary ID of each layer when bitstreamInScope may have one or more layers carrying auxiliary information (such as depth or alpha).
[0085] bitstreamInScope is a sequence of AUs that, in the order of decoding, includes the AU containing the current scalability dimension SEI message, followed by zero or more AUs, up to but not including any subsequent AUs containing the scalability dimension SEI message.
[0086] The increment of sdi_max_layers_minus1 indicates the maximum number of layers in bitstreamInScope.
[0087] A value of 1 for `sdi_multiview_info_flag` indicates that `bitstreamInScope` may be a multiview bitstream, and that the `sdi_view_id_val[]` syntax element exists in the scalability dimension SEI message. A value of 0 for `sdi_multiview_flag` indicates that `bitstreamInScope` is not a multiview bitstream, and that the `sdi_view_id_val[]` syntax element does not exist in the scalability dimension SEI message.
[0088] A value of 1 for `sdi_auxiliary_info_flag` indicates that auxiliary information may exist, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element exists in the scalability dimension SEI message. A value of 0 for `sdi_auxiliary_info_flag` indicates that no auxiliary information exists, carried by one or more layers in `bitstreamInScope`, and that the `sdi_aux_id[]` syntax element does not exist in the scalability dimension SEI message.
[0089] The sdi_view_id_len parameter specifies the length of the sdi_view_id_val[i] syntax element in bits.
[0090] `sdi_view_id_val[i]` specifies the view ID of the i-th level in `bitstreamInScope`. The length of the `sdi_view_id_val[i]` syntax element is `sdi_view_id_len` bits. If it does not exist, the value of `sdi_view_id_val[i]` is inferred to be equal to 0.
[0091] A value of 0 for sdi_aux_id[i] indicates that the i-th layer in bitstreamInScope does not contain auxiliary images. A value greater than 0 for sdi_aux_id[i] indicates the type of auxiliary image in the i-th layer of bitstreamInScope as specified in Table 1.
[0092] Table 1 – Mapping from sdi_aux_id[i] to auxiliary image types
[0093]
[0094]
[0095] Note 1 - Explanation of the auxiliary picture associated with sdi_aux_id in the range of 128 to 159 (inclusive) by means other than the sdi_aux_id value.
[0096] For this version of the bitstream conforming to this specification, sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive). Although the value of sdi_aux_id[i] should be in the range of 0 to 2 (inclusive) or 128 to 159 (inclusive), in this version of the specification, the decoder should allow the value of sdi_aux_id[i] to be in the range of 0 to 255 (inclusive).
[0097] The syntax and semantics of MAI SEI message 326 are as follows.
[0098] MAI SEI message syntax.
[0099]
[0100]
[0101] MAI SEI message semantics.
[0102] The Multi-View Acquisition Information (SEI) message specifies various parameters of the acquisition environment. Specifically, it specifies the parameters of the internal and external cameras. These parameters can be used to process and decode the view before rendering it on a 3D display.
[0103] The following semantics apply to each nuh_layer_id targetLayerId in the nuh_layer_id value applied to the SEI message for multi-view information retrieval.
[0104] When present, the SEI message for multiview acquisition applied to the current layer should be included in the access unit containing the intra random access picture (IRAP), which is the first picture of the current layer's CLVS. The signaling notification information in the SEI message is applied to the CLVS.
[0105] When a multi-view information acquisition SEI message is included in a scalable nested SEI message, the syntax elements sn_ols_flag and sn_all_layers_flag in the scalable nested SEI message should be equal to 0.
[0106] The variable numViewsMinus1 is exported as follows:
[0107] --If the multi-view information SEI message is not included in the scalable nested SEI message, then numViewsMinus1 is set to equal to 0.
[0108] --Otherwise (if the multi-view information SEI message is included in the scalable nested SEI message), numViewsMinus1 is set to equal sn_num_layers_minus1.
[0109] The multi-view retrieval information contained in the multi-view retrieval information (SEI) message may include some views that may not exist.
[0110] In the following semantics, index i refers to the syntax element and variable applied to the layer whose nuh_layer_id is equal to NestingLayerId[i].
[0111] The external camera parameters are defined according to the right-hand coordinate system, where the top-left corner of the image is the origin, i.e., the coordinates (0, 0), and the other corners of the image have non-negative coordinates. Based on these specifications, for the i-th camera, the 3D world point wP = [xyz] is mapped to the 2D camera point cP[i] = [uv 1], according to:
[0112] s*cP[i]=A[i]*R -1 [i]*(wP-T[i])(X)
[0113] Where A[i] represents the internal camera parameter matrix, R -1 [i] denotes the inverse of the rotation matrix R[i], T[i] denotes the translation vector, and s (scalar value) is an arbitrary scale factor chosen so that the third coordinate of cP[i] is equal to 1. The elements of A[i], R[i], and T[i] are determined according to the syntax elements of the signaling notification in the SEI message, and are specified as follows.
[0114] An intrinsic_param_flag value of 1 indicates the existence of internal camera parameters. An intrinsic_param_flag value of 0 indicates the absence of internal camera parameters.
[0115] An extrinsic_param_flag value of 1 indicates the presence of external camera parameters. An extrinsic_param_flag value of 0 indicates the absence of external camera parameters.
[0116] An intrinsic_params_equal_flag value of 1 indicates that all cameras have the same intrinsic camera parameters, and there is only one set of intrinsic camera parameters. An intrinsic_params_equal_flag value of 0 indicates that each camera has different intrinsic camera parameters, and each camera has its own set of intrinsic camera parameters.
[0117] prec_focal_length is specified as 2 -prec_focal_length The exponent of the maximum permissible truncation error for focal_length_x[i] and focal_length_y[i]. The value of prec_focal_length should be in the range of 0 to 31 (inclusive).
[0118] prec_principal_point specifies that 2 -prec_principal_point The exponent of the maximum permissible truncation error for principal_point_x[i] and principal_point_y[i]. The value of prec_principal_point should be in the range of 0 to 31 (inclusive).
[0119] prec_skew_factor is specified by 2 -prec_skew_factor The exponent of the maximum permissible truncation error given the skew factor. The value of prec_skew_factor should be in the range of 0 to 31 (inclusive).
[0120] A value of 0 for sign_focal_length_x[i] indicates that the focal length of the i-th camera in the horizontal direction is positive. A value of 1 for sign_focal_length_x[i] indicates that the focal length is negative.
[0121] `exponent_focal_length_x[i]` specifies the exponential portion of the focal length of the i-th camera in the horizontal direction. The value of `exponent_focal_length_x[i]` should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should interpret value 63 as an indication of an unspecified focal length.
[0122] `mantissa_focal_length_x[i]` specifies the mantissa portion of the focal length of the i-th camera in the horizontal direction. The length of the `mantissa_focal_length_x[i]` syntax element is variable and is determined as follows:
[0123] --If exponent_focal_length_x[i] equals 0, then the length is Max(0,prec_focal_length-30).
[0124] --Otherwise (exponent_focal_length_x[i] is in the range of 0 to 63 (inclusive), the length is Max(0,exponent_focal_length_x[i]+prec_focal_length-31).
[0125] A value of 0 for sign_focal_length_y[i] indicates that the focal length in the vertical direction of the i-th camera is positive. A value of 1 for sign_focal_length_y[i] indicates that the focal length is negative.
[0126] `exponent_focal_length_y[i]` specifies the exponential portion of the focal length in the vertical direction for the i-th camera. The value of `exponent_focal_length_y[i]` should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should interpret value 63 as an indication of an unspecified focal length.
[0127] `mantissa_focal_length_y[i]` specifies the last part of the focal length of the i-th camera in the vertical direction.
[0128] The length of the mantissa_focal_length_y[i] syntax element is variable and is determined as follows:
[0129] --If exponent_focal_length_y[i] equals 0, then the length is Max(0,prec_focal_length-30).
[0130] --Otherwise (exponent_focal_length_y[i] is in the range of 0 to 63 (excluding the end value), the length is Max(0,exponent_focal_length_y[i]+prec_focal_length-31).
[0131] A sign_principal_point_x[i] equal to 0 indicates that the sign of the principal point of the i-th camera is positive in the horizontal direction. A sign_principal_point_x[i] equal to 1 indicates that the sign is negative.
[0132] `exponent_principal_point_x[i]` specifies the exponent portion of the principal point of the i-th camera in the horizontal direction. The value of `exponent_principal_point_x[i]` should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should treat value 63 as indicating an unspecified principal point.
[0133] `mantissa_principal_point_x[i]` specifies the mantissa portion of the principal point of the i-th camera in the horizontal direction. The length of the `mantissa_principal_point_x[i]` syntax element, in bits, is variable and is determined as follows:
[0134] --If exponent_principal_point_x[i] equals 0, then the length is Max(0, prec_principal_point-30).
[0135] --Otherwise (exponent_principal_point_x[i] is in the range of 0 to 63 (excluding the end value), the length is Max(0,exponent_principal_point_x[i]+prec_principal_point-31).
[0136] A value of 0 for sign_principal_point_y[i] indicates that the sign of the principal point of the i-th camera is positive in the vertical direction. A value of 1 for sign_principal_point_y[i] indicates that the sign is negative.
[0137] `exponent_principal_point_y[i]` specifies the exponential portion of the principal point of the i-th camera in the vertical direction. The value of `exponent_principal_point_y[i]` should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should treat value 63 as an indication of an unspecified principal point.
[0138] `mantissa_principal_point_y[i]` specifies the mantissa portion of the principal point of the i-th camera in the vertical direction. The length of the `mantissa_principal_point_y[i]` syntax element, in bits, is variable and is determined as follows:
[0139] --If exponent_principal_point_y[i] equals 0, then the length is Max(0, prec_principal_point-30).
[0140] --Otherwise (exponent_principal_point_y[i] is in the range of 0 to 63 (excluding the end value), the length is Max(0,exponent_principal_point_y[i]+prec_principal_point-31).
[0141] The value of sign_skew_factor[i] being 0 indicates that the sign of the tilt factor of the i-th camera is positive.
[0142] The sign_skew_factor[i] being equal to 1 indicates that the sign is negative.
[0143] `exponent_skew_factor[i]` specifies the exponent portion of the tilt factor for the i-th camera. The value of `exponent_skew_factor[i]` should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should interpret value 63 as an indication of an unspecified tilt factor.
[0144] `mantissa_skew_factor[i]` specifies the mantissa portion of the tilt factor for the i-th camera. The length of the `mantissa_skew_factor[i]` syntax element is variable and is determined as follows:
[0145] --If exponent_skew_factor[i] equals 0, then the length is Max(0,prec_skew_factor-30).
[0146] --Otherwise (exponent_skew_factor[i] is in the range of 0 to 63 (excluding the end value), the length is Max(0,exponent_skew_factor[i]+prec_skew_factor-31).
[0147] The internal matrix A[i] of the i-th camera is represented by the following formula.
[0148]
[0149] prec_rotation_param specifies that 2 -prec_rotation_param The exponent of the maximum permissible truncation error for r[i][j][k]. The value of prec_rotation_param should be in the range of 0 to 31 (inclusive).
[0150] prec_translation_param specifies that it is 2 -prec_translation_param The exponent of the maximum permissible truncation error for t[i][j]. The value of prec_translation_param should be in the range of 0 to 31 (inclusive).
[0151] A sign_r[i][j][k] equal to 0 indicates that the (j, k) component of the rotation matrix of the i-th camera has a positive sign. A sign_r[i][j][k] equal to 1 indicates that the sign is negative.
[0152] The exponent_r[i][j][k] specifies the exponent portion of the (j, k) component of the rotation matrix for the i-th camera. The value of the exponent r[i][j][k] should be in the range of 0 to 62 (inclusive). Value 63 is reserved by ITU-T|ISO / IEC for future use. The decoder should interpret value 63 as indicating an unspecified rotation matrix.
[0153] `mantissa_r[i][j][k]` specifies the mantissa portion of the (j, k) component of the rotation matrix for the i-th camera. The length of the `mantissa_r[i][j][k]` syntax element, in bits, is variable and is determined as follows:
[0154] --If exponent_r[i] equals 0, then the length is Max(0,prec_rotation_param-30).
[0155] --Otherwise (exponent_r[i] is in the range of 0 to 63 (excluding the end value), the length is Max(0,exponent_r[i]+prec_rotation_param-31).
[0156] The rotation matrix R[i] of the i-th camera is represented as follows:
[0157]
[0158] sign_t[i][j] equal to 0 indicates that the sign of the j-th component of the translation vector of the i-th camera is positive. sign_t[i][j] equal to 1 indicates that the sign is negative.
[0159] exponent_t[i][j] specifies the exponent part of the j-th component of the translation vector of the i-th camera. The value of exponent_t[i][j] shall be in the range from 0 to 62 inclusive. The value 63 is reserved for future use by ITU-T|ISO / IEC. The decoder shall treat the value 63 as indicating an unspecified translation vector.
[0160] mantissa_t[i][j] specifies the mantissa part of the j-th component of the translation vector of the i-th camera. The length v of the mantissa_t[i][j] syntax element in bits is variable and is determined as follows:
[0161] -- If exponent_t[i] is equal to 0, the length v is set equal to Max(0, prec_translation_param-30).
[0162] -- Otherwise (0 < exponent_t[i] < 63), the length v is set equal to Max(0, exponent_t[i] + prec_translation_param-31).
[0163] The translation vector T[i] of the i-th camera is represented by the following formula:
[0164]
[0165] The association between the camera parameter variables and the corresponding syntax elements is specified in Table ZZ. Each component of the internal matrix and the rotation matrix as well as the translation vector are obtained from the variables specified in Table ZZ, and the variable x is calculated as follows:
[0166] -- If e is in the range from 0 to 63 exclusive, x is set equal to (-1) s *2 e-31 *(1+n÷2 v ).
[0167] -- Otherwise (e is equal to 0), x is set equal to (-1) s *2 -(30+v) *n.
[0168] NOTE - The above specification is similar to the specification of IEC 60559:1989.
[0169] Table ZZ – Association between camera parameter variables and syntax elements.
[0170]
[0171]
[0172] The syntax and semantics of DRI SEI message 328 are as follows.
[0173] DRI SEI message syntax.
[0174]
[0175]
[0176]
[0177] DRI SEI message semantics.
[0178] The syntax elements in the Depth Representation Information (SEI) message specify various parameters for the auxiliary image of type AUX_DEPTH, so that the primary and auxiliary images are processed and decoded before being rendered on a 3D display, such as for view composition. Specifically, the depth or parallax range of the depth image is specified.
[0179] When present, the Depth Representation Information (SEI) message should be associated with one or more layers whose sdi_aux_id value is equal to AUX_DEPTH. The following semantics apply to each nuh_layer_id targetLayerId in the nuh_layer_id value to which the Depth Representation Information (SEI) message is applied.
[0180] When present, the depth representation information (SEI) message can be included in any access cell. It is recommended that, when present, the SEI message be included in an access cell for random access purposes, where the encoded / decoded image with nuh_layer_id equal to targetLayerId is an IRAP image.
[0181] For an auxiliary picture where sdi_aux_id[targetLayerId] equals AUX_DEPTH, for all j values in the range of 0 to 2 (inclusive) and 4 to 15 (inclusive), the associated primary picture (if any) is the picture in the same access unit where sdi_aux_id[nuhLayerIdB] equals 0, such that ScalabilityId[LayerIdxInVps[targetLayerId]][j] equals ScalabilityId[LayerIdxInVps[nuhLayerIdB]][j].
[0182] The information indicated in the SEI message applies to all images whose nuh_layer_id is equal to targetLayerId, starting from the access unit containing the SEI message, up to but not including the next image, in the decoding order, associated with the depth representation information SEI message applicable to targetLayerId, or up to the end of the CLVS where nuh_layer_id is equal to targetLayerId, whichever is earlier in the decoding order.
[0183] A z_near_flag value of 0 indicates that the syntax structure does not contain a syntax element that specifies the nearest depth value. A z_near_flag value of 1 indicates that the syntax structure contains a syntax element that specifies the nearest depth value.
[0184] A z_far_flag value of 0 indicates that the syntax structure does not contain any syntax elements that specify the farthest depth value. A z_far_flag value of 1 indicates that the syntax structure contains any syntax elements that specify the farthest depth value.
[0185] A value of 0 for d_min_flag indicates that the syntax structure does not contain any syntax element specifying the minimum disparity value. A value of 1 for d_min_flag indicates that the syntax structure contains a syntax element specifying the minimum disparity value.
[0186] A value of 0 for d_max_flag indicates that the syntax structure does not contain a syntax element that specifies the maximum disparity value. A value of 1 for d_max_flag indicates that the syntax structure contains a syntax element that specifies the maximum disparity value.
[0187] The `depth_representation_type` specifies the representation definition of the decoded luminance samples of the auxiliary image as defined in Table Y1. In Table Y1, parallax specifies the horizontal displacement between two texture views, while the Z value specifies the distance from the camera.
[0188] The variable maxVal is set to equal to (1<<(8+sps_bitdepth_minus8))-1, where sps_bitdepth_minus8 is the active SPS contained in the layer whose nuh_layer_id is equal to targetLayerId or its inferred value.
[0189] Definition of Table Y1 – depth_representation_type
[0190]
[0191]
[0192] disparity_ref_view_id specifies the ViewId value on which the disparity value is derived.
[0193] Note 1 - Disparity_ref_view_id exists only when d_min_flag equals 1 or d_max_flag equals 1, and is useful for depth_representation_type values of 1 and 3.
[0194] The variables in column x of table Y2 are derived from the corresponding variables in columns s, e, n, and v of table Y2, as shown below:
[0195] --If the value of e is in the range of 0 to 127 (excluding extreme values), then x is set to equal to (-1). s *2 e-31 *(1+n÷2 v ).
[0196] Otherwise (if e equals 0), x is set to equal to (-1). s *2 -(30+v) *n.
[0197] Note 1 - The above specifications are similar to those in IEC 60559:1989.
[0198] Table Y2 – Relationship between depth parameter variables and syntax elements
[0199]
[0200]
[0201] When DMin and DMax values exist, the DMin and DMax values are specified in units of the luminance sample width of the encoded and decoded image, where ViewId is equal to the ViewId of the auxiliary image.
[0202] When they exist, the units of ZNear and ZFar values are the same, but the ZNear and ZFar values are not specified.
[0203] depth_nonlinear_representation_num_minus1 plus 2 specifies the number of piecewise linear segments used to map depth values to a scale uniformly quantized according to disparity.
[0204] For i ranging from 0 to depth_nonlinear_representation_num_minus1+2 (inclusive), the depth_nonlinear_representation_model[i] is defined to map the decoded luminance sample values of the auxiliary image to piecewise linear segments of a scale uniformly quantized according to disparity. The values of depth_nonlinear_representation_model[0] and depth_nonlinear_representation_model[depth_nonlinear_representation_num_minus1+2] are both inferred to be equal to 0.
[0205] Note 2 - When depth_representation_type equals 3, the auxiliary image contains depth samples with a non-linear transformation. The variable DepthLUT[i], as specified below, is used to transform the decoded depth sample values from a non-linear representation to a linear representation, i.e., uniformly quantized disparity values. The shape of this transformation is defined by approximating a line segment in the non-linear disparity space from a two-dimensional linear disparity. The first (0, 0) and last (maxVal, maxVal) nodes of the curve are predefined. The positions of the additional nodes are transmitted in the form of deviations from the straight curve (depth_nonlinear_representation_model[i]). These deviations are uniformly distributed across the entire range from 0 to maxVal (inclusive), with spacing depending on the value of nonlinear_depth_representation_num_minus1.
[0206] The variable DepthLUT[i] for i in the range of 0 to maxVal (inclusive) is defined as follows:
[0207]
[0208]
[0209] When depth_representation_type equals 3, DepthLUT[dS] represents the disparity uniformly quantized to the range of 0 to maxVal (inclusive) for all decoded luminance sample values dS of the auxiliary image in the range of 0 to maxVal (inclusive).
[0210] The syntax structure specifies the values of elements in the depth representation information SEI message.
[0211] The syntax structure sets the values of the variables OutSign, OutExp, OutMantissa, and OutManLen, which represent floating-point values. When this syntax structure is contained within another syntax structure, the variable names OutSign, OutExp, OutMantissa, and OutManLen are interpreted as being replaced by the variable names used when the syntax structure was contained.
[0212] A value of 0 for da_sign_flag indicates a positive sign for the floating-point value. A value of 1 for da_sign_flag indicates a negative sign. The variable OutSign is set to equal da_sign_flag.
[0213] `da_exponent` specifies the exponent of the floating-point value. The value of `da_exponent` should be between 0 and 2. 7 The range is -2 (inclusive). Value 2 7 -1 is reserved for future use by ITU-T|ISO / IEC. The decoder should set the value to 2. 7 -1 is considered an unspecified value. The variable OutExp is set to equal da_exponent.
[0214] The increment of 1 in `da_mantissa_len_minus1` specifies the number of bits in the `da_mantissa` syntax element. The value of `da_mantissa_len_minus1` should be in the range of 0 to 31 (inclusive). The variable `OutManLen` is set to equal to `da_mantissa_len_minus1 + 1`.
[0215] `da_mantissa` specifies the mantissa of a floating-point value. The variable `OutMantissa` is set to be equal to `da_mantissa`.
[0216] The syntax and semantics of ACI SEI message 300 are as follows.
[0217] ACI SEI message syntax.
[0218]
[0219]
[0220] ACI SEI message semantics.
[0221] The α channel information SEI message provides information about the α channel sample values and the post-processing applied to the decoded α plane, which is encoded and decoded in an auxiliary picture of type AUX_ALPHA and one or more associated primary pictures.
[0222] For an auxiliary picture where nuh_layer_id equals nuhLayerIdA and sdi_aux_id[nuhLayerIdA] equals AUX_ALPHA, for all j values in the range of 0 to 2 (inclusive) and 4 to 15 (inclusive), the associated primary picture (if any) is the picture in the same access unit where sdi_aux_id[nuhLayerIdB] equals 0, such that scalabilityid[LayerIdxInVps[nuhLayerIdA]][j] equals scalabilityid[LayerIdxInVps[nuhLayerIdB]][j].
[0223] When the access unit contains an auxiliary image picA with nuh_layer_id equal to nuhLayerIdA and sdi_aux_id[nuhLayerIdA] equal to AUX_ALPHA, the alpha channel sample values of picA continue in output order until one or more of the following conditions are true:
[0224] -- Output the next image whose nuh_layer_id is equal to nuhLayerIdA, in the order of output.
[0225] --End of CLVS including auxiliary image picA.
[0226] --End of bitstream.
[0227] --nuh_layer_id equals nuhLayerIdA for the end of any related primary layer's CLVS for auxiliary image layers.
[0228] The following semantics apply to each nuh_layer_id targetLayerId in the nuh_layer_id value applied to the SEI message of the α channel information.
[0229] An alpha_channel_cancel_flag value of 1 indicates that the alpha channel information SEI message cancels the persistence of any previous alpha channel information SEI messages applied to the current layer in the order of output. An alpha_channel_cancel_flag value of 0 indicates that an alpha channel message follows.
[0230] Let currPic be the image associated with the alpha channel information SEI message. The semantics of the alpha channel information SEI message are continuously applied to the current layer in the order of output until one or more of the following conditions are true:
[0231] --A new CLVS for the current layer begins.
[0232] --End of bitstream.
[0233] --After the decoding process of calling the image sequence count of picB, immediately output the image picB in the access unit where numh_layer_id is equal to targetLayerId. This access unit contains an α channel information SEI message where numh_layer_id is equal to targetLayerId, and its PicOrderCnt(picB) is greater than PicOrderCnt(currPic). PicOrderCnt(picB) and PicOrderCnt(currPic) are the PicOrderCntVal values of picB and currPic, respectively.
[0234] An alpha_channel_use_idc value of 0 indicates that, for alpha mixing purposes, the decoded samples of the associated primary picture should be multiplied by the interpreted sample value of the auxiliary codec picture during display after the output from the decoding process. An alpha_channel_use_idc value of 1 indicates that, for alpha mixing purposes, the decoded samples of the associated primary picture should not be multiplied by the interpreted sample value of the auxiliary codec picture during display after the output from the decoding process. An alpha_channel_use_idc value of 2 indicates that the use of auxiliary pictures is not specified. Values of alpha_channel_use_idc greater than 2 are reserved for future use by ITU-T|ISO / IEC. When it does not exist, the value of alpha_channel_use_idc is inferred to be equal to 2.
[0235] `alpha_channel_bit_depth_minus8` plus 8 specifies the bit depth of the luminance sample array of the auxiliary image. `alpha_channel_bit_depth_minus8` should be in the range of 0 to 7 (inclusive). `alpha_channel_bit_depth_minus8` should be equal to `bit_depth_luma_minus8` of the relevant primary image.
[0236] `alpha_transparent_value` specifies the interpreted sample value for the luminance sample of the auxiliary codec image. For this auxiliary codec image luminance sample, the correlated luminance and chrominance samples of the primary codec image are considered transparent for alpha blending purposes. The number of bits used to represent the `alpha_transparent_value` syntax element is `alpha_channel_bit_depth_minus8+9`.
[0237] `alpha_opaque_value` specifies the interpreted sample value for the luminance sample of the auxiliary codec image. For this auxiliary codec image luminance sample, the correlated luminance and chrominance samples of the primary codec image are considered opaque and used for alpha blending. The number of bits used to represent the `alpha_opaque_value` syntax element is `alpha_channel_bit_depth_minus8+9`.
[0238] An `alpha_channel_incr_flag` value of 0 indicates that, for alpha mixing purposes, the interpreted sample value for each decoded auxiliary image luminance sample value is equal to the decoded auxiliary image sample value. An `alpha_channel_incr_flag` value of 1 indicates that, for alpha mixing purposes, after decoding the auxiliary image samples, any auxiliary image luminance sample value greater than `Min(alpha_opaque_value, alpha_transparent_value)` should be incremented by 1 to obtain the interpreted sample value for the auxiliary image sample, and any auxiliary image luminance sample value less than or equal to `Min(alpha_opaque_value, alpha_transparent_value)` should be used unchanged as the interpreted sample value for the decoded auxiliary image sample value. When it does not exist, the value of `alpha_channel_incr_flag` is inferred to be 0.
[0239] An alpha_channel_clip_flag value of 0 indicates that no cropping operation is applied to obtain the interpreted sample values of the decoding aid image. An alpha_channel_clip_flag value of 1 indicates that the interpreted sample values of the decoding aid image are changed according to the cropping process described by the alpha_channel_clip_type_flag syntax element. When it does not exist, the value of alpha_channel_clip_flag is inferred to be 0.
[0240] An `alpha_channel_clip_type_flag` value of 0 indicates that, for alpha blending purposes, after decoding the auxiliary image samples, any auxiliary image luminance sample greater than (alpha_opaque_value - alpha_transparent_value) / 2 is set to equal to `alpha_opaque_value` to obtain the interpreted sample value, and any auxiliary image luminance sample less than or equal to (alpha_opaque_value - alpha_transparent_value) / 2 is set to equal to `alpha_transparent_value` to obtain the interpreted sample value. An `alpha_channel_clip_type_flag` value of 1 indicates that, for alpha blending purposes, after decoding the auxiliary image samples, any auxiliary image luminance sample greater than `alpha_opaque_value` is set to equal to `alpha_opaque_value` to obtain the interpreted sample value, and any auxiliary image luminance sample less than or equal to `alpha_transparent_value` is set to equal to `alpha_transparent_value` to obtain the interpreted sample value.
[0241] Note - When both alpha_channel_incr_flag and alpha_channel_clip_flag are equal to 1, the cropping operation specified by alpha_channel_clip_type_flag should be applied first, and then the changes specified by alpha_channel_incr_flag should be applied to obtain the interpretation sample values of the auxiliary image brightness samples.
[0242] Unfortunately, the current design has at least the following problems with signaling notifications of scalability dimension information, depth representation information, and alpha channel information in SEI messages.
[0243] 1) There is an issue with the current persistence range specification for Scalability Dimension Information (SDI) SEI messages: there is no good way to indicate a set of AUs that does not indicate SDI if it follows another set of AUs that indicates SDI.
[0244] 2) Currently, it is stipulated that the value of sdi_view_id_val[i] is inferred to be equal to 0 when it does not exist. While this is good for contexts where SDI SEI messages exist, it is bad for contexts where SDI SEI messages do not exist, in which case the value of the view ID should not be assumed or inferred.
[0245] 3) Currently, the value of sdi_aux_id[i] is not specified when the syntax element does not exist. However, when sdi_auxiliary_info_flag is equal to 0 (which means that the SDI SEI message exists), for each value of i, the value of sdi_aux_id[i] needs to be inferred to be equal to 0 in order to infer that there is no auxiliary image.
[0246] 4) The Multi-View Acquisition Information (MAI) SEI message carries information about all views in the multi-view bitstream, and therefore should not be specified as layer-specific (as is the case now). Instead, the scope should be the current CVS, not the current CLVS.
[0247] 5) Currently, when an access unit contains both SDI SEI messages and MAI SEI messages, the MAI SEI messages can precede the SDI SEI messages in the decoding order. However, the existence and interpretation of the MAI SEI messages depend on the SDI SEI messages. Therefore, within the same AU, it makes more sense to require the SDI SEI messages to precede the MAI SEI messages in the decoding order.
[0248] 6) Currently, when an access unit contains both SDI SEI messages and Depth Representation Information (DRI) SEI messages, the DRI SEI messages may precede the SDI SEI messages in the decoding order. However, the presence and interpretation of the DRI SEI messages depend on the SDI SEI messages. Therefore, within the same AU, it makes more sense to require the SDI SEI messages to precede the DRI SEI messages in the decoding order.
[0249] 7) Currently, when an access unit contains both SDI SEI messages and Alpha Channel Information (ACI) SEI messages, the ACI SEI messages can precede the SDI SEI messages in the decoding order. However, the presence and interpretation of the ACI SEI messages depend on the SDI SEI messages. Therefore, within the same AU, it makes more sense to require the SDI SEI messages to precede the ACI SEI messages in the decoding order.
[0250] 8) Currently, SDI SEI messages can be included in scalable nested SEI messages. However, since SDI SEI messages contain information from all layers, it makes less sense to include them in scalable nested SEI messages.
[0251] This document discloses techniques for addressing one or more of the aforementioned problems. For example, this disclosure provides techniques for specifying the duration of Multi-View Acquisition Information (MAI) SEI messages, specifying the position of MAI SEI messages within a codec video sequence (CVS), and specifying the content of all MAI SEI messages within the same CVS. By specifying the duration, position, and content of MAI SEI messages, the video encoding and decoding process is improved.
[0252] To address the aforementioned issues, methods summarized below are disclosed. These techniques should be considered as examples for explaining general concepts, rather than interpreted in a narrow sense. Furthermore, these techniques can be applied individually or in combination in any way.
[0253] Example 1
[0254] To address issue 1, the persistence range specification for the Scalability Dimension Information (SDI) SEI message is specified as one of the following:
[0255] a. SDI SEI messages continue in the decoding order from the current AU to the next AU containing an SDI SEI message whose content differs from the current SDI SEI message, or until the end of the bit stream.
[0256] b. The persistence range of the SDI SEI message is defined as the current CVS (i.e., the CVS containing the SDI SEI message).
[0257] c. If at least one AU in the current CVS following the current AU in decoding order is associated with an SDI SEI message, then the bitstreamInScope applied to the SDI SEI message is an AU sequence that includes the current AU in decoding order, followed by zero or more AUs, including all subsequent AUs, up to but excluding any subsequent AU containing the SDI SEI message. Otherwise, bitstreamInScope is an AU sequence that includes the current AU in decoding order, followed by zero or more AUs, including all subsequent AUs, up to and including the last AU in the current CVS in decoding order.
[0258] d. Add a cancellation flag and / or a persistence flag to the SDI SEI message syntax, and specify the persistence range of the SDI SEI message based on the cancellation flag and / or persistence flag.
[0259] Example 2
[0260] 2) In one example, it is specified that if an SDI SEI message exists in any AU of CVS, the SDI SEI message should exist in the first AU of CVS.
[0261] Example 3
[0262] 3) In one example, it is specified that all SDI SEI messages applied to the same CVS should have the same content.
[0263] Example 4
[0264] 4) To solve problem 2, it is stipulated that when sdi_multiview_info_flag equals 0, the value of sdi_view_id_val[i] is inferred to be equal to 0.
[0265] Example 5
[0266] 5) To solve problem 3, it is stipulated that when sdi_auxiliary_info_flag is equal to 0, the value of sdi_aux_id[i] is inferred to be equal to 0.
[0267] Example 6
[0268] 6) To solve problem 4, it is stipulated that the Multi-View Acquisition Information (MAI) SEI message continues in the decoding order from the current AU to the next AU containing a MAI SEI message whose content is different from the current MAI SEI message, or continues until the end of the bit stream.
[0269] Example 7
[0270] 7) In one example, it is specified that when the MAI SEI message exists in any AU of CVS, the MAI SEI message should appear in the first AU of CVS.
[0271] Example 8
[0272] 8) In one example, it is specified that all MAI SEI messages applied to the same CVS should have the same content.
[0273] Example 9
[0274] 9) To solve problem 5, it is stipulated that when AU contains both SDI SEI messages and MAI SEI messages, the SDI SEI messages should precede the MAI SEI messages according to the decoding order.
[0275] Example 10
[0276] 10) To solve problem 6, it is stipulated that for at least one i value, when the AU contains both an SDI SEI message with sdi_aux_id[i] equal to 2 and a Depth Representation Information (DRI) SEI message, the SDI SEI message should precede the DRI SEI message in the decoding order.
[0277] Example 11
[0278] 11) To solve problem 7, it is stipulated that for at least one i value, when the AU contains both an SDI SEI message with sdi_aux_id[i] equal to 1 and an α channel information (ACI) SEI message, the SDI SEI message should precede the ACI SEI message in the decoding order.
[0279] Example 12
[0280] 12) To address issue 8, it is stipulated that SDI SEI messages should not be included in scalable nested SEI messages.
[0281] The following are some example implementations of some aspects summarized above.
[0282] This embodiment can be applied to VVC. Most of the relevant parts that have been added or modified are indicated in bold, and some deleted parts are indicated in bold italics. There may be other editorial changes, which are not highlighted.
[0283] Scalability dimension SEI message semantics.
[0284] The Scalability Dimension Information (SDI) SEI message provides the SDI for each layer in bitstreamInScope, such as 1) the view ID of each layer when bitstreamInScope may be a multi-view bitstream; and 2) the auxiliary ID of each layer when bitstreamInScope may contain auxiliary information (such as depth or α) carried by one or more layers.
[0285] `bitstreamInScope` is an AU sequence that follows the decoding order, including the AU containing the current SDI SEI message, followed by zero or more subsequent AUs, including all subsequent AUs, up to but not including any subsequent AU containing an SDI SEI message. When an SDI SEI message exists in any AU of a CVS, the first AU of the CVS should also contain an SDI SEI message. All SDI SEI messages applied to the same CVS should have the same content.
[0286] SDI SEI messages should not be included in scalable nested SEI messages.
[0287] `sdi_view_id_val[i]` specifies the view ID of the i-th level in `bitstreamInScope`. The length of the `sdi_view_id_val[i]` syntax element is `sdi_view_id_len_minus1+1` bits. When there is no `sdi_multiview_info_flag` equal to 0, the value of `sdi_view_id_val[i]` is inferred to be equal to 0.
[0288] A value of 0 for sdi_aux_id[i] indicates that the i-th layer in bitstreamInScope does not contain auxiliary images. A value greater than 0 for sdi_aux_id[i] indicates the type of auxiliary image in the i-th layer of bitstreamInScope as specified in Table 1. When sdi_auxiliary_info_flag is equal to 0, the value of sdi_aux_id[i] is inferred to be equal to 0.
[0289] Multi-view information retrieval SEI message semantics.
[0290] The Multi-View Acquisition Information (MAI) SEI message specifies various parameters of the acquisition environment. Specifically, it specifies the parameters of the internal and external cameras. These parameters can be used to process the decoded view before rendering it on a 3D display.
[0291] The following semantics apply to each nuh_layer_id targetLayerId in the nuh_layer_id value applied to the SEI message for multi-view information retrieval.
[0292] When present, the SEI message for multi-view acquisition applied to the current layer should be included in the access unit containing the IRAP image, which is the first image of the current layer's CLVS. The signaling notification information in the SEI message is applied to the CLVS.
[0293] MAI SEI messages continue in decoding order from the current AU to the next AU containing a MAI SEI message whose content differs from the current MAI SEI message, or until the end of the bitstream. When a MAI SEI message exists in any AU of a CVS, a MAI SEI message should be provided for the first AU of the CVS. All MAI SEI messages applied to the same CVS should have the same content.
[0294] When an AU contains both SDI SEI messages and MAI SEI messages, the SDI SEI messages should precede the MAI SEI messages according to the decoding order.
[0295] The multi-view retrieval information contained in the multi-view retrieval information (SEI) message may include some views that may not exist.
[0296] The depth representation information SEI message semantics.
[0297] The syntax elements in the Depth Representation Information (SEI) message specify various parameters for the auxiliary image of type AUX_DEPTH, so that the primary and auxiliary images are processed and decoded before being rendered on a 3D display, such as for view composition. Specifically, the depth or parallax range of the depth image is specified.
[0298] When an AU contains both an SDI SEI message and a DRI SEI message where sdi_aux_id[i] equals 2, for at least one value of i, the SDI SEI message should precede the DRI SEI message in the decoding order.
[0299] α-channel information SEI message semantics.
[0300] The Alpha Channel Information (ACI) SEI message provides information about the Alpha Channel sample values and the post-processing applied to the decoded Alpha plane, which is encoded and decoded in an auxiliary picture of type AUX_ALPHA and one or more associated primary pictures.
[0301] When an AU contains both an SDI SEI message and an ACI SEI message where sdi_aux_id[i] equals 1, for at least one value of i, the SDI SEI message should precede the ACI SEI message in the decoding order.
[0302] Figure 4 This is a block diagram of an example video processing system 400 that can implement the various techniques disclosed herein. Various implementations may include some or all of the components in the video processing system 400. The video processing system 400 may include an input 402 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8 or 10-bit multi-component pixel values), or in a compressed or encoded format. Input 402 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces (such as Ethernet, Passive Optical Networking (PON), etc.) and wireless interfaces (such as Wi-Fi or cellular interfaces).
[0303] Video processing system 400 may include codec component 404 capable of implementing the various codec or encoding methods described herein. Codec component 404 can reduce the average bit rate of the video from input 402 to the output of codec component 404 to produce a codec representation of the video. Therefore, codec techniques are sometimes referred to as video compression or video transcoding techniques. The output of codec component 404 may be stored or transmitted via connected communication, as represented by component 406. The stored or communicated bitstream (or codec) representation of the video received at input 402 may be used by component 408 to generate pixel values or displayable video that is sent to display interface 410. The process of generating user-visible video from the bitstream representation is sometimes referred to as video decompression. Furthermore, although some video processing operations are referred to as “codec” operations or tools, it should be understood that the codec tool or operation is used at the encoder, and the corresponding decoding tool or operation will be inverted by the decoder to retrieve the result of the codec.
[0304] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or DisplayPort. Examples of storage interfaces include SATA (Serial Advanced Technology Accessory), Peripheral Component Interconnect (PCI), and Integrated Drive Electronic Devices (IDE) interfaces. The technologies described in this document can be implemented in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of digital data processing and / or video display.
[0305] Figure 5 This is a block diagram of a video processing apparatus 500. Apparatus 500 can be used to implement one or more of the methods described herein. Apparatus 500 can be implemented in smartphones, tablets, computers, Internet of Things (IoT) receivers, etc. Apparatus 500 may include one or more processors 502, one or more memories 504, and video processing hardware 506 (also referred to as video processing circuitry). The processors(multiple) 502 can be configured to implement one or more methods described herein. The memories(multiple) 504 can be used to store data and code used to implement the methods and techniques described herein. The video processing hardware 506 can be used to implement some of the techniques described herein in hardware circuitry. In some embodiments, hardware 506 may be partially or entirely located within processor 502, such as a graphics processor.
[0306] Figure 6 This is a block diagram illustrating an example video encoding / decoding system 600 that can utilize the techniques disclosed herein. Figure 6As shown, the video encoding / decoding system 600 may include a source device 610 and a destination device 620. The source device 610 generates encoded video data and may be referred to as a video encoding device. The destination device 620 can decode the encoded video data generated by the source device 610 and may be referred to as a video decoding device.
[0307] The source device 610 may include a video source 612, a video encoder 614, and an input / output (I / O) interface 616.
[0308] Video source 612 may include sources such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system that generates video data, or a combination of these sources. Video data may include one or more pictures. Video encoder 614 encodes the video data from video source 612 to generate a bitstream. The bitstream may include a sequence of bits forming a codec representation of the video data. The bitstream may include codec pictures and associated data. A codec picture is a codec representation of a picture. Associated data may include sequence parameter sets, picture parameter sets, and other syntax elements. I / O interface 616 includes a modulator / demodulator (modem) and / or a transmitter. Encoded video data may be transmitted directly to destination device 620 via network 630 through I / O interface 616. Encoded video data may also be stored on storage medium / server 640 for access by destination device 620.
[0309] Destination device 620 may include I / O interface 626, video decoder 624, and display device 622.
[0310] I / O interface 626 may include a receiver and / or a modem. I / O interface 626 may acquire encoded video data from source device 610 or storage medium / server 640. Video decoder 624 may decode the encoded video data. Display device 622 may display the decoded video data to a user. Display device 622 may be integrated with destination device 620 or may be external to destination device 620 configured to connect to an external display device.
[0311] The video encoder 614 and the video decoder 624 can operate according to video compression standards such as the High Efficiency Video Codec (HEVC) standard, the Multi-Functional Video Codec (VVC) standard, and other current and / or other standards.
[0312] Figure 7 This is a block diagram illustrating an example of a video encoder 700, which can be... Figure 6 The video encoder 614 in the video encoding and decoding system 600 shown in the figure.
[0313] The video encoder 700 can be configured to perform any or all of the techniques disclosed herein. Figure 7 In the example, the video encoder 700 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video encoder 700. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.
[0314] The functional components of the video encoder 700 may include a segmentation unit 701, a prediction unit 702 (which may include a mode selection unit 703, a motion estimation unit 704, a motion compensation unit 705, and an intra-frame prediction unit 706), a residual generation unit 707, a transform unit 708, a quantization unit 709, an inverse quantization unit 710, an inverse transform unit 711, a reconstruction unit 712, a buffer 713, and an entropy coding unit 714.
[0315] In other examples, the video encoder 700 may include more, fewer, or different functional components. In one example, the prediction unit 702 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in IBC mode, where at least one reference picture is the picture in which the current video block is located.
[0316] Furthermore, some components, such as the motion estimation unit 704 and the motion compensation unit 705, can be highly integrated, but for illustrative purposes... Figure 7 The examples are shown separately.
[0317] The segmentation unit 701 can segment an image into one or more video blocks. Figure 6 The video encoder 614 and video decoder 600 can support various video block sizes.
[0318] The mode selection unit 703 can, for example, select one of the intra-frame or inter-frame encoding / decoding modes based on the error result, and provide the obtained intra-frame or inter-frame encoded / decoded blocks to the residual generation unit 707 to generate residual block data and to the reconstruction unit 712 to reconstruct the encoded / decoded blocks for use as reference images. In some examples, the mode selection unit 703 can select a combined intra-frame and inter-frame prediction (CIIP) mode, where prediction is based on inter-frame prediction signals and intra-frame prediction signals. The mode selection unit 703 can also select the resolution of the motion vector (e.g., sub-pixel or integer pixel precision) for blocks in the inter-frame prediction case.
[0319] To perform inter-frame prediction for the current video block, motion estimation unit 704 can generate motion information for the current video block by comparing one or more reference frames from buffer 713 with the current video block. Motion compensation unit 705 can determine the predicted video block for the current video block based on the motion information of the image from buffer 713 (rather than the image associated with the current video block) and decoded samples.
[0320] The motion estimation unit 704 and motion compensation unit 705 can perform different operations on the current video block, depending on whether the current video block is in an I-band, P-band, or B-band. I-bands (or I-frames) have the lowest compression ratio but do not require other video frames for decoding. S-bands (or P-frames) can use data from previous frames for decompression and are easier to compress than I-frames. B-bands (or B-frames) can use both previous and preceding frames as data references to achieve the highest data compression.
[0321] In some examples, motion estimation unit 704 can perform unidirectional prediction of the current video block, and can search for a reference video block for the current video block in the reference images of list 0 or list 1. Motion estimation unit 704 can then generate a reference index indicating that the reference image in list 0 or list 1 contains the reference video block, and a motion vector indicating the spatial displacement between the current video block and the reference video block. Motion estimation unit 704 can output the reference index, prediction direction indicator, and motion vector as motion information for the current video block. Motion compensation unit 705 can generate a predicted video block for the current block based on the reference video block indicated by the motion information of the current video block.
[0322] In other examples, motion estimation unit 704 can perform bidirectional prediction of the current video block. Motion estimation unit 704 can search for a reference video block for the current video block in the reference images of list 0 and can also search for another reference video block for the current video block in the reference images of list 1. Motion estimation unit 704 can then generate a reference index indicating that the reference images in list 0 or list 1 contain the reference video block, and a motion vector indicating the spatial displacement between the reference video block and the current video block. Motion estimation unit 704 can output the reference index and the motion vector of the current video block as the motion information of the current video block. Motion compensation unit 705 can generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.
[0323] In some examples, the motion estimation unit 704 can output the complete set of motion information for the decoder's decoding process.
[0324] In some examples, the motion estimation unit 704 may not output the complete set of motion information for the current video. Instead, the motion estimation unit 704 may signal the motion information of the current video block by referencing the motion information of another video block. For example, the motion estimation unit 704 may determine that the motion information of the current video block is sufficiently similar to the motion information of adjacent video blocks.
[0325] In one example, the motion estimation unit 704 may indicate in the syntax structure associated with the current video block that the video decoder 624 indicates that the current video block has the same motion information value as another video block.
[0326] In another example, motion estimation unit 704 can identify another video block and motion vector difference (MVD) in the syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicating video block. Video decoder 624 can use the motion vector of the indicating video block and the motion vector difference to determine the motion vector of the current video block.
[0327] As discussed above, the video encoder 614 can predictively signal motion vectors. Two examples of predictive signaling notification techniques that can be implemented by the video encoder 614 include Advanced Motion Vector Prediction (AMVP) and merge pattern signaling notification.
[0328] The intra-prediction unit 706 can perform intra-prediction on the current video block. When performing intra-prediction on the current video block, the intra-prediction unit 706 can generate prediction data for the current video block based on the decoded samples of other video blocks in the same frame. The prediction data for the current video block may include the predicted video block and various syntax elements.
[0329] The residual generation unit 707 can generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) multiple predicted video blocks from the current video block. The residual data for the current video block can include residual video blocks corresponding to different sample components of the samples in the current video block.
[0330] In other examples, such as in skip mode, residual data for the current video block may not exist, and the residual generation unit 707 may not perform a subtraction operation.
[0331] Transform unit 708 can generate one or more transform coefficient video blocks of the current video block by applying one or more transforms to the residual video block associated with the current video block.
[0332] After the transform unit 708 generates a transform coefficient video block associated with the current video block, the quantization unit 709 can quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values associated with the current video block.
[0333] The inverse quantization unit 710 and the inverse transform unit 711 can apply inverse quantization and inverse transform to the transform coefficient video block respectively to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 712 can add the reconstructed residual video block to the corresponding samples of one or more predicted video blocks generated by the prediction unit 702 to produce a reconstructed video block associated with the current block for storage in the buffer 713.
[0334] After the video block is reconstructed by reconstruction unit 712, a loop filtering operation can be performed to reduce video block artifacts in the video block.
[0335] The entropy encoding unit 714 can receive data from other functional components of the video encoder 700. When the entropy encoding unit 714 receives data, it can perform one or more entropy encoding operations to generate entropy-encoded data and output a bitstream including the entropy-encoded data.
[0336] Figure 8 This is a block diagram illustrating an example of a video decoder 800, which can be... Figure 6 The video decoder 624 in the video encoding and decoding system 600 shown in the figure.
[0337] The video decoder 800 can be configured to perform any or all of the techniques disclosed herein. Figure 8 In the example, the video decoder 800 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video decoder 800. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.
[0338] exist Figure 8 In the example, the video decoder 800 includes an entropy decoding unit 801, a motion compensation unit 802, an intra-frame prediction unit 809, an inverse quantization unit 804, an inverse transform unit 805, a reconstruction unit 806, and a buffer 807. In some examples, the video decoder 800 can perform operations related to the video encoder 614 ( Figure 6 The decoding process is the overall inversion of the encoding process described.
[0339] The entropy decoding unit 801 can retrieve the encoded bitstream. The encoded bitstream may include entropy-encoded video data (e.g., encoded blocks of video data). The entropy decoding unit 801 can decode the entropy-encoded video, and based on the entropy-encoded video data, the motion compensation unit 802 can determine motion information including motion vectors, motion vector precision, reference image list index, and other motion information. The motion compensation unit 802 can determine such information, for example, by performing AMVP and merge mode signaling notifications.
[0340] The motion compensation unit 802 can generate motion compensation blocks, possibly based on interpolation filters. The identifier of the interpolation filter to be used at sub-pixel precision can be included in the syntax element.
[0341] The motion compensation unit 802 can use the interpolation filter used by the video encoder 614 during the encoding of the video block to calculate the interpolation values of a sub-integer number of pixels of the reference block. The motion compensation unit 802 can determine the interpolation filter used by the video encoder 614 based on the received syntax information and use the interpolation filter to generate the prediction block.
[0342] The motion compensation unit 802 can use some syntax information to determine: the size of the blocks used to encode (multiple) frames and / or (multiple) stripes of the encoded video sequence, segmentation information describing how each macroblock of the image of the encoded video sequence is segmented, the mode indicating how each segment is encoded, one or more reference frames (and a list of reference frames) for each inter-frame coded block, and other information for decoding the encoded video sequence.
[0343] Intra-prediction unit 803 can use, for example, an intra-prediction mode received in the bitstream to form prediction blocks from spatially adjacent blocks. Inverse quantization unit 804 inverse quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 801. Inverse transform unit 805 applies the inverse transform.
[0344] The reconstruction unit 806 can sum the residual blocks using the corresponding prediction blocks generated by the motion compensation unit 802 or the intra-frame prediction unit 803 to form a decoded block. As desired, a deblocking filter can also be applied to filter the decoded block to remove block artifacts. The decoded video block is then stored in a buffer 807, which provides a reference block for subsequent motion compensation / intra-frame prediction and also produces the decoded video for presentation on the display device.
[0345] Figure 9This is a video data encoding / decoding method 900 according to embodiments of the present disclosure. Method 900 can be executed by an encoding / decoding device (e.g., an encoder) having a processor and memory. Method 900 can be implemented when information is transmitted in a bitstream using SEI messages.
[0346] In block 902, whenever any Multi-View Acquisition Information (MAI) Supplemental Enhancement Information (SEI) message is present in any first access unit (AU) of the codec video sequence (CVS), it is determined that an MAI SEI message exists in an AU of the CVS.
[0347] In block 904, the codec device performs a conversion between video and a video bitstream based on the MAI SEI message. When implemented in the encoder, the conversion includes receiving the video and encoding it into a bitstream that includes the SEI message. When implemented in the decoder, the conversion includes receiving the bitstream that includes the SEI message and decoding the bitstream that includes the SEI message to reconstruct the video.
[0348] In this embodiment, the MAI SEI message is one of any MAI SEI messages, and the first AU is one of any AU. In some embodiments, the first AU means the first AU encountered in the CVS and / or bitstream in the decoding order. In some embodiments, the decoding order means, for example, in Figure 1-3 The direction is from left to right. In an embodiment, the MAI SEI message specifies internal and external camera parameters. In an embodiment, the internal and external camera parameters are used to process the decoded view before rendering on a 3D display. In some embodiments, the internal camera parameters are parameters necessary to link the pixel coordinates of image points with corresponding coordinates in a camera reference frame. In some embodiments, the external camera parameters are parameters that define the position and orientation of the camera reference frame relative to a known world reference frame.
[0349] In this embodiment, all MAI SEI messages applicable to CVS have the same content. In some embodiments, "same content" means that the content is substantially similar or identical. In one embodiment, the MAI SEI message is one of all MAI SEI messages. In this embodiment, MAI SEI messages continue in decoding order from the current AU to the subsequent AU containing subsequent MAI SEI messages. In this embodiment, subsequent MAI SEI messages contain content different from the original MAI SEI messages.
[0350] In this embodiment, the MAI SEI messages continue from the current AU to the end of the bitstream in the decoding order. In this embodiment, the MAI SEI messages follow the Scalability Dimension Information (SDI) SEI messages in the decoding order.
[0351] In one embodiment, the MAI SEI message includes internal parameter flags. In another embodiment, the MAI SEI message includes external parameter flags. A flag is a variable or a single bit syntax element that can take one of two possible values: 0 or 1.
[0352] In this embodiment, the MAI SEI message applies to the entire CVS, not just the codec layer video sequence (CLVS). In this embodiment, the MAI SEI message is contained within a scalable nested SEI message. A scalable nested SEI message is an SEI message within a scalable nested SEI message. A scalable nested SEI message is a message containing multiple scalable nested SEI messages corresponding to one or more layers in one or more output layer sets or multi-layer bitstreams.
[0353] In one embodiment, an internal parameter flag of 1 indicates the presence of an internal camera parameter, and an internal parameter flag of 0 indicates the absence of an internal camera parameter. In another embodiment, an external parameter flag of 1 indicates the presence of an external camera parameter, and an external parameter flag of 0 indicates the absence of an external camera parameter.
[0354] In one embodiment, method 900 further includes encoding the MAI SEI message into a bitstream by a video codec. In another embodiment, method 900 further includes decoding the bitstream by the video codec to obtain the MAI SEI message.
[0355] In an embodiment, method 900 may utilize or combine one or more features or processes of other methods disclosed herein.
[0356] The following provides a list of preferred solutions for some embodiments.
[0357] The following solutions illustrate example embodiments of the techniques discussed in this disclosure (e.g., Example 1).
[0358] 1. A video processing method comprising: performing a conversion between a video and a bitstream of the video; wherein a Scalability Dimension Information (SDI) Supplemental Enhancement Information (SEI) message is indicated for the video; and wherein rules define the duration range of the SDI SEI message or constraints on the SDI SEI message.
[0359] 2. The method according to claim 1, wherein the rule stipulates that the SDI SEI messages continue from the current access unit (AU) in the decoding order until the next AU containing another SDI SEI message whose content is different from the SDI SEI message, or until the end of the bit stream.
[0360] 3. The method of claim 1, wherein the rule specifies that the SDI SEI message persists for the codec video sequence (CVS) including the SDI SEI message.
[0361] 4. The method according to any one of claims 1-3, wherein the rule defines a constraint that the SDI SEI message exists in the first access unit (AU) of the CVS when the SDI SEI message exists in the codec video sequence (CVS).
[0362] 5. The method according to any one of claims 1-4, wherein the rule defines a constraint that all SDI SEI messages in the encoded / decoded video sequence have the same content.
[0363] 6. The method according to any one of claims 1-5, wherein the rule specifies a constraint that, in response to (a) a flag indicating that there is no multi-view information in the bitstream, or (b) a flag indicating that there is no auxiliary information in the bitstream, the value of the identifier of the SDI SEI message is inferred to be zero.
[0364] 7. The method according to any one of the preceding claims, wherein the rule specifies that SDI SEI messages are not allowed to be located within constraints of scalable nested SEI messages.
[0365] 8. A video processing method comprising: performing a conversion between a video and a bitstream of the video; wherein a Multi-View Acquisition Information (MAI) Supplemental Enhancement Information (SEI) message is provided for the video; and wherein rules define the duration range of the MAI SEI message or constraints on the MAI SEI message.
[0366] 9. The method of claim 8, wherein the rule defines the duration of the MAI SEI message in the decoding order from the current access unit (AU) containing the MAI SEI message to the next AU containing another MAI SEI message with different content, or until the end of the bit stream.
[0367] 10. The method according to any one of claims 8-9, wherein the rule defines a constraint that the MAI SEI message exists in the first access unit (AU) of the CVS when the MAI SEI message exists in the codec video sequence (CVS).
[0368] 11. A method for video processing, comprising: performing a conversion between a video and a video bitstream; wherein a Scalability Dimension Information (SDI) Supplemental Enhancement Information (SEI) message and a second SEI message are indicated for the video; and wherein a rule defines the format of the SDI SEI message and the second SEI message.
[0369] 12. The method of claim 11, wherein the rule specifies that the second SEI message is a Multi-View Acquisition Information (MAI) SEI message and the MAI SEI messages appear in the order following the Scalable Dimension Information (SDI) SEI message in the decoding order.
[0370] 13. The method of claim 11, wherein the second SEI message is a Depth Representation Information (DRI) SEI message, and wherein a rule specifies that, in response to the SDI SEI message having a layer identifier value of 2, the SDI SEI message precedes the DRI SEI message in the decoding order.
[0371] 14. The method of claim 11, wherein the second SEI message is an Alpha Channel Information (ACI) Information (DRI) SEI message, and wherein, according to the rule, in response to the SDI SEI message having a layer identifier value of 1, the SDI SEI message precedes the DRI SEI message in the decoding order.
[0372] 15. The method according to any one of claims 1-14, wherein the conversion comprises generating a bitstream from a video or generating a video from a bitstream.
[0373] 16. A video decoding apparatus comprising a processor configured to implement the method of one or more of claims 1 to 15.
[0374] 17. A video encoding apparatus comprising a processor configured to implement the method of one or more of claims 1 to 15.
[0375] 18. A computer program product having computer code stored thereon, which, when executed by a processor, causes the processor to implement the method of any one of claims 1 to 15.
[0376] 19. A computer-readable medium having stored thereon a bit stream generated according to any one of claims 1 to 15.
[0377] 20. A method comprising generating a bitstream according to any one of claims 1 to 15 and writing the bitstream to a computer-readable medium.
[0378] 21. A method, an apparatus, or a bit stream generated according to the disclosed method or system described in this document.
[0379] The following documents may include additional details related to the technology disclosed herein:
[0380] [1]ITU-T and ISO / IEC, “Efficient video coding and decoding”, Rec.ITU-T H.265|ISO / IEC 23008-2 (current version).
[0381] [2] J. Chen, E. Alshina, G. J. Sullivan, J.-R. Ohm, J. Boyce, “Algorithmic description of Joint Exploration Test Model 7 (JEM7)”, JVET-G1001, August 2017.
[0382] [3]Rec.ITU-T H.266|ISO / IEC 23090-3, “Multi-functional video codec”, 2020.
[0383] [4] B. Bross, J. Chen, S. Liu, Y.-K. Wang (eds.), “Multi-functional video codec (draft 10)”, JVET-S2001.
[0384] [5] Rec.ITU-T Rec.H.274|ISO / IEC 23002-7, “Multifunctional supplementary enhancement information messages for encoding and decoding video bitstreams”, 2020.
[0385] [6] J. Boyce, V. Drugeon, G. Sullivan, Y.-K. Wang (eds.), “Multifunctional supplementary enhancement information messages for encoding and decoding video bitstreams (draft 5)”, JVET-S 2007.
[0386] The disclosures and other schemes, examples, embodiments, modules, and functional operations described in this document can be implemented in digital electronic circuits or in computer software, firmware, or hardware, containing the structures disclosed in this document and their equivalents, or combinations thereof. The disclosed and other embodiments can be implemented as one or more computer program products encoded on a computer-readable medium, i.e., one or more computer program instruction modules for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a complex influencing machine-readable propagating signals, or combinations thereof. The term "data processing apparatus" encompasses all means, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or combinations thereof. Propagating signals are artificially generated signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device.
[0387] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language, including compiled or interpreted languages, and can be deployed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on one computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a communications network.
[0388] The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be performed by special-purpose logic circuitry (e.g., field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs)), and the apparatus can be implemented as special-purpose logic circuitry (e.g., FPGAs or ASICs).
[0389] Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors in any type of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magneto-optical, magneto-optical, or optical disc) for storing data, or operatively coupled to receive data from or transfer data to a mass storage device (e.g., magneto-optical, magneto-optical, or optical disc), or both. However, a computer does not necessarily need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and optical disks. Processors and memory may be supplemented by or incorporated into special-purpose logic circuitry.
[0390] While this patent document contains numerous details, these details should not be construed as limiting any subject matter or the scope of the claims, but rather as descriptions of features specific to particular embodiments of a particular art. In this patent document, certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in various suitable sub-combinations. Furthermore, although features may be described above as operating in certain combinations and even initially claimed in the same manner, in certain circumstances one or more features from the claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.
[0391] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order or sequence shown, or to perform all the operations shown, in order to achieve the desired result. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.
[0392] Only a few implementations and examples are described, and other implementations, enhancements and variations can be made based on what is described and shown in this patent document.
Claims
1. A method for processing video data, comprising: For the conversion between the video and the bitstream of the video, whenever any Multi-View Acquisition Information (MAI) Supplemental Enhancement Information (SEI) message exists in any Access Unit (AU) of the CVS of the codec video sequence, it is determined that the first AU of the CVS contains a MAI SEI message. as well as The conversion is performed based on the MAI SEI message; All MAI SEI messages applicable to the CVS have the same content.
2. The method of claim 1, wherein, The MAI SEI message is one of all MAI SEI messages, and the first AU is one of any access unit AU.
3. The method according to claim 1, wherein, The MAI SEI message specifies the internal and external camera parameters.
4. The method according to claim 3, wherein, The internal and external camera parameters are used to process the decoded view before it is rendered on a 3D display.
5. The method of claim 1, wherein, The MAI SEI messages continue from the current AU to the subsequent AU containing subsequent MAI SEI messages in the decoding order.
6. The method of claim 1, wherein, The MAI SEI messages continue from the current AU to the end of the bit stream in the decoding order.
7. The method of claim 1, wherein, When the AU contains both the Scalable Dimension Information (SDI) SEI message and the MAI SEI message, the MAI SEI message follows the SDI SEI message in the decoding order.
8. The method of claim 1, wherein, The MAI SEI message includes internal parameter flags.
9. The method of claim 8, wherein, The internal parameter flag being equal to 1 indicates the presence of internal camera parameters, and the internal parameter flag being equal to 0 indicates the absence of the internal camera parameters.
10. The method of claim 1, wherein, The MAI SEI message includes external parameter flags.
11. The method of claim 10, wherein, The external parameter flag being equal to 1 indicates the presence of an external camera parameter, and the external parameter flag being equal to 0 indicates the absence of the external camera parameter.
12. The method of claim 1, wherein, The MAI SEI message applies to the entire CVS, not just the codec layer video sequence (CLVS).
13. The method of claim 1, wherein, The MAI SEI message is contained within a scalable nested SEI message.
14. The method of any one of claims 1-13, wherein, The conversion includes encoding the video into the bitstream.
15. The method of any one of claims 1-13, wherein, The conversion includes decoding the video from the bitstream.
16. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: For the conversion between the video and the video bitstream, whenever any Multi-View Acquisition Information (MAI) Supplemental Enhancement Information (SEI) message exists in any Access Unit (AU) of the CVS (Catalyst Video Sequence), it is determined that a MAI SEI message exists in the first AU of the CVS; and The conversion is performed based on the MAI SEI message; wherein All MAI SEI messages applicable to the CVS have the same content.
17. The apparatus of claim 16, wherein, The MAI SEI message specifies the internal and external camera parameters; The internal and external camera parameters are used to process the decoded view before it is presented on a three-dimensional (3D) display; Wherein, when the AU contains both the Scalable Dimension Information (SDI) SEI message and the MAI SEI message, the MAI SEI message follows the SDI SEI message in the decoding order; The MAI SEI message includes internal parameter flags; The MAI SEI message includes external parameter flags; The MAI SEI message applies to the entire CVS, not just the codec layer video sequence (CLVS). Wherein, an internal parameter flag of 1 indicates the presence of internal camera parameters, and an internal parameter flag of 0 indicates the absence of internal camera parameters; and Wherein, the external parameter flag equal to 1 indicates the existence of external camera parameters, and the external parameter flag equal to 0 indicates the absence of external camera parameters.
18. The apparatus of claim 16, wherein, The MAI SEI message is one of all the MAI SEI messages.
19. A non-transitory computer-readable storage medium for storing instructions, said instructions causing a processor to: For the conversion between the video and the video bitstream, whenever any Multi-View Acquisition Information (MAI) Supplemental Enhancement Information (SEI) message exists in any Access Unit (AU) of the CVS (Catalyst Video Sequence), it is determined that a MAI SEI message exists in the first AU of the CVS; and The conversion is performed based on the MAI SEI message; wherein All MAI SEI messages applicable to the CVS have the same content.
20. The non-transitory computer-readable storage medium of claim 19, wherein, The MAI SEI message specifies the internal and external camera parameters; The internal and external camera parameters are used to process the decoded view before it is presented on a three-dimensional (3D) display; Wherein, when the AU contains both the Scalable Dimension Information (SDI) SEI message and the MAI SEI message, the MAI SEI message follows the SDI SEI message in the decoding order; The MAI SEI message includes internal parameter flags; The MAI SEI message includes external parameter flags; The MAI SEI message applies to the entire CVS, not just the codec layer video sequence (CLVS). Wherein, an internal parameter flag of 1 indicates the presence of internal camera parameters, and an internal parameter flag of 0 indicates the absence of internal camera parameters; and Wherein, the external parameter flag equal to 1 indicates the existence of external camera parameters, and the external parameter flag equal to 0 indicates the absence of external camera parameters.
21. A non-transitory computer-readable recording medium storing a bitstream of video generated by a method performed by a video processing apparatus, wherein the method comprises: Whenever any Multi-View Acquisition Information (MAI) Supplemental Enhancement Information (SEI) message exists in any Access Unit (AU) of the CVS (Cut-out Video Sequence), it is determined that the first AU of the CVS contains an MAI SEI message. as well as The video bitstream is generated based on the MAI SEI message; All MAI SEI messages applicable to the CVS have the same content. 22.The non-transitory computer-readable recording medium of claim 21, wherein, The MAI SEI message specifies the internal and external camera parameters; The internal and external camera parameters are used to process the decoded view before it is presented on a three-dimensional (3D) display; Wherein, when the AU contains both the Scalable Dimension Information (SDI) SEI message and the MAI SEI message, the MAI SEI message follows the SDI SEI message in the decoding order; The MAI SEI message includes internal parameter flags; The MAI SEI message includes external parameter flags; The MAI SEI message applies to the entire CVS, not just the codec layer video sequence (CLVS). Wherein, an internal parameter flag of 1 indicates the presence of internal camera parameters, and an internal parameter flag of 0 indicates the absence of internal camera parameters; and Wherein, the external parameter flag equal to 1 indicates the existence of external camera parameters, and the external parameter flag equal to 0 indicates the absence of external camera parameters.
23. A method for storing a bitstream of video, comprising: Whenever any Multi-View Acquisition Information (MAI) Supplemental Enhancement Information (SEI) message exists in any Access Unit (AU) of the CVS (Cut-out Video Sequence), it is determined that the first AU of the CVS contains an MAI SEI message. The video bitstream is generated based on the MAI SEI message; as well as The bitstream is stored in a non-transitory computer-readable recording medium; All MAI SEI messages applicable to the CVS have the same content.
Citation Information
Patent Citations
Method and device used for playing video from breaking point
CN102984560A
Image data transmission device, image data transmission method, and image data receiving device
JP2013255207A
Extensions of motion-constrained tile sets SEI message for interactivity
US20150016504A1
