Method, Apparatus, and Article of Manufacture for Generating Packed Video Frames for Volumetric and Immersive Video Bitstreams

Abstract

Methods, apparatus, systems, and articles of manufacture for generating packed video frames are disclosed. A video encoding system disclosed herein includes a configuration determiner that creates a packed video frame layout including regions where video components will be placed. The system also includes a frame generator that forms a packed video frame including video components placed in different regions. The encoding system further includes a frame information generator that generates packed video frame information that identifies characteristics of the packed video frame, including information identifying (i) the identities of regions included in the packed video frame layout, (ii) the types of video components included in the regions, or (iii) the locations and dimensions of the regions. A video encoder of the encoding system encodes the frame and includes the packed video frame information signaling the inclusion of the packed video frame in an encoded bitstream.

Description

[Technical field]

[0001] [Related Applications] This patent claims the benefit of U.S. Patent Application No. 63 / 040369, filed June 17, 2020. U.S. Patent Application No. 63 / 040369 is hereby incorporated by reference in its entirety. Priority to U.S. Patent Application No. 63 / 040369 is hereby claimed.

[0002] The present disclosure relates generally to immersive video, and more particularly, to a method and apparatus for generating data packs for an immersive video bitstream. [Background technology]

[0003] Volumetric video is captured using multiple video processing tools and cameras, and the results are formatted in three degrees of freedom. Thus, the user is provided with a sense and perspective of being immersed in another time / place through the virtual environment depicted by the volumetric video. In addition, the user can move around in the virtual environment, further creating the feeling that the user has been transported to another space or time. Thus, commercial applications using volumetric video, including virtual / augmented / mixed reality applications, are improving rapidly, and the popularity and promise of volumetric video technology has caused a surge in interest in this field. [Brief description of the drawings]

[0004] [Figure 1A] FIG. 2 is a diagram of a first exemplary packing structure in which atlas information may be packed in accordance with the present invention.

[0005] [Figure 1B] FIG. 13 is a diagram of an exemplary second packing configuration in which the atlas information may be packed.

[0006] [Figure 1C]FIG. 13 is a diagram of an exemplary third packing configuration in which atlas information may be packed.

[0007] [Diagram 2] 1 is a block diagram of an exemplary video encoder having an exemplary frame packing unit in accordance with the present invention;

[0008] [Diagram 3] 3 is a block diagram of an example frame packing unit of FIG. 2 according to an example embodiment of the present invention.

[0009] [Figure 4A] 13 is a table depicting example modifications to V3C unit header types to include unit header types representing packed video data.

[0010] [Figure 4B] 11 is pseudocode for identifying V3C unit information to obtain and / or generate when a V3C unit header indicates that the V3C unit contains packed video data.

[0011] [Figure 4C] 11 is pseudocode identifying how a V3C parameter set can be modified to identify when a V3C unit contains packed video and corresponding pack information.

[0012] [Figure 4D] 1 is a table representing a V3C profile definition showing possible values ​​of syntax elements for the MIV main profile and the MIV extended profile.

[0013] [Diagram 5] 4 is pseudocode for implementing a pack information generation unit of the frame packing unit of FIGS. 2 and 3;

[0014] [Figure 6]4 is pseudocode for implementing an example packed information generator of the example frame packing portion of FIGS. 2 and 3.

[0015] [Figure 7] FIG. 2 is a block diagram of an exemplary decoder having an exemplary frame unpacking unit in accordance with the present invention.

[0016] [Figure 8] 8 is a block diagram of a portion of the decoder of FIG. 7 including the example frame unpacking portion of FIG. 7 according to an example embodiment of the present invention.

[0017] [Figure 9] 8 is pseudocode that can be used to implement at least a portion of the example frame unpacker of FIGS. 6 and 7.

[0018] [Figure 10A] 4 is a flow chart illustrating an example program for implementing the example encoder of FIGS. 2 and 3. [Figure 10B] 4 is a flow chart illustrating an example program for implementing the example encoder of FIGS. 2 and 3.

[0019] [Figure 11] 4 is a flowchart illustrating an example program for implementing the frame packer, encoder, and multiplexer of FIGS. 2 and 3.

[0020] [Figure 12] 4 is a flow chart illustrating an example program for implementing a decoder number verifier of the encoder of FIGS. 2 and 3.

[0021] [Figure 13] 9 is pseudocode that can be used to implement the example decoder verifier of FIG. 8.

[0022] [Figure 14]9 is a flow chart illustrating an example program for implementing the decoder of FIGS. 7 and 8.

[0023] [Figure 15] 9 is a flow chart illustrating an example program for implementing the frame unpacking portion of FIGS. 7 and 8.

[0024] [Figure 16] FIG. 13 is a block diagram of an example processing platform configured to execute the instructions of FIGS. 10A, 10B, 11, and 12 to implement an example encoder.

[0025] [Figure 17] FIG. 16 is a block diagram of an example processing platform configured to execute the instructions of FIGS. 14 and 15 to implement an example decoder.

[0026] The figures are not to scale. Generally, the same reference numbers are used throughout the drawings and the accompanying written specification to refer to the same or similar parts. As used herein, connection references (e.g., attached, coupled, connected, and joined) may include intermediate members between the elements referenced by the connection reference and / or relative movement between those elements, unless otherwise indicated. Thus, a connection reference does not necessarily infer that two elements are directly connected and / or in fixed relationship to each other.

[0027] Unless otherwise noted, descriptors such as "first," "second," "third," etc. are used herein without in any way inferring or otherwise indicating any sense of priority, physical order, placement within a list, and / or ordering, but are merely used as labels and / or arbitrary names to distinguish elements to facilitate understanding of the disclosed examples. In some instances, the descriptor "first" may be used to refer to an element in the detailed description, but the same element may be referred to in the claims using a different descriptor, such as "second" or "third." In such instances, it should be understood that such descriptors are merely used to clearly distinguish between elements that may otherwise share the same name, for example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Visual volumetric video refers to a collection of visual volumetric video coding (V3C) frames. The V3C frames are encoded by converting the 3D information of the V3C frames into a collection of 2D images. The 2D images can then be encoded using any video and image coding specification (such as, for example, ISO / IEC 14492-10, ISO / IEC 23008-2, etc.). The V3C data associated with the 2D encoded images can be used to reconstruct the V3C to recreate the 3D video.

[0029] A view captured by a camera can be represented as a base view or an additional view. A base view includes all of the video data corresponding to a view being captured by a camera. In contrast, an additional view includes video data corresponding to one or more patches (portions) of a captured view that differs with respect to a reprojection of another view onto the position of the base view. Therefore, instead of retransmitting in its entirety all of the information necessary to reconstruct each view after one or more base views have been obtained, only information about the portions (patches) of the base view that have changed between time points is transmitted.

[0030] The portions of a view that are transmitted are called patches and are contained / represented in an atlas. Within an atlas or attratile, a patch represents a component at a particular location in a view. An atlas contains one or more attratiles. Components are various types of video data and can include data related to geometry, color, reflectance, surface information, texture, occupancy, materials, objects, etc. An access unit may contain one or more atlases corresponding to multiple views captured on multiple cameras at the same point in time, resulting in an access unit containing a collection of atlases that all correspond to the same point in time.

[0031] In many cases, to achieve a high-quality immersive experience for users, V3C videos are captured using many cameras, resulting in a large amount of video data. Furthermore, in some cases, attributes (and other types of information) associated with V3C videos and / or MPEG immersive video data will be explicitly signaled in separate video sub-bitstreams. For example, an MPEG immersive (MIV) video access unit refers to all sub-bitstream configuration units that share the same decoding order count. The sub-bitstream configuration units may include sub-bitstream configuration units containing occupancy information, sub-bitstream configuration units containing attributes, sub-bitstream configuration units containing geometry information, etc. Therefore, multiple sub-bitstreams, each requiring a separate decoder, may be included in the coded visual volumetric video transmission.

[0032] Due to the number of bitstreams and the large amount of video data, multiple decoder instantiations are typically required to reconstruct the video and synchronize the operation of the decoder instantiations. While high-end devices such as personal computers generally have enough decoder instantiations to synchronously handle various bitstreams of visual volumetric video data, mobile electronic devices typically do not have the capability to synchronize multiple video decoder instantiations. As a result, mobile electronic devices are often unable to provide users with a high-quality immersive video experience.

[0033] The systems, methods, apparatus, and articles of manufacture disclosed herein use a frame packing technique before encoding V3C video data. Using the frame packing technique disclosed herein results in a smaller bitstream and therefore less decoder instantiation is required. The frame packing technique results in one or more packed video frames that contain information to be used in a decoder to reconstruct a three-dimensional image from raw video data. As used herein, a "packed video frame" refers to a frame-packed video layout that includes multiple regions, individual regions. Each region includes a type of video component data (also referred to herein as a "video data type" or "type of video data") associated with an atlas or atlas style (e.g., geometry, occupancy, attributes, etc.). Each region can represent a geometry, occupancy, or attribute video sub-bitstream of an atlas or atlas style. Each region represents data that would otherwise be carried by a video sub-bitstream that includes only one type of component data (e.g., geometry, occupancy, attributes, etc.). Furthermore, in some examples, the packed video frame need not be completely occupied by the region, allowing for padding of unoccupied space if necessary (e.g., to make the encoded output data more friendly to a video codec). As used herein, the terms rect, rectangle, reg, and region are used interchangeably. As described herein, a region can take a rectangular geometric shape, but a region can instead take any shape and is in no way limited to a rectangular shape. As used herein, some of the variables include "rect," and in such examples, "rect" is used to refer to a region (also abbreviated as "reg").

[0034] The frame packing techniques disclosed herein use packed frame video information to which the encoded video sub-bitstreams of volumetric video data are supplied. The packed frame video information is used in the decoder to identify regions contained within the packed frame, information about the location and dimensions of the regions, information about the type of video components contained within the regions, mapping information by which the regions can be mapped to a sequence of video frames to be rendered, etc. The generation and provision of this information to the encoded volumetric video data allows the generation of packed frames at the encoder and subsequent unpacking at the decoder. Furthermore, in some examples, independently decodable regions can be identified and supplemental enhancement information (SEI) messages can be generated to signal the presence of such regions and provide information about where to locate them. In such examples, multiple independently decodable regions can be decoded in parallel by multiple decoders.

[0035] The frame packing techniques disclosed herein reduce the number of decoder instantiations required to decode the encoded video because the information contained in the atlas (which would normally be carried in multiple sub-bitstreams each representing a component type) can be inserted into a single packed video frame that can be decoded using a single decoder, thereby lowering the number of bitstreams required to carry the encoded video signal. Even when a frame packing technique results in the creation of more than one packed video frame, the number of packed video frames created and the corresponding number of decoders required to decode the packed video frames are less than the number of decoders that would be required if the encoded video were not packed video frames, as described further below.

[0036] FIG. 1A is a diagram of an exemplary first packed video frame structure 100A into which atlas information can be packed in accordance with the present invention. In the example of FIG. 1A, metadata regarding texture, geometry and occupancy data of different atlases are included in regions that are inserted into a set of packed video frames (e.g., pack 0A, pack 1A, pack 2A, pack 3A, pack 4A and pack 5A). In the example of FIG. 1A, an initial region "r0" 102A of pack 1A is populated with texture data of a first atlas "a0", a second region "r1" 104A of pack 0A is populated with geometry data of atlas a0, and a portion 106A of pack 0 is unpopulated. Atlas a0 is assumed to represent the base view since the base view does not contain occupancy data.

[0037] In the example of FIG. 1A, a packed video frame ("Pack 1A") includes data from atlases a0 and a1 contained within three regions (e.g., region r0 108A, region r1 110A, and region r2 112A). Region r0 of Pack 1A includes texture data from atlas "a1". Region r1 110A of Pack 0A includes geometry data from atlas a0. In some examples, region r2 112 of Pack 1A includes occupying data from atlas a1. Additionally, region r2 occupies a portion 114A of an otherwise empty region.

[0038] In the example of FIG. 1A, region "r0" 116A of pack 2A is populated with texture data for atlas a2, region "r1" 118A is populated with geometry data for atlas a2, and portion 120A of pack 2A is unpopulated.

[0039] 1A, region r0 122A of pack 3A is populated with texture data for atlas "a3," region r1 124A is populated with geometry data for atlas a3, and region r2 126A is populated with occupancy data for atlas a3, while portion 128A (where region r2 is located) is otherwise unpopulated.

[0040] In the example of FIG. 1A, region r0 130A of pack 4 is populated with texture data for atlas "a4," region r1 132A is populated with geometry data for atlas a4, and portion 134A of pack 4 is unpopulated.

[0041] 1A, region r1 136A of pack 5A is populated with texture data for atlas "a5," region r1 138A is populated with geometry data for atlas a5, and region r2 140A is populated with occupancy data for atlas a5, while portion 142A including region r2 140A is otherwise unpopulated.

[0042] Thus, the frame pack configuration 100A of FIG. 1A shows a sequence of six packed video frames of data from six different atlases (Pack 0A, Pack 1A, Pack 2A, Pack 3A, Pack 4A, and Pack 5A). As shown, data associated with six atlases is packed, three of the atlases are associated with two types of data (texture and geometry), and three of the atlases are associated with three types of data (texture, geometry, and occupancy). If the data represented in the six packed video frames of FIG. 1A were transmitted using conventional techniques, 15 (e.g., (3*2)+(3*3)=15) sub-bitstreams would be required to carry the data, and 15 different decoders would be required to decode the data. By using the configuration 100A of FIG. 1A, only six bitstreams (one for each packed video frame) are required to carry the data, and therefore only six decoders are required to decode the six bitstreams. As described above, the first packed video frame configuration 100A illustrates packing of different video components of the same atlas within the same packed video frame of a sequence of packed video frames including six packed video frames (Pack 0, Pack 1, Pack 2, Pack 3, Pack 4, Pack 5).

[0043] As shown in FIG. 1A, each packed video frame is formed from a set of blocks having a specific location in a video pack as shown in pack 2. In some examples, the information of each respective block is ultimately mapped to a volumetric video sequence. Examples of such mappings are described below. Blocks are associated with each region, but are only shown in pack 2 for clarity.

[0044] 1B is a diagram of an example second packed video frame structure 100B that can pack information / data from multiple atlases. The second packed video frame structure includes three packed video frames (e.g., packed video frame pack 0B, packed video frame pack 1B, and packed video frame pack 2B). In the second packed video frame structure 100B, the first packed video frame pack 0B includes six regions (region r0 102B, region r1 104B, region r2 106B, region r3 108B, region r4 110B, and region r5 112B). In the example second configuration of FIG. 1B, all six of the regions of Pack 0B contain texture data, and the texture data contained within each region is associated with a different attra tile (e.g., attra tile a0, attra tile a1, attra tile a2, attra tile a3, attra tile a4, and attra tile a5, respectively).

[0045] The second packed video frame 1B also includes six regions: region r0 114B, region r1 116B, region r2 118B, region r3 120B, region r4 122B, and region r5 124B. All six regions include geometry data, and the geometry data included within each region of Pack 1B is associated with a different attra tile (e.g., attra tile a0, attra tile a1, attra tile a2, attra tile a3, attra tile a4, and attra tile a5, respectively). The third packed video frame, Pack 2B, includes three regions: region r0 126B, region r1 128B, and region r3 130B, each of which includes occupancy data for a different attra tile (attra tile a0, attra tile a3, and attra tile a5, respectively). The second frame-packed video frame structure 100B therefore includes atlas data for six atlas styles, three of which include geometry and texture data, and three of which include geometry, texture and occupancy data.

[0046] 1C illustrates an example third packed video frame structure 100C that includes pack 0C and pack 1C. In the third packed video frame structure 100C, pack 0C includes six regions (reg r0 102C, reg r1 104C, reg r2 106C, reg r3 108C, reg r4 110C, reg r5 112C). reg r0 102C, reg r1 104C, and reg r2 106C each occupy one-quarter of the area of ​​pack 0C. reg r0 102C, reg r1 104C, and reg r2 106C each include texture data corresponding to atla tile a0, atla tile a2, and atla tile a4, respectively. Pack 0C also contains three regions (reg r3 108C, r4 110C, r5 114C) that each occupy 1 / 16th of the area of ​​Pack 0C. Each of the three regions (reg r3 108C, r4 110C, r5 114C) contains geometry data from atla style a0, atla style a2, and atla style a4, respectively. Therefore, Pack 0 contains texture and geometry data for atla style a0, atla style a2, and atla style a4, each of which represents a base view. One-sixteenth of Pack 0C does not contain any data.

[0047] Pack 1C in FIG. 1C includes three regions (reg r0 116C, reg r1 118C, and reg r2 120C) that each occupy a quarter of the area of ​​Pack 1C. Each of the three regions (reg r0 116C, reg r1 118C, and reg r2 120C) contains texture data corresponding to atla tile a1, atla tile a3, and atla tile a5, respectively. Pack 1C also includes three regions (reg r3 122C, reg r4 124C, and reg r5 126C) that each occupy a sixteenth of the area of ​​Pack 1C. Each of the three regions (reg r3 122C, reg r4 124C, and reg r5 126C) contains geometry data from atla style a1, atla style a3, and atla style a5, respectively. In addition, pack 1C contains three regions (reg r6 126C, reg r7 128C, and reg r8 130C). reg r6 and reg r8 each occupy 1 / 32 of the area of ​​pack 1C, and reg r7 occupies an area less than 1 / 32 of the area of ​​pack 1C. Each of the three regions (reg r6 126C, reg r7 128C, and reg r8 130C) contains occupancy data from atla style a1, atla style a3, and atla style a5, respectively. Thus, Pack 1C in FIG. 1C includes texture and geometry data for the atlases (atlas a0, atlas a2, and atlas a4), and also includes occupancy data for the atlases (atlas a1, atlas a3, and atlas a5). Pack 1C further includes a portion 134C that is devoid of data. Note that individual components packed together within the same packed video frame may be coded at different frame rates (e.g., a packed video frame of a texture atlas may be transmitted at 60 fps, while a packed video frame of a geometry atlas may be transmitted at 30 fps to achieve additional compression or to support desired features depending on the application).Although the example configurations of Figures 1A, 1B, and 1C are assumed to be based on atlases or atlas tiles of the same size, the atlases or atlas tiles may instead be of different sizes.

[0048] An encoded video bitstream includes an encoded representation of video data. More specifically, the bitstream includes a series of Network Abstraction Layer (NAL) units. Each of the NAL units includes a NAL unit header and a Raw Byte Sequence Payload (RBSP). Information also included in the bitstream can be used to identify the end of the RBSP. In some examples, a unit can include more than one packed video frame (e.g., a sequence of packed video frames).

[0049] For purposes of illustration only, the regions in Figures 1A, 1B and 1C are rectangular. In practice, the regions can be any shape, provided that the regions do not overlap.

[0050] Reference is now made to Figure 2, which is a block diagram of a simplified encoding system 200 according to the teachings of this disclosure. The simplified encoding system 200 includes an example captured video data and metadata storage 202, an example video and metadata pre-processor 204, an example parameter / atlas data provider 206, an example geometry video data provider 208, an example attribute video data provider 210, an example occupation video data provider 212, an example bitstream formatter 216, an example frame packer 218, an example encoder 220, and an example multiplexer 222.

[0051] In some examples, the captured video data and metadata are stored in an example captured video data and metadata storage 202. An example video metadata pre-processor 204 inspects the video data and metadata and separates the metadata (e.g., parameters and atlas data) from different types of video data (e.g., geometry, attributes, occupancy, etc.) contained within the video data. Additionally, the video and metadata pre-processor 204 separates the different types of video data (geometry, attributes, and occupancy data from each other) and provides each of them to a geometry video feed 208, an attribute video data feed 210, and an occupancy video feed 212, respectively. In some examples, the parameters and atlas data are provided to a parameter and atlas data feed 206. The geometry video data provider 208, the attribute video data provider 210 and the occupation video provider 212 respectively provide geometry video data, attribute video data and occupation video data to an exemplary frame packing unit 218, and the parameter / atlas data provider 206 provides parameter / atlas data to an exemplary bitstream formatter 216. Video data types are also referred to herein as video component types.

[0052] In some examples, the geometry video data, attribute video data, and occupation video data are provided to an example packed video frame packing unit 218. Additionally, the parameter / atlas data is provided to an example bitstream formatting unit 216. In some examples, the bitstream formatting unit 216 and the packed video frame packing unit 218 share information needed to determine a packed video frame configuration (e.g., the packed video frame configuration of FIG. 1A, the packed video frame configuration of FIG. 1B, and / or the packed video frame configuration of FIG. 1C), including the geometry, attribute, and occupation data provided to the packed video frame packing unit 218. In some examples, the packed video frame packing unit 218 provides the packed video frames to an example encoder 220 for encoding using any suitable encoding method. In addition to generating packed video frames, the packed video frame packing unit 218 also generates packed video frame information for use by a decoder in determining where regions of video contained within the packed video frames are to be located for the purpose of generating a volumetric video sequence for rendering.

[0053] The example bitstream formatter 216 determines a bitstream format for the parameter / atlas data 206 corresponding to the video data included in the packed video frames generated by the frame packer 218. In some examples, the parameter / atlas data 206 further includes an SEI message. In some examples, the SEI message information is provided to the bitstream formatter 216 by the packed video frame packer 218. In some examples, the bitstream formatter 216 provides the bitstream format information, and the encoder 220 provides the encoded packed video frames and the packed video frame information to an example multiplexer 222, which operates to multiplex the bitstream format information with the encoded packed video frames. In some examples, the SEI message identifies one or more regions of one or more packed video frames that may be decoded independently from other regions. In some examples, such regions may be decoded in parallel by respective decoders.

[0054] Figure 3 is a block diagram of an example frame packing unit 218 of Figure 2. In some examples, the frame packing unit 218 of Figure 3 includes an example profile / tier / level storage 302, an example packed video frame flag setter 304, an example packed video frame pack configuration determiner 306, an example atlas / atlas style identifier 307, an example geometry video collector 308, an example attribute video reg collector generator 310, an example occupancy video collector generator 312, an example packed video frame generator 314, an example packed video frame information generator 316, an example media player 318, an example 3D video player 319, an example 3D video player 320, an example 3D video player 321, an example 3D video player 322, an example 3D video player 323, an example 3D video player 324, an example 3D video player 325, an example 3D video player 326, an example 3D video player 327, an example 3D video player 328, an example 3D video player 329, an example 3D video player 330, an example 3D video player 331, an example 3D video player 332, an example 3D video player 333, an example 3D video player 334, an example 3D video player 335, an example 3D video player 336, an example 3D video player 337, an example 3D video player 338, an example 3D video player 339, an example 3D video player 340, an example 3D video player 341, an example 3D video player 342, an example 3D video player 343, an example 3D video player 344, an example 3D video player 345, an example 3D video player 346, an example 3D video player 347 The packed video frame frame storage 318 includes an exemplary packed video frame storage 322, an exemplary metadata provider 324, an exemplary packed video frame provider 326, an exemplary decoder number verifier 328, an exemplary evaluation unit 330, an exemplary packed video frame flag storage 332, an exemplary packed video frame flag setting unit 333, an exemplary packed video frame flag adder 334, an exemplary packed video frame checker 336, an exemplary reg subtracter 338, and an exemplary comparator 340.

[0055] In some examples, the encoder 220 operates according to a video coding standard having a syntax that defines a profile field, a hierarchical field, and a level field. In some examples, a "profile" of a video coding standard defines which coding tools may be used. For example, for the encoding system 200, a profile may specify a set of coding tools that the encoding system 200 may use to generate a coded bitstream. Similarly, an encoder profile may specify a type of video decoder that may decode video data generated by the video encoder, including a set of coding tools that the video decoder must have in order to be able to decode a bitstream associated with the profile. In some examples, a level is a defined set of constraints on values ​​that may be taken by syntax elements and variables of a video coding standard. A hierarchical level is a specified category of level constraints that are imposed on values ​​of syntax elements or variables in a bitstream. Level constraints are nested within a hierarchy, such that a decoder that complies with a particular hierarchy and level is able to decode all bitstreams that comply with the same hierarchy or lower hierarchical levels at that level or any level below that level. Thus, a level of a hierarchy is a specified set of constraints imposed on the values ​​of syntax elements in a bitstream or variables used in decoding the bitstream. In some examples, the profile / tier / level information specifies a constraint on the number of decoders required to decode the bitstream at the decoder. In some examples, the constraint identifies the maximum number of encoders that can be used to decode the bitstream, since if the decoder had fewer decoders, it would be impossible to decode the bitstream. In some examples, the encoding system can use the profile / tier / level information to identify the number of decoders required and ensure that the encoding performed in the encoding system does not violate the constraint on the number of decoders.In some examples, the encoding system can determine the number of decoders that will be required to decode a bitstream based on characteristics of the information that needs to be encoded to form a properly formatted bitstream with the corresponding video component.

[0056] In some examples, as described further below, to limit the number of decoder instantiations required to decode an encoded bitstream, the profile / tier / level syntax structure is modified to include a set of bits defined to indicate a maximum number of such decoder instantiations. In some examples, the profile / tier / level syntax may include, for example, a set of bits (e.g., 4) represented by the variable "ptl_max_decodes_idc" and used to indicate a constraint on the number of sub-bitstreams that the coded video sequence requires to conform to a video decoder instantiation. The bitstream includes a value of "ptl_max_decodes_idc" as specified below. For example, the four bits of the variable "ptl_profile_codec_group_idc" may indicate one of the values ​​0, 1, 2, 3, 4, 5, 6, 7, 8, 9, each of which may correspond to a different maximum number of decoder instantiations (e.g., 1, 2, 3, 4, 6, 8, 12, 16, 24, 32, respectively). Values ​​10-14 may be reserved for future use, and value 15 may be unconstrained. Some devices (decoders) may support multiple simultaneous decodings, while others may not have such capabilities. As a result, the same content may be encoded according to multiple profiles such that the output bitstream can be decoded by decoding devices with different decoding capabilities. Therefore, the addition of a profile / tier / level corresponding to a maximum number of decodings allows the use of multiple profiles to support multiple encodings of the same bitstream such that the bitstream can be decoded by decoding devices with different decoding capabilities. In some examples, example profiles / tiers / levels supported by encoding system 200 are stored in example profile / tier / level storage 302.As described above, in some examples, the encoding system 200 may be capable of encoding multiple video bitstreams, each corresponding to a different one of multiple profiles, each having a different maximum number of decoders (represented by the variable “max_decode_idc”) to support decoding devices having different levels of decoding capabilities.

[0057] In some examples, the information stored in the profile / layer / level signal storage is set by a manufacturer of the encoding system 200. In some examples, the information stored in the profile / layer / level signal storage can be populated by an operator of the encoding system 200 and / or can be reconfigured / revised by such an operator. In some examples, the information stored in the profile / layer / level signal storage can be received from another of the encoder blocks included in FIG. 2. In some examples, the packed frame configuration used by the frame packing unit 218 to form a packed video frame of a region is at least partially governed by the information stored in the profile / layer / level signal storage, since each packed video frame included in the configuration of the packed video frame requires a decoder instantiation at the decoding side. Therefore, the profile / layer / level information by specifying the maximum number of decoder instantiations can be a consideration used when generating a packed video frame.

[0058] In some examples, when a network abstraction layer unit (e.g., a NAL unit) of a video is to be packed, the packed video frame flag setter 304 sets a packed frame video flag. In some examples, setting the packed frame video flag includes setting one or more bits in the metadata storage 318. In some examples, based on information in the profile / tier / level storage 302 or based on information provided by any of the other blocks of FIG. 2, the packed video frame configuration determiner 306 determines a packed video frame configuration that identifies regions of the packed video frame, dimensions of the regions, locations of the regions, types of data included in each of the regions, etc. Exemplary packed video frame configurations are shown in FIG. 1A, FIG. 1B, and / or FIG. 1C. In some examples, any of a variety of packed frame video configurations can be determined based on aspects / characteristics of the video stream to be packed into the packed video frame. For example, the determined packed video frame configuration may depend on the number of available views and the typical number of decoder instantiations of a product for the use case.

[0059] As described above, in some examples, the packed video frame configuration includes information identifying the number of regions to be included in the packed video frame, the dimensions (e.g., width, height) of each of the regions, the type of data to be included in the regions, etc. In some examples, the packed video frame configuration uses the atlas or atlas tile identifier 307 to determine the number of atlases or atlas tiles associated with the incoming geometry, attribute and occupancy video data to be processed. In some examples, the incoming geometry data, attribute data and occupancy data each include information identifying the atlas or atlas tile to which the information corresponds. In some examples, the atlas tile identifier 307 examines the incoming geometry, attribute and occupancy data to determine the atlas or atlas tile to which each of the types of data corresponds. In some examples, all geometry, attribute and occupancy data associated with a first atlas tile are received at the packed video frame packer 218 at the same time (or within the same period of time). In some such examples, the atlas identifier 307 may tag or otherwise associate the incoming geometry, attribute and occupancy data with an atlas style number / type (e.g., the atlas style identifier 307, information that identifies the atlas style as being associated with a base view or an additional view, etc.).

[0060] In some examples, the example packed video frame generator 314 uses the packed video frame information from the packed video frame information generator 316 to generate a packed video frame having separate regions that include geometry data provided by the example geometry data collector 308, attribute data provided by the example attribute data collector 310, and occupancy data (if present) provided by the example occupancy data collector 312. In some examples, each region has a dimension (determined by the packed video frame configuration determiner 306) and / or an amount of geometry data, attribute data, and occupancy data that is collected. Note that some atlases may not have occupancy data, such that occupancy regions do not always exist for all geometry and attribute regions present.

[0061] In some examples, the packed video frame generator 314 uses the determined packed video frame configuration and / or the packed video frame information generator 316 to ensure that sufficient packed video frame information exists for a decoder to locate a region of the corresponding packed video frame. In some examples, the packed video frame generator 314 stores the packed video frames in an example packed video frame storage 322. In addition, the packed video frame generator 314 includes the packed video frame information as metadata to be carried with the corresponding packed video frame or as carried within metadata associated with the corresponding packed video frame. In some examples, the packed video frame information metadata can include dimensional information regarding the size of the regions, the location of the regions within a sequence of volumetric video frames, atlas information identifying the atlas or atlas tile to which each of the regions corresponds, and any other information required by a decoder to unpack the packed video frame of video data such that the unpacked data can be used to reconstruct a three-dimensional image / view. In some examples, the example packed video frame information generator 316 stores the packed video frame information in the metadata storage 318, and the metadata provider 324 provides the packed video frame information to the packed video frame storage for inclusion with the corresponding packed video frame. In some examples, the metadata information and the corresponding video are not stored in the packed video frame storage, but are provided directly to the example encoder 220 for encoding. In some examples, the packed video frames and the corresponding packed video frames are provided to an example packed video frame provider, which provides them to the example encoder 220.In some examples, the metadata provider 324 may provide information corresponding to an atlas or atlas style of the packed video frame (or multiple packed video frames) to the example bitstream formatter of Figure 2. In some examples, the bitstream formatter of Figure 2 formats parameter data and / or SEI messages associated with the corresponding packed video frame. In some examples, the example packed video frame provider 326 provides the packed video frame (obtained from the packed video frame storage 322) to the example encoder 220 of the example encoding system 200 of Figure 2 where the packed video frame is encoded.

[0062] In addition, the example multiplexer 222 multiplexes the bitstream of metadata / parameter / SEI message data with the corresponding encoded data for transport to the decoder for decoding and rendering. Note that in some examples, the packed video frame generator 314 does not generate the packed video frames itself, but rather correlates the packed video information 314 with the incoming collected video data so that the video data can be decoded by use of the packed video information. In some examples, multiple packed video frames are generated simultaneously (e.g., correlated to corresponding packed video frame information) such that multiple packed video frames are generated at the same time (or within the same period of each other). Thus, the packed video frame includes packed video frame information that signals (to a receiving decoder) the presence of the packed video frame and further signals how the volumetric video data included in the packed video frame is to be arranged for inclusion in a reconstructed volumetric video view.

[0063] As briefly described above, in some examples, the encoding system 200 includes features that specify regions of the packed video frame that can be decoded independently. In some examples, the encoding system aligns these regions with independently coded regions of the video codec, such as tiles or subpictures. In some such examples, the bitstream formatter creates an SEI message to signal the correspondence with the independently decodable regions of the packed video frame. In some such examples, the SEI message signals tile indexes for the top-left and bottom-right tiles within a rectangular region of tile rows and columns, such as those used in the temporal motion constrained tile sets of the HEVC encoding standard. In some examples, the SEI message signals subpicture IDs, such as those used in the V3C encoding standard.

[0064] The above enables use cases where a media aware network element (MANE) or MIV decoder can discard parts of a video bitstream while allowing the remaining bitstream to be decoded. An encoder may choose to select different quantization parameter (QP) values ​​for different regions within the same packed video frame.

[0065] FIG. 4A illustrates a set of V3C unit header types, and FIG. 4B illustrates example information associated with a packed video frame unit of data. In some examples, a video coding standard includes information to identify the type of data included within a packed video frame unit. For example, the V3C standard includes the following unit types: V3C parameter set data, atlas data, occupied video data, attribute video data, and packed video frame data. In some examples, a unit of data associated with or included in the packed video frame data is identified using the header "V3C_PVD" shown in FIG. 4A. In some examples, when the encoded data is identified using a V3C_PVD unit header, the metadata associated with the packed video frame data unit header includes a set of bits (e.g., 4 bits) represented by the variable "vuh_v3c_parameter_set_id" that identifies the location of a parameter set associated with the packed video frame data unit. (See FIG. 4B). In addition, the encoded data identified using the V3C_PVD unit header is associated with metadata including a set of bits (e.g., 4 bits) represented by a variable "vuh_pack_index". "vuh_pack_index" identifies a number assigned to a packed video frame associated with the packed video frame data unit. Thus, in some examples, the bitstream formatting portion of FIG. 2 is configured to include a "vuh_v3c_parameter_set_id" bit and a "vuh_pack_index" bit in the metadata associated with the packed video frame data unit. In some examples, each packed video frame sub-bitstream is associated with a delimiter value of "vuh_pack_index" and is associated with a codec specified by the variable "pi_pack_codec_id".

[0066] 4C illustrates V3C parameter set data and a set of bits that are set or not set depending on whether various types of data are present in the bitstream to be decoded. In some examples, to accommodate the processing of packed video frame data units, the V3C parameter set is modified to include a "vps_pack_video_enabled_flag" 402C. If the flag is set as determined by pseudocode line 404C, the bitstream to be decoded (e.g., the corresponding video packed video frame data unit) includes packed video frames generated by the frame packer 218 of FIG. 2. As described above, the packed video frame flag setter 304 sets the flag by generating (or inverting) a bit that is included in the metadata to be provided to the example bitstream formatter. 8, in some examples, if vps_pack_video_enabled_flag indicates that the corresponding bitstream contains packed video frame data, a decoder decoding the bitstream obtains packed video frame information that includes information needed to unpack the packed video frames (pseudocode line 406C). In some examples, the information included in the packed video frame information is generated, at least in part, by pack information generator 316, as described below.

[0067] Referring to FIG. 4D, FIG. 4D illustrates how to modify the MIV extended profile to account for the presence of packed video frames. In some examples, the "vuh unit type" of the MIV extended profile is modified to include a fifth bit, which, when set, indicates that packed frame video is included in the corresponding video bitstream. In some examples, the syntax value of the profile toolset idc (of the MIV extended profile) is modified to be equal to 65 if packed video frames are included in the video stream being processed. In addition, a vme embedded occupancy variable is added, which may take a value of 0 or 1, indicating whether the corresponding bitstream includes embedded occupancy data, where occupancy information is embedded in the geometry component. A vps occupied video present flag is added, which may take a value of 0 or 1, indicating whether the corresponding bitstream includes occupancy video data. A vps packing enabled flag is also added, which may take a value of 0 or 1, indicating whether the corresponding bitstream includes packed video frames.

[0068] 3, in some examples, the packed video frame information generated by the example packed video information generator 306 includes a set of bits (e.g., 4) that provide an identifier of a set of V3C parameter set data corresponding to the packed video frame. In some examples, the packed video frame information generator 316 generates a unique packed video frame index number for each packed video frame generated and provides the packed video frame index number to a metadata provider 324 for delivery to the example bitstream formatter 216. In some examples, the information is instead converted to metadata for inclusion within the corresponding encoded packed video frame.

[0069] Still referring to FIG. 3, the example decoder validator 328 stores flag information in the example flag storage 332, the flag information including bits corresponding to a set of flags. The flags are identified in a parameter set associated with the packed video frame unit. Each flag included in the set of flags corresponds to a different type of video data (e.g., geometry, auxiliary, attribute, occupancy, etc.), and each flag indicates whether the corresponding type of video data is present in the corresponding packed video frame unit. In some examples, the attribute data may include subcategories of attribute data, and the number of subcategories included in the attribute video data is represented using the variable "ai_attribute_count."

[0070] In some examples, the number of decodings verifier 328 also includes an example flag adder 334 that sums the number of flags associated with each atlas / atlas style of the packed video frame unit to determine a total number of types of video data included in the packed video frame unit. The sum of the flags is the number of decodings that can be decoded by a decoder receiving the encoded video, and is equal to the variable "NumDecodes". Once the total number of flags is summed, an example packed video frame checker 336 checks whether the packed video frame video flag indicates that the video unit includes a packed video frame unit. Provided that the packed video frame video flag indicates that the video unit includes a packed video frame, for each packed video frame, an example region subtracter 338 subtracts from the "NumDecodes" value the number of regions included in each packed video frame minus 1. The number of regions included in each packed video frame is represented by the variable "pi_num_rect[k]-1", where the variable "k" represents the number of packed video frames included in the total number of packed video frames. The example comparator 340 then compares the value of NumDecodes to a value corresponding to the maximum number of decodes that the receiving decoder is capable of decoding. The maximum number of decodes is represented by the variable "MaxDecodes". In some examples, the frame packer 218 is pre-programmed with information identifying corresponding MaxDecodes values ​​for various types of decoders to which the video stream may be transmitted. In some examples, the maximum number of decodes is included in the profile / tier / level syntax information.

[0071] If the example comparator 340 determines that the value of NumDecodes is less than or equal to the value of MaxDecodes, the frame packer 218 may proceed to generate a packed video frame. If the comparator 340 determines that the value of NumDecodes is not less than or equal to the value of MaxDecodes, the example encoder 220 does not encode the corresponding video data. Therefore, the example decoder number verifier 328 determines whether the number of decoders required to decode the bitstream being encoded will exceed the constraint MaxDecodes. In some examples, the decoder number verifier 328 is included within the decoding system to ensure that the decoding system has a sufficient number of decoders. In some examples, the number of decoders required is provided in the parameter data transmitted by the encoding system of FIG. 2, and the decoding system of FIG. 7 and FIG. 8 simply compares the required number with the available number and does not decode the incoming bitstream if the available number is insufficient.

[0072] 5 is pseudocode that may be used by the packed video frame information generator 316 to generate packed video frame information for a set of packed video frames. In some examples, the packed video frame information generated by the packed video frame information generator 316 (for use by a decoder) specifies the number of V3C_PVD units for which region information is signaled, represented by the variable "pi_num_packs". The packed video frame information also specifies, for each packed video frame, an identifier of the codec (represented as the encoding system 200 in FIG. 2) used to compress the corresponding packed video frame, represented by the variable "pi_pack_codec_id[k]", where "k" is an index that references the number of packed video frames. The value of "pi_pack_codec_id" ranges from 0 to 255, inclusive. In some examples, the codec identifier is transmitted to the decoder in a supplemental enhancement (SEI) message associated with the packed video frame (eg, via a component codec mapping SEI message) or via any other means.

[0073] In some examples, the packed video frame information also indicates a nominal 2D bit depth to which the packed video frame for the kth packed video frame is converted, represented by a variable "pi_pack_nominal_2d_bitdepth_minus1[k] plus 1". In some examples, the value of "pi_pack_nominal_2d_bitdepth_minus1[k]" ranges from 0 to 31, inclusive. The nominal 2d bit depth represents the number of bits used to represent the component information. Thus, for example, geometry data of a packed video frame may be represented by a number of bits "A" and attribute data corresponding to the same packed video frame may be represented by a number of bits "B". In some such examples, the packed video frame information generator 316 may determine that all components of the packed video frame will be represented by the same number of bits (e.g., the greater of the A and B values) and may pad all empty bit positions.

[0074] The packed video frame information may also indicate the width and height, respectively, of each of the packed video frames. The width and height may be represented by the variables "pi_pack_width[k]" and "pi_pack_height[k]", respectively. The packed video frame information also specifies the number of regions contained within the kth packed video frame, represented by the variable pi_num_rect[k]. The packed video frame information also specifies the vertical (x) and horizontal (y) positions of the top left corner of each region contained within each packed video frame in units of "AtlasPatchPackingBlockSize" samples. The variable "AtlasPatchPackingBlockSize" is a unit representing the size of the blocks contained within the patch of the atlas currently being processed. The size of all patches, atlases, and atlas tiles is an integer multiple of AtlasPatchPackingBlockSize, so the sizes can be more efficiently expressed in those units rather than in single pixels. In some examples, the patches corresponding to the atlas are configured to be stored in blocks of regions that are activated on. The vertical and horizontal positions can be represented using the variables "pi_rect_pos_x[k][i]" and "pi_rect_pos_y[k][i]".

[0075] The packed video frame information also specifies the width and height, respectively, of each packed video frame's region in units of "AtlasPatchPackingBlockSize" samples. The width and height of the region are represented by the variables "pi_rect_width[k][i]" and "pi_rect_height[k][i]", respectively. The following conditions can be applied to restrict the location and size of the region so that it fits within the size of the packed video frame: the value of "pi_rect_pos_x[k][i]" + "pi_rect_width[k][i]" shall be less than or equal to pi_pack_width[k], and the value of pi_rect_pos_y[k][i] + pi_rect_height[k][i] shall be less than or equal to pi_pack_height[k].

[0076] Similarly, to ensure that no regions overlap, the frame packed video frame 218 may be equipped with an evaluator 330 to ensure that for all values ​​of "m" and "n", there exists at most one value of "i" (representing an index number for the packed video frame) that satisfies the following statement to be evaluated: [(pi_rect_pos_x[k][i]<=m <pi_rect_pos_x[k][i]+pi_rect_width[k][i])]&&[(pi_rect_pos_y[k][i]<=n<pi_rect_pos_y[k][i]+pi_rect_height[k][i])]。

[0077] The packed video frame information also identifies the rotation, if any, of the ith region of the kth packed video frame. The rotation may be represented by the variable "pi_rect_orientation[k][i]." In some examples, the packed video frame generator 314 may rotate one or more regions when including them in a packed video frame to result in a packed video frame having a smaller size than would be achieved without the rotation.

[0078] The packed video frame information may also include an atlas or atlas tile identifier (ID) for the ith region of the kth packed video frame, represented by the variable "pi_rect_atlas_id[k][i]". The atlas ID identifies which of multiple atlases is the source of the data stored in the corresponding rect.

[0079] In some examples, the packed video frame information also identifies which of multiple maps (each represented by a different "map index" number) is associated with the i-th region of the k-th packed video frame. The map index number to be used to identify the map is represented by the variable "pi_rect_map_index[k][i]."

[0080] In some examples, the packed video frame information includes a flag that identifies whether the i-th region of the k-th packed video frame includes auxiliary video data. In some examples, the variable "pi_rect_auxiliary_video_flag[k][i]" represents the auxiliary video flag of the i-th region of the k-th packed video frame.

[0081] The packed video frame information also specifies a V3C type of the data contained within the ith region of the kth packed video frame atlas. The V3C type of the data may be represented by a variable "pi_rect_type_id[k][i]". In some examples, the value of "i_rect_type_id[k][i]" ranges from 2 to 4. In some examples, the V3C type of the data contained within the region may be an occupancy type, a geometry type, an attribute type, etc.

[0082] In the packed video frame information, the variable "pi_rect_attr_index[k][i]" specifies the attribute index of the ith region of the kth packed video frame. The attribute index represents which of multiple types of attribute data is contained in the ith region. In some examples, there are various types of data that are classified as attribute data, and the various types are numbered in the index such that the attribute index number reflects the type of attribute data that is contained in the ith region.

[0083] "pi_rect_attr_partition_index[k][i]", if present, specifies the attribute partition index of the i-th region of the k-th packed video frame. If not present, the value of "pi_rect_attr_partition_index[k][i]" is inferred to be equal to 0. The attribute partition index identifies which of a set of subcategories of the attribute type is contained within the attribute.

[0084] Therefore, the variables identified above represent examples of packed video frame information generated by the example packed information generator 316 before, during and / or after generation of the packed video frames. Further, as will be explained, the packed video frame information is provided to a bitstream formatter and to a multiplexer for multiplexing with corresponding packed video frames in accordance with the standard governing operation of the encoding system 200.

[0085] Still referring to FIG. 5, in some examples, the packed video frame information generator 316 generates packed video frame information for each packed video frame of the plurality of k packed video frames by generating a packed video frame codec index number, a packed video frame nominal 2D bit depth-1, a packed video frame width, a packed video frame height, and a number of regions to be included in the packed video frame. Then, for each region included in each region, the packed video frame information generator 316 generates a region position of the region in the packed video frame as an x, y position in the packed video frame. The packed video frame information generator 316 also generates a width, height, orientation, an atlas index number and / or atlas style index number, a data type index number, a map index number, an auxiliary data flag (to be set if auxiliary data is included in the region), and an attribute index number. The evaluator 330 of FIG. 3 can then multiply the value of the attribute dimension partition −1 by the value of the atlas index number (or atlas tile index number) and use it to determine whether the value multiplied by the attribute index number is greater than zero. If so, the packed video frame information generator 316 generates a region attribute partition index number for the attribute data. The packed video frame information generator 316 continues with the next region contained within the packed video frame until data has been generated for all regions of the packed video frame. The packed video frame information generator 316 then selects the next set of incoming geometry, attribute and / or occupancy data and generates the data described above for the next packed video frame. Region information is generated for each region contained within the next packed video frame in the manner described above, and the packed video frame information generator 316 continues in this manner until the incoming video stream or video stream sequence is encoded.

[0086] FIG. 6 illustrates information to be generated by the packed video frame information generator 316 when the packed video frame independent region SEI message is used to identify a correspondence between a region of the packed video frame and an independently decodable region in a video sub-bitstream. Additionally, FIG. 6 is pseudocode that can be used to generate an SEI message to indicate a correspondence of a region of the packed video frame to an independently decodable region in a video sub-bitstream. In some such examples, the example packed video frame information generator 316 can generate signaling to be included in the packed video frame independent region SEI message. Thus, the example packed video frame information generator 316 generates a set of variables such as those included in the pseudocode of FIG. 6. In some examples, the set of variables includes a variable "pir_num_pack" that represents the number of packed video frames for which the independently decodable region information is signaled. The variable "pir_description_type_idc[k] equal to 0" indicates that the tile indexes of the top-left and bottom-right tiles of an independently decodable region (e.g., a temporal motion constraint tile set) are signaled for the kth packed video frame region. Based on some coding standards, the atlas may be subdivided into tiles and may operate on a tiled basis. The variable "pir_description_type_idc equal to 1" indicates that a sub-picture ID is signaled for the kth packed video frame region. The variable "pir_description_type_idc[k]" ranges from 0 to 1. Other values ​​(e.g., values ​​in the range 2..3) may be reserved for future use.

[0087] In some examples, the variable "pir_num_rect[k]" specifies the number of regions of the kth packed video frame for which independently decodable region information is signaled. The set of variables "pir_top_left_tile_idx[k][i]" and "pir_bottom_right_tile_idx[k][i]", if present, identify the tile indexes of the top left and bottom right tiles, respectively, of an independently decodable region (e.g., a temporal motion constraint tile set) that correspond to the ith region of the video sub-bitstream of the kth packed video frame, in tile raster scan order. The variable "pir_subpic_id[k][i]", if present, identifies the sub-picture ID that corresponds to the ith region of the video sub-bitstream of the kth packed video frame.

[0088] Still referring to FIG. 6, in some examples, information identifying independent regions (pir) of a packed video frame can be generated by a packed video frame information generator using the pseudocode of FIG. 6. In some such examples, for each packed video frame of the plurality of packed video frames "k" for which information corresponding to an independently decodable region is signaled, the packed video frame information generator generates packed video frame information. In some such examples, a pir description type is obtained that, if equal to 0 (as described above), indicates that tile indices of the top left tile and bottom right tile of the pir independently decodable region (e.g., temporal motion constraint tile set) are signaled for a region of the kth pir packed video frame. The packed video frame information generator then determines, e.g., based on a packed video frame configuration, the number of regions of the kth pir packed video frame for which independently decodable region information is to be signaled. Then, for each packed video frame with pir description type equal to zero, the packed video frame information generator generates / determines tile indices for the top-left tile and bottom-right tile of the pir independently decodable region corresponding to each of the i regions of the video sub-bitstream of the kth packed video frame. If the value of the variable pir_description_type_idc is equal to 1, the packed video frame information generator generates sub-picture IDs for each region of each pir packed video frame.

[0089] 7 is a block diagram of portions of an example packed video frame video decoding system 700. In some examples, the packed video frame video decoding system 700 includes an example demultiplexer 702, an example packed video frame video stream parser 704, an example frame decoder 708, an example frame unpacker 710, an example reconstructor 712, an example renderer 714, an example metadata / SEI message extractor 716, and an example metadata / SEI message decoder 718.

[0090] In some examples, the example demultiplexer 702 receives the encoded video stream from the multiplexer of the video encoding system 200 of FIG. 2. The demultiplexer 702 separates the metadata / SEI messages from the encoded video stream and provides the separated information to the example metadata / SEI message extractor 716. In some examples, the metadata / SEI message extractor 716 examines the extracted information to identify SEI messages contained therein and to identify individual data fields and associated data that correspond to the video stream parser 704. In some examples, the video stream parser 704 uses a vuh unit header ("vuh_pack_index") that corresponds to a unit of a packed video frame sub-bitstream to identify a unit that includes a packed video frame. In some examples, the video stream parser 704 identifies the codec (using the variable "pi_pack_codec_id"). In some examples, values ​​of the variables vuh_pack_index and pi_pack_codec_id are provided to the video stream parser 704 by an example metadata / SEI message decoder 718. In some examples, the vuh_pack_index and pi_pack_codec_id information may be included in the packed video frame sub-bitstream.

[0091] In some examples, the video stream parser 704 uses information provided by the metadata / SEI message decoder 718 to identify units of the sub-bitstream that correspond to the packed video frames and to identify information corresponding to the units. In some examples, the information provided by the metadata / SEI message decoder 718 includes, for each packed video frame, a color component index, a bit depth, a packed video frame width, and a packed video frame height. In addition, the metadata / SEI message decoder 718 provides the number of map streams associated with the packed video frame and, if necessary, obtains a map count associated with a currently enabled packed frame.

[0092] In some examples, the example video stream parser 704 uses the collected / determined information to assign values ​​to a set of variables associated with the decoded packed video frame, including 1) "decPackFrame[frameIdx][compIdx][y][x]", 2) "decPackBitdepth[frameIdx]", 3) "decPackWidth[frameIdx]", and 4) "decPackHeight[frameIdx]", and 5) "mapIdx". The value represented by the variable "frameIdx" is an index of the decoded packed video frame, and the value represented by the variable "compIdx" is a color component index. The variable "y" is a row index within the decoded packed video frame and ranges from 0 to the value represented by the variable "decPackHeight[frameIdx]-1" (inclusive), and the variable "x" is a column index within the decoded frame and ranges from 0 to "decPackWidth[frameIdx]-1" (inclusive). The value of the map index ("mapIdx") falls within a range that starts at zero and extends to the value that the following statement evaluates to: "vps_multiple_map_streams_present_flag? vps_map_count_minus1:0" (inclusive). In some examples, the above statement evaluates to the value of the variable "vps_mpa_count_minus 1" if the value of the variable "vps_multiple_map_streams_present_flag" is true (or not equal to zero), and to zero otherwise. The variable "vps_map_count_minus1" is a value that represents the number of maps contained within or otherwise associated with the packed video frame with which the current packed video frame unit is associated.

[0093] In some examples, the packed video frame information is included with the packed video frame and parsed by a video stream parser 704, where the packed video frame information is parsed from the encoded video. The encoded video is provided to an example frame decoder 708 for decoding, and the example packed video frame information is provided to a frame unpacker 710. In some examples, the frame decoder 708 decodes the packed video frame units according to any video decoding standard, and provides the resulting decoded packed video frame units to an example frame unpacker 710. The frame unpacker 710 unpacks the packed video frame units using the packed video frame information associated with the decoded packed video frames, variables identified by the video stream parser 704, etc., to unpack the decoded packed video frame units. The information extracted from the unpacked frames is then provided to a reconstructor 712 to reconstruct the original 3D image for rendering in a renderer 714. In some examples, the frame unpacker 710 operates to use the packed frame information to identify regions, region characteristics, blocks of the regions, and / or maps to be used to map the blocks in real-time or near real-time to a volumetric video sequence for display by a renderer. In some such examples, the frame unpacker 710 deploys the operations described with respect to the decoding system of FIG. 8, although it is not necessary to perform the operations in the same order or in exactly the same manner.

[0094] 8 is a block diagram of an example metadata / SEI message decoder 718, an example frame unpacker 710, and an example reconstructor 712 (all of which are also shown in FIG. 7). The metadata / SEI message decoder 718 decodes information used to identify a packed video frame unit (including packed video frame information associated with the packed video frame unit to be unpacked), an atlas data header corresponding to the packed video frame unit, a decoded patch unit for the packed video frame unit, a decoded block-to-patch map, etc., as described above.

[0095] In some examples, the example frame unpacking unit 710 includes an example reg (region) counting unit 801, an example variable value setting unit 802, an example packed video frame unit fetching unit 804, an example reg type determination unit 806, an example reg selection unit 807, an example reg extraction unit 808, an example reg height and reg width increment unit 810, an example reg positioning determination unit 812, and an example mapping unit 814.

[0096] In some examples, the example packed video frame unit fetcher 804 receives and / or fetches a packed video frame unit (from the frame decoder 708 of FIG. 7) to be processed. In some examples, there are "k" packed video frames contained within the packed video frame unit, and each of the k packed video frames is turned on by a block of the frame unpacker 710 in a manner described below. In some examples, the packed video frame unit fetcher 804 does not receive (or fetch) a frame, but rather receives information regarding the location of the packed frame video within the packed video provided by the decoder.

[0097] Each one of the packed video frames is referred to as the kth packed video frame, where the value of k is implemented each time a packed video frame is processed by the frame unpacker 710. In some examples, the example region selector 807 selects one of a set of "i" regions associated with the kth packed video frame. Each one of the regions is referred to as the ith rect. An atlas index number (or atlas style index number) corresponding to the ith region of the kth packed video frame is then identified, and a map identifier corresponding to the ith region of the kth packed video frame is identified. As described above, in some examples, packed video frame information is received from the metadata / SEI message decoder 718 and / or from inspection of the packed video frame units and / or from information provided by the frame decoder 708 and used to identify the region from which the ith region is to be selected.

[0098] In some examples, the location of the block within the selected i-th region is determined by an example region positioning determiner 812. The determined location is represented by variables "x,y", where the value of x is the vertical location of the block within the region and y is the horizontal location of the block within the region. In some examples, the blocks correspond to equally sized squares contained within the region such that all of the blocks together form the region. Next, the frame index number and component index number of the currently turned on packed video frame unit (e.g., the frame into which the i-th region of the k-th packed video frame is to be inserted) are obtained.

[0099] In some examples, the region type determiner 806 determines a type of video data (e.g., occupancy data, geometry data, attribute data, etc.) contained within the ith region of the kth packed video frame. In some examples, when the region type determiner 806 determines the data type, the region type determiner 806 also generates information indicating that the type of data present at the x, y position of the ith region of the kth packed video frame is the type of data determined by the region type determiner. In some examples, the example frame unpacker 710 provides the decoded video data of the ith region of the kth packed video frame and the corresponding collected information as output to the example reconstructor 712. In some examples, the output provided to the reconstructor 712 includes an atlas index number of the ith region, a map index number of the ith region, a frame index number of the current packed video frame unit, a component index number, and x, y position information. The example reconstructor 712 uses these information to place the corresponding raw video data in the image being reconstructed. In some examples, if the type of data is determined to be attribute data, the frame unpacking unit 710 generates the information identified above and an attribute index number that identifies which of the various attributes are present at the indicated location. In some examples, after the information output is generated, the frame unpacking unit 710 performs the same operations for the next block at the next location (e.g., (x+1, y=1) of the i region of the k packed video frame). In some examples, after all of the blocks of the i region have been processed, the next region (e.g., the (i+1) region of the k packed video frame) is processed in the manner described above. Similarly, once all of the regions of the packed video frame have been turned on, the next packed video frame (e.g., the (k+1) packed video frame unit) is obtained and the operations of the frame unpacking unit 610 are repeated.In some examples, once all of the packed video frames of a packed video frame unit have been processed, the next set of packed video frames associated with another packed video frame unit are processed as described above. In some examples, once all of the packed video frames of a packed video frame unit have been turned on, the packed video frame unit fetcher 804 responds by fetching or receiving the next packed video frame unit and associated information from the video decoder.

[0100] In some examples, for blocks of regions having occupancy type data, the frame unpacking unit 610 generates the variables "decOccFrame[atlasIdx][mapIdx][frameIdx][compIdx][y][x]" and corresponding values. For blocks of regions having geometry type data, the frame unpacking unit 610 generates the variables "decGeoFrame[atlasIdx][mapIdx][frameIdx][compIdx][y][x]" and corresponding variables. Similarly, for blocks of regions having attribute data, the frame packing unit 610 generates the variables ("decAttrFrame[atlasIdx][attrIdx][mapIdx][partIdx][frameIdx][compIdx][y][x]") and corresponding data. As described above, this information is used by the reconstructor 712 to reconstruct the view represented by the packed video frame unit. In some examples, the frame unpacker may be implemented according to the example pseudocode of FIG.

[0101] 2 and 3, one or more of the elements, processes and / or devices shown in FIG. 2 and 3 may be combined, divided, rearranged, omitted, removed, and / or implemented in any other manner. In addition, the exemplary captured video data and metadata storage 202, the exemplary video and metadata pre-processor 204, the exemplary parameter / atlas data provider 206, the exemplary geometry video data provider 208, the exemplary attribute video data provider 210, the exemplary occupancy video data provider 212, the exemplary bitstream formatter 216, the exemplary frame packer 218, the exemplary encoder 220 and the exemplary multiplexer 222, the exemplary profile / tier / level storage 302, the exemplary packed video flag setter 304, the exemplary pack configuration selector 306, the exemplary atlas identifier 307, the exemplary geometry region generator 308, the exemplary attribute region generator 310, the exemplary The occupied area generator 312, the example pack generator 314, the example pack information generator 316, the example metadata storage 318, the example pack storage 322, the example metadata provider 324, the example pack provider 326, the example decoder verifier 328, the example evaluator 330, the example flag storage 332, the example flag setter 333, the example flag adder 334, the example packed video checker 336, the example area subtracter 338, and the example comparator 340, and / or, more generally, the example encoding system 200 of Figures 2 and 3 may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware.Thus, for example, the exemplary captured video data and metadata storage 202, the exemplary video and metadata pre-processor 204, the exemplary parameter / atlas data provider 206, the exemplary geometry video data provider 208, the exemplary attribute video data provider 210, the exemplary occupation video data provider 212, the exemplary bitstream formatter 216, the exemplary frame packing unit 218, the exemplary encoder 220 and the exemplary multiplexer 222, the exemplary profile / tier / level storage 302, the exemplary packed video flag setter 304, the exemplary pack configuration selector 306, the exemplary atlas identifier 307, the exemplary geometry region generator 308, the exemplary attribute region generator 310, the exemplary occupation region generator 312, the exemplary pack generator 314, the exemplary pack information generator 316, the exemplary metadata Any of the storage 318, the example pack storage 322, the example metadata provider 324, the example pack provider 326, the example decoder verifier 328, the example evaluator 330, the example flag storage 332, the example flag setter 333, the example flag adder 334, the example packed video checker 336, the example region subtracter 338, the example comparator 340, and / or, more generally, the example encoding system 200 may be implemented by one or more analog or digital circuits, logic circuits, programmable processors, programmable controllers, graphics processing units (GPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), programmable logic devices (PLDs), and / or field programmable logic devices (FPLDs).Any of the apparatus or system claims of this patent encompassing a purely software and / or firmware implementation may be read to include an example captured video data and metadata storage 202, an example video and metadata pre-processor 204, an example parameter / atlas data provider 206, an example geometry video data provider 208, an example attribute video data provider 210, an example occupancy video data provider 212, an example bitstream formatter 216, an example frame packing unit 218, an example encoder 220 and an example multiplexer 222, an example profile / tier / level storage 302, an example packed video flag setter 304, an example pack configuration selector 306, an example atlas identifier 307, an example geometry region generator 308, an example attribute region generator 309, an example frame packing unit 219, an example frame packing unit 220, an example frame packing unit 230, an example frame packing unit 232, an example frame packing unit 234, an example frame packing unit 236, an example frame packing unit 238, an example frame packing unit 238, an example frame packing unit 239, an example frame packing unit 240, an example frame packing unit 242, an example frame packing unit 244, an example frame packing unit 246, an example frame packing unit 248, an example frame packing unit 248, an example frame packing unit 248, an example frame packing unit 249, an example frame packing unit 250, an example frame packing unit 252, an example frame packing unit 254, an example frame packing unit 256, an example frame packing unit 258, an example frame packing unit 259, an example frame packing unit 260, an example frame packing unit 262, an example frame packing unit 264, an example frame packing unit 266, an example frame packing unit At least one of 310, example occupied area generator 312, example pack generator 314, example pack information generator 316, example metadata storage 318, example pack storage 322, example metadata provider 324, example pack provider 326, example decoder verifier 328, example evaluator 330, example flag storage 332, example flag setting unit 333, example flag adder 334, example packed video check unit 336, example area subtracter 338, and example comparator 340 is expressly defined hereby to include a non-transitory computer-readable storage device or storage disk, such as a memory, digital versatile disk (DVD), compact disk (CD), Blu-ray® disk, etc., including software and / or firmware. Still further, the example encoding system 200 of FIGS. 2 and 3 may include one or more elements, processes and / or devices in addition to or instead of those shown in FIGS. 2 and 3, and / or may include more than one of any or all of the elements, processes and devices shown.As used herein, the phrase "in communication," including variations thereof, encompasses direct communication and / or indirect communication through one or more intervening components and does not require direct physical (e.g., wired) communication and / or constant communication, but rather additionally includes selective communication at periodic intervals, scheduled intervals, aperiodic intervals, and / or one-time events.

[0102] Flowcharts representative of example hardware logic, machine-readable instructions, hardware-implemented state machines, and / or any combination thereof implementing the encoding system 200 of Figures 2 and 3 are shown in Figures 10A, 10B, 11, and 12. The machine-readable instructions may be one or more executable programs or portions of executable programs for execution by a computer processor and / or processor circuitry, such as the processor 1600 shown in the example processor platform 1600 discussed below in connection with Figure 16. The programs may be embodied in software stored on a non-transitory computer-readable storage medium, such as a CD-ROM, floppy disk, hard drive, DVD, Blu-ray® disk, or memory associated with the processor 1612, although the entire program and / or portions thereof may alternatively be executed by a device other than the processor 1612 and / or embodied in firmware or dedicated hardware. Further, although the example program is described with reference to the flowcharts shown in Figures 10A, 10B, 11, and 12, many other ways of implementing the example encoding system 200 may alternatively be used. For example, the order of execution of the blocks may be changed, and / or some of the described blocks may be changed, removed, or combined. Additionally or alternatively, any or all of these blocks may be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (opamps), logic circuits, etc.) configured to perform the corresponding operations without executing software or firmware. The processor circuits may be distributed across different network locations and / or local to one or more devices (e.g., multi-core processors in a single machine, multiple processors distributed across server racks, etc.).

[0103] The machine-readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a fragmented format, a compiled format, an executable format, a packaged format, etc. The machine-readable instructions as described herein may be stored as data or data structures (e.g., portions of instructions, code, representations of code, etc.) that can be utilized to create, manufacture, and / or generate machine-executable instructions. For example, the machine-readable instructions may be fragmented and stored in one or more storage devices and / or computing devices (e.g., servers) located at the same or different locations of a network or collection of networks (e.g., in a cloud, in an edge device, etc.). The machine-readable instructions may require one or more of installing, modifying, adapting, updating, combining, supplementing, configuring, decrypting, decompressing, unpacking, distributing, reassigning, compiling, etc. to make the machine-readable instructions directly readable, interpretable, and / or executable by a computing device and / or other machines. For example, machine-readable instructions may be stored in multiple portions that are individually compressed, encrypted and stored on separate computing devices, which portions, when decrypted, decompressed and combined, form a set of executable instructions that implement one or more functions that may together form a program such as those described herein.

[0104] In another example, machine-readable instructions may be stored in a state that can be read by a processor circuit, but may require the addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc., to execute the instructions on a particular computing device or other device. In another example, the machine-readable instructions may need to be configured (e.g., storing settings, entering data, recording network addresses, etc.) before the machine-readable instructions and / or corresponding program can be executed in whole or in part. Thus, a machine-readable medium, as used herein, may include machine-readable instructions and / or programs, regardless of the particular format or state of the machine-readable instructions and / or programs when stored or otherwise stationary or in motion.

[0105] The machine-readable instructions described herein may be expressed by any past, present, or future command language, scripting language, programming language, etc. For example, the machine-readable instructions may be expressed using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, HyperText Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0106] As mentioned above, the example processes of Figures 10A, 10B, 11 and 12 may be implemented using executable instructions (e.g., computer and / or machine readable instructions) stored on a non-transitory computer and / or machine readable medium, such as a hard disk drive, a flash memory, a read-only memory, a compact disk, a digital versatile disk, a cache, a random access memory and / or any other storage device or storage disk in which information is stored for any duration (e.g., long term, permanently, short term, for temporary buffering, and / or for caching of information). As used herein, the term non-transitory computer readable medium is expressly defined to include any type of computer readable storage device and / or storage disk and to exclude propagating signals and to exclude transmission media.

[0107] "Including" and "comprising" (and all their forms and tenses) are used herein as open-ended terms. Thus, whenever a claim employs any form of "include" or "comprise" (e.g., comprises, includes, comprising, including, having, etc.) as a preamble or within any type of claim recitation, it is to be understood that additional elements, terms, etc. may be present without departing from the scope of the corresponding claim or recitation. As used herein, the phrase "at least," when used as a transitional term, for example, in the preamble of a claim, is open-ended in the same way that the terms "comprising" and "comprising" are open-ended. The term "and / or," when used in the form of, for example, A, B, and / or C, refers to any combination or subset of A, B, and C, such as, for example, (1) A only, (2) B only, (3) C only, (4) A and B, (5) A and C, (6) B and C, and (7) A, B, and C. When used herein in the context of describing a structure, component, item, object, and / or thing, the phrase "at least one of A and B" is intended to refer to an implementation that includes either (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, when used herein in the context of describing a structure, component, item, object, and / or thing, the phrase "at least one of A or B" is intended to refer to an implementation that includes either (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. When used herein in the context of describing the implementation or performance of a process, instruction, action, activity, and / or step, the phrase “at least one of A and B” is intended to refer to implementations that include any of: (1) at least one A; (2) at least one B; and (3) at least one A and at least one B.Similarly, when used herein in the context of describing the implementation or performance of a process, instruction, action, activity, and / or step, the phrase “at least one of A or B” is intended to refer to implementations that include any of: (1) at least one A; (2) at least one B; and (3) at least one A and at least one B.

[0108] As used herein, singular references (e.g., "a", "an", "first", "second", etc.) do not exclude a plurality. The term "a" or "an" entity as used herein refers to one or more of that entity. The terms "a" (or "an"), "one or more" and "at least one" can be used interchangeably herein. Furthermore, although individually listed, a plurality of means, elements or method actions may be implemented by, for example, a single unit or processor. In addition, although individual features may be included in different examples or claims, they may be combined in some cases, and their inclusion in different examples or claims does not imply that a combination of features is not feasible and / or advantageous.

[0109] Example ways of implementing a decoder 700 are shown in FIGS. 7 and 8, although one or more of the elements, processes and / or devices shown in FIGS. 7 and 8 may be combined, divided, rearranged, omitted, removed, and / or implemented in any other manner. Further, the example demultiplexer 702, the example packed video stream parser 704, the example frame decoder 708, the example frame unpacker 710, the example reconstructor 712, the example renderer 714, the example metadata / SEI message extractor 716, and the example metadata / SEI message decoder 718, the example region counter 801, the example variable value setter 802, the example packed frame unit fetcher 804, the example region type determiner 806, the example region selector 807, the example region extractor 808, the example region height and region width incrementer 810, and the example region positioner 812, the example mapping unit 814, and / or, more generally, the example decoder 700 of Figures 7 and 8 may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware.Thus, for example, an example demultiplexer 702, an example packed video stream parser 704, an example frame decoder 708, an example frame unpacker 710, an example reconstructor 712, an example renderer 714, an example metadata / SEI message extractor 716, and an example metadata / SEI message decoder 718, an example region counter 801, an example variable value setter 802, an example packed frame unit fetcher 804, an example region type determiner 806, an example region selector 807, an example region extractor 808, an example region height and the region width increment unit 810, and any of the example region positioning determination unit 812, the example mapping unit 814, and / or, more generally, the example decoder 700, may be implemented by one or more analog or digital circuits, logic circuits, programmable processors, programmable controllers, graphics processing units (GPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), programmable logic devices (PLDs), and / or field programmable logic devices (FPLDs).

[0110] When reading any of the apparatus or system claims of this patent that encompass purely software and / or firmware implementations, the following may be included: an example demultiplexer 702, an example packed video stream parser 704, an example frame decoder 708, an example frame unpacker 710, an example reconstructor 712, an example renderer 714, an example metadata / SEI message extractor 716, and an example metadata / SEI message decoder 718, an example region counter 801, an example variable value setter 802, an example packed frame unit fetcher 804, an example frame decoder 708, an example frame unpacker 710, an example reconstructor 712, an example rendering unit 714, an example metadata / SEI message extractor 716, and an example metadata / SEI message decoder 718, an example region counter 801, an example variable value setter 802, an example packed frame unit fetcher 804, an example frame decoder 706, an example frame unpacker 710, an example frame unpacker 712, an example frame unpacker 714, an example frame unpacker 710, an example frame unpacker 712 ...0, an example frame unpacker 712, an example frame unpacker 710, an example frame unpacker 710, an example frame unpacker 710, an example frame unpacker 710, an example frame unpacker 710, an example frame unpacker 710, an example frame unpack At least one of unit 804, example region type determiner 806, example region selector 807, example region extractor 808, example region height and region width incrementer 810, and example region positioning determiner 812, and example mapper 814 are expressly defined hereby to include a non-transitory computer-readable storage device or storage disk, such as a memory, digital versatile disk (DVD), compact disk (CD), Blu-ray disk, etc., including software and / or firmware. Still further, the example decoder 700 of Figures 7 and 8 may include one or more elements, processes and / or devices in addition to or instead of those shown in Figures 7 and 8, and / or may include more than one of any or all of the elements, processes and devices shown. As used herein, the phrase "in communication," including variations thereof, encompasses direct communication and / or indirect communication through one or more intervening components and does not require direct physical (e.g., wired) communication and / or constant communication, but rather additionally includes selective communication at periodic intervals, scheduled intervals, aperiodic intervals, and / or one-time events.

[0111] Flowcharts representative of example hardware logic, machine-readable instructions, hardware-implemented state machines, and / or any combination thereof implementing the decoder 700 of Figures 7 and 8 are shown in Figures 14 and 15. The machine-readable instructions may be one or more executable programs or portions of executable programs for execution by a computer processor and / or processor circuitry, such as the processor 1712 shown in the example processor platform 1700 discussed below in connection with Figure 17. The programs may be embodied in software stored on a non-transitory computer-readable storage medium, such as a CD-ROM, floppy disk, hard drive, DVD, Blu-ray® disk, or memory associated with the processor 1712, although the entire program and / or portions thereof may alternatively be executed by a device other than the processor 1712 and / or embodied in firmware or dedicated hardware. Furthermore, although the example programs are described with reference to the flowcharts shown in Figures 14 and 15, many other ways of implementing the example decoder 700 may alternatively be used. For example, the order of execution of the blocks may be changed and / or some of the described blocks may be changed, eliminated, or combined. Additionally or alternatively, any or all of these blocks may be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (opamps), logic circuits, etc.) configured to perform the corresponding operations without executing software or firmware. Processor circuitry may be distributed across different network locations and / or local to one or more devices (e.g., a multi-core processor within a single machine, multiple processors distributed across server racks, etc.).

[0112] The machine-readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a fragmented format, a compiled format, an executable format, a packaged format, etc. The machine-readable instructions as described herein may be stored as data or data structures (e.g., portions of instructions, code, representations of code, etc.) that can be utilized to create, manufacture, and / or generate machine-executable instructions. For example, the machine-readable instructions may be fragmented and stored in one or more storage devices and / or computing devices (e.g., servers) located at the same or different locations of a network or collection of networks (e.g., in a cloud, in an edge device, etc.). The machine-readable instructions may require one or more of installing, modifying, adapting, updating, combining, supplementing, configuring, decrypting, decompressing, unpacking, distributing, reassigning, compiling, etc. to make the machine-readable instructions directly readable, interpretable, and / or executable by a computing device and / or other machines. For example, machine-readable instructions may be stored in multiple portions that are individually compressed, encrypted and stored on separate computing devices, which portions, when decrypted, decompressed and combined, form a set of executable instructions that implement one or more functions that may together form a program such as those described herein.

[0113] In another example, machine-readable instructions may be stored in a state that can be read by a processor circuit, but may require the addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc., to execute the instructions on a particular computing device or other device. In another example, the machine-readable instructions may need to be configured (e.g., storing settings, entering data, recording network addresses, etc.) before the machine-readable instructions and / or corresponding program can be executed in whole or in part. Thus, a machine-readable medium, as used herein, may include machine-readable instructions and / or programs, regardless of the particular format or state of the machine-readable instructions and / or programs when stored or otherwise stationary or in motion.

[0114] The machine-readable instructions described herein may be expressed by any past, present, or future command language, scripting language, programming language, etc. For example, the machine-readable instructions may be expressed using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, HyperText Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0115] As mentioned above, the example processes of Figures 14 and 15 may be implemented using executable instructions (e.g., computer and / or machine readable instructions) stored on a non-transitory computer and / or machine readable medium, such as a hard disk drive, a flash memory, a read-only memory, a compact disk, a digital versatile disk, a cache, a random access memory, and / or other storage device or disk in which information is stored for any duration (e.g., long term, permanently, short term, for temporary buffering, and / or for caching of information). As used herein, the term non-transitory computer readable medium is expressly defined to include any type of computer readable storage device and / or disk, and to exclude propagating signals and to exclude transmission media.

[0116] 10A and 10B show a program that may be executed to implement the example encoding system 200 of FIG. 2. The program of FIG. 10 includes a block 1002 in which captured video data and parameter / atlas data are stored in the example captured video data and metadata storage 202. The example video and metadata preprocessor 204 separates the parameter / atlas data from different types of video (e.g., geometry, attributes, occupancy, etc.) contained within the video data (block 1004). The different types of video data are then provided to the packed video frame packer 218 via the video data providers 208, 210, and 212, and the parameter and atlas data are provided to the bitstream formatter 216 via the parameter and atlas data provider 206 (block 1006).

[0117] The example packed video frame packer 218 determines a packed video frame configuration based in part on the type and size of the video data provided by the video data providers 208, 210, 212, or based on any of a variety of other factors (block 1008). In some examples, the example packed video frame packer 218 determines the number of decodes required to decode the bitstream generated by the encoding system 200 (block 1010), as described with reference to FIG. 3. The packed video frame packer 218 tests to determine whether the number of decodes will meet a threshold (e.g., is less than or equal to a maximum number of decodes) (block 1012). In some examples, if the maximum number of decodes is exceeded, the packed video frame packer 218 does not encode the video and the program ends.

[0118] If the maximum number of decodes has been met (eg, the number of decodes is less than or equal to the maximum number of decodes), the packed video frame packer 218 generates a packed video frame (block 1014).

[0119] In some examples, when (or before or after) generating the packed video frame, the example packed video frame packer 218 (FIG. 2) generates packed video frame information (block 1016). In some examples, the packed video frame includes the packed video frame in addition to the types of data contained within the regions of the packed video frame. In some examples, the packed video frame information includes any of a variety of information about the packed video frame, including the regions contained therein, the types of video content within each of the regions, and the atlas or atlas style corresponding to each of the regions. The bitstream formatter 216 (FIG. 2) formats the parameter / atlas data (and any SEI messages that apply to the packed video frame) (block 1018). In some examples, the parameter data may include parameter data corresponding to camera views of the video used to generate the packed video frame, and / or camera related data identifying, for example, the number of cameras for which data is being collected, the views associated with each of the cameras, etc. In some examples, as described above, the packed video frame information may include information regarding the collective packed video frame, the individual packed video frames, the individual regions contained within each of the packed video frames, the dimensions of the regions, etc. The packed video frames are encoded in the encoder 220 (block 1020), and an example multiplexer 222 operates to multiplex the bitstream information with the encoded packed video frames (block 1022). The multiplexed bitstream is then transmitted to the decoder (block 1024), and the program terminates.

[0120] FIG. 11 is a program 1100 for implementing the example packed video frame packer 218 of FIG. 2 and FIG. 3. The program 1100 begins at block 1102, where profile / tier / level information is stored in a profile / tier / level storage. The example packed video flag setter 304 then sets a video flag indicating that packed video frame data is to be processed (block 1104). In some examples, setting the packed video flag includes setting one or more bits in the metadata storage 318. In some examples, based on information in the profile / tier / level storage 302 or based on information provided by any of the other blocks of FIG. 2, the packed video frame configuration selector 306 determines a video frame configuration / layout (block 1106). In some examples, any of the packed video frame configurations shown in FIG. 1A, FIG. 1B, and / or FIG. 1C may be selected. In some instances, any of a variety of configurations other than (or in addition to) those of Figures 1A, 1B, and 1 may be determined.

[0121] Next, the geometry region generator 308 generates a geometry rect / reg, the attribute region generator 310 generates a region, and the example occupancy region generator 312 generates an occupancy rect (block 1108). In some examples, the packed video frame generator 314 uses the selected packed video frame configuration and the generated region to generate a packed video frame (or inserts the region into one or more packed video frames according to the selected packed video frame configuration) (block 1110). In some examples, the generated packed video frame identifies the location of the region in the volumetric video being encoded and associates the location (and other information) with the location of the volumetric video in the bitstream. In addition, the packed video frame information generator 316 generates packed video frame information for each packed video frame created by the packed video frame generator 314, as described above (block 1112). The packed video frame information can be stored in the metadata storage 318. In some examples, the metadata provider 324 provides the metadata (packed video frame information, parameter / atlas data, etc., corresponding to the packed video frame (or multiple packed video frames)) to the example bitstream formatter 2016 of FIG. 2 (block 1114). In some examples, the packed video frame is attached to or otherwise associated with the corresponding packed video frame to be encoded. In some examples, the example packed video frame provider 326 provides the packed video frame (obtained from the packed storage 322) to the example encoding system 200 of FIG. 2 where the packed video frame is encoded (block 1116). Additionally, the packed video frame information may be associated with the encoded packed frame.In addition, the example multiplexer multiplexes the metadata bitstream with the corresponding encoded data for transport to the decoder for decoding and rendering (block 1118), and the multiplexed stream is sent to the decoder (block 1120). The program 1100 then repeats or ends.

[0122] In some examples, the packed video frame generator 314 inserts regions into different packed video frames simultaneously such that multiple packed video frames are generated at the same time (or within the same time period of each other). In some examples, the example packed video frame information generator 316 examines packed video frames stored in the packed video frame storage 322 to determine the packed video frame information, and in some examples, the packed video frame generator 314 provides one or more types of packed video frame information to the packed video frame information generator 316. In some examples, the packed video frame information generator 316 determines an order to arrange the packed video frame information according to a video encoding standard.

[0123] Figure 12 is a program 1200 that implements the example decoder verifier 328 of Figure 3. The example program 1000 of Figures 10A and 10B illustrates how the decoder verifier 328 operates within the context of the encoding system 200 operation. For clarity, the operation of the decoder verifier 328 is not reflected in the program 11 of Figure 11, and the specifics of the decoder verifier 328 are not reflected in Figure 10. Instead, the operation of the decoder verifier 328 is described with reference to the program 1200 of Figure 12. The broader context of how the example program 1200 operates to perform blocks 1010 and 1012 is illustrated with reference to Figures 10A and 10B.

[0124] The example program may include block 1202, in which the example flag setting unit 333 of FIG. 3 sets flags in the example flag storage 332. Each flag corresponds to a different type of video data, and each flag indicates whether the corresponding type of video data is present in the corresponding packed video frame video unit. The different types of video data include auxiliary video data, occupancy video data, geometry video data, and attribute video data. In some examples, the attribute data may include subcategories of attribute data, and the number of subcategories included in the attribute video data is represented using the variable "ai_attribute_count". In addition, the example flag adding unit 334 sums up the number of flags associated with each atlas of the packed video frame unit to determine a total number of types of video data included in the packed video frame unit (block 1204). The sum of the flags is the number of decodes that can be decoded by a decoder receiving the encoded video, and is equal to the variable "NumDecodes". Once the number of flags for all atlases are summed, the example packed frame video checker 336 checks whether the packed video frame flag indicates that the packed video frame unit contains a packed video frame (block 1206). If the packed video frame video flag does not indicate that the packed video frame unit contains a packed video frame, the program 1200 ends. Provided that the packed video frame video flag indicates that the packed video frame video unit contains a packed video frame, for each packed video frame, the example region subtracter 338 subtracts the number of regions contained within each packed video frame from the value of NumDecodes (block 1208) until the regions of all packed video frames have been subtracted.The number of regions contained within each packed video frame minus one may be represented by a variable "pi_num_rect[k]-1," where the variable "k" represents the number of packed video frames contained within the total number of packed video frames. The example comparator 340 then compares the value of NumDecodes to a value corresponding to the maximum number of decodes that the receiving decoder is capable of decoding (block 1210). The maximum number of decodes is represented by a variable "MaxDecodes" (also referred to as a threshold). In some examples, the frame packer 218 is preprogrammed with information identifying corresponding MaxDecodes values ​​for various types of decoders to which the video stream may be transmitted.

[0125] If the maximum number of decodes (also referred to as a threshold) has been exceeded (eg, the threshold is not met as determined in block 1212), the corresponding video is not encoded and program 1200 ends.

[0126] If the threshold is met (as determined in block 1212), the encoder proceeds with generating the packed video frame (block 1214) and program 1200 ends. In some examples, the pseudocode of FIG. 13 can be used to implement or replace program 1200 of FIG. 12. The purpose of program 1200 of FIG. 12 is to determine the number of video decodes required to decode the bitstream. For every atlas, if a particular component type exists, it is added. Then, if packing is used, the #-1 of regions can be subtracted from the total number of decodes, because the entire packed frame requires only one decoder, regardless of the number of regions it contains.

[0127] FIG. 14 is a program 1400 for implementing the example packed video frame decoder 600 of FIG. 6. In some examples, the program 1400 includes a block 1402 in which the example demultiplexer 702 of FIG. 7 receives and demultiplexes the encoded video bitstream generated by the encoding system 200 of FIG. 2 (block 1402). In addition, in block 1402, the demultiplexer 702 separates metadata / SEI messages from the encoded video sub-bitstreams and provides the metadata / SEI messages to an example metadata / SEI message extractor 716. In addition, the demultiplexer 702 provides the encoded video sub-bitstreams and the packed video frame information corresponding to the metadata / SEI messages to an example video stream parser 604.

[0128] In some examples, the example video stream parser 704 uses information provided by the example metadata / SEI message decoder 718 to identify units of the sub-bitstream that correspond to the packed video frames and to identify information corresponding to the packed video frames. In addition, the video stream parser 704 parses the packed video frame information from the encoded packed video sub-bitstream received from the encoder (block 1404). In some examples, the information of the metadata / SEI message decoder 718 includes, for each packed video frame, a color component index, a bit depth, a pack width, and a pack height. In addition, the metadata / SEI message decoder 718 provides the number of map streams associated with the packed video frames and, if necessary, obtains map counts associated with one or more packed video frames that are currently turned on.

[0129] In some examples, the example decoding process 1200 includes parsing the packed frame video information from the bitstream using the video stream parser 704 (block 1404). In some examples, the video stream parser associates the packed video frame information with a set of variables describing the packed video frame, including 1) "decPackFrame[frameIdx][compIdx][y][x]", 2) "decPackBitdepth[frameIdx]", 3) "decPackWidth[frameIdx]" and 4) "decPackHeight[frameIdx]", and 5) "mapIdx". The value represented by the variable "frameIdx" is an index of the decoded packed video frame, and the value represented by the variable "compIdx" is a color component index. The variable "y" is a row index within the packed video frame and ranges from 0 to the value represented by the variable "decPackHeight[frameIdx]-1" inclusive, and the variable "x" is a column index within the packed video frame and ranges from 0 to "decPackWidth[frameIdx]-1" inclusive. The value of the map index ("mapIdx") falls within a range that starts at zero and extends to the value to which the following statement evaluates: "vps_multiple_map_streams_present_flag? vps_map_count_minus1:0" (inclusive). In some examples, the above statement evaluates to the value of the variable "vps_mpa_count_minus 1" if the value of the variable "vps_multiple_map_streams_present_flag" is true (or not equal to zero), and to zero otherwise. The variable "vps_map_count_minus1" is a value that represents the number of maps contained within or otherwise associated with the currently turned on packed video frame.

[0130] The decoded packed video frames are provided to an exemplary frame decoder 608. The decoded packed video information is then provided to a frame parser. The frame decoder 608 performs any decoding process on the packed video frame stream according to any video decoding standard and provides the resulting decoded packed video frame units to an exemplary frame unpacker 610 (block 1406). The packed video frame unpacker 610 unpacks the packed video frame units using the packed video frame information associated with the decoded packed video frames, variables identified by the video stream parser 604, etc., to unpack the decoded packed video frames (block 1408).

[0131] An example frame mapper uses the decoded packed video stream and the unpacked frame information to map regions of the decoded packed video stream into a volumetric video sequence (block 1410). The mapped video is provided in a stream to an example reconstructor 712 (see FIG. 7), which uses this information to perform any additional processing (block 1412) required to reconstruct a three-dimensional image for display in an example renderer (block 1414).

[0132] Figure 15 is a program 1500 that can be used to implement the example frame unpacking unit of Figure 8. In some examples, the program 1500 begins at block 1502, where the metadata / SEI message decoder 618 decodes metadata (including packed video frame information) and SEI messages generated by the frame packing unit of the example encoder. The metadata / SEI messages can include packed video frame information associated with the packed video frame units to be unpacked, an atlas data header corresponding to the packed video frame units, decoded patch units for the packed video frame units, decoded block-to-patch maps, etc. The information of the metadata / SEI message decoder 618 can be provided to the frame unpacking unit as needed or as required by the frame unpacking unit.

[0133] In some examples, the example packed video frame unit fetcher 704 receives and / or fetches (from the frame decoder 608) a packed video frame unit to be processed (block 1504). In some examples, for each region of each packed video frame, information describing each region is collected from the metadata / SEI message decoder (block 1508), as described above with respect to FIG. 7. For each block of each rect, a position is determined and corresponding information is obtained (block 1510), as described above with respect to FIG. 7. The block position, corresponding block information, and raw data are provided to a reconstructor for placement in the image, as described above with respect to FIG. 7 (block 1512). Blocks 1508, 1510, and 1512 are repeated until all blocks of all regions of all packed video frames have been processed (block 1514). The program 1500 then ends. In some examples, the program 1500 can be implemented using the pseudocode of FIG. 9.

[0134] Figure 16 is a block diagram of an example processor platform 1600 configured to execute the instructions of Figures 10A, 10B, 11, and 12 to implement the encoders of Figures 2 and 3. The processor platform 1600 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., neural networks), a mobile device (e.g., a mobile phone, a smartphone, a tablet such as an iPad®™), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray® player, a gaming console, a personal video recorder, a set-top box, a headset or other wearable device, or any other type of computing device.

[0135] The illustrated example processor platform 1600 includes a processor 1612. The illustrated example processor 1612 is hardware. For example, the processor 1612 may be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor-based (e.g., silicon-based) device. In this example, the processor includes an example video and metadata pre-processor 204, an example parameter / atlas data provider 206, an example geometry video data provider 208, an example attribute video data provider 210, an example occupancy video data provider 212, an example bitstream formatter 216, an example frame packer 218, an example encoder 220 and an example multiplexer 222, an example packed video frame video flag setter 304, an example packed video frame composition selector 306, an example atlas identifier 307, an example geometry region 218, an example frame packer 219, an example frame packer 220, an example frame packer 222, an example frame packer 224, an example frame packer 226, an example frame packer 228, an example frame packer 229, an example frame packer 230, an example frame packer 232, an example frame packer 234, an example frame packer 236, an example frame packer 238, an example frame packer 239, an example frame packer 240, an example frame packer 242, an example frame packer 244, an example frame packer 246, an example frame packer 248, an example frame packer 249, an example frame packer 250, an example frame packer 252, an example frame packer 254, an example frame packer 256, an example frame packer 258, an example frame packer 259, an example frame packer 260 generator 308, an example attribute region generator 310, an example occupied region generator 312, an example packed video frame generator 314, an example packed video frame information generator 316, an example metadata provider 324, an example packed video frame provider 326, an example decoder verifier 328, an example evaluator 330, an example flag setter 333, an example flag adder 334, an example packed video frame video checker 336, an example region subtracter 338, and an example comparator 340, and / or more generally, implements the example encoding system 200.

[0136] The processor 1612 of the illustrated example includes a local memory 1613 (e.g., a cache). The processor 1612 of the illustrated example communicates with a main memory, including a volatile memory 1614 and a non-volatile memory 1616, via a bus 1618. The volatile memory 1614 may be implemented by synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS® Dynamic Random Access Memory (RDRAM®), and / or any other type of random access memory device. The non-volatile memory 1616 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1614, 1616 is controlled by a memory controller.

[0137] The processor platform 1600 of the depicted example also includes an interface circuit 1620. The interface circuit 1620 may be implemented with any type of interface standard, such as an Ethernet interface, a Universal Serial Bus (USB), a Bluetooth interface, a Near Field Communication (NFC) interface, and / or a PCI Express interface.

[0138] In the depicted example, one or more input devices 1622 are connected to the interface circuit 1620. The input devices 1622 allow a user to enter data and / or commands into the processor 1612. The input devices may be implemented, for example, by audio sensors, microphones, cameras (still or video), keyboards, buttons, mice, touch screens, track pads, track balls, isopoints, and / or voice recognition systems.

[0139] One or more output devices 1624 are also connected to the interface circuitry 1620 of the illustrated example. The output device(s) 1624 may be implemented by, for example, a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touch screen, etc.), a haptic output device, a virtual reality headset, etc. Thus, the interface circuitry 1620 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0140] The interface circuitry 1620 of the illustrated example also includes communications devices such as a transmitter, receiver, transceiver, modem, residential gateway, wireless access point, and / or network interface to facilitate the exchange of data with external machines (e.g., any type of computing device) via a network 1626. Communications may be via, for example, an Ethernet connection, a Digital Subscriber Line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-sight wireless system, a cellular telephone system, etc.

[0141] The processor platform 1600 of the depicted example also includes one or more mass storage devices 1628 for storing software and / or data. Examples of such mass storage devices 1628 include floppy disk drives, hard drive disks, compact disk drives, Blu-ray® disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0142] The machine-executable instructions 1632 of Figures 10A, 10B, 11 and 12 may be stored on the mass storage device 1628, the volatile memory 1614, the non-volatile memory 1616, and / or a removable non-transitory computer-readable storage medium such as a CD or DVD.

[0143] Figure 17 is a block diagram of an exemplary processor platform 1700 configured to execute the instructions of Figures 14 and 15 to implement the decoders of Figures 7 and 8. The processor platform 1700 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., neural networks), a mobile device (e.g., a mobile phone, a smart phone, a tablet such as an iPad®™), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray® player, a gaming console, a personal video recorder, a set-top box, a headset or other wearable device, a virtual reality system, or any other type of computing device.

[0144] The processor platform 1700 of the illustrated example includes a processor 1712. The processor 1712 of the illustrated example is hardware. For example, the processor 1712 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. A hardware processor can be a semiconductor-based (e.g., silicon-based) device. In this example, the processor implements an exemplary demultiplexer 702, an exemplary packed video frame video stream parser 704, an exemplary frame decoder 708, an exemplary frame unpacker 710, an exemplary reconstructor 712, an exemplary renderer 714, an exemplary metadata / SEI message extractor 716, and an exemplary metadata / SEI message decoder 718, an exemplary region counter 801, an exemplary variable value setter 802, an exemplary packed video frame unit fetcher 804, an exemplary region type determiner 806, an exemplary region selector 807, an exemplary region extractor 808, an exemplary region height and region width incrementer 810, an exemplary region positioning determiner 812, and an exemplary mapping unit 814.

[0145] The processor 1712 of the illustrated example includes a local memory 1713 (e.g., a cache). The processor 1712 of the illustrated example communicates with a main memory, including a volatile memory 1714 and a non-volatile memory 1716, via a bus 1718. The volatile memory 1714 may be implemented by synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS® Dynamic Random Access Memory (RDRAM®), and / or any other type of random access memory device. The non-volatile memory 1716 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1714, 1716 is controlled by a memory controller.

[0146] The processor platform 1700 of the depicted example also includes an interface circuit 1720. The interface circuit 1720 may be implemented with any type of interface standard, such as an Ethernet interface, a Universal Serial Bus (USB), a Bluetooth interface, a Near Field Communication (NFC) interface, and / or a PCI Express interface.

[0147] In the depicted example, one or more input devices 1722 are connected to the interface circuit 1720. The input devices 1722 allow a user to enter data and / or commands into the processor 1712. The input devices may be implemented by, for example, audio sensors, microphones, cameras (still or video), keyboards, buttons, mice, touch screens, track pads, track balls, isopoints, and / or voice recognition systems.

[0148] One or more output devices 1724 are also connected to the interface circuitry 1720 of the illustrated example. The output device(s) 1724 may be implemented by, for example, a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touch screen, etc.), a haptic output device, a virtual reality headset, etc. Thus, the interface circuitry 1720 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0149] The interface circuitry 1720 of the depicted example also includes communications devices such as a transmitter, receiver, transceiver, modem, residential gateway, wireless access point, and / or network interface to facilitate the exchange of data with external machines (e.g., any type of computing device) via a network 1726. Communications may be via, for example, an Ethernet connection, a Digital Subscriber Line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-sight wireless system, a cellular telephone system, etc.

[0150] The processor platform 1700 of the depicted example also includes one or more mass storage devices 1728 for storing software and / or data. Examples of such mass storage devices 1728 include floppy disk drives, hard drive disks, compact disk drives, Blu-ray® disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0151] The machine-executable instructions 1732 of Figures 14 and 15 may be stored on the mass storage device 1728, the volatile memory 1714, the non-volatile memory 1716, and / or a removable non-transitory computer-readable storage medium, such as a CD or DVD.

[0152] From the above, it will be appreciated that exemplary methods, apparatus, and articles of manufacture for generating data packed video frames for immersive video bitstreams have been disclosed. The disclosed methods, apparatus, and articles of manufacture improve the efficiency of using computing devices by reducing the number of decoders required to decode a video frame, thereby expanding the pool of devices capable of decoding the video frame. In addition, the methods, apparatus, and articles of manufacture disclosed herein also enable different components of the same or different atlases to be included in the same "packed video frame", which may be coded at different frame rates to achieve further compression or support desired features depending on the application (e.g., packed video frames of texture atlases may be transmitted at 60 fps, while packed video frames of geometry atlases may be transmitted at 30 fps). In addition, the present invention enables the use of such encoders and decoders to operate on volumetric video data and / or immersive data without the need to modify standard encoders or decoders. Thus, the disclosed methods, apparatus, and articles of manufacture are directed to one or more improvements in computer functionality.

[0153] SUMMARY Exemplary methods, apparatus, systems, and articles of manufacture for generating packed video frames for volumetric and immersive video bitstreams are disclosed herein.

[0154] Further examples and combinations thereof include:

[0155] Example 1 includes a volumetric video encoding system, the volumetric video encoding system including a configuration determiner that creates a packed video frame layout, the packed video frame layout including regions in which video components are to be placed, and a packed video frame generator that forms a packed video frame including the video components placed in different ones of the regions of the packed video frame layout. Additionally, a packed video frame information generator is provided that generates packed video frame information that identifies characteristics of the packed video frame including at least one of: (i) identities of regions included in the packed video frame layout, (ii) respective types of video components included in each of the regions, or (iii) information identifying locations and dimensions of the regions. The volumetric video encoding system also includes a video encoder that encodes the packed video frame into an encoded bitstream, the encoded bitstream including the packed video frame information, the packed video frame information signaling inclusion of the packed video frame in the encoded bitstream.

[0156] Example 2 includes the volumetric video encoding system of example 1, wherein the type of video component includes at least one of auxiliary data, attribute data, geometry data, or occupancy data.

[0157] Example 3 includes the volumetric video encoding system of Example 1, wherein the regions of the packed video frame do not overlap, and the packed video frame information further includes information identifying respective types of video components associated with each of the regions, and at least one of an atlas identifier or an atlas style identifier.

[0158] Example 4 includes the volumetric video encoding system of Example 1, wherein the volumetric video data within the same packed video frame has the same sampling format and bit depth.

[0159] Example 5 includes the volumetric video encoding system of Example 1, further comprising a decoder number verifier that verifies that a constraint on the number of decoders to be used to decode the bitstream is not violated.

[0160] Example 6 includes the volumetric video encoding system of Example 1, further comprising: an independently decodable region identifier that identifies (i) a region of the packed video frame that can be independently decodable and (ii) a corresponding region identifier; a message generator that generates a message including the region identifier and indicating that the region corresponding to the region identifier can be independently decodable; and a multiplexer that multiplexes the message with the encoded bitstream for transmission to a decoder.

[0161] Example 7 includes one or more non-transitory computer-readable media including instructions that, when executed by at least one processor, cause the at least one processor to perform at least a procedure for creating a packed video frame layout. The packed video frame layout includes regions in which video components are to be placed. In addition, the processor forms a packed video frame including the video components placed in different ones of the regions of the packed video frame layout, and generates packed video frame information identifying characteristics of the packed video frame including at least one of: (i) identities of regions included in the packed video frame layout, (ii) respective types of video components included in each one of the regions, or (iii) information identifying locations and dimensions of the regions. The processor also encodes the packed video frame into an encoded bitstream including the packed video frame information. The packed video frame information signals the inclusion of the packed video frame in the encoded bitstream.

[0162] Example 8 includes the one or more non-transitory computer-readable media of Example 6, wherein the type of video component includes at least one of auxiliary data, attribute data, geometry data, or occupancy data.

[0163] Example 9 includes the one or more non-transitory computer-readable media of Example 6, wherein the regions of the packed video frame do not overlap and the packed video frame information further includes information identifying respective video component types associated with each of the regions and at least one of an atlas identifier or an atlas style identifier.

[0164] Example 10 includes the one or more non-transitory computer-readable media of Example 6, wherein the volumetric video data disposed within the same packed video frame has the same sampling format and bit depth.

[0165] Example 11 includes the one or more non-transitory computer-readable media of Example 6, further comprising a decoder number verification unit that verifies that a constraint on the number of decoders to be used to decode the bitstream is not violated.

[0166] Example 12 includes the one or more non-transitory computer-readable media of Example 6, wherein the instructions, when executed, further cause the processor to: (i) identify a region of the packed video frame that can be independently decoded and (ii) a corresponding region identifier; and generate a message including the region identifier and indicating that the region corresponding to the region identifier can be independently decoded. The instructions also cause the processor to multiplex the message with the encoded bitstream for transmission to a decoder.

[0167] Example 13 includes one or more non-transitory media as described in Example 6, the instructions, when executed, causing the at least one processor to identify regions of the packed video frame that can be independently decoded, generate a message identifying the regions of the packed video frame to be independently decoded, and multiplex the message with the encoded bitstream for transmission to a decoder.

[0168] Example 15 includes a volumetric video decoding system, comprising a parser for extracting packed video frame information from a sub-bitstream that includes a packed video frame, the packed video frame information including (i) a packed video frame layout, (ii) a region identifier that uniquely identifies a region of the packed video frame layout, and (iii) region location and dimension information, a video decoder for decoding the packed video frame of the sub-bitstream, and a mapping unit for mapping the region of the packed video frame to at least a portion of a volumetric video sequence to be rendered based on the packed video frame information.

[0169] Example 16 includes the volumetric video decoding system of Example 15, wherein each of the regions includes a respective video component type, the video component type including at least one of auxiliary data, attribute data, geometry data, or occupancy data.

[0170] Example 17 includes the volumetric video decoding system of Example 15, wherein the regions of the packed video frame do not overlap and the packed video frame information further includes information identifying respective video component types associated with each of the regions and at least one of an atlas identifier or an atlas style identifier.

[0171] Example 18 includes the volumetric video decoding system of Example 15, wherein the parsing unit further extracts a message from the sub-bitstream, the message including supplemental enhancement information having a region identifier and a region corresponding to the region identifier that is to be coded independently.

[0172] Example 19 includes the volumetric video decoding system of Example 18, wherein the size and location of the region to be independently coded is determined based on a tile index included in the packed video frame information.

[0173] Example 20 includes the volumetric video decoding system of Example 18, wherein a size and location of a region to be independently coded is determined based on a sub-picture identifier included in the packed video frame information.

[0174] Example 21 includes a volumetric video decoding system, comprising a video bitstream parsing unit that parses a parameter set associated with a video bitstream. The parameter set indicates a number of video decoders that will be used to decode the bitstream. A decoder validator determines whether a number of video decoders of the volumetric video decoding system is greater than or equal to the number of decoders indicated by the parameter set based on a profile, tier, and level associated with the volumetric video decoding system. In addition, a decoding controller stops decoding the sub-bitstream if the number of video decoders is less than the number of decoders indicated by the parameter set, and continues decoding the sub-bitstream if the number of video decoders is greater than or equal to the number of decoders indicated by the parameter set.

[0175] Example 22 includes a method of decoding volumetric video, comprising extracting packed video frame information from a sub-bitstream that includes a packed video frame, the packed video frame information including (i) a packed video frame layout, (ii) a region identifier that uniquely identifies a region of the packed video frame layout, and (iii) region location and dimension information, the method also comprising decoding the packed video frame of the sub-bitstream and mapping the region of the packed video frame to at least a portion of a volumetric video sequence to be rendered based on the packed video frame information.

[0176] Example 23 includes the volumetric video decoding method of Example 22, wherein each of the regions includes a respective video component type, the video component type including at least one of auxiliary data, attribute data, geometry data, or occupancy data.

[0177] Example 24 includes the volumetric video decoding method of Example 22, wherein the regions of the packed video frame do not overlap and the packed video frame information further includes information identifying respective video component types associated with each of the regions and at least one of an atlas identifier or an atlas style identifier.

[0178] Example 25 includes the volumetric video decoding method of Example 22, wherein the parsing unit further extracts a message from the sub-bitstream, the message including supplemental enhancement information having a region identifier and a region corresponding to the region identifier that is to be coded independently.

[0179] Example 26 includes the volumetric video decoding method of Example 22, in which a size and location of a region to be independently coded is determined based on a tile index included in the packed video frame information.

[0180] Example 27 includes the volumetric video decoding method of Example 22, wherein a size and location of a region to be independently coded is determined based on a sub-picture identifier included in the packed video frame information.

[0181] Example 28 is Example 1 including any one of Examples 2 to 6.

[0182] Example 29 is Example 7 including any one of Examples 8 to 13.

[0183] Example 30 is Example 15 including any one of Examples 16 to 17, 19, and 20.

[0184] Example 31 includes Example 21 and further includes any one of Examples 22 to 27.

[0185] Example 32 includes one or more non-transitory media containing instructions that, when executed, cause a processor to perform the method of any one of Examples 22-27.

[0186] Although certain example methods, apparatus and articles of manufacture have been disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all methods, apparatus and articles of manufacture that fairly fall within the scope of the claims of this patent.

[0187] The following claims are hereby incorporated into this detailed description by reference, with each claim standing on its own as a separate embodiment of this disclosure. [Other possible items] [Item 1] 1. A volumetric video coding system, comprising: a configuration determiner for creating a packed video frame layout, the packed video frame layout including regions into which video components will be placed; a packed video frame generator for forming a packed video frame including the video components disposed in different ones of the regions of the packed video frame layout; a packed video frame information generator for generating packed video frame information identifying characteristics of the packed video frame including at least one of: (i) identities of regions included in the packed video frame layout, (ii) respective types of video components included in each of the regions, or (iii) information identifying locations and dimensions of the regions; and a video encoder for encoding the packed video frame into an encoded bitstream, the encoded bitstream including the packed video frame information, the packed video frame information signaling inclusion of the packed video frame in the encoded bitstream; A volumetric video encoding system comprising: [Item 2] 2. The volumetric video encoding system of claim 1, wherein the type of video component includes at least one of auxiliary data, attribute data, geometry data, or occupancy data. [Item 3] 2. The volumetric video coding system of claim 1, wherein the regions of the packed video frame do not overlap, and the packed video frame information further includes information identifying respective types of video components associated with each of the regions, and at least one of an atlas identifier or an atlas style identifier. [Item 4] 2. The volumetric video encoding system of claim 1, wherein the volumetric video data within the same packed video frame has the same sampling format and bit depth. [Item 5] 2. The volumetric video coding system of claim 1, further comprising a decoder number verification unit for verifying that a constraint on the number of decoders to be used to decode the bitstream is not violated. [Item 6] an independently decodable region identifier that identifies (i) regions of the packed video frame that can be independently decodable and (ii) corresponding region identifiers; a message generator for generating a message including the region identifier and indicating that the region corresponding to the region identifier can be independently decoded; a multiplexer for multiplexing said message with said encoded bitstream for transmission to a decoder; Item 2. The volumetric video encoding system of item 1, further comprising: [Item 7] One or more non-transitory computer-readable media containing instructions that, when executed by at least one processor, cause the at least one processor to at least: creating a packed video frame layout, the packed video frame layout including regions in which video components will be placed; forming a packed video frame that includes the video components arranged in different ones of the regions of the packed video frame layout; generating packed video frame information identifying characteristics of the packed video frame including at least one of: (i) identities of regions included in the packed video frame layout; (ii) respective types of video components included in each of the regions; or (iii) information identifying locations and dimensions of the regions; encoding the packed video frame into an encoded bitstream, the encoded bitstream including the packed video frame information, the packed video frame information signaling inclusion of the packed video frame within the encoded bitstream; One or more non-transitory computer-readable media that cause [Item 8] 7. The one or more non-transitory computer-readable media of claim 6, wherein the type of video component includes at least one of auxiliary data, attribute data, geometry data, or occupancy data. [Item 9] 7. The one or more non-transitory computer-readable media of claim 6, wherein the regions of the packed video frame do not overlap, and the packed video frame information further includes information identifying a respective video component type associated with each of the regions and at least one of an atlas identifier or an atlas style identifier. [Item 10] 7. The one or more non-transitory computer-readable media of claim 6, wherein the volumetric video data arranged within the same packed video frame has the same sampling format and bit depth. [Item 11] 7. The one or more non-transitory computer-readable media of claim 6, further comprising a decoder number verifier that verifies that a constraint on the number of decoders to be used to decode the bitstream is not violated. [Item 12] The instructions, when executed, cause the processor to: (i) identifying regions of the packed video frame that can be independently decoded and (ii) corresponding region identifiers; generating a message including the region identifier and indicating that the region corresponding to the region identifier can be independently decoded; multiplexing said message with said encoded bitstream for transmission to a decoder; 7. The one or more non-transitory computer-readable media of item 6, further comprising: [Item 13] The instructions, when executed, cause the at least one processor to: identifying regions of the packed video frame that can be independently decoded; generating a message identifying the region of the packed video frame to be independently decoded; multiplexing said message with said encoded bitstream for transmission to a decoder; 7. One or more non-transitory media according to item 6, [Item 15] 1. A volumetric video decoding system, comprising: a parser for extracting packed video frame information from a sub-bitstream containing a packed video frame, the packed video frame information including: (i) a packed video frame layout, (ii) region identifiers that uniquely identify regions of the packed video frame layout, and (iii) region location and dimension information; a video decoder for decoding the packed video frames of the sub-bitstream; a mapping unit for mapping the region of the packed video frame onto at least a portion of a volumetric video sequence to be rendered based on the packed video frame information; A volumetric video decoding system comprising: [Item 16] Item 16. The volumetric video decoding system of item 15, wherein each of the regions includes a respective video component type, the video component type including at least one of auxiliary data, attribute data, geometry data, or occupancy data. [Item 17] 16. The volumetric video decoding system of claim 15, wherein the regions of the packed video frame do not overlap, and the packed video frame information further includes information identifying respective video component types associated with each of the regions, and at least one of an atlas identifier or an atlas style identifier. [Item 18] 16. The volumetric video decoding system of claim 15, wherein the parser further extracts a message from the sub-bitstream, the message including a region identifier, supplemental enhancement information having a region corresponding to the region identifier that is to be coded independently. [Item 19] 20. The volumetric video decoding system of claim 18, wherein the size and location of the region to be independently coded is determined based on a tile index included in the packed video frame information. [Item 20] Item 19. The volumetric video decoding system of item 18, wherein the size and location of the region to be independently coded is determined based on a sub-picture identifier included in the packed video frame information. [Item 21] 1. A volumetric video decoding system, comprising: a video bitstream parsing unit for parsing a parameter set associated with a video bitstream, the parameter set indicating a number of video decoders that will be used to decode the bitstream; a decoder verification unit that determines whether a number of video decoders of the volumetric video decoding system is greater than or equal to the number of decoders indicated by the parameter set based on a profile, a tier, and a level associated with the volumetric video decoding system; a decoding controller that stops decoding the sub-bitstream when the number of video decoders is less than the number of decoders indicated by the parameter set and continues decoding the sub-bitstream when the number of video decoders is greater than or equal to the number of decoders indicated by the parameter set; A volumetric video decoding system comprising: [Item 22] 1. A method for decoding volumetric video, comprising: extracting packed video frame information from a sub-bitstream containing a packed video frame, the packed video frame information including: (i) a packed video frame layout, (ii) region identifiers that uniquely identify regions of the packed video frame layout, and (iii) region location and dimension information; decoding the packed video frames of the sub-bitstream; mapping the region of the packed video frame based on the packed video frame information onto at least a portion of a volumetric video sequence to be rendered; 13. A method for decoding volumetric video, comprising: [Item 23] 23. The volumetric video decoding method of claim 22, wherein each of the regions includes a respective video component type, the video component type including at least one of auxiliary data, attribute data, geometry data, or occupancy data. [Item 24] 23. The volumetric video decoding method of claim 22, wherein the regions of the packed video frame do not overlap, and the packed video frame information further includes information identifying respective video component types associated with each of the regions, and at least one of an atlas identifier or an atlas style identifier. [Item 25] 23. The volumetric video decoding method of claim 22, wherein the parsing unit further extracts a message from the sub-bitstream, the message including a region identifier, supplemental enhancement information having a region corresponding to the region identifier that is to be coded independently. [Item 26] 23. The volumetric video decoding method of claim 22, wherein the size and location of the region to be independently coded is determined based on a tile index included in the packed video frame information. [Item 27] 23. The volumetric video decoding method of claim 22, wherein a size and location of a region to be independently coded is determined based on a sub-picture identifier included in the packed video frame information.

Claims

1. 1. A volumetric video coding system, comprising: a configuration determiner for creating a packed video frame layout, the packed video frame layout including regions into which video components will be placed; a packed video frame generator for forming a packed video frame including the video components disposed in different ones of the regions of the packed video frame layout; a packed video frame information generator for generating packed video frame information identifying characteristics of the packed video frame, the packed video frame information including at least one of an atlas identifier or an atlas style identifier; a video encoder for encoding the packed video frame into an encoded bitstream, the encoded bitstream including the packed video frame information, the packed video frame information signaling inclusion of the packed video frame in the encoded bitstream; A volumetric video encoding system comprising:

2. 1. A volumetric video coding system, comprising: a configuration determiner for creating a packed video frame layout, the packed video frame layout including regions into which video components will be placed; a packed video frame generator for forming a packed video frame including the video components disposed in different ones of the regions of the packed video frame layout; a packed video frame information generator for generating packed video frame information identifying characteristics of the packed video frame including at least one of: (i) identities of regions included in the packed video frame layout; (ii) respective types of video components included in each of the regions; or (iii) information identifying locations and dimensions of the regions; a video encoder for encoding the packed video frame into an encoded bitstream, the encoded bitstream including the packed video frame information, the packed video frame information signaling inclusion of the packed video frame in the encoded bitstream; and a decoder number verifier for verifying that a constraint on the number of decoders to be used to decode the bitstream is not violated; A volumetric video encoding system comprising:

3. 1. A volumetric video coding system, comprising: a configuration determiner for creating a packed video frame layout, the packed video frame layout including regions into which video components will be placed; a packed video frame generator for forming a packed video frame including the video components disposed in different ones of the regions of the packed video frame layout; a packed video frame information generator for generating packed video frame information identifying characteristics of the packed video frame including at least one of: (i) identities of regions included in the packed video frame layout; (ii) respective types of video components included in each of the regions; or (iii) information identifying locations and dimensions of the regions; a video encoder for encoding the packed video frame into an encoded bitstream, the encoded bitstream including the packed video frame information, the packed video frame information signaling inclusion of the packed video frame in the encoded bitstream; and (i) an independently decodable region identifier that identifies regions of the packed video frame that can be independently decoded and (ii) a corresponding region identifier; a message generator for generating a message including the region identifier and indicating that the region corresponding to the region identifier can be independently decoded; a multiplexer for multiplexing said message with said encoded bitstream for transmission to a decoder; A volumetric video encoding system comprising:

4. The volumetric video encoding system of claim 1, wherein the packed video frame information further includes at least one of: (i) identities of regions contained within the packed video frame layout, (ii) respective types of video components contained within each of the regions, or (iii) information identifying locations and dimensions of the regions.

5. The volumetric video coding system of any one of claims 1 to 4, wherein the types of video components include at least one of the following: auxiliary data, attribute data, geometry data, or occupancy data.

6. 6. The volumetric video coding system of claim 1 , wherein the regions of the packed video frame do not overlap, and the packed video frame information further comprises information identifying respective types of video components associated with respective ones of the regions.

7. The volumetric video encoding system of any one of claims 1 to 6, wherein volumetric video data within a same packed video frame has the same sampling format and bit depth.

8. At least one processor includes at least creating a packed video frame layout, the packed video frame layout including regions in which video components will be placed; forming a packed video frame that includes the video components arranged in different ones of the regions of the packed video frame layout; generating packed video frame information identifying characteristics of the packed video frame, the packed video frame information including at least one of an atlas identifier or an atlas style identifier; encoding the packed video frame into an encoded bitstream, the encoded bitstream including the packed video frame information, the packed video frame information signaling inclusion of the packed video frame within the encoded bitstream; A computer program for executing the following:

9. At least one processor includes at least creating a packed video frame layout, the packed video frame layout including regions in which video components will be placed; forming a packed video frame that includes the video components arranged in different ones of the regions of the packed video frame layout; generating packed video frame information identifying characteristics of the packed video frame including at least one of: (i) identities of regions included in the packed video frame layout; (ii) respective types of video components included in each of the regions; or (iii) information identifying locations and dimensions of the regions; encoding the packed video frame into an encoded bitstream, the encoded bitstream including the packed video frame information, the packed video frame information signaling inclusion of the packed video frame within the encoded bitstream; verifying that a constraint on the number of decoders that will be used to decode the bitstream is not violated; A computer program for executing the following:

10. At least one processor includes at least creating a packed video frame layout, the packed video frame layout including regions in which video components will be placed; forming a packed video frame that includes the video components arranged in different ones of the regions of the packed video frame layout; generating packed video frame information identifying characteristics of the packed video frame including at least one of: (i) identities of regions included in the packed video frame layout; (ii) respective types of video components included in each of the regions; or (iii) information identifying locations and dimensions of the regions; encoding the packed video frame into an encoded bitstream, the encoded bitstream including the packed video frame information, the packed video frame information signaling inclusion of the packed video frame within the encoded bitstream; (i) identifying regions of the packed video frame that can be independently decoded and (ii) corresponding region identifiers; generating a message including the region identifier and indicating that the region corresponding to the region identifier can be independently decoded; multiplexing said message with said encoded bitstream for transmission to a decoder; A computer program for executing the following:

11. At least one processor includes at least creating a packed video frame layout, the packed video frame layout including regions in which video components will be placed; forming a packed video frame that includes the video components arranged in different ones of the regions of the packed video frame layout; generating packed video frame information identifying characteristics of the packed video frame including at least one of: (i) identities of regions included in the packed video frame layout; (ii) respective types of video components included in each of the regions; or (iii) information identifying locations and dimensions of the regions; encoding the packed video frame into an encoded bitstream, the encoded bitstream including the packed video frame information, the packed video frame information signaling inclusion of the packed video frame within the encoded bitstream; identifying regions of the packed video frame that can be independently decoded; generating a message identifying the region of the packed video frame to be independently decoded; multiplexing said message with said encoded bitstream for transmission to a decoder; A computer program for executing the following:

12. The computer program of claim 8, wherein the packed video frame information further includes at least one of: (i) identities of regions contained within the packed video frame layout, (ii) respective types of video components contained within each of the regions, or (iii) information identifying locations and dimensions of the regions.

13. A computer program product according to any one of claims 8 to 12, wherein the types of video components include at least one of the following: auxiliary data, attribute data, geometry data, or occupancy data.

14. 14. The computer program product of claim 8, wherein the regions of the packed video frame do not overlap, and wherein the packed video frame information further comprises information identifying respective video component types associated with respective ones of the regions.

15. A computer program product according to any one of claims 8 to 14, wherein volumetric video data located within the same packed video frame have the same sampling format and bit depth.

16. 1. A volumetric video decoding system, comprising: a parser for extracting packed video frame information from a sub-bitstream containing a packed video frame, the packed video frame information including: (i) a packed video frame layout, (ii) region identifiers that uniquely identify regions of the packed video frame layout, and (iii) region location and dimension information; a video decoder for decoding the packed video frames of the sub-bitstream; a mapping unit for mapping the region of the packed video frame onto at least a portion of a volumetric video sequence to be rendered based on the packed video frame information; A volumetric video decoding system comprising:

17. 20. The volumetric video decoding system of claim 16, wherein each of the regions includes a respective video component type, the video component type including at least one of auxiliary data, attribute data, geometry data, or occupancy data.

18. 18. The volumetric video decoding system of claim 16 or claim 17, wherein the regions of the packed video frame do not overlap, and the packed video frame information further comprises information identifying a respective video component type associated with each of the regions, and at least one of an atlas identifier or an atlas style identifier.

19. The volumetric video decoding system of any one of claims 16 to 18, wherein the parser further extracts a message from the sub-bitstream, the message including a region identifier, supplemental enhancement information having a region corresponding to the region identifier that is to be coded independently.

20. 20. The volumetric video decoding system of claim 19, wherein a size and location of a region to be independently coded is determined based on a tile index included in the packed video frame information.

21. 20. The volumetric video decoding system of claim 19, wherein a size and location of a region to be independently coded is determined based on a sub-picture identifier included in the packed video frame information.

22. 1. A method for decoding volumetric video, comprising: extracting packed video frame information from a sub-bitstream that includes a packed video frame, the packed video frame information including: (i) a packed video frame layout, (ii) region identifiers that uniquely identify regions of the packed video frame layout, and (iii) region location and dimension information; decoding the packed video frames of the sub-bitstream; mapping the region of the packed video frame based on the packed video frame information onto at least a portion of a volumetric video sequence to be rendered; 13. A method for decoding volumetric video, comprising:

23. 23. The method of claim 22, wherein each one of the regions includes a respective video component type, the video component type including at least one of auxiliary data, attribute data, geometry data, or occupancy data.

24. 24. The method of claim 22 or claim 23, wherein the regions of the packed video frame do not overlap, and the packed video frame information further comprises information identifying a respective video content type associated with each of the regions, and at least one of an atlas identifier or an atlas style identifier.

25. A method according to any one of claims 22 to 24, wherein the parsing unit further extracts a message from the sub-bitstream, the message including a region identifier, supplemental enhancement information having a region corresponding to the region identifier to be coded independently.

26. 26. The method of claim 25, wherein sizes and locations of regions to be independently coded are determined based on tile indexes included in the packed video frame information.

27. 26. The method of claim 25, wherein sizes and locations of regions to be independently coded are determined based on sub-picture identifiers included within the packed video frame information.

28. One or more non-transitory computer readable media storing a computer program according to any one of claims 8 to 15.

Images