Carriage of coded base mesh and displacement data of video-based dynamic mesh coding in isobmff media containers

Abstract

Examples of the systems and methods discussed herein provide an encapsulation scheme that may contain a base mesh bitstream, sub-mesh(es), and / or displacement information, providing efficient storage and transmission of the content in an extensible container, such as an ISO Base Media File Format (ISOBMFF) container. In particular, examples of the systems and methods discussed herein provide a generic and scalable design to support carrying sub-components of a video-based dynamic mesh coding (V-DMC) or similar bitstream, such as a base mesh, sub-meshes, and displacement information, in an ISOBMFF media container.

Description

CARRIAGE OF CODED BASE MESH AND DISPLACEMENT DATA OF VIDEO-BASED DYNAMIC MESH CODING IN ISOBMFF MEDIA CONTAINERSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 622,977, filed January 19, 2024, the contents of which are incorporated herein by reference.TECHNICAL FIELD

[0002] The present systems, devices and methods generally relate to video encoding and transport. In particular, the present systems, devices and methods relate to carriage of coded base mesh and displacement data for videobased dynamic mesh coding.BACKGROUND

[0003] New three-dimensional video encoding standards include video-based dynamic mesh coding (V-DMC), promulgated by Working Group 7 of the Motion Pictures Experts Group (MPEG) as ISO / IEC 23090-29, which is an extension of the Visual Volumetric Video-based Coding (V3C) specification (ISO / IEC 23090-5). Communicating V- DMC and / or V3C data between an encoder and decoder or other such devices raises problems due to the variety of data and potential for different data sub-types (e.g., two dimensional data, three-dimensional data, mesh data, attribute data, etc.) as well as the large bandwidth and throughput typically required. Existing file types may be inadequate to describe these varied forms of data and / or may be bloated and inefficient.SUMMARY

[0004] In some aspects, the present disclosure is directed to examples of systems and methods for extensible and scalable encapsulation of mesh and sub-mesh information for V-DMC or V3C video data in an extensible container, such as an ISO Base Media File Format (ISOBMFF) container. In particular, examples of the systems and methods discussed herein provide a generic and scalable design to support carrying sub-components of a V-DMC or similar bitstream, such as a base mesh, sub-meshes, and displacement information, in an ISOBMFF media container.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] A more detailed understanding may be had from the following description, given by way of example in conjunction with the accompanying drawings, wherein like reference numerals in the figures indicate like elements, and wherein:

[0006] FIG. 1 is a schematic of an implementation of an ISOBMFF media container for V-DMC bitstreams, according to some examples;

[0007] FIG. 2 is a schematic of another implementation of an ISOBMFF media container for V-DMC bitstreams, according to some examples;

[0008] FIG. 3 illustrates a block diagram of a system within which aspects of the present examples may be implemented;

[0009] FIG. 4 illustrates a block diagram of an implementation of a video-based point cloud compression (V-PCC) encoder; and

[0010] FIG. 5 illustrates a block diagram of an implementation of a V-PCC decoder.DETAILED DESCRIPTION

[0011] The following ISO / IEC specification (s) and standard(s), including any draft versions of such standard(s), are hereby incorporated herein by reference in their entirety and are made part of the present disclosure for all purposes: ISO / IEC 23090-5 "Information technology - Coded representation of immersive media - Part 5: Visual volumetric video-based coding (V3C) and video-based point cloud compression (V-PCC)”; ISO / IEC 23090-10, “Information technology - Coded representation of immersive media - Part 10: Carriage of visual volumetric videobased coding data”; ISO / IEC 23090-29 “Information technology - Coded representation of immersive media - Part 29: Video-based dynamic mesh coding (V-DMC); and ISO / IEC 14496-12, “Information technology - Coding of Audio- Visual Objects -- Part 12: ISO Base Media File Format”. Although this disclosure may reference aspects of these standard(s), the disclosure is in no way limited by these standard(s).

[0012] Video-based dynamic mesh coding (V-DMC), which extends the Visual Volumetric Video-based Coding (V3C) specification, provides for encoding, communication, and decoding of three-dimensional video bit-streams. These bit-streams may include subcomponents including sequences of dynamic meshes, which may represent entities such as objects, environmental features, people, animals, machines, etc. These dynamic meshes may be represented by a number of V3C components, including a base mesh, a set of displacements, 2D representation of the attributes, and an atlas.

[0013] In some examples, a base mesh component may include a simplified low-resolution approximation of the original mesh The base mesh component can be encoded using any mesh codec. For example, in some examples, a static mesh codec may be used to code static mesh instances, and a motion decoder may be used to temporally transform the static mesh over a period (e.g., as an entity moves over time). In some examples, a base mesh codec may delegate the compression process to a static mesh codec and a motion codec. In many examples, the V3C codec may be codec-agnostic regarding the base mesh - that is, any static mesh / motion codec can be used to code the base mesh bitstream.

[0014] In some examples, a displacement component provides displacement vectors that can be encoded as a V3C geometry video component using any suitable video codec. In some examples, the profile may indicate that the displacement component is encoded using arithmetic coding, video coding, or any other suitable type of coding.

[0015] In some examples, an atlas component provides information to a V3C decoding and / or rendering system on how to perform inverse reconstruction. For example, the atlas component may specify how to subdivide the base mesh and how to apply the displacement vectors to the subdivided mesh vertices, and how to apply attributes to the reconstructed mesh.

[0016] In some examples, a sub-mesh may be represented by a set of vertices, their connectivity, and the associated mesh attributes. In many examples, sub-meshes can be decoded completely independently of each other. Each base mesh may include one or more sub-meshes, each of which may be associated with a unique identifier in a base mesh sub-bitstream frame parameter set. Similarly in many examples, each patch in an atlas tile may be associated with the corresponding sub-mesh identifier.

[0017] Communicating V-DMC and / or V3C data between an encoder and decoder or other such devices raises problems due to the variety of data and potential for different data sub-types (e.g., two dimensional data, three- dimensional data, mesh data, attribute data, etc.) as well as the large bandwidth and throughput typically required. Existing file types may be inadequate to describe these varied forms of data and may lack scalability or extensibility.

[0018] The present description provides examples of an encapsulation scheme that may contain a base mesh bitstream and expose its high-level features such as a sub-mesh, providing efficient storage and transmission of the base mesh content. Examples of the encapsulation scheme may also contain arithmetic-coded displacement information, which may fulfill the requirements of a V-DMC profile in which displacement information is arithmetically coded rather than video-based. By using flexible encapsulation, access to the sub-meshes in the bitstream is enabled, and the bitstream may be multiplexed along with displacement bitstreams, atlas and base mesh bitstreams in an extensible container, such as an ISO Base Media File Format (ISOBMFF) container. In particular, examples of the systems and methods discussed herein provide a generic and scalable design to support carrying subcomponents of a V-DMC or similar bitstream, such as a base mesh, sub-meshes, and displacement information, in an ISOBMFF media container.

[0019] For clarity, sub-mesh information are part of the base-mesh sub-stream. The sub-streams are then encapsulated (at the encoder / packager side) by structuring them in a media container as separate tracks (in the case of timed media). Generally, a description is provided for the base-mesh and displacement sub-streams. In the case of a base-mesh sub-stream that contains multiple sub-meshes, methods to store these sub-meshes in separate submesh tracks that enable a player to only select the tracks for the sub-meshes it needs or wants are provided.

[0020] A system and method for communication of mesh and displacement data are provided. The system and method include receiving, by an encoder of a device, a plurality of sub-streams of an item of media content, theplurality of sub-streams including an atlas sub-stream, one or more video sub-streams, a base-mesh sub-stream, one or more sub-mesh sub-streams, and a displacement sub-stream; generating, by the encoder, a track reference identifying an association between the base-mesh sub-stream and the one or more sub-mesh sub-streams, wherein the track reference is encapsulated in a header of a first sub-stream of the plurality of sub-streams; and generating, by the encoder, the plurality of sub-streams in a container file format, wherein a header of a first sub-stream generated in the container file format includes a reference identifier of each of the one or more video sub-streams, the base-mesh sub-stream, the one or more sub-mesh sub-streams, and the displacement sub-stream. The container file format may be an International Standards Organization Base Media File Format (ISOBMFF). The base-mesh sub-stream, the one or more sub-mesh sub-streams, or the displacement sub-stream may be timed data in synchronization with the atlas sub-stream and one or more video sub-streams. The base-mesh sub-stream, the one or more sub-mesh sub-streams, or the displacement sub-stream may be un-timed data. At least one of the basemesh sub-stream, the one or more sub-mesh sub-streams, or the displacement sub-stream may be timed data in synchronization with the atlas sub-stream and one or more video sub-streams, and at least one other of the basemesh sub-stream, the one or more sub-mesh sub-streams, or the displacement sub-stream may be un-timed data. The sub-streams may correspond to tracks in the generated container file format. A header of the generated basemesh sub-stream comprises an identifier of each sub-mesh sub-stream. The base-mesh sub-stream may identify a plurality of base meshes of the item of media content, each base mesh associated with a different time period within the item of media content. The generated one or more sub-mesh sub-streams may be identified in a header of the atlas sub-stream as members of a track group.

[0021] A device for communication of mesh and displacement data is also disclosed. The device include an encoder; and an input / output (I / O) device operably coupled to the encoder. The device may be configured to receive a plurality of sub-streams of an item of media content, the plurality of sub-streams including an atlas sub-stream, one or more video sub-streams, a base-mesh sub-stream, one or more sub-mesh sub-streams, and a displacement substream The device may be further configured to generate a track reference identifying an association between the base-mesh sub-stream and the one or more sub-mesh sub-streams, wherein the track reference is encapsulated in a header of a first sub-stream of the plurality of sub-streams. The device may be further configured to generate the plurality of sub-streams in a container file format, wherein a header of a first sub-stream generated in the container file format includes a reference identifier of each of the one or more video sub-streams, the base-mesh sub-stream, the one or more sub-mesh sub-streams, and the displacement sub-stream. The container file format may be an International Standards Organization Base Media File Format (ISOBMFF). The base-mesh sub-stream, the one or more sub-mesh sub-streams, or the displacement sub-stream may be timed data in synchronization with the atlas sub-stream and one or more video sub-streams. The base-mesh sub-stream, the one or more sub-mesh substreams, or the displacement sub-stream may be un-timed data. At least one of the base-mesh sub-stream, the one or more sub-mesh sub-streams, or the displacement sub-stream may be timed data in synchronization with the atlassub-stream and one or more video sub-streams, and at least one other of the base-mesh sub-stream, the one or more sub-mesh sub-streams, or the displacement sub-stream may be un-timed data. The sub-streams may correspond to tracks in the generated container file format. A header of the generated base-mesh sub-stream comprises an identifier of each sub-mesh sub-stream. A header of one sub-mesh track comprises an identifier of each sub-mesh sub-stream. The base-mesh sub-stream may identify a plurality of base meshes of the item of media content, each base mesh associated with a different time period within the item of media content. The generated one or more sub-mesh sub-streams may be identified in a header of the atlas sub-stream as members of a track group.

[0022] Additional methods are also described including a method for communication of mesh and displacement data. This method includes receiving, by a decoder of a device, information including a container file format, the container format including a plurality of sub-streams of an item of media content, the plurality of sub-streams including an atlas sub-stream, one or more video sub-streams, a base-mesh sub-stream, one or more sub-mesh substreams, and a displacement sub-stream; decoding, by the decoder, the information including the container file format including the plurality of sub-streams; parsing the plurality of sub-streams from the container file format, wherein a header of a first sub-stream generated in the container file format includes a reference identifier of each of the one or more video sub-streams, the base-mesh sub-stream, the one or more sub-mesh sub-streams, and the displacement sub-stream; parsing a track reference encapsulated in a header of a first sub-stream of the plurality of sub-streams to identify an association between the base-mesh sub-stream and the one or more sub-mesh substreams; and render the media content using the track reference. The rendering of the media content may occur on an external display.

[0023] Within the ISO / IEC 14496 (MPEG-4) standard there are several parts that define file formats for the storage of time-based media. These parts are all based on and derived from the ISO Base Media File Format (ISOBMFF), which is a structural, media-independent definition. FIG. 1 illustrates a schematic of an implementation of an ISOBMFF media container 100 for timed data of V-DMC bitstreams. ISOBMFF 100 contains structural and media data information mainly for timed presentations of media data such as audio, video, etc. There is also support for un-timed data, such as meta-data at different levels within the file structure. The logical structure of the file is of a movie that in turn contains a set of time-parallel tracks 110 (e.g., track 1 110i, track 2 11O2, track 3 11O3, collectively referred to a tracks 110). The time structure of the file is that the tracks 110 contain sequences of samples in time, and those sequences are mapped into the timeline of the overall movie. ISOBMFF 100 is based in the concept of box-structured files. A box-structured file consists of a series of boxes (sometimes called atoms), which have a size and a type. The types are 32-bit values and usually chosen to be four printable characters, also known a four- character code (4CC) (e.g., “v3vg” for a V-DMC geometry bitstream, “v3va” for a V-DMC attribute bitstream, in the illustrated implementation). Un-timed data may be contained in a metadata box, at the file level, or attached to the movie box or one of the streams of timed data, called tracks, within the movie.

[0024] Among the top-level boxes within an ISOBMFF container 100 is the MovieBox ('moov') which contains metadata for the continuous media streams present in the file. These metadata are signaled within the hierarchy of boxes in the MovieBox, e.g., within the TrackBox ('trak'). A track represents a continuous media stream that is present in the file. The media stream itself consists of a sequence of samples, such as audio or video access units of an elementary media stream, and are enclosed within a MediaDataBox ('mdat') that is present at the top-level of the container 100. The metadata for each track includes a list of sample description entries, each providing the coding or encapsulation format used in the track and the initialization data for processing that format. Each sample may be associated with one of the sample description entries of the track. ISO / I EC 14496-12 provides a tool for defining an explicit timeline map for each track. This is known as an edit list and is signaled using an EditListBox with the following syntax, where each entry defines part of the track time-line: by mapping part of the composition timeline, or by indicating ‘empty’ time (portions of the presentation timeline that map to no media, an 'empty' edit): aligned(8) class EditListBox extends FullBox('elst', version, flags) { unsigned int(32) entry _count; for (1=1 ; I <= entry _count; i++) { if (version==1) { unsigned int(64) edi t_duration; int(64) mediajime;} else { II version==0 unsigned int(32) edi t_duration; int(32) mediajime;} int(16) media_ratejnteger; int(16) media_rate_fraction = 0;} }

[0025] The ISO / IEC 23090-10 specification also includes extensions to the ISO / IEC 14496-12 specification to enable encapsulation (carriage) of V3C bitstreams in ISOBMFF media containers.

[0026] FIG. 2 is a schematic of another implementation of an ISOBMFF media container 200 for V-DMC bitstreams with additional tracks 210 (e.g., track 1 210i, track 2 2102, track 3 2103, track 4 2104, track 5 210s, track 6 210e, collectively referred to as tracks 210) defined for a base-mesh bitstream, sub-mesh bitstream, and displacement bitstream, according to some examples. Although shown with one each of a base-mesh 2104, sub-mesh 2105, and displacement bitstream 210e, in some examples, greater or fewer numbers of these may be present. For example, in one implementation, the media container 200 may include a base-mesh bitstream and a plurality of sub-mesh bitstreams (e.g., for different sub-meshes of the base mesh).

[0027] In some examples, to carry the data of the base mesh sub-bitstream of a V-DMC bitstream, a Base Mesh track 2104 is included. The base mesh track 2104iriay have a sample entry of type BaseMeshSampleEntry, which derives from the VolumetricVisualSampleEntry, and has an ISOBMFF box type identified by a four-character code (4CC) such as ‘vbm1 ', ‘vdmb’, or 'vbmg' or a similar identifier. In some examples, a BaseMeshSampleEntry may contain a BaseMeshConfigurationBox comprising configuration information needed to initial the base mesh decoder. The decoder initialization information may be signaled in a BaseMeshDecoderConfigurationRecord structure inside the BaseMeshConfigurationBox.

[0028] In some examples, the BaseMeshSampleEntry and BaseMeshConfigurationBox may be defined as follows: aligned(8) class BaseMeshSampleEntry(type) extends VolumetricVisualSampleEntry { BaseMeshConfigurationBoxQ; aligned(8) class BaseMeshConfigurationBox extends FullBox('bmcC, version, flags=0) { BaseMeshDecoderConfigurationRecord(); aligned(8) class BaseMeshDecoderConfigurationRecord {II version 0 unsigned int(3) unit_size_precision_bytes_minus1; unsigned int(8) num_of_setup_unit_arrays; for (int j=0; j < num_of_setup_unit_arrays; j++) { unsigned int(1) array_completeness; bit(1) reserved = 0; unsigned int(6) nal_unit_type; unsigned int(8) num_nal_units; for (int i=0; i < num_nal_units; i++) { unsigned int(16) setup_unit_length; / / bmesh_nal_unit(size) as defined in ISO / IEC 23090-29bmesh_nal_unit setup_unit(setup_unit_length);}II additional fields

[0029] The semantics of the fields of the BaseMeshConfigurationBox are as follows:

[0030] unit_size_precision_bytes_minus1 / plus1 specifies the precision, in bytes, of the sample stream network abstraction layer (NAL) unit to which this configuration record applies. The value of this field may be conditional on the 4CC-code of the sample entry. For example, in some examples, for base mesh tracks unit_size_precision_bytes_minus1 may be equal or equivalent to ssnh_unit_size_precision_bytes_minus1 in the sample_stream_nal_header().

[0031] num_of_setup_unit_arrays indicates the number of arrays of base mesh NAL units of the indicated type(s)

[0032] array_completeness, when equal to 1, may indicate that all base mesh NAL units of the given type are in the following array and none are in the stream; and when equal to 0 may indicate that additional base mesh NAL units of the indicated type may be in the stream. The default and permitted values may be constrained by the sample entry name

[0033] nal_unit_type indicates the type of the base mesh NAL units in the following array, which may all be of that type, or may be a mix of types in some examples (in which case, multiple nal_unit_type entries may be present, or a type may be predefined as a mix). The types may be encoded as a predetermined value, such as those defined in ISO / IEC 23090-29. In some examples, the field may be restricted to take one of the values indicating a NAL_BMSPS, NAL_BMFPS, NAL_PREFIX_ESEI, NAL_PREFIX_NSEI, NAL_SUFFIX_ESEI, or NAL_SUFFIX_NSEI base mesh NAL unit.

[0034] num_nal_units indicates the number of base mesh NAL units of type nal_unit_type included in the configuration record for the stream to which this configuration record applies.

[0035] setup_unit_length indicates the size, in bytes, of the setup_unit field. The length field includes the size of both the NAL unit header and the NAL unit payload but in many examples does not include the length of the length field itself.

[0036] setup_unit may contain a NAL unit according to related nal_unit_type. When present in setupjjnit, a NAL_PREFIX_ESEI, NAL_PREFIX_NSEI, NAL_SUFFIX_ESEI, or NAL_SUFFIX_NSEI may contain supplemental enhancement information (SEI) messages of a ‘declarative’ nature, that is, those that provide information about the stream as a whole. An example of such an SEI could be a user-data SEI.

[0037] In some examples, the semantics of the fields inherited from the VolumetricVisualSampleEntry are as follows

[0038] compressorname in the base class VolumetricVisualSampleEntry indicates the name of the compressor used and may include padding or a length indicator. For example, given a value “\012VBM Coding”, the first byte is a count of the remaining bytes, here represented by \012, which (being octal 12) is 10 (decimal), the number of bytes in the rest of the string

[0039] In some examples, a base mesh 2104 contains one or more sub-meshes 2105 that are independently decodable. When the base mesh sub-bitstream 2105 contains more than one sub-mesh, the data for each sub-mesh may be carried separate tracks in the ISOBMFF container 200. For example, while FIG. 2 only shows one sub-mesh track 2105, in some examples, multiple sub-mesh tracks may be included (e.g., one for each sub-mesh) This is particularly useful when the spatial partitioning of the base mesh into sub-meshes does not change for the duration of the dynamic mesh sequence. These tracks may be referred to as Sub-Mesh tracks and may include a SubMeshSampleEntry with the 4CC 'vbsT (and, in some examples, 'vbs2' for a second sub-mesh track, 'vbs3‘ for a third sub-mesh track, etc., although other identifiers may be utilized). A Sub-Mesh track 2105 carries sub-mesh data for one or more sub-meshes in the base mesh sub-bitstream. The identifiers for all the sub-meshes carried by a SubMesh track (e.g , when multiple sub-meshes are used or when the sub-meshes change over the course of the dynamic mesh sequence) may be signaled in the SubMeshConfigurationBox in the SubMeshSampleEntry.

[0040] In some examples, a track reference type such as ‘vlvb’ may be utilized to link the Base Mesh track 2104, which contains the decoder configuration information necessary for instantiating and initializing the base mesh decoder, to the associated Sub-Mesh tracks 2105. In some examples, a TrackReferenceTypeBox with the reference type 'vlvb' may be added to a TrackReferenceBox within the TrackBox of the Base Mesh track 2IO4. The TrackReferenceTypeBox may contain an array or string of trackJDs comprising the identifiers for the referenced Sub-Mesh tracks. In another embodiment, another track reference may be added from the Sub-Mesh track 2105 to the Base Mesh track 2104.

[0041] In some examples, the SubMeshSampleEntry may be defined as follows:Sample Entry Type: 'vbsTContainer: SampleDescriptionBox Mandatory: YesQuantity: One or more aligned(8) class SubMeshSampleEntry('vbsT) extends VolumetricVisualSampleEntry { SubMeshConfigurationBoxO;aligned(8) class SubMeshConfigurationBox extends FullBox('smcC, version, flags=O) { unsigned int(3) unit_size_precision_bytes_minus1 ; bit(7) reserved = 0; unsigned int(6) num_submeshes; for(int i=0; i < num_submeshes; i++){ unsigned int(16) submesh Jd;II additional sub-mesh configuration information

[0042] In some examples, the semantics of the fields of SubMeshConfigurationBox may be defined as follows:

[0043] unit_size_precision_bytes_minus1 / pl us 1 may specify the precision, in bytes, of the sample stream NAL unit to which the sample entry containing this configuration box applies. The value of this field may be equal to ssnh_unit_size_precision_bytes_minus1 in sample_stream_nal_header() for the base mesh component bitstream.

[0044] num_submeshes may indicate the number of sub-meshes carried in this track.

[0045] submeshjd may specify the sub-mesh ID of the sub-mesh present in the track. The value of submeshjd may be equal to the value of the corresponding bmsi_submesh_id syntax element in bmesh_sub_mesh_information(), defined in ISO / IEC 23090-29.

[0046] compressorname in the base class VolumetricVisualSampleEntry may indicate the name of the compressor used (along with padding or other indicators if necessary). For example, given the value "\014VBM Submesh", the first byte is a count of the remaining bytes, here represented by \014, which (being octal 14) is 12 (decimal), the number of bytes in the rest of the string.

[0047] A sync sample in a Base Mesh track 2104 or a Sub-Mesh track 210s is a sample that contains an intra random access point (IRAP) coded base mesh access unit as defined in ISO / IEC 23090-29. Base mesh parameter sets and SEI messages can be repeated, if needed, at a sync sample to allow for random access.

[0048] A V-DMC decoder typically requires information from the atlas sub-bitstream 210i to reconstruct the base mesh. In order efficiently access and decode certain sub-meshes from the base mesh sub-bitstream 2104, in some examples, the atlas information 210i pertaining to these sub-meshes may be placed in independent atlas tiles. To signal the association between a Sub-Mesh track 210s and a V3C Atlas Tile track containing the atlas information 210i for that sub-mesh, a new track group VDMCSubMeshTrackGroupBox which extends the TrackGroupTypeBox, defined in ISO / IEC 14496-12, may be defined as follows:Box Type: 'vdmg'Container: TrackGroupBoxMandatory: NoQuantity: Zero or more aligned(8) class VDMCSubMeshTrackGroupBox extends TrackGroupTypeBox('potg') {II track jroupJd is inherited from TrackGroupTypeBox unsigned int(16) submesh Jd; unsigned int(16) numjiles; for(int i=0; i < numjiles; I++) { unsigned int(16) tilejd;}

[0049] The semantics of the fields of the VDMCSubMeshTrackGroupBox may be as follows:

[0050] submesh Jd may comprise the identifier for the sub-mesh for this track group instance. The value of submeshjd may be equal to the value of the corresponding bmsi_submeshjd syntax element in bmesh_sub_meshjnformation(), defined in ISO / IEC 23090-29.

[0051] numjiles may indicate the number of atlas tiles associated with sub-mesh for this track group instance.

[0052] tilejd may specify the atlas tile ID of an atlas tile carrying patches associated with the sub-mesh for this track group instance.

[0053] In another embodiment, a VDMCSubMeshGroupBox may represent an entity group comprising each group of V-DMC tracks belonging to the same sub-mesh. The definition of VDMCSubMeshGroupBox is as follows:Box Type: 'vdmg'Container: GroupListBox in a MetaBox on Movie levelMandatory: NoQuantity: Zero or more aligned (8) class VDMCSubMeshGroupBoxQ extends EntityToGroupBoxfvdmg', version=0, flags) { unsigned int(16) submeshjd; unsigned int(16) numjiles;for(int i=0; i < numjiles; I++) { unsigned int(16) tilejd;

[0054] The semantics of the fields of the VDMCSubMeshGroupBox may be as follows:

[0055] submeshjd may comprise the identifier for the sub-mesh for this entity group instance. The value of submeshjd may be equal to the value of the corresponding bmsi_submesh_id syntax element in bmesh_sub_mesh_information(), defined in ISO / IEC 23090-29.

[0056] numjiles may indicate the number of atlas tiles associated with sub-mesh for this entity group instance.

[0057] tilejd may specify the atlas tile ID of an atlas tile carrying patches associated with the sub-mesh for this entity group instance.

[0058] A V-DMC Displacement track 210e may carry data from the V-DMC displacement sub-bitstream. If the V- DMC displacement component is coded using a traditional 2D video codec, the Displacement track 210e may comprise a restricted video track that includes a VisualSampleEntry with the 4CC 'resv' and contains a RestrictedSchemelnfoBox.

[0059] In some examples, displacement tracks 210e which are arithmetically coded may be carried as restricted volumetric visual tracks. These tracks use a generic restricted VolumetricVisualSampleEntry with the 4CC Tes3' which includes a RestrictedSchemelnfoBox (e.g., as shown in the implementation of FIG. 2).

[0060] In many examples, the following additional requirements apply to both video-coded and arithmetically coded Displacement tracks 210e: a SchemeTypeBox is present in RestrictedSchemelnfoBox and the schemejype of that box is set to 'vvbm'; and a SchemelnformationBox is present in RestrictedSchemelnfoBox and contains aV3CUnitHeaderBox

[0061] In the track header of the Displacement track 210e, the trackjnjnovie flag may be set to 0, to indicate that this track should not be presented alone.

[0062] In some examples, a displacement track sample entry may also include a VDMCDisplacementConfigurationBox that is defined as follows: aligned(8) class VDMCDisplacementConfigurationBox extends FullBox('vdcC, version, flags=0) { unsigned int(3) unit_size_precision_bytes_minus1 ; bit(5) reserved = 0;VDMCDisplacementDecoderConfigurationRecord();II additional displacement configuration information aligned(8) class VDMCDisplacementDecoderConfigurationRecord {II version 0 unsigned int(3) unit_size_precision_bytes_minus1; unsigned int(8) num_of_setup_unit_arrays; for (int j=0; j < num_of_setup_unit_arrays; j++) { unsigned int(1) array_completeness; bit(1) reserved = 0; unsigned int(6) nal_unit_type; unsigned int(8) num_nal_units; for (int i=0; i < num_nal_units; i++) { unsigned int(16) setup_unit_length; / / displ_nal_unit(size) as defined in ISO / I EC 23090-29 displ_nal_unit setup_unit(setup_unit_length);}II additional fields

[0063] The semantics of the fields of the VDMCDisplacementConfigurationBox may be as follows:

[0064] unit size preci tion bytes minusl plus 1 specifies the precision, in bytes, of the sample stream NAL unit to which this configuration record applies. The value of this field shall be conditional on the 4CC-code of the sample entry. For base mesh tracks unit_size_precision_bytes_minus1 shall be equal to ssnh_unit_size_precision_bytes_minus1 in sample_stream_nal_header().

[0065] num_of_setup_unit_arrays may indicate the number of arrays of displacement NAL units of the indicated type(s)

[0066] array_completeness when equal to 1 may indicate that all displacement NAL units of the given type are in the following array and none are in the stream; and when equal to 0 indicates that additional displacement NAL units of the indicated type may be in the stream. The default and permitted values may be constrained by the sample entry name.

[0067] nal_unit_type may indicate the type of the displacement NAL units in the following array, which may all be of that type, or may be a mix of types in some examples (in which case, multiple nal _unit_type entries may be present, or a type may be predefined as a mix). The nal_unit_type may comprise a value which may be predetermined to represent a type of displacement NAL unit, such as those defined in ISO / IEC 23090-29; in some examples, it may be restricted to take one of the values indicating a NALJDSPS, NAL_DFPS, NAL_PREFIX_ESEI, NAL_PREFIX_NSEI, NAL_SUFFIX_ESEI, or NAL_SUFFIX_NSEI displacement NAL unit.

[0068] num_nal junits may indicate the number of base mesh NAL units of type nal_unit_ty pe included in the configuration record for the stream to which this configuration record applies.

[0069] setup_unit_length may indicate the size, in bytes, of the setupjunit field. The length field may include the size of both the NAL unit header and the NAL unit payload but in many examples does not include the length field itself.

[0070] setup_uni t contains a NAL unit according to related nal _unit_type. When present in setupjunit,NAL_PREFIX_ESEI, NAL_PREFIX_NSEI, NAL_SUFFIX_ESEI, or NAL_SUFFIX_NSEI contain SEI messages of a ‘declarative’ nature, that is, those that provide information about the stream as a whole. An example of such an SEI could be a user-data SEI.

[0071] Each sample in a V-DMC Displacement track 210e may correspond to a single coded displacement access unit. In some examples, a V-DMC Displacement sample may be defined as follows: aligned(8) class VDMCDisplacementSample {II sample_size value is the size of the sample from the SampleSizeBox for (int i=0; I < sample_size; ) { sample_stream_nal_unit ss_nal_unit; / / as defined in ISO / IEC 23090-5, ss_nal_unit containing nal unit of displacement as defined in Annex J of 23090-29 i += ss_nal_unit.ssnu_nal_unit_size + VDMCDisplacementDecoderConfigurationRecord.unit_size_precision_bytes_minus1 + 1 }

[0072] Accordingly, examples of the systems and methods discussed herein provide an encapsulation scheme that may contain a base mesh bitstream 2104, sub-mesh(es) 2105, and / or displacement information 2106, providing efficient storage and transmission of the content in an extensible container, such as an ISOBMFF container 200. In particular, examples of the systems and methods discussed herein provide a generic and scalable design to support carrying sub-components of a V-DMC or similar bitstream, such as a base mesh 2104, sub-meshes 2105, and displacement information 210e, in an ISOBMFF media container 200.

[0073] FIG. 3 illustrates a block diagram of an example of a system 300 in which various aspects and examples can be implemented. System 300 may be embodied as a device including the various components described below and is configured to perform one or more of the aspects described in this application. Examples of such devices, include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 300, singly or in combination, may be embodied in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one example, the processing and encoder / decoder elements of system 300 are distributed across multiple ICs and / or discrete components. In various examples, system 300 is communicatively coupled to other systems, or to other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports. In various examples, system 300 is configured to implement one or more of the aspects described in this application.

[0074] System 300 includes at least one processor 310 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this application. Processor 310 may include embedded memory, input output interface, and various other circuitries as known in the art. System 300 includes at least one memory 320 (e.g., a volatile memory device, and / or a non-volatile memory device). System 300 includes a storage device 340, which may include non-volatile memory and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drive, and / or optical disk drive. Storage device 340 may include an internal storage device, an attached storage device, and / or a network accessible storage device, as nonlimiting examples.

[0075] System 300 includes an encoder / decoder module 330 configured, for example, to process data to provide an encoded video / 3D object or decoded video / 3D object, and encoder / decoder module 330 may include its own processor and memory. Encoder / decoder module 330 represents module(s) that may be included in a device to perform the encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. Additionally, encoder / decoder module 330 may be implemented as a separate element of system 300 or may be incorporated within processor 310 as a combination of hardware and software as known to those skilled in the art.

[0076] Program code to be loaded onto processor 310 or encoder / decoder 330 to perform the various aspects described in this application, may be stored in storage device 340 and subsequently loaded onto memory 320 for execution by processor 310. In accordance with various examples, one or more of processor 310, memory 320, storage device 340, and encoder / decoder module 330 may store one or more of various items during the performance of the processes described in this application. Such stored items may include, but are not limited to, the input video / 3D object, the decoded video / 3D object or portions of the decoded video / 3D object, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0077] In several examples, memory inside of processor 310 and / or encoder / decoder module 330 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding. In other examples, a memory external to the processing device (for example, the processing device may be either processor 310 or encoder / decoder module 330) is used for one or more of these functions. The external memory may be memory 320 and / or storage device 340, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several examples, an external non-volatile flash memory is used to store the operating system of a television. In at least one example, a fast external dynamic volatile memory such as a RAM is used as working memory for coding and decoding operations, such as for instance MPEG-2, HEVC, or WC.

[0078] The input to the elements of system 300 may be provided through various input devices as indicated in block 305. Such input devices include, but are not limited to, (i) an RF portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0079] In various examples, the input devices of block 305 have associated respective input processing elements as known in the art. For example, the RF portion may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) down converting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select (for example) a signal frequency band which may be referred to as a channel in certain examples, (iv) demodulating the down converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various examples includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion may include a tuner that performs various of these functions, including, for example, down converting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box example, the RF portion and its associated input processing element receives an RF signal transmitted over a wired (for example, cable) medium, and performs frequency selection by filtering, down converting, and filtering again to a desired frequency band. Various examples rearrange the order of the above-described (and other) elements, removesome of these elements, and / or add other elements performing similar or different functions. Adding elements may include inserting elements in between existing elements, for example, inserting amplifiers and an analog-to-digital converter. In various examples, the RF portion includes an antenna.

[0080] Additionally, the USB and / or HD M I terminals may include respective interface processors for connecting system 300 to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within processor 310 as necessary. Similarly, aspects of USB or HDMI interface processing may be implemented within separate interface ICs or within processor 310 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 310, and encoder / decoder 330 operating in combination with the memory and storage elements to process the data stream as necessary for presentation on an output device.

[0081] Various elements of system 300 may be provided within an integrated housing. Within the integrated housing, the various elements may be interconnected and transmit data therebetween using suitable connection arrangement, for example, an internal bus as known in the art, including the I2C bus, wiring, and printed circuit boards.

[0082] System 300 includes a communication interface 350 that enables communication with other devices via a communication channel 390. Communication interface 350 may include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 390. Communication interface 350 may include, but is not limited to, a modem or network card and communication channel 390 may be implemented, for example, within a wired and / or a wireless medium.

[0083] Data is streamed to system 300, in various examples, using a Wi-Fi network such as IEEE 802.11 . The Wi-Fi signal of these examples is received over communications channel 390 and communications interface 350 which are adapted for Wi-Fi communications. Communications channel 390 of these examples is typically connected to an access point or router that provides access to outside networks including the Internet for allowing streaming applications and other over-the-top communications. Other examples provide streamed data to system 300 using a set-top box that delivers the data over the HDMI connection of input block 305. Still other examples provide streamed data to system 300 using the RF connection of input block 305.

[0084] System 300 may provide an output signal to various output devices, including a display 365, a set of audio outputs, such as speakers 375, and other peripheral devices 385. The other peripheral devices 385 include, in various examples, one or more of a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide a function based on the output of system 300. In various examples, control signals are communicated between system 300 and the display 365, speakers 375, or other peripheral devices 385 using signaling such as AV. Link, CEC, or other communications protocols that enable device- to-device control with orwithout user intervention. The output devices may be communicatively coupled to system 300 via dedicated connections through respective interfaces, such as a display interface 360, an audio interface 370, and a peripheral interface 380. The output devices may be connected to system 300 using communications channel 390 via communications interface 350. Display 365 and speakers 375 may be integrated in a single unit with the other components of system 300 in an electronic device, for example, a television. In various examples, display interface 360 includes a display driver, for example, a timing controller (T Con) chip.

[0085] Display 365 and speaker 375 may alternatively be separate from one or more of the other components, for example, if the RF portion of input 305 is part of a separate set-top box. In various examples in which display 365 and speakers 375 are external components, the output signal may be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0086] In many examples of two-dimensional or three-dimensional environments, environmental details and / or objects or entities may be represented by point clouds or meshes of vertices and / or edges forming a large number of triangles or other shapes. For example, for three-dimensional virtual reality, augmented reality, or extended reality (VR, AR, or XR) environments, virtual objects within the environment may be represented as a mesh of triangles to represent the geometry of the object. In order to provide fine detail, the triangles or other shapes utilized in the mesh may be very small, with the mesh having a correspondingly large number of vertices and edges, potentially in the millions or billions of vertices for a complex object. This may require a correspondingly large amount of memory for storage and bandwidth for transmission. When used for three-dimensional video or animation, the problem provided above may be even worse: even a relatively simple environment with 1 million vertices, at 30 frames per second, may require a total bandwidth of 3.6 gigabits per second if uncompressed.

[0087] Different encoding and decoding standards and examples have been described, including video-based point cloud compression (V-PCC), geometry-based point cloud compression (GPCC), and video-based dynamic mesh coding (V-DMC), promulgated by the Motion Picture Experts Group (MPEG); P11 dynamic mesh coding, developed by Apple Inc.; and others. A mesh may comprise one or more of the following features: a list of vertex positions; a topology defining the connection between the vertices, for instance a list of faces; and optionally photometric data, such as a texture map or color values associated with vertices or faces. The 3D mesh can be derived from a point cloud of the 3D object. The faces defined by connected vertices can be triangles or any other possible polygons or combinations of polygons. In many examples, photometric data may be projected on texture map so that the texture map can be encoded as a video image.

[0088] FIG. 4 illustrates a block diagram of an implementation of a V-PCC encoder 400. In brief overview, in some examples, volumetric 3D data or a mesh may be split into a plurality of sub-meshes or patches. In some examples, the frame image may be separated into a set of 3D projections in different components, such as far and near components for geometry and corresponding attribute components. In some examples, a 2D occupancy mapmay be created to indicate parts of an image that should be used as a texture or projection on the mesh. The 2D projection may comprise a plurality of independent patches based on geometry characteristics of the input point cloud frame. After the patches have been generated and 2D projection frames for video encoding are created, the occupancy map, geometry information, attribute information and the auxiliary information may be compressed and, in many examples, post-processed (e.g., smoothed, normalized, scaled, etc.). The separate bitstreams (e.g., patch sub stream 410, attribute sub stream 420, geometry sub stream 430, and occupancy sub stream 440) may be multiplexed into an output compressed bit stream 450.

[0089] FIG. 5 illustrates a block diagram of an implementation of a V-PCC decoder 500. The incoming bitstream 510 may be demultiplexed to recover the patch 520, geometry 540, attribute 550, and occupancy 530 sub streams. In some examples, additional auxiliary information 560 may be included in the multiplexed bitstream, such as a sequence parameter set (SPS) sub stream 560. Such auxiliary information stream 560 may be entropy coded and compressed. The occupancy map may be compressed using video compression 535, and in some examples, may be upscaled to a nominal resolution using any suitable techniques (e.g., nearest neighbor, etc.). The geometry stream 540 may be decoded 545 and, in combination with the occupancy map and auxiliary information, may be smoothed or interpolated to reconstruct the point cloud geometry information 590. Based on the decoded attribute video stream 550, reconstructed geometry information 540, occupancy map 530, and auxiliary information 560, the point cloud topology and its attributes (e.g., texture, color, lighting, etc.) can be reconstructed.

[0090] In the present application, the terms "reconstructed” and “decoded” may be used interchangeably, the terms “encoded” or “coded" may be used interchangeably, the terms “pixel” or “sample” may be used interchangeably, and the terms “image,” “picture" and “frame” may be used interchangeably. Usually, but not necessarily, the term “reconstructed” is used at the encoder side while “decoded” is used at the decoder side.

[0091] Although discussed above in terms of examples using V-PCC codecs, the systems and methods discussed herein are not limited to such codecs, and may be used to encode and decode any suitable two- dimensional or three-dimensional mesh or point cloud, and accordingly, the encoder and decoder of FIGs. 4 and 5 are provided by way of example only.

[0092] Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as “first”, “second”, etc. may be used in various examples to modify an element, component, step, operation, etc., for example, a “first decoding” and a “second decoding”. Use of such terms does not imply an ordering to the modified operations unless specifically required. So, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or in an overlapping time period with the second decoding.

[0093] Moreover, the present aspects are not limited to V-PCC, G-PCC, P11 , or V-DMC, and can be applied, for example, to other standards and recommendations, and extensions of any such standards and recommendations. Unless indicated otherwise, or technically precluded, the aspects described in this application can be used individually or in combination. Various numeric values are used in the present application. The specific values are for example purposes and the aspects described are not limited to these specific values.

[0094] Various examples involve decoding "Decoding,” as used in this application, may encompass all or part of the processes performed, for example, on a received encoded sequence in order to produce a final output suitable for display. In various examples, such processes include one or more of the processes typically performed by a decoder, for example, entropy decoding, inverse quantization, inverse transformation, and differential decoding. Whether the phrase “decoding process” is intended to refer specifically to a subset of operations or generally to the broader decoding process will be clear based on the context of the specific descriptions and is believed to be well understood by those skilled in the art.

[0095] Various examples involve encoding. In an analogous way to the above discussion about “decoding”, “encoding” as used in this application may encompass all or part of the processes performed, for example, on an input video sequence in order to produce an encoded bitstream.

[0096] The examples and aspects described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed may also be implemented in other forms (for example, an apparatus or program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, for example, computers, cell phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.

[0097] Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation”, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation”, as well any other variations, appearing in various places throughout this application are not necessarily all referring to the same embodiment. Additionally, this application may refer to “determining” various pieces of information. Determining the information may include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0098] Further, this application may refer to "accessing” various pieces of information. Accessing the information may include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0099] Additionally, this application may refer to "receiving” various pieces of information. Receiving is, as with "accessing”, intended to be a broad term. Receiving the information may include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, “receiving” is typically involved, in one way or another, during operations, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0100] It is to be appreciated that the use of any of the following 7”, “and / or”, and “at least one of, for example, in the cases of “A / B”, “A and / or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as is clear to one of ordinary skill in this and related arts, for as many items as are listed.

[0101] Also, as used herein, the word “signal” refers to, among other things, indicating something to a corresponding decoder. For example, in certain examples the encoder signals a quantization matrix for dequantization. In this way, in an examples the same parameter is used at both the encoder side and the decoder side. Thus, for example, an encoder can transmit (explicit signaling) a particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter as well as others, then signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual functions, a bit savings is realized in various examples. It is to be appreciated that signaling can be accomplished in a variety of ways.

[0102] For example, one or more syntax elements, flags, and so forth are used to signal information to a corresponding decoder in various examples. While the preceding relates to the verb form of the word “signal”, the word “signal” can also be used herein as a noun.

[0103] As will be evident to one of ordinary skill in the art, examples may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method, or data produced by one of the described examples. For example, a signal maybe formatted to carry the bitstream of a described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor- readable medium.

Claims

CLAIMSWhat is Claimed:1 . A method for communication of mesh and displacement data, comprising: receiving, by an encoder of a device, a plurality of sub-streams of an item of media content, the plurality of sub-streams including an atlas sub-stream, one or more video sub-streams, a base-mesh sub-stream, one or more sub-mesh sub-streams, and a displacement sub-stream; generating, by the encoder, a track reference identifying an association between the base-mesh sub-stream and the one or more sub-mesh sub-streams, wherein the track reference is encapsulated in a header of a first substream of the plurality of sub-streams; and generating, by the encoder, the plurality of sub-streams in a container file format, wherein a header of a first sub-stream generated in the container file format includes a reference identifier of each of the one or more video substreams, the base-mesh sub-stream, the one or more sub-mesh sub-streams, and the displacement sub-stream.

2. The method of claim 1 , wherein the container file format is an International Standards Organization Base Media File Format (ISOBMFF).

3. The method of claim 1, wherein the base-mesh sub-stream, the one or more sub-mesh sub-streams, or the displacement sub-stream are timed data in synchronization with the atlas sub-stream and one or more video sub-streams.

4. The method of claim 1 , wherein the base-mesh sub-stream, the one or more sub-mesh sub-streams, or the displacement sub-stream are un-timed data.

5. The method of claim 1 , wherein at least one of the base-mesh sub-stream, the one or more submesh sub-streams, or the displacement sub-stream are timed data in synchronization with the atlas sub-stream and one or more video sub-streams, and at least one other of the base-mesh sub-stream, the one or more sub-mesh substreams, or the displacement sub-stream are un-timed data.

6. The method of claim 1 , wherein the sub-streams correspond to tracks in the generated container file format.

7. The method of claim 1 , wherein a header of the generated base-mesh sub-stream comprises an identifier of each sub-mesh sub-stream.

8. The method of claim 1 , wherein the base-mesh sub-stream identifies a plurality of base meshes of the item of media content, each base mesh associated with a different time period within the item of media content.

9. The method of claim 1 , wherein the generated one or more sub-mesh sub-streams are identified in a header of the atlas sub-stream as members of a track group.

10. A device for communication of mesh and displacement data, the device comprising: an encoder; and an input / output (I / O) device operably coupled to the encoder, the device configured to receive a plurality of sub-streams of an item of media content, the plurality of substreams including an atlas sub-stream, one or more video sub-streams, a base-mesh sub-stream, one or more submesh sub-streams, and a displacement sub-stream; the device further configured to generate a track reference identifying an association between the base-mesh sub-stream and the one or more sub-mesh sub-streams, wherein the track reference is encapsulated in a header of a first sub-stream of the plurality of sub-streams; and the device further configured to generate the plurality of sub-streams in a container file format, wherein a header of a first sub-stream generated in the container file format includes a reference identifier of each of the one or more video sub-streams, the base-mesh sub-stream, the one or more sub-mesh sub-streams, and the displacement sub-stream.

11. The device of claim 10, wherein the container file format is an International Standards Organization Base Media File Format (ISOBMFF).

12. The device of claim 10, wherein the base-mesh sub-stream, the one or more sub-mesh sub-streams, or the displacement sub-stream are timed data in synchronization with the atlas sub-stream and one or more video sub-streams.

13. The device of claim 10, wherein the base-mesh sub-stream, the one or more sub-mesh sub-streams, or the displacement sub-stream are un-timed data.

14. The device of claim 10, wherein at least one of the base-mesh sub-stream, the one or more submesh sub-streams, or the displacement sub-stream are timed data in synchronization with the atlas sub-stream and one or more video sub-streams, and at least one other of the base-mesh sub-stream, the one or more sub-mesh substreams, or the displacement sub-stream are un-timed data.

15. The device of claim 10, wherein a header of the generated base-mesh sub-stream comprises an identifier of each sub-mesh sub-stream.

16. The device of claim 10, wherein a header of one sub-mesh track comprises an identifier of each submesh sub-stream.

17. The device of claim 10, wherein the base-mesh sub-stream identifies a plurality of base meshes of the item of media content, each base mesh associated with a different time period within the item of media content.

18. The device of claim 10, wherein the generated one or more sub-mesh sub-streams are identified in a header of the atlas sub-stream as members of a track group.

19. A method for communication of mesh and displacement data, comprising: receiving, by a decoder of a device, information including a container file format, the container format including a plurality of sub-streams of an item of media content, the plurality of sub-streams including an atlas sub-stream, one or more video sub-streams, a base-mesh sub-stream, one or more sub-mesh sub-streams, and a displacement substream; decoding, by the decoder, the information including the container file format including the plurality of substreams;parsing the plurality of sub-streams from the container file format, wherein a header of a first sub-stream generated in the container file format includes a reference identifier of each of the one or more video sub-streams, the base-mesh sub-stream, the one or more sub-mesh sub-streams, and the displacement sub-stream; parsing a track reference encapsulated in a header of a first sub-stream of the plurality of sub-streams to identify an association between the base-mesh sub-stream and the one or more sub-mesh sub-streams; and render the media content using the track reference.

20. The method of claim 19, wherein the rendering of the media content occurs on an external display.

Images