Volumetric video experiences using dynamic mesh coding

Abstract

Methods and bitstream syntax are described for supporting volumetric video experiences using video-based mesh coding (V-DMC). For the V-DMC file and streaming formats, example syntax is provided for extending the ISOBMFF standard for both single-track and multi-track encapsulations of all key parameters in a V-DMC bitstream, such as Atlas, Base Mesh, Displacement, and Attribute sub-streams. Example descriptions are also provided for DASH Media Presentation Description (MPD) files, a V-DMC player, and a V-DMC streaming client.

Description

VOLUMETRIC VIDEO EXPERIENCES USING DYNAMIC MESH CODINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from U.S. Provisional Application No. 63 / 573,018, filed on 2 April 2024, which is incorporated by reference herein in its entirety.TECHNOLOGY

[0002] The present document relates generally to images and video. More particularly, embodiments of the present invention relate to technologies for volumetric video experiences using dynamic mesh coding.BACKGROUND

[0003] Advances in 3D capture, modelling, and rendering allow now for more immersive user experiences. Using 3D meshes is one of many ways to represent immersive, volumetric content. The Motion Pictures Experts Group (MPEG) is actively working on defining standards for the coded representation of immersive media, especially video-based dynamic mesh coding (V-DMC) (Ref. [1]). As appreciated by the inventors, novel technologies for the storage, delivery, and streaming of V-DMC content are described herein.

[0004] The term “metadata” herein relates to any auxiliary information transmitted either as part of the coded bitstream or along with it that assists a decoder to render or interpret one or more decoded images.

[0005] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, issues identified with respect to one or more approaches should not assume to have been recognized in any prior art on the basis of this section, unless otherwise indicated.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] An embodiment of the present invention is illustrated by way of example, and not in way by limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0007] FIG. 1A depicts an example encoding process for a V-DMC bitstream;

[0008] FIG. IB depicts an example decoding process for a coded V-DMC bitstream;

[0009] FIG. 2A depicts an example of an Atlas NAL unit in V-DMC coding;

[0010] FIG. 2B depicts an example of a Base-mesh NAL unit in V-DMC coding;

[0011] FIG. 3 depicts an example of a V-DMC player process according to an embodiment of this invention;

[0012] FIG. 4 A depicts an example of a single-track encapsulation of a V-DMC bitstream comprising atlas, base mesh, displacement, and texture sub-bitstreams according to an embodiment of this invention;

[0013] FIG. 4B depicts an example of a single-track encapsulation of a V-DMC bitstream comprising atlas, base mesh, and displacement and texture sub-bitstreams according to an embodiment of this invention;

[0014] FIG. 5A depicts an example of a multi-track encapsulation of a V-DMC bitstream according to an embodiment of this invention;

[0015] FIG. 5B depicts an example of a multi-track encapsulation of a V-DMC bitstream with a packed video bitstream according to an embodiment of this invention;

[0016] FIG. 6 depicts an example workflow for V-DMC file decapsulation according to an embodiment of this invention;

[0017] FIG. 7 A depicts an example of grouping V-DMC components using preselection according to an embodiment of this invention;

[0018] FIG. 7B depicts an example of grouping atlas, base mesh, and packed video component adaptation sets using preselection according to an embodiment of this invention; and

[0019] FIG. 8 depicts an example of a workflow for a V-DMC streaming client according to an embodiment of this invention.DESCRIPTION OF EXAMPLE EMBODIMENTS

[0020] Example embodiments that relate to technologies for video-based dynamic mesh coding (V-DMC) are described herein. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments of present invention. It will be apparent, however, that the various embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustivedetail, in order to avoid unnecessarily occluding, obscuring, or obfuscating embodiments of the present invention.SUMMARY

[0021] Example embodiments described herein relate to technologies for video-based dynamic mesh coding (V-DMC), including technologies defining the carriage of video based dynamic mesh coding data for storage and delivery purposes. More specifically, examples describe how to provide immersive volumetric video experiences by extending the ISO-based media file format (ISOBMFF) for storage and the MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH) format for streaming using both single-track and multi-track encapsulations. Embodiments for a V-DMC player and a V-DMC DASH client are also presented.VIDEO-BASED DYNAMIC MESH CODING (V-DMC)

[0022] As depicted in FIG. 1A, a dynamic mesh sequence (103) is composed of: a base mesh (130), a set of displacements (135), the attributes (texture) (140), and an atlas (125). The base mesh component is a simplified low-resolution approximation of the original mesh. The base mesh component can be encoded using any mesh codecs. The displacement component provides displacement vectors, that can be encoded by using any video codec. The attribute components provide additional properties, e.g., texture or material information, and can be encoded by any video codecs. Finally, the atlas component provides information to a decoding and / or rendering system on how to perform inverse reconstruction, e.g., how to perform the subdivision of base mesh and how to apply the displacement vectors to the subdivided mesh vertices and how to apply attributes to the reconstructed mesh. The encoding process generates a V-DMC bitstream (205) containing a collection of subbitstreams representing atlas, base mesh, displacement, and attributes.

[0023] As depicted in FIG. IB, a decoding process starts by demultiplexing of the input V-DMC bitstream (205) into components that carry the base mesh, the displacement geometry refinement, the attributes, and the atlas sub-streams. Next, each sub-bitstream component is decoded. The base mesh sub-bitstream is decoded by a mesh decoder, the displacement and attribute sub-bitstream are decoded by a 2D video codec. When thedecoding of all the components is finished, an approximation of the original mesh with texture (103b) can be reconstructed.

[0024] A V-DMC bitstream is a single visual volumetric video (V3C) bitstream consisting of a sequence of syntax structures called V3C units (Ref. [2]). Each V3C unit has a V3C unit header and a V3C unit payload. The V3C unit header describes the V3C unit type and additional information, e.g., attribute identification, if needed.

[0025] A particular type of V3C units contain the V3C parameter set (VPS) which contains the parameter set applied to a whole sequence.

[0026] The payload of the V3C units carry the displacement and attribute video corresponding to video data units, e.g., NAL (the network abstraction layer) units, that could be decoded by an appropriate video decoder. Each video NAL unit contains either the displacement or attribute video, or both of them.

[0027] The payload of the V3C unit carrying the atlas contains atlas NAL units. Each atlas NAL unit contains one of an atlas sequence parameter set, an atlas frame parameter set, SEI, and atlas tile data, as described in FIG. 2A (Ref. [6]).

[0028] The payload of the V3C unit carrying the bash mesh contains the base mesh NAL unit. Each base mesh NAL unit contains one of: a base mesh sequence parameter set, a base mesh frame parameter set, and mesh data, as described in FIG. 2B (Ref. [1]).Metadata requirements

[0029] To assist a V-DMC player to select which tracks to be extracted from a file or which segment(s) to be downloaded, the metadata to transmit through the channel, i.e., in a file / segment or Media Presentation Description (MPD), includes the following information:• In a file o The entry point track. o Required capability information to only allow the player which supports the required capability to decode and to process the V-DMC bitstream extracted from track(s). o When multiple tracks need to be extracted, the number of tracks and track indication which need to be extracted. o Decoder configuration information to be used for the V-DMC decoder. o post extraction operation information when any required operations after extracting samples from the track(s) are required.o Sample format information indicating how the V-DMC data unit is encapsulated in the sample of the track.• In MPEG DASH MPD o Indication whether multiple segments need to be prefetched. o Among the available segments, which segment(s) need to be requested for providing immersive experience. o Information to enable the player to request the particular segments according to the device capability, network bandwidth, user viewport, and so on.V-DMC Player

[0030] FIG. 3 depicts an example V-DMC player according to an embodiment. The player may include sensors and a user interface (UI) to be used in two example playback scenarios: via a streaming client (308) or via a media file (310), e.g., an MP4 file on a user’s computer. In the streaming scenario, metadata (302), as described earlier, is signaled in the DASH MPD, which contains all the information necessary to enable the player to determine which segments need to be requested among all segments available. The player receives the DASH MPD and determines which segments are fit to be extracted under the current viewing situation, such as, based on device capability or a viewport. Then the player requests and receives the segments accordingly via the segment interface (305). Alternatively, in the fileplayback scenario, to enable the player to determine whether it can play the file, the metadata may also be also carried in the file itself (310).

[0031] The file / segment decapsulation module (315) extracts the sequence of V3C units and parses the metadata from the file or the segments in the streaming client. Depending on how the sequence of V3C units is encapsulated in a file or in a streaming segment, sometimes a player may require post operations after extracting samples from the track. For example, when the sequence of V3C units is carried in multiple tracks within a file or a segment, it requires a V3C unit construction by combining sample entry information and sample data units from samples of the track before the V-DMC bitstream is generated. Alternatively, when the sequence of V3C units is encapsulated into a single track, the extracted sequence of V3C units, identical to the V-DMC bitstream, can be fed to the V-DMC decoder without any post extraction operation. After optional V-DMC bitstream composition, the V-DMC bitstream is decoded (320) into multiple frames and a mesh sequence is reconstructed fromthe decoded signals. The mesh sequence is finally rendered (325) as the output viewport according to the current user viewport including viewing position and orientation.Track encapsulation of timed V-DMC in ISOBMFF

[0032] This section specifies the carriage of a single V-DMC bitstream into a file by utilizing the existing capabilities of the ISO base media file format (ISOBMFF) (Ref. [3]), and defining, when needed, proposed extensions.

[0033] Two methods are specified, including the sample entry definition and how the V- DMC bitstream is divided into samples as specified by a sample format; a single track encapsulation, where the entire V-DMC bitstream is stored as one track, and multi-track encapsulation, where the V-DMC bitstream is stored into multiple tracks.

[0034] The proposed V-DMC box types described in this section are listed in bold in Table 1. Tabic 1. Box types described in this section and their relation to ISOBMFF boxes.Single-track encapsulation

[0035] For a single-track encapsulation, the entire V-DMC bitstream is directly stored in a bitstream track and V3C unit data structures are kept in the bitstream. A bitstream samplecontains one or more V3C units which belong to the same presentation time and each V3C unit corresponds to a sub-sample.

[0036] V-DMC -bitstream decoder- specific information, which is used to initialize the decoders, is carried in the sample entry of a bitstream track (e.g., 402). In an embodiment, but without limitation, the bitstream track uses the Volumet ri cVi sualSampleEnt ry with a sample entry type of ' v3e2 ' . The ‘v3e2 ' sample entry contains aV3CConf igurat ionBox which contains V3C parameter sets, atlas parameter sets (with the NAL unit type as NAL_ASF S, NAL_AAP S, NAL_AFP S) , displacement parameter sets(with the NAL unit type as NAL_DSP S, NAL_DFPS), and base mesh parameter sets (with the NAL unit type as NAL_BSP S, NAL_BFP S) which apply to samples referring to this sample entry. The ‘v3e2 ' sample entry also contains SEI messages for essential processing of a V-DMC bitstream. The ‘v3e2 ' sample entry contains 2D video decoder configuration information box, e.g., HEVCcon f iguart ionBox, WCconf iguart ionBox, as defined in ISO / IEC 14496-15 (Ref.[4]), to signal the information for video decoder configuration and initialization. An example of a v3e2 entry is depicted in Table 2.Table 2.v3e2.

[0037] In Table 2, unit_size_precision_bytes_minus l plus 1 specifies the precision, in bytes, of a sample stream V3C unit to which this configuration record applies. The value of this field is equal to ssvh_unit_size_precision_bytes_minusl in the sample_ st ream— v3c_ header ( ) as defined below .ssvh_unit_size_precision_bytes_minusl plus 1 specifies the precision, in bytes, of the ssvh_v3c_unit_size element in all sample stream V3C units. num_of_v3 c_parameter_set s specifies the number of V3C parameter set units signalled in the decoder configuration record. v3c_parameter_set_lenqth indicates the size, in bytes, of the v3c_parameter_set field. v3c_parameter_set is a V3C unit payload for V3C unit of type V3C_VPS which contains a V3C parameter set. num_of_setup_unit_arrays indicates the number of arrays of atlas NAL units of the indicated type(s). array_completenes s when equal to 1 indicates that all NAL units of the given type are in the following array and none are in the stream; when equal to 0 indicates that additional NAL units of the indicated type may be in the stream; the default and permitted values are constrained by the sample entry' name.nal_unit_type indicates the type of the NAL units in the following array (which shall be all of that type); it takes one of the values indicating a NAL_ASP S, NAL_AAP S, NAL_AFP S, NAL_PREFIX_ESE I, NAL_PREF IX_NSE I , NAL_PREF IX_ESE I , NAL_PREF IX_NSE I , NAL_SUFF IX_ESE I, NAL_SUFF IX_NSE I , NAL_DSP S , NAL_DFP S , NAL_BMSP S , NAL_BMFP S NAL unit, . num_nal_unit s indicates the number of NAL units of type nal_unit_type included in the configuration record for the stream to which this configuration record applies. setup_unit_length indicates the size, in bytes, of the setup_unit field. The length field includes the size of both the NAL unit header and the NAL unit payload but does not include the length field itself. setup_unit contains a NAL unit according to related nal_unit_type.The setup_unit arrays include atlas parameter sets and base mesh sub-bitstream parameter sets that are constant for the coded atlas bitstream referred to by the sample entry in which the decoder configuration record is present as well as SEI messages.

[0038] A bitstream sample contains one or more V3C units which belong to the same presentation time and each V3C unit corresponds to a sub-sample defined asThe s s_v3 c_unit contains a single V3C unit in V3C unit sample stream format defined as:The s svu_v3c_unit_s ize specifies the size, in bytes, of the sample stream V3C unit. It is equal to(V3 CDs coder Con figur at ionRecord . unit_s i ze_preci s ion_byt e s_minus l + 1 ) * 8.

[0039] To enable the file decoder to support the sub-sample level access, the bitstream track contains one SubSample lnf ormationBox ( ' subs ' ) , which lists the V3C unit header of sub-samples, in the movie header of the bitstream track. The 4-bytes v3c_unit_header header(), as defined in 1SO / 1EC 23090-5 (Ref. [6 J) which represents the sub-sample is copied to the codec_specif ic_parameters field in the SubSample lnformat ionBox and the V3C unit type of each sub-sample is identified by parsing the codec_speci f ic_parameters field in the SubSample lnformat ionBox.

[0040] FIG. 4A depicts an example embodiment of a single-track encapsulation of the V- DMC bitstream which contains atlas, base mesh, displacement, and texture sub-bitstreams. Here, the sample entry (402) contains V3C parameter sets, V-DMC decoder configuration information, video decoder configuration information, e.g., HEVC configuration information, and mesh decoder configuration information. A sample contains multiple sub-samples (405) corresponding to the same presentation time, and each sub-sample contains one V3C unit, which is consisting of V3C unit header and payload, whose pay load contains one of an atlas NAL unit, a base mesh NAL unit, a displacement NAL unit, and a texture NAL unit. In this scenario, a decoder may need to have two video decoders: one to decode displacement information and one to decode attribute (texture) information.

[0041] FIG. 4B depicts an example of the single-track encapsulation of a V-DMC bitstream consisting of an atlas, a base mesh, and a packed video sub-bitstream of displacement and texture. This is the case when one video coder is used to code the packed video bitstream of displacement and attribute. All decoder initialization information, i.e., V3C parameter sets and all decoder configuration information, are carried in the sample entry (402). All NAL units corresponding to the same presentation time are carried in the same sample and each sub-sample contains one V3C unit, which contains one of atlas NAL unit, base mesh NAL unit, and packed video NAL unit containing displacement and attribute video data.Multi-track encapsulation

[0042] In an embodiment, as depicted in FIGs 5A and 5B, a multi-track encapsulated V- DMC bitstream container for a single V-DMC content includes the following:• One atlas track which contains a V3C parameter set and may contain atlas parameter sets in the sample entry and atlas NAL units in the samples.• Zero or one V3C video component tracks where the samples contain access units of video-coded elementary streams for displacement data (i.e., payloads of V3C units of type equal to V3C_GVD).• Zero or one V3C video component tracks where the samples contain access units of video-coded elementary streams for texture data (i.e., payloads of V3C units of type equal to V3C_AVD).• Zero or one V3C video component tracks where the samples contain access units of video-coded elementary streams for packed data (i.e., payloads of V3C units of type equal to V3C_PVD). When this track is present, V3C video component tracks where the samples contain access units of video-coded elementary streams for both of displacement and texture data are not present, vice versa, this track is not present when V3C video component tracks where the samples contain access units of video-coded elementary streams for both of displacement and texture data are present.• One base mesh track which contains base mesh parameter sets in the sample entry and base mesh NAL units in the samples.For multiple track encapsulation, the conditions below need to be satisfied:• An atlas track is an entry point and includes track references to other tracks carrying the payloads of video compressed V3C units (i.e., V3C unit types equal to V3C_GVD, and V3C_AVD, or V3C_PVD) or to base mesh V3C units (i.e., V3C unit type V3C_BMD).• In each track sample entry, a single V3CUnitHeaderBox is present to indicate the role of the bitstream contained in the track.• When a track where the samples contain access units of video-coded elementary streams for packed data is present, the V3C video component tracks which contain displacement data or texture data are not present in a same file. Vice versa, when V3C video component tracks which contain displacement data or texture data are present, a track where the samples contain access units of video-coded elementary streams for packed data is not present in a same file.

[0043] A single V3CUnitHeaderBox is present in the sample entry of a V3C atlas track, a V3C bash mesh track, and in s cheme_speci f ic_data Box array of the Scheme lnf ormat ionBox of all video component tracks. V3CUnitHeaderBox contains the V3C unit header describing the samples carried by the respective track. An example of a box-type “vunt” is depicted in Table 3. An example of a v3c_unit_header is depicted in Table 4.Table 3. Example of a “vunt” Box typeTab e 4. Example of a v3c_unit_header()The vuh_unit_type indicates the V3C unit type as depicted in the following Table.vuh_v3c_parameter_set_id specifies the value of vps_v3c_parameter_set_id for the active V3C VPS. vuh_atlas_id specifies the ID of the atlas that corresponds to the current V3C unit. vuh_attribute_index indicates the index of the attribute data carried in the Attribute Video Data unit. vuh_attribute_partition_index indicates the index of the attribute dimension group carried in the attribute video data unit. vuh_map_index when present, indicates the map index of the current geometry or attribute stream. vuh auxiliary video flag equal to 1 indicates that the associated geometry or attribute video data unit is a RAW and / or EOM coded points video only sub-bitstream. vuh_auxiliary_video_flag equal to 0 indicates that the associated geometry or attribute video data unit may contain RAW and / or EOM coded points.Atlas track encapsulation

[0044] An atlas track uses V3CAtlas SampleEnt ry which extendsVolumet ri cVi sualSampleEnt ry with a sample entry type of ' v3cl ' , ' v3 c2 ' , or ' v3 cg ' . The following restrictions are set for the atlas track:• In track header the track_in_movie flag is set to 1.• A V3C atlas track sample entry contains a V3 CAt lasConf igurationBox, and a V3 CUnit Heade rBox.• V3CAt lasCon f igurationBox contains V-DMC decoder configuration and initialization information which apply to the decode a whole V-DMC bitstream. It includes V3C parameter sets and atlas parameter sets that are constant for the coded atlas bitstream referred to by the sample entry as well as SEI messages.• V3CUnitHeaderBox contains the 32-bit V3C unit header corresponding to atlas NAL units carried in samples of the track.An example of a V3C Atlas sample entry (504) is depicted in Table 5.Table 5. Example of V3C Atlas sample entry.

[0045] In Table 5 , unit— size_ recision— bytes— minus l plus 1 specifies the precision, in bytes, of the sample stream NAL unit or sample stream V3C unit to which this configuration record applies. For V3C atlas tracks unit_size_precision_byte s_minus l is equal to ssnh_unit_size_precision_bytes_minusl in sample_st ream_nal_header ( ) defined as:ssnh_unit_size_precision_bytes_minusl plus 1 specifies the precision, in bytes, of the ssnh_nal_unit_size element in all sample stream NAL units. num_of_v3 c_parameter_set s specifies the number of V3C parameter set units signalled in the decoder configuration record. v3c_parameter_set_length indicates the size, in bytes, of the v3c_ parameter— set field. v3c_parameter_set is a V3C unit payload for V3C unit of type V3C_VPS which contains a V3C parameter set. num_of_setup_unit_arrays indicates the number of arrays of atlas NAL units of the indicated type(s). array_completenes s when equal to 1 indicates that all NAL units of the given type are in the following array and none are in the stream; when equal to 0 indicates that additional NAL units of the indicated type may be in the stream; the default and permitted values are constrained by the sample entry name. nal_unit_type indicates the type of the NAL units in the following array (which shall be all of that type); it takes one of the values indicating a NAL_ASPS, NAL—AAPS, NAL—AFPS, NAL_PREFIX_ESEI, NAL_PREFIX_NSEI, NAL_SUFFIX_ESEI, NAL_SUFF IX_NSE I NAL unit.num_nal_unit s indicates the number of NAL units of type nal_unit_type included in the configuration record for the stream to which this configuration record applies. setup_unit_length indicates the size, in bytes, of the setup_unit field. The length field includes the size of both the NAL unit header and the NAL unit pay load but does not include the length field itself. setup_unit contains a NAL unit according to related nal_unit_type.The setup_unit arrays include atlas parameter sets that are constant for the CVS referred to by the sample entry as well as SEI messages.

[0046] Each sample in a V3C atlas track corresponds to a single coded atlas access unit consisting of atlas NAL units as described in:s s_nal_unit contains a single atlas NAL unit in NAL unit sample stream format defined as:s snu_nal_unit_s i ze specifies the size, in bytes, of the sample stream NAL unit. The number of bits used to represent ssnu_nal_unit_size is equal to( V3CDecoderConfigurationRecord.unit_size_precision_bytes_minusl + 1 ) * 8.

[0047] To link a V3C atlas track to other tracks, two or more track reference type boxes are added to a track reference box within the track box of the V3C atlas track. The track reference type box contains an array of t rack_IDs designating the other tracks which the V3C atlas track references. The re f erence_type of a track reference type box identifies the type of the video component (i.e., displacement, texture, or packed) and the base mesh component. The four-character code (4CC) of these track reference types are:— 'v3vg': the referenced track(s) contain the video-coded displacement component— 'v3va': the referenced track(s) contain the video-coded texture component— 'v3vp': the referenced track(s) contain the video-coded packed component containing displacement and texture component— 'v3bm': the referenced track(s) contain the coded base mesh componentThe type of the V3C component carried by the referenced restricted video track, signaled in the restricted scheme info box of the track, needs to match the reference type of the track reference from the V3C atlas track.Base mesh encapsulation

[0048] A base mesh track uses BaseMeshSampleEnt ry which extends Volumet ri cVi sualSampleEnt ry with a sample entry type of ‘vbml’. The following restrictions are set for base mesh tracks:• A base mesh sample entry contains a BaseMe shConf igurat ionBox, and a V3CUnitHeaderBox.• BaseMe shCon f igurationBox contains base mesh parameter sets (with the NAL unit type as NAL_BSP S, NAL_BFP S) that are constant for the coded mesh bitstream referred to by the sample entry as well as SEI messages.• V3CUnitHeaderBox contains the 32-bit V3C unit header corresponding to base mesh NAL units carried in samples of the track.

[0049] Table 6 describes an example of a Base mesh sample entry (510).Table 6. Example of Base mesh

[0050] In Table 6, unit_size_precision_bytes_minusl plus 1 specifies the precision, in bytes, of the sample stream NAL unit or sample stream V3C unit to which this configuration record applies. num_of_setup_unit_arrays indicates the number of arrays of base mesh NAL units of the indicated type(s). array_completenes s when equal to 1 indicates that all base mesh NAL units of the given type are in the following array and none are in the stream; when equal to 0 indicates that additional NAL units of the indicated type may be in the stream; the default and permitted values are constrained by the sample entry name. nal_unit_type indicates the type of the NAL units in the following array (which shall be all of that type); it takes one of the values indicating a NAL_BMSP S , NAL_BMFPS, NAL_PREFIX_ESEI, NAL_PREFIX_NSEI, NAL_SUFF IX_ESE I, NAL_SUFFIX_NSEI NAL unit, . num_nal_units indicates the number of NAL units of type nal_unit_type included in the configuration record for the stream to which this configuration record applies. setup_unit_length indicates the size, in bytes, of the setup_unit field. The length field includes the size of both the NAL unit header and the NAL unit payload but does not include the length field itself. setup_unit contains a NAL unit according to related nal_unit_type.The setup_unit arrays include base mesh parameter sets that are constant for the coded base mesh bitstream referred to by the sample entry as well as SEI messages.

[0051] Each sample (515) in base mesh track corresponds to a single coded base mesh access unit consisting of base mesh NAL units as described in:s s_ nal_ unit contains a single base mesh NAL unit in NAL unit sample stream format. s snu_nal_unit_s i ze specifies the size, in bytes, of the sample stream NAL unit.The number of bits used to represent ssnu_nal_unit_size is equal to( V3CDecoderConfigurationRecord.unit_size_precision_bytes_minusl + 1 ) * 8.V3C video component encapsulation

[0052] A V3C video component track carries 2D video encoded data of V3C video component, i.e., displacement, texture, or packed video of displacement and texture. TheV3C video component track is represented in the file as restricted video and uses a generic restricted sample entry 'resv' with additional requirements:• SchemeTypeBox is present in Re strict edScheme lnf oBox and scheme_type is set to 'vvvc'• Scheme lnf ormationBox is present in Rest rictedScheme lnfoBox and contain a V3 CUnit HeaderBox.• In track header the track_in_movie flag is set to 0, to indicate that this track should not be presented alone.• V3CL'nit Header Box contains the 32-bit V3C unit header corresponding to samples of the track.

[0053] Either restricted video sample entry or sample entry contains 2D video decoder configuration information box, e.g., HEVCconf iguart ionBox,WCconf iguart ionBox , as defined in 1SO / 1EC 14496-15 (Ref. |4|), to signal the information for video decoder configuration and initialization to decode samples in the track.For displacement video track, either restricted video sample entry or sample entry contains a decoder configuration information box which contains displacement parameter sets (with the NAL unit type as NAL_D SP S, NAL_DFP S) as well as SEI messages.Each sample in video component track corresponds to a single coded video access unit.Examples of multi-track encapsulation

[0054] FIG. 5 A depicts an example of a multi-track encapsulation of a V-DMC bitstream. Pay loads of V3C units of a V-DMC bitstream are mapped to individual tracks within the multi-track container file based on their types. Each sample entry contains a single V3CUnitHeaderBox to indicate the role of the bitstream contained in the track. Atlas track sample entry contains V3C parameter sets which describes the required capability to decode and to process the whole V-DMC bitstream, and atlas parameter sets and SEIs which apply to the samples in the track. The base mesh sample entry contains base mesh decoder configuration information including base mesh parameters and SEIs which apply to base mesh samples in the track. The restricted video sample entry of displacement and texture video track contains 2D video decoder configuration information to signal the information to decode video samples in the track.

[0055] When the V-DMC bitstream contains the packed video sub-stream which the displacement and texture are packed into video frames, only one video component track is presented in a file as described in FIG. 5B. The restricted video sample entry of the packed video track contains 2D video decoder configuration information to signal the information to decode video samples in the track.V-DMC file decapsulation workflow

[0056] FIG. 6 depicts an example embodiment of a V-DMC file decapsulation workflow (600). The file decapsulation starts by reading a movie header in a file and determines whether track(s) in a file can be processed or decoded. If the file can be processed / decoded, the file decapsulation determines the number of tracks that need to be extracted in the file. If it is a single track, it extracts the sequence of V3C units from the samples and decodes them. If it is required to extract multiple tracks, then the file decapsulation finds the atlas track and extracts the V3C unit header and V3C unit payloads from samples. Next, it reads track references in the atlas track. Based on the references, it finds the referenced tracks, and extracts the V3C unit headers and V3C unit payloads from samples in the referenced tracks.Based on the extracted data, it constructs the V3C units, composites them to a V-DMC bitstream, and decodes the V-DMC bitstream. Then, the dynamic mesh is reconstructed and rendered from the decoded signal.Encapsulation and signaling in MPEG DASH

[0057] The single-track mode in DASH enables streaming of V-DMC ISOBMFF files where the V-DMC bitstream is stored using single-track encapsulation. The single-track mode in DASH is represented as one Adaptation Set with one or more Representations. Representations within the sole Adaptation Set use the same codec for the corresponding video components and base mesh component.

[0058] If a Representation consists of more than one Media Segment, an Initialization Media Segment is present. The Initialization Segment contains a V3CConf igurat ionBox and the V3CConf igurat ionBox contains V3C parameter sets as well as other parameter sets for V-DMC components, e.g., atlas parameter sets, base mesh parameter sets, which applies to Media segments. The Initialization Segment also contains 2D video decoder configuration information box, e.g.,HEVCconf iguart ionBox, WCconf iguart ionBox, to signal the information for video decoder configuration and initialization.

[0059] The ^ codecs attribute starts with four-character codes, e.g., v3e2 , to indicate the segment is in a single-track mode, and contains strings describing the information of the video coding and base mesh coding and its parameters. The value of ^ codecs provides three fields, separated by a comma:— four-character codes, e.g., v3e2 , to indicate the segment is in a single-track mode— video coding information, e.g., resv . vvvc . hvc l when HEVC coding is applied for encoding video components— bash mesh coding information, e.g., drac . 1 . 5 when Google’s Draco (major version is 1 and minor version is 5) is applied for mesh coding.Media Segments for the Representation of this Adaptation Set contain one or more track fragments of the single track described earlier.

[0060] The following restrictions on some of the attributes are applied:— The @mimeType is ‘application / mp4’— The @ codecs is present on the adaptation set level and signals the maximum required capability to decode any Representation in the Adaptation Set. The@ codecs parameter is signalled on the representation level if different from the one on the adaptation set level. The @ codecs on a representation level signals the required capability to decode any component in the Representation.— The @maxWidth and @maxHeight are not signalled for any Adaptation Set.— The @ f rameRate is signalled only in the AdaptationSet element, i.e., the value is not different for different Representations in one Adaptation Set.— The @ width and @ height are not signalled for any Representation.

[0061] The example of MPD signaling depicted in Table 7 demonstrates three different bitrate versions of same V-DMC content are available and each contains a particular bitrate version of V-DMC sequence in single track mode which video components are coded, as an example, by HEVC and mesh component is coded, as an example, by Google Draco for mesh.Table 7. Example of MPD single-track signaling supporting multiple bitrates.Multi-track Mode

[0062] In the multi-track mode, atlas, base mesh, and video components are represented as a separate Adaptation Set in the DASH manifest (MPD) file. To identify the type of V- DMC Component Adaptation Set, a V-DMC Component descriptor is used. A VDMCComponent descriptor parameter is an Essential Property element with the @ s cheme IdUri set to "urn : mpeg : mpeg I : v3 c : 2024 : Component".At Adaptation Set level, the VDMCComponent descriptor is signalled for the type of V-DMC component that is present in the Representations of the Adaptation Set. The @ value of the VDMCComponent descriptor is not present. The VDMCComponent descriptor includes elements and attributes as specified in Table 8.Table 8. Elements and attributes for the V-DMCComponent descriptor

[0063] An example of an XML schema for the VDMCComponent descriptor is shown in Table 9.Table 9. Example XML schema for the VDMCCComponent descriptor

[0064] To provide a single experience, it needs to enable for the DASH client to provide a single experience from a combination of adaptation sets. The Preselection is used to enable for the DASH client to fetch multiple Adaptation Sets. The V-DMC preselection either is signalled in MPD using a PreSelection element within the Period element or a Preselection descriptor at the Adaptation Set level. A VDMC PreSelection element is signalled with an id list for the Gpresele ctionComponent s attribute including the id of the Main Adaptation Set followed by the ids of the Component Adaptation Sets. The@ codecs attribute for the Preselection is set to ' v3 cl ' , ' v3 c2 ' , ' v3cg ' or ' v3 cb ' indicating that the media represented by the Preselection is mesh-based coding media.

[0065] A SupplementalProperty element with a @ s cheme l dUri equal to" urn : mpeg : mpegl : v3c : 2020 : v3c" is referred to as a V3C descriptor. At most oneV3C descriptor may be present in Main Adaptation Set or VDMC Preselection. The V3C descriptor contain the attributes defined in Table 10.Table 10 — Attributes for the V3C descriptor

[0066] The data types for the attributes are as defined in the XML schema. An XML schema for the V3C descriptor is shown in Table 11.Table 11. XML schema for a V3C

[0067] Multiple versions of the same V-DMC media is signalled using separate Preselections. When more than one V-DMC Preselection for the same V-DMC content are present in the MPD, a V3C descriptor with the @vld attribute is signalled in each V-DMC Preselection. Preselections that represent alternative versions of the same V-DMC contentcontains a V3C descriptor with the same @ vid value. At most one V3C descriptor is present at the preselection level. These Preselections are therefore alternatives to each other and the id of the main Adaptation Set of the Preselection, first id in the list of Adaptation Set ids for the @pre sele ctionComponent s , may be different.

[0068] FIG. 7 A illustrates an exemplary DASH configuration for grouping V-DMC component Adaptation sets belonging to a single V-DMC content. The atlas, base mesh, displacement video, attribute video are presented as a separate Adaptation Set, respectively. By using the Preselection element, it indicates the grouping of four Adaptation Sets to provide a single experience.

[0069] FIG. 7B illustrates another example to indicate the grouping of atlas, base mech, and packed video component Adaptation Sets belonging to a single V-DMC content.The following may be applied in MPEG DASH MPD:— Either the above PreSelection element or a Preselection descriptor at the Adaptation Set level is present.— An Adaptation Set for atlas information serves as the Main Adaptation Set for the V- DMC content.— At Adaptation Set level except the Main Adaptation Set, a VDMCComponent descriptor is signalled for the type of V-DMC component that is present in the Representations of the Adaptation Set.— An atlas Adaptation Set for atlas information has the @ co decs attribute set to' v3 c 1 ' , ' v3 c2 ' , or ' v3 eg ' . The atlas Adaptation Set contains a single Initialization Segment at the adaptation set level. The Initialization Segment contains all parameter sets needed to initialize the V-DMC decoder, including V3C parameter sets as well as other parameter sets for V-DMC components, e.g., atlas parameter sets, base mesh parameter sets, and 2D video decoder configuration information. Media Segments for the Representation of the atlas Adaptation Set contain one or more track fragments of the atlas track described earlier.— The @ codecs attribute of a base mesh Adaptation set provides three fields, separated by a dot: four-character codes to indicate the mesh codec, e.g., drac when Google’s Draco is used, major version of the mesh codec, and minor version if applicable. Media Segments for the Representations of base mesh Adaptation Sets contains oneor more track fragments of the corresponding video component track as described earlier— The @ codecs attribute for the Video Component Adaptation Sets, or Representations of these Adaptation Sets, if ^codecs is not signalled for the AdaptationSet element, is set based on the respective codec used for encoding the component. The value of @ codec s is set to ' re s v . vvvc . XXXX ' , where XXXX corresponds to the four-character code (4CC) of the video codec from the original_f ormat field in RestrictedSchemelnf oBox of Sample Entry (e.g., ' avcl ' or ' hvcl ' ). Media Segments for the Representations of Video Component Adaptation Sets contains one or more track fragments of the corresponding video component track described earlier.

[0070] Table 12 describes an example of DASH MPD for FIG. 7A.Table 12. Example DASH MPD for structure depicted in FIG. 7A

[0071] Table 13 depicts an example of DASH MPD for FIG. 7B.Table 13. DASH MPD for structure depicted in FIG. 7BV-DMC streaming client workflow

[0072] When the V-DMC streaming client joins the session to start the playback, a DASH MPD will be sent to the client. FIG. 8 depicts an example of a V-DMC streaming client workflow after receiving the DASH MPD. The V-DMC streaming client parses the MPD and determines whether multiple segments or a single segment need to be fetched based on the existence of the above VDMC preselection in the MPD. When it needs to prefetch multiple segments, the V-DMC streaming client determine the combination of Adaptation Sets among those available. According to the device capability and bandwidth, the V-DMC streaming client determines which segment(s), among those available, need to be requested for providing a dynamic mesh-based immersive experience. Then, it requests and retrievessegments to the server and starts to decapsulate the received segments. The segments are decapsulated by the workflow (600) described earlier. The decapsulated data is decoded into the V-DMC bitstream and the dynamic mesh is reconstructed and rendered from the decoded signal.ReferencesEach one of the references listed herein is incorporated by reference in its entirety.[1 J ISO / 1EC 23090-29:2024(E), Information technology — Coded representation of immersive media — Part 29: Video-based dynamic mesh coding (V-DMC), Working Draft 2024-03-15.[2] ISO / IEC 23090-10:2021(E), Information technology — Coded representation of immersive media — Part 10: Carriage of visual volumetric video-based coding data, FDIS.[3] ISO / IEC 14496-12:2022 (DIS), Information technology, Coding of audio- visual objects, Part 12: ISO base media file format (ISOBMFF).[4] ISO / IEC 14496-15:2022(E), Information technology — Coding of audio- visual objects— Part 15: Carriage of network abstraction layer (NAL) unit structured video in the ISO base media file format, FDIS.[5] ISO / IEC 23009-1, Information Technology - Dynamic Adaptive Streaming over HTTP (DASH) - Part 1: Media Presentation Description and Segment Formats, Fifth edition, 2022-08.[6] ISO / IEV 23090:5:2023(E), Information technology — Coded representation of immersive media — Part 5: Visual volumetric video-based coding (V3C) and videobased point cloud compression (V-PCC), 2023-02-10, FDIS.EXAMPLE COMPUTER SYSTEM IMPLEMENTATION

[0073] Embodiments of the present invention may be implemented with a computer system, systems configured in electronic circuitry and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA), or another configurable or programmable logic device (PED), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or apparatus that includes one or more of suchsystems, devices or components. The computer and / or IC may perform, control, or execute instructions relating to technologies for video-based dynamic mesh coding (V-DMC), such as those described herein. The computer and / or IC may compute any of a variety of parameters or values that relate to technologies for video-based dynamic mesh coding (V-DMC) described herein. The image and video embodiments may be implemented in hardware, software, firmware and various combinations thereof.

[0074] Certain implementations of the invention comprise computer processors which execute software instructions which cause the processors to perform a method of the invention. For example, one or more processors in a display, an encoder, a set top box, a transcoder, or the like may implement methods related to technologies for video-based dynamic mesh coding (V-DMC) as described above by executing software instructions in a program memory accessible to the processors. Embodiments of the invention may also be provided in the form of a program product. The program product may comprise any non- transitory and tangible medium which carries a set of computer-readable signals comprising instructions which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according to the invention may be in any of a wide variety of non-transitory and tangible forms. The program product may comprise, for example, physical media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted. Where a component (e.g. a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, reference to that component (including a reference to a "means") should be interpreted as including as equivalents of that component any component which performs the function of the described component (e.g., that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated example embodiments of the invention.EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS

[0075] Example embodiments that relate to technologies for video-based dynamic mesh coding (V-DMC) are thus described. In the foregoing specification, embodiments of the present invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and what is intended by the applicants to be the invention, is the set of claimsthat issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

[0076] Various aspects of the present disclosure may be appreciated from the following Enumerated Example Embodiments (EEEs):EEE1. A method to store a video-based dynamic mesh-coding (V-DMC) bitstream according to a media file format, the method comprising: generating a V-DMC bitstream comprising atlas, base mesh, displacement, and attribute sub-streams; packing the V-DMC bitstream as a sequence of visual volumetric video (V3C) units, wherein each V3C unit comprises a V3C unit header and a V3C unit payload, wherein for the displacement and attribute sub-streams, the V3C unit pay load comprises video data units in a network abstraction layer (NAL) representation, for atlas information, the V3C pay load comprises atlas NAL units, and for the base mesh sub-stream the V3C payload comprises base mesh NAL units; and, in a single-track encapsulation, a V-DMC bitstream track includes header information comprising a “v3e2” entry that comprises V3C parameter sets, V-DMC decoder configuration, video decoder configuration, and mesh decoder configuration.EEE2. The method of EEE1, wherein each of the V3C units comprises: a V3C_AD header for atlas NAL units; a V3C_BMD header for base mesh NAL units; a V3C_PVD header for NAL units that combine displacement and attribute (texture) information in a single V3C unit or a V3C_GVD header for displacement NAL units and a V3C_AVD header for attribute NAL units, when attribute and displacement sub-samples are separate in the V-DMC bitstream.EEE3. A method to store a video-based dynamic mesh-coding (V-DMC) bitstream according to a media file format, the method comprising: generating a V-DMC bitstream comprising atlas, base mesh, displacement, andattribute sub-streams; packing the V-DMC bitstream as a sequence of V3C units, wherein each V3C unit comprises a V3C unit header and a V3C unit payload, and for a multi-track encapsulation, the V-DMC bitstream comprises: an atlas track which contains a V3C parameter set and may contain atlas parameter sets and atlas NAL units; an optional V3C video component track comprising units of video-coded elementary streams for displacement data; an optional V3C video component track comprising access units of video-coded elementary streams for texture data; an optional V3C video component track comprising access units of video-coded elementary streams for packed data; and a base mesh track comprising base mesh parameter sets and base mesh NAL units, wherein the atlas track includes track references to other tracks carrying the payloads of video compressed V3C units.EEE4. The method of EEE3, wherein the atlas track references comprise one or more tracks with a four-character code of any one of:'v3vg': denoting the referenced track(s) contain a video-coded displacement component;'v3va': denoting the referenced track(s) contain a video-coded texture component;'v3vp': denoting the referenced track(s) contain a video-coded packed component containing displacement and texture component; and'v3bm': denoting the referenced track(s) contain a coded base mesh component.EEE5. The method of EEE1 or EEE3, wherein the media file format is an ISOBMFF format.EEE6. A method to playback a video-based dynamic mesh-coding (V-DMC) bitstream, the method comprising: accessing a V-DMC bitstream (305 or 310); accessing user controls or sensors to generate a desired viewport; accessing metadata related to the V-DMC bitstream, wherein the metadata identifywhich segments of the V-DMC bitstream are required for the desired viewport; applying a decapsulation process (315) of the V-DMC segments of the desired viewport to generate decapsulated segments; generating a decoded V-DMC stream based on the decapsulated segments; generating a mesh sequence based on the decoded V-DMC stream; and rendering the mesh sequence on a display.EEE7. The method of EEE6, wherein applying the decapsulation process comprises: determining how many tracks need to be extracted from the bitstream; if it is a single track, then extracting V3C unit headers and payloads for atlas, base mesh, displacement, and texture samples from the single track; else: extracting a V3C unit header and samples from an atlas track; generating reference tracks from the atlas track; generating V3C unit headers and samples from the reference tracks; and generating atlas, base mesh, displacement, and texture samples based on the headers and samples in the reference tracks.EEE8. A method to enable video-based dynamic mesh-coding (V-DMC) streaming according to DASH, the method comprising: generating a V-DMC bitstream comprising atlas, base mesh, displacement, and attribute sub-streams; generating a single-track encapsulation comprising an Adaptation set and one or more Representations, wherein Representations within the Adaptation Set use the same codec for the corresponding video components and a base mesh component, wherein if a Representation consists of more than one Media Segment, an Initialization Media Segment is present, wherein the Initialization Segment comprises a v3e2 entry comprising V3C parameter sets, atlas parameter sets, and base mesh parameter sets.EEE9. The method of EEE8, wherein the Initialization Media Segment further comprises a 2D video decoder configuration information box to signal information for a video decoder configuration and initialization.EEE10. The method of EEE8, wherein the Adaptation Set includes a @ codecs attribute comprising three fields:— a four-character code to indicate the segment is in a single-track mode;— video coding information; and— bash mesh coding information.EEE11. The method of EEE 10, wherein the four-character code is v3e2, the video coding information is resv.vvvc.hvcl for HEVC coding, and the base mesh information is drac.1.5 for Google Draco version 1.5 mesh coding.EEE12. A method to enable video-based dynamic mesh-coding (V-DMC) streaming according to DASH, the method comprising: generating a V-DMC bitstream comprising atlas, base mesh, displacement, and attribute sub-streams; generating a multi-track encapsulation wherein the atlas, base mesh, and video component are represented as a separate Adaptation Set in the DASH manifest (MPD) file, together with a V-DMC Component descriptor and a VDMC Preselection element, wherein the VDMC Preselection element enables a player to fetch multiple Adaptation Sets for a single V-DMC stream.EEE13. The method of EEE 12, wherein the V-DMC component descriptor comprises a VDMCComponent descriptor parameter defining an Essential Property element with a @ s cheme I dUri parameter set to "urn : mpeg : mpegl : v3c : 2024 : Component ".EEE14. The method of EEE12, wherein the VDMC Preselection element is signalled either in the MPD using a PreSelection element within a Period element or a with Preselection descriptor at the Adaptation Set level.EEE 15. The method of EEE 14, wherein the VDMC Preselection element is signalled with an id list for a @preselect ionComponent s attribute including an id of the Main Adaptation Set followed by ids of Component Adaptation Sets.EEE16. A method to enable video-based dynamic mesh-coding (V-DMC) playback according to DASH, the method comprising:accessing a DASH media presentation description (MPD) file; parsing the DASH MPD file to determine a number of segments to be fetched based on a VDMC Preselection element comprising an id list for a@pre selectioncomponent s attribute including an id of a Main Adaptation Set followed by ids of Component Adaptation Sets; selecting one or more Adaptation Sets for a selected viewport; accessing from a server segments according to the one or more Adaptation Sets; applying a decapsulation process to the segments of the selected viewport to generate decapsulated segments; generating a decoded V-DMC stream based on the decapsulated segments; generating a mesh sequence based on the decoded V-DMC stream; and rendering the mesh sequence on a display.EEE17. A non- transitory computer-readable storage medium having stored thereon computer-executable instructions for executing with one or more processors a method in accordance with any one of EEE1-EEE16.EEE18. An apparatus comprising a processor and configured to perform any one of the methods recited in EEE1-EEE16.

Claims

CLAIMSWhat is claimed is:

1. A method to store a video-based dynamic mesh-coding (V-DMC) bitstream according to a media file format, the method comprising: generating a V-DMC bitstream comprising atlas, base mesh, displacement, and attribute sub-streams; packing the V-DMC bitstream as a sequence of visual volumetric video (V3C) units, wherein each V3C unit comprises a V3C unit header and a V3C unit payload, wherein for the displacement and attribute sub-streams, the V3C unit payload comprises video data units in a network abstraction layer (NAL) representation, for atlas information, the V3C payload comprises atlas NAL units, and for the base mesh sub-stream the V3C payload comprises base mesh NAL units; and, in a single-track encapsulation, a V-DMC bitstream track includes header information comprising a “v3e2” entry that comprises V3C parameter sets, V-DMC decoder configuration, video decoder configuration, and mesh decoder configuration.

2. The method of claim 1, wherein each of the V3C units comprises: a V3C_AD header for atlas NAL units; a V3C_BMD header for base mesh NAL units; a V3C_PVD header for NAL units that combine displacement and attribute (texture) information in a single V3C unit or a V3C_GVD header for displacement NAL units and a V3C_AVD header for attribute NAL units, when attribute and displacement sub-samples are separate in the V-DMC bitstream.

3. A method to store a video-based dynamic mesh-coding (V-DMC) bitstream according to a media file format, the method comprising: generating a V-DMC bitstream comprising atlas, base mesh, displacement, and attribute sub-streams; packing the V-DMC bitstream as a sequence of V3C units, wherein each V3C unit comprises a V3C unit header and a V3C unit payload, and for a multi-track encapsulation, the V-DMC bitstream comprises: an atlas track which contains a V3C parameter set and may contain atlas parameter setsand atlas NAL units; an optional V3C video component track comprising units of video-coded elementary streams for displacement data; an optional V3C video component track comprising access units of video-coded elementary streams for texture data; an optional V3C video component track comprising access units of video-coded elementary streams for packed data; and a base mesh track comprising base mesh parameter sets and base mesh NAL units, wherein the atlas track includes track references to other tracks carrying the payloads of video compressed V3C units.

4. The method of claim 3, wherein the atlas track references comprise one or more tracks with a four-character code of any one of:'v3vg': denoting the referenced track(s) contain a video-coded displacement component;'v3va': denoting the referenced track(s) contain a video-coded texture component;'v3vp': denoting the referenced track(s) contain a video-coded packed component containing displacement and texture component; and'v3bm': denoting the referenced track(s) contain a coded base mesh component.

5. The method of claim 1 or claim 3, wherein the media file format is an ISOBMFF format.

6. A method to playback a video-based dynamic mesh-coding (V-DMC) bitstream, the method comprising: accessing a V-DMC bitstream (305 or 310); accessing user controls or sensors to generate a desired viewport; accessing metadata related to the V-DMC bitstream, wherein the metadata identify which segments of the V-DMC bitstream are required for the desired viewport; applying a decapsulation process (315) of the V-DMC segments of the desired viewport to generate decapsulated segments; generating a decoded V-DMC stream based on the decapsulated segments; generating a mesh sequence based on the decoded V-DMC stream; andrendering the mesh sequence on a display.

7. The method of claim 6, wherein applying the decapsulation process comprises: determining how many tracks need to be extracted from the bitstream; if it is a single track, then extracting V3C unit headers and payloads for atlas, base mesh, displacement, and texture samples from the single track; else: extracting a V3C unit header and samples from an atlas track; generating reference tracks from the atlas track; generating V3C unit headers and samples from the reference tracks; and generating atlas, base mesh, displacement, and texture samples based on the headers and samples in the reference tracks.

8. A method to enable video-based dynamic mesh-coding (V-DMC) streaming according to DASH, the method comprising: generating a V-DMC bitstream comprising atlas, base mesh, displacement, and attribute sub-streams; generating a single-track encapsulation comprising an Adaptation set and one or more Representations, wherein Representations within the Adaptation Set use the same codec for the corresponding video components and a base mesh component, wherein if a Representation consists of more than one Media Segment, an Initialization Media Segment is present, wherein the Initialization Segment comprises a v3e2 entry comprising V3C parameter sets, atlas parameter sets, and base mesh parameter sets.

9. The method of claim 8, wherein the Initialization Media Segment further comprises a 2D video decoder configuration information box to signal information for a video decoder configuration and initialization.

10. The method of claim 8, wherein the Adaptation Set includes a @ codecs attribute comprising three fields:— a four-character code to indicate the segment is in a single-track mode;— video coding information; and— bash mesh coding information.

11. The method of claim 10, wherein the four-character code is v3e2, the video coding information is resv.vvvc.hvcl for HEVC coding, and the base mesh information is drac. 1.5 for Google Draco version 1.5 mesh coding.

12. A method to enable video-based dynamic mesh-coding (V-DMC) streaming according to DASH, the method comprising: generating a V-DMC bitstream comprising atlas, base mesh, displacement, and attribute sub-streams; generating a multi-track encapsulation wherein the atlas, base mesh, and video component are represented as a separate Adaptation Set in the DASH manifest (MPD) file, together with a V-DMC Component descriptor and a VDMC Preselection element, wherein the VDMC Preselection element enables a player to fetch multiple Adaptation Sets for a single V-DMC stream.

13. The method of claim 12, wherein the V-DMC component descriptor comprises a VDMCComponent descriptor parameter defining an Essential Property element with a @ s cheme !dUri parameter set to "urn : mpeg : mpegI : v3c : 2024 : Component".

14. The method of claim 12, wherein the VDMC Preselection element is signalled either in the MPD using a PreSelection element within a Period element or a with Preselection descriptor at the Adaptation Set level.

15. The method of claim 14, wherein the VDMC Preselection element is signalled with an id list for a @preselect ionComponent s attribute including an id of the Main Adaptation Set followed by ids of Component Adaptation Sets.

16. A method to enable video-based dynamic mesh-coding (V-DMC) playback according to DASH, the method comprising: accessing a DASH media presentation description (MPD) file; parsing the DASH MPD file to determine a number of segments to be fetched based on a VDMC Preselection element comprising an id list for a@pre selectioncomponent s attribute including an id of a Main Adaptation Set followed by ids of Component Adaptation Sets;selecting one or more Adaptation Sets for a selected viewport; accessing from a server segments according to the one or more Adaptation Sets; applying a decapsulation process to the segments of the selected viewport to generate decapsulated segments; generating a decoded V-DMC stream based on the decapsulated segments; generating a mesh sequence based on the decoded V-DMC stream; and rendering the mesh sequence on a display.

17. A non-transitory computer-readable storage medium having stored thereon computer- executable instructions for executing with one or more processors a method in accordance with any one of claims 1-16.

18. An apparatus comprising a processor and configured to perform any one of the methods recited in claims 1-16.

Images