Volume video experience using dynamic mesh encoding

Abstract

Methods and bitstream syntax for supporting volumetric video experiences using video-based mesh coding (V-DMC) are described. For V-DMC file and streaming formats, example syntax is provided for extending the ISOBMFF standard for both single-track encapsulation and multi-track encapsulation of all key parameters in a V-DMC bitstream, such as atlases, base meshes, displacement, and attribute substreams. Example descriptions for DASH media presentation description (MPD) files, V-DMC players, and V-DMC streaming clients are also provided.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 573,018, filed April 2, 2024, which is incorporated herein by reference in its entirety. Technical Field

[0002] This document generally relates to images and videos. More specifically, embodiments of the invention relate to techniques for volumetric video experiences using dynamic mesh coding. Background Technology

[0003] Advances in 3D capture, modeling, and rendering now enable more immersive user experiences. Using 3D meshes is one of many ways to represent immersive volumetric content. The Moving Picture Experts Group (MPEG) is actively working to define coding representation standards for immersive media, particularly video-based dynamic mesh coding (V-DMC) (Reference [1]). As the inventors recognize, this paper describes new techniques for the storage, delivery, and streaming of V-DMC content.

[0004] The term “metadata” in this document refers to any auxiliary information that is transmitted as part of or with the encoded bitstream to assist the decoder in rendering or interpreting one or more decoded images.

[0005] The methods described in this section are permissible but not necessarily methods that have been previously conceived or employed. Therefore, unless otherwise instructed, no method described in this section should be considered prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise instructed, any issues concerning one or more methods should not be considered to be in any prior art based on this section. Attached Figure Description

[0006] Embodiments of the invention are illustrated in the accompanying drawings by way of example rather than limitation, and similar reference numerals refer to similar elements, and in the drawings: Figure 1A An example encoding process for V-DMC bitstreams is described; Figure 1B An example decoding process for the V-DMC bitstream used for encoding is described; Figure 2A An example of a map-based NAL unit in V-DMC encoding is depicted; Figure 2B An example of the basic grid NAL cell in V-DMC encoding is depicted; Figure 3 An example of the V-DMC player process according to an embodiment of the present invention is described; Figure 4A An example of a single-track encapsulation of a V-DMC bitstream according to an embodiment of the present invention is depicted, the V-DMC bitstream comprising an atlas, a base grid, shift and texture sub-bitstreams; Figure 4B An example of a single-track encapsulation of a V-DMC bitstream according to an embodiment of the present invention is depicted, the V-DMC bitstream comprising an atlas, a base grid, and displacement and texture sub-bitstreams; Figure 5A An example of multitrack encapsulation of a V-DMC bitstream according to an embodiment of the present invention is depicted; Figure 5B An example of multitrack encapsulation of a V-DMC bitstream with packed video bitstreams according to an embodiment of the present invention is depicted; Figure 6 An example workflow for decapsulating V-DMC files according to an embodiment of the present invention is described; Figure 7A An example of using pre-selection to group V-DMC components according to an embodiment of the present invention is described; Figure 7B An example of grouping atlases, base grids, and packing video component adaptive sets using pre-selected pairs according to embodiments of the present invention is depicted; and Figure 8 An example of the workflow of a V-DMC streaming client according to an embodiment of the present invention is described. Detailed Implementation

[0007] This document describes example embodiments of techniques relating to video-based dynamic mesh coding (V-DMC). In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of various embodiments of the invention. However, it will be apparent that various embodiments of the invention can be practiced without these specific details. In other instances, well-known structures and devices are not described in detail to avoid unnecessarily obscuring, obscuring, or confusing embodiments of the invention.

[0008] Overview The example embodiments described herein relate to techniques for video-based dynamic mesh coding (V-DMC), including techniques for defining the bearer of video-based dynamic mesh coding data for storage and delivery purposes. More specifically, examples describe how to provide an immersive volumetric video experience by extending both single-track and multi-track encapsulation using the ISO-based Media File Format (ISOBMFF) for storage and the MPEG-DASH format for streaming over HTTP. Embodiments for V-DMC players and V-DMC DASH clients are also presented.

[0009] Video-based Dynamic Mesh Coding (V-DMC) like Figure 1A The depicted dynamic mesh sequence (103) consists of the following components: a base mesh (130), a set of displacements (135), attributes (textures) (140), and an atlas (125). The base mesh components are simplified low-resolution approximations of the original mesh. The base mesh components can be encoded using any mesh codec. The displacement components provide displacement vectors, which can be encoded using any video codec. The attribute components provide additional properties, such as texture or material information, and can be encoded by any video codec. Finally, the atlas components provide the decoding and / or rendering system with information on how to perform inverse reconstruction, such as how to subdivide the base mesh, how to apply the displacement vectors to the subdivided mesh vertices, and how to apply the attributes to the reconstructed mesh. The encoding process generates a V-DMC bitstream (205) containing a set of sub-bitstreams representing the atlas, base mesh, displacements, and attributes.

[0010] like Figure 1B The decoding process, as described, begins by demultiplexing the input V-DMC bitstream (205) into components carrying the base mesh, displacement geometry refinement, attribute, and atlas substreams. Next, each sub-bitstream component is decoded. The base mesh sub-bitstream is decoded by the mesh decoder, and the displacement and attribute sub-bitstreams are decoded by the 2D video codec. When all components are decoded, an approximation of the original mesh with texture can be reconstructed (103b).

[0011] The V-DMC bitstream is a single visual volumetric video (V3C) bitstream, which consists of a sequence of syntax structures called V3C units (reference [2]). Each V3C unit has a V3C unit header and a V3C unit payload. The V3C unit header describes the V3C unit type and additional information, such as attribute identifiers (if needed).

[0012] A specific type of V3C unit contains a V3C parameter set (VPS), which contains a set of parameters applied to the entire sequence.

[0013] The payload of a V3C unit carries displacement and attribute video corresponding to video data units (e.g., NAL (Network Abstraction Layer) units), which can be decoded by an appropriate video decoder. Each video NAL unit contains displacement or attribute video, or both.

[0014] The payload of the V3C cell carrying the atlas contains atlas NAL cells. Each atlas NAL cell contains one of the following: atlas sequence parameter set, atlas frame parameter set, SEI, and atlas tile data, such as... Figure 2A As described in (reference [6]).

[0015] The payload of the V3C cell carrying the base mesh comprises the base mesh NAL cell. Each base mesh NAL cell contains one of the following: a base mesh sequence parameter set, a base mesh frame parameter set, and mesh data, such as... Figure 2B As described in (reference [1]).

[0016] Metadata requirements To assist the V-DMC player in selecting which tracks to extract from a file or which segment(s) to download, the metadata to be transmitted over the channel (i.e., in the file / segment or Media Presentation Description (MPD)) includes the following information: • In the file ○ Entry point track.

[0017] ○ Required capability information to allow only players that support the required capabilities to decode and process V-DMC bitstreams extracted from one or more tracks.

[0018] ○ When multiple tracks need to be extracted, specify the number of tracks to be extracted and the track indicators.

[0019] ○ Decoder configuration information for the V-DMC decoder.

[0020] ○ When any desired operation is required after extracting samples from one or more orbits, extract post-operation information.

[0021] ○ Sample format information, indicating how the V-DMC data unit is encapsulated in the sample of the track.

[0022] • In MPEG DASH MPD ○ Indicates whether multiple segments need to be prefetched.

[0023] ○ Among the available segments, which segment(s) need to be requested to provide an immersive experience.

[0024] ○ Enables the player to request information about specific segments based on device capabilities, network bandwidth, user viewport, etc.

[0025] V-DMC Player Figure 3An example V-DMC player according to an embodiment is depicted. The player may include sensors and a user interface (UI) for two example playback scenarios: via a streaming client (308) or via a media file (310), such as an MP4 file on a user's computer. In the streaming scenario, as previously described, metadata (302) is signaled in a DASH MPD containing all the information needed to enable the player to determine which segments need to be requested from all available segments. The player receives the DASH MPD and determines which segments are appropriate to retrieve in the current viewing situation, such as based on device capabilities or viewport. The player then requests and receives these segments accordingly via a segment interface (305). Alternatively, in the file playback scenario, metadata may also be carried in the file itself (310) to enable the player to determine whether it can play the file.

[0026] The file / fragment decapsulation module (315) extracts the V3C unit sequence and parses the metadata of the file or fragment from the streaming client. Depending on how the V3C unit sequence is encapsulated in the file or streaming fragment, the player may sometimes need to perform post-processing after extracting samples from the track. For example, when the V3C unit sequence is carried across multiple tracks within a file or fragment, V3C unit construction is required by combining sample entry information and sample data units from the tracks before generating the V-DMC bitstream. Alternatively, when the V3C unit sequence is encapsulated in a single track, the extracted V3C unit sequence is identical to the V-DMC bitstream and can be fed to the V-DMC decoder without any post-extraction processing. After optional V-DMC bitstream combination, the V-DMC bitstream is decoded (320) into multiple frames, and the mesh sequence is reconstructed from the decoded signal. The mesh sequence is finally rendered (325) as the output viewport based on the current user viewport, including viewing position and orientation.

[0027] Track package for timing V-DMC in ISOBMFF This section specifies how to carry a single V-DMC bitstream into a file by utilizing the existing capabilities of the ISO Basic Media File Format (ISOBMFF) (Reference [3]), and defines the proposed extensions as needed.

[0028] Two methods are specified, including sample entry definition and how to divide the V-DMC bitstream into samples as specified in the sample format; single-track encapsulation, in which the entire V-DMC bitstream is stored as a single track, and multi-track encapsulation, in which the V-DMC bitstream is stored in multiple tracks.

[0029] The proposed V-DMC box types described in this section are listed in bold in Table 1.

[0030] Table 1. The types of boxes described in this section and their relationship to ISOBMFF boxes.

[0031]

[0032] Single-track package For single-track packaging, the entire V-DMC bitstream is stored directly in the bitstream track, and the V3C cell data structure is preserved in the bitstream. A bitstream sample contains one or more V3C cells belonging to the same presentation time, and each V3C cell corresponds to a subsample.

[0033] V-DMC bitstream decoder-specific information used to initialize the decoder is carried in a sample entry (e.g., 402) of the bitstream track. In embodiments, but not limited to, the bitstream track uses a VolumetricVisualSampleEntry of sample entry type 'v3e2'. The 'v3e2' sample entry contains a V3CConfigurationBox, which contains a V3C parameter set, an atlas parameter set (where NAL cell types are NAL_ASPS, NAL_AAPS, and NAL_AFPS), a displacement parameter set (where NAL cell types are NAL_DSPS and NAL_DFPS), and a base mesh parameter set (where NAL cell types are NAL_BSPS and NAL_BFPS), which are applicable to the samples referencing the sample entry. The 'v3e2' sample entry also contains an SEI message for basic processing of the V-DMC bitstream. The 'v3e2' sample entry contains 2D video decoder configuration information boxes, such as HEVCconfiguationBox and VVCconfiguationBox, as defined in ISO / IEC 14496-15 (reference [4]), to send information for video decoder configuration and initialization using signals. Examples of v3e2 entries are depicted in Table 2.

[0034] Table 2. Examples of sample entry type v3e2.

[0035]

[0036] In Table 2, incrementing `unit_size_precision_bytes_minus1` by 1 specifies the precision of the sample stream V3C unit to which this configuration record applies, in bytes. The value of this field is equal to `ssvh_unit_size_precision_bytes_minus1` in `sample_stream_v3c_header()` as defined below.

[0037]

[0038] The increment of 1 in ssvh_unit_size_precision_bytes_minus1 specifies the precision of the ssvh_v3c_unit_size element in all sample stream V3C units, in bytes.

[0039] num_of_v3c_parameter_sets specifies the number of V3C parameter set units that are sent by signals in the decoder configuration record.

[0040] v3c_parameter_set_length indicates the size of the v3c_parameter_set field in bytes.

[0041] v3c_parameter_set is the V3C cell payload of type V3C_VPS, which contains the V3C parameter set.

[0042] num_of_setup_unit_arrays indicates the number of arrays of NAL units of one or more types of atlases.

[0043] When array_completeness equals 1, it indicates that all NAL cells of the given type are in the subsequent array and none are in the stream; when it equals 0, it indicates that additional NAL cells of the indicated type may be in the stream; the default value and allowed values ​​are constrained by the sample entry name.

[0044] `nal_unit_type` indicates the type of NAL units in the subsequent array (which should all be of this type); it takes one of the values ​​indicating NAL_ASPS, NAL_AAPS, NAL_AFPS, NAL_PREFIX_ESEI, NAL_PREFIX_NSEI, NAL_PREFIX_ESEI, NAL_PREFIX_NSEI, NAL_SUFFIX_ESEI, NAL_SUFFIX_NSEI, NAL_DSPS, NAL_DFPS, NAL_BMSPS, and NAL_BMFPS NAL units.

[0045] num_nal_units indicates the number of NAL units of type nal_unit_type included in the configuration record used for the stream to which it applies.

[0046] The setup_unit_length indicates the size of the setup_unit field in bytes. The length field includes the size of both the NAL unit header and the NAL unit payload, but does not include the length field itself.

[0047] setup_unit contains NAL units based on the relevant nal_unit_type.

[0048] The setup_unit array includes a set of atlas parameters and a set of base grid sub-bitstream parameters that remain constant for the encoded atlas bitstream referenced by sample entries in which decoder configuration records exist, as well as SEI messages.

[0049] A bitstream sample contains one or more V3C units belonging to the same presentation time, and each V3C unit corresponds to a subsample, defined as follows:

[0050] ss_v3c_unit contains a single V3C unit, using the V3C unit sample stream format defined below:

[0051] `ssvu_v3c_unit_size` specifies the size of a V3C unit in the sample stream, in bytes. It is equal to... (V3CDecoderConfigurationRecord.unit_size_precision_bytes_minus1 + 1 ) 8.

[0052] To enable the file decoder to support subsample-level access, the bitstream track includes a SubSampleInformationBox('subs') in the movie header of the bitstream track, which lists the V3C unit headers of the subsamples. The 4-byte v3c_unit_header header() representing the subsample, as defined in ISO / IEC 23090-5 (reference [6]), is copied to the codec_specific_parameters field in the SubSampleInformationBox, and the V3C unit type of each subsample is identified by parsing the codec_specific_parameters field in the SubSampleInformationBox.

[0053] Figure 4AAn example embodiment of a single-track encapsulation of a V-DMC bitstream comprising atlas, base mesh, displacement, and texture sub-bitstreams is depicted. Here, a sample entry (402) contains a V3C parameter set, V-DMC decoder configuration information, video decoder configuration information (e.g., HEVC configuration information), and mesh decoder configuration information. The sample comprises multiple subsamples (405) corresponding to the same presentation time, and each subsample contains a V3C unit consisting of a V3C unit header and a payload, the payload of which contains one of an atlas NAL unit, a base mesh NAL unit, a displacement NAL unit, and a texture NAL unit. In this scenario, the decoder may need to have two video decoders: one for decoding displacement information and one for decoding attribute (texture) information.

[0054] Figure 4B An example of a single-track encapsulation of a V-DMC bitstream is depicted, consisting of an atlas, a base mesh, and packed video sub-bitstreams of displacement and texture. This is the case when a single video encoder is used to encode the packed video bitstreams of displacement and attributes. All decoder initialization information (i.e., the V3C parameter set and all decoder configuration information) is carried in the sample entry (402). All NAL units corresponding to the same presentation time are carried in the same sample, and each subsample contains a V3C unit, which contains one of the atlas NAL unit, the base mesh NAL unit, and the packed video NAL unit containing the displacement and attribute video data.

[0055] Multi-track packaging In an embodiment, such as Figure 5A and Figure 5B The V-DMC bitstream container described, used for multi-track encapsulation of a single V-DMC content, includes the following: • An atlas track containing a V3C parameter set, and the atlas parameter set can be included in sample entries, and atlas NAL cells can be included in samples.

[0056] • Zero or one V3C video component track, wherein the sample contains access units for the video-coded elementary stream of displacement data (i.e., payloads of V3C units of type V3C_GVD).

[0057] • Zero or one V3C video component track, wherein the sample contains access units for the video-coded elementary stream of texture data (i.e., payloads of V3C units of type V3C_AVD).

[0058] • Zero or one V3C video component track, wherein the sample contains access units for the Video Coded Elementary Stream (V3C Elementary Stream) used to pack the data (i.e., payloads of V3C Elementary Streams of type V3C_PVD). When this track exists, a V3C video component track containing access units for both translation and texture data does not exist. Conversely, when a V3C video component track containing access units for both translation and texture data exists, this track does not exist.

[0059] • A base grid orbital that contains a base grid parameter set in the sample entries and base grid NAL cells in the samples.

[0060] For multi-track packaging, the following conditions must be met: • The atlas track is the entry point and includes track references to other tracks that carry the payload of the video compression V3C unit (i.e., V3C unit type equal to V3C_GVD and V3C_AVD, or V3C_PVD), or include track references to the base grid V3C unit (i.e., V3C unit type V3C_BMD).

[0061] • In each track sample entry, there is a single V3CUnitHeaderBox to indicate the role of the bitstream contained in the track.

[0062] • When a sample contains a track with access units for the Video Codec Basic Stream (V3C Stream) used to pack data, a V3C Stream component track containing displacement or texture data does not exist in the same file. Conversely, when a sample contains a V3C Stream component track with displacement or texture data, a V3C Stream component track containing access units for the Video Codec Basic Stream (V3C Stream) used to pack data does not exist in the same file.

[0063] A single V3CUnitHeaderBox exists within the sample entries of V3C atlas tracks, V3C base mesh tracks, and the scheme_specific_data box array of SchemeInformationBoxes for all video component tracks. The V3CUnitHeaderBox contains the V3C unit header describing the sample carried by the corresponding track. Table 3 depicts examples of the "vunt" box type. Table 4 depicts examples of the v3c_unit_header.

[0064] Table 3. Examples of the “vunt” box type

[0065] Table 4. Examples of v3c_unit_header()

[0066] vuh_unit_type indicates the V3C unit type as described in the table below.

[0067]

[0068] vuh_v3c_parameter_set_id specifies the value of vps_v3c_parameter_set_id for an active V3C VPS.

[0069] vuh_atlas_id specifies the atlas ID corresponding to the current V3C unit.

[0070] vuh_attribute_index indicates the index of the attribute data carried in the attribute video data unit.

[0071] vuh_attribute_partition_index indicates the index of the attribute dimension group carried in the attribute video data unit.

[0072] When vuh_map_index exists, it indicates the mapping index of the current geometry or attribute stream.

[0073] A `vuh_auxiliary_video_flag` value of 1 indicates that the associated geometric or attribute video data unit is a video sub-bitstream consisting only of RAW and / or EOM coded points. A `vuh_auxiliary_video_flag` value of 0 indicates that the associated geometric or attribute video data unit may contain RAW and / or EOM coded points.

[0074] Atlas Track Packaging The atlas track uses V3CAtlasSampleEntry, which extends the VolumetricVisualSampleEntry with sample entry types of 'v3c1', 'v3c2', or 'v3cg'. The following constraints are set for the atlas track: • In the track header, track_in_movie is set to 1.

[0075] • V3C Atlas track sample entries contain V3CAtlasConfigurationBox and V3CUnitHeaderBox.

[0076] • The V3CAtlasConfigurationBox contains V-DMC decoder configuration and initialization information applicable to decoding the entire V-DMC bitstream. This includes the V3C parameter set and atlas parameter set, which remain constant for the encoded atlas bitstream referenced by the sample entry, as well as the SEI message.

[0077] • The V3CUnitHeaderBox contains a 32-bit V3C unit header corresponding to the NAL unit of the atlas carried in the track sample.

[0078] Table 5 shows an example of a sample entry (504) in the V3C atlas.

[0079] Table 5. Examples of sample entries in the V3C atlas.

[0080]

[0081] In Table 5, incrementing `unit_size_precision_bytes_minus1` by 1 specifies the precision, in bytes, of the sample stream NAL unit or sample stream V3C unit to which this configuration record applies. For V3C atlas tracks... unit_size_precision_bytes_minus1 is equal to the following definition sample_stream_nal_header() ssnh_unit_size_precision_bytes_minus1:

[0082] The increment of 1 in ssnh_unit_size_precision_bytes_minus1 specifies the precision of the ssnh_nal_unit_size element in all sample stream NAL units, in bytes.

[0083] num_of_v3c_parameter_sets specifies the number of V3C parameter set units that are sent by signals in the decoder configuration record.

[0084] v3c_parameter_set_length indicates the size of the v3c_parameter_set field in bytes.

[0085] v3c_parameter_set is the V3C cell payload of type V3C_VPS, which contains the V3C parameter set.

[0086] num_of_setup_unit_arrays indicates the number of arrays of NAL units of one or more types of atlases.

[0087] When array_completeness equals 1, it indicates that all NAL cells of the given type are in the subsequent array and none are in the stream; when it equals 0, it indicates that additional NAL cells of the indicated type may be in the stream; the default value and allowed values ​​are constrained by the sample entry name.

[0088] nal_unit_type indicates the type of NAL units in the subsequent array (which should all be of this type); it takes one of the values ​​indicating NAL_ASPS, NAL_AAPS, NAL_AFPS, NAL_PREFIX_ESEI, NAL_PREFIX_NSEI, NAL_SUFFIX_ESEI, or NAL_SUFFIX_NSEI NAL units.

[0089] num_nal_units indicates the number of NAL units of type nal_unit_type included in the configuration record used for the stream to which it applies.

[0090] The setup_unit_length indicates the size of the setup_unit field in bytes. The length field includes the size of both the NAL unit header and the NAL unit payload, but does not include the length field itself.

[0091] setup_unit contains NAL units based on the relevant nal_unit_type.

[0092] The setup_unit array includes a set of atlas parameters that remain constant for the CVS referenced by the sample entry, as well as SEI messages.

[0093] Each sample in the V3C atlas track corresponds to a single encoded atlas access unit consisting of atlas NAL units as described below:

[0094] ss_nal_unit contains a single atlas NAL unit in a NAL unit sample stream format defined as follows:

[0095] `ssnu_nal_unit_size` specifies the size of the NAL unit in the sample stream, in bytes. The number of bits used to represent `ssnu_nal_unit_size` is equal to... (V3CDecoderConfigurationRecord.unit_size_precision_bytes_minus1 + 1) 8.

[0096] To link a V3C Atlas track to other tracks, two or more track reference type boxes are added to the track reference box within the V3C Atlas track's track box. The track reference type box contains an array of track_IDs specifying the other tracks referenced by the V3C Atlas track. The reference_type of the track reference type box identifies the type of the video components (i.e., displacement, texture, or packing) and the underlying mesh components. The four-character codes (4CC) for these track reference types are: — 'v3vg': The referenced track contains the video-coded displacement components. — 'v3va': The referenced track contains the texture components of the video encoding. — 'v3vp': The referenced track contains a packed component of video encoding with displacement and texture components. — 'v3bm': The referenced orbitals contain encoded underlying grid components. The type of V3C component transmitted by signal in the restricted scheme information box of the track, carried by the referenced restricted video track, needs to match the reference type of the track reference from the V3C atlas track.

[0097] Basic mesh packaging The base mesh track uses BaseMeshSampleEntry, which extends the VolumetricVisualSampleEntry with sample entry type 'vbm1'. The following constraints are set for the base mesh track: • The basic mesh sample entry contains a BaseMeshConfigurationBox and a V3CUnitHeaderBox.

[0098] • The BaseMeshConfigurationBox contains a set of base mesh parameters (where the NAL cell types are NAL_BSPS and NAL_BFPS) that remain constant for the encoded mesh bitstream referenced by the sample entry, as well as the SEI message.

[0099] • The V3CUnitHeaderBox contains a 32-bit V3C unit header corresponding to the underlying NAL cell in the orbit sample.

[0100] Table 6 describes an example of the basic grid sample entry (510).

[0101] Table 6. Examples of basic grid sample entries.

[0102]

[0103] In Table 6, incrementing `unit_size_precision_bytes_minus1` by 1 specifies the precision, in bytes, of the sample stream NAL unit or sample stream V3C unit to which this configuration record applies.

[0104] num_of_setup_unit_arrays indicates the number of arrays of one or more base grid NAL cells of the indicated type.

[0105] When array_completeness equals 1, it indicates that all basic grid NAL cells of the given type are in the subsequent array and none are in the stream; when it equals 0, it indicates that additional NAL cells of the indicated type can be in the stream; the default value and allowed values ​​are constrained by the sample entry name.

[0106] nal_unit_type indicates the type of NAL units in the subsequent array (which should all be of this type); it takes one of the values ​​indicating NAL_BMSPS, NAL_BMFPS, NAL_PREFIX_ESEI, NAL_PREFIX_NSEI, NAL_SUFFIX_ESEI, or NAL_SUFFIX_NSEI NAL units.

[0107] num_nal_units indicates the number of NAL units of type nal_unit_type included in the configuration record used for the stream to which it applies.

[0108] The setup_unit_length indicates the size of the setup_unit field in bytes. The length field includes the size of both the NAL unit header and the NAL unit payload, but does not include the length field itself.

[0109] setup_unit contains NAL units based on the relevant nal_unit_type.

[0110] The setup_unit array includes a set of base grid parameters that remain constant for the encoded base grid bitstream referenced by the sample entry, as well as the SEI message.

[0111] Each sample (515) in the base grid orbit corresponds to a single encoded base grid access cell consisting of base grid NAL cells as described below:

[0112] ss_nal_unit contains a single base grid NAL cell in NAL cell sample stream format.

[0113] `ssnu_nal_unit_size` specifies the size of the NAL unit in the sample stream, in bytes. The number of bits used to represent `ssnu_nal_unit_size` is equal to... (V3CDecoderConfigurationRecord.unit_size_precision_bytes_minus1 + 1) 8.

[0114] V3C Video Component Packaging V3C video component tracks carry 2D video encoded data for V3C video components, i.e., displacement, texture, or a packed video of displacement and texture. V3C video component tracks are represented as restricted videos in the file and use the generic restricted sample entry 'resv' with additional requirements: • The SchemeTypeBox exists within the RestrictedSchemeInfoBox, and the scheme_type is set to 'vvvc'. • The SchemeInformationBox exists within the RestrictedSchemeInfoBox and contains the V3CUnitHeaderBox.

[0115] • In the track header, the track_in_movie flag is set to 0 to indicate that the track should not exist alone.

[0116] • The V3CUnitHeaderBox contains a 32-bit V3C unit header corresponding to the track sample.

[0117] The restricted video sample entry or sample entry contains a 2D video decoder configuration information box, such as HEVCconfiguationBox, VVCconfiguationBox, as defined in ISO / IEC 14496-15 (reference [4]), to send information on video decoder configuration and initialization for decoding samples in the track. For displacement video tracks, the restricted video sample entry or sample entry contains a decoder configuration information box containing a set of displacement parameters (where the NAL unit types are NAL_DSPS, NAL_DFPS) and an SEI message.

[0118] Each sample in the video component track corresponds to a single coded video access unit.

[0119] Example of multi-track packaging Figure 5A An example of multitrack encapsulation of V-DMC bitstreams is depicted. The payload of the V3C unit of the V-DMC bitstream is mapped to the individual tracks within the multitrack container file based on its type. Each sample entry contains a single V3CUnitHeaderBox to indicate the role of the bitstream contained in the track. Atlas track sample entries contain a V3C parameter set describing the capabilities required to decode and process the entire V-DMC bitstream, as well as the atlas parameter set and SEI applicable to the samples in the track. Base mesh sample entries contain base mesh decoder configuration information, including base mesh parameters and the SEI applicable to the base mesh samples in the track. Constrained video sample entries for displacement and texture video tracks contain 2D video decoder configuration information to signal information for decoding the video samples in the track.

[0120] When the V-DMC bitstream contains a packed video substream in which shifts and textures are packed into video frames, such as Figure 5B As described, only one video component track exists in the file. The restricted video sample entries for the packaged video track contain 2D video decoder configuration information to send signals for decoding the video samples in the track.

[0121] V-DMC file decompression workflow Figure 6 An example embodiment of a V-DMC file decapsulation workflow (600) is described. File decapsulation begins by reading the movie header of the file and determining whether one or more tracks in the file can be processed or decoded. If the file can be processed / decoded, file decapsulation determines the number of tracks to be extracted from the file. If it is a single track, the V3C unit sequence is extracted from the sample and decoded. If multiple tracks need to be extracted, file decapsulation finds the atlas track and extracts the V3C unit header and V3C unit payload from the sample. Next, it reads the track references in the atlas track. Based on these references, it finds the referenced track and extracts the V3C unit header and V3C unit payload from the sample in the referenced track. Based on the extracted data, it constructs V3C units, combines them into a V-DMC bitstream, and decodes the V-DMC bitstream. Then, the dynamic mesh is reconstructed from the decoded signal and rendered.

[0122] Encapsulation and Signaling in MPEG DASH The single-track mode in DASH enables the streaming of V-DMC ISOBMFF files, where the V-DMC bitstream is stored using a single-track encapsulation. A single-track mode in DASH is represented as an adaptive set with one or more representations. Representations within a unique adaptive set use the same codec for the corresponding video components and underlying mesh components.

[0123] If the representation consists of more than one media segment, an initialization media segment is present. The initialization segment contains a V3CConfigurationBox, which contains the V3C parameter set and other parameter sets for the V-DMC components, such as the atlas parameter set and the base mesh parameter set applicable to the media segment. The initialization segment also contains 2D video decoder configuration information boxes, such as HEVCconfigurationBox and VVCconfigurationBox, to signal information for video decoder configuration and initialization.

[0124] The `@codecs` attribute begins with a four-character code (e.g., `v3e2`) to indicate that the clip is in single-track mode and contains a string describing the video encoding and underlying mesh encoding and their parameters. The value of `@codecs` provides three fields, separated by commas: — A four-character code, such as v3e2, is used to indicate that the fragment is in single-track mode. — Video encoding information, for example, when HEVC encoding is applied to encode video components, it is resv.vvvc.hvc1 — Basic trellis encoding information, such as drac.1.5 when Google's Draco (major version 1 and minor version 5) is applied to trellis encoding.

[0125] The media segments represented by this adaptive set contain one or more track segments of the single track previously described.

[0126] The following restrictions apply to certain properties: — @mimeType is 'application / mp4' — @codecs exist at the adaptive set level and signal the maximum capability required to decode any representation within the adaptive set. If the @codecs parameter differs from that at the adaptive set level, it is signaled at the representation level. @codecs at the representation level signal the capability required to decode any component of the representation.

[0127] — For any adaptive set, @maxWidth and @maxHeight do not require signaling.

[0128] — @frameRate is signaled only within the AdaptationSet element, meaning that the value is not different for different representations of an adaptive set.

[0129] — For any representation, @width and @height are not signaled.

[0130] The example of MPD signaling depicted in Table 7 shows that three different bitrate versions of the same V-DMC content are available, and each version contains a specific bitrate version of the V-DMC sequence in single-track mode, where the video components are encoded, for example, by HEVC, and the grid components are encoded, for example, by Google Draco for grids.

[0131] Table 7. Examples of MPD single-track signaling supporting multiple bit rates.

[0132]

[0133] Multitrack mode In multitrack mode, the atlas, base mesh, and video components are represented as separate adaptive sets in the DASH manifest (MPD) file. To identify the type of V-DMC component adaptive set, a V-DMC component descriptor is used. The VDMCComponent descriptor parameter is an Essential Property element with its @schemeIdUri set to "urn:mpeg:mpegI:v3c:2024:Component".

[0134] At the adaptive set level, for the type of V-DMC component present in the representation of the adaptive set, the VDMCComponent descriptor is signaled. The @value of the VDMCComponent descriptor does not exist. The VDMCComponent descriptor includes elements and attributes as specified in Table 8.

[0135] Table 8. Elements and properties used in the V-DMCComponent descriptor

[0136] Table 9 shows an example of the XML schema for the VDMCComponent descriptor.

[0137] Table 9. Example XML schemas for the VDMCCComponent descriptor

[0138] To provide a single experience, DASH clients need to be able to receive a single experience from a combination of adaptive sets. Preselection is used to enable DASH clients to acquire multiple adaptive sets. V-DMC preselection is signaled either using a PreSelection element within a Period element in the MPD or using a Preselection descriptor at the adaptive set level. The VDMCPreSelection element is signaled via a list of ids in the @preselectionComponents attribute, which includes the id of the primary adaptive set followed by the id of the component adaptive set. The @codecs attribute of the preselection is set to 'v3c1', 'v3c2', 'v3cg', or 'v3cb', indicating that the media represented by the preselection is grid-based encoded media.

[0139] A SupplementalProperty element with a @schemeIdUri equal to "urn:mpeg:mpegI:v3c:2020:v3c" is called a V3C descriptor. At most one V3C descriptor can exist in the master adaptive set or VDMC preselection. A V3C descriptor contains the properties defined in Table 10.

[0140] Table 10 – Attributes used for V3C descriptors

[0141] The data types of the attributes are as defined in the XML schema. Table 11 shows the XML schema for V3C descriptors.

[0142] Table 11. Example XML schemas for V3C descriptors

[0143] Multiple versions of the same V-DMC media are sent using separate preselection signals. When more than one V-DMC preselection exists in the MPD for the same V-DMC content, a V3C descriptor with the @vId attribute is signaled for each V-DMC preselection. Preselections representing alternative versions of the same V-DMC content contain V3C descriptors with the same @vId value. At most one V3C descriptor exists at the preselection level. Therefore, these preselections are alternatives to each other, and the ID of the primary adaptive set of the preselections (the first ID in the adaptive set ID list of @preselectionComponents) can be different.

[0144] Figure 7AThe illustration shows an exemplary DASH configuration for grouping V-DMC component adaptive sets belonging to a single V-DMC content. The atlas, base mesh, displacement video, and attribute video exist as separate adaptive sets. A preselection element is used to indicate that the four adaptive sets are grouped to provide a single experience.

[0145] Figure 7B The illustration shows another example of how to indicate the grouping of atlases, base grids, and packing video component adaptive sets belonging to a single V-DMC content.

[0146] The following can be applied to MPEG DASH MPD: — The above PreSelection elements or Preselection descriptors at the adaptive set level exist.

[0147] — The adaptive set used for atlas information is used as the main adaptive set for V-DMC content.

[0148] — At the adaptive set level other than the primary adaptive set, the VDMCComponent descriptor is signaled for the type of the V-DMC component present in the representation of the adaptive set.

[0149] — The atlas adaptation set used for atlas information sets the @codecs attribute to 'v3c1', 'v3c2', or 'v3cg'. The atlas adaptation set contains a single initialization fragment at the adaptation set level. The initialization fragment contains all the parameter sets required to initialize the V-DMC decoder, including the V3C parameter set and other parameter sets for V-DMC components, such as the atlas parameter set, the base mesh parameter set, and 2D video decoder configuration information. The media fragment used for the representation of the atlas adaptation set contains one or more track fragments of the previously described atlas tracks.

[0150] The `@codecs` attribute of the base mesh adaptive set provides three fields, separated by dots: a four-character code indicating the mesh codec, such as `drac` when using Google's Draco; the major version of the mesh codec; and the minor version (if applicable). The media segment represented by the base mesh adaptive set contains one or more track segments corresponding to the video component tracks as previously described.

[0151] — The @codecs attribute of the video component adaptive set or a representation of such adaptive sets, if @codecs is not signaled for the AdaptationSet element, is set based on the corresponding codec used to encode the component. The value of @codecs is set to 'resv.vvvc.XXXX', where XXXX corresponds to the four-character code (4CC) of the video codec from the original_format field in the RestrictedSchemeInfoBox of the sample entry (e.g., 'avc1' or 'hvc1'). The media segment represented by the video component adaptive set contains one or more track segments of the corresponding video component tracks described earlier.

[0152] Table 12 describes the uses for Figure 7A An example of DASH MPD.

[0153] Table 12. Used for Figure 7A The example structure depicted in DASH MPD

[0154] Table 13 describes the use of Figure 7B An example of DASH MPD.

[0155] Table 13. Used for Figure 7B The example structure depicted in DASH MPD

[0156] V-DMC Streaming Client Workflow When a V-DMC streaming client joins a session to begin playback, the DASH MPD will be sent to that client. Figure 8An example of the V-DMC streaming client workflow after receiving a DASH MPD is described. The V-DMC streaming client parses the MPD and determines whether to fetch multiple segments or a single segment based on the presence of the aforementioned VDMC prefetch in the MPD. When it needs to prefetch multiple segments, the V-DMC streaming client determines a combination of adaptive sets from the available adaptive sets. Based on device capabilities and bandwidth, the V-DMC streaming client determines which segment(s) ...

[0157] References Each reference listed in this article is cited in its full text.

[0158] [1] ISO / IEC 23090-29:2024(E), Information technology — Coded representation of immersive media — Part 29: Video-based dynamic mesh coding (V-DMC) [Information technology — Coded representation of immersive media — Part 29: Video-based dynamic mesh coding (V-DMC)], Working draft 2024-03-15.

[0159] [2] ISO / IEC 23090-10:2021(E), Information technology — Coded representation of immersive media — Part 10: Carriage of visual volumetric video-based coding data, FDIS.

[0160] [3] ISO / IEC 14496-12:2022 (DIS), Information technology, Coding of audio-visual objects, Part 12: ISO base media file format (ISOBMFF).

[0161] [4] ISO / IEC 14496-15:2022(E), Information technology — Coding of audio-visual objects — Part 15: Carriage of network abstraction layer (NAL) unit structured video in the ISO base media file format, FDIS.

[0162] [5] ISO / IEC 23009-1, Information Technology – Dynamic Adaptive Streaming over HTTP (DASH) – Part 1: Media Presentation Description and Segment Formats, 5th Edition, 2022-08.

[0163] [6] ISO / IEV 23090:5:2023(E), Information technology — Coded representation of immersive media — Part 5: Visual volumetric video-based coding (V3C) and video-based point cloud compression (V-PCC) [Information technology — Coded representation of immersive media — Part 5: Visual volumetric video-based coding (V3C) and video-based point cloud compression (V-PCC)], 2023-02-10, FDIS.

[0164] Example computer system implementation Embodiments of the present invention may be implemented using computer systems, systems configured with electronic circuits and components, integrated circuit (IC) devices (such as microcontrollers, field-programmable gate arrays (FPGAs) or other configurable or programmable logic devices (PLDs), discrete-time or digital signal processors (DSPs), application-specific integrated circuits (ASICs)), and / or means including one or more of such systems, devices, or components. The computer and / or IC may execute, control, or implement instructions related to techniques for video-based dynamic mesh coding (V-DMC), such as those described herein. The computer and / or IC may calculate any of the various parameters or values ​​related to the techniques for video-based dynamic mesh coding (V-DMC) described herein. Image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0165] Some embodiments of the present invention include a computer processor that executes software instructions that cause the processor to perform the methods of the present invention. For example, one or more processors, such as those in a display, encoder, set-top box, transcoder, etc., can implement the methods described above related to techniques for video-based dynamic mesh coding (V-DMC) by executing software instructions in a processor-accessible program memory. Embodiments of the present invention can also be provided in the form of a program product. The program product may include any non-transitory tangible medium carrying a set of computer-readable signals including instructions that, when executed by a data processor, cause the data processor to perform the methods of the present invention. The program product according to the present invention can take any of a variety of non-transitory and tangible forms. The program product may include, for example, physical media, such as magnetic data storage media including floppy disks and hard disk drives, optical data storage media including CD-ROMs and DVDs, electronic data storage media including ROMs and flash RAMs, etc. The computer-readable signals on the program product may optionally be compressed or encrypted. In the case of the components mentioned above (e.g., software modules, processors, components, devices, circuits, etc.), unless otherwise specified, references to these components (including references to “devices”) should be interpreted as including any component that performs the function of the described component as an equivalent of that component (e.g., functionally equivalent), including components that are structurally different from those that perform the functions in the illustrated exemplary embodiments of the invention.

[0166] Equivalents, extensions, alternatives and miscellaneous Therefore, example embodiments relating to techniques for video-based dynamic mesh coding (V-DMC) are described. In the foregoing description, embodiments of the invention have been described with reference to numerous specific details, which may vary depending on the implementation. Therefore, the sole and exclusive indication of the invention and the applicant's inventive intent is the set of claims published in specific form according to this application, wherein such claim publication includes any subsequent corrections. Any definitions expressly set forth herein with respect to terms contained in such claims shall govern the meaning of such terms as used in the claims. Therefore, any limitations, elements, properties, characteristics, advantages, or attributes not expressly referenced in the claims should not in any way limit the scope of such claims. Therefore, this specification and drawings should be viewed in an illustrative rather than restrictive sense.

[0167] Various aspects of this disclosure can be understood from the following enumerated example embodiments (EEE): EEE1. A method for storing a video-based dynamic mesh coding (V-DMC) bitstream according to a media file format, the method comprising: Generate a V-DMC bitstream that includes atlas, base mesh, displacement, and attribute substreams; The V-DMC bitstream is packaged into a sequence of Visual Volumetric Video (V3C) units, wherein each V3C unit includes a V3C unit header and a V3C unit payload. For the displacement and attribute substreams, the V3C unit payload includes video data units represented by a Network Abstraction Layer (NAL); for atlas information, the V3C payload includes atlas NAL units; and for the base mesh substream, the V3C payload includes base mesh NAL units. Furthermore, in a single-track encapsulation, the V-DMC bitstream track includes header information, which includes a "v3e2" entry. The "v3e2" entry includes a V3C parameter set, V-DMC decoder configuration, video decoder configuration, and mesh decoder configuration.

[0168] EEE2. The method as described in EEE1, wherein each V3C unit in the V3C unit comprises: V3C_AD header for NAL units in atlas; V3C_BMD header for the base mesh NAL cell; The V3C_PVD header is used for NAL units that combine displacement and attribute (texture) information in a single V3C unit, or the V3C_GVD header is used for displacement NAL units and the V3C_AVD header is used for attribute NAL units when attribute and displacement subsamples are separated in the V-DMC bitstream.

[0169] EEE3. A method for storing a video-based Dynamic Mesh Coding (V-DMC) bitstream according to a media file format, the method comprising: Generate a V-DMC bitstream that includes atlas, base mesh, displacement, and attribute substreams; The V-DMC bitstream is packaged into a sequence of V3C units, wherein each V3C unit includes a V3C unit header and a V3C unit payload, and for multi-track encapsulation, the V-DMC bitstream includes: Atlas track, which contains V3C parameter set and can contain atlas parameter set and atlas NAL unit; Includes optional V3C video component tracks for the unit of the video-coded elementary stream used for displacement data; Includes optional V3C video component tracks for accessing the video-encoded elementary stream for texture data; Includes optional V3C video component tracks for accessing the video encoded elementary stream used to pack data; and A base grid track, comprising a base grid parameter set and base grid NAL cells, wherein the atlas track includes track references to other tracks, which carry the payload of the video compression V3C unit.

[0170] EEE4. The method as described in EEE3, wherein the atlas track reference includes one or more tracks having any of the following four-character codes: 'v3vg': indicates that one or more tracks being referenced contain video-coded displacement components; 'v3va': indicates that one or more tracks referenced contain texture components encoded in the video; 'v3vp': indicates that one or more tracks referenced contain packed components of video encoding with displacement and texture components; and 'v3bm': Indicates that one or more referenced tracks contain encoded underlying grid components.

[0171] EEE5. The method as described in EEE1 or EEE3, wherein the media file format is ISOBMFF format.

[0172] EEE6. A method for playing back a video-based Dynamic Mesh Coding (V-DMC) bitstream, the method comprising: Access the V-DMC bitstream (305 or 310); Access user controls or sensors to generate the desired viewport; Access metadata associated with the V-DMC bitstream, wherein the metadata identifies which segments of the V-DMC bitstream are required by the desired viewport; The V-DMC segment of the desired viewport is subjected to a decapsulation process (315) to generate a decapsulated segment; Generate a decoded V-DMC stream based on the decapsulated fragment; A grid sequence is generated based on the decoded V-DMC stream; and The grid sequence is rendered on the display.

[0173] EEE7. The method as described in EEE6, wherein applying the decapsulation process includes: Determine how many tracks need to be extracted from the bitstream; If it is a single track, extract the V3C cell header and payload for atlas, base mesh, displacement and texture samples from the single track; otherwise: Extract V3C unit headers and samples from the atlas tracks; Generate a reference track from the atlas track; Generate V3C cell headers and samples from the referenced track; and Atlases, base meshes, displacements, and texture samples are generated based on the heads and samples in the referenced orbit.

[0174] EEE8. A method for implementing video-based dynamic mesh coding (V-DMC) streaming according to DASH, the method comprising: Generate a V-DMC bitstream that includes atlas, base mesh, displacement, and attribute substreams; Generate a single-track encapsulation comprising an adaptive set and one or more representations, wherein the representations within the adaptive set use the same codec for the corresponding video components and the underlying mesh components. If the representation consists of more than one media segment, then there exists an initialization media segment, wherein the initialization segment includes a v3e2 entry, the v3e2 entry including a V3C parameter set, an atlas parameter set, and a base mesh parameter set.

[0175] EEE9. The method as described in EEE8, wherein the initialization media segment further includes a 2D video decoder configuration information box to signal information for video decoder configuration and initialization.

[0176] EEE10. The method as described in EEE8, wherein the adaptive set includes the @codecs attribute, the @codecs attribute including three fields: — A four-character code indicating that the segment is in single-track mode; — Video encoding information; and — Basic grid coding information.

[0177] EEE11. The method as described in EEE10, wherein the four-character code is v3e2, the video encoding information is resv.vvvc.hvc1 for HEVC encoding, and the base grid information is drac.1.5 for Google Draco version 1.5 grid encoding.

[0178] EEE12. A method for implementing video-based dynamic mesh coding (V-DMC) streaming according to DASH, the method comprising: Generate a V-DMC bitstream that includes atlas, base mesh, displacement, and attribute substreams; A multi-track encapsulation is generated, wherein the atlas, base mesh, and video components, together with V-DMC component descriptors and VDMC Preselection elements, are represented as separate adaptive sets in a DASH manifest (MPD) file, wherein the VDMC Preselection element enables the player to acquire multiple adaptive sets for a single V-DMC stream.

[0179] EEE13. The method as described in EEE12, wherein the V-DMC component descriptor includes a VDMCComponent descriptor parameter that defines the EssentialProperty element, wherein the @schemeIdUri parameter is set to "urn:mpeg:mpegI:v3c:2024:Component".

[0180] EEE14. The method as described in EEE12, wherein the VDMC Preselection element is signaled using a PreSelection element within a Period element in the MPD, or signaled using a Preselection descriptor at the adaptive set level.

[0181] EEE15. The method as described in EEE14, wherein the VDMC Preselection element is signaled via a list of ids of the @preselectionComponents attribute, the list of ids including the id of the master adaptive set followed by the id of the component adaptive set.

[0182] EEE16. A method for implementing video-based dynamic mesh coding (V-DMC) playback according to DASH, the method comprising: Access the DASH Media Presentation Description (MPD) file; The DASH MPD file is parsed to determine the number of segments to be acquired based on the VDMC Preselection element, which includes a list of ids for the @preselectionComponents attribute, the list of ids including the id of the primary adaptive set followed by the id of the component adaptive set; Select one or more adaptive sets for the selected viewport; Access fragments from the server based on one or more adaptive sets; The selected viewport segment is subjected to a decapsulation process to generate a decapsulated segment; Generate a decoded V-DMC stream based on the decapsulated fragment; A grid sequence is generated based on the decoded V-DMC stream; and The grid sequence is rendered on the display.

[0183] EEE17. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for performing, using one or more processors, a method as described in any one of EEE1 to EEE16.

[0184] EEE18. An apparatus including a processor and configured to perform any of the methods described in EEE1 through EEE16.

Claims

1. A method for storing a video-based dynamic mesh coding (V-DMC) bitstream according to a media file format, the method comprising: Generate a V-DMC bitstream that includes atlas, base mesh, displacement, and attribute substreams; The V-DMC bitstream is packaged into a sequence of Visual Volumetric Video (V3C) units, wherein each V3C unit includes a V3C unit header and a V3C unit payload. For the displacement and attribute substreams, the V3C unit payload includes video data units represented by a Network Abstraction Layer (NAL); for atlas information, the V3C payload includes atlas NAL units; and for the base mesh substream, the V3C payload includes base mesh NAL units. Furthermore, in a single-track encapsulation, the V-DMC bitstream track includes header information, which includes a "v3e2" entry. This "v3e2" entry includes a V3C parameter set, V-DMC decoder configuration, video decoder configuration, and mesh decoder configuration.

2. The method as described in claim 1, wherein, Each V3C unit in the V3C unit includes: V3C_AD header for NAL units in atlas; V3C_BMD header for the base mesh NAL cell; The V3C_PVD header is used for NAL units that combine displacement and attribute (texture) information in a single V3C unit, or the V3C_GVD header is used for displacement NAL units and the V3C_AVD header is used for attribute NAL units when attribute and displacement subsamples are separated in the V-DMC bitstream.

3. A method for storing a video-based dynamic mesh coding (V-DMC) bitstream according to a media file format, the method comprising: Generate a V-DMC bitstream that includes atlas, base mesh, displacement, and attribute substreams; The V-DMC bitstream is packaged into a sequence of V3C units, wherein each V3C unit includes a V3C unit header and a V3C unit payload, and for multi-track encapsulation, the V-DMC bitstream includes: Atlas track, which contains V3C parameter set and can contain atlas parameter set and atlas NAL unit; Includes optional V3C video component tracks for the unit of the video-coded elementary stream used for displacement data; Includes optional V3C video component tracks for accessing the video-encoded elementary stream for texture data; Includes optional V3C video component tracks for accessing the video encoded elementary stream used to pack data; and A base grid track, comprising a base grid parameter set and base grid NAL cells, wherein the atlas track includes track references to other tracks, which carry the payload of the video compression V3C unit.

4. The method of claim 3, wherein, The atlas track references include one or more tracks with any of the following four-character codes: 'v3vg': indicates that one or more tracks being referenced contain video-coded displacement components; 'v3va': indicates that one or more tracks referenced contain texture components encoded in the video; 'v3vp': indicates that one or more tracks referenced contain packed components of video encoding with displacement and texture components; and 'v3bm': Indicates that one or more referenced tracks contain encoded underlying grid components.

5. The method as claimed in claim 1 or claim 3, wherein, The media file format is ISOBMFF.

6. A method for playing back a video-based dynamic mesh coding (V-DMC) bitstream, the method comprising: Access the V-DMC bitstream (305 or 310); Access user controls or sensors to generate the desired viewport; Access metadata associated with the V-DMC bitstream, wherein the metadata identifies which segments of the V-DMC bitstream are required by the desired viewport; The V-DMC segment of the desired viewport is subjected to a decapsulation process (315) to generate a decapsulated segment; Generate a decoded V-DMC stream based on the decapsulated fragment; A grid sequence is generated based on the decoded V-DMC stream; and Render the grid sequence on the display.

7. The method of claim 6, wherein, The application of the decapsulation process includes: Determine how many tracks need to be extracted from the bitstream; If it is a single track, extract the V3C cell header and payload for atlas, base mesh, displacement and texture samples from the single track; otherwise: Extract V3C cell headers and samples from the atlas tracks; Generate a reference track from the atlas track; Generate V3C cell headers and samples from the referenced track; and Atlases, base meshes, displacements, and texture samples are generated based on the heads and samples in the referenced orbit.

8. A method for implementing video-based dynamic mesh coding (V-DMC) streaming according to DASH, the method comprising: Generate a V-DMC bitstream that includes atlas, base mesh, displacement, and attribute substreams; Generate a single-track encapsulation comprising an adaptive set and one or more representations, wherein the representations within the adaptive set use the same codec for the corresponding video components and the underlying mesh components. If the representation consists of more than one media segment, then there exists an initialization media segment, wherein the initialization segment includes a v3e2 entry, the v3e2 entry including a V3C parameter set, an atlas parameter set, and a base mesh parameter set.

9. The method of claim 8, wherein, The initialization media segment further includes a 2D video decoder configuration information box to signal information for video decoder configuration and initialization.

10. The method of claim 8, wherein, The adaptive set includes the @codecs attribute, which comprises three fields: — A four-character code indicating that the segment is in single-track mode; — Video encoding information; as well as — Basic grid coding information.

11. The method of claim 10, wherein, The four-character code is v3e2. For HEVC encoding, the video encoding information is resv.vvvc.hvc1, and for Google Draco version 1.5 mesh encoding, the base mesh information is drac.1.

5.

12. A method for implementing video-based dynamic mesh coding (V-DMC) streaming according to DASH, the method comprising: Generate a V-DMC bitstream that includes atlas, base mesh, displacement, and attribute substreams; A multi-track encapsulation is generated, wherein the atlas, base mesh, and video components, together with the V-DMC component descriptors and VDMCPreselection elements, are represented as separate adaptive sets in the DASH manifest (MPD) file, wherein the VDMCPreselection elements enable the player to acquire multiple adaptive sets for a single V-DMC stream.

13. The method of claim 12, wherein, The V-DMC component descriptor includes the VDMCComponent descriptor parameter that defines the EssentialProperty element, where the @schemeIdUri parameter is set to "urn:mpeg:mpegI:v3c:2024:Component".

14. The method of claim 12, wherein, The VDMC Preselection element is used to signal within the Period element in the MPD, or to signal using a Preselection descriptor at the adaptive set level.

15. The method of claim 14, wherein, The VDMC Preselection element is signaled using a list of ids from the @preselectionComponents attribute, which includes the id of the primary adaptive set followed by the id of the component adaptive set.

16. A method for implementing video-based dynamic mesh coding (V-DMC) playback according to DASH, the method comprising: Access the DASH Media Presentation Description (MPD) file; The DASH MPD file is parsed to determine the number of segments to be acquired based on the VDMC Preselection element, which includes a list of ids for the @preselectionComponents attribute, the list of ids including the id of the primary adaptive set followed by the id of the component adaptive set; Select one or more adaptive sets for the selected viewport; Access fragments from the server based on one or more adaptive sets; The selected viewport segment is subjected to a decapsulation process to generate a decapsulated segment; Generate a decoded V-DMC stream based on the decapsulated fragment; A grid sequence is generated based on the decoded V-DMC stream; as well as Render the grid sequence on the display.

17. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for performing the method of any one of claims 1 to 16 using one or more processors.

18. An apparatus comprising a processor and configured to perform any one of the methods described in claims 1 to 16.

Images