Storing multiple atlases from one V-PCC elementary stream in ISOBMFF

Abstract

An apparatus includes means for providing signal information to identify an atlas identifier (602) on a sample of a volumetric media track or on a volumetric media track in a multi-track container; wherein the signal information enables a file parser to link the volumetric media track with a different atlas identifier derived from a volumetric media elementary stream (604); and wherein the file parser is capable of reconstructing the volumetric media elementary stream (606) based on the signal information and data encapsulated in the multi-track container.

Description

Technical Field

[0001] The exemplary and non-limiting embodiments relate generally to multimedia and software, and more particularly to storing multiple atlases from one V-PCC elementary stream in ISOBMFF. Background Art

[0002] Performing video encoding and decoding is known. Summary of the Invention

[0003] According to one aspect, an apparatus includes means for providing signal information to identify an atlas identifier on a sample of a volumetric media track or on a volumetric media track in a multi-track container; wherein the signal information enables a file parser to link the volumetric media track with different atlas identifiers originating from a volumetric media elementary stream; and wherein the file parser is capable of reconstructing the volumetric media elementary stream based on the signal information and data encapsulated in the multi-track container.

[0004] According to one aspect, a method includes providing signal information to identify an atlas identifier on a sample of a volumetric media track or on a volumetric media track in a multi-track container; wherein the signal information enables a file parser to link the volumetric media track with different atlas identifiers originating from a volumetric media elementary stream; and wherein the file parser is capable of reconstructing the volumetric media elementary stream based on the signal information and data encapsulated in the multi-track container.

[0005] According to one aspect, a non-transitory program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform operations comprising: providing signal information to identify an atlas identifier on a sample of or on a volumetric media track in a multi-track container; wherein the signal information enables a file parser to link the volumetric media track with a different atlas identifier derived from a volumetric media elementary stream; and wherein the file parser is capable of reconstructing the volumetric media elementary stream based on the signal information and data encapsulated in the multi-track container. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The foregoing aspects and other features are explained in the following description in conjunction with the accompanying drawings, in which:

[0007] Figure 1 is a block diagram of an example V-PCC elementary stream structure based on the 23090-5 specification.

[0008] Figure 2A and Figure 2B (collectively referred to as FIG2 ) depicts an example ISOBMFF multi-track container structure according to ISO / IEC 23090-10.

[0009] Figure 3A and Figure 3B(collectively referred to as FIG3 ) depicts an example of a multi-track container structure with a V-PCC parameter track.

[0010] Figure 4A and Figure 4B (collectively, FIG4 ) depicts an example of a multi-track container structure with confined V-PCC tracks.

[0011] Figure 5 An example apparatus configured to implement storage of multiple atlases from one V-PCC elementary stream in ISOBMFF.

[0012] Figure 6 is an example method for implementing storage of multiple atlases from one V-PCC elementary stream in ISOBMFF. DETAILED DESCRIPTION

[0013] The following acronyms and abbreviations that may be found in the specification and / or drawings are defined as follows:

[0014] 2D or 2d two-dimensional

[0015] 3D or 3d three-dimensional

[0016] 3DoF+ 3 degrees of freedom and greater

[0017] 3GPP Third Generation Partnership Project

[0018] 4CC four-character code

[0019] CD Committee Draft

[0020] HEVC High-Efficiency Video Codec

[0021] H.26x is a series of video codec standards in the ITU-T field.

[0022] id or ID identifier

[0023] IEC International Electrotechnical Commission

[0024] I / F interface

[0025] I / O Input / Output

[0026] ISO International Organization for Standardization

[0027] ISOBMFF ISO Base Media File Format

[0028] MIV MPEG Immersive Video

[0029] MPEG Moving Picture Experts Group

[0030] MPEG-I MPEG Immersive

[0031] MP4 file format for MPEG-4 Part 14 files

[0032] NAL Network Abstraction Layer

[0033] NW Network

[0034] V3C Visual Volumetric Video Codec

[0035] V-PCC or VPCC video-based point cloud compression

[0036] Box structure file format The box structure and layered file format concept have been widely used for media storage and sharing. The most famous file formats in this regard are the ISO Base Media File Format (ISOBMFF) and its variants such as MP4 and 3GPP file formats.

[0037] ISOBMFF allows for the storage of audio / video media streams captured in time, called media tracks. The metadata describing the tracks is separate from the encoded bitstream itself. The format provides a mechanism for accessing the media data in a codec-independent manner from the perspective of the file parser.

[0038] The basic building block in ISOBMFF is called a box. Each box has a header and a payload. The box header indicates the box type and box size in bytes. The box type is typically identified by an unsigned 32-bit integer, interpreted as a four-character code (4CC). Boxes can contain other boxes, and the ISO file format specifies which box types are allowed within a particular type of box. Furthermore, the presence of certain boxes in each file may be mandatory, while the presence of other boxes may be optional. Furthermore, for certain box types, multiple boxes may be allowed in a file. Therefore, ISOBMFF can be considered to specify a hierarchy of boxes.

[0039] In files conforming to the ISO base media file format, media data may be provided in one or more instances of MediaDataBox (mdat), and MovieBox (moov) may be used to encapsulate metadata for timed media. In some cases, both the "mdat" and "moov" boxes may need to be present for the file to be operable. A "moov" box may contain one or more tracks, and each track may reside in a corresponding TrackBox ("trak"). Each track is associated with a handler identified by a four-character code that specifies the track type.

[0040] Tracks contain samples, such as audio or video frames, volumetric information, or metadata. For video tracks, media samples may correspond to coded pictures or access units. For volumetric media tracks, samples may correspond to coded atlas access units. Media / volume media tracks refer to samples formatted according to the media / volume media compression format (and its encapsulation of the ISO base media file format) (also referred to as media / volume media samples). Timed metadata tracks may refer to samples that describe the referenced media samples.

[0041] The SampleDescriptionBox ("stsd") is contained in the SampleTableBox and provides detailed information about the type of codec being used, as well as any initialization information required by that codec. The information stored in the SampleDescriptionBox after the entry count is specific to the track type and may also have variations within a track type (e.g., different codecs may use different specific information after certain common fields, even within a video track). Which type of sample entry form is used is determined by the media handler, using the appropriate form defined in ISO 14496-12, in a derived specification, or from a registry.

[0042]

[0043]

[0044] A SampleEntry box may contain "additional boxes" that are not explicitly defined in the box syntax of ISO / IEC 14496-12. If present, such a box shall follow all defined fields and shall follow any defined contained boxes. A decoder shall assume that a SampleEntry box may contain additional boxes and shall continue parsing as if they were present until the length of the containing box is exhausted.

[0045] Exactly one TrackReferenceBox ("tref") may be contained within a TrackBox. If this box is not present, the track does not reference any other tracks in any way. The size of the reference array may be filled with reference type boxes. A TrackReferenceBox provides a reference from the containing track to another track in the presentation. These references are typed using TrackReferenceTypeBoxes, where there should be at most one TrackReferenceTypeBox of a given type within a TrackReferenceBox.

[0046]

[0047] For example, a TrackReferenceTypeBox of reference_type 'hint' will link from the containing hint track to the media data it hints to, i.e. the track indicated by the track_IDs array within the TrackReferenceTypeBox.

[0048] The SubSampleInformationBox ("subs") is a box designed to contain information specific to a subsample, where a subsample is a contiguous range of bytes within a sample. A specific definition of subsamples is provided for a given codec (e.g., for ISO / IEC 14496-10, Advanced Video Codec).

[0049]

[0050]

[0051] A TrackGroupBox ("trgr") is contained within a TrackBox. A TrackGroupBox enables the indication of track groups, where each group shares certain characteristics or where the tracks within the group have certain relationships. A TrackGroupBox contains zero or more boxes, and the specific characteristics or relationships are indicated by the box type of the contained boxes. Contained boxes include identifiers that can be used to summarize tracks that belong to the same track group. Tracks that contain contained boxes of the same type within a TrackGroupBox and that have the same identifier value within those contained boxes belong to the same track group. Track groups are not used to indicate dependencies between tracks. Instead, a TrackReferenceBox is used for such a purpose.

[0052] The syntax of TrackGroupBox is as follows:

[0053]

[0054] item An ISOBMFF-compliant file may contain any non-timed object, called an item, meta-item, or metadata item, within a MetaBox ("meta"). Although the name MetaBox refers to metadata, an item may generally contain either metadata or media data. MetaBoxes may reside at the top level of a file, within a MovieBox ("moov"), and within a TrackBox ("trak"), but at most one MetaBox may appear at each level: file level, movie level, or track level.

[0055] An entity group is a grouping of items, which can also group tracks. The entities in an entity group share specific characteristics or have specific relationships, as indicated by the grouping type.

[0056] Entity groups are indicated in the GroupsListBox ("grpl"). Entity groups specified in the GroupsListBox of a file-level MetaBox refer to track- or file-level items. Entity groups specified in the GroupsListBox of a movie-level MetaBox refer to movie-level items. Entity groups specified in the GroupsListBox of a track-level MetaBox refer to track-level items for that track. When a GroupsListBox exists in a file-level MetaBox, there is no item_ID value in the ItemInfoBox ("iinf") in any file-level MetaBox that is equal to the track_ID value in any TrackHeaderBox.

[0057] The GroupsListBox contains EntityToGroupBoxes, each of which specifies an entity group. The four-character box type of the EntityToGroupBox represents the defined grouping type.

[0058] To handle situations where the file author requires the player or renderer to perform certain operations, ISO / IEC 14496-12 specifies a mechanism that enables the player to simply inspect the file to find out such requirements for rendering the bitstream and stop a legacy player from decoding and rendering the file that requires further processing. In other words, the content (track) can only be decoded by a player that renders it correctly.

[0059] A restricted sample entry is defined as a sample entry to which the following transformation process is applied:

[0060] • The four-character code for the sample entry is replaced with a new sample entry code "resv" indicating restricted video.

[0061] The RestrictedSchemeInfoBox is added to the sample description, while all other boxes remain unmodified.

[0062] The original sample entry type is stored in the OriginalFormatBox contained in the RestrictedSchemeInfoBox.

[0063] The original sample entry type is contained in an OriginalFormatBox ("frma") which is located in a RestrictedSchemeInfoBox ("rinf"). The exact nature of the restriction is defined in a SchemeTypeBox ("schm"), and the data required for the scheme is stored in a SchemeInformationBox ("schi").

[0064] Volumetric Video Coding . In the area of ​​volumetric video compression and 3 degrees of freedom and greater (3DoF+) video, a character or scene captured using a set of depth cameras or a character or scene comprehensively modeled and animated as a 3D scene can be encoded as a volumetric video. Volumetric video compression typically partitions the 3D content into a set of 2D tiles, or tiles containing color, geometry, and optional occupancy data, which can then be compressed using a standard 2D video compression format. Therefore, color, geometry, and occupancy data can be considered as components of the volumetric video. Volumetric video compression is currently being explored and standardized in the MPEG-I standardization work.

[0065] Volumetric video compression in 23090-12 can use 3D scene segmentation to generate views that can be packaged into atlases and efficiently encoded using existing 2D compression technologies (such as H.265 or H.264). In order for end users to consume such content, it may be necessary to define a standard metadata format that efficiently describes the information required for view synthesis. Volumetric video compression in the 23090-5 specification defines a similar metadata structure, but limits the number of atlases to one.

[0066] V-PCC basic stream . In addition to defining volumetric video compression, the 23090-5 specification also defines elementary streams that can be used to carry data compressed not only by the standard itself, but also by other standards. An example here is the possibility of using a V-PCC elementary stream to carry data compressed using the 23090-12 specification. The nomenclature of some of the data structures described herein may change during the standardization process. For example, as reflected in ISO / IEC 23090-5, ISO / IEC 23090-10 and ISO / IEC 23090-12, a V-PCC elementary stream is now called a V3C (Visual Volumetric Video Based Codec) elementary stream. Therefore, the use of a V-PCC elementary stream or a V PCC elementary stream, as described and claimed herein, may be used interchangeably with (e.g., as a replacement for) a V3C elementary stream.

[0067] Figure 1 is a block diagram 100 of an example V-PCC basic stream structure based on the 23090-5 specification. Figure 1As shown, a V-PCC elementary stream 102 includes a set of V-PCC units. Each V-PCC unit 104 has a V-PCC unit header and a V-PCC unit payload. The V-PCC unit header describes the V-PCC unit type 106. Supported V-PCC unit types 106 are VPCC_VPS, VPCC_OVD, VPCC_AVD, VPCC_GVD, and VPCC_AD, corresponding to V-PCC parameter sets, occupancy video data, attribute video data, geometry video data, and atlas data, respectively.

[0068] The VPCC_VPS unit type carries the V-PCC parameter set. The V-PCC parameter set contains information that allows the content of the V-PCC elementary stream to be understood and correctly consumed. The V-PCC parameter set is shared among all the graph sets in the V-PCC elementary stream.

[0069] VPCC_OVD, VPCC_AVD, VPCC_GVD unit types carry occupancy, attribute, and geometry video data. This data corresponds to a video data unit that can be decoded by an appropriate video decoder (e.g., HEVC NAL unit).

[0070] The VPCC_AD unit type carries atlas data. This data corresponds to an atlas NAL unit containing metadata information. This metadata is used together with the V-PCC parameter set and video data to reconstruct the coded volume structure (e.g., point cloud when the metadata conforms to 23090-5, or 3DoF+ video when the metadata conforms to 23090-12).

[0071] When multiple atlases exist in a V-PCC elementary stream, each atlas can be uniquely identified and parsed using the vuh_atlas_id defined in the VPCC unit header.

[0072]

[0073] Design of ISOBMFF for V-PCC. According to the current CD version of 23090-10, the multi-track container structure is to encapsulate a V-PCC elementary stream in ISOBMFF into a V-PCC track (a form of volume media track) and multiple V-PCC component tracks (a form of video media track), see Figure 2. In particular, Figure 2 is an example ISOBMFF multi-track container structure 200 according to ISO / IEC 23090-10. The V-PCC track 202 contains V-PCC parameter sets and NAL units carrying atlas data information. The V-PCC component tracks contain video coding information (occupancy 204, geometry 206, attributes 208).

[0074] When encapsulating a V-PCC elementary stream in ISOBMFF, each V-PCC unit header is stored in a VPCCUnitHeaderBox ("vunt"). In V-PCC track 202, the VPCCUnitHeaderBox is stored in a VPCCSampleEntry. In V-PCC component tracks (such as entries 204, 206, and 208), the VPCCUnitHeaderBox is in the SchemeInformation in the RestrictedVideoSampleEntry. The VPCCUnitHeaderBox allows identification of the components carried by the V-PCC component track. In both cases, it is assumed that the data in the VPCCUnitHeaderBox does not change for the entire duration of the track.

[0075] Multiple atlases in a V-PCC elementary stream . MPEG 128 compliant V-PCC elementary bitstreams can contain more than one atlas (starting from ISO / IEC 23090-5, ISO / IEC 23090-10 and ISO / IEC 23090-12, V-PCC elementary bitstreams are now called V3C (Visual Volumetric Video based Codec) elementary streams). This functionality was added to carry data encoded according to the MIV specification (23090-12). To enable this functionality, vuh_atlas_id is added to the V-PCC unit header of the V-PCC unit with the types: VPCC_AD, VPCC_GVD, VPCC_OVD and VPCC_AVD.

[0076] When it comes to V-PCC tracks, the addition of vuh_atlas_id has an impact on the design of the multi-track container structure. For example (although the examples described in this article address issues not limited to the following description):

[0077] A V-PCC sample is only allowed to store one V-PCC unit payload. Therefore,

[0078] A V-PCC track must be created for each atlas.

[0079] There is no functionality for linking multiple V-PCC tracks to the same V-PCC elementary stream.

[0080] No information about how VPCC_VPS is stored (e.g. whether each V-PCC track will be duplicated)

[0081] When there is more than one V-PCC track, there is no function to signal to the main V-PCC track.

[0082] There is no function for enabling sharing of common data between V-PCC tracks.

[0083] V-PCC component tracks can be created without modification because from their perspective, vuh_atlas_id is just another identifier for the track, similar to vuh_unit_type, vuh_attribute_index, vuh_map_index and vuh_attribute_dimension_index.

[0084] Partial access in V-PCC ISOBMFF. The CD-Text of 23090-10 also introduces an advanced solution for partial access to 3D space, where tracks are grouped based on the content to which the spatial region data in these tracks belong. However, the specification does not mention whether these tracks come from one V-PCC elementary stream or from independent V-PCC elementary streams. In the case of one V-PCC elementary stream: the tool only allows splitting video tracks based on the content to which the spatial region data in these tracks belong (occupancy, geometry, attributes). The atlas data in the V-PCC tracks remains in its original form. Therefore, even if only one spatial region is to be displayed to the end user, all atlas data needs to be downloaded / decoded. Splitting the atlas data into a number of atlases corresponding to certain spatial regions will help in partial access scenarios. However, as mentioned above, storing multiple atlases in a multi-track container structure is not fully supported.

[0085] So, in summary, the examples described in this article address the following issues:

[0086] There is no functionality for storing multiple atlases (i.e. V-PCC units of type VPCC_AD with different vuh_atlas_id) in one V-PCC track

[0087] There is no functionality for linking multiple V-PCC tracks to the same V-PCC elementary stream.

[0088] oThis function is required to access some point clouds

[0089] o Alternatively, multiple atlases may be stored, i.e. not in one V-PCC track.

[0090] If multiple V-PCC tracks are created, there is no design on how the VPCC_VPS is stored if multiple atlases exist (e.g., will it be duplicated for each V-PCC track?).

[0091] · It does not consider how to store the camera_params_list of MIV (23090-12) common to multiple atlases in ISOBMFF

[0092] US Provisional Application No. 62 / 853903, filed by the assignee of the examples described herein (Nokia), describes a single-track packaging scheme using a similar (but different) approach to the one of the embodiments described herein (see Storing Multiple Atlases in One V-PCC Track - Embodiment 4).

[0093] The examples described herein provide signaling information to identify samples in a multi-track container or atlas IDs on a V-PCC track. Additionally, this signaling ensures that a file parser can link a V-PCC track with different atlas IDs originating from a single V-PCC elementary stream. Thus, a file parser can reconstruct the original V-PCC elementary stream based on the signaling information and the data encapsulated in the multi-track container.

[0094] Storing multiple atlases in one V-PCC track .

[0095] 1. In one embodiment ,

[0096] · VPCCUnitHeaderBox is not stored in VPCCSampleEntry as currently defined in the CD text of 23090-10.

[0097] A new definition of VPCCSample is specified for the V-PCC track, where "Each sample in a V-PCC track corresponds to a frame. The samples corresponding to that frame in the respective component tracks shall have the same composition time as the samples of the V-PCC track. Each V-PCC sample shall contain one or more V-PCC unit payloads of type VPCC_AD, which have the same composition time as defined in ISO / IEC 23090-5 [VPCC], which may include one or more atlas NAL units."

[0098] vuh_atlas_id maps to the atlas_id in the VPCCSample structure of each nalUnit

[0099]

[0100] 2. In the second embodiment ,

[0101] · VPCCUnitHeaderBox is not stored in VPCCSampleEntry as currently defined in the CD text of 23090-10.

[0102] A new definition of VPCCSample is specified for the V-PCC track, where "Each sample in a V-PCC track corresponds to a single frame. The samples corresponding to that frame in the respective component tracks shall have the same composition time as the V-PCC track samples. Each V-PCC sample shall contain one or more V-PCC unit payloads of type VPCC_AD, which have the same composition time as defined in ISO / IEC 23090-5 [VPCC], which may include one or more atlas NAL units."

[0103] vuh_atlas_id maps to the atlas_id in the VPCCSample structure of a set of nalUnits

[0104]

[0105] 3. In another embodiment ,

[0106] · VPCCUnitHeaderBox is not stored in VPCCSampleEntry as currently defined in the CD text of 23090-10.

[0107] A new definition of VPCCSample is specified for the V-PCC track, where "Each sample in a V-PCC track corresponds to a single frame. The samples corresponding to that frame in the respective component tracks shall have the same composition time as the V-PCC track samples. Each V-PCC sample shall contain one or more V-PCC unit payloads of type VPCC_AD, which have the same composition time as defined in ISO / IEC 23090-5 [VPCC], which may include one or more atlas NAL units."

[0108] vuh_atlas_id is implicitly mapped to nalUnits in VPCCSample

[0109]

[0110] 4. In another embodiment,

[0111] · VPCCUnitHeaderBox is not stored in VPCCSampleEntry as currently defined in the CD text of 23090-10.

[0112] A new definition of VPCCSample is specified for the V-PCC track, where "Each sample in a V-PCC track corresponds to a frame. The samples corresponding to that frame in the respective component tracks shall have the same composition time as the samples of the V-PCC track. Each V-PCC sample shall contain one or more V-PCC unit payloads of type VPCC_AD, which have the same composition time as defined in ISO / IEC 23090-5 [VPCC], which may include one or more atlas NAL units."

[0113] VPCCSample is constructed according to 23090-10 or embodiment 3, but does not assume implicit mapping.

[0114] The SubSampleInformationBox is part of the V-PCC track and contains the VPCCUnitHeaderBox in the codec_specific_parameters field. The VPCCUnitHeaderBox links the subsample nalUnits in the VPCCSample to the vuh_atlas_id.

[0115] 5. In another embodiment ,

[0116] · VPCCUnitHeaderBox is not stored in VPCCSampleEntry as currently defined in the CD text of 23090-10.

[0117] A new definition of VPCCSample is specified for V-PCC tracks, where a sample no longer contains an atlas_sub_bitstream but a sequence of vpcc_units with both payload and header.

[0118] "Each sample in a V-PCC track corresponds to a frame. The samples corresponding to that frame in the component tracks shall have the same composition time as the V-PCC track samples. Each V-PCC sample shall contain one or more V-PCC units of type VPCC_AD with the same composition time as defined in ISO / IEC 23090-5 [VPCC], which may include one or more atlas NAL units."

[0119]

[0120] Specify a new grouping type Utilizing the new grouping types (entity grouping, track grouping) allows for faster access to information, as parsing track groups comes earlier in the parsing hierarchy than parsing the tracks themselves.

[0121] 6. In another embodiment ,

[0122] The grouping structure should contain information about the atlas id and an explicit indicator to signal whether the track in question should be considered the main track

[0123] A new grouping type (e.g., "vpes") is specified for the tracking group mechanism, as shown below.

[0124] definition

[0125] Box type: 'vpes'

[0126] Container: TrackGroupBox

[0127] Mandatory: No

[0128] Quantity: zero or more

[0129] grammar

[0130]

[0131] Semantics

[0132] track_group_id is an integer specifying a track group or set. If VPCCElementaryStreamGroupBox is not present, there is no information about whether this track is linked with other V-PCC tracks. track_group_id should be the same for tracks derived from the same V-PCC elementary stream.

[0133] main_track equal to 1 indicates that this V-PCC track contains common data of all other V-PCC tracks belonging to the same group (eg, contains VPCC parameter sets in VPCCDecoderConfigurationRecord), while main_track equal to 0 indicates that this V-PCC track does not contain common data.

[0134] atlas_id is an integer that identifies the atlas to which the data carried in this V-PCC track belongs.

[0135] 7. In another embodiment,

[0136] The grouping structure should contain an atlas id. A track with a specific atlas id can be implicitly considered as the primary track, for example, atlas_id == 0.

[0137] A new grouping type (e.g., "vpes") is specified for the tracking group mechanism, as shown below.

[0138] definition

[0139] Box type: 'vpes'

[0140] Container: TrackGroupBox

[0141] Mandatory: No

[0142] Quantity: zero or more

[0143] grammar

[0144]

[0145] Semantics

[0146] track_group_id is an integer specifying a track group or set. If VPCCElementaryStreamGroupBox is not present, there is no information about whether this track is linked with other V-PCC tracks. track_group_id should be the same for tracks derived from the same V-PCC elementary stream.

[0147] atlas_id is an integer that identifies the atlas to which the data carried in this V-PCC track belongs.

[0148] 8. In another embodiment,

[0149] The packet structure shall contain a vpcc unit header which contains information about the atlas id and an explicit indicator of the primary track.

[0150] A new grouping type (e.g., "vpes") is specified for the tracking group mechanism, as shown below.

[0151] definition

[0152] Box type: 'vpes'

[0153] Container: TrackGroupBox

[0154] Mandatory: No

[0155] Quantity: zero or more

[0156] grammar

[0157]

[0158] Semantics

[0159] track_group_id is an integer specifying a track group or set. If VPCCElementaryStreamGroupBox is not present, there is no information about whether this track is linked with other V-PCC tracks. track_group_id should be the same for tracks derived from the same V-PCC elementary stream.

[0160] main_track equal to 1 indicates that this V-PCC track contains common data of all other V-PCC tracks belonging to the same group (eg, contains VPCC parameter sets in VPCCDecoderConfigurationRecord), while main_track equal to 0 indicates that this V-PCC track does not contain common data.

[0161] VPCCUnitHeaderBox() is as defined in 23090-10 9. In another embodiment,

[0162] A track group should only contain the VPCC unit header for the track in question.

[0163] A new grouping type (e.g., "vpes") is specified for the tracking group mechanism, as shown below.

[0164] definition

[0165] Box type: 'vpes'

[0166] Container: TrackGroupBox

[0167] Mandatory: No

[0168] Quantity: zero or more

[0169] grammar

[0170]

[0171] Semantics

[0172] track_group_id is an integer specifying a track group or set. If VPCCElementaryStreamGroupBox is not present, there is no information about whether this track is linked with other V-PCC tracks. track_group_id should be the same for tracks derived from the same V-PCC elementary stream.

[0173] VPCCUnitHeaderBox() is as defined in 23090-10. In another embodiment,

[0174] Two new grouping types are specified for the tracking group mechanism (e.g., "vpes" and

[0175] "atls"), as shown below.

[0176] definition

[0177] Box type: 'vpes'

[0178] Container: TrackGroupBox

[0179] Mandatory: No

[0180] Quantity: zero or more

[0181] grammar

[0182]

[0183] Semantics

[0184] track_group_id is an integer specifying a track group or set. If VPCCElementaryStreamGroupBox is not present, there is no information about whether this track is linked to other tracks. track_group_id should be the same for tracks derived from the same V-PCC elementary stream.

[0185] definition

[0186] Frame type: 'atls'

[0187] Container: TrackGroupBox

[0188] Mandatory: No

[0189] Quantity: zero or more

[0190] grammar

[0191]

[0192] Semantics

[0193] track_group_id is an integer specifying the track group or collection. If AtlasGroupBox is not present, there is no information about which atlas this track belongs to. track_group_id should be the same for tracks belonging to the same atlas.

[0194] atlas_id is an integer that identifies the atlas to which the data carried in this track belongs.

[0195] Specify a new V-PCC parameter track FIG3 depicts an example of a multi-track container structure 300 with a V-PCC parameter track 302 .

[0196] 11. In another embodiment,

[0197] A new V-PCC sample entry “vpcp” 304 and a new track reference of type “pccp” 306 specifying a V-PCC track (V-PCC parameter track 302 ).

[0198] VPCCUnitHeaderBox 308 is not stored in VPCCSampleEntry of type "vpcp"

[0199] A V-PCC track (V-PCC parameter track 302) with a VPCCSampleEntry of type "vpcp" 304 contains data common to all VPCC Tracks originating from the same V-PCC elementary stream (e.g., MIV camera parameters common to the VPCC as a whole or all atlases of type VPCC_VPS stored in a VPCCDecoderConfigurationRecord)

[0200] A track reference of type "pccp" 306 in the V-PCC track reference table 310 (referring to the V-PCC parameter track 302) is used to connect from a V-PCC track 312 to the V-PCC parameter track 302 (FIG. 3).

[0201] Designation of new restricted V-PCC tracks FIG. 4 is an example of a multi-track container structure 400 with confined V-PCC tracks 420 .

[0202] 12. In another embodiment,

[0203] A new V-PCC sample entry "vpcr" of type "pccr" 426 and a new restrictedVPCCampleEntry are specified for the V-PCC track 424.

[0204] "pcrr"428.

[0205] VPCCUnitHeaderBox 430 is not stored in VPCCSampleEntry of type "vpcr"

[0206] A V-PCC track 424 with a VPCCSampleEntry of type "vpcr" contains only common data 432 for all RestrictedVPCCTracks (e.g., MIV camera parameters common to the VPCC as a whole or all atlases of type VPCC_VPS stored in a VPCCDecoderConfigurationRecord)

[0207] Restricted VPCCSampleEntry "pcrr" 428 contains RestrictedSchemeInfo

[0208] 434, the RestrictedSchemeInfo 434 contains SchemeInformationBox

[0209] 436, SchemeInformationBox 436 eventually contains VPCCUnitHeaderBox

[0210] 430.

[0211] A track reference of type “pcrr” 426 in a V-PCC track 424 with a VPCCSampleEntry of type “vpcr” is used to connect from the V-PCC track 424 to all RestrictedVPCCTracks 420 ( FIG. 4 ).

[0212] Figure 5 1 is an exemplary apparatus 500 based on the examples described herein, which can be implemented in hardware and configured to implement storage of multiple atlases from a V-PCC elementary stream in ISOBMFF. The apparatus 500 includes a processor 502 and at least one non-transitory memory 504 including computer program code 505, wherein the at least one memory 504 and the computer program code 505 are configured to, together with the at least one processor 502, cause the apparatus to implement atlas signaling, circuitry, encoders, decoders, processes, components, modules, or functions (collectively, signaling 506) to implement storage of multiple atlases from a V-PCC elementary stream in ISOBMFF based on the examples described herein. The apparatus 500 optionally includes a display and / or I / O interface 508, which can be used to display output or content as a result of the signaling 506 during rendering. The apparatus 500 optionally includes one or more network (NW) interfaces (I / Fs) 510. NW I / F(s) 510 can be wired and / or wireless and communicate via the Internet / other network(s) using any communication technology. NW I / F(s) 510 can include one or more transmitters and one or more receivers. N / WI / F(s) 510 can include standard, well-known components such as amplifiers, filters, frequency converters, (de)modulators and encoder / decoder circuits, as well as one or more antennas.

[0213] Device 500 can be a remote, virtual, or cloud device. Device 500 can be an encoder or a decoder, or both. Memory 504 can be implemented using any suitable data storage technology, such as semiconductor-based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. Memory 504 can include a database for storing data.

[0214] Figure 6An example method 600 for storing multiple atlases from a V-PCC elementary stream in ISOBMFF based on an example implementation described herein is provided. At 602, the method includes providing signal information to identify an atlas identifier on a sample of a volumetric media track or on a volumetric media track in a multi-track container. At 604, the method includes wherein the signal information enables a file parser to link the volumetric media track with different atlas identifiers originating from the volumetric media elementary stream. At 606, the method includes wherein the file parser is able to reconstruct the volumetric media elementary stream based on the signal information and data encapsulated in the multi-track container.

[0215] References to "computers," "processors," and the like should be understood to include not only computers having different architectures, such as single / multi-processor architectures and sequential (von Neumann) / parallel architectures, but also specialized circuits, such as field programmable gate arrays (FPGAs), application specific circuits (ASICs), signal processing devices, and other processing circuit systems. References to computer programs, instructions, code, and the like should be understood to encompass software for a programmable processor or firmware, for example, the programmable content of a hardware device, such as instructions for a processor, or configuration settings for a fixed-function device, gate array, or programmable logic device, and the like.

[0216] As used herein, the term "circuitry" may refer to any of the following: (a) a hardware circuit implementation, such as an implementation using analog and / or digital circuitry, and (b) a combination of circuitry and software (and / or firmware), such as, as applicable: (i) a combination of processor(s), or (ii) portions of processor(s) / software, including digital signal processor(s), software, and memory(s), which work together to enable the device to perform various functions, and (c) circuitry, such as microprocessor(s) or portions of microprocessor(s), that requires software or firmware to operate, even if the software or firmware is not physically present. This description of "circuitry" applies to the use of the term in this application. As a further example, as used herein, the term "circuitry" would also encompass an implementation of only a processor(s) or a portion of a processor(s) and its(their) accompanying software and / or firmware. For example, if applicable to the particular component, the term "circuitry" would also encompass a baseband integrated circuit or applications processor integrated circuit for a mobile phone, or a similar integrated circuit in a server, cellular network device, or other network device.

[0217] An example apparatus includes at least one processor; and at least one non-transitory memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least perform: providing signal information to identify an atlas identifier on a sample of a volumetric media track or on the volumetric media track in a multi-track container; wherein the signal information enables a file parser to link the volumetric media track with a different atlas identifier derived from a volumetric media elementary stream; and wherein the file parser is capable of reconstructing the volumetric media elementary stream based on the signal information and data encapsulated in the multi-track container.

[0218] The apparatus may further include, wherein the signal information comprises: a volumetric media sample entry for a volumetric media parameter track, comprising common data of a volumetric media track originating from a volumetric media elementary stream; and a track reference in a volumetric media parameter track reference table, which references the volumetric media track and is used to connect from the volumetric media parameter track to the volumetric media track.

[0219] The apparatus may further include where the volumetric media unit header atlas identifier is mapped to an atlas identifier within a volumetric media sample structure for each of the one or more network abstraction layer units.

[0220] The apparatus may further include, wherein the volumetric media unit header atlas identifier is mapped to an atlas identifier within a volumetric media sample structure for a set of network abstraction layer units.

[0221] The apparatus may further include, wherein the volumetric media unit header atlas identifier is implicitly mapped to a network abstraction layer unit within the volumetric media sample structure.

[0222] The apparatus may further include, wherein the volumetric media track comprises a subsample information box, and wherein the subsample information box comprises a volumetric media unit header box in a codec specific parameter field, the volumetric media unit header box linking a subsample network abstraction layer unit in the volumetric media sample to a volumetric media unit header atlas identifier.

[0223] The apparatus may further include where a sample of the volumetric media track comprises a sequence of volumetric media units having a payload and a header, and the sample corresponds to a single point cloud frame, wherein the samples within the one or more component tracks corresponding to the single point cloud frame have a composition time corresponding to a composition time of the sample of the volumetric media track.

[0224] The apparatus may further include, wherein the signal information comprises a grouping structure including information about an atlas identifier; and an indicator for signaling whether the volumetric media track is a master track having common data for other volumetric media tracks belonging to the group.

[0225] The apparatus may further include, wherein the signal information comprises a grouping structure, the grouping structure comprising an atlas identifier, wherein a track having a particular atlas identifier is implicitly considered to be a master track having common data for other volumetric media tracks belonging to the group.

[0226] The apparatus may further include, wherein the signal information comprises a packet structure, the packet structure comprising a volumetric media unit header, wherein the volumetric media unit header comprises information about an atlas identifier and an indicator of a main track having common data for other volumetric media tracks belonging to the group.

[0227] The apparatus may further include, wherein the signal information comprises a track group including a volumetric media unit header for the volumetric media track.

[0228] The apparatus may further include, wherein the signal information provides at least one grouping type to specify a track grouping mechanism.

[0229] The apparatus may further include, wherein the signal information comprises: a volumetric media sample entry comprising common data of a volumetric media track originating from a volumetric media elementary stream; and a track reference in a volumetric media track reference table, which references the volumetric media track and is used to connect from the volumetric media parameter track to the volumetric media track.

[0230] The apparatus may further include, wherein the signal information comprises: a volumetric media sample entry comprising common data of a certain type of constrained volumetric media track; a constrained sample entry comprising constrained scheme information; and a track reference in a volumetric media parameter track having a sample entry for connecting from the volumetric media parameter track to the constrained volumetric media track.

[0231] An example method includes providing signal information to identify an atlas identifier on a sample of a volumetric media track or on the volumetric media track in a multi-track container; wherein the signal information enables a file parser to link the volumetric media track with a different atlas identifier derived from a volumetric media elementary stream; and wherein the file parser is capable of reconstructing the volumetric media elementary stream based on the signal information and data encapsulated in the multi-track container.

[0232] Other aspects of the method may include the following. The signal information may include: a volumetric media sample entry for a volumetric media parameter track, comprising common data from the volumetric media tracks of the volumetric media elementary stream; and a track reference in a volumetric media parameter track reference table, which references the volumetric media track and is used to connect from the volumetric media parameter track to the volumetric media track. For each of one or more network abstraction layer units, a volumetric media unit header atlas identifier may be mapped to an atlas identifier within a volumetric media sample structure. The volumetric media unit header atlas identifier may be mapped to an atlas identifier within a volumetric media sample structure for a group of network abstraction layer units.

[0233] An example apparatus includes means for providing signal information to identify an atlas identifier on a sample of a volumetric media track or on a volumetric media track in a multi-track container; wherein the signal information enables a file parser to link the volumetric media track with different atlas identifiers derived from a volumetric media elementary stream; and wherein the file parser is capable of reconstructing the volumetric media elementary stream based on the signal information and data encapsulated in the multi-track container.

[0234] Other aspects of the apparatus may include the following. The signal information may include: a volumetric media sample entry for a volumetric media parameter track, comprising common data from a volumetric media track of a volumetric media elementary stream; and a track reference in a volumetric media parameter track reference table, which references the volumetric media track and is used to connect from the volumetric media parameter track to the volumetric media track. For each of one or more network abstraction layer units, a volumetric media unit header atlas identifier may be mapped to an atlas identifier within a volumetric media sample structure. The volumetric media unit header atlas identifier may be mapped to an atlas identifier within a volumetric media sample structure for a group of network abstraction layer units. The volumetric media unit header atlas identifier may be implicitly mapped to a network abstraction layer unit within the volumetric media sample structure. The volumetric media track may include a subsample information box, and the subsample information box may include a volumetric media unit header box in a codec specific parameter field, the volumetric media unit header box linking the subsample network abstraction layer unit in the volumetric media sample to the volumetric media unit header atlas identifier. A sample of a volumetric media track may include a sequence of volumetric media units having a payload and a header, and the sample may correspond to a single point cloud frame, wherein samples corresponding to the single point cloud frame within one or more component tracks have a composition time corresponding to a composition time of the sample of the volumetric media track. Signal information may include: a grouping structure including information about an atlas identifier; and an indicator for signaling whether the volumetric media track is a master track, the master track having common data for other volumetric media tracks belonging to the group. Signal information may include a grouping structure including an atlas identifier, wherein a track with a particular atlas identifier is implicitly considered a master track having common data for other volumetric media tracks belonging to the group. Signal information may include a grouping structure including a volumetric media unit header, wherein the volumetric media unit header includes information about the atlas identifier and an indicator of a master track having common data for other volumetric media tracks belonging to the group. Signal information may include a track group, wherein the track group includes a volumetric media unit header for the volumetric media track. Signal information may provide at least one grouping type to specify a track grouping mechanism. The signal information may include: a volumetric media sample entry including common data for a certain type of constrained volumetric media track; a constrained sample entry including constrained scheme information; and a track reference in a volumetric media parameter track having a sample entry for connecting from the volumetric media parameter track to the constrained volumetric media track.

[0235] An exemplary non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform operations may be provided, the operations comprising: providing signal information to identify an atlas identifier on a sample of or on a volumetric media track in a multi-track container; wherein the signal information enables a file parser to link the volumetric media track with a different atlas identifier derived from a volumetric media elementary stream; and wherein the file parser is capable of reconstructing the volumetric media elementary stream based on the signal information and data encapsulated in the multi-track container.

[0236] Other aspects of the non-transitory program storage device may include the following. The signal information may include: a volumetric media sample entry for a volumetric media parameter track, including common data from a volumetric media track of a volumetric media elementary stream; and a track reference in a volumetric media parameter track reference table, which references the volumetric media track and is used to connect from the volumetric media parameter track to the volumetric media track. For each of the one or more network abstraction layer units, a volumetric media unit header atlas identifier may be mapped to an atlas identifier within the volumetric media sample structure.

[0237] An example apparatus includes means for providing signal information to identify an atlas identifier on a sample of a volumetric media track or on the volumetric media track in a multi-track container; wherein the signal information configures a file parser to link the volumetric media track with a different atlas identifier derived from a volumetric media elementary stream; and wherein the file parser is configured to reconstruct the volumetric media elementary stream based on the signal information and data encapsulated in the multi-track container.

[0238] An example apparatus includes at least one processor; and at least one non-transitory memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least perform: providing signal information to identify an atlas identifier on a sample of a volumetric media track or on the volumetric media track in a multi-track container; wherein the signal information configures a file parser to link the volumetric media track with a different atlas identifier derived from a volumetric media elementary stream; and wherein the file parser is configured to reconstruct the volumetric media elementary stream based on the signal information and data encapsulated in the multi-track container.

[0239] An example method includes providing signal information to identify an atlas identifier on a sample of a volumetric media track or on the volumetric media track in a multi-track container; wherein the signal information configures a file parser to link the volumetric media track with a different atlas identifier derived from a volumetric media elementary stream; and wherein the file parser is configured to reconstruct the volumetric media elementary stream based on the signal information and data encapsulated in the multi-track container.

[0240] An example non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform operations comprising: providing signal information to identify atlas identifiers on samples of or on a volumetric media track in a multi-track container; wherein the signal information configures a file parser to link the volumetric media track with different atlas identifiers derived from a volumetric media elementary stream; and wherein the file parser is configured to reconstruct the volumetric media elementary stream based on the signal information and data encapsulated in the multi-track container.

[0241] It should be understood that the foregoing description is illustrative only. Those skilled in the art may devise various alternatives and modifications. For example, the features recited in the various dependent claims may be combined with one another in any suitable combination. Furthermore, features from the different embodiments described above may be selectively combined to form new embodiments. Therefore, this description is intended to encompass all such alternatives, modifications, and variations that fall within the scope of the appended claims.

Claims

1. An apparatus for volumetric media elementary stream processing, comprising: means for providing signal information to identify an atlas identifier on a sample of or on a volumetric media track in a multi-track container; wherein the signal information allows a file parser to link volumetric media tracks with different atlas identifiers derived from a volumetric media elementary stream; and The file parser is capable of reconstructing the volumetric media elementary stream based on the signal information and the data encapsulated in the multi-track container.

2. The apparatus according to claim 1, wherein the signal information comprises: a volumetric media sample entry of a volumetric media parameter track comprising common data originating from the volumetric media tracks of said volumetric media elementary stream; as well as A track reference in the volumetric media parameter track reference table that references a volumetric media track and is used to connect from the volumetric media parameter track to the volumetric media track.

3. The apparatus of any one of claims 1 to 2, wherein the volumetric media unit header atlas identifier is mapped to an atlas identifier within a volumetric media sample structure for each of the one or more network abstraction layer units.

4. The apparatus of any one of claims 1 to 2, wherein a volumetric media unit header atlas identifier is mapped to an atlas identifier within a volumetric media sample structure for a set of network abstraction layer units. 5 . The apparatus according to claim 1 , wherein a volumetric media unit header atlas identifier is implicitly mapped to a network abstraction layer unit within a volumetric media sample structure.

6. The apparatus of claim 1 , wherein the volumetric media track comprises a subsample information box, and wherein the subsample information box comprises a volumetric media unit header box in a codec specific parameter field, the volumetric media unit header box linking a subsample network abstraction layer unit in a volumetric media sample to a volumetric media unit header atlas identifier.

7. An apparatus according to any one of claims 1 to 2, wherein the samples of the volumetric media track comprise a sequence of volumetric media units having a payload and a header, and the samples correspond to a single point cloud frame, wherein the samples within one or more component tracks corresponding to the single point cloud frame have a composition time corresponding to the composition time of the samples of the volumetric media track.

8. The apparatus according to any one of claims 1 to 2, wherein the signal information comprises: a grouping structure comprising information about the atlas identifier; as well as An indicator for signaling whether the volumetric media track is a master track having common data for other volumetric media tracks belonging to a group.

9. An apparatus according to any one of claims 1 to 2, wherein the signal information comprises a grouping structure including the atlas identifier, wherein a track with a specific atlas identifier is implicitly considered to be a master track having common data for other volumetric media tracks belonging to a group.

10. An apparatus according to any one of claims 1 to 2, wherein the signal information comprises a grouping structure, the grouping structure comprising a volume media unit header, wherein the volume media unit header comprises information about the atlas identifier and an indicator of a main track having common data for other volume media tracks belonging to a group.

11. The apparatus according to any one of claims 1 to 2, wherein the signal information comprises a track group including a volume media unit header for the volume media track.

12. The apparatus according to any one of claims 1 to 2, wherein the signal information provides at least one grouping type to specify a track grouping mechanism.

13. The apparatus according to any one of claims 1 to 2, wherein the signal information comprises: A volumetric media sample entry includes data common to a certain type of constrained volumetric media track; restricted sample entries, including restricted protocol information; as well as A track reference in the volumetric media parameter track has sample entries for connecting from the volumetric media parameter track to the constrained volumetric media track.

14. The apparatus according to any one of claims 1 to 2, wherein the volumetric media elementary stream is a V-PCC elementary stream.

15. The apparatus according to any one of claims 1 to 2, wherein the volumetric media elementary stream is a V3C elementary stream.

16. The device of claim 2, wherein the volumetric media parameter track is a V-PCC track.

17. A method for volumetric media elementary stream processing, comprising: providing signal information to identify an atlas identifier on a sample of or on a volumetric media track in a multi-track container; wherein the signal information allows a file parser to link volumetric media tracks with different atlas identifiers derived from a volumetric media elementary stream; and The file parser is capable of reconstructing the volumetric media elementary stream based on the signal information and the data encapsulated in the multi-track container.

18. A non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform operations comprising: providing signal information to identify an atlas identifier on a sample of or on a volumetric media track in a multi-track container; wherein the signal information allows a file parser to link volumetric media tracks with different atlas identifiers derived from a volumetric media elementary stream; and The file parser is capable of reconstructing the volumetric media elementary stream based on the signal information and the data encapsulated in the multi-track container.

Images