Method, apparatus, and computer program for improving multi-track encapsulation of point cloud data

Abstract

According to some embodiments of the present disclosure, a method is provided for encapsulating a bitstream in a media file including different tracks, the bitstream including point cloud data, the point cloud data including slice-based point cloud frames, and slices of the point cloud frames including different types of data units. After obtaining a first data unit of a first slice of one point cloud frame and a second data unit of a second slice of the point cloud frame, each of the obtained first and second data units is encapsulated in a track of the media file according to the type of the data unit. At least one information item characterizing a relative position in the track of the media file of the first data unit with respect to the second data unit is obtained and encapsulated in the media file.

Description

[Technical Field]

[0001] Field of Disclosure The present disclosure relates to encapsulating data, particularly point cloud data, in a standard and interoperable format, for example, for storing or transmitting slice-based point cloud frames of 3D points as a set of tracks. [Background technology]

[0002] Disclosure process The Moving Picture Experts Group (MPEG) has standardized the compression and storage of point cloud data (also known as volumetric media data), which consists of a set of 3D points with associated attribute information such as color, reflectance, and frame index.

[0003] First, MPEG-I Part-9 (ISO / IEC 23090-9) specifies geometry-based point cloud compression (G-PCC) and specifies a bitstream syntax for point cloud information. According to MPEG-I Part-9, a point cloud is an unordered list of points that includes geometry information, optional attributes, and associated metadata. Geometry information describes the location of the points in a three-dimensional Cartesian coordinate system. Attributes are typed properties of each point, such as color or reflectance. Metadata are items of information used to interpret the geometry information and attributes. According to MPEG-I Part-9, the G-PCC compression specification (MPEG-I Part-9) defines specific attributes, such as a frame index attribute or a frame number attribute, with reserved attribute label values ​​(3 for frame index and 4 for frame number attribute), which recall that a point cloud frame is a set of points at a specific time instance. A point cloud frame can be partitioned into one or more ordered subframes, tiles, or slices. A subframe is a partial representation of a point cloud frame consisting of points with the same frame number or frame index attribute value. For example, a sub-frame may be a set of points with attributes within a point cloud frame that share a common acquisition, capture, or rendering time. As another example, a sub-frame may be a set of points with those attributes within a point cloud frame that are acquired or captured consecutively during a given time range or that are to be rendered in a given time range. As yet another example, a sub-frame may be a set of points with those attributes within a point cloud frame that are acquired according to a laser shot direction or that correspond to a portion of a 3D sensor scan path. In MPEG-I Part-9, point cloud frames are also indicated by the FrameCtr variable using a parameter (frame_ctr_lsb syntax element) in a frame boundary marker data unit or some data unit headers.

[0004] Recall that a tile is a set of slices, identified by a common slice_tag syntax element value, whose geometry is contained within a bounding box that may be specified in a tile inventory data unit. Each tile consists of a single bounding box and an identifier (tileId). Tile information is not used in the decoding process specified in ISO / IEC 23090-9. A slice corresponds to the geometry and attributes of a portion of an encoded point cloud frame or of the entire encoded point cloud frame. Each slice should contain at least one geometry data unit (GDU) that encodes slice geometry and attribute data units (ADUs), or a default attribute data unit (DU) that encodes slice attributes. A slice is identified by a GDU slice_id. ISO / EC 23090-9 specifies the slice decoding process as a four-step process: - Decoding points within a slice, -Decoding default attributes, if any - Decoding point attributes, if any, and - Point position offset and scaling.

[0005] Second, MPEG-I Part-18 (ISO / IEC 23090-18) specifies a media format that enables the storage and distribution of geometry-based point cloud compressed data. It also supports flexible extraction of geometry-based point cloud compressed data during distribution and / or decoding. According to MPEG-I Part-18, a point cloud frame is encapsulated within one or more G-PCC tracks, and samples are contained within a G-PCC track corresponding to a single point cloud frame. Each sample contains one or more G-PCC units belonging to the same presentation time. A G-PCC unit is a type-length-value (TLV) encapsulation structure that contains at least one of the following: a sequence parameter set (SPS), a geometry parameter set (GPS), an attribute parameter set (APS), a tile inventory, a frame boundary marker, a geometry data unit (GDU), an attribute data unit (ADU), a default attribute data unit, a frame-specific attribute property (FSAP) data unit, and a user data unit. The syntax of the TLV encapsulation structure is defined in Annex B of ISO / IEC 23090-9. Summary of the Invention [Problem to be solved by the invention]

[0006] While the ISO Base Media File Format has proven efficient for encapsulating point cloud data, there is a need to improve the encapsulation efficiency, particularly to improve multi-track encapsulation of slice-based point cloud frames of 3D points (i.e., point cloud frames comprising at least one slice). [Means for solving the problem]

[0007] Disclosure Overview The present disclosure is designed to address one or more of the aforementioned concerns. In this regard, a solution is provided for improving the encapsulation of point cloud data.

[0008] According to a first aspect of the present disclosure, there is provided a method for encapsulating a bitstream into a media file comprising different tracks, the bitstream comprising point cloud data, the point cloud data comprising slice-based point cloud frames, the slices of the point cloud frames comprising different types of data units, the method comprising: Obtain a first data unit of a first slice of one point cloud frame; for each of the obtained first data units, encapsulating the first data unit within a track of the media file, the track being selected as a function of a type of the first data unit; obtaining a second data unit of a second slice of the point cloud frame; for each of the obtained second data units, encapsulating the second data unit within a track of the media file, the track being selected as a function of a type of the second data unit; obtaining at least one information item characterizing the relative order within the bitstream of the first data unit with respect to the second data unit; encapsulating said retrieved at least one information item within said media file; A method is provided, comprising:

[0009] Thus, the disclosed method allows for describing the encapsulated data units of a bitstream comprising a slice-based point cloud data frame, allowing a parser to generate a bitstream with properly ordered data units.

[0010] According to some embodiments, at least one of the obtained at least one information item is encapsulated as a sample group within the geometry track of the media file.

[0011] Furthermore, according to some embodiments, at least one of the retrieved at least one information item is encapsulated in each of the different tracks of the media file.

[0012] Furthermore, according to some embodiments, at least one information item encapsulated in each of the different tracks of the media file is a slice separator.

[0013] Furthermore, according to some embodiments, at least one information item encapsulated in each of the different tracks of the media file comprises a description of the structure of data units within a sample, each data unit being associated with slice information.

[0014] Furthermore, according to some embodiments, at least one item of information encapsulated in each of said different tracks of said media file comprises a description of the structure of a slice within a point cloud frame.

[0015] Furthermore, according to some embodiments, the media file complies with the ISOBMF format, the different tracks comprise a geometry track and at least one attribute track, and the data units comprise geometry data units and attribute data units.

[0016] According to a second aspect of the present disclosure, there is provided a method of analyzing a media file including encapsulated point cloud data, wherein the point cloud data includes slice-based point cloud frames, the slices of the point cloud frames include data units of different types, and the data units are encapsulated in the media file in different tracks according to their types, the method comprising: Retrieving data units from the track on a slice basis; concatenating the acquired data units on a slice basis such that all the acquired data units related to a given slice of a given track are contiguous with all the acquired data units related to a slice corresponding to the given slice in the other of the different tracks of the given track; generating a bitstream comprising the concatenated data units; A method is provided, comprising:

[0017] Thus, the disclosed method enables parsing slice-based point cloud frames encapsulated in a multi-track media file to generate a bitstream in which data units are properly ordered, where data units belonging to a slice in a track and data units belonging to a corresponding slice in another track belong to the same slice in the bitstream.

[0018] According to some embodiments, concatenating the obtained data units comprises: obtaining a first set of at least one first data unit from a first track of different tracks, wherein the data units of the first set belong to a first slice; obtaining a second set of at least one second data unit from a second track, the second set being different from the first track, the second data unit belonging to a second slice, the second slice corresponding to the first slice; The data units of the first set and the data units of the second set are concatenated so that the data units of the first set and the data units of the second set are contiguous.

[0019] According to some further embodiments, the method further comprises obtaining a third set of at least one third data unit from the first track, wherein the data units of the third set belong to a third slice, the third slice being different from the first slice and following the first slice, the data units of the first set and the data units of the third set belong to the same sample, and the data units of the third set are concatenated after the data units of the first set and the data units of the second set in the generated bitstream.

[0020] Further, according to some embodiments, obtaining the first set of at least one first data unit comprises: identifying a first data unit of a first set; comparing the information item of the identified first data unit with the information item of the data unit following the identified first data unit in the first track; and determining, as a result of the comparison, whether the data unit following the identified first data unit belongs to the first set.

[0021] Further, according to some embodiments, obtaining the first set of at least one first data unit comprises: Identifying a data unit in the first track; Obtain a slice identifier associated with the identified data unit; The method includes comparing the obtained slice identifier with a current slice identifier, and determining whether the identified data unit belongs to the first set as a result of the comparison.

[0022] Further, according to some embodiments, obtaining the first set of at least one first data unit comprises: A slice encapsulation description (data unit per slice description) is obtained from the media file, and at least one first data unit is obtained from the first track as a function of the obtained slice encapsulation description (data unit per slice description).

[0023] Further, according to some embodiments, a slice encapsulation description (a data unit per slice description) is obtained as a function of a reference to a sample comprising a plurality of data units, and the data units of the first set are obtained from the plurality of data units.

[0024] Further, according to some embodiments, obtaining the first set of at least one first data unit comprises: identifying a first data unit of the first set in the first track; obtaining a type of a data unit following the identified first data unit in the first track; If the obtained type corresponds to a slice separator, an information item is provided indicating that all the first data units of the first set have been obtained.

[0025] Furthermore, according to some embodiments, the media file complies with the ISOBMF format, the different tracks comprise a geometry track and at least one attribute track, and the data units comprise geometry data units and attribute data units.

[0026] Additionally, according to some embodiments, the data units per slice description provides the number of slices and, for each slice, provides a list of the number of data units per track associated with the slice.

[0027] Furthermore, according to some embodiments, the list has a length equal to the number of track references from the geometry track to the attribute track containing the sample group plus one.

[0028] Furthermore, according to some embodiments, the list is ordered starting with the number of the data units for the geometry track.

[0029] According to a third aspect of the present disclosure, there is provided a method for encapsulating a bitstream into a media file comprising different tracks, the bitstream comprising point cloud data, the point cloud data comprising slice-based point cloud frames, the slices of the point cloud frames comprising different types of data units, the method comprising: Obtain a first data unit of a first slice of one point cloud frame; for each of the obtained first data units, encapsulating the first data unit within a track of a media file, the track being selected as a function of a type of the first data unit; Obtain a second data unit of a second slice of the point cloud frame; for each of the obtained second data units, encapsulating the second data unit within a track of the media file, the track being selected as a function of a type of the second data unit; obtaining at least one information item characterizing the relative order within the bitstream of the first data unit with respect to the second data unit; encapsulating the retrieved at least one information item in a media file; A method is provided, comprising:

[0030] Thus, the disclosed method allows for describing the encapsulated data units of a bitstream comprising a slice-based point cloud data frame, allowing a parser to generate a bitstream with properly ordered data units.

[0031] According to some embodiments, at least one of the obtained at least one information item is encapsulated as a sample group within the geometry track of the media file.

[0032] Furthermore, according to some embodiments, the sample group provides a number of slices and, for each slice, a list of the number of data units per track associated with said slice.

[0033] Furthermore, according to some embodiments, the list has a length equal to the number of track references from the geometry track to the attribute track containing the sample group plus one.

[0034] Furthermore, according to some embodiments, the list is ordered starting with the number of data units for the geometry track.

[0035] Furthermore, according to some embodiments, the media file complies with the ISOBMF format, the different tracks comprise a geometry track and at least one attribute track, and the data units comprise geometry data units and attribute data units.

[0036] According to another aspect of the present disclosure, there is provided a device comprising a processing unit configured to perform the steps of the above-described method.

[0037] This aspect of the disclosure has similar advantages to those discussed above.

[0038] At least a portion of the methods according to the present disclosure may be computer-implemented. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be generally referred to herein as a "circuit," "module," or "system." Furthermore, the present disclosure may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium.

[0039] Since the solutions of the present disclosure can be implemented in software, the solutions of the present disclosure can be embodied as computer-readable code for provision to a programmable apparatus on any suitable carrier medium. The tangible carrier medium may comprise a storage medium such as a floppy disk, a CD-ROM, a hard disk drive, a magnetic tape device, or a solid-state memory device. The ephemeral carrier medium may be an electric signal, an electronic signal, an optical signal, an acoustic signal, a magnetic signal, or an electromagnetic signal, such as a microwave or RE signal. [Brief explanation of the drawings]

[0040] Some embodiments of the present disclosure will now be described, by way of example only, with reference to the following drawings: [Figure 1] FIG. 1 shows an example of a system in which the present invention can be implemented. [Figure 2] FIG. 2 shows an example of encapsulating bitstreams in several tracks of a media file and parsing the latter to generate bitstreams that comply with a given format, such as MPEG-I Part-9. [Figure 3] FIG. 3 shows a first example of the structure of frames of a sequence of frames of an encapsulated and parsed G-PCC bitstream, where all slices of a point cloud frame have the same structure. [Figure 4] FIG. 4 shows a second example of the frame structure of a sequence of frames of an encapsulated and parsed G-PCC bitstream, where not all slices of a point cloud frame have the same structure. [Figure 5] FIG. 5 illustrates example steps in an encapsulation process according to some embodiments of the present invention, allowing a parser to encapsulate point cloud data into multiple tracks in an interoperable manner to generate a compliant G-PCC bitstream. [Figure 6] FIG. 6 illustrates example steps in a parsing process according to some embodiments of the present invention. [Figure 7] FIG. 7 illustrates an example of adding static or semi-static instructions to a multi-track media file encapsulating slice-based point cloud frames to assist a parser in reconstructing a bitstream that conforms to a predetermined format, for example, to reconstruct a G-PCC bitstream. [Figure 8] FIG. 8 illustrates an example of adding dynamic instructions to a multi-track media file encapsulating slice-based point cloud frames to assist a parser in reconstructing a bitstream that conforms to a predetermined format, for example, to reconstruct a G-PCC bitstream. [Figure 9] Figure 9 shows an example of adding slice separators to a multi-track media file encapsulating slice-based point cloud frames to assist the parser in reconstructing a bitstream that conforms to a predetermined format, e.g., reconstructing a G-PCC bitstream. [Figure 10] FIG. 10 implements at least one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0041] Detailed Description of Disclosure The inventors have observed that the data units of a slice-based point cloud frame in a bitstream should be encapsulated according to certain constraints or with additional metadata that enables a parser to generate a usable bitstream that conforms to a given format, i.e., a bitstream with properly ordered data units.

[0042] According to some embodiments of the present disclosure, data units of slice-based point cloud frames of a bitstream are encapsulated within a multi-track media track such that a parser can parse the encapsulated data units to provide a bitstream that conforms to a predetermined format, e.g., a bitstream that conforms to MPEG-I Part-9.

[0043] Furthermore, according to some embodiments of the present disclosure, additional metadata is provided in tracks of the media file, e.g., G-PCC tracks, to enable distinguishing data units of a given slice from data units of another slice. Such additional metadata can include slice end indications in the tracks, or descriptions of data unit organization within the slice, whether such organization is static or dynamic.

[0044] 1 shows an example of a system in which the present invention can be implemented. More precisely, the present invention can be used in a media file writer, such as media file writer 100, or a media player, such as media player 130, or both.

[0045] As shown, media file writer 100 takes as input point cloud data (or volume data), such as point cloud data 150. Point cloud data 150 may be acquired from a 3D sensor, as described with reference to FIG. 2. The point cloud data may be received as uncompressed raw data or as a compressed bitstream, for example, a compressed bitstream conforming to the MPEG-I Part-9 standard. Media file writer 100 includes an encapsulation module 102.

[0046] The media file writer 100 can be connected via a network interface (not shown) to a communications network 120 which can also be connected via a network interface (not shown) to a media player (or reader) 130 equipped with a decapsulation module 132.

[0047] Media file writer 100 may be used to provide media files using protocols for dynamic adaptive streaming over HTTP, such as Dynamic Adaptive Streaming over HTTP (DASH) or HTTP Live Streaming. These protocols require a streaming manifest, such as a media presentation description (MPD) or a playlist. When used to stream encapsulated media content, media file writer 100 may include a manifest generation module, such as manifest generation module 104. Media file writer 100 may also include a compression module, such as compression module 106, for compressing input point cloud data into a compressed bitstream, using a point cloud compression algorithm such as that described in MPEG-I Part-9.

[0048] For interoperability purposes, encapsulation module 102 can encapsulate the received point cloud data according to an ISOBMFF-based format, such as MPEG-I Part-18, to generate a media file, such as media file 152, which can be stored for later use by a player or by an image analysis tool, or transmitted to a media player or streaming client. The encapsulation process performed in encapsulation module 102 is further described with reference to FIG. 5.

[0049] Media file writer 100 may be controlled and parameterized by a user, e.g., via a graphical user interface, or by an application, e.g., by application code or script. To process compressed point cloud data, e.g., to process a bitstream of compressed point cloud data that conforms to MPEG-I Part-9, encapsulation module 102 may include a g-PCC unit parser that can read the header of a g-PCC unit and, e.g., determine the length (e.g., in bytes, as in the tlv_num_payload_bytes syntax element) of the data corresponding to the unit or unit type (e.g., the tlv_type syntax element). The G-PCC unit parser may also be capable of parsing header information for some G-PCC units, e.g., attribute headers (to obtain their types), and may also be capable of parsing parameter sets to obtain general information about the bitstream. To process uncompressed point cloud data, e.g., data acquired directly from a 3D sensor, the encapsulation module 102 can include a point cloud data parser that can read multiple point locations and their attributes directly from the captured raw data (e.g., a ply or pcd file parser). The media writer may be a recording device, a multi-sensor camera device, onboard a vehicle in which the 3D sensor is mounted, or may be part of a software tool in a studio where the volume data is acquired.

[0050] The media file 152 may consist of a single media file or a set of media segment files, for example, as an ISOBMFF segment (an ISO base media file containing one or more segments). The media file may also be a fragmented file, for example, for live acquisition or capture and encapsulation or live (or low-latency) streaming. It may conform to the ISOBMFF standard or a standard specification derived from ISOBMFF (e.g., MPEG-I Part-18).

[0051] The media player (or reader) 130 may be a streaming client, with multiple streaming functions handled by a streaming module, such as streaming module 134 implementing a DASH or HLS client for requesting media files, such as media file 152, and adapting transmission parameters. The media player may also implement an RTP or RTC web client, particularly for live transmissions that allow transmission loss experimentation. The media player 130 may also include a decompression module 136 that takes as input a bitstream representing compressed point cloud data, e.g., an MPEG-I Part-9 compliant bitstream, and generates the point cloud data (or data) for rendering or analysis. The media file 152 may be read from a storage location or streamed using the streaming module 134. The data may be read all at once or in chunks, segments, or fragments and provided to the decapsulation module (or analysis module) 132.

[0052] The decapsulation module (or analysis module) 132 then extracts the encapsulated point cloud data, or a subset of the encapsulated point cloud data, depending on the player configuration, a selection from the user, or parameters of the application using the media player. The extracted point cloud data can result in a bitstream, such as an MPEG-I Part-9 compliant bitstream. In such a case, the bitstream is provided to a decompression module (external or internal, e.g., internal decompression module 136) for reconstruction of point cloud data 154 for use by a user or application (e.g., visualization or analysis). The analysis process is further described with reference to FIG. 6. The media player can be installed in a display device (e.g., a smartphone, tablet, PC, vehicle with a multimedia screen, etc., or a software tool in a studio for volumetric data generation).

[0053] Figure 2 shows an example of encapsulating a bitstream in several tracks of a media file and parsing the latter to generate a bitstream conforming to a given format, such as MPEG-I Part-9. The encapsulated bitstream may be a G-PCC bitstream, and the encapsulated media file may be an ISOBMFF media file, e.g., the 'gpc1' and / or 'gpcg' tracks of ISO / IEC 23090-18. Note that the media data of the encapsulated bitstream may be tiled media data, possibly encapsulated as 'gpcb' tile base tracks referencing the G-PCC tile track 'gpt1', each carrying one G-PCC component (geometry or attribute).

[0054] As shown, G-PCC bitstream 200 comprises a sequence of multiple point cloud frames, specifically frame N referenced at 205. An encoded point cloud frame here comprises a sequence of zero or more slices that share the same FrameCtr value (a kind of timestamp). For example, point cloud frame N comprises two slices referenced at 210-1 and 210-2. Each slice comprises one geometry data unit (possibly followed by optional duplicate geometry data units) that encodes geometry, and may comprise one or more attribute data units or one or more default attribute data units (DUs) that encode slice attributes. The number of data units may vary from frame to frame (e.g., different number of slices, new parameter sets, FSAP data units, etc.). For illustrative purposes, slice 210-1 comprises one geometry data unit 215-1 and two attribute data units 220-11 and 220-12. Similarly, slice 210-2 comprises one geometry data unit 215-2 and two attribute data units 220-21 and 220-22.

[0055] Slices are identified by the slice_id in the GDU. Slices may be repeated within a coded point cloud frame, but those repetitions should not change the value of slice_id. Slice division may be used to enable parallelization, to improve coding efficiency, and / or to enable other functionality such as error resilience and progressive decoding.

[0056] During encapsulation of a G-PCC bitstream 200 within a media file 230, all items of geometry information of the G-PCC bitstream may be stored and described in a geometry track, and each attribute of the G-PCC bitstream may be stored and described in a corresponding attribute track. For example, geometry data units may be stored and described in geometry track 235-1, attribute data units of a first attribute may be stored and described in attribute track 235-2, and attribute data units of a second attribute may be stored and described in attribute track 235-3. It is observed that tracks belonging to the same G-PCC sequence or bitstream are preferably time-aligned (i.e., each track contains samples, each of which contains a sample time (e.g., in an 'stts' or 'ctts' box) for storing a time value, and there is one sample in each track associated with a given time value. For example, samples 245-1, 245-2, and 245-3 are time-aligned samples. Thus, each point cloud frame, e.g., point cloud frame 205, can be described as a set of samples in the ISOBMFF, such as sample set 240, with one sample in each track, and each sample in the same track represents the same type of information or the same component (geometry information or specific attribute information). For example, sample 245-1 in track 235-1 comprises geometry information, sample 245-2 in track 235-2 comprises attribute information of a first attribute, and sample 245-3 in track 235-3 comprises attribute information of a second attribute.

[0057] According to the multi-track encapsulation specification ISO / IEC 23090-18, a media file ultimately consists of samples in different tracks. Each sample in a given track contains data units of a given type from different slices in the original bitstream 200. For example, slices 210-1 and 210-2 have their geometry data units and geometry-related information (e.g., GPS or tile inventory) in track 235-1 (i.e., GDUs 215-1 and 215-2), and the same slices have their attribute data units and attribute-related information (e.g., APS or frame-specific attribute data units) in attribute tracks 235-2 (i.e., ADUs 220-11 and 220-21) and 235-3 (i.e., ADUs 220-12 and 220-22). Note that the representation of media file 230 is a simplified view of ISOBMFF. It should be understood that track and sample descriptions are available under the 'moov' box and 'trak' box indexing data (GDU or ADU). Similarly, data is understood to be within media data boxes (e.g., 'mdat', 'imda', or 'idat' boxes). If the media file is a fragmented ISO-based media file, the track fragment and sample descriptions of this fragment store the sample data, with their associated 'mdat' or 'imda' boxes within the 'moof' and 'traf' boxes. It should also be noted that each sample comprises data units from different slices. The inventors have observed that there are boundaries within each sample, which may be implicit or signaled, that allow for distinguishing data units from one slice from another, as indicated by dotted line 250.

[0058] Here, it is observed that the bitstream reconstructed by the parser should satisfy several rules to ensure that the decoder is able to decode and / or render the point cloud data. Examples of such rules related to TLV unit order, TLV constraints, and / or TLV slice constraints are as follows: - each slice should contain a single GDU, followed by any duplicate GDUs, followed by any ADUs or default attribute DUs; - DUs belonging to different slices should not be interleaved - Every slice should have a corresponding ADU or default attribute data unit for every attribute listed in the SPS.

[0059] To generate a valid G-PCC bitstream from a media file 230 and according to specifications ISO / IEC 23090-18, e.g., G-PCC bitstream 255, some mechanism should be used to concatenate data units stored in different tracks of the media file in the correct order, following the rule from ISO / IEC 23090-9 that data units belonging to different slices should not be interleaved. To ensure that any parser, independent of its implementation, generates a compliant bitstream for a G-PCC decoder, at least from the slice ordering point of view, one of the following mechanisms can be used: - A restructuring mechanism that implements restructuring rules that take slices (parser side) into account - an encapsulation mechanism that implements the constraints of multi-track encapsulation (used on the writer side and parser side); a signaling mechanism that provides instructions to be used during the reassembly process (implemented on the writer side and used on the parser side); these instructions are ·static, ·dynamic, Hybrid (static and redefined for some specific / exceptional samples or some fragments) Interactive within the geometry track Spread within each component track (tiled or untiled).

[0060] The mechanism used may depend on the configuration of the G-PCC bitstream considered for encapsulation and parsing, for example, whether all slices of a point cloud frame have the same structure. For example, when the slice structure is stable (e.g., it comprises the same number of slices across multiple samples and the same number of TLVs per slice), it may not be necessary to provide instructions for samples or sample-based groups, but rather for all samples of a track or track fragment, i.e., by using a static approach. The mechanism used may also depend on the ISO / IEC 23090-18 specification level: it may or may not constrain the writer regarding slice ordering in each component track. Depending on the level of specification, the parser may or may not have some a priori discretion. If this cannot be assumed a priori, the parser may then use additional instructions from the media file.

[0061] FIG. 3 shows a first example of the structure of frames of a sequence of frames of a G-PCC bitstream to be encapsulated and parsed, where all slices of a point cloud frame have the same structure.

[0062] Here, we assume that the SPS of a point cloud sequence declares two attributes (as in Figure 2). According to the example in Figure 3, a single structure of data units within a slice is used for all slices of the frame of the considered G-PCC bitstream. In other words, there is a fixed TLV pattern per slice.

[0063] As shown in FIG. 3, each slice 305-1, 305-2, 305-3, and 305-4 of frame 300 comprises one geometry data unit (GDU) followed by two attribute data units (ADUs) or default attribute data units, also referred to as Def ADUs (for clarity, only one slice is shown for frame 300, but multiple slices may exist, here assumed to have the same TLV pattern: one GDU followed by two ADUs or default ADUs).

[0064] While the structure shown in FIG. 3 may be advantageously used to encapsulate and / or parse a G-PCC bitstream, the inventors have observed that it may be advantageous for such a structure to vary from one slice to another to accommodate data loss (at the writer or parser) or to account for when partial reconstruction occurs at the receiver. For example, while an encapsulated and parsed G-PCC bitstream may contain multiple slices with the same structure, it may be considered that an encapsulated and parsed G-PCC bitstream does not contain slices with the same structure, so that parsing can be performed properly even when data units are lost. This may be the case, for example, when the transmission is unreliable, such as when using RTP-based delivery for live or real-time transmission.

[0065] Figure 4 shows a second example of the structure of a frame in a sequence of frames of a G-PCC bitstream to be encapsulated and parsed, where not all slices in a point cloud frame have the same structure. Again, assume that the SPS for the point cloud sequence declares two attributes (and for simplicity, only one slice for frame 400 is shown, but there can be multiple slices for a frame of the point cloud sequence). However, according to this example, a slice can contain a number of attributes or default attribute data units that differ from the expected number of attributes or default attribute data units. For illustrative purposes, slice 405-1 of frame 400 comprises one geometry data unit and one attribute data unit, while slice 405-3 comprises one geometry data unit and two attribute data units. In fact, ISO / IEC JTC1 / SC29 / WG11 MPEG m53681 proposes signaling default attribute data units by adding default attribute values ​​to the SPS attribute description to handle the omission of coded attribute data, avoiding the need to encode (or decode) attribute data where all attribute values ​​are identical. Slice 405-1 uses default SPS attribute values ​​for the second attribute, while slice 405-5 uses another concept, the frame-specific attribute property data unit (FSAP). When an FSAP data unit contains an attribute property with a type equal to 4, it means that the FSAP data unit defines a default value for a given attribute that applies to a particular frame. Therefore, the attribute data unit for the second attribute of slice 405-5 may be omitted. For other frames, the default value or the actual coded value may apply.

[0066] Note that losses may occur before encapsulation or during transmission. When the bitstream is damaged, an advanced writer can analyze the bitstream and detect missing data units. When the writer can detect missing attribute data units, the writing device can take the lead in padding the bitstream with default attribute data units. Similarly, on the receiver or client side, when the parser detects that some attribute data units are missing, the parser can take the lead in padding the default attribute data units to generate a compliant bitstream having the number of attributes declared in the SPS. This last point also applies to partial extraction. When the bitstream is encapsulated into multiple tracks, for example, one geometric track and N attribute tracks, if the parser extracts only a subset of the attribute tracks having the geometric track, for example, M tracks where M < N, in order to avoid rewriting the SPS and declaring the number of attributes (declaring N attributes), N - M default attribute data units can be embedded. It can embed the attributes in any order, provided that it does not interleave data units from different slices.

[0067] Encapsulation Process FIG. 5 shows an example of steps of an encapsulation process according to some embodiments of the present invention, enabling a parser to encapsulate point cloud data into multiple tracks in an interoperable manner to generate a compliant G-PCC bitstream. For illustration purposes, the steps shown in FIG. 5 may be executed by the encapsulation module 102 of FIG. 1.

[0068] According to the illustrated example, the first step (step 500) is directed to initializing the reception of point cloud data (or volume data) for encapsulation. Next, the encapsulation module is configured (step 505) according to the type of data to be processed (e.g., depending on whether the point cloud data is compressed, whether the point cloud data is to be analyzed, whether the point cloud data includes tiles or slices). This can be done by a user or an application. For illustrative purposes, configuration can include selecting single-track or multi-track encapsulation, selecting live or offline encapsulation, and selecting a description mode, e.g., describing tiles or not. Within the scope of this disclosure, multi-track encapsulation is configured with or without tile tracks.

[0069] The configuration step 505 may also be used to indicate whether the same configuration parameters should be applied to encapsulate the entire point cloud data (static configuration) or whether they may change when encapsulating the point cloud data (dynamic configuration). If the media file writer also includes a compression module, such as compression module 106 of FIG. 1, the configuration step may include setting parameters for the encoder, such as setting the slice or tile structure. The compression module may encode these configuration parameters, for example, as additional syntax elements of the G-PCC bitstream, such as in a sequence parameter set (SPS) or in dedicated TLV types, such as in a tile inventory data unit for the description of bounding boxes for tiles. For example, the SPS may provide information about slice dependencies for bitstream parsing and decoding (such as the entropy_continuation parameter). When point cloud data is received by the encapsulation module as a G-PCC bitstream (e.g., a bitstream generated by compression module 106 of FIG. 1 or a bitstream generated by an external compression module), the configuration of the encapsulation module may use information from the parameter set of the bitstream, or supplemental information (sometimes called Supplemental Enhancement Information messages (SEI)) associated with the bitstream, or information provided in a user data unit. Supplemental information refers to coding parameters that are not required to decode the point cloud data but can assist applications that use these point cloud data by providing additional information.

[0070] In addition to the configuration of the encapsulation module, metadata structures for the media file, such as top-level boxes (e.g., 'ftyp' or 'styp', 'moov' and its 'trak' sub-box, 'mdat' box, and boxes for sample descriptions such as 'stbl' or 'stsd'), are created during an initialization step (step 510). Such an initialization step may include reading parameter sets (e.g., sequence parameter set, geometry parameter set, and / or attribute parameter set) from the encoded bitstream of the point cloud data, or obtaining information about the sensor used to acquire the point cloud data, such as some points, the type of attributes associated with the points (e.g., color, reflectance, timestamp, region of interest, etc.), or obtaining information about encoder settings. Note that some of the configuration parameters defined in the configuration step 505 may be reflected in the track description or sample description. Similarly, user data or supplemental information, if available, may be included in whole or in part in the media file. Portions of the configuration information (configuration parameters, parameter sets, user data, or supplementary information) may be collected in metadata structures, for example, sample entries (e.g., 'gpc1', 'gpcg', 'gpeb', or any 4CC for a G-PCC component track) or structures dedicated to sample descriptions, such as configuration boxes (e.g., 'gpcC'). The configuration information may include information about the slice configuration in the bitstream, i.e., the presence or absence of slices, optionally the number of slices per frame, whether this number is constant or not, whether slices are repeated or not, the presence or absence of slice sets, and whether there are slice reordering constraints. Having this information may impose fewer constraints on the parser (e.g., if slice_reordering_constraint=0, there is no need to check the slice order in the reconstructed bitstream).Some of this information may be obtained from the parameter set (e.g., slice reordering constraints), and some may require bitstream analysis (such as the number of slices and the number of slices per frame). Similarly, entropy_continuation_enabled from the SPS may be useful to reflect in the sample description slices, as it may also constrain the reconstruction process in terms of inter-slice ordering. This may be determined by the encapsulation module during initialization step 510, or it may be calculated all along the encapsulation and written to a file at the end of the encapsulation (this latter mode is less relevant for live packaging during live or low-latency streaming).

[0071] An example metadata structure for providing slice configuration information can be defined as a new box identified by the four-character code 'scfg', e.g. inheriting from FullBox, e.g. SliceConfigurationInformationBox, as follows: JPEG0007825066000001.jpg86101 During the ceremony slice_present indicates whether the sample's data contains slices. When set to 1, this parameter indicates that the sample may contain slices. When set to 0, this parameter indicates that the sample does not contain slices. slice_reordering_constraint has the same semantics as ISO / IEC 23090-9, entropy_continuation_enabled has the same semantics as ISO / IEC 23090-9, repeated_slices indicates whether slices can be repeated within some samples.

[0072] When set to 1, this parameter indicates that the sample may contain repeated slices. When set to 0, this parameter indicates that the sample does not contain repeated slices. same_num_slices_per_frame indicates whether all samples have the same number of slices. When set to 1, it indicates that the samples have the same number of slices. When set to 0, this parameter indicates that at least two samples do not have the same number of slices. num_slices_per_frame indicates the number of slices per sample (if it is a constant or not present), max_num_slices_per_frame indicates the maximum number of slices for a sample in a track.

[0073] This box may be stored in any G-PCC track as any box in the sample entry. Alternatively, the parameters present in the slice configuration information may be stored in the GPCCDecoderConfigurationRecord in a 'gpcC' box.

[0074] Following initialization step 510, the writer may optionally decide whether indication of reconstruction rules is required, and if so, whether they should be static (or implicit, i.e., apply to all samples) or dynamic (or explicit, i.e., one rule per sample, per group of samples, or per fragment or segment, as the case may be) (step 515). By default, or at least for live packaging, explicit reconstruction rules may be considered, as they allow the writer to handle the majority of configurations, such as those described with reference to Figures 3 and 4.

[0075] The following steps describe encapsulation with explicit reconstruction rules or instructions that are effective for reconstructing a valid bitstream from multiple tracks.

[0076] The encapsulation process then enters a processing loop to process each frame of point cloud data.

[0077] After reading a point cloud frame (step 520), e.g., after reading data unit by data unit from the compressed point cloud data bitstream, the encapsulation module determines whether the read data corresponds to geometry data (test 525). If the read data is determined to be geometry data, the read data unit is stored in the geometry track (step 530). In the case of explicit reconstruction rules, the slice_id and optional slice_tag from the GDU header are decoded and stored. The size of the read data unit may also be stored along with the slice_id and optional prev_slice_id (an identifier of a slice on which the current slice may depend). When the SPS indicates that entropy_continuation is active, the writer may also decode the value of the slice_entropy_continuation parameter, and if set to 1, may decode the value of the prev_slice_id parameter and store this latter value (to ensure correct inter-slice ordering).

[0078] When a read geometry data unit is detected as a duplicate GDU (e.g., by calculating the difference of multiple data units or by checking whether they have the same slice_id), depending on the encapsulation mode, the writer can either store this duplicate GDU during step 530 or discard it and store only one instance of the duplicate GDU. This encapsulation choice (presence or absence of duplicated GDUs) can be indicated as a track property or in the sample description of the geometry track (e.g., sample entry or configuration box, e.g., GPCCConfigurationBox or GPCCComponentInfoBox).

[0079] Determining the type of the read data unit may comprise determining the type of the G-PCC unit (or TLV unit) by reading its type using a G-PCC unit parser (not shown).

[0080] Returning to test 525, if it is determined that the read data is not geometry data, another test is performed to determine whether the read data corresponds to attribute data (test 535). If the read data corresponds to attribute data, the type of attribute data is obtained by the G-PCC unit parser (step 540) to determine the track in which the read data unit is to be stored (step 545). The writer preferably stores the size of the read attribute data unit and may decode its ADU_slice_id (i.e., the identifier of the slice that depends on the ADU). It is recalled that for a G-PCC bitstream, the attribute data unit corresponds to an ADU-type TLV unit, an APS (attribute parameter set), a default attribute data unit, or an FSAP data unit (frame-specific attribute property). If the read data does not correspond to attribute data (step 535), for example, if the read data corresponds to an SPS, a frame boundary marker, or user data, the read data unit is stored in the geometry track (step 530).

[0081] Once a data unit is stored in a track (step 550), the writer stores information about this data unit and associated slice information. The metadata structures and types of information in which this information item is stored are described below with reference to Figures 7, 8, and 9. As shown, the process is repeated across data units within a frame (test 555) and across multiple frames (test 560) of the point cloud sequence to encapsulate them.

[0082] Handling replicated data Note that the instruction to preserve or discard duplicated data units (geometry and / or attribute data units) can also be applied to single tracks or tile tracks that do not have component tracks (which still have an instruction at the track level, or in the sample description, e.g., in the sample entry or configuration box if a GPCCConfigurationBox or GPCCComponentInfoBox is present). This can be indicated, for example, by a two-bit parameter, where the first bit, when set to 1, indicates the presence of duplicated GDUs, and the second bit, when set to 1, indicates the presence of duplicated ADUs. When stored in a GPCCComponentInfoBox, this can be indicated by some reserved flag value in the box. This informs the parser that, for bitstream reconstruction, a choice is also possible regarding these duplicated data units, which can be kept or discarded by the parser in the reconstructed bitstream. Duplicate data units can occur when repeated slices are present. The same instruction as above can also be applied to repeated slices, and the third bit, when set to 1, can indicate that the repeated slice has been removed from the media file (writer) or the reconstructed bitstream (parser). Again, the parser can decide to keep or discard repeated slices. The values ​​of these parameters may be set during the configuration step of the encapsulation module (step 505), or may be determined by the encapsulation module itself during this configuration. For example, when the writer (or server) knows that the transmission is reliable (e.g., via HTTP), duplicated data units are not kept, and an indication that duplicated data units are to be discarded is set in at least one of the encapsulated tracks. Alternatively, when the writer (or server) knows that the transmission may be unreliable (e.g., using RTP), the writer (or server) may keep duplicated data units or keep repeated slices.When the indication that replication units are preserved is set, the writer can replicate the APS within the attribute tracks (instead of encapsulating the APS in only one of the attribute tracks), which provides some robustness in case of loss and also provides robustness of reconstruction from a subset of tracks compared to the complete set of tracks.

[0083] Analysis Process Figure 6 shows an example of steps in a parsing process according to some embodiments of the present invention. For illustrative purposes, these steps may be performed by decapsulation module 132 of Figure 1, which may also be referred to as a parser or reader. The parsing process of Figure 6 enables the parser to reconstruct a compliant G-PCC bitstream from a multi-track encapsulation (tiled or non-tiled) by respecting the TLV order and rules for slicing defined in ISO / IEC 23090-9.

[0084] As shown, the first step is directed to receiving a media file to parse (step 600). It may be streamed using a streaming module, e.g., streaming module 134 of FIG. 1, or read from a storage location. Streaming may use a reliable or unreliable (possibly lossy) protocol, depending on the application. Next, a parser is initialized (step 605). Initialization may be performed by parsing the media file's boxes, e.g., the top-level 'moov' box, 'trak' box, and sample description boxes. If the media player includes a decoder (or decompression module, e.g., decompression module 136 of FIG. 1), the decoder may also be initialized during this step, e.g., using decoder configuration information from the sample description (e.g., G-PCC configuration box 'gpcC'). During this step, the parser may obtain information about the slice configuration indicated in the media file by the writer (e.g., step 510 of FIG. 5). Initialization of the parser may include reading instructions written by the encapsulation module (e.g., according to one of the embodiments associated with Figures 7, 8, and 9). As discussed with respect to the writer, the instructions may be implicit (or static) or dynamic (or explicit). The steps shown in Figure 6 focus on the case where the instructions are explicit and dynamic. This means that the parser needs to check the instructions on each sample, group of samples, movie fragment, or media segment. If the instructions are static, they can be read once for all during step 610.

[0085] Next, slice reading begins (step 610). This step involves updating the slice counter or slice index from the value of GDU_slice_id decoded from the first GDU of the current sample (read in step 615), or, for example, starting from the value 1. To find the data unit to read for the current sample, the parser (or decapsulation module) reads the sample description from the metadata portion of the received media file and finds the corresponding media data in the media data boxes of the received media file. It also identifies the geometry track and attribute tracks by inspecting the TrackReferenceBox. Reading the sample consists of reading data units from the samples chronologically ordered in the geometry track and associated attribute tracks. Reading is done on a slice-by-slice basis to ensure that data units from different slices are not interleaved. This may be dictated by the reconstruction rules.

[0086] In step 615, the parser reads the data units of the current slice in the geometry track. To do so, it can read GDU data units until their payload changes (to collect duplicated GDUs), or read several GDU data units based on instructions in the media file, if present. The parser then records the read position in the 'mdat' box of the next slice. Optionally, the parser can decode the gdu_slice_id or obtain it from instructions in the media file, if present, and check that it matches the current slice_id initialized in step 610. The parser may also compare the header of the current GDU with the header of the next GDU to determine whether there are any changes in the header, i.e., whether at least the slice identifier has changed, or obtain the slice change from instructions in the media file, if present.

[0087] Next, in step 620, the parser reads the data unit from the first reference track (i.e., the first attribute track referenced by the geometry track). For a GDU, the parser may collect a duplicated ADU by checking that the data unit payload has not changed, if present, or by decoding the ADU_slice_id (or defattr_slice_id for a default ADU) and check that it matches the current slice index, or obtain it from an instruction in the media file, if present. The parser may also compare the header of the current ADU with the header of the next ADU to determine whether there is a change in the header, i.e., whether at least the slice identifier has changed, or obtain the slice change from an instruction in the media file when present.

[0088] Next, in step 625, the parser checks whether any other attribute tracks are referenced by the geometry track. If there is at least one other attribute track referenced by the geometry track, the parser repeats steps 620 and 625 to process all referenced attribute tracks. When the last attribute track is reached in step 625, the read data unit (corresponding to a single slice) is appended to the bitstream being reconstructed (step 630). The bitstream can then be processed by a (running) decoder for rendering, or can be stored for later on-demand decoding or rendering.

[0089] If there is a mismatch between the decoded slice_id and the current slice_id, this may be interpreted as an error. However, the parser may store the read data units and later reorder the data units in the proper slice order, then stream these read data into the bitstream being reconstructed.

[0090] Once a data unit of a slice has been parsed, the reader checks whether it has reached the end of the sample (step 635). This can be determined from the payload bytes indicated in each G-PCC unit. If the end of the sample has not been reached, the slice counter is incremented (step 640), and the algorithm loops back to step 615 to parse the data unit of the next slice. Conversely, if the end of the sample has been reached, a test is performed to determine whether there is at least one remaining sample to process (step 645). If there is at least one remaining sample to process, the algorithm loops back to step 610 to begin reading the first slice of a new sample. The algorithm terminates when the last sample of the point cloud sequence is reached.

[0091] When collecting attribute data units in step 620, the parser may check at the end of the loop on the attribute track whether it has collected the expected number of attributes, i.e., the same number of attribute data units as indicated in the SPS. If they are different, the parser can, for example, take the initiative to pad the current slice with some default attribute data units to reach the expected number of attributes before step 630. To identify missing attribute data units, the parser checks the attrIdx from the collected attribute data units and compares it with the list declared in the SPS. Missing attribute data units are created as default data units and are given either the default value of any attribute (e.g., Exp2(attr_bitdepth_minus1[attrIdx]) as defined in ISO / IEC 23090-9) or the default value displayed in the SPS, if present. This particular step may depend on parser settings, such as the integrity check mode, which may or may not be activated (by the user, by the application, by a script, or by default). This integrity check mode can also cause the parser to store corrupted media files for point cloud data in a partial file format, possibly with indications of complete, corrupted, and missing segments.

[0092] Another setting consists in setting the parser to 'purge' mode. If this mode is set, for media files in which duplicated data units exist (either in a dedicated metadata structure or as determined by the parser), the parser will remove these duplicated data units and will not pass them to the point cloud decoder. In this mode, the parser may do the same for repeated slices. The two modes for duplicating data units or repeated slices may correspond to separate configurations for the parser, or may be combined into a single 'purge' mode.

[0093] Below we describe some embodiments for interoperability between ISO / IEC 23090-18 writers and readers. - encapsulating point cloud data in a constrained and standardized way so that a parser can unambiguously reconstruct a valid bitstream from multiple tracks, or - Encapsulates point cloud data with instructions in the media file on how to reconstruct it from multiple tracks without interleaving data units from different slices. Constraints applied to multi-track encapsulation of slice-based point cloud frames (allowing analysis without additional signals) According to this embodiment, encapsulating a G-PCC bitstream comprising slice-based point cloud frames within a multi-track media file does not require the addition of specific signaling within the media file, but does involve encapsulation constraints so that a parser can properly parse the data units. Such constraints apply to component tracks (or attribute tracks), whether tiled or not. The description of multi-track encapsulation is further specified in the following statements: -The slice order of each attribute track matches the slice order of the geometry track -The slice number of each attribute track matches the slice number of the geometry track.

[0094] Optionally, when entropy continuation is in use or when a slice_reordering constraint is indicated in the G-PCC bitstream, the inter-slice ordering between geometry tracks and attribute tracks follows the entropy continuation instruction it references.

[0095] The parser can then rely on these constraints (at least the first constraint) to collect the appropriate number of TLVs per slice in each component track and optionally determine several slices. To determine the number of TLVs per slice from the geometry track or from the attribute track, the parser may maintain the current value of slice_id while reading data units and verify that the read data unit matches this current slice_id (as described with reference to Figure 6). Alternatively, the parser may collect ADUs or GDUs in a track until the payload changes. To detect a possible change in slice_id, only the first byte may be used.

[0096] Reconstruction of a point cloud frame from a geometry track ('gpc1' or 'gpcg') referencing an attribute track ('gpc1' or 'gpcg') can be performed from slice-based time-ordered samples by concatenating G-PCC units in the following order: - any G-PCC units declared in the sample entry, - any G-PCC unit with type set to SPS (value 0) or GPS (value 1) from the geometry track (from the sample entry or within the sample), - If present, any G-PCC units from the geometry track with type set to Tile Inventory Data Unit (value 5) or User Data Data Unit (value 9) For each slice, -All GDUs have the same slice_id and for any G-PCC unit from the geometry track that has type set to GDU (value 2) and for each attribute track referenced from the geometry track via the 'gpca' track reference type: - any G-PCC unit with type set to APS (value 3) from the attribute track, - any frame-specific attribute characteristic data unit (value 8) from the attribute track, - any G-PCC unit with type set to Attribute Data Unit (value 4) or Default Attribute Data Unit (value 7) from an attribute track with the same value of adu_slice_id or defattr_slice_id, - any G-PCC unit with type set to User Data data unit (value 9) from the Attributes Track, and - After the ADUs collected for all slices, if present, any G-PCC units with type set to Frame Boundary Marker Data Unit (value 6) from the Geometry Track.

[0097] The above encapsulation constraints may also be applied to G-PCC tile tracks ('gpt1') referenced from a 'gpcb' tile base track. In this case, for sets of geometry and associated attribute tracks corresponding to the same set of one or more tiles, the slice order in each attribute track matches the slice order in the geometry track, and the slice number in each attribute track matches the slice number in the geometry track. Optionally, inter-slice ordering may also be applied when slice_reordering_constraint is set or when entropy_continuation is set. Reconstruction is the same as above, but applied to each set of geometry and associated attribute tracks for a different tile or set of tiles referenced by the 'gpcb' tile base track.

[0098] Parsing multi-track media files that encapsulate slice-based point cloud frames without specific signaling According to this embodiment, as well as the embodiments described above, encapsulating a G-PCC bitstream comprising slice-based point cloud frames within a multi-track media file does not require the addition of specific signaling within the media file. Additionally, it does not expect specific behavior from the writer. According to this embodiment, the parser checks the slice identifiers of data units each time they are read to ensure that the data units are concatenated in the proper order in the reconstructed bitstream.

[0099] The reconstruction of a point cloud frame from a geometry track (e.g., 'gpc1' or 'gpcg') referencing an attribute track (e.g., 'gpc1' or 'gpcg') may be performed from the time-ordered samples on a slice basis, taking into account that curSliceID is the slice identifier of the current slice (i.e., the slice being processed), and curSliceID is incremented after all data units corresponding to this slice identifier have been processed. For the first sample, the reconstruction is performed by concatenation of G-PCC units in the following order, after the data units from the sample entry are appended to the bitstream: - any G-PCC unit whose type is set to SPS (value 0) or GPS (value 1) from the geometry track (sample entry or within the sample), - If present, any G-PCC units with type set to Tile Inventory Data Unit (value 5) or User Data Data Unit (value 9) from the Geometry Track, For each slice, - Any G-PCC unit with slice_id equal to curSliceID and type set to GDU (value 2) from the geometry track And for each attribute track referenced from a geometry track via a 'gpca' track reference type: - any G-PCC unit with type set to APS (value 3) from the attribute track, - any frame-specific attribute characteristic data unit (value 8) from the attribute track, any G-PCC unit with adu_slice_id or defattr_slice_id equal to curSliceID and with type set to Attribute Data Unit (value 4) or Default Attribute Data Unit (value 7) from the Attribute Track, - any G-PCC unit with type set to User Data Data Unit (value 9) from the Attribute Track, and - After the ADUs collected for all slices, if present, any G-PCC units with type set to Frame Boundary Marker Data Unit (value 6) from the geometry track.

[0100] For G-PCC tile tracks carrying one G-PCC component, referenced from a 'gpcb' tile base track, the reconstruction is the same as above, but applied to each set of geometry and associated attribute tracks of a different tile or set of tiles referenced by the 'gpcb' tile base track.

[0101] Encapsulating slice-based point cloud frames into multi-track media files with signaling information for improved analysis FIG. 7 shows an example of adding static or quasi-static instructions to a multi-track media file encapsulating slice-based point cloud frames to assist a parser in reconstructing a bitstream that conforms to a predetermined format, e.g., to reconstruct a G-PCC bitstream.

[0102] It should be noted that although FIG. 7 shows a particular embodiment in which the additional instructions are located within the geometry track, the additional instructions may be split across different geometry and attribute tracks.

[0103] Additional instructions are based on sample groupings.

[0104] According to the illustrated example, the G-PCC bitstream is encapsulated in a geometry track referenced 700 and two attribute tracks referenced 705 and 710. As shown, samples 1 through N of geometry track 700 include two GDUs (indicated by horizontal hatching) for the first slice and one GDU (indicated by vertical hatching) for the second slice. Samples N+1 through M of the same geometry track have different patterns: one GDU in the first slice (still represented by horizontal hatching) and one GDU in the second slice (still represented by vertical hatching). To further illustrate, attribute track 705 comprises different data unit or TLV unit patterns for each slice of samples 1 through N and samples N+1 through M. According to the illustrated example, attribute track 710 includes duplicated ADUs.

[0105] While encapsulating these data units, the writer may generate metadata structures such as metadata structures 715 and 720. These structures may group samples from ISOBMFF into group and group description boxes, which may be included in sample descriptions at the start of the file or movie fragments over time. Metadata structure 715 may, for example, include a grouping_type equal to 'tlvs' for 'TLV per Slice' that maps groups of samples to sample group entries in metadata structure 720. Sample group entries may inherit from VolumetricVisualSampleGroupEntry defined in ISOBMFF. Sample groups provide a mapping of TLVs to slices, which can assist readers in extracting the correct number of TLV units from each of the geometry and associated attribute tracks.

[0106] For example, samples 1 through N are mapped to the first entry, and samples N+1 through M are mapped to the second entry in metadata structure 720. Note that when all samples share the same pattern, a default sample grouping can be used (in such cases, metadata structure 715 can be omitted and only metadata structure 720 is present). Furthermore, when the configuration does not change over time, the sample grouping can be declared as static using an appropriate flag value for metadata structure 720 defined in ISOBMFF. The contents of metadata structure (or box) 720 include a list of entries indicated by the entry_count parameter. The grouping type parameter of this box, if present, is also 'tlvs' (e.g., or any dedicated 4CC) that pairs this box with the corresponding box in metadata structure 715. Each entry in box 720 includes a list of patterns (e.g., one per slice) and indicates the number of GDUs and ADUs to collect in different tracks. The declarations within each sample group entry follow the same declaration order as the 'gpca' track references that link geometry tracks to attribute tracks. Note that depending on the configuration or settings or application needs, the writer may have declared only one instance of a given ADU (when filtering mode is activated, duplicate data units are not saved). In such a case, the last value in the pattern of the first slice in both entries (referenced in 725 and 730) is 1, and the last entry in the pattern of the second slice in both entries 725 and 730 is also 1.

[0107] The metadata structure 725 or 730 may be defined as follows: definition Group type: 'tlvs' Container: Sample Group Description Box (´sgpd´) Required: No Quantity: Zero or more The use of 'tlvs' for grouping_type in sample grouping represents the assignment of samples in a G-PCC track to TLVs for slice mapping information carried in this sample group. When there is a SampleGroupDescriptionBox with grouping_type equal to 'tlvs', there may be an accompanying SampleToGroupBox with the same grouping type (when the default sample grouping is not applicable). The grouping_type_parameter of this SampleToGroupBox is undefined.

[0108] A 'tlvs' sample group can exist in a 'gpc1' or 'gpcg' geometry track, or in a 'gpt1' geometry tile track referenced by a 'gpcb' tile base track.

[0109] syntax JPEG0007825066000002.jpg3396 semantic num_slices indicates the number of slices present in the samples associated with this sample group, and num_tlvs provides a list of the number of consecutive TLV units per track associated with a given slice in the samples associated with this sample group. The length of the list is equal to one plus the number of track references from the geometry track containing this sample group to the attribute tracks. The list starts with the number of TLV units in the geometry track, followed by the number of TLV units in each attribute track referenced by a 'gpca' track reference, followed by these track references.

[0110] The following reconstruction rules based on metadata structure 720 or 720 plus 715 are proposed for inclusion in section 7.4 of ISO / IEC 23090-18.

[0111] The reconstruction of a point cloud frame from a geometry track ('gpc1' or 'gpcg') referencing an attribute track ('gpc1' or 'gpcg') is performed from slice-based, time-ordered samples by concatenation of G-PCC units in the order indicated by the 'tlvs' sample groups of the geometry track.

[0112] A point cloud frame is reconstructed from multiple component tracks by: -Identification of the corresponding samples within the sample group description; -Identifies the mapping of TLVs from the geometry track's 'tlvs' sample group description box to slices of this sample -For each slice of this sample, read the number of TLV units from the geometry track corresponding to the number of TLV units indicated in the first value of the 'tlvs' entry - For each attribute track referenced from the geometry track via the 'gpca' track reference type, read some number of TLV units from the current attribute track that correspond to the next value of the 'tlvs' entry.

[0113] The following reconstruction rules are proposed for inclusion in section 7.5 of ISO / IEC 23090-18:

[0114] For 'gpt1' geometry tracks referenced by 'gpcb' tile base tracks, the same reconstruction as proposed for inclusion in Section 7.4 of ISO / IEC 23090-18 is applied to each set of geometry tile tracks and their associated attribute tile tracks containing two or more slices (if a tile track contains only one slice, a sample-based reconstruction may be used, i.e., the concatenation of time-ordered samples from the geometry track and its associated attribute tracks).

[0115] FIG. 8 shows an example of adding dynamic instructions to a multi-track media file encapsulating slice-based point cloud frames to assist a parser in reconstructing a bitstream that conforms to a predetermined format, e.g., to reconstruct a G-PCC bitstream.

[0116] According to this embodiment, a description of the TLV-to-slice mapping on a subsample basis is provided. To do so, the writer can use a 'subs' box from the ISOBMFF, e.g., 'subs' box 800, which has a specific flag value indicating that the subsample actually corresponds to a slice (this allows for a lighter description when some data units are duplicated). As shown by references 800-1 to 800-3, such a mapping can be provided for each track, e.g., the geometry track 805 and the attribute tracks 810 and 815. As an example, the flag value 0x000004 (hex) is reserved to indicate that the subsample described in the 'subs' box of a G-PCC track is a G-PCC slice-based subsample. For example, the value of the codec_specific_parameters field of the SubsampleInformationBox can be defined as follows: JPEG0007825066000003.jpg95112 where: slice_id provides the identifier of the slice identifier of the subsample, prev_slice_id indicates the slice identifier of the previous slice on which the subsample depends (if the subsample cannot have a dependency, then prev_slice_id is set to 0xFFFF, the default value indicating that there is no previous slice on which the subsample depends). The prev_slice_id parameter is optional and can be replaced by 16 reserved bits set to the value 0. This is not useful for bitstreams where entropy_continuation_enabled is indicated in the SPS as not active, i.e. when the entropy analysis of a data unit in the current slice does not depend on the final entropy analysis state of a data unit in the previous slice.

[0117] Existing flag values ​​may also be reused, such as the value 0, which indicates TLV-based (or G-PCC unit-based) subsampling. In this case, the codec_specific_parameters field of the SubsampleInformationBox may be defined as follows: JPEG0007825066000004.jpg68113 where slice_id provides the identifier of the subsample (TLV or G-PCC unit). Optionally, in case of a GDU TLV (when payloadType=2), the identifier of the previous slice is given in the delta_prev_slice_id parameter, coded as slice_id minus prev_slice_id, and represented on the remaining 8 bits of codec_specific_parameters. Note that a subsample may not have a dependency, then delta_prev_slice_id is set to 0xFFFF, the default value indicating that there is no previous slice on which the subsample depends. In a variant that does not indicate a possible identifier for the previous slice, the 8 remaining bits are simply reserved and set to 0.

[0118] The slice identifier may be provided in codec_specific_parameter, for example in 16 bits. Optionally, previous_slice_id may be present (possibly coded as a delta to slice_id in 8 bits if bits are needed for other parameters when entropy continuation is activated). In some variants, especially in the case of flag values ​​of subs dedicated to slices, the slice identifier may be omitted and codec_specific_parameters may be equal to 0. The parser should then infer that the corresponding subsample size corresponds to one slice and the next subsample size corresponds to the next slice (this ensures compliance with the increasing order or entropy continuation). This is why 'subs' with a dedicated flag value is safer. To reuse the value 0, it is necessary to follow the ISO / IEC 23090-18 specification (October 13, 2021). th The properties of the component track need to be modified so that it can contain a 'subs' box with a flag value set to 0, which is not allowed in the current version of GPCCComponentInfoBox. To avoid issues with flag values, the 'subs' box 800 can be allowed as an optional box in the GPCCComponentInfoBox. The metadata structure 800 can be generated by the writer during step 545 of Figure 5.

[0119] These additional instructions can be specified for many samples in a sequence, allowing the writer to use an optimized version of the 'subs' box, if available, or perhaps a compressed or deflated version indicated by a particular 4cc. Then, when the parser wants to use the subs information, it must first decompress or uncompress this version of the 'subs' box to obtain the actual subsample sizes and offsets in the media data portion.

[0120] FIG. 9 shows an example of adding slice separators to a multi-track media file encapsulating slice-based point cloud frames to assist a parser in reconstructing a bitstream that conforms to a predetermined format, e.g., a G-PCC bitstream.

[0121] According to this embodiment, specific TLV units that function as separators are added within the media data portion of the file (e.g., within the 'mdat box). Such separators may be added by the writer to each track (i.e., the geometry track and the attribute track) when it detects that a slice has been read (this may be detected by comparing the first byte of data units of the same type or by decoding the first three bytes of a data unit, which may contain a slice identifier such as a GDU, an ADU, or a default attribute data unit). As shown in Figure 9, a new TLV unit 905 is placed between GDUs of two different slices in the geometry track 900. Similarly, new TLV units 910 and 915 are placed between ADUs of two different slices in the attribute tracks 920 and 925, respectively. The bytes for these separator TLVs may be taken into account when calculating the sample size and sample or chunk offsets in the media portion of the file.

[0122] These TLV units are used by the parser to distinguish data units from one slice from data units from another slice, allowing the data units to be ordered properly. When the parser encounters such TLV units in the media data portion of a media file, it should interpret and then discard these TLV units. In other words, the parser should not include these separator TLVs in the reconstructed bitstream. It is observed that such TLV units may not be needed after the last slice of a sample (because the parser may detect the end of the sample as the end of the last slice).

[0123] According to a particular embodiment, this particular TLV unit simply contains a TLV type and an empty payload: JPEG0007825066000005.jpg29112 In a variant, the slice identifier of the preceding data unit may be provided as the payload of the slice separator TLV. Then, tlv_num_payload_bytes is set to 2 (bytes), followed by the value of the slice identifier. When entropy continuation is activated, prev_slice_id may also be provided, allowing the parser to perform a deeper check (without decoding the GDU or ADU header). In this case, tlv_num_payload_bytes may be set to 4 (bytes), and the slice identifier value follows in two bytes, as does the 2-byte value of prev_slice_id.

[0124] This particular TLV unit may only be defined and used at the system level between the writer and the parser: a G-PCC encoder may not need to generate this kind of separator, and a G-PCC decoder may not understand it.

[0125] Optionally, there may be delimiters for slice sets and for repeated slices that easily allow the parser to not include them in the reconstructed bitstream. These can be defined as slice separators with a specific tlv_type and no payload. Similarly, there may be separators between slices with the same slice_tag value. The parser can then check that it presents the data for a given slice tag to the decoder as a contiguous byte range.

[0126] Tiled slice-based point cloud frames It is observed that processing a tiled slice-based point cloud frame is similar to processing a non-tiled slice-based point cloud frame, and the process described above is applied to each tile. However, it is observed that here, the geometry and attribute tracks are divided such that there are several configurations for the tile tracks, where some tiles are organized according to the process described above and some tiles are not organized according to this process (e.g., if the latter contains only one slice).

[0127] Hardware for performing steps of some embodiments of the present disclosure 10 is a schematic block diagram of a computing device 1000 for implementing one or more embodiments of the present disclosure. The computing device 1000 may be a device such as a microcomputer, a workstation, or a light portable device. The computing device 1000 comprises a communication bus 1002 connected to: -CPU (Central Processing Unit) 1004 such as a microprocessor; - a RAM (Random Access Memory) 1008 for storing the executable code of the methods of the embodiments of the present disclosure, as well as registers adapted to record variables and parameters necessary to implement the methods for encapsulating, indexing, decapsulating and / or accessing data, e.g., the memory can have its memory capacity expanded by an optional RAM connected to an expansion port; - a ROM (Read Only Memory) 1006 for storing a computer program for implementing embodiments of the present disclosure; The network interface 1012 is typically connected to a communications network 1014 over which digital data to be processed is transmitted or received. The network interface 1012 may be a single network interface or may consist of a set of different network interfaces (e.g., wired and wireless interfaces, or different types of wired or wireless interfaces). Data is written to the network interface for transmission or read from the network interface for reception under the control of a software application running on the CPU 1004; - A user interface (UI) 1016 for receiving input from a user or displaying information to a user; -Hard disk (HD) 1010; and / or - An I / O module 1018 for sending and receiving data to and from external devices such as video sources and displays.

[0128] The executable code may be stored either in the read-only memory 1006, on the hard disk 1010 or on a removable digital medium such as a disk. According to a variant, the executable code of the program may be received by means of a communications network via the network interface 1012 in order to be stored in one of the storage means of the communications device 1000, such as the hard disk 1010, before being executed.

[0129] The central processing unit 1004 is adapted to control and direct the execution of instructions or portions of software code of a program according to an embodiment of the present disclosure, the instructions being stored in one of the aforementioned storage means. After power-on, the CPU 1004 can execute instructions from the main RAM memory 1008 relating to a software application, after the instructions have been loaded, for example, from a program ROM 1006 or a hard disk (HD) 1010. Such software application, when executed by the CPU 1004, causes the execution of the steps of the flowcharts shown in the previous figures.

[0130] In this embodiment, the device is a programmable device that uses software to perform the methods of the present disclosure, however, alternatively, the methods of the present disclosure may be implemented in hardware (e.g., in the form of an application specific integrated circuit or ASIC).

[0131] Although the present disclosure has been described with reference to particular embodiments, the present disclosure is not limited to the particular embodiments, and modifications will be apparent to those skilled in the art that are within the scope of the present disclosure.

[0132] Many further modifications and variations will be suggested to those skilled in the art by reference to the exemplary embodiments described above, which are not intended to limit the scope of the present disclosure, which is determined solely by the appended claims. In particular, different features from the various embodiments can be interchanged as desired.

[0133] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.

Claims

1. A method for generating a media file comprising a bitstream encapsulated in multiple tracks, the bitstream comprising point cloud data or volume data, the point cloud data or volume data comprising slice-based frames, slices of the frames comprising different types of data units, the method comprising: Obtain a first data unit belonging to a first slice of a frame; the obtained first data units are encapsulated in the media file, and the track is selected according to a type of the first data units; obtaining a second data unit belonging to a second slice of the frame; the obtained second data units are encapsulated in the media file, and the track is selected according to a type of the second data units; obtaining at least one information item characterizing an order within a sample of a track in the bitstream of the second data unit and the first data unit; generating the media file encapsulating the retrieved at least one information item; A method comprising:

2. The method of claim 1 , wherein at least one of the obtained at least one information item is encapsulated as a sample group within a geometry track of the media file.

3. The method of claim 1 , wherein at least one of the retrieved at least one information item is encapsulated in each track of the plurality of tracks of the media file.

4. The method of claim 2 , wherein the sample group provides a number of slices and, for each slice, a list of the number of data units per track associated with the slice.

5. The method of claim 4 , wherein the list has a length equal to the number of track references from the geometry track to the attribute track containing the sample group plus one.

6. The method of claim 4, wherein the list is ordered starting with the number of the data units for a geometry track.

7. The method of claim 1 , wherein the media file conforms to the ISOBMF format, the plurality of tracks comprises a geometry track and at least one attribute track, and the data units comprise geometry data units and attribute data units.

8. 1. A method of analyzing a media file containing encapsulated point cloud or volume data, wherein the point cloud or volume data comprises slice-based frames, slices of the frames being encapsulated in the media file such that different types of data units are contained in different tracks according to their types, the method comprising: obtaining from said media file at least one information item characterizing an order within samples of a track for a first data unit belonging to a first slice and a second data unit belonging to a second slice; obtaining at least one first data unit belonging to said first slice from a first one of the different tracks based on said at least one information item; obtaining, based on the at least one information item, at least one second data unit belonging to a second slice from a second track different from the first track; Concatenating the first data unit and the second data unit so that they are consecutive; A method comprising:

9. 9. The method of claim 8, further comprising obtaining at least one third data unit belonging to a third slice from the first track, the third slice being different from the first slice and located after the first slice, the first data unit and the third data unit belonging to the same sample, and the third data unit being concatenated after the first data unit and the second data unit in the generated bitstream.

10. 9. The method of claim 8, wherein the information item is obtained by reference to a sample comprising a plurality of data units, and the first data unit is obtained from the plurality of data units.

11. The method of claim 8 , wherein the information item provides a number of slices and, for each slice, a list of the number of data units per track associated with the slice.

12. The method of claim 11 , wherein the list has a length equal to the number of track references from the geometry track to the attribute track containing the sample group plus one.

13. The method of claim 11 , wherein the list is ordered starting with the number of the data units for a geometry track.

14. The method of claim 8 , wherein the media file conforms to the ISOBMF format, the different tracks comprise a geometry track and at least one attribute track, and the data units comprise geometry data units or attribute data units.

15. A program for causing a computer to function as a method according to any one of claims 1 to 14.

Images