Method, apparatus, and computer program for improving multi-track encapsulation of point cloud data - Patents.com

Abstract

According to some embodiments of the present disclosure, a method is provided for encapsulating a bitstream in a media file including different tracks, the bitstream including point cloud data, the point cloud data including slice-based point cloud frames, and slices of the point cloud frames including different types of data units. After obtaining a first data unit of a first slice of one point cloud frame and a second data unit of a second slice of the point cloud frame, each of the obtained first and second data units is encapsulated in a track of the media file according to the type of the data unit. At least one information item characterizing a relative position in the track of the media file of the first data unit with respect to the second data unit is obtained and encapsulated in the media file.

Description

[Technical field]

[0001] Field of Disclosure The present disclosure relates to encapsulation of data, particularly point cloud data, in a standard and interoperable format, for example for storing or transmitting slice-based point cloud frames of 3D points as a set of tracks. [Background technology]

[0002] Background to disclosure The Moving Picture Experts Group (MPEG) has standardized the compression and storage of point cloud data (also called volumetric media data) information, which consists of a set of 3D points with associated attribute information such as color, reflectance, and frame index.

[0003] First, MPEG-I Part-9 (ISO / IEC 23090-9) specifies geometry-based point cloud compression (G-PCC) and specifies a bitstream syntax for point cloud information. According to MPEG-I Part-9, a point cloud is an unordered list of points that includes geometry information, optional attributes, and associated metadata. The geometry information describes the location of the points in a three-dimensional Cartesian coordinate system. The attributes are typed properties of each point, such as color or reflectance. The metadata are items of information used to interpret the geometry information and attributes. The G-PCC compression specification (MPEG-I Part-9) defines certain attributes such as a frame index attribute or a frame number attribute with reserved attribute label values ​​(3 for indicating a frame index and 4 for indicating a frame number attribute) that recall that a point cloud frame is a set of points at a particular time instance according to MPEG-I Part-9. A point cloud frame may be partitioned into one or more ordered subframes, tiles, or slices. A subframe is a partial representation of a point cloud frame that consists of points with the same frame number or frame index attribute value. For example, a sub-frame may be a set of points with attributes in a point cloud frame that share a common acquisition, capture, or rendering time. As another example, a sub-frame may be a set of points with those attributes in a point cloud frame that are acquired or captured consecutively during a given time range, or to be rendered in a given time range. As yet another example, a sub-frame may be a set of points with those attributes in a point cloud frame that are acquired according to a laser shot direction, or that correspond to a portion of a scan path of a 3D sensor. In MPEG-I Part-9, point cloud frames are also indicated by the FrameCtr variable, using a parameter (frame_ctr_lsb syntax element) in the frame boundary marker data unit or some data unit headers.

[0004] It is recalled that a tile is a set of slices, identified by a common slice_tag syntax element value, whose geometry is contained within a bounding box that may be specified in a tile inventory data unit. Each tile consists of a single bounding box and an identifier (tileId). The tile information is not used in the decoding process specified in ISO / IEC 23090-9. A slice corresponds to the geometry and attributes of a part of an encoded point cloud frame or of the entire encoded point cloud frame. Each slice should contain at least one geometry data unit (GDU) that encodes a slice geometry and attribute data unit (ADU), or a default attribute data unit (DU) that encodes slice attributes. A slice is identified by a GDU slice_id. ISO / EC 23090-9 specifies the slice decoding process as a four-step process: - Decoding points in a slice, - Decoding default attributes, if any - Decoding point attributes, if any, and - Point position offset and scaling.

[0005] Second, MPEG-I Part-18 (ISO / IEC 23090-18) specifies a media format that allows for the storage and delivery of geometry-based point cloud compressed data. It also supports flexible extraction of geometry-based point cloud compressed data during delivery and / or decoding. According to MPEG-I Part-18, a point cloud frame is encapsulated within one or more G-PCC tracks, and a sample is within a G-PCC track corresponding to a single point cloud frame. Each sample contains one or more G-PCC units that belong to the same presentation time. A G-PCC unit is a type-length-value (TLV) encapsulation structure that contains at least one of the following: sequence parameter set (SPS), geometry parameter set (GPS), attribute parameter set (APS), tile inventory, frame boundary marker, geometry data unit (GDU), attribute data unit (ADU), default attribute data unit, frame specific attribute property (FSAP) data unit, and user data unit. The syntax of the TLV encapsulation structure is defined in Annex B of ISO / IEC 23090-9. Summary of the Invention [Problem to be solved by the invention]

[0006] While the ISO Base Media File Format has proven efficient for encapsulating point cloud data, there is a need to improve the encapsulation efficiency, especially to improve multi-track encapsulation of slice-based point cloud frames of 3D points (i.e., point cloud frames comprising at least one slice). [Means for solving the problem]

[0007] Disclosure Summary The present disclosure is designed to address one or more of the above-mentioned concerns. In this regard, a solution is provided for improving the encapsulation of point cloud data.

[0008] According to a first aspect of the present disclosure, there is provided a method for encapsulating a bitstream into a media file including different tracks, the bitstream including point cloud data, the point cloud data including slice-based point cloud frames, the slices of the point cloud frames including different types of data units, the method comprising: Obtain a first data unit of a first slice of one point cloud frame; for each of the obtained first data units, encapsulating the first data unit within a track of the media file, the track being selected as a function of a type of the first data unit; Obtaining a second data unit of a second slice of the point cloud frame; for each of the obtained second data units, encapsulating the second data unit within a track of the media file, the track being selected as a function of a type of the second data unit; obtaining at least one information item characterizing the relative order within the bitstream of the first data unit with respect to the second data unit; encapsulating said retrieved at least one information item within said media file; A method is provided, comprising:

[0009] Thus, the methods of the present disclosure enable describing the encapsulated data units of a bitstream comprising a slice-based point cloud data frame, enabling a parser to generate a bitstream having properly ordered data units.

[0010] According to some embodiments, at least one of the obtained at least one information item is encapsulated within the geometry track of the media file as a sample group.

[0011] Further according to some embodiments, at least one of the retrieved at least one information item is encapsulated in each of the different tracks of the media file.

[0012] Further according to some embodiments, at least one information item encapsulated in each of the different tracks of the media file is a slice separator.

[0013] Furthermore, according to some embodiments, at least one information item encapsulated in each of the different tracks of the media file comprises a description of the structure of a data unit within a sample, each data unit being associated with slice information.

[0014] Furthermore, according to some embodiments, at least one item of information encapsulated in each of said different tracks of said media file includes a description of the structure of a slice within a point cloud frame.

[0015] Furthermore, according to some embodiments, the media file complies with the ISOBMF format, and the different tracks comprise a geometry track and at least one attribute track, and the data units comprise geometry data units and attribute data units.

[0016] According to a second aspect of the present disclosure, there is provided a method of analysing a media file including encapsulated point cloud data, the point cloud data including slice-based point cloud frames, the slices of the point cloud frames including different types of data units, the data units being encapsulated in the media file in different tracks according to their type, the method comprising: Obtaining data units from said tracks on a slice basis; concatenating the acquired data units on a slice basis such that all the acquired data units related to a given slice of a given track are contiguous with all the acquired data units related to a slice corresponding to the given slice in the other of the different tracks of the given track; generating a bitstream comprising the concatenated data units; A method is provided, comprising:

[0017] Thus, the disclosed method enables parsing slice-based point cloud frames encapsulated in a multi-track media file to generate a bitstream in which data units are properly ordered, where data units belonging to a slice of a track and data units belonging to a corresponding slice of another track belong to the same slice in the bitstream.

[0018] According to some embodiments, concatenating the obtained data units comprises: obtaining a first set of at least one first data unit from a first track of different tracks, the data units of the first set belonging to a first slice; obtaining a second set of at least one second data unit from a second track, the second track being different from the first track, the data units of the second set belonging to a second slice, the second slice corresponding to the first slice; The data units of the first set and the data units of the second set are concatenated so that the data units of the first set and the data units of the second set are contiguous.

[0019] According to further some embodiments, the method further comprises obtaining a third set of at least one third data unit from the first track, wherein the data units of the third set belong to a third slice, the third slice being different from the first slice and following the first slice, the data units of the first set and the data units of the third set belong to the same sample, and the data units of the third set are concatenated after the data units of the first set and the data units of the second set in the generated bitstream.

[0020] Further, according to some embodiments, obtaining the first set of at least one first data unit comprises: Identifying a first data unit of a first set; comparing the information item of the identified first data unit with the information item of the data unit following the identified first data unit in the first track; determining, as a result of the comparison, whether a data unit following the identified first data unit belongs to the first set.

[0021] Further, according to some embodiments, obtaining the first set of at least one first data unit comprises: Identifying a data unit in the first track; Obtain a slice identifier associated with the identified data unit; The method includes comparing the obtained slice identifier with a current slice identifier, and determining whether the identified data unit belongs to the first set as a result of the comparison.

[0022] Further, according to some embodiments, obtaining the first set of at least one first data unit includes: Obtaining a slice encapsulation description (data unit per slice description) from the media file and obtaining at least one first data unit from the first track as a function of the obtained slice encapsulation description (data unit per slice description).

[0023] Further according to some embodiments, a slice encapsulation description (a data unit per slice description) is obtained as a function of reference to a sample comprising a plurality of data units, and the data units of the first set are obtained from the plurality of data units.

[0024] Further, according to some embodiments, obtaining the first set of at least one first data unit comprises: identifying a first data unit of the first set in the first track; Obtaining a type of a data unit following the identified first data unit in the first track; providing an information item indicating that all the first data units of the first set have been obtained if the obtained type corresponds to a slice separator;

[0025] Furthermore, according to some embodiments, the media file complies with the ISOBMF format, and the different tracks comprise a geometry track and at least one attribute track, and the data units comprise geometry data units and attribute data units.

[0026] Additionally, according to some embodiments, the data unit per slice description provides the number of slices and, for each slice, provides a list of the number of data units per track associated with the slice.

[0027] Further according to some embodiments, the list has a length equal to the number of track references from the geometry track to the attribute track that contains the sample group plus one.

[0028] Furthermore, according to some embodiments, the list is ordered starting with the number of the data units for the geometry track.

[0029] According to a third aspect of the present disclosure, there is provided a method for encapsulating a bitstream into a media file comprising different tracks, the bitstream comprising point cloud data, the point cloud data comprising slice-based point cloud frames, the slices of the point cloud frames comprising different types of data units, the method comprising: Obtain a first data unit of a first slice of one point cloud frame; for each of the obtained first data units, encapsulating the first data unit within a track of a media file, the track being selected as a function of a type of the first data unit; Obtain a second data unit of a second slice of the point cloud frame; for each of the obtained second data units, encapsulating the second data unit within a track of the media file, the track being selected as a function of a type of the second data unit; obtaining at least one information item characterizing the relative order within the bitstream of the first data unit with respect to the second data unit; encapsulating said retrieved at least one information item in a media file; A method is provided, comprising:

[0030] Thus, the methods of the present disclosure enable describing the encapsulated data units of a bitstream comprising a slice-based point cloud data frame, enabling a parser to generate a bitstream having properly ordered data units.

[0031] According to some embodiments, at least one of the obtained at least one information item is encapsulated within the geometry track of the media file as a sample group.

[0032] Furthermore, according to some embodiments, the sample group provides a number of slices and, for each slice, a list of the number of data units per track associated with the slice.

[0033] Further according to some embodiments, the list has a length equal to the number of track references from the geometry track to the attribute track that contains the sample group plus one.

[0034] Further according to some embodiments, the list is ordered starting with the number of data units for the geometry track.

[0035] Furthermore, according to some embodiments, the media file complies with the ISOBMF format, and the different tracks comprise a geometry track and at least one attribute track, and the data units comprise geometry data units and attribute data units.

[0036] According to another aspect of the present disclosure, there is provided a device comprising a processing unit configured to perform the steps of the above-mentioned method.

[0037] This aspect of the disclosure has similar advantages to those discussed above.

[0038] At least some of the methods according to the present disclosure may be computer-implemented. Thus, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be generally referred to herein as a "circuit," "module," or "system." Furthermore, the present disclosure may take the form of a computer program product embodied in any tangible medium of expression having computer usable program code embodied in the medium.

[0039] Since the solutions of the present disclosure can be implemented in software, the solutions of the present disclosure can be embodied as computer readable code for providing to a programmable device on any suitable carrier medium. The tangible carrier medium can comprise a storage medium such as a floppy disk, a CD-ROM, a hard disk drive, a magnetic tape device, or a solid-state memory device. The ephemeral carrier medium can be an electric signal, an electronic signal, an optical signal, an acoustic signal, a magnetic signal, or an electromagnetic signal, such as a microwave or RE signal. [Brief description of the drawings]

[0040] Some embodiments of the present disclosure will now be described, by way of example only, with reference to the following drawings: [Figure 1] FIG. 1 shows an example of a system in which the present invention can be implemented. [Diagram 2] FIG. 2 shows an example of encapsulating bitstreams in several tracks of a media file and parsing the latter to generate bitstreams compliant with a given format, such as MPEG-I Part-9. [Diagram 3] FIG. 3 shows a first example of a frame structure of a sequence of frames of an encapsulated and parsed G-PCC bitstream, where all slices of a point cloud frame have the same structure. [Figure 4] FIG. 4 shows a second example of a frame structure of a sequence of frames of a G-PCC bitstream that is encapsulated and parsed, where not all slices of a point cloud frame have the same structure. [Diagram 5] FIG. 5 illustrates example steps of an encapsulation process according to some embodiments of the present invention, enabling a parser to encapsulate point cloud data into multiple tracks in an interoperable manner to generate a compliant G-PCC bitstream. [Figure 6] FIG. 6 illustrates example steps of a parsing process according to some embodiments of the present invention. [Figure 7] FIG. 7 illustrates, for example, an example of adding static or semi-static instructions to a multi-track media file encapsulating slice-based point cloud frames to assist a parser in reconstructing a bitstream that conforms to a predefined format in order to reconstruct a G-PCC bitstream. [Figure 8] FIG. 8 illustrates, for example, an example of adding dynamic instructions to a multi-track media file encapsulating slice-based point cloud frames to assist a parser in reconstructing a bitstream that conforms to a predefined format in order to reconstruct a G-PCC bitstream. [Figure 9] FIG. 9 shows an example of adding slice separators to a multi-track media file encapsulating slice-based point cloud frames to assist the parser in reconstructing a bitstream that conforms to a predefined format, e.g., reconstructing a G-PCC bitstream. [Figure 10] FIG. 10 implements at least one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] Detailed Description of the Disclosure The inventors have observed that the data units of slice-based point cloud frames in a bitstream should be encapsulated according to certain constraints or with additional metadata that enables a parser to generate a usable bitstream that conforms to a given format, i.e., a bitstream with properly ordered data units.

[0042] According to some embodiments of the present disclosure, data units of slice-based point cloud frames of a bitstream are encapsulated within a multi-track media track such that a parser can parse the encapsulated data units to provide a bitstream that conforms to a predefined format, e.g., a bitstream that conforms to MPEG-I Part-9.

[0043] Furthermore, according to some embodiments of the present disclosure, additional metadata is provided in a track of the media file, for example a G-PCC track, to enable identification of data units of a given slice to data units of another slice. Such additional metadata can include a slice end indication in the track, or a description of the data unit organization within the slice, where such organization can be static or dynamic.

[0044] 1 shows an example of a system in which the present invention can be implemented. More precisely, the present invention can be used in a media file writer, such as media file writer 100, or a media player, such as media player 130, or both.

[0045] As shown, the media file writer 100 takes as input point cloud data (or volume data), such as point cloud data 150. The point cloud data 150 may be acquired from a 3D sensor, as described with reference to FIG. 2. The point cloud data may be received as uncompressed raw data or as a compressed bitstream, for example a compressed bitstream conforming to the MPEG-I Part-9 standard. The media file writer 100 comprises an encapsulation module 102.

[0046] The media file writer 100 can be connected via a network interface (not shown) to a communications network 120 which can also be connected via a network interface (not shown) to a media player (or reader) 130 equipped with a decapsulation module 132.

[0047] The media file writer 100 may be used to provide media files using a protocol for dynamic adaptive streaming over HTTP, such as Dynamic Adaptive Streaming over HTTP (DASH) or HTTP Live Streaming. These protocols require a streaming manifest, such as a media presentation description (MPD) or a playlist. When used to stream encapsulated media content, the media file writer 100 may include a manifest generation module, such as manifest generation module 104. The media file writer 100 may also include a compression module, such as compression module 106 for compressing the input point cloud data into a compressed bitstream, using a point cloud compression algorithm, such as described in MPEG-I Part-9.

[0048] The encapsulation module 102 can encapsulate the received point cloud data according to an ISOBMFF-based format, such as MPEG-I Part-18, for interoperability purposes, to generate a media file, such as media file 152, which can be stored for later use by a player or by an image analysis tool, or transmitted to a media player or streaming client. The encapsulation process performed in the encapsulation module 102 is further described with reference to FIG. 5.

[0049] The media file writer 100 may be controlled and parameterized by a user, e.g., via a graphical user interface, or by an application, e.g., by application code or script. To process compressed point cloud data, e.g., to process a bitstream of compressed point cloud data conforming to MPEG-I Part-9, the encapsulation module 102 may include a g-PCC unit parser that can read the header of a g-PCC unit and determine, e.g., the length of data (e.g., in bytes, as in the tlv_num_payload_bytes syntax element) corresponding to the unit or unit type (e.g., the tlv_type syntax element). The G-PCC unit parser may also be capable of parsing header information for some G-PCC units, e.g., the attribute header (to obtain its type), and may also be capable of parsing parameter sets to obtain general information about the bitstream. To process uncompressed point cloud data, e.g., data acquired directly from a 3D sensor, the encapsulation module 102 can include a point cloud data parser that can read multiple point locations and their attributes directly from the captured raw data (e.g., a ply or pcd file parser). The media writer may be onboard a recording device, a multi-sensor camera device, a vehicle in which the 3D sensor is mounted, or may be part of a software tool in a studio where the volume data is acquired.

[0050] The media file 152 may consist of a single media file or a set of media segment files, for example as an ISOBMFF segment (ISO base media file containing one or more segments). The media file may be a fragmented file, for example for live acquisition or capture and encapsulation or live (or low latency) streaming. It may conform to the ISOBMFF standard or a standard specification derived from ISOBMFF (e.g., MPEG-I Part-18).

[0051] The media player (or reader) 130 may be a streaming client, where multiple streaming functions are handled by a streaming module, such as streaming module 134 implementing a DASH or HLS client for requesting media files, such as media file 152, and adapting transmission parameters. The media player may also implement an RTP or RTC web client, particularly for live transmissions where transmission losses can be experimented with. The media player 130 may also include a decompression module 136 that takes as input a bitstream representing compressed point cloud data, such as a bitstream conforming to MPEG-I Part-9, and generates the point cloud data (or data) for rendering or analysis. The media file 152 may be read from a storage location or may be streamed using the streaming module 134. The data may be read all at once or by chunks, segments, or fragments and provided to the decapsulation module (or analysis module) 132.

[0052] The decapsulation module (or analysis module) 132 then extracts the encapsulated point cloud data, or a subset of the encapsulated point cloud data, depending on the player configuration, or depending on a selection from the user, or depending on multiple parameters of the application using the media player. The extracted point cloud data can result in a bitstream, such as a bitstream that complies with MPEG-I Part-9. In such a case, the bitstream is provided to a decompression module (external decompression module or internal decompression module, e.g., internal decompression module 136) for reconstruction of the point cloud data 154 for use by a user or application (e.g., visualization or analysis). The analysis process is further described with reference to FIG. 6. The media player can be on board a display device (e.g., a smartphone, a tablet, a PC, a vehicle with a multimedia screen, etc., or a software tool in a studio for volumetric data generation).

[0053] Figure 2 shows an example of encapsulating bitstreams in several tracks of a media file and parsing the latter to generate a bitstream conforming to a given format such as MPEG-I Part-9. The encapsulated bitstream may be a G-PCC bitstream and the encapsulated media file may be an ISOBMFF media file, e.g. the 'gpc1' and / or 'gpcg' tracks of ISO / IEC 23090-18. Note that the media data of the encapsulated bitstream may be tiled media data, possibly encapsulated as 'gpcb' tile base tracks referencing the G-PCC tile track 'gpt1', each carrying one G-PCC component (geometry or attributes).

[0054] As shown, the G-PCC bitstream 200 comprises a sequence of multiple point cloud frames, in particular frame N, referenced at 205, where the encoded point cloud frame comprises a sequence of zero or more slices sharing the same value of FrameCtr (a kind of timestamp). For example, point cloud frame N comprises two slices, referenced at 210-1 and 210-2. Each slice comprises one geometry data unit (possibly followed by any replicated geometry data units) that codes the geometry, and may comprise one or more attribute data units or one or more default attribute data units (DUs) that code slice attributes. The number of data units may vary from frame to frame (e.g., different number of slices, new parameter sets, FSAP data units, etc.). For illustration, slice 210-1 comprises one geometry data unit 215-1 and two attribute data units 220-11 and 220-12. Similarly, slice 210-2 comprises one geometry data unit 215-2 and two attribute data units 220-21 and 220-22.

[0055] Slices are identified by the slice_id in the GDU. Slices may be repeated within a coded point cloud frame, but those repetitions should not change the value of slice_id. Slice divisions may be used to allow parallelization, to improve coding efficiency, and / or to enable other functionality such as error resilience and progressive decoding.

[0056] During encapsulation of the G-PCC bitstream 200 in the media file 230, all items of geometry information of the G-PCC bitstream may be stored and described in a geometry track, and each attribute of the G-PCC bitstream may be stored and described in a corresponding attribute track. For example, geometry data units may be stored and described in geometry track 235-1, attribute data units of a first attribute may be stored and described in attribute track 235-2, and attribute data units of a second attribute may be stored and described in attribute track 235-3. It is observed that tracks belonging to the same G-PCC sequence or bitstream are preferably time-aligned (i.e., each track comprises samples, each of which comprises a sample time (e.g., in an 'stts' or 'ctts' box) for storing a time value, and there is one sample of each track associated with a given time value. For example, samples 245-1, 245-2, and 245-3 are time-aligned samples. Thus, each point cloud frame, e.g., point cloud frame 205, can be described as a set of samples in the ISOBMFF, such as sample set 240, comprising one sample in each track, and each sample in the same track represents the same type of information or the same component (geometry information or specific attribute information). For example, sample 245-1 in track 235-1 comprises geometry information, sample 245-2 in track 235-2 comprises attribute information of a first attribute, and sample 245-3 in track 235-3 comprises attribute information of a second attribute.

[0057] According to the specification ISO / IEC 23090-18 on multi-track encapsulation, a media file ends up with samples in different tracks. Each sample in a given track contains data units of a given type from different slices in the original bitstream 200. For example, slices 210-1 and 210-2 have their geometry data units and geometry related information (e.g. GPS or tile inventory) in track 235-1 (i.e. GDUs 215-1 and 215-2), and the same slices have their attribute data units and attribute related information (e.g. APS or frame specific attribute data units) in attribute tracks 235-2 (i.e. ADUs 220-11 and 220-21) and 235-3 (i.e. ADUs 220-12 and 220-22). Note that the representation of the media file 230 is a simplified view of the ISOBMFF. It should be appreciated that the track and sample descriptions are available under the 'moov' and 'trak' boxes indexing data (GDU or ADU). Similarly, it should be appreciated that the data is within the media data boxes (e.g. 'mdat', 'imda', or 'idat' boxes). If the media file is a fragmented ISO-based media file, the track fragment and the sample descriptions of this fragment store the sample data and their associated 'mdat' or 'imda' boxes are within the 'moof' and 'traf' boxes. It should also be noted that each sample comprises data units from different slices. The inventors have observed that there are boundaries within each sample, which may be implicit or signaled, that allow for distinguishing data units from one slice to data units from another slice, as indicated by the dotted line 250.

[0058] Here, it is observed that the bitstream reconstructed by the parser should satisfy several rules to ensure that the decoder is able to decode and / or render the point cloud data. Examples of such rules related to TLV unit order, TLV constraints, and / or TLV slice constraints are as follows; - each slice should contain a single GDU, followed by any duplicate GDUs, followed by any ADUs or default attribute DUs; - DUs belonging to different slices should not be interleaved - Every slice should have a corresponding ADU or default attribute data unit for every attribute listed in the SPS.

[0059] To generate a valid G-PCC bitstream from a media file 230 and according to specifications ISO / IEC 23090-18, e.g., G-PCC bitstream 255, some mechanism should be used to concatenate data units stored in different tracks of the media file in the correct order and following the rule from ISO / IEC 23090-9 that data units belonging to different slices should not be interleaved. To ensure that any parser, independent of its implementation, generates a compliant bitstream for a G-PCC decoder, at least from the slice ordering point of view, one of the following mechanisms may be used: - A reconstructor that implements reconstruction rules that take slices (parser side) into account - an encapsulation mechanism that implements the constraints of multi-track encapsulation (used on the writer side, used on the parser side); a signaling mechanism that provides instructions to be used during the reassembly process (implemented on the writer side, used on the parser side); these instructions are ·static, ·dynamic, Hybrid (static and redefined for some specific / exceptional samples or some fragments) -Interactive in the geometry track · Spread within each component track (tiled or untiled).

[0060] The mechanism used may depend on the configuration of the G-PCC bitstream considered to be encapsulated and parsed, for example, on whether all slices of a point cloud frame have the same structure. For example, when the slice structure is stable (e.g., it comprises the same number of slices across samples and the same number of TLVs per slice), it may not be necessary to provide instructions for samples or sample-based groups, but rather for all samples of a track or track fragment, i.e., by using a static approach. The mechanism used may also depend on the ISO / IEC 23090-18 specification level: it may or may not constrain the writer regarding slice ordering in each component track. Depending on the level of specification, the parser may or may not have some a priori. If it cannot be assumed a priori, the parser may then use additional instructions from the media file.

[0061] FIG. 3 shows a first example of a frame structure of a sequence of frames of a G-PCC bitstream to be encapsulated and parsed, where all slices of a point cloud frame have the same structure.

[0062] Here, we assume that the SPS of a point cloud sequence declares two attributes (as in Figure 2). According to the example in Figure 3, a single structure of data units within a slice is used for all slices of a frame of the considered G-PCC bitstream. In other words, there is a fixed TLV pattern per slice.

[0063] As shown in FIG. 3, each slice 305-1, 305-2, 305-3, and 305-4 of frame 300 comprises one geometry data unit (GDU) followed by two attribute data units (ADUs) or default attribute data units, also referred to as Def ADUs (for clarity, only one slice is shown for frame 300, but there can be multiple slices, here assuming they have the same TLV pattern: one GDU followed by two ADUs or default ADUs).

[0064] While the structure shown in Fig. 3 may be advantageously used to encapsulate and / or parse a G-PCC bitstream, the inventors have observed that it may be expedient to consider when such a structure may change from one slice to another to accommodate data loss (at the writing or parsing side) or when partial reconstruction is performed at the receiver side. For example, it may be considered that the encapsulated and parsed G-PCC bitstream contains multiple slices with the same structure, but the encapsulated and parsed G-PCC bitstream does not contain slices with the same structure, so that parsing can be performed properly even when data units are lost. This may be the case, for example, when the transmission is unreliable, e.g., when using an RTP-based delivery for live or real-time transmission.

[0065] FIG. 4 shows a second example of the structure of a frame of a sequence of frames of a G-PCC bitstream to be encapsulated and parsed, where not all slices of a point cloud frame have the same structure. Again, assume that the SPS for the point cloud sequence declares two attributes (and for simplicity, only one slice for frame 400 is represented, but there can be multiple slices for the frames of the point cloud sequence). However, according to this example, a slice can contain a number of attributes or default attribute data units that differ from the expected number of attributes or default attribute data units. For illustration, slice 405-1 of frame 400 comprises one geometry data unit and one attribute data unit, slice 405-3 comprises one geometry data unit and two attribute data units. In fact, in ISO / IEC JTC1 / SC29 / WG11 MPEG m53681, in order to avoid the need to encode (or decode) attribute data where all attribute values ​​are identical, it is proposed to signal a default attribute data unit by adding a default attribute value to the SPS attribute description to handle omissions of encoded attribute data. Slice 405-1 uses default SPS attribute values ​​for the second attribute, while slice 405-5 uses another concept, which is the frame-specific attribute property data unit (FSAP). When an FSAP data unit contains an attribute property with a type equal to 4, it means that the FSAP data unit defines a default value for a given attribute that applies to a particular frame. Therefore, the attribute data unit for the second attribute of slice 405-5 may be omitted. For other frames, the default value or the actual coded value may be applied.

[0066] Note that losses may occur before encapsulation or during transmission. When the bitstream is damaged, an advanced writer can analyze the bitstream and detect missing data units. When the writer can detect the missing attribute data units, the writing device can be in charge of padding the bitstream with default attribute data units. Similarly, on the receiver or client side, when the parser detects that some attribute data units are missing, the parser can be in charge of padding with default attribute data units to generate a conforming bitstream having the number of attributes declared in the SPS. This last point also applies to partial extraction. When the bitstream is encapsulated into multiple tracks, e.g., one geometric track and N attribute tracks, if the parser extracts only a subset of the attribute tracks having the geometric track, e.g., M tracks where M < N, in order to avoid rewriting the SPS and declaring the number of attributes (declaring N attributes), N - M default attribute data units can be embedded. It can embed the attributes in any order, provided that it does not interleave data units from different slices.

[0067] Encapsulation Process FIG. 5 shows an example of steps of an encapsulation process according to some embodiments of the present invention, enabling a parser to encapsulate point cloud data into multiple tracks in an interoperable way to generate a conforming G-PCC bitstream. For illustration purposes, the steps shown in FIG. 5 may be executed by the encapsulation module 102 of FIG. 1.

[0068] According to the illustrated example, a first step (step 500) is directed to initializing the reception of point cloud data (or volume data) for encapsulation. The encapsulation module is then configured (step 505) according to the type of data to be processed (e.g., depending on whether the point cloud data is compressed or not, depending on whether the point cloud data is to be analyzed or not, depending on whether the point cloud data includes tiles or slices or not). This can be done by a user or an application. For illustrative purposes, the configuration can include selecting single track or multi-track encapsulation, selecting live or offline encapsulation, selecting a description mode, e.g., describing tiles or not. Within the scope of this disclosure, multi-track encapsulation is configured with or without tile tracks.

[0069] The configuration step 505 may also be used to indicate whether the same configuration parameters should be applied to encapsulate the entire point cloud data (static configuration) or whether they may change when encapsulating the point cloud data (dynamic configuration). If the media file writer also includes a compression module, e.g., compression module 106 of FIG. 1, the configuration step may include setting parameters for the encoder, e.g., setting the slice or tile structure. The compression module may encode these configuration parameters, e.g., as additional syntax elements of the G-PCC bitstream, e.g., in a sequence parameter set (SPS), or in dedicated TLV types, e.g., in a tile inventory data unit for the description of bounding boxes for tiles. For example, the SPS may provide information about slice dependencies for parsing and decoding the bitstream (such as the entropy_continuation parameter). When the point cloud data is received by the encapsulation module as a G-PCC bitstream (e.g., a bitstream generated by the compression module 106 of FIG. 1 or a bitstream generated by an external compression module), the configuration of the encapsulation module may use information from the parameter set of the bitstream, or supplemental information (sometimes called Supplemental Enhancement Information messages (SEI)) associated with the bitstream, or information provided in a user data unit. The supplemental information refers to coding parameters that are not required to decode the point cloud data, but can assist applications that use these point cloud data by providing additional information.

[0070] In addition to the configuration of the encapsulation module, metadata structures of the media file, such as top-level boxes (e.g., 'ftyp' or 'styp', 'moov' and its 'trak' sub-box, 'mdat' box, and boxes for sample descriptions such as 'stbl' or 'stsd') are created during an initialization step (step 510). Such an initialization step may include reading parameter sets (e.g., sequence parameter set, geometry parameter set, and / or attribute parameter set) from the encoded bitstream of point cloud data, or may include obtaining information about the sensor used to obtain the point cloud data, such as a number of points, the type of attributes associated with the points (e.g., color, reflectance, timestamp, region of interest, etc.), or may include obtaining information about encoder settings. Note that some of the setting parameters defined in the configuration step 505 may be reflected in the track description or sample description. Similarly, user data or supplementary information, if available, may be included in the media file in whole or in part. Portions of the configuration information (configuration parameters, parameter sets, user data, or supplementary information) may be collected in metadata structures, for example, sample entries (e.g., 'gpc1', 'gpcg', 'gpeb', or any 4CC for G-PCC component tracks) or structures dedicated to sample description, such as configuration boxes (e.g., 'gpcC'). The configuration information may include information about the slice configuration in the bitstream, i.e., the presence or absence of slices, optionally the number of slices per frame, whether this number is constant or not, whether slices are repeated or not, the presence or absence of slice sets, whether there is a slice reordering constraint or not. Having this information may impose fewer constraints on the parser (e.g., if slice_reordering_constraint=0, there is no need to check the slice order in the reconstructed bitstream).Some of this information may be obtained from the parameter set (e.g. slice reordering constraints) and some may require bitstream analysis (like the number of slices and the number of slices per frame). Similarly, entropy_continuation_enabled from the SPS may be useful to reflect in the sample description slices, as it may also constrain the reconstruction process in terms of inter-slice ordering. This may be determined by the encapsulation module during the initialization step 510, or it may be calculated all along the encapsulation and written to a file at the end of the encapsulation (this latter mode is less relevant for live packaging during live or low latency streaming).

[0071] An example metadata structure for providing slice configuration information may be defined as a new box identified by the four-letter code 'scfg', inheriting, for example, from FullBox, e.g., SliceConfigurationInformationBox, as follows: JPEG2025510565000002.jpg86101 During the ceremony slice_present indicates whether the sample's data contains slices or not. When set to 1, this parameter indicates that the sample may contain slices. When set to 0, this parameter indicates that the sample does not contain slices. slice_reordering_constraint has the same semantics as ISO / IEC 23090-9, entropy_continuation_enabled has the same semantics as ISO / IEC 23090-9, repeated_slices indicates whether slices may be repeated within some samples.

[0072] When set to 1, it indicates that the sample may contain repeated slices. When set to 0, this parameter indicates that the sample does not contain repeated slices. same_num_slices_per_frame indicates whether all samples have the same number of slices. When set to 1, it indicates that the samples have the same number of slices. When set to 0, this parameter indicates that at least two samples do not have the same number of slices. num_slices_per_frame indicates the number of slices per sample (if it is a constant or not present), max_num_slices_per_frame indicates the maximum number of slices for a sample in a track.

[0073] This box may be stored in any G-PCC track as any box in the sample entry. Alternatively, the parameters present in the slice configuration information may be stored in the GPCCDecoderConfigurationRecord in a 'gpcC' box.

[0074] Following the initialization step 510, the writer may optionally decide whether indication of reconstruction rules is required, and if so, whether they should be static (or implicit, i.e., applied to all samples) or dynamic (or explicit, i.e., one rule per sample, per group of samples, or per fragment or segment, as the case may be) (step 515). By default, or at least for live packaging, explicit reconstruction rules may be considered, as they allow the writer to handle the majority of configurations, such as those described with reference to Figures 3 and 4.

[0075] The following steps describe the encapsulation with explicit reconstruction rules or instructions that are useful for reconstructing a valid bitstream from multiple tracks.

[0076] The encapsulation process then enters into a processing loop to process each frame of the point cloud data.

[0077] After reading the point cloud frame (step 520), e.g., after reading data unit by data unit from the bitstream of compressed point cloud data, the encapsulation module determines whether the read data corresponds to geometry data (test 525). If it is determined that the read data is geometry data, the read data unit is stored in the geometry track (step 530). In case of an explicit reconstruction rule, the slice_id and any slice_tag of the GDU header are decoded and stored. The size of the read data unit may also be stored along with the slice_id and any prev_slice_id (an identifier of a slice on which the current slice may depend). When the SPS indicates that entropy_continuation is active, the writer may also decode the value of the slice_entropy_continuation parameter, and when set to 1, may decode the value of the prev_slice_id parameter and store this latter value (this is to ensure correct inter-slice ordering).

[0078] When a read geometry data unit is detected as a duplicate GDU (e.g., by calculating the difference of multiple data units or by checking whether they have the same slice_id), depending on the encapsulation mode, the writer can either store this duplicate GDU during step 530 or discard it and store only one instance of the duplicate GDU. This encapsulation choice (presence or absence of duplicated GDUs) can be indicated as a track property or in the sample description of the geometry track (e.g., in a sample entry or configuration box, e.g., GPCCConfigurationBox or GPCCComponentInfoBox).

[0079] Determining the type of the read data unit may comprise determining the type of the G-PCC unit (or TLV unit) by reading its type using a G-PCC unit parser (not shown).

[0080] Returning to test 525, if it is determined that the read data is not geometry data, another test is performed to determine whether the read data corresponds to attribute data (test 535). If the read data corresponds to attribute data, the type of attribute data is obtained by the G-PCC unit parser (step 540) to determine the track in which the read data unit is stored (step 545). The writer preferably stores the size of the read attribute data unit and may decode its ADU_slice_id (i.e., the identifier of the slice depending on the ADU). It is recalled that for a G-PCC bitstream, the attribute data unit corresponds to a TLV unit of ADU type, an attribute parameter set (APS), a default attribute data unit, or a frame-specific attribute property (FSAP) data unit. If the read data does not correspond to attribute data (step 535), for example if the read data corresponds to an SPS, a frame boundary marker, or user data, the read data unit is stored in the geometry track (step 530).

[0081] Once a data unit has been stored in a track (step 550), the writer stores the data unit's information and associated slice information. The metadata structures and types of information in which this information item is stored are described below with reference to Figures 7, 8, and 9. As shown, the process is repeated across data units within a frame (test 555) and across multiple frames (test 560) of the point cloud sequence to encapsulate them.

[0082] Dealing with replicated data It should be noted that the indication to keep or discard duplicated data units (geometry or attribute data units or both) can also be applied to single tracks or tile tracks that do not have component tracks (still have an indication at the track level or in the sample description, e.g. in the sample entry or configuration box if GPCCConfigurationBox or GPCCComponentInfoBox are present). It can be indicated for example by a parameter on two bits, where the first bit, when set to 1, indicates the presence of duplicated GDUs and the second bit, when set to 1, indicates the presence of duplicated ADUs. When stored in the GPCCComponentInfoBox, this can be indicated by some reserved flag value of the box. This informs the parser that for bitstream reconstruction, a choice is possible also regarding these duplicated data units, which can be kept or discarded by the parser in the reconstructed bitstream. Duplicate data units can occur when repeating slices are present. The same indication as above can also be applied to repeating slices, where the third bit, when set to 1, can indicate that the repeating slices have been removed from the media file (writer) or the reconstructed bitstream (parser). Again, the parser can decide to keep or discard the repeated slices. The values ​​of these parameters may be set during the configuration step of the encapsulation module (step 505) or may be determined by the encapsulation module itself during this configuration. For example, when the writer (or server) knows that the transmission is reliable (e.g., via HTTP), the duplicated data units are not kept and an indication that the duplicated data units are discarded is set in at least one of the encapsulated tracks. Alternatively, when the writer (or server) knows that the transmission may not be reliable (e.g., using RTP), the writer (or server) may preserve the duplicated data units or may preserve the repeated slices.When the indication that duplicate units are preserved is set, the writer can duplicate the APS within the attribute tracks (instead of encapsulating the APS in only one of the attribute tracks), which provides some robustness in case of loss and also provides robustness of reconstruction from a subset of tracks compared to the complete set of tracks.

[0083] Analysis process Figure 6 illustrates an example of steps of a parsing process according to some embodiments of the present invention. For illustrative purposes, these steps may be performed by the decapsulation module 132 of Figure 1, which may also be referred to as a parser or a reader. The parsing process of Figure 6 enables the parser to reconstruct a compliant G-PCC bitstream from a multi-track encapsulation (tiled or non-tiled) by respecting the TLV order and rules for slicing defined in ISO / IEC 23090-9.

[0084] As shown, the first step is directed to receiving a media file to parse (step 600). It may be streamed using a streaming module, e.g., streaming module 134 of FIG. 1, or may be read from a storage location. The streaming may use a reliable or unreliable (possibly lossy) protocol, depending on the application. Next, a parser is initialized (step 605). Initialization may be performed by parsing the boxes of the media file, e.g., the top-level 'moov' box, the 'trak' box, and the sample description box. When the media player includes a decoder (or a decompression module, e.g., decompression module 136 of FIG. 1), the decoder may also be initialized during this step, e.g., using decoder configuration information from the sample description (e.g., G-PCC configuration box 'gpcC'). During this step, the parser may obtain information about the slice configuration indicated in the media file by the writer (e.g., step 510 of FIG. 5). Initialization of the parser may include reading the instructions written by the encapsulation module (e.g., according to one of the embodiments associated with Figures 7, 8, and 9). As described with respect to the writer, the instructions may be implicit (or static) or dynamic (or explicit). The steps shown in Figure 6 focus on the case where the instructions are explicit and dynamic. This means that the parser needs to check the instructions on each sample, group of samples, movie fragment, or media segment. If the instructions are static, they can be read once for all during step 610.

[0085] Next, the reading of the slice is started (step 610). This step includes updating the slice counter or slice index from the value of GDU_slice_id decoded from the first GDU of the current sample (read in step 615) or, for example, starting from the value 1. To find the data units to read for the current sample, the parser (or decapsulation module) reads the sample description from the metadata part of the received media file and finds the corresponding media data in the Media Data Box of the received media file. It also identifies the geometry track and the attribute track by inspecting the TrackReferenceBox. The reading of the sample consists of reading data units from the sample ordered chronologically in the geometry track and the associated attribute track. The reading is done on a slice basis to ensure that data units from different slices are not interleaved. This may be dictated by the reconstruction rules.

[0086] In step 615, the parser reads the data units of the current slice in the geometry track. To do so, it can read GDU data units until their payload changes (to collect duplicated GDUs) or read several GDU data units based on the instructions in the media file when present. The parser then records the read position in the 'mdat' box of the next slice. Optionally, the parser can decode the gdu_slice_id or obtain it from the instructions in the media file when present and check that it matches the current slice_id initialized in step 610. The parser may also compare the header of the current GDU with the header of the next GDU to determine whether there is a change in the header, i.e., whether at least the slice identifier has changed, or obtain the slice change from the instructions in the media file when present.

[0087] Next, in step 620, the parser reads the data unit from the first reference track (i.e., the first attribute track referenced by the geometry track). For the GDU, the parser may collect the duplicated ADU by checking that the data unit payload is unchanged, if present, or by decoding the ADU_slice_id (or defattr_slice_id for default ADUs) and check that it matches the current slice index, or may obtain it from an instruction in the media file, if present. The parser may also compare the header of the current ADU with the header of the next ADU to determine whether there is a change in the header, i.e., whether at least the slice identifier has changed, or obtain the slice change from an instruction in the media file when present.

[0088] Next, in step 625, the parser checks whether other attribute tracks are referenced by the geometry track. If there is at least one other attribute track referenced by the geometry track, the parser repeats steps 620 and 625 to process all referenced attribute tracks. When the last attribute track is reached in step 625, the read data unit (corresponding to a single slice) is appended to the bitstream being reconstructed (step 630). The bitstream can then be processed by a (running) decoder for rendering, or stored for later on-demand decoding or rendering.

[0089] If there is a mismatch between the decoded slice_id and the current slice_id, this may be interpreted as an error, however the parser may store the read data units and later reorder the data units in the proper slice order, then stream these read data into the reconstructed bitstream.

[0090] Once a data unit of a slice has been parsed, the reader checks whether it has reached the end of the sample (step 635). This can be determined from the payload bytes indicated in each G-PCC unit. If the end of the sample has not been reached, the slice counter is incremented (step 640) and the algorithm loops to step 615 to parse the data unit of the next slice. Conversely, if the end of the sample has been reached, a test is performed to determine whether there is at least one remaining sample to process (step 645). If there is at least one remaining sample to process, the algorithm loops to step 610 to start reading the first slice of a new sample. The algorithm ends when the last sample of the point cloud sequence has been reached.

[0091] When collecting attribute data units in step 620, the parser may check whether it has collected the expected number of attributes, i.e., the same number of attribute data units as indicated in the SPS, at the end of the loop on the attribute track. If different, the parser may take the initiative, for example, before step 630, to pad the current slice with some default attribute data units to reach the expected number of attributes. To identify missing attribute data units, the parser checks the attrIdx from the collected attribute data units and compares it with the list declared in the SPS. The missing attribute data units are created as default data units and taken as either the default value of any attribute (e.g., Exp2(attr_bitdepth_minus1[attrIdx]) as defined in ISO / IEC 23090-9), or the default value indicated in the SPS if present. This particular step may depend on the parser's settings, such as the integrity check mode, which may or may not be activated (by the user, by the application, by the script, or by default). This integrity check mode can also cause the parser to store corrupted media files for point cloud data in a partial file format, possibly with an indication of complete, corrupted, and missing segments.

[0092] Another setting consists in setting the parser to 'purge' mode. If this mode is set, for media files where duplicated data units exist (either in a dedicated metadata structure or determined by the parser), the parser will remove these duplicated data units and will not pass them to the point cloud decoder. In this mode, the parser may do the same for repeated slices. The two modes for duplicating data units or repeated slices may correspond to separate configurations for the parser or may be combined into a single 'purge' mode.

[0093] In the following, several embodiments for interoperability between ISO / IEC 23090-18 writers and readers are described. - Encapsulating the point cloud data in a constrained and standardized way so that a parser can unambiguously reconstruct a valid bitstream from multiple tracks, or - Encapsulates point cloud data with instructions in the media file on how to reconstruct it from multiple tracks, without interleaving data units from different slices. Constraints imposed on multi-track encapsulation of slice-based point cloud frames (allowing analysis without the use of additional signals) According to this embodiment, encapsulating a G-PCC bitstream comprising slice-based point cloud frames within a multi-track media file does not require the addition of specific signaling within the media file, but does involve encapsulation constraints so that a parser can properly parse the data units. Such constraints apply to component tracks (or attribute tracks), whether tiled or not. The description of multi-track encapsulation is further clarified in the following statement: -The slice order of each attribute track matches the slice order of the geometry track -The slice number of each attribute track matches the slice number of the geometry track.

[0094] Optionally, when entropy continuation is in use or when a slice_reordering constraint is indicated in the G-PCC bitstream, the inter-slice ordering between geometry tracks and attribute tracks follows the entropy continuation instruction it references.

[0095] The parser can then rely on these constraints (at least the first constraint) to collect the appropriate number of TLVs per slice in each component track and optionally determine the number of slices. To determine the number of TLVs per slice from the geometry track or on the attribute track, the parser may maintain the current value of slice_id while reading the data unit and verify that the read data unit matches this current slice_id (as described with reference to FIG. 6). Alternatively, the parser may collect ADUs or GDUs in the track until the payload changes. To detect a possible change in slice_id, only the first byte may be used.

[0096] Reconstruction of a point cloud frame from a geometry track ('gpc1' or 'gpcg') referencing an attribute track ('gpc1' or 'gpcg') can be performed from slice-based time-ordered samples by concatenating G-PCC units in the following order: - any G-PCC units declared in the sample entry, - any G-PCC unit with type set to SPS (value 0) or GPS (value 1) from the geometry track (from the sample entry or within the sample), - Any G-PCC units with type set to Tile Inventory Data Units (value 5) or User Data Data Units (value 9) from the geometry track, if present For each slice, -All GDUs have the same slice_id and for any G-PCC unit with type set to GDU (value 2) from the geometry track and for each attribute track referenced from the geometry track via the 'gpca' track reference type: - any G-PCC unit with the type from the attribute track set to APS (value 3), - any frame-specific attribute characteristic data unit (value 8) from the attribute track, - any G-PCC unit with type set to attribute data unit (value 4) or default attribute data unit (value 7) from an attribute track with the same value of adu_slice_id or defattr_slice_id, - any G-PCC unit with its type set to User Data data unit (value 9) from the Attribute Track, and - After the ADUs collected for all slices, if present, any G-PCC units with type set to the Frame Boundary Marker Data Unit (value 6) from the Geometry Track.

[0097] The above encapsulation constraints may also be applied to G-PCC tile tracks ('gpt1') referenced from a 'gpcb' tile base track. In this case, for a set of geometry and associated attribute tracks corresponding to the same set of one or more tiles, the slice order in each attribute track matches the slice order in the geometry track, and the slice numbers in each attribute track match the slice numbers in the geometry track. Optionally, inter-slice ordering may also be applied when slice_reordering_constraint is set or when entropy_continuation is set. The reconstruction is the same as above, but applied to each set of geometry and associated attribute tracks of a different tile or set of tiles referenced by the 'gpcb' tile base track.

[0098] Parsing multi-track media files that encapsulate slice-based point cloud frames without specific signaling According to this embodiment, and others such as those described above, encapsulating a G-PCC bitstream with slice-based point cloud frames in a multi-track media file does not require the addition of specific signaling in the media file. In addition, it does not expect any specific behavior of the writer. According to this embodiment, the parser checks the slice identifier of a data unit every time the data unit is read to ensure that the data units are concatenated in the proper order in the reconstructed bitstream.

[0099] The reconstruction of a point cloud frame from a geometry track (e.g., 'gpc1' or 'gpcg') referencing an attribute track (e.g., 'gpc1' or 'gpcg') may be performed from the time-ordered samples on a slice basis, taking into account that curSliceID is the slice identifier of the current slice (i.e. the slice being processed), which is incremented after all data units corresponding to this slice identifier have been processed. For the first sample, the reconstruction is performed by concatenation of G-PCC units in the following order after the data units from the sample entry are appended to the bitstream: - any G-PCC unit whose type is set to SPS (value 0) or GPS (value 1) from the geometry track (sample entry or within the sample), - any G-PCC units with type set to Tile Inventory Data Units (value 5) or User Data Data Units (value 9) from the geometry track, if present; For each slice, - Any G-PCC unit with slice_id equal to curSliceID and type set to GDU (value 2) from the geometry track And for each attribute track referenced from a geometry track via a 'gpca' track reference type: - any G-PCC unit with the type from the attribute track set to APS (value 3), - any frame-specific attribute characteristic data unit (value 8) from the attribute track, - any G-PCC unit with adu_slice_id or defattr_slice_id equal to curSliceID and with type set to attribute data unit (value 4) or default attribute data unit (value 7) from the attribute track, - any G-PCC unit with type set to User Data Data Unit (value 9) from the Attribute Track, and - After the ADUs collected for all slices, if present, any G-PCC units with type set to Frame Boundary Marker Data Unit (value 6) from the geometry track.

[0100] For G-PCC tile tracks carrying one G-PCC component, referenced from a 'gpcb' tile base track, the reconstruction is the same as above, but applied to each set of geometry and associated attribute tracks of a different tile or set of tiles referenced by the 'gpcb' tile base track.

[0101] Encapsulating slice-based point cloud frames into multi-track media files using signaling information for improved analysis FIG. 7 shows an example of adding static or quasi-static instructions to a multi-track media file encapsulating slice-based point cloud frames to assist a parser in reconstructing a bitstream that conforms to a predefined format, e.g., to reconstruct a G-PCC bitstream.

[0102] It should be noted that although FIG. 7 shows a particular embodiment in which the additional instructions are located within the geometry track, the additional instructions may be split across different geometry and attribute tracks.

[0103] Additional instructions are based on sample groupings.

[0104] According to the illustrated example, the G-PCC bitstream is encapsulated in a geometry track referenced at 700 and two attribute tracks referenced at 705 and 710. As shown, samples 1 to N of the geometry track 700 include two GDUs for the first slice (represented by horizontal hatching) and one GDU for the second slice (represented by vertical hatching). Samples N+1 to M of the same geometry track have different patterns: one GDU in the first slice (still represented by horizontal hatching) and one GDU in the second slice (still represented by vertical hatching). To further illustrate, the attribute track 705 comprises different data unit or TLV unit patterns for the slices of samples 1 to N and samples N+1 to M. According to the illustrated example, the attribute track 710 includes duplicated ADUs.

[0105] While encapsulating these data units, the writer may generate metadata structures such as metadata structures 715 and 720. These structures may sample from ISOBMFF to group and group description boxes that may be included in the sample description or movie fragment at the start of the file over time. Metadata structure 715 may include a grouping_type equal to 'tlvs' for example 'TLV per Slice' that maps the groups of samples to sample group entries in metadata structure 720. Sample group entries may inherit from VolumetricVisualSampleGroupEntry defined in ISOBMFF. Sample groups provide a mapping of TLVs to slices that may assist the reader to extract the correct number of TLV units from each of the geometry and associated attribute tracks.

[0106] For example, samples 1 to N are mapped to the first entry, samples N+1 to M are mapped to the second entry of the metadata structure 720. Note that when all samples share the same pattern, a default sample grouping may be used (in such a case, the metadata structure 715 may be omitted and only the metadata structure 720 is present). Furthermore, when the configuration does not change over time, the sample grouping may be declared as static using an appropriate flag value for the metadata structure 720 defined in ISOBMFF. The content of the metadata structure (or box) 720 contains a list of entries indicated by the entry_count parameter. The grouping type parameter of this box, if present, is also a 'tlvs' (e.g., or any dedicated 4CC) that pairs this box with the corresponding box in the metadata structure 715. Each entry in the box 720 contains a list of patterns (e.g., one per slice) and indicates the number of GDUs and ADUs to collect in the different tracks. The declarations in each sample group entry follow the same declaration order as the 'gpca' track references that link the geometry track to the attribute track. Note that depending on configuration or settings or application needs, a writer may have declared only one instance of a given ADU (when filtering mode is activated, duplicate data units are not stored). In such a case, the last value in the pattern of the first slice in both entries (referenced at 725 and 730) is 1, and the last entry in the pattern of the second slice in both entries 725 and 730 is also 1.

[0107] The metadata structure 725 or 730 may be defined as follows: definition Group type: 'tlvs' Container: Sample Group Description Box (´sgpd´) Required: No Quantity: Zero or more The use of 'tlvs' for grouping_type in a sample grouping represents the assignment of samples in a G-PCC track to TLVs for slice mapping information carried in this sample group. When a SampleGroupDescriptionBox with grouping_type equal to 'tlvs' is present, there may be an associated SampleToGroupBox with the same grouping type (when the default sample grouping is not applicable). The grouping_type_parameter of this SampleToGroupBox is undefined.

[0108] A 'tlvs' sample group can exist in a 'gpc1' or 'gpcg' geometry track, or in a 'gpt1' geometry tile track referenced by a 'gpcb' tile base track.

[0109] syntax JPEG2025510565000003.jpg3396 semantic num_slices indicates the number of slices present in the samples associated with this sample group, and num_tlvs provides a list of the number of consecutive TLV units per track associated with a given slice in the samples associated with this sample group. The length of the list is equal to the number of track references from the geometry track containing this sample group to attribute tracks plus one. The list starts with the number of TLV units in the geometry track, followed by the number of TLV units in each attribute track referenced by a 'gpca' track reference, followed by these track references.

[0110] The following reconstruction rules based on metadata structure 720 or 720 plus 715 are proposed for inclusion in section 7.4 of ISO / IEC 23090-18.

[0111] The reconstruction of a point cloud frame from a geometry track ('gpc1' or 'gpcg') referencing an attribute track ('gpc1' or 'gpcg') is performed from slice-based time-ordered samples by concatenation of G-PCC units in the order indicated by the 'tlvs' sample groups of the geometry track.

[0112] A point cloud frame is reconstructed from multiple component tracks by: -Identification of the corresponding sample within the sample group description; -Identifies the mapping of TLVs from the geometry track's 'tlvs' sample group description box to slices of this sample. -For each slice of this sample, read the number of TLV units from the geometry track that corresponds to the number of TLV units indicated in the first value of the 'tlvs' entry - For each attribute track referenced from the geometry track via the 'gpca' track reference type, read some number of TLV units from the current attribute track that correspond to the next value of its 'tlvs' entry.

[0113] The following reconstruction rules are proposed for inclusion in section 7.5 of ISO / IEC 23090-18:

[0114] For 'gpt1' geometry tracks referenced by 'gpcb' tile base tracks, the same reconstruction as proposed for inclusion in section 7.4 of ISO / IEC 23090-18 is applied to each set of geometry tile tracks and their associated attribute tile tracks that contain two or more slices (if a tile track contains only one slice, a sample-based reconstruction may be used, i.e. the concatenation of time-ordered samples from the geometry track and its associated attribute tracks).

[0115] FIG. 8 shows an example of adding dynamic instructions to a multi-track media file encapsulating slice-based point cloud frames to assist a parser in reconstructing a bitstream that conforms to a predefined format, e.g., to reconstruct a G-PCC bitstream.

[0116] According to this embodiment, a description of TLV to slice mapping on a subsample basis is provided. For that purpose, the writer can use a 'subs' box from ISOBMFF, e.g. 'subs' box 800, which here has a specific flag value indicating that the subsample actually corresponds to a slice (this allows for a lighter description when some data units are duplicated). As shown by references 800-1 to 800-3, such a mapping can be provided for each track, e.g. geometry track 805, as well as attribute tracks 810 and 815. As an example, the flag value 0x000004 (hex) is reserved to indicate that the subsamples described in the 'subs' box of a G-PCC track are G-PCC slice-based subsamples. For example, the value of the codec_specific_parameters field of the SubsampleInformationBox can be defined as follows: JPEG2025510565000004.jpg95112 Where: slice_id provides the identifier of the slice identifier of the subsample, prev_slice_id indicates the slice identifier of the previous slice on which the subsample depends (if the subsample cannot have a dependency then prev_slice_id is set to 0xFFFF, the default value indicating that there is no previous slice on which the subsample depends). The prev_slice_id parameter is optional and can be replaced by 16 reserved bits set to value 0. This is not useful for bitstreams where entropy_continuation_enabled is indicated in the SPS as not active, i.e. when the entropy analysis of a data unit of the current slice does not depend on the final entropy analysis state of a data unit in a previous slice.

[0117] Existing flag values ​​may also be reused, such as the value 0, which indicates TLV-based (or G-PCC unit-based) subsampling. In this case, the codec_specific_parameters field of the SubsampleInformationBox may be defined as follows: JPEG2025510565000005.jpg68113 Here, slice_id provides an identifier of the subsample (TLV or G-PCC unit). Optionally, in case of GDU TLV (when payloadType=2), an identifier of the previous slice is given in the delta_prev_slice_id parameter, coded as slice_id minus prev_slice_id, and represented on the remaining 8 bits of codec_specific_parameters. Note that a subsample may not have a dependency, then delta_prev_slice_id is set to 0xFFFF, the default value indicating that there is no previous slice on which the subsample depends. In a variant that does not indicate a possible identifier for the previous slice, the 8 remaining bits are simply reserved and set to 0.

[0118] The slice identifier may be provided in codec_specific_parameter, for example in 16 bits. Optionally, previous_slice_id may be present (possibly encoded as a delta to slice_id in 8 bits if bits are needed for other parameters when entropy continuation is activated). In some variants, especially in the case of slice-dedicated sub flag values, the slice identifier may be omitted and codec_specific_parameters may be equal to 0. The parser should then infer that the corresponding subsample size corresponds to one slice and the next subsample size corresponds to the next slice (this ensures compliance with the increasing order or entropy continuation). This is why 'subs' with a dedicated flag value is safer. To reuse the value 0, the specification of ISO / IEC 23090-18 (October 13, 2021) requires that the slice identifier be omitted and the codec_specific_parameters may be equal to 0. th The properties of the component track need to be modified so that it can contain a 'subs' box with the flag value set to 0, which is not allowed in the current version of GPCC. To avoid issues with the flag value, the 'subs' box 800 can be allowed as an optional box in the GPCCComponentInfoBox. The metadata structure 800 can be generated by the writer during step 545 of Figure 5.

[0119] These additional instructions can be specified for many samples in a sequence, so that the writer can use an optimized version of the 'subs' box, if available, or perhaps a compressed or deflated version, indicated by a particular 4cc. Then, when the parser wants to use the subs information, it must first decompress or uncompress this version of the 'subs' box to obtain the actual subsample sizes and offsets in the media data portion.

[0120] FIG. 9 shows an example of adding slice separators to a multi-track media file encapsulating slice-based point cloud frames to assist a parser in reconstructing a bitstream that conforms to a predefined format, e.g., a G-PCC bitstream.

[0121] According to this embodiment, a specific TLV unit that acts as a separator is added in the media data portion of the file (e.g., in the 'mdat box). Such a separator may be added by the writer to each track (i.e., the geometry track and the attribute track) when it is detected that a slice has been read (this may be detected by comparing the first byte of data units of the same type or by decoding the first three bytes of a data unit, which may contain a slice identifier such as a GDU, an ADU, or a default attribute data unit). As shown in FIG. 9, a new TLV unit 905 is placed between the GDUs of two different slices in the geometry track 900. Similarly, new TLV units 910 and 915 are placed between the ADUs of two different slices in the attribute tracks 920 and 925, respectively. The bytes for these separator TLVs may be taken into account when calculating the sample size and the sample or chunk offset in the media portion of the file.

[0122] These TLV units are used by the parser to distinguish data units from one slice from data units from another slice, allowing the data units to be properly ordered. When the parser encounters such TLV units in the media data portion of a media file, it should interpret and then discard these TLV units. In other words, the parser should not include these separator TLVs in the reconstructed bitstream. It is observed that such TLV units may not be needed after the last slice of a sample (because the parser may detect the end of the sample as the end of the last slice).

[0123] According to a particular embodiment, this particular TLV unit simply contains a TLV type and an empty payload: JPEG2025510565000006.jpg29112 In a variant, the slice identifier of the preceding data unit may be provided as the payload of the slice separator TLV. Then, tlv_num_payload_bytes is set to 2 (bytes) followed by the value of the slice identifier. When entropy continuation is activated, prev_slice_id may also be provided, allowing the parser to perform a deeper check (without decoding the GDU or ADU header). In this case, tlv_num_payload_bytes may be set to 4 (bytes) and the value of the slice identifier follows in 2 bytes, as does the value of prev_slice_id in 2 bytes.

[0124] This particular TLV unit may only be defined and used at the system level between the writer and the parser: a G-PCC encoder may not need to generate this kind of separator, and a G-PCC decoder may not understand it.

[0125] Optionally, there may be delimiters for slice sets and for repeated slices that easily allow the parser to not include them in the reconstructed bitstream. These can be defined as slice separators with a specific tlv_type and no payload. Similarly, there may be separators between slices with the same slice_tag value. The parser can then check that it presents the data for a given slice tag to the decoder as a contiguous byte range.

[0126] Tiled slice-based point cloud frames It is observed that processing a tiled slice-based point cloud frame is similar to processing a non-tiled slice-based point cloud frame, and the process described above in this specification is applied to each tile. However, it is observed that there are several configurations for the tile tracks where the geometry and attribute tracks are split such that some tiles are organized according to the process described above and some tiles are not organized according to this process (e.g., if the latter contains only one slice).

[0127] Hardware for performing steps of some embodiments of the present disclosure 10 is a schematic block diagram of a computing device 1000 for implementing one or more embodiments of the present disclosure. The computing device 1000 may be a device such as a microcomputer, a workstation, or a light portable device. The computing device 1000 comprises a communication bus 1002 connected to: -CPU (Central Processing Unit) 1004 such as a microprocessor; - a RAM (Random Access Memory) 1008 for storing the executable code of the methods of the embodiments of the present disclosure, as well as registers adapted to record variables and parameters necessary to implement the methods for encapsulating, indexing, decapsulating and / or accessing data, e.g. the memory can be expanded in its memory capacity by an optional RAM connected to an expansion port; - a ROM (Read Only Memory) 1006 for storing a computer program for implementing the embodiments of the present disclosure; - the network interface 1012 is typically connected to a communications network 1014 over which the digital data to be processed is transmitted or received. The network interface 1012 may be a single network interface or may be composed of a set of different network interfaces (e.g. wired and wireless interfaces, or different types of wired or wireless interfaces). Data is written to the network interface for transmission or read from the network interface for reception under the control of a software application executing in the CPU 1004; - A user interface (UI) 1016 for receiving input from a user or displaying information to a user; -Hard Disk (HD) 1010; and / or - An I / O module 1018 for sending and receiving data to and from external devices such as video sources and displays.

[0128] The executable code may be stored either in the read-only memory 1006, in the hard disk 1010 or on a removable digital medium, such as a disk. According to a variant, the executable code of the program may be received by means of a communications network, via the network interface 1012, in order to be stored in one of the storage means of the communications device 1000, such as the hard disk 1010, before execution.

[0129] The central processing unit 1004 is adapted to control and direct the execution of instructions or parts of the software code of a program according to an embodiment of the present disclosure, which instructions are stored in one of the aforementioned storage means. After power-on, the CPU 1004 can execute instructions from the main RAM memory 1008 relating to a software application, for example after the instructions are loaded from a program ROM 1006 or a hard disk (HD) 1010. Such a software application, when executed by the CPU 1004, causes the execution of the steps of the flowcharts shown in the previous figures.

[0130] In this embodiment, the device is a programmable device that uses software to perform the methods of the present disclosure, however, alternatively, the methods of the present disclosure may be implemented in hardware (e.g., in the form of an application specific integrated circuit or ASIC).

[0131] Although the present disclosure has been described with reference to specific embodiments, the present disclosure is not limited to the specific embodiments, and modifications will be apparent to those skilled in the art that are within the scope of the present disclosure.

[0132] Many further modifications and variations will be suggested to those skilled in the art by reference to the exemplary embodiments described above, which are not intended to limit the scope of the present disclosure, which is determined solely by the appended claims. In particular, different features from the various embodiments can be interchanged, as appropriate.

[0133] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.

Claims

1. 1. A method of encapsulating a bitstream into a media file comprising different tracks, the bitstream comprising point cloud data, the point cloud data comprising slice-based point cloud frames, the slices of the point cloud frames comprising different types of data units, the method comprising: Obtain a first data unit of a first slice of one point cloud frame; for each of the obtained first data units, encapsulating the first data unit within a track of the media file, the track being selected as a function of a type of the first data unit; Obtaining a second data unit of a second slice of the point cloud frame; for each of the obtained second data units, encapsulating the second data unit within a track of the media file, the track being selected as a function of a type of the second data unit; obtaining at least one information item characterizing the relative order within the bitstream of the first data unit with respect to the second data unit; encapsulating said retrieved at least one information item within said media file; A method comprising:

2. The method of claim 1 , wherein at least one of the obtained at least one information item is encapsulated as a sample group within the geometry track of the media file.

3. The method of claim 1 , wherein at least one of the retrieved at least one information item is encapsulated in each of the different tracks of the media file.

4. The method of claim 2 , wherein the sample group provides a number of slices and, for each slice, provides a list of a number of data units per track associated with the slice.

5. The method of claim 4 , wherein the list has a length equal to the number of track references from the geometry track to the attribute track that contains the sample group plus one.

6. The method of claim 4, wherein the list is ordered starting with the number of the data units for the geometry track.

7. 7. The method of claim 1, wherein the media file complies with the ISOBMF format, the different tracks comprise a geometry track and at least one attribute track, and the data units comprise geometry data units and attribute data units.

8. 1. A method of analysing a media file including encapsulated point cloud data, the point cloud data including slice-based point cloud frames, the slices of the point cloud frames including different types of data units, the data units being encapsulated in the media file in different tracks according to their type, the method comprising: Obtaining data units from said tracks on a slice basis; concatenating the acquired data units on a slice basis such that all the acquired data units related to a given slice of a given track are contiguous with all the acquired data units related to a slice corresponding to the given slice in the other of the different tracks of the given track; generating a bitstream comprising the concatenated data units; A method comprising:

9. Obtaining data units and concatenating the obtained data units includes: obtaining a first set of at least one first data unit from a first track of different tracks, the data units of the first set belonging to a first slice; obtaining a second set of at least one second data unit from a second track, the second track being different from the first track, the data units of the second set belonging to a second slice, the second slice corresponding to the first slice; 9. The method of claim 8, comprising concatenating the data units of the first set with the data units of the second set such that the data units of the first set and the data units of the second set are contiguous.

10. 10. The method of claim 8 or 9, further comprising obtaining a third set of at least one third data unit from the first track, wherein the data units of the third set belong to a third slice, the third slice being different from the first slice and following the first slice, the data units of the first set and the data units of the third set belong to a same sample, and the data units of the third set are concatenated after the data units of the first set and the data units of the second set in the generated bitstream.

11. Obtaining the first set of at least one first data unit includes: obtaining a data unit for each slice description from the media file; 11. A method according to claim 9 or 10, comprising retrieving the at least one first data unit from the first track as a function of the retrieved data unit per slice description.

12. The method of claim 11 , wherein the data units for each slice description are obtained as a function of reference of a sample comprising a plurality of data units, and the first set of data units are obtained from the plurality of data units.

13. The method of claim 11 , wherein the data unit per slice description provides a number of slices and, for each slice, provides a list of a number of data units per track associated with the slice.

14. The method of claim 13 , wherein the list has a length equal to the number of track references from the geometry track to the attribute track that contains the sample group plus one.

15. The method of claim 13 , wherein the list is ordered starting with the number of the data units for the geometry track.

16. 16. The method of any one of claims 8 to 15, wherein the media file complies with the ISOBMF format, the different tracks comprise a geometry track and at least one attribute track, and the data units comprise a geometry data unit and an attribute data unit.

17. A signal carrying an information dataset for media data, said information dataset comprising encoded media data encapsulated according to the method of claims 1 to 6.

18. 17. A computer program product for a programmable device, the computer program product comprising a series of instructions for implementing each of the steps of the method according to any one of claims 1 to 16, when the computer program product is loaded into and executed by the programmable device.

19. A computer readable storage medium having stored thereon computer program instructions for implementing each of the steps of the method according to any one of claims 1 to 16.

20. 17. An apparatus comprising a processing unit configured to implement each of the steps of the method according to any one of claims 1 to 16.

Images