Method, apparatus and computer program for improving multi-track encapsulation of point cloud data
By selecting tracks according to data unit type in a multi-track media file and describing the strip structure using additional metadata, the problem of low efficiency and interleaving of point cloud data packaging is solved, and efficient point cloud frame bitstream encapsulation and parsing is achieved, and correct support is supported. decoding and rendering.
Patent Information
- Application Number
- CN202380031576.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-05
- Filing Date
- 2023-04-03
- Publication Date
- 2024-11-12
AI Technical Summary
The existing point cloud data packaging technology is inefficient in multi-track media files, especially for multi-track packaging of striped point cloud frames, which affects the correct decoding and rendering of the parser.
Ensure the correct sorting and parsing of data units by selecting appropriate tracks in the media file for encapsulation based on the type of data units and using additional metadata to provide relative position information and structural description of the stripes.
It realizes efficiently encapsulating and parsing the bitstream of striped point cloud frames in multi-track media files, ensuring that the parser generates appropriately sorted data units, and supports bitstream decoding and rendering that complies with MPEG-I standards.
Smart Images

Figure CN118946906A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to packaging data, particularly point cloud data, in a standard and interoperable format to store or transmit slice-based point cloud frames of 3D points, for example as a collection of tracks. Background Art
[0002] The Moving Picture Experts Group (MPEG) is standardizing the compression and storage of point cloud data (also called volumetric media data) information. Point cloud information consists of a collection of 3D points with associated attribute information (such as color, reflectivity, frame index, etc.).
[0003] In the first aspect, MPEG-I Part 9 (ISO / IEC 23090-9) specifies Geometry-based Point Cloud Compression (G-PCC) and specifies a bitstream syntax for point cloud information. According to MPEG-I Part 9, a point cloud is an unordered list of points that includes geometric information, optional attributes, and associated metadata. The geometric information describes the position of the points in a three-dimensional Cartesian coordinate system. Attributes are typed properties of each point, such as color or reflectivity. Metadata are information items used to interpret the geometric information and attributes. The G-PCC compression specification (MPEG-I Part 9) defines specific attributes such as a frame index attribute or a frame number attribute using reserved attribute tag values (3 indicates a frame index, and 4 indicates a frame number attribute) (recall that according to MPEG-I Part 9, a point cloud frame is a collection of points at a specific time instance). A point cloud frame can be divided into one or more ordered subframes, tiles, or strips. A subframe is a partial representation of a point cloud frame consisting of points with the same frame number or frame index attribute value. For example, a subframe may be a collection of points within a point cloud frame having the attribute of sharing a common acquisition, capture or rendering time. As another example, a subframe may be a collection of points within a point cloud frame having the attribute of being continuously acquired or captured during a given time range or should be rendered within a given time range. As another example, a subframe may be a collection of points within a point cloud frame having the attribute of being acquired according to a laser emission direction or corresponding to a portion of a scanning path of a 3D sensor. Still in MPEG-I Part 9, it is possible to use a frame boundary marker data unit or parameters in some data unit headers (frame_ctr_lsb syntax element) to indicate a point cloud frame using the FrameCtr variable.
[0004] To recap, a tile is a collection of slices identified by a common slice_tag syntax element value whose geometry is contained within a bounding box that can be specified in a tile directory data unit. Each tile consists of a single bounding box and an identifier (tileId). Tile information is not used by the decoding process specified in ISO / IEC 23090-9. A slice corresponds to the geometry and attributes of a portion of a coded point cloud frame or the entire coded point cloud frame. Each slice shall consist of at least one geometry data unit (GDU) encoding the slice geometry and an attribute data unit (ADU) or a default attribute data unit (DU) encoding the slice attributes. A slice is identified by the GDU slice_id. ISO / EC 23090-9 specifies the slice decoding process as a four-step process:
[0005] -decode the points within the strip,
[0006] - decode the default attributes (if present),
[0007] - decode the point attributes (if present), and
[0008] -Offset and scale the position of a point.
[0009] In a second aspect, MPEG-I Part 18 (ISO / IEC 23090-18) specifies a media format that enables the storage and delivery of geometry-based point cloud compressed data. It also supports flexible extraction of geometry-based point cloud compressed data when delivering and / or decoding time. According to MPEG-I Part 18, a point cloud frame is encapsulated in one or more G-PCC tracks, and samples in a G-PCC track correspond to a single point cloud frame. Each sample includes one or more G-PCC units belonging to the same presentation time. The G-PCC unit is a type-length-value (TLV) encapsulation structure that includes at least one of a sequence parameter set (SPS), a geometry parameter set (GPS), an attribute parameter set (APS), a block list, a frame boundary marker, a geometry data unit (GDU), an attribute data unit (ADU), a default attribute data unit, a frame specific attribute property (FSAP) data unit, and a user data data unit. The syntax of the TLV encapsulation structure is defined in Annex B of ISO / IEC 23090-9.
[0010] While the ISO base media file format has proven to be effective for packaging point cloud data, there is a need to improve the packaging efficiency, in particular to improve multi-track packaging of strip-based point cloud frames (i.e., point cloud frames comprising at least one strip) of 3D points. Summary of the invention
[0011] The present disclosure has been designed to address one or more of the foregoing concerns.
[0012] In this context, a scheme for improving the encapsulation of point cloud data is provided.
[0013] According to a first aspect of the present disclosure, a method for encapsulating a bitstream in a media file including different tracks is provided, wherein the bitstream includes point cloud data, the point cloud data includes a slice-based point cloud frame, and the slice of the point cloud frame includes different types of data units, the method comprising:
[0014] Obtain a first data unit of a first stripe of a point cloud frame;
[0015] For each first data unit in the obtained first data units, encapsulate the first data unit in a track of the media file, where the track is selected according to a type of the first data unit;
[0016] Obtain a second data unit of a second stripe of the one point cloud frame;
[0017] For each of the obtained second data units, encapsulate the second data unit in a track of the media file, where the track is selected according to a type of the second data unit;
[0018] obtaining at least one item of information characterizing a relative position of the first data unit with respect to the second data unit in a track of the media file; and
[0019] The obtained at least one information item is encapsulated in the media file.
[0020] Therefore, the method of the present disclosure makes it possible to describe the packed data units of a bitstream comprising a strip-based point cloud data frame, thereby enabling a parser to generate a bitstream with properly ordered data units.
[0021] According to some embodiments, at least one of the obtained at least one information item is encapsulated in a geometry track of the media file as a sample group.
[0022] Also in accordance with some embodiments, at least one of the obtained at least one information item is encapsulated in a respective one of different tracks of the media file.
[0023] Also in accordance with some embodiments, at least one item of information in each of the tracks encapsulated in different tracks of the media file is a stripe separator.
[0024] Also in accordance with some embodiments, at least one information item in each of the different tracks of the media file encapsulated comprises a description of the structure of data units within the sample, each data unit being associated with slice information.
[0025] Also in accordance with some embodiments, at least one information item in each of the different tracks encapsulated in the media file comprises a description of the structure of the strips within the point cloud frame.
[0026] Also in accordance with some embodiments, the media file conforms to the ISOBMF format, and wherein the different tracks include a geometry track and at least one attribute track, and the data units include geometry data units and attribute data units.
[0027] According to a second aspect of the present disclosure, a method for parsing a media file including encapsulated point cloud data is provided, wherein the point cloud data includes a stripe-based point cloud frame, wherein the stripe of the point cloud frame includes different types of data units, and the data units are encapsulated in different tracks in the media file according to their types, and the method includes:
[0028] Obtaining data units from the track in units of stripes;
[0029] concatenating the obtained data units on a stripe basis such that all obtained data units associated with a given stripe of a given track and all obtained data units associated with stripes corresponding to the given stripe in tracks other than the given track in the different tracks are continuous; and
[0030] A bit stream including the concatenated data units is generated.
[0031] Therefore, the method of the present disclosure makes it possible to parse a stripe-based point cloud frame encapsulated in a multi-track media file to generate a bitstream with properly ordered data units. It should be noted here that the data units belonging to a stripe of a track and the data units belonging to the corresponding stripe of another track belong to the same stripe in the bitstream.
[0032] According to some embodiments, obtaining data units and concatenating the obtained data units comprises:
[0033] Obtaining a first set of at least one first data unit from a first track of the different tracks, the data units of the first set belonging to a first stripe;
[0034] obtaining a second set of at least one second data unit from a second track different from the first track among the different tracks, the data units of the second set belonging to a second stripe, the second stripe corresponding to the first stripe, and
[0035] The data units of the first set and the data units of the second set are concatenated so that the data units of the first set and the data units of the second set are continuous.
[0036] Also according to some embodiments, the method further includes obtaining a third set of at least one third data unit from the first track, the data units of the third set belonging to a third stripe, the third stripe being different from the first stripe and following the first stripe, the data units of the first set and the data units of the third set belonging to the same sample, and the data units of the third set being concatenated after the data units of the first set and the data units of the second set in the generated bitstream.
[0037] Also according to some embodiments, obtaining a first set of at least one first data unit includes:
[0038] identifying a first data unit of a first set;
[0039] comparing the information item of the identified first data unit with the information items of data units following the identified first data unit in the first track; and
[0040] As a result of the comparison, it is determined whether the data unit following the identified first data unit belongs to the first set.
[0041] Also according to some embodiments, obtaining a first set of at least one first data unit includes:
[0042] identifying data units in the first track;
[0043] obtaining a stripe identifier associated with the identified data unit;
[0044] comparing the obtained stripe identifier with the current stripe identifier; and
[0045] As a result of the comparison, it is determined whether the identified data unit belongs to the first set.
[0046] Also according to some embodiments, obtaining a first set of at least one first data unit includes:
[0047] Get the slice encapsulation description (data unit per slice description) from the media file, and
[0048] According to the obtained stripe encapsulation description (the description of the data unit of each stripe), the at least one first data unit is obtained from the first track.
[0049] Also in accordance with some embodiments, a slice encapsulation description (a description of the data units for each slice) is obtained from a reference to a sample comprising a plurality of data units from which the data units of the first set are obtained.
[0050] Also according to some embodiments, obtaining a first set of at least one first data unit includes:
[0051] identifying a first data unit of a first set in a first track;
[0052] obtaining the type of data units following the identified first data unit in the first track; and
[0053] If the obtained type corresponds to a stripe delimiter, an information item for notifying that all first data units of the first set have been obtained is provided.
[0054] Also in accordance with some embodiments, the media file conforms to the ISOBMF format, and wherein the different tracks include a geometry track and at least one attribute track, and the data units include geometry data units and attribute data units.
[0055] Also in accordance with some embodiments, the description of the data units for each stripe provides the number of stripes, and for each stripe, a listing of the number of data units for each track associated with the stripe is provided.
[0056] Also in accordance with some embodiments, the list has a length equal to the number of track references to the attribute track from the geometry track containing the sample group plus one.
[0057] Also in accordance with some embodiments, the list is ordered starting from the number of data units for the geometry track.
[0058] According to a third aspect of the present disclosure, a method for encapsulating a bitstream in a media file including different tracks is provided, wherein the bitstream includes point cloud data, the point cloud data includes a slice-based point cloud frame, and the slice of the point cloud frame includes different types of data units, the method comprising:
[0059] Obtain a first data unit of a first stripe of a point cloud frame;
[0060] For each first data unit in the obtained first data units, encapsulate the first data unit in a track of the media file, where the track is selected according to a type of the first data unit;
[0061] Obtain a second data unit of a second stripe of the one point cloud frame;
[0062] For each of the obtained second data units, encapsulate the second data unit in a track of the media file, where the track is selected according to a type of the second data unit;
[0063] obtaining at least one item of information characterizing a relative order of the first data unit with respect to the second data unit in the bitstream; and
[0064] The obtained at least one information item is encapsulated in the media file.
[0065] Therefore, the method of the present disclosure makes it possible to describe the packed data units of a bitstream comprising a strip-based point cloud data frame, thereby enabling a parser to generate a bitstream with properly ordered data units.
[0066] According to some embodiments, at least one of the obtained at least one information item is encapsulated in a geometry track of the media file as a sample group.
[0067] Also in accordance with some embodiments, the sample group provides a number of stripes and, for each stripe, provides a list of the number of data units per track associated with the stripe.
[0068] Also in accordance with some embodiments, the list has a length equal to the number of track references to the attribute track from the geometry track containing the sample group plus one.
[0069] Also in accordance with some embodiments, the list is ordered starting from the number of data units for the geometry track.
[0070] Also in accordance with some embodiments, the media file conforms to the ISOBMF format, and wherein the different tracks include a geometry track and at least one attribute track, and the data units include geometry data units and attribute data units.
[0071] According to another aspect of the present disclosure, there is provided a device including a processing unit configured to execute each step of the above method.
[0072] This aspect of the present disclosure has similar advantages as the above-described aspect.
[0073] At least a portion of the method according to the present disclosure may be computer-implemented. Thus, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or a combination of software and hardware embodiments, which may be collectively referred to herein as a "circuit," "module," or "system." In addition, the present disclosure may take the form of a computer program product embodied in any tangible medium of expression having a computer-usable program code embodied in the medium.
[0074] Since the scheme of the present disclosure can be implemented in software, the scheme of the present disclosure can be embodied as a computer readable code for providing to a programmable device on any suitable carrier medium. Tangible carrier media may include storage media such as floppy disks, CD-ROMs, hard disk drives, magnetic tape devices, or solid-state memory devices. Temporary carrier media may include signals such as electrical signals, electronic signals, optical signals, acoustic signals, magnetic signals, or electromagnetic signals, such as microwave or RF signals. BRIEF DESCRIPTION OF THE DRAWINGS
[0075] Some embodiments of the present disclosure will now be described, by way of example only, and with reference to the following drawings, in which:
[0076] Figure 1 An example of a system in which the present invention may be implemented is illustrated;
[0077] Figure 2 illustrates an example of encapsulating a bitstream in several tracks of a media file and parsing the tracks to generate a bitstream conforming to a given format such as MPEG-1 Part 9;
[0078] Figure 3 A first example of the structure of a frame in a frame sequence of a G-PCC bitstream to be packed and parsed is illustrated, wherein all slices of a point cloud frame have the same structure;
[0079] Figure 4 A second example of the structure of a frame in a frame sequence of a G-PCC bitstream to be packed and parsed is illustrated, wherein all slices of a point cloud frame do not have the same structure;
[0080] Figure 5 An example of steps of a packing process according to some embodiments of the present invention is illustrated, thereby enabling point cloud data to be packed into multiple tracks in an interoperable manner for a parser to generate a conforming G-PCC bitstream;
[0081] Figure 6 illustrates an example of steps of a parsing process according to some embodiments of the present invention;
[0082] Figure 7 An example of adding static or semi-static indications in a multi-track media file encapsulating slice-based point cloud frames to help a parser reconstruct a bitstream conforming to a predetermined format (e.g., reconstructing a G-PCC bitstream) is illustrated;
[0083] Figure 8 An example of adding dynamic indications in a multi-track media file encapsulating slice-based point cloud frames to help a parser reconstruct a bitstream conforming to a predetermined format (e.g., reconstructing a G-PCC bitstream) is illustrated;
[0084] Fig. 9 An example of adding a slice delimiter in a multi-track media file encapsulating slice-based point cloud frames to help a parser reconstruct a bitstream conforming to a predetermined format (e.g., reconstructing a G-PCC bitstream); and
[0085] Fig.10 A processing device configured to implement at least one embodiment of the present disclosure is schematically illustrated. DETAILED DESCRIPTION
[0086] The inventors have observed that data units of a strip-based point cloud frame of a bitstream should be packed according to certain constraints or with additional metadata to enable a parser to generate a usable bitstream conforming to a given format (that is, a bitstream comprising properly ordered data units).
[0087] According to some embodiments of the present disclosure, data units of slice-based point cloud frames of a bitstream are encapsulated in a multi-track media track in such a manner that a parser can parse the encapsulated data units to provide a bitstream conforming to a predetermined format (e.g., a bitstream conforming to MPEG-I Part 9).
[0088] Also according to some embodiments of the present disclosure, additional metadata is provided in a track of a media file (e.g., in a G-PCC track) to enable identification of data units of a given slice relative to data units of another slice. Such additional metadata may include an indication of the end of a slice within a track or a description of the organization of data units within a slice, such organization being static or dynamic.
[0089] Figure 1 An example of a system in which the present invention may be implemented is illustrated. More specifically, the present invention may be used in a media file writer, such as media file writer 100, or in a media player, such as media player 130, or in both.
[0090] As shown, the media file writer 100 takes point cloud data (or volume data) such as point cloud data 150 as input. Figure 2 As described, point cloud data 150 may be obtained from a 3D sensor. The point cloud data may be received as uncompressed raw data or as a compressed bitstream (eg, a compressed bitstream compliant with the MPEG-I Part 9 standard). The media file writer 100 includes a packaging module 102.
[0091] The media file writer 100 may be connected to the communication network 120 via a network interface (not shown), and the media player (or reader) 130 including the decapsulation module 132 may also be connected to the communication network 120 via a network interface (not shown).
[0092] The media file writer 100 may be used to serve media files, for example, using a protocol for dynamic adaptive streaming over HTTP, such as DASH (Dynamic Adaptive Streaming over HTTP) or HTTP real-time streaming. These protocols require a streaming manifest, such as a media presentation description (MPD), or a playlist. When used to stream encapsulated media content, the media file writer 100 may include a manifest generation module, such as the manifest generation module 104. The media file writer 100 may also include a compression module, such as the compression module 106, to compress the input point cloud data into a compressed bitstream, for example, using a point cloud compression algorithm, such as the point cloud compression algorithm described in MPEG-I Part 9.
[0093] For interoperability purposes, the packaging module 102 may package the received point cloud data according to a format based on ISOBMFF (e.g., MPEG-I Part 18) to generate a media file (e.g., media file 152), which may be stored for later use by a player or image analysis tool, or may be transmitted to a media player or streaming client. Figure 5 The packaging process performed in the packaging module 102 is further described.
[0094] The media file writer 100 may be controlled and parameterized by a user, for example, via a graphical user interface, or by an application (e.g., via application code or scripts). To process compressed point cloud data, for example, to process a bitstream of compressed point cloud data compliant with MPEG-I Part 9, the encapsulation module 102 may include a G-PCC unit parser that may read a header of a G-PCC unit, for example, to determine the length of the data corresponding to the unit (e.g., in bytes, such as a tlv_num_payload_bytes syntax element) or the unit type (e.g., a tlv_type syntax element). The G-PCC unit parser may also be able to parse some header information of the G-PCC unit (e.g., such as an attribute header (to obtain its type)), and may also be able to parse parameter sets to obtain general information about the bitstream. To process uncompressed point cloud data (e.g., data obtained directly from a 3D sensor), the encapsulation module 102 may include a point cloud data parser (e.g., a .ply or .pcd file parser) that may be able to read point positions and their attributes directly from the captured raw data. The media writer may be embedded in a recording device, in a multi-sensor camera device, on a vehicle embedded with a 3D sensor, or be part of a software tool in a studio that acquires volumetric data.
[0095] The media file 152 may consist of a single media file or a collection of media segment files (e.g., as ISOBMFF segments) (ISO base media files containing one or more segments). The media file may be, for example, a fragmented file for real-time acquisition or capture and packaging or real-time (or low-latency) streaming. The media file may conform to the ISOBMFF standard or a standard specification derived from the ISOBMFF (e.g., MPEG-I Part 18).
[0096] The media player (or reader) 130 may be a streaming client for requesting media files such as the media file 152 and for adapting transmission parameters, the streaming characteristics being handled by a streaming module such as the streaming module 134 (which implements, for example, a DASH or HLS client). The media player may also implement an RTP or RTC web client, particularly for real-time transmissions that may experience transmission losses. The media player 130 may also include a decompression module 136 that takes as input a bitstream representing compressed point cloud data (e.g., a bitstream compliant with MPEG-I Part 9) and generates point cloud data (or volume data) for rendering or analysis. The media file 152 may be read from a storage location or streamed using the streaming module 134. The data may be read at once, or in blocks, segments, or fragments, and provided to the decapsulation module (or parsing module) 132.
[0097] The decapsulation module (or parsing module) 132 then extracts the encapsulated point cloud data or a subset of the encapsulated point cloud data based on the player configuration, or based on a selection from the user, or based on parameters of an application using the media player. The extracted point cloud data may produce a bitstream (such as a bitstream compliant with MPEG-I Part 9). In this case, the bitstream is provided to a decompression module (an external decompression module or an internal decompression module (e.g., internal decompression module 136)) for reconstructing the point cloud data 154 for use by a user or application (e.g., visualization or analysis). Reference Figure 6 The parsing process is further described.The media player may be embedded in a display device (eg, a smartphone, tablet, PC, vehicle with multimedia display, etc., or a software tool in a studio for volume data generation).
[0098] Figure 2An example of encapsulating a bitstream in several tracks of a media file and parsing the tracks to generate a bitstream conforming to a given format such as MPEG-1 Part 9 is illustrated. The bitstream to be encapsulated may be a G-PCC bitstream, and the encapsulated media file may be an ISOBMFF media file, the tracks of which are, for example, ISO / IEC 23090-18 "gpc1" and / or "gpcg" tracks. Note that the media data of the bitstream to be encapsulated may be tile media data, possibly encapsulated as "gpcb" tile base tracks referencing G-PCC tile tracks "gpt1" each carrying one G-PCC component (geometry or attribute).
[0099] As shown, the G-PCC bitstream 200 includes a sequence of point cloud frames, particularly frame N (labeled 205), where the encoded point cloud frame includes a sequence of zero or more slices that share the same value of FrameCtr (a timestamp). For example, point cloud frame N includes two slices labeled 210-1 and 210-2. Each slice includes a geometry data unit that encodes the geometry (which may be followed by any optional replicated geometry data units) and may include one or more attribute data units or one or more default attribute data units (DU) that encode the attributes of the slice. The number of data units may vary from one frame to another (e.g., different numbers of slices, new parameter sets, FSAP data units, etc.). For illustration, slice 210-1 includes one geometry data unit 215-1 and two attribute data units 220-11 and 220-12. Furthermore, the stripe 210 - 2 includes one geometry data unit 215 - 2 and two attribute data units 220 - 21 and 220 - 22 .
[0100] The slice_id of the GDU is used to identify the slice. The slice can be repeated within the coded point cloud frame, but its repetition should not change the value of slice_id. Slice division can be used to allow parallelization to improve coding efficiency and / or implement other functions such as error resilience and progressive decoding.
[0101] During encapsulation of the G-PCC bitstream 200 in the media file 230, all geometry information items of the G-PCC bitstream may be stored and described in a geometry track, and individual attributes of the G-PCC bitstream may be stored and described in corresponding attribute tracks. For example, geometry data units may be stored and described in geometry track 235-1, attribute data units of a first attribute may be stored and described in attribute track 235-2, and attribute data units of a second attribute may be stored and described in attribute track 235-3. It is observed that tracks belonging to the same G-PCC sequence or bitstream are preferably time-aligned (i.e., each track includes samples, each of which includes a sample time for storing a time value (e.g., in a "stts" or "ctts" box)) such that there is one sample of each track associated with a given time value. For example, samples 245-1, 245-2, and 245-3 are time-aligned samples. Therefore, each point cloud frame (e.g., point cloud frame 205) can be described as a set of samples in ISOBMFF (e.g., sample set 240), which includes a sample in each track, and each sample in the same track represents the same kind of information or the same component (geometric information or specific attribute information). For example, sample 245-1 of track 235-1 includes geometric information, sample 245-2 of track 235-2 includes attribute information of a first attribute, and sample 245-3 of track 235-3 includes attribute information of a second attribute.
[0102] According to the ISO / IEC 23090-18 specification on multi-track encapsulation, a media file ends up with samples in different tracks. Each sample in a given track contains data units of a given type from different stripes in the initial bitstream 200. For example, stripes 210-1 and 210-2 have their geometry data units and geometry related information (e.g., GPS or block directory) in track 235-1 (i.e., GDU 215-1 and 215-2), while the same stripes have their attribute data units and attribute related information (e.g., APS or frame specific attribute data units) in attribute tracks 235-2 (i.e., ADU 220-11 and 220-21) and 235-3 (i.e., ADU 220-12 and 220-22). It should be noted that the representation of the media file 230 is a simplified view of ISOBMFF. It should be understood that the track and sample descriptions are available under the "moov" box and the "trak" box index data (GDU or ADU). Furthermore, it should be understood that the data is in a media data box (e.g., an "mdat", "imda" or "idat" box). When the media file is a fragmented ISO base media file, the description of the track fragments and the samples of the fragments are in the "moof" and "traf" boxes, where the associated "mdat" or "imda" boxes of these boxes store the data of the samples. It should also be noted that each sample includes data units from different slices. The inventors have observed that there are boundaries in each sample, which can be implicit or can be signaled, so that data units from one slice can be distinguished from data units from another slice (as shown by the dotted line 250).
[0103] It is observed here that the bitstream reconstructed by the parser should satisfy some rules to ensure that the decoder is able to decode and / or render the point cloud data. Examples of such rules related to TLV unit order, to TLV constraints and / or to TLV stripe constraints are as follows:
[0104] - Each stripe shall contain a single GDU followed by any optional replicated GDUs and then any ADUs or default attribute DUs,
[0105] - DUs belonging to different slices should not be interleaved, and
[0106] - For each attribute enumerated in the SPS, each stripe shall have a corresponding ADU or default attribute data unit.
[0107] In order to generate a valid G-PCC bitstream (e.g., G-PCC bitstream 255) from the media file 230 and from the specification ISO / IEC 23090-18, some mechanism should be used to concatenate data units stored in different tracks of the media file in the correct order to comply with the rules from ISO / IEC 23090-9, according to which data units belonging to different slices should not be interleaved. In order to ensure that any parser generates a conforming bitstream for a G-PCC decoder (at least from a slice ordering point of view) independently of its implementation, one of the following mechanisms may be used:
[0108] - implement a reconstruction mechanism (parser side) that takes into account the reconstruction rules of the stripes,
[0109] - an encapsulation mechanism that implements constraints on multi-track encapsulation (writer side, used by the parser side),
[0110] - Provides a signaling mechanism (implemented on the writer side and used on the parser side) for indications to be used during the reconstruction process, these indications are:
[0111] ○ Static,
[0112] ○ Dynamic,
[0113] ○ Mixed (static and redefined for some specific / exceptional samples or certain fragments)
[0114] ○Shared in the geometry track,
[0115] ○ Expand within each component track (block or non-block).
[0116] The mechanism to be used may depend on the configuration of the considered G-PCC bitstream to be encapsulated and parsed, for example may depend on whether all slices of a point cloud frame have the same structure. For example, when the slice structure is stable (e.g. comprising the same number of slices across samples and the same number of TLVs for each slice), it may not be necessary to provide indications on a sample or sample group basis, but rather for all samples of a track or track fragment, i.e. by using a static approach. The mechanism to be used may also depend on the ISO / IEC 23090-18 specification level: whether the writer is constrained to respect the ordering of slices in the individual component tracks. Depending on the specification level, the parser may or may not have some priors. When no priors can be assumed, the parser may then use added indications from the media file.
[0117] Figure 3 A first example of the structure of a frame of a frame sequence of a G-PCC bitstream to be packed and parsed is illustrated, wherein all slices of a point cloud frame have the same structure.
[0118] Here we assume that two attributes are declared for the SPS of the point cloud sequence (as in Figure 2 ). Figure 3 In the example of , a single structure of the data unit within a slice is used for all slices of the frame of the G-PCC bitstream under consideration. In other words, there is a fixed TLV pattern for each slice.
[0119] like Figure 3 As shown, each stripe 305-1, 305-2, 305-3 and 305-4 of the frame 300 includes a geometry data unit (GDU), which is followed by two attribute data units (ADU) or a default attribute data unit (also represented as Def ADU) (for clarity, only one stripe is presented for the frame 300, but there may be multiple stripes, and it is assumed here that these stripes have the same TLV pattern: a GDU followed by two ADUs or default ADUs).
[0120] Although Figure 3 The structure shown in can be advantageously used for encapsulating and / or parsing a G-PCC bitstream, but the inventors have observed that it may be convenient to consider that such a structure may change from one stripe to another to cope with data loss (at the writer or parser side) or when a partial reconstruction is performed at the receiver side. For example, although the G-PCC bitstream to be encapsulated and parsed includes stripes with the same structure, it may be considered that the G-PCC bitstream to be encapsulated and parsed does not include stripes with the same structure, so that parsing can be performed appropriately even when data units are lost. This may be the case, for example, when the transmission is unreliable, such as when using RTP-based delivery for real-time or live transmission.
[0121] Figure 4A second example of the structure of a frame that illustrates the frame sequence of a G-PCC bitstream to be encapsulated and parsed, where all stripes of a point cloud frame do not have the same structure. Again, it is assumed here that two attributes are declared for the SPS of the point cloud sequence (and again, only one stripe for frame 400 is shown for simplicity, but there can be multiple stripes for the frames of the point cloud sequence). However, according to this example, a stripe can contain a number of attributes or default attribute data units different from the expected number of attribute or default attribute data units. For illustration, stripe 405-1 of frame 400 includes one geometric data unit and one attribute data unit, while stripe 405-3 includes one geometric data unit and two attribute data units. In fact, in ISO / IEC JTC1 / SC29 / WG11 MPEG m53681, it has been proposed to signal default attribute data units by adding default attribute values to the SPS attribute description to handle the omission of encoded attribute data, to avoid the need to encode (or decode) attribute data with all the same attribute values. Stripe 405-1 uses the default SPS attribute value for the second attribute, while stripe 405-5 uses another concept as a frame-specific attribute property data unit (FSAP). When the FSAP data unit contains an attribute property with type equal to 4, this means that the FSAP data unit defines the default value of a given attribute to be applied to a specific frame. Thus, the attribute data unit for the second attribute for stripe 405-5 can be omitted. For other frames, the default value or the actual encoded value can be applied.
[0122] It should be noted that there are cases where loss occurs before encapsulation or during transmission. When the bitstream is corrupted, the high-level writer can analyze the bitstream and detect the lost data units. When the writer can detect a lost attribute data unit, it can actively fill the bitstream with default attribute data units. Additionally, on the receiver or client side, when the parser detects that some attribute data units are missing, it can actively fill the default attribute data units to generate a conforming bitstream with the number of attributes declared in the SPS. This last point also applies to partial extraction: when the bitstream is encapsulated in multiple tracks (e.g., one geometric track and N attribute tracks), when the parser only extracts a subset of the attribute tracks with the geometric track (e.g., M tracks, where M < N), N - M default attribute data units can be used to fill to avoid rewriting the SPS and the declaration of the number of attributes (declaring N attributes). The attributes can be filled in any order without interleaving the data units from different stripes.
[0123] Encapsulation process
[0124] Figure 5An example of the steps of a packing process according to some embodiments of the present invention is illustrated, which enables packing of point cloud data in multiple tracks in an interoperable manner for a parser to generate a conforming G-PCC bitstream. For illustration, Figure 5 The steps shown in can be Figure 1 The encapsulation module 102 in is executed.
[0125] According to the example shown, the first step (step 500) involves initializing the reception of point cloud data (or volume data) for packaging. Next, the packaging module is configured (step 505) according to the type of data to be processed (e.g., depending on whether the point cloud data is compressed, depending on whether the point cloud data should be analyzed, depending on whether the compressed point cloud contains blocks or strips). This can be done by the user or the application. For illustration, the configuration can include selecting a single track or multi-track packaging, selecting a real-time or offline packaging, selecting a description mode (e.g., describing blocks or not describing blocks). Within the scope of the present disclosure, a multi-track packaging is provided with a block track or without a block track.
[0126] The configuration step 505 can also be used to indicate whether the same configuration parameters are applied to the entire point cloud data package (static configuration) or can be changed when the point cloud data is packaged (dynamic configuration). Figure 1 In the case of a compression module 106 in the G-PCC bitstream, the configuration step may include setting parameters of the encoder: for example, setting the slice or tile structure. The compression module may encode these configuration parameters as additional syntax elements of the G-PCC bitstream, such as in a sequence parameter set (SPS), or in a dedicated TLV type, such as a tile directory data unit for describing the bounding box of a tile. For example, the SPS may provide information about the slice dependencies for bitstream parsing and decoding (such as the entropy_continuation parameter). When the point cloud data is transmitted as a G-PCC bitstream (e.g., by Figure 1 When a bitstream (a compression module 106 in the image processing unit or a bitstream generated by an external compression module) is received by the encapsulation module, the configuration of the encapsulation module may use information from a parameter set of the bitstream or supplementary information associated with the bitstream (sometimes referred to as SEI (Supplementary Enhancement Information message)) or information provided in a user data unit. Supplementary information refers to the following coded parameters that do not force the decoding of the point cloud data but can help applications use these point cloud data by providing additional information.
[0127] With regard to the configuration of the encapsulation module, metadata structures of the media file, such as top-level boxes (e.g., "ftyp" or "styp", "moov" and its "trak" subbox, "mdat" box, and boxes for sample descriptions (e.g., "stbl" or "stsd")) are created during an initialization step (step 510). Such an initialization step may include reading parameter sets (e.g., sequence parameter sets, geometry parameter sets, and / or attribute parameter sets) from an encoded bitstream of point cloud data, or may include obtaining information about a sensor used to obtain point cloud data (e.g., the number of points, the type of attributes associated with the points (e.g., color, reflectivity, timestamp, region of interest, etc.)), or may include obtaining information about encoder settings. Note that some of the setting parameters defined in the configuration step 505 may be reflected in the track description or sample description. In addition, user data or supplementary information (when available) may be included in whole or in part in the media file. Parts of the configuration information (configuration parameters, parameter sets, user data, or supplementary information) may be collected in a metadata structure, such as a structure dedicated to sample description, such as a sample entry (e.g., "gpc1", "gpcg", "gpeb", or any 4CC of a G-PCC component track) or a configuration box (e.g., "gpcC"). The configuration information may contain information about the slice configuration in the bitstream: presence or absence of slices, optionally the number of slices for each frame, whether the number is constant, whether the slices are repeated, whether a slice setup exists, whether there are slice reordering constraints. Having this information may impose fewer constraints on the parser (e.g., if slice_reordering_constraint=0, then there is no need to check the order of slices in the reconstructed bitstream). Some of this information may be obtained from the parameter set (e.g., slice reordering constraints), and some may require bitstream analysis (such as the number of slices and the number of slices for each frame). In addition, entropy_continuation_enabled from the SPS may be used to reflect in the sample description, as this may also constrain the reconstruction process from the point of view of the order between slices. This may be determined by the encapsulation module during the initialization step 510, or may be calculated throughout encapsulation and written to a file at the end of encapsulation (the latter mode is less relevant for real-time packaging during real-time or low-latency streaming).
[0128] An example of a metadata structure providing slice configuration information may be defined as, for example, a new box that inherits from FullBox, such as SliceConfigurationInformationBox identified by the "scfg" four-character code, as shown below:
[0129]
[0130]
[0131] in,
[0132] slice_present indicates whether the sample's data contains slices. When set to 1, this parameter indicates that the sample may contain slices. When set to 0, this parameter indicates that the sample does not contain any slices.
[0133] slice_reordering_constraint has the same semantics as in ISO / IEC 23090-9,
[0134] entropy_continuation_enabled has the same semantics as in ISO / IEC 23090-9,
[0135] repeated_slices indicates whether slices may be repeated within some samples. When set to 1, this parameter indicates that the sample may contain repeated slices. When set to 0, this parameter indicates that the sample does not contain repeated slices.
[0136] same_num_slices_per_frame indicates whether all samples have the same number of slices. When set to 1, this parameter indicates that the samples have the same number of slices. When set to 0, this parameter indicates that at least two samples do not have the same number of slices.
[0137] num_slices_per_frame indicates the number of slices for each sample (when constant, otherwise absent), and
[0138] max_num_slices_per_frame indicates the maximum number of slices of samples for a track.
[0139] This box may be stored in any G-PCC track as an optional box of a sample entry. Alternatively, the parameters present in the slice configuration information may be stored in a GPCDecoderConfigurationRecord within a 'gpcC' box.
[0140] After the initialization step 510, the writer may optionally determine (step 515) whether an indication of a reconstruction rule is required, and if so, whether the indication should be static (or implicit, i.e., applied to all samples) or dynamic (or explicit, i.e., one rule per sample, per group of samples, or per fragment or segment). By default, or at least for real-time packaging, the writer may consider explicit reconstruction rules, because this rule allows handling most configurations, such as reference Figure 3 and Figure 4 Describes the configuration.
[0141] The following steps describe encapsulation with explicit reconstruction rules or instructions that facilitate reconstruction of a valid bitstream from multiple tracks.
[0142] Next, the packaging process enters a processing loop for processing each frame of point cloud data.
[0143] After reading the point cloud frame from the bitstream of compressed point cloud data, for example, data unit by data unit (step 520), the encapsulation module determines whether the read data corresponds to geometric data (test 525). If it is determined that the read data is geometric data, the read data unit is stored in the geometry track (step 530). In the case of explicit reconstruction rules, the slice_id and optional slice_tag of the GDU header are decoded and stored. The size of the read data unit can also be stored together with the slice_id and optional prev_slice_id (an identifier of the stripe that the current stripe may depend on). When the SPS indicates that entropy_continuation is active, the writer can also decode the value of the slice_entropy_continuation parameter, and when the parameter is set to 1, decode and store the value of the prev_slice_id parameter (this is to ensure the correct inter-slice order).
[0144] When a read geometry data unit is detected as a duplicate GDU (e.g., by computing a difference on the data unit, or by checking whether the data units have the same slice_id), depending on the encapsulation mode, the writer may store the duplicate GDU during step 530 or discard the duplicate GDU and store only one instance of the duplicate GDU. The encapsulation choice (presence or absence of duplicate GDUs) may be indicated as a track property or in the sample description of the geometry track (e.g., in a sample entry or in a configuration box, such as in GPCCConfigurationBox or GPCCComponentInfoBox).
[0145] Determining the type of the read data unit may include determining the type of the G-PCC unit (or TLV unit) by reading the type of the G-PCC unit (or TLV unit) using a G-PCC unit parser (not shown).
[0146] Returning to test 525, if it is determined that the read data is not geometry data, another test is performed to determine whether the read data corresponds to attribute data (test 535). If the read data corresponds to attribute data, the type of attribute data can be obtained by the G-PCC unit parser (step 540) to determine the track in which the read data unit is to be stored (step 545). The writer preferably stores the size of the read attribute data unit and can decode its adu_slice_id (i.e., the identifier of the stripe on which the ADU depends). Recall that for a G-PCC bitstream, the attribute data unit corresponds to a TLV unit of type ADU, an APS (attribute parameter set), a default attribute data unit, or a FSAP data unit (frame specific attribute property). If the read data does not correspond to attribute data (step 535), for example, if the read data corresponds to an SPS, a frame boundary marker, or user data, the read data unit is stored in the geometry track (step 530).
[0147] When a data unit is stored in a track (step 550), the writer stores information about the data unit and associated stripe information. Figure 7 , Figure 8 and Fig. 9 Describes the metadata structure and type of information storing the information item. As shown, the process iterates over data units within a frame (test 555) and within a frame of a point cloud sequence (test 560) for encapsulation.
[0148] Disposing of replicated data units
[0149] It is to be noted that the indication to keep or discard duplicated data units (geometry data units or attribute data units or both) may also be applied to single tracks or tile tracks without component tracks (still with an indication at the track level or in the sample description such as a sample entry or a configuration box, e.g. in a GPCCConfigurationBox or a GPCCComponentInfoBox when present). The indication may be indicated by a parameter, e.g. on 2 bits, where the first bit when set to 1 indicates the presence of a duplicated GDU and the second bit when set to 1 indicates the presence of a duplicated ADU. When stored in the GPCCComponentInfoBox this may be indicated with some reserved flag value of the box. This informs the parser that for the bitstream reconstruction a choice regarding these duplicated data units is also possible: the parser may keep or discard these duplicated data units in the reconstructed bitstream. Duplicate data units may appear when duplicated stripes are present. The same indication as above may also apply to duplicated stripes: the third bit when set to 1 may indicate that the duplicated stripe has been removed from the media file (writer) or from the reconstructed bitstream (parser). Again, the parser can decide to keep or discard duplicate stripes. The values of these parameters can be set during the configuration step (step 505) of the encapsulation module, or can be determined by the encapsulation module itself during that configuration. For example, when the writer (or server) knows that the transmission is reliable (e.g., over HTTP), the duplicated data units are not kept, and an indication to discard the duplicated data units is set in at least one encapsulated track. Alternatively, when the writer (or server) knows that the transmission may be unreliable (e.g., using RTP), the duplicated data units can be retained or the duplicated stripes can be retained. When the indication to retain duplicated units is set, the writer can duplicate the APS in the attribute track (rather than encapsulating it in only one of the attribute tracks). This provides some robustness in the event of a loss, and also provides robustness in reconstructing from a subset of the tracks compared to the complete set of tracks.
[0150] Parsing
[0151] Figure 6 Examples of the steps of the parsing process according to some embodiments of the present invention are illustrated. For illustration, these steps can be represented by Figure 1 The decapsulation module 132 in the processor is executed, and the decapsulation module 132 may also be referred to as a parser or a reader. Figure 6 The parsing process above allows the parser to reconstruct a compliant G-PCC bitstream from a multi-track encapsulation (chunked or non-chunked) by following the TLV order and rules for slices defined in ISO / IEC 23090-9.
[0152] As shown, the first step involves receiving a media file for parsing (step 600). A streaming module (e.g. Figure 1 The media file may be streamed using the streaming module 134 in the media player, or the media file may be read from a storage location. Depending on the application, streaming may use a reliable or unreliable (possibly with transmission loss) protocol. Next, the parser is initialized (step 605). Initialization may be performed by parsing the boxes of the media file (e.g., the top-level "moov" box, the "trak" box, and the sample description box). When a media player includes a decoder (or decompression module, such as Figure 1 The decoder may also be initialized during this step, for example, using decoder configuration information (e.g., G-PCC configuration box "gpcC") from the sample description, when the decoder is initialized during this step. During this step, the parser may obtain information about the media file (e.g., in the media file) provided by the writer. Figure 5 Initialization of the parser may include reading the instructions written by the encapsulation module (e.g., according to the instructions in step 510 of FIG. 1 ). Figure 7 , Figure 8 and Fig. 9 One of the related embodiments.) As described with reference to the writer, the indication may be implicit (or static) or may be dynamic (or explicit). Figure 6 The steps shown in focus on the case where the indication is explicit and dynamic. This means that the parser may need to check the indication on each sample, sample group, animation clip or media segment. In the case where the indication is static, the indication can be read once for all during step 610.
[0153] Next, the reading of the stripe is started (step 610). This step includes, for example, updating a stripe counter or stripe index starting from a value of 1, or from the value of the gdu_slice_id decoded from the first GDU of the current sample (which can be read at step 615). In order to locate the data unit to be read for the current sample, the parser (or decapsulation module) reads the sample description from the metadata portion of the received media file to locate the corresponding media data in the media data box of the received media file. The geometry track and attribute track are also identified by checking the TrackReferenceBox. Then, reading the sample includes reading data units from the time-aligned samples in the geometry track and the associated attribute track. Reading is performed on a stripe basis to ensure that data units from different stripes are not interleaved. This is where the indication of the reconstruction rule may apply.
[0154] In step 615, the parser reads the data units for the current stripe in the geometry track. To do this, the GDU data units can be read until their payload changes (to collect the copied GDUs), or a number of GDU data units based on the indication in the media file (when present) can be read. The parser then records its read position in the "mdat" box of the next stripe. Optionally, the parser can decode the gdu_slice_id or obtain the gdu_slice_id from the indication in the media file (when present) and check whether it matches the current slice_id initialized at step 610. The parser can also compare the header of the current GDU with the header of the next GDU to determine whether there is a change in the header (which means that at least the stripe identifier has changed), or the stripe change can be obtained from the indication in the media file (when present).
[0155] Next, in step 620, the parser reads data units from the first referenced track (i.e., the first attribute track referenced by the geometry track). For GDUs, the parser can collect duplicate ADUs (when present) by checking that the data unit payload has not changed or by decoding adu_slice_id (or defattr_slice_id for the default ADU) and checking that adu_slice_id matches the current slice index, or can obtain duplicate ADUs (when present) from indications in the media file (when present). The parser can also compare the header of the current ADU with the header of the next ADU to determine if there is a change in the header (which means that at least the slice identifier has changed), or can obtain the slice change from indications in the media file (when present).
[0156] Next, in step 625, the parser checks whether the geometry track references other attribute tracks. If there is at least one other attribute track referenced by the geometry track, the parser iterates over steps 620 and 625 to process all referenced attribute tracks. When the last attribute track is reached in step 625, the read data unit (corresponding to a single slice) is appended to the reconstructed bitstream (step 630). The bitstream can then be processed by the decoder (on the fly) for rendering, or the bitstream can be stored for later decoding or rendering on demand.
[0157] When there is a mismatch between the decoded slice_id and the current slice_id, this can be interpreted as an error. However, the parser can store the read data units to reorder the data units in the proper slice order later and then flush these read data to the bitstream being reconstructed.
[0158] Once the data units of a stripe have been parsed, the reader checks whether the end of the sample has been reached (step 635). This can be determined based on the payload bytes indicated in the individual G-PCC units. If the end of the sample has not been reached, the stripe counter is incremented (step 640) and the algorithm loops over step 615 to parse the data units of the next stripe. In contrast, if the end of the sample has been reached, a test is performed to determine if there is at least one remaining sample to be processed (step 645). If there is at least one remaining sample to be processed, the algorithm loops over step 610 to begin reading the first stripe of a new sample. When the last sample of the point cloud sequence is reached, the algorithm ends.
[0159] When collecting attribute data units in step 620, the parser may check whether the expected number of attributes, i.e. the same number of attribute data units as indicated in the SPS, has been collected at the end of the loop over the attribute track. If this is not the case, the parser may proactively fill the current strip with a number of default attribute data units that reaches the expected number of attributes, for example before step 630. In order to identify missing attribute data units, the parser checks the attrIdx from the collected attribute data units and compares it with the list declared in the SPS. Missing attribute data units are created as default data units and the default value of any attribute (e.g., Exp2(attr_bitdepth_minus1[attrIdx]) as defined in ISO / IEC 23090-9) or the default value indicated in the SPS (when present) is used as the value. This particular step may depend on the settings of the parser, such as an integrity check mode that is activated or not activated (by the user, by an application, by a script or by default). The integrity check mode may also cause the parser to store corrupted media files of point cloud data as a partial file format (possibly with indications of complete segments, corrupted segments, and missing segments).
[0160] Another setting may include setting the parser to a "purge" mode. When this mode is set, for media files where duplicate data units exist (either in a dedicated metadata structure or as determined by the parser), the parser removes these duplicate data units and does not pass them to the point cloud decoder. In this mode, the parser may do the same for duplicate stripes. The two modes for duplicate data units or duplicate stripes may correspond to different configurations of the parser, or may be combined into a single "purge" mode.
[0161] In the following, several embodiments of interoperability between ISO / IEC 23090-18 writers and readers are described, such that:
[0162] - Packaging point cloud data in a constrained and standardized way, allowing the parser to unambiguously reconstruct a valid bitstream from multiple tracks, or
[0163] - Encapsulate point cloud data in a media file to indicate how to reconstruct from multiple tracks without interleaving data units from different stripes.
[0164] Constraints to be applied to multi-track packaging of strip-based point cloud frames (to achieve resolution without additional signaling)
[0165] According to this embodiment, encapsulation of a G-PCC bitstream including slice-based point cloud frames in a multi-track media file does not require adding specific signaling in the media file, but involves encapsulation constraints so that the parser can correctly parse the data units. Such constraints are applied to component tracks (or attribute tracks) (blocked or non-blocked). The description of multi-track encapsulation is further specified by the following statement:
[0166] - the stripe order in each attribute track matches the stripe order in the geometry track; and
[0167] -The strip numbers in each attribute track match the strip numbers in the geometry track.
[0168] Optionally, when entropy continuation is used or when slice_reordering constraint is indicated in the G-PCC bitstream, the inter-slice order in the geometry track and its referenced attribute tracks conforms to the entropy_continuation indication.
[0169] The parser can then rely on these constraints (at least the first constraint) to collect the appropriate number of TLVs for each slice in the various component tracks, and optionally determine the number of slices. To determine the number of TLVs for each slice from the geometry track or the attribute track, the parser can maintain the current value of slice_id when reading a data unit and verify that the data unit read matches the current slice_id (as referenced in Figure 6 Alternatively, the parser may collect ADUs or GDUs within a track until the payload changes. Even only the first byte may be used to detect a possible change in slice_id.
[0170] Reconstructing a point cloud frame from a geometry track ("gpc1" or "gpcg") referencing an attribute track ("gpc1" or "gpcg") can be done on a strip-by-strip basis from time-aligned samples by concatenating G-PCC units in the following order:
[0171] - any G-PCC units declared in the sample entry,
[0172] - any G-PCC cell from a geometry track (either from a sample entry or within a sample) with type set to SPS (value 0) or GPS (value 1),
[0173] - any G-PCC unit from the geometry track with type set to either a chunk directory data unit (value 5) or a user data data unit (value 9), when present,
[0174] For each strip,
[0175] - any G-PCC unit from the geometry track with type set to GDU (value 2), all GDUs have the same slice_id,
[0176] And for each attribute track referenced from the geometry track via the "gpca" track reference type:
[0177] - any G-PCC unit from this attribute track with type set to APS (value 3),
[0178] - any frame specific attribute property data elements (value 8) from this attribute track,
[0179] - any G-PCC unit from this attribute track with type set to Attribute Data Unit (value 4) or Default Attribute Data Unit (value 7) with the same value of adu_slice_id or defattr_slice_id,
[0180] - any G-PCC units from this attribute track with type set to User Data Unit (value 9), and
[0181] - Any G-PCC unit (when present) from the geometry track with type set to Frame Boundary Marker Data Unit (value 6) after the collected ADU for all slices.
[0182] The above packing constraints can also be applied to G-PCC tile tracks ("gpt1") referenced from the "gpcb" tile base track. In this case, for sets of geometry tracks and associated attribute tracks corresponding to the same set of one or more tiles, the strip order in the individual attribute tracks matches the strip order in the geometry track, and the strip numbers in the individual attribute tracks match the strip numbers in the geometry track. Optionally, inter-strip order can also be applied when slice_reordering_constraint is set or when entropy_continuation is set. The reconstruction is the same as above, but is applied to individual sets of geometry tracks and associated attribute tracks for different tiles or sets of tiles referenced by the "gpcb" tile base track. Parsing multi-track media files that pack strip-based point cloud frames without specific signaling
[0183] According to this embodiment and similar to the previous embodiment, encapsulating a G-PCC bitstream including slice-based point cloud frames in a multi-track media file does not require adding specific signaling in the media file. In addition, no specific behavior of the writer is expected. According to this embodiment, each time a data unit is read, the parser checks the slice identifier of the data unit to ensure that the data units are concatenated in the proper order in the reconstructed bitstream.
[0184] Reconstructing a point cloud frame from a geometry track (e.g., "gpc1" or "gpcg") that references an attribute track (e.g., "gpc1" or "gpcg") may be performed on a slice basis from time-aligned samples, with curSliceID being considered the slice identifier of the current slice (i.e., the slice being processed), which is incremented after all data units corresponding to that slice identifier have been processed. Reconstruction is performed by concatenating G-PCC units in the following order after the data units from the sample entry are appended to the bitstream:
[0185] For the first sample:
[0186] - any G-PCC cell from a geometry track (either from a sample entry or within a sample) with type set to SPS (value 0) or GPS (value 1),
[0187] - any G-PCC unit from the geometry track with type set to either a chunk directory data unit (value 5) or a user data data unit (value 9), when present,
[0188] For each strip:
[0189] - any G-PCC unit from the geometry track with type set to GDU (value 2) and with slice_id equal to curSliceID, and
[0190] For each attribute track referenced from the geometry track via the "gpca" track reference type:
[0191] - any G-PCC unit from this attribute track with type set to APS (value 3),
[0192] - any frame specific attribute property data elements (value 8) from this attribute track,
[0193] - any G-PCC unit from this attribute track with type set to Attribute Data Unit (value 4) or Default Attribute Data Unit (value 7) with adu_slice_id or defattr_slice_id equal to curSliceID,
[0194] - any G-PCC units from this attribute track with type set to User Data Unit (value 9), and
[0195] - Any G-PCC unit (when present) from the geometry track with type set to Frame Boundary Marker Data Unit (value 6) after the collected ADU for all slices.
[0196] For a G-PCC tile track carrying one G-PCC component referenced from a "gpcb" tile base track, the reconstruction will be the same as above, but applied to the respective sets of geometry tracks and associated attribute tracks of different tiles or sets of tiles referenced by the "gpcb" tile base track.
[0197] Use signaling information to pack strip-based point cloud frames into multi-track media files for improved parsing
[0198] Figure 7 An example of adding static or semi-static indications in a multi-track media file encapsulating slice-based point cloud frames to help a parser reconstruct a bitstream conforming to a predetermined format (eg, reconstructing a G-PCC bitstream) is illustrated.
[0199] Although Figure 7 A specific embodiment is illustrated in which the added indication is located in the geometry track, but it should be noted that the added indication may be split into different geometry and attribute tracks.
[0200] The added indications are based on sample grouping.
[0201] According to the example shown, the G-PCC bitstream is encapsulated in a geometry track (labeled 700) and two attribute tracks (labeled 705 and 710). As shown, samples 1 to N of the geometry track 700 include two GDUs for the first stripe (indicated by horizontal hatching) and one GDU for the second stripe (indicated by vertical hatching). Samples N+1 to M of the same geometry track have different patterns: one GDU in the first stripe (still indicated by horizontal hatching) and one GDU in the second stripe (still indicated by vertical hatching). Still for illustration, the attribute track 705 includes different data unit or TLV unit patterns for each stripe for samples 1 to N and samples N+1 to M. According to the example shown, the attribute track 710 contains a replicated ADU.
[0202] When encapsulating these data units, the writer may generate metadata structures such as metadata structures 715 and 720. Over time, these structures may be sample-to-group, and the sample group description box from ISOBMFF may be included in the sample description at the beginning of the file or in the animation clip. Metadata structure 715 contains a grouping_type equal to, for example, "tlvs" for "TLV for each stripe" to map the sample groups to the sample group entries in metadata structure 720. The sample group entries may inherit from the VolumetricVisualSampleGroupEntry defined in ISOBMFF. The sample group provides a mapping of TLVs to stripes, which may help the reader extract the exact number of TLV units from each of the geometry track and the associated attribute track.
[0203] For example, in metadata structure 720, samples 1 to N are mapped to the first entry, while samples N+1 to M are mapped to the second entry. It should be noted that when all samples share the same pattern, a default sample grouping may be used (in which case metadata structure 715 may be omitted and only metadata structure 720 may be present). In addition, when the configuration does not change over time, the sample grouping may be declared static using an appropriate flag value of metadata structure 720 defined in ISOBMFF. The content of metadata structure (or box) 720 contains a list of entries indicated by the entry_count parameter. The grouping type parameter of the box is also "tlvs" (e.g., or any dedicated 4CC) to pair the box with the corresponding box in metadata structure 715 (when present). Each entry in box 720 contains a list of patterns (e.g., one for each stripe) and indicates the number of GDUs and ADUs to be collected in different tracks. The declarations within each sample group entry follow the same declaration order as the "gpca" track reference that links the geometry track to the attribute track. It is to be noted here that, depending on the configuration or setup or application needs, the writer may have declared only one instance of a given ADU (no duplicated data units are retained when filtering mode is activated). In this case, the last value in the pattern for the first stripe in these two entries (labeled as 725 and 730) will be 1, and the last entry in the pattern for the second stripe in these two entries 725 and 730 will also be 1.
[0204] The metadata structure 725 or 730 may be defined as follows:
[0205] definition
[0206] Group Type: "tlvs"
[0207] Container: Sample Group Description Box ("sgpd")
[0208] Mandatory: None
[0209] Quantity: zero or more
[0210] The use of "tlvs" for grouping_type in a sample group indicates that the samples in the G-PCC track are assigned to the TLV-to-slice mapping information carried in the sample group. When there is a SampleGroupDescriptionBox with grouping_type equal to "tlvs", there may be a companion SampleToGroupBox with the same grouping type (when default sample grouping cannot be applied). The grouping_type_parameter in this SampleToGroupBox is undefined.
[0211] The 'tlvs' sample group can exist in a 'gpc1' or 'gpcg' geometry track, or in a 'gpt1' geometry tile track referenced by a 'gpcb' tile base track.
[0212] syntax
[0213]
[0214]
[0215] Semantics
[0216] num_slices indicates the number of slices present in the samples associated with this sample group, and
[0217] num_tlvs provides a list of the number of consecutive TLV units for each track associated with a given stripe in the samples associated with this sample group. The length of the list is equal to the number of track references to the attribute track from the geometry track containing this sample group plus 1. The list is ordered starting with the number of TLV units for the geometry track, followed by the number of TLV units for each attribute track referenced by the "gpca" track reference (in the order of these track references).
[0218] The following reconstruction rules based on metadata structure 720 or 720 plus 715 are proposed to be included in Section 7.4 of ISO / IEC 23090-18.
[0219] A point cloud frame is reconstructed from a geometry track ("gpc1" or "gpcg") that references an attribute track ("gpc1" or "gpcg") on a strip-by-strip basis from time-aligned samples by concatenating G-PCC units in the order indicated by the "tlvs" sample group of the geometry track.
[0220] Reconstruct the point cloud frame from multiple component tracks by following these steps:
[0221] - Identify its corresponding sample in the sample group description;
[0222] - Identify the mapping of the TLV to the stripe for that sample from the "tlvs" sample group description box of the geometry track; and
[0223] - for each stripe in the sample, read from the geometry track a number of TLV units corresponding to the number of TLV units indicated in the first value of the "tlvs" entry;
[0224] - For each attribute track referenced from the geometry track via the "gpca" track reference type, read a number of TLV elements from the current attribute track corresponding to the next value in the "tlvs" entry.
[0225] The following reconstruction rules are proposed for inclusion in section 7.5 of ISO / IEC 23090-18.
[0226] For a 'gpt1' geometry track referenced by a 'gpcb' tile base track, the same reconstruction as proposed for inclusion in section 7.4 of ISO / IEC 23090-18 applies to geometry tile tracks containing more than one slice and their associated attribute tile tracks (when a tile track contains only one slice, a sample-based reconstruction, i.e. the concatenation of time-aligned samples from the geometry track and its associated attribute track, may be used).
[0227] Figure 8 An example of adding dynamic indications in a multi-track media file encapsulating slice-based point cloud frames to help a parser reconstruct a bitstream conforming to a predetermined format (eg, reconstructing a G-PCC bitstream) is illustrated.
[0228] According to this embodiment, a description of a subsample-based TLV-to-strip mapping is provided. To this end, the writer can use a "subs" box from ISOBMFF (e.g., "subs" box 800) with a specific flag value to indicate that the subsamples here actually correspond to stripes (this allows a lighter description when some data units are copied). As shown in the figure numerals 800-1 to 800-3, such a mapping can be provided for each track (e.g., for the geometry track 805 and for the attribute tracks 810 and 815). As an example, the reserved flag value 0x000004 (hexadecimal) indicates that the subsamples described in the "subs" box of the G-PCC track are stripe-based subsamples of the G-PCC. For example, the value of the codec_specific_parameters field of the SubsampleInformationBox can be defined as follows:
[0229]
[0230] in,
[0231] slice_id provides the identifier of the slice identifier of the subsample, and
[0232] prev_slice_id indicates the slice identifier of the previous slice on which the subsample depends (a subsample may have no dependencies, then prev_slice_id is set to 0xFFFF (default value) to indicate that there is no previous slice on which the subsample depends). The prev_slice_id parameter is optional and may be replaced by 16 reserved bits set to the value 0. It is not used for bitstreams where entropy_continuation_enabled is indicated as inactive in the SPS, i.e., when the entropy parsing of data units of the current slice does not depend on the final entropy parsing state of data units in the previous slice.
[0233] It is also possible to reuse existing flag values, such as a value of 0 indicating a TLV-based (or G-PCC unit-based) subsample. In this case, the codec_specific_parameters field of the SubsampleInformationBox may be defined as follows:
[0234]
[0235] Where slice_id provides an identifier of the subsample (TLV or G-PCC unit). Optionally, for the GDU TLV (when payloadType=2), the identifier of the previous slice is provided in the delta_prev_slice_id parameter, which is encoded as slice_id minus prev_slice_id to be represented on the 8 remaining bits of codec_specific_parameters. Note that the subsamples may not have dependencies, then delta_prev_slice_id is set to 0xFFFF (default value) to indicate that there is no previous slice on which the subsample depends. In a variant where no possible identifier for the previous slice is indicated, the 8 remaining bits are simply reserved and set to 0.
[0236] The slice identifier may be provided in the codec_specific_parameter, for example on 16 bits. Optionally, a previous_slice_id may be present (possibly encoded as an increment of slice_id on 8 bits when entropy continuation is activated, if other parameters require bits). In some variants, in particular for flag values of subs dedicated to slices, the slice identifier may even be omitted and codec_specific_parameters may be equal to 0. The parser should then infer that the corresponding subsample size corresponds to one slice and that the next subsample size will correspond to the next slice (which does guarantee increasing order or compliance with entropy continuation). This is why "subs" with a dedicated flag value may be safer. Using the value 0 again would also require changing the properties of the component tracks to allow component tracks to contain "subs" boxes with the flag value set to 0, which is not allowed in the current version (October 13, 2021) of the ISO / IEC 23090-18 specification. To avoid issues regarding flag values, the "subs" box 800 can be made optional in the GPCCComponentInfoBox. The metadata structure 800 can be written by the writer in Figure 5 Generated during step 545 in .
[0237] Since these added indications can be specified for many samples in a sequence, a writer can use an optimized version of the "subs" box if available, or a shrunken or reduced version (perhaps with a specific 4cc indication). Then, when a parser wants to use the subs information, it must first de-shrunken or decompress that version of the "subs" box to obtain the actual subsample size and offset in the media data portion.
[0238] Fig. 9 An example of adding a slice delimiter in a multi-track media file encapsulating slice-based point cloud frames to help a parser reconstruct a bitstream conforming to a predetermined format (eg, reconstructing a G-PCC bitstream) is illustrated.
[0239] According to this embodiment, special TLV units are added as delimiters within the media data portion of the file (e.g. in the "mdat" box). Such delimiters may be added by the writer in the various tracks (i.e., in the geometry track and the attribute track) when the writer detects that a stripe has been read (this may be detected by comparing the first byte of data units of the same type, or by decoding the first 3 bytes of data units that may contain a stripe identifier (e.g., GDU, ADU) or a default attribute data unit). Fig. 9As shown, new TLV unit 905 is located between GDUs of two different stripes within geometry track 900. In addition, new TLV units 910 and 915 are located between ADUs of two different stripes within attribute tracks 920 and 925, respectively. The bytes of these delimiter TLVs may be taken into account when calculating sample sizes and sample or block offsets in the media portion of the file.
[0240] The parser uses these TLV units to identify data units from another stripe relative to data units from a stripe so that the data units can be ordered correctly. When the parser encounters such TLV units in the media data portion of the media file, these TLV units should be interpreted and then discarded. In other words, the parser should not include these delimiter TLVs in the reconstructed bitstream. It can be observed that such a TLV unit may not be needed after the last stripe of samples (because the parser can detect the end of the sample as the end of the last stripe).
[0241] According to a specific embodiment, this specific TLV unit contains only the TLV type and an empty payload:
[0242]
[0243] In a variation, the stripe identifier of the previous data unit may be provided as the payload of the stripe delimiter TLV. Then tlv_num_payload_bytes is set to 2 (bytes) and followed by the value of the stripe identifier. When entropy continuation is activated, prev_slice_id may also be provided, allowing the parser to perform a deeper inspection (without decoding the GDU or ADU header). In this case, tlv_num_payload_bytes may be set to 4 (bytes) and followed by the value of the stripe identifier on 2 bytes, and the value of prev_slice_id on 2 bytes.
[0244] This specific TLV unit may be defined and used only at the system level between the writer and the parser.A G-PCC encoder may not need to generate such a delimiter, and a G-PCC decoder may not understand such a delimiter.
[0245] Optionally, there may also be delimiters for slice sets and for repeated slices to easily allow the parser to not include them in the reconstructed bitstream. These delimiters may be defined like slice delimiters (with a specific tlv_type and no payload). Furthermore, there may be delimiters between slices with the same slice_tag value. The parser may then check if the data for a given slice tag is provided to the decoder as a continuous byte range.
[0246] Strip-based point cloud frame of blocks
[0247] It is observed that handling a tiled strip-based point cloud frame is similar to handling a non-tile-based point cloud frame, with the processing described above being applied to each tile. However, it is observed here that there are some configurations for the tile track where the geometry track and the attribute track are split so that some tiles are organized according to the processing described above and some tiles are not organized according to the processing (for example, if the tiles contain only one strip).
[0248] Hardware for performing the steps of some embodiments of the present disclosure
[0249] Fig.10 1 is a schematic block diagram of a computing device 1000 for implementing one or more embodiments of the present disclosure. The computing device 1000 may be a device such as a microcomputer, a workstation, or a light portable device. The computing device 1000 includes a communication bus 1002 connected to:
[0250] - a central processing unit (CPU) 1004, such as a microprocessor;
[0251] - Random Access Memory (RAM) 1008, which is used to store executable code of the method of the embodiment of the present disclosure and registers suitable for recording variables and parameters required to implement the method for encapsulating, indexing, decapsulating and / or accessing data, and its memory capacity can be expanded by, for example, an optional RAM connected to an expansion port;
[0252] - a read-only memory (ROM) 1006 for storing a computer program for implementing the embodiments of the present disclosure;
[0253] - A network interface 1012, which in turn is typically connected to a communication network 1014, through which the digital data to be processed is transmitted or received. The network interface 1012 may be a single network interface, or may consist of a collection of different network interfaces (e.g., wired and wireless interfaces, or different kinds of wired or wireless interfaces). Data is written to the network interface for transmission, or read from the network interface for reception, under the control of a software application running in the CPU 1004;
[0254] - A user interface (UI) 1016 for receiving input from a user or displaying information to a user;
[0255] - Hard disk (HD) 1010; and / or
[0256] - An I / O module 1018 for receiving / sending data from / to an external device such as a video source or a display.
[0257] The executable code may be stored in the read-only memory 1006, on the hard disk 1010 or on a removable digital medium such as a disk. According to a variant, the executable code of the program may be received over the communication network via the network interface 1012 to be stored in one of the storage devices of the communication device 1000 such as the hard disk 1010 before execution.
[0258] The central processing unit 1004 is adapted to control and direct the execution of instructions or software code portions of one or more programs according to embodiments of the present disclosure, which instructions are stored in one of the aforementioned storage devices. After power-on, the CPU 1004 is capable of executing instructions related to software applications from the main RAM memory 1008, after loading these instructions, for example, from the program ROM 1006 or the hard disk (HD) 1010. When executed by the CPU 1004, such software applications cause the steps of the flowcharts shown in the previous figures to be performed.
[0259] In this embodiment, the device is a programmable device that uses software to implement the method of the present disclosure. However, alternatively, the method of the present disclosure may be implemented in hardware (for example, in the form of an application specific integrated circuit or ASIC).
[0260] Although the present disclosure has been described above with reference to specific embodiments, the present disclosure is not limited to the specific embodiments, and modifications within the scope of the present disclosure will be apparent to those skilled in the art.
[0261] Many further modifications and variations will occur to those skilled in the art when referring to the foregoing illustrative embodiments, which are given by way of example only and are not intended to limit the scope of the present disclosure, which is determined solely by the appended claims. In particular, different features from different embodiments may be interchanged where appropriate.
[0262] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.
Claims
1. A method for encapsulating a bitstream in a media file comprising different tracks, the bitstream comprising point cloud data, the point cloud data comprising a slice-based point cloud frame, the slices of the point cloud frame comprising different types of data units, the method comprising: Obtain a first data unit of a first stripe of a point cloud frame; For each first data unit in the obtained first data units, encapsulate the first data unit in a track of the media file, where the track is selected according to a type of the first data unit; Obtain a second data unit of a second stripe of the one point cloud frame; For each of the obtained second data units, encapsulate the second data unit in a track of the media file, where the track is selected according to a type of the second data unit; obtaining at least one item of information characterizing a relative order of the first data unit with respect to the second data unit in the bitstream; as well as The obtained at least one information item is encapsulated in the media file.
2. The method according to claim 1, wherein: At least one of the obtained at least one information item is encapsulated in a geometry track of the media file as a sample group.
3. The method according to claim 1, wherein: At least one of the obtained at least one information item is encapsulated in a respective track of different tracks of the media file.
4. The method according to claim 2, wherein: The sample group provides the number of stripes and, for each stripe, a list of the number of data units per track associated with the stripe.
5. The method according to claim 4, wherein: The list has a length equal to the number of track references to the attribute track from the geometry track containing the sample group plus one.
6. The method according to claim 4, wherein: The list is ordered starting with the number of data units for the geometry track.
7. The method according to any one of claims 1 to 6, wherein: The media file complies with the ISOBMF format, and wherein the different tracks include a geometry track and at least one attribute track, and the data units include geometry data units and attribute data units.
8. A method for parsing a media file comprising encapsulated point cloud data, wherein the point cloud data comprises a stripe-based point cloud frame, wherein the stripe of the point cloud frame comprises different types of data units, and the data units are encapsulated in different tracks in the media file according to their types, the method comprising: Obtaining data units from the track in units of stripes; concatenating the obtained data units on a stripe basis such that all obtained data units associated with a given stripe of a given track and all obtained data units associated with stripes corresponding to the given stripe in tracks other than the given track in the different tracks are continuous; and A bit stream including the concatenated data units is generated.
9. The method according to claim 8, wherein: Obtaining data units and concatenating the obtained data units includes: Obtaining a first set of at least one first data unit from a first track of the different tracks, the data units of the first set belonging to a first stripe; obtaining a second set of at least one second data unit from a second track different from the first track among the different tracks, the data units of the second set belonging to a second stripe, the second stripe corresponding to the first stripe; and The data units of the first set and the data units of the second set are concatenated so that the data units of the first set and the data units of the second set are continuous.
10. The method according to claim 8 or 9, further comprising obtaining a third set of at least one third data unit from the first track, the data units of the third set belonging to a third stripe, the third stripe being different from the first stripe and following the first stripe, the data units of the first set and the data units of the third set belonging to the same sample, and the data units of the third set being concatenated after the data units of the first set and the data units of the second set in the generated bitstream.
11. The method according to claim 9 or 10, wherein: Obtaining the first set of at least one first data unit comprises: obtaining a description of the data unit for each slice from the media file; and The at least one first data unit is obtained from the first track according to the obtained description of the data unit for each stripe.
12. The method according to claim 11, wherein: The description of the data units for each stripe is obtained from a reference to a sample comprising a plurality of data units from which the data units of the first set are obtained.
13. The method according to claim 11, wherein: The description of the data units for each stripe provides the number of stripes and, for each stripe, provides a listing of the number of data units for each track associated with the stripe.
14. The method according to claim 13, wherein: The list has a length equal to the number of track references to the attribute track from the geometry track containing the sample group plus one.
15. The method according to claim 13, wherein: The list is sorted starting from the number of data units for the geometry track.
16. The method according to any one of claims 8 to 15, wherein: The media file complies with the ISOBMF format, and wherein the different tracks include a geometry track and at least one attribute track, and the data units include geometry data units and attribute data units.
17. A signal carrying an information data set for media data, the information data set comprising encoded media data encapsulated according to the method of claims 1 to 6.
18. A computer program product for a programmable device, the computer program product comprising an instruction sequence for implementing the steps of the method according to any one of claims 1 to 16 when the instruction sequence is loaded into the programmable device and executed by the programmable device.
19. A non-transitory computer-readable storage medium storing instructions of a computer program for implementing the steps of the method according to any one of claims 1 to 16.
20. An apparatus comprising a processing unit configured to perform the steps of the method according to any one of claims 1 to 16.
