File format for point cloud data

Abstract

A method for processing point cloud data, comprising: determining a 3D region of the point cloud data and a 2D region of an atlas frame of a point cloud orbit, one or more points in the point cloud data being projected onto the 2D region of the atlas frame of the point cloud orbit; and reconstructing the 3D region of the point cloud data based on atlas frame data of the point cloud orbit included in the 2D region and video frame data of a corresponding point cloud component orbit.

Description

Technical Field

[0001] This patent application is directed to multimedia data processing and transmission technologies, as well as to methods, apparatuses, and systems for processing point cloud data. Background Art

[0002] Video coding uses compression tools to encode two-dimensional video frames into a compressed bitstream representation, which is more efficient for storage or transmission over a network. Traditional video coding techniques using two-dimensional video frames are sometimes inefficient for representing the visual information of three-dimensional visual scenes. Summary of the Invention

[0003] This patent application particularly describes technologies for encoding and decoding digital video carrying visual information related to multi-dimensional images.

[0004] In one exemplary aspect, a method for processing point cloud data is disclosed. The method includes: determining a 3D region of the point cloud data and a 2D region of a point cloud orbit atlas frame, projecting one or more points in the 3D region onto the 2D region of the point cloud orbit atlas frame, and reconstructing the 3D region of the point cloud data based on the atlas frame data included in the 2D region and the video frame data of the corresponding point cloud component orbit.

[0005] In another exemplary aspect, another method for processing point cloud data is disclosed. The method includes: determining a 3D region of the point cloud data and a group of point cloud component orbits corresponding to the 3D region, and reconstructing the 3D region of the point cloud data based on the video frame data of the point cloud component orbits from the group of point cloud component orbits and the corresponding point cloud orbit atlas frame data.

[0006] In another exemplary aspect, an apparatus for processing point cloud data is disclosed.

[0007] In yet another exemplary aspect, a computer program storage medium is disclosed. The computer program storage medium includes code stored thereon. When executed by a processor, the code causes the processor to implement the described method.

[0008] This application describes these aspects and other aspects. Brief Description of the Drawings

[0009] Figure 1 is a flowchart of an exemplary method for 3D point cloud data processing.

[0010] Figure 2 is a flowchart of an exemplary method for 3D point cloud data processing.

[0011] Figure 3 is a flowchart of an exemplary method for 3D point cloud data processing.

[0012] Figure 4It is a flowchart of an example method for 3D point cloud data processing.

[0013] Figure 5 It is a flowchart of an example method for 3D point cloud data processing.

[0014] Figure 6 It shows an example of data syntax.

[0015] Figure 7 It shows another example of data syntax.

[0016] Figure 8 It is a block diagram of an example hardware device. Detailed Description

[0017] The chapter headings are used in this application only for improving readability and do not limit the scope of the embodiments and techniques disclosed in each chapter to that chapter. Examples using the H.264 / AVC (High Efficiency Video Coding) and H.265 / HEVC and MPEG standards are used to describe certain features. However, the applicability of the disclosed techniques is not limited only to H.264 / AVC or H.265 / HEVC systems.

[0018] In this application, various syntax elements for point cloud data processing are disclosed in different chapters. However, it should be understood that, unless otherwise specified, syntax elements with the same name will have the same format and syntax as those used in different chapters. Additionally, in various embodiments, different syntax elements and structures described under different chapter headings can be combined together. Moreover, although specific structures are described as examples of embodiments, it should be understood that, unless otherwise specified in this application, the order of various entries of the syntax structure can be changed.

[0019] Video-based Point Cloud Compression (VPCC or V-PCC) represents volumetric encoding of point cloud visual information. The V-PCC bitstream containing the encoded point cloud sequence (CPCS) consists of a VPCC unit carrying sequence parameter set (SPS) data, an atlas information bitstream, a 2D video-coded occupancy map bitstream, a 2D video-coded geometry bitstream, and zero or more 2D video-coded attribute bitstreams.

[0020] Users typically have 6 degrees of freedom (DoF) to view a point cloud object. At any given point in time, each user can only see a part of the available point cloud object, depending on the user's position, viewport, field of view, etc. For many applications, the entire point cloud object data does not have to be transferred, decoded, and rendered. To support partial access and transfer of the point cloud object, it is necessary to support identifying one or more 3D spatial sub-regions to fully cover all possibilities of the DoF and orientation from which the user wishes to view the point cloud object.

[0021] Example Structure of V-PCC ISOBMFF Container

[0022] In the V-PCC elementary stream, V-PCC units are mapped to respective tracks in the ISOBMFF file based on their types. There are two types of tracks in a multi-track ISOBMFF V-PCC container: V-PCC tracks and V-PCC component tracks.

[0023] A V-PCC track is a track that carries volumetric visual information in the V-PCC bitstream, which includes an atlas information sub-bitstream and a sequence parameter set.

[0024] A V-PCC component track is a restricted video profile track that carries 2D video coding data of the occupancy map, geometry, and attribute sub-bitstreams of the V-PCC bitstream.

[0025] Based on this layout, the V-PCC ISOBMFF container shall include the following:

[0026] A V-PCC track that contains a sequence parameter set (in the sample entry) and samples that carry the payloads of sequence parameter set V-PCC units (unit type VPCC_SPS) and atlas data group V-PCC units (unit type VPCC_PDG). This track also includes track references to other tracks that carry the payloads of video-compressed V-PCC units (i.e., unit types VPCC_OVD, VPCC_GVD, and VPCC_AVD).

[0027] A restricted video profile track where the samples contain access units of the video coding elementary stream for occupancy map data (i.e., the payload of V-PCC units of type VPCC_OVD).

[0028] One or more restricted video profile tracks where the samples contain access units of the video coding elementary stream for geometry data (i.e., the payload of V-PCC units of type VPCC_GVD).

[0029] Zero or more restricted video profile tracks where the samples contain access units of the video coding elementary stream for attribute data (i.e., the payload of V-PCC units of type VPCC_AVD).

[0030] Brief Discussion

[0031] The technology disclosed in this application can be used to encode (and decode) point clouds into bitstream representations using a file format compatible with ISO BMFF, while also allowing omnidirectional video processing techniques that implement natural switching between omnidirectional videos of different viewpoints, thus enhancing the user viewing experience.

[0032] The technical solution of the embodiment of the present invention provides view group information for an omnidirectional video track, and points out that all omnidirectional video tracks belonging to the same viewpoint constitute a track group corresponding to the view. During the omnidirectional video viewpoint switching process, the consistency of the content of the omnidirectional video window before and after the viewpoint switching is ensured, and the natural switching between omnidirectional videos of different viewpoints is realized, thereby improving the user viewing experience.

[0033] Generally speaking, embodiments based on the disclosed technology can be used for video data processing. In some embodiments, omnidirectional video data is stored in a file based on the ISO (International Organization for Standardization) base media file format. Among them, ISO base media file formats such as the restricted scheme information box, track reference box, and track group box can operate with reference to the ISO / IEC JTC1 / SC29 / WG11 Moving Picture Experts Group (MPEG) MPEG-4 Part 12 ISO base media file format.

[0034] All data of the ISO base file format is installed in boxes. The ISO base file format represented by the MP4 file consists of several box groups. Each box has a type and a length and can be regarded as a data object. A box can contain another box called a container box. The MP4 file initially has only one box of the "ftyp" type as a marker of the file format and contains some information about the file. There is exactly one box of the "MOOV" type (movie box), which is a container box, and its sub-boxes contain metadata information of the media. The media data of the MP4 file is included in a media box (media data box) of the "mdat" type, which is also a container box and may or may not be available (when the media data references other files). The structure of the media data consists of metadata groups.

[0035] The timed metadata track is a mechanism in the ISO base media file format (ISOBMFF) that establishes timed metadata associated with specific samples. The timed metadata is less coupled with the media data and is usually "descriptive".

[0036] In this application, several technical solutions are provided to allow the 3D or spatial region of point cloud data (such as MPEG's V-PCC data) to be represented in a format compatible with traditional 2D video formats (such as MP4 or ISOBMFF formats). An advantageous aspect of the proposed solution is the ability to reuse traditional 2D video technologies and syntax to implement new functions.

[0037] Solution 1

[0038] As Figure 1 shown in the example flowchart of, the method includes the following steps:

[0039] Step S110 includes: determining a 3D region of the point cloud data and a 2D region of the point cloud atlas frame, and one or more points in the 3D region are projected onto the 2D region of the point cloud atlas frame.

[0040] Step S120 includes: reconstructing the 3D region of the point cloud data according to the atlas frame data in the included 2D region and the video frame data of the corresponding point cloud component track.

[0041] In some embodiments, the determination operation in S110 can be performed by using multiple 2D regions corresponding to one or more points in the 3D region.

[0042] In some embodiments, the spatial region (e.g., 3D region) of the point cloud data includes at least one of the following information or is described using at least one of the following information:

[0043] SpatialRegionInfoStruct() provides information about the spatial region, and the information of the spatial region includes the x, y, z coordinate offsets of the spatial region, the width, height, and depth of the region, and its source bounding box information.

[0044] 1) 3D bounding box parameters (3D spatial position) of the point cloud data, and the 3D bounding box parameters include: the x, y, z coordinate offsets of the point cloud data, and the width, height, and depth of the point cloud data;

[0045] 2) 3D region (or 3D block) parameters of the point cloud data, and the 3D region parameters include: the x, y, z coordinate offsets of the 3D region of the point cloud data, and the width, height, and depth of the 3D region of the point cloud data;

[0046] 3) The number of points in the point cloud data;

[0047] 4) The label of the point cloud data;

[0048] 5) The number and type of point cloud attributes.

[0049] Alternatively, the spatial region (e.g., 3D region) of the point cloud data at least includes the following information:

[0050] 1) The 3D bounding box width, height, and depth of the point cloud data, which are used to represent the size of the bounding box of the point cloud data, and the number of 3D blocks into which the 3D bounding box of the point cloud data is divided in the x, y, and z coordinate directions, and the width, height, and depth of the 3D blocks; or

[0051] 2) The 3D bounding sphere radius of the point cloud data, which specifies the radius of the bounding sphere containing the point cloud data, and the number of 3D sub - spheres into which the 3D bounding sphere of the point cloud data is divided in the row, column, and layer directions, and the width, height, and thickness of the 3D sub - spheres.

[0052] SpatialRegionInfoStruct() provides the spatial region information of the point cloud data. The spatial region information of the point cloud data includes the 3D region of the point cloud data and the 2D region of the point cloud orbit atlas frame. One or more points in the 3D region are projected onto the 2D region of the point cloud orbit atlas frame and are defined as follows:

[0053] Syntax example

[0054]

[0055]

[0056] Semantic example

[0057] number_points specifies the number of points of the point cloud data.

[0058] label specifies the label of the point cloud data.

[0059] number_attributes specifies the number of attributes of the point cloud data.

[0060] attributes_type[i] specifies the type of the i-th attribute of the point cloud data.

[0061] 3d_bounding_box_present_flag indicates whether the 3D bounding box parameters of the point cloud data exist.

[0062] 2d_bounding_box_update_flag indicates whether the 2D region of the point cloud orbit atlas frame is updated, and one or more points in the 3D region of the point cloud data are projected onto the 2D region of the point cloud orbit atlas frame.

[0063] region_id specifies the identifier of the 3D region of the point cloud data.

[0064] 3d_bounding_box_x, 3d_bounding_box_y, and 3d_bounding_box_z respectively specify the minimum values of the x, y, and z offsets of the 3D region corresponding to the 3D spatial part of the source bounding box in the Cartesian coordinate system.

[0065] delta_included_flag indicates whether 3d_bounding_box_delta_x, 3d_bounding_box_height, and 3d_bounding_box_delta_z exist in 3DRegionStruct().

[0066] The 3d_bounding_box_delta_x, 3d_bounding_box_delta_y, and 3d_bounding_box_delta_z respectively specify the maximum values of the x, y, and z offsets of a 3D region corresponding to a 3D spatial portion of the source bounding box in a Cartesian coordinate system.

[0067] The 3d_bounding_box_width, 3d_bounding_box_height, and 3d_bounding_box_depth respectively indicate the width, height, and depth of a spatial region corresponding to a 3D spatial portion of the source bounding box in a Cartesian coordinate system.

[0068] The bounding_box_top and bounding_box_left respectively specify the distances from the top and left sides of a rectangular 2D region to the top edge of the atlas frame.

[0069] The bounding_box_width and bounding_box_height respectively specify the width and height of a rectangular 2D region in the atlas frame.

[0070] The splitting of the atlas frame, tiles, and tile groups can be performed as follows.

[0071] An atlas frame from a V-PCC track can be split into tiles and tile groups.

[0072] The atlas frame is divided into one or more tile rows and one or more tile columns. A tile is a rectangular region of the atlas frame.

[0073] A tile group contains multiple tiles of the atlas frame.

[0074] In some embodiments, only rectangular tile groups are supported. In this mode, a tile group contains multiple tiles of the atlas frame, and these tiles together form a rectangular region of the atlas frame.

[0075] Therefore, the 2D region of the point cloud track atlas frame is generated at least based on the following partitioning principle: one or more points in the 3D region of the point cloud data are projected onto the 2D region of the point cloud track atlas frame.

[0076] 1. The 2D region of the point cloud track atlas frame is defined based on the atlas position on the atlas frame, and one or more points in the 3D region of the point cloud data are projected onto the 2D region of the point cloud track atlas frame.

[0077] 2. The 2D region of the point cloud atlas frame is defined based on the tile position on the atlas frame divided into tiles, and one or more points in the 3D region of the point cloud data are projected onto the 2D region of the point cloud atlas frame.

[0078] 3. The 2D region of the point cloud atlas frame is defined based on the tile group position on the atlas frame divided into tile groups, and one or more points in the 3D region of the point cloud data are projected onto the 2D region of the point cloud atlas frame.

[0079] Correspondingly, the 2D region on the point cloud atlas frame includes at least one of the following information:

[0080] 1) 2D region position parameters, including: the x and y offsets of the 2D bounding box, and the width and height of the 2D bounding box.

[0081] 2) 2D region layer index information.

[0082] 3) 2D region description information (such as intent).

[0083] 4) Point cloud atlas information (such as the number of atlases).

[0084] 5) Point cloud tile information.

[0085] 6) Point cloud tile group information.

[0086] In some embodiments, the point cloud reconstruction operation in S120 can be performed as follows.

[0087] The input of the point cloud reconstruction process is:

[0088] – The mapping information of the block to the atlas included in the 2D region,

[0089] - The decoded geometric video frame of the corresponding point cloud component track at the nominal resolution,

[0090] - The decoded attribute video frame of the corresponding point cloud component track at the nominal resolution,

[0091] - The decoded occupancy map video frame of the corresponding point cloud component track at the nominal resolution, and / or

[0092] The frame index in the output order.

[0093] In some embodiments, in order to provide visually accurate coverage of the point cloud data, the above input can be based on a magnification operation.

[0094] The output of the point cloud reconstruction process is:

[0095] - A container for saving a point list in the reconstructed point cloud frame for the 3D region, and

[0096] - The number of points in the reconstructed point cloud for the 3D region.

[0097] Example 1

[0098] As Figure 2 shown, step S210 describes determining the 3D region of the point cloud based on the point cloud track sample entry or elements in the sample. The sample entry and sample format of the point cloud track will be described below in connection with alternative embodiments.

[0099] V-PCC track

[0100] Different from the video track that stores traditional planar 2D video information, the V-PCC track is a new type of track for storing 3D volumetric visual information.

[0101] Volumetric visual track

[0102] Each volumetric visual scene is represented by a unique volumetric visual track. An ISOBMFF file may contain multiple scenes, so there may be multiple volumetric visual tracks in the file.

[0103] The volumetric visual track can be identified by the volumetric visual media processor type 'volv' in the HandlerBox of the MediaBox, which is defined in ISO / IEC 14496-12.

[0104] V-PCC track sample entry

[0105] Sample entry types: 'vpc1', 'vpcg'

[0106] Container: SampleDescriptionBox ('stsd')

[0107] Mandatory: The 'vpc1' or 'vpcg' sample entry is mandatory Number: There may be one or more sample entries

[0108] The V-PCC track shall use the sample entry VPCCSampleEntry, and the VPCCSampleEntry entry extends from VolumetricVisualSampleEntry, and its sample entry type is 'vpc1' or 'vpcg'.

[0109] The VPCC volumetric sample entry shall contain the VPCCConfigurationBox, which includes the VPCCDecoderConfigurationRecord as defined herein.

[0110]

[0111] Alternative Example 1

[0112] The 3D region of the point cloud data is associated with one or more 2D regions of the point cloud atlas frame, which may contain atlas data, 2D tile data, or tile group data, for reconstructing the 3D region of the point cloud data using VolumetricTileInfoBox() defined as follows.

[0113] Syntax example

[0114]

[0115]

[0116] Semantic example

[0117] num_regions specifies the number of 3D regions of the point cloud data.

[0118] region_intent[i] specifies the intent of region [i] of the point cloud data.

[0119] layer_index[i] specifies the layer index of region [i] of the point cloud data.

[0120] mapping_type[i] specifies the mapping type of region [i] of the point cloud data.

[0121] num_patches[i] specifies the number of atlases in region [i] of the point cloud data.

[0122] num_tiles[i] specifies the number of 2D tiles in region [i] of the point cloud data.

[0123] tile_track_group_id[i][j] specifies the array group of 2D tile track group identifiers of region [i] of the point cloud data.

[0124] num_tile_groups[i] specifies the number of tile groups in region [i] of the point cloud data.

[0125] tile_group_id[i][j] specifies the array group of tile group identifiers of region [i] of the point cloud data.

[0126] Alternative embodiment 2

[0127] The 3D region of the point cloud data is associated with one or more 2D regions of the point cloud atlas frame, which contains atlas data, for reconstructing the 3D region of the point cloud data using 3DRegionToPatchBox() defined in this application.

[0128] Grammar Example

[0129]

[0130] Semantic Example

[0131] num_regions specifies the number of 3D regions of the point cloud data.

[0132] region_intent[i] specifies the intent of region [i] of the point cloud data.

[0133] layer_index[i] specifies the layer index of region [i] of the point cloud data.

[0134] num_patches[i] specifies the number of atlases in region [i] of the point cloud data.

[0135] Alternative Embodiment 3

[0136] The 3D regions of the point cloud data are associated with one or more 2D regions of a point cloud atlas frame that contains 2D tile data for reconstructing the 3D regions of the point cloud data using 3DRegionTo2DTileBox() defined as follows.

[0137] Grammar Example

[0138]

[0139]

[0140] Semantic Example

[0141] num_regions specifies the number of 3D regions of the point cloud data.

[0142] region_intent[i] specifies the intent of region [i] of the point cloud data.

[0143] layer_index[i] specifies the layer index of region [i] of the point cloud data.

[0144] num_tiles[i] specifies the number of 2D tiles in region [i] of the point cloud data.

[0145] tile_track_group_id[i][j] specifies the array of 2D tile track group identifiers for region [i] of the point cloud data.

[0146] Alternative Embodiment Four

[0147] The 3D region of the point cloud data is associated with one or more 2D regions of a point cloud orbit atlas frame, which contains tile group data for reconstructing the 3D region of the point cloud data using 3DRegionToPatchTileGroupBox() defined as follows.

[0148] Syntax example

[0149]

[0150]

[0151] Semantic example

[0152] num_regions specifies the number of 3D regions of the point cloud data.

[0153] region_intent[i] specifies the intent of region [i] of the point cloud data.

[0154] layer_index[i] specifies the layer index of region [i] of the point cloud data.

[0155] num_tile_groups[i] specifies the number of tile groups in region [i] of the point cloud data.

[0156] tile_group_id[i][j] specifies the array of tile group identifiers for region [i] of the point cloud data.

[0157] V-PCC orbit sample format

[0158] Each atlas sample in the V-PCC orbit corresponds to a single point cloud frame. The samples corresponding to the frames in each component video orbit should have the same composition time as the V-PCC orbit atlas samples. Each V-PCC atlas sample should contain only one V-PCC unit payload of type VPCC_PDG, which may include one or more atlas sequence unit payloads.

[0159] Alternative embodiment 1

[0160] The 3D region of the point cloud data is associated with one or more 2D regions of a point cloud orbit atlas frame, which may contain atlas data, 2D tile data, or tile group data for reconstructing the 3D region of the point cloud data using VolumetricTileInfoBox() defined as follows.

[0161] Syntax example

[0162]

[0163] Semantic example

[0164] vpcc_unit_payload_size provides the number of bytes for vpcc_unit_payload().

[0165] vpcc_unit_payload() is the payload of a V-PCC unit of type VPCC_PDG and shall contain an instance of patch_data_group().

[0166] Alternative Embodiment 2

[0167] A 3D region of point cloud data is associated with one or more 2D regions of a point cloud orbit atlas frame that contains atlas data for reconstructing the 3D region of the point cloud data using 3DRegionToPatchBox() defined as follows.

[0168] Syntax Example

[0169]

[0170]

[0171] Alternative Embodiment 3

[0172] A 3D region of point cloud data is associated with one or more 2D regions of a point cloud orbit atlas frame that contains 2D tile data for reconstructing the 3D region of the point cloud data using 3DRegionTo2DTileBox() defined as follows.

[0173] Syntax Example

[0174]

[0175] Alternative Embodiment 4

[0176] A 3D region of point cloud data is associated with one or more 2D regions of a point cloud orbit atlas frame that contains tile group data for reconstructing the 3D region of the point cloud data using 3DRegionToPatchTileGroupBox() defined as follows.

[0177] Syntax Example

[0178]

[0179] Embodiment 2

[0180] As Figure 3As shown, step S310 describes determining a 3D region of point cloud data based on elements in a timing metadata track; this timing metadata track contains a specific track reference to the point cloud track. The timing metadata track is described below in connection with alternative embodiments.

[0181] V-PCC Timing Metadata Track

[0182] By using a TrackReferenceBox of the 'cdsc' track reference type, the V-PCC spatial region timing metadata track is linked to the corresponding V-PCC track, and this V-PCC spatial region timing metadata track indicates the corresponding spatial region information of the point cloud data that changes dynamically over time.

[0183] The sample entry and its sample format of the V-PCC timing metadata track are defined as follows:

[0184] The 3D region of the point cloud data is associated with one or more 2D regions of a point cloud track atlas frame, and this point cloud track atlas frame may contain atlas data, 2D tile data, or tile group data for reconstructing the 3D region of the point cloud data using a VolumetricTileInfoBox() defined as follows.

[0185] Syntax Example

[0186]

[0187]

[0188] Solution 2

[0189] This example provides a method for processing point cloud data. As Figure 4 shown, the method includes the following steps:

[0190] Step S410 includes determining a 3D region of the point cloud data and a point cloud component track group corresponding to this 3D region.

[0191] Step S420 includes reconstructing the 3D region of the point cloud data based on the video frame data from the point cloud component tracks in the point cloud component track group and the corresponding point cloud track atlas frame data.

[0192] In some embodiments, the spatial region (e.g., 3D region) of the point cloud data includes a format similar to the format described previously for Solution 1 or is described using a format similar to the format described previously for Solution 1 (see paragraph

[0042] and subsequent paragraphs).

[0193] In some embodiments, the point cloud reconstruction operation in S420 can be performed as follows.

[0194] The inputs to the point cloud reconstruction process are as follows:

[0195] – Decoded geometric video frames from point cloud component tracks in a point cloud component track group at nominal resolution,

[0196] - Decoded attribute video frames from point cloud component tracks in a point cloud component track group at nominal resolution,

[0197] - Decoded occupancy map video frames from point cloud component tracks in a point cloud component track group at nominal resolution,

[0198] - Block-to-atlas mapping information corresponding to the point cloud track, and / or

[0199] - Frame indices in the output order.

[0200] In some embodiments, in order to provide an accurate mapping between video and other parameters and point cloud data, upscaling may be applied to the above inputs.

[0201] The outputs of the point cloud reconstruction process are as follows:

[0202] - A container that holds a list of points in the reconstructed point cloud frame for a 3D region, and

[0203] - The number of points in the point cloud for the reconstruction of the 3D region.

[0204] Embodiment 3

[0205] As Figure 5 shown, step S510 describes determining a 3D region of the point cloud data and a point cloud component track group corresponding to the 3D region based on elements in the track group data box of the point cloud component tracks; wherein, the point cloud component track group includes: an occupancy map track, a geometry track, and an attribute track.

[0206] V-PCC component tracks for video coding

[0207] Since it may not make sense to display decoded frames based on attribute, geometry, or occupancy map tracks without reconstructing the point cloud on the player side, a restricted video scheme type is defined for these video coding tracks.

[0208] V-PCC component video tracks can be represented as restricted video in a file and are identified by 'pccv' in the scheme_type field of the SchemeTypeBox of the RestrictedSchemeInfoBox of their restricted video sample entry.

[0209] There should be a SchemeInformationBox in the V-PCC component video track, and the V-PCC component video track includes a VPCCUnitHeaderBox.

[0210] The V-PCC component video track includes at least: a 2D video coding occupancy map track, a 2D video coding geometry track, and zero or more 2D video coding attribute tracks.

[0211] Alternative Embodiment 1

[0212] The track grouping tool of ISO / IEC 14496-12 can be used to group together all V-PCC component tracks (occupancy map tracks, geometry tracks, and attribute tracks) corresponding to the same spatial region of point cloud data by adding a specific type of VPCCTrackGroupBox ('pctg') to all component tracks. The VPCCTrackGroupBox in the V-PCC component video track is described in conjunction with alternative embodiments below.

[0213] VPCCTrackGroupBox

[0214] Box type: 'pctg'

[0215] Container: TrackBox

[0216] Mandatory: No

[0217] Quantity: Zero or one

[0218] A TrackGroupTypeBox with track_group_type equal to 'pctg' is a VPCCTrackGroupBox, which indicates a V-PCC component track group.

[0219] V-PCC component tracks with the same track_group_id value within a TrackGroupTypeBox with track_group_type equal to 'pctg' belong to the same track group with a 3D spatial relationship (e.g., corresponding to the same 3D region of point cloud data). The track_group_id within a TrackGroupTypeBox with track_group_type equal to 'pctg' can be used as an identifier for the spatial region of point cloud data.

[0220] V-PCC component tracks corresponding to the same spatial region have the same track_group_id value for track_group_type 'pctg', and the track_group_id of tracks from one spatial region is different from the track_group_id of tracks from any other spatial region.

[0221] Syntax example

[0222] aligned(8)class VPCCTrackGroupBox extends TrackGroupTypeBox('pctg'){

[0223] SpatialRegionInfoStruct();

[0224] }

[0225] V-PCC component video tracks may include a TrackGroupTypeBox with track_group_type equal to 'pctg', that is, a VPCCTrackGroupBox, and alternative embodiments for V-PCC component video tracks including a VPCCTrackGroupBox are described below.

[0226] Figure 6 is a schematic diagram of a VPCCTrackGroupBox in a V-PCC component track according to an embodiment.

[0227] As Figure 6 shown, the point cloud data file includes: an occupancy map component video track, a geometry component video track, and zero or more attribute component video tracks. Different V-PCC component video tracks include a TrackGroupTypeBox with track_group_type equal to 'pctg', that is, a VPCCTrackGroupBox, and have the same track_group_id value, which indicates that the V-PCC occupancy map, geometry, and attribute component video tracks form a V-PCC component track group corresponding to the same spatial region (e.g., a 3D region of the point cloud) of the point cloud data.

[0228] Alternative embodiment 2

[0229] The track grouping tool of ISO / IEC 14496-12 can be used to group all V-PCC component 2D tile tracks (occupying Figure 2 D tile tracks, geometry 2D tile tracks, and attribute 2D tile tracks) corresponding to the same spatial region of the point cloud data by adding a specific type of VPCC2DTileGroupBox (‘pcti’) to all component 2D tile tracks. The VPCC2DTileGroupBox in the V-PCC component video track will be described in conjunction with the optional embodiments below.

[0230] VPCC2DTileGroupBox

[0231] Box type: ‘pcti’

[0232] Container: TrackBox

[0233] Mandatory: No

[0234] Quantity: Zero or one

[0235] The TrackGroupTypeBox with track_group_type equal to ‘pcti’ is the VPCC2DTileGroupBox, which indicates a V-PCC component 2D tile track group.

[0236] V-PCC component 2D tile tracks having the same track_group_id value within a TrackGroupTypeBox with track_group_type equal to ‘pcti’ belong to the same track group having a 3D spatial relationship (e.g., corresponding to the same 3D region of the point cloud data). The track_group_id within a TrackGroupTypeBox with track_group_type equal to ‘pcti’ can be used as an identifier for the spatial region of the point cloud data.

[0237] V-PCC component 2D tile tracks corresponding to the same spatial region have the same track_group_id value for track_group_type ‘pcti’, and the track_group_id of tracks from one spatial region is different from the track_group_id of tracks from any other spatial region.

[0238] Syntax example

[0239] aligned(8)class VPCC2DTileGroupBox extends TrackGroupTypeBox('pcti'){

[0240] SpatialRegionInfoStruct();

[0241] }

[0242] Alternative Embodiment 3

[0243] The point cloud component tile group tracks in all point cloud components (geometry component, occupancy map component, attribute component) corresponding to the same point cloud 3D region can be composed of a specific type of track group data box (track group box). Point cloud component tile group track group of the 3D region. The point cloud component tile group data box in the point cloud component tile group track is described below in conjunction with alternative embodiments.

[0244] The track grouping tool of ISO / IEC 14496-12 can be used to group together all V-PCC component tile group tracks (occupancy map tile group track, geometry tile group track, and attribute tile group track) corresponding to the same spatial region of the point cloud data by adding a specific type of VPCCTileGroupBox ('ptgg') to all component 2D tile tracks. The VPCCTileGroupBox in the V-PCC component video track will be described below in conjunction with alternative embodiments.

[0245] VPCCTileGroupBox

[0246] Box type: 'ptgg'

[0247] Container: TrackBox

[0248] Mandatory: No

[0249] Quantity: Zero or one

[0250] The track group type data box (TrackGroupTypeBox) whose track group type (track_group_type) is equal to 'ptgg' is the point cloud component Tile Group group data box, which indicates the point cloud component Tile Group track group in the point cloud component. All point cloud components of the point cloud component TileGroup data box with the same track group identifier (track_group_id) value belong to the track group with a spatial region relationship (for example, corresponding to the 3D region of the same point cloud). The track group identifier of the point cloud component group data box can also be used as the point cloud spatial region identifier (region_id), and the values of the track group identifiers corresponding to different point cloud spatial regions should be different.

[0251] The TrackGroupTypeBox with track_group_type equal to 'ptgg' is the VPCCTileGroupBox, and this VPCCTileGroupBox indicates a V-PCC component tile group track group.

[0252] V-PCC component tile group tracks within a TrackGroupTypeBox with track_group_type equal to 'ptgg' and having the same track_group_id value belong to the same track group with a 3D spatial relationship (e.g., corresponding to the same 3D region of point cloud data). The track_group_id within a TrackGroupTypeBox with track_group_type equal to 'ptgg' can be used as an identifier for the spatial region of point cloud data.

[0253] V-PCC component tile group tracks corresponding to the same spatial region have the same track_group_id value for track_group_type 'ptgg', and the track_group_id of tracks from one spatial region is different from the track_group_id of tracks from any other spatial region.

[0254] Syntax example

[0255]

[0256] Semantic example

[0257] tile_group_id specifies the tile group identification of the V-PCC component tile group track group.

[0258] Example 4

[0259] V-PCC timing metadata track

[0260] The V-PCC spatial region timing metadata track is linked to the corresponding V-PCC component track group by using a TrackReferenceBox of the 'cdsc' track reference type, and this V-PCC spatial region timing metadata track indicates the corresponding spatial region information of the point cloud data that changes dynamically over time.

[0261] The group component V-PCC spatial region timing metadata track contains a 'cdsc' track reference to the track_group_id value of the V-PCC component track group to describe each V-PCC component track in the track group respectively.

[0262] The sample entry and sample format of the V-PCC timing metadata track are defined as follows:

[0263] Syntax example

[0264]

[0265]

[0266] Figure 6 An example file format in which video samples are arranged as media data is shown. The video samples are sorted according to tracks (shown as trak). An example of a track includes a geometry track of point cloud data identified based on a track group entry and a specific track group type (pctg). Another example of the shown track is an attribute track of point cloud data. A third example of a track is an occupancy map trak of point cloud data. For example, the occupancy map contains data that provides mapping of individual points in a 3D region to corresponding one or more 2D regions.

[0267] Figure 7 It is a schematic diagram of a V-PCC spatial region temporal metadata track that references a V-PCC component track group according to an embodiment of the present invention.

[0268] As Figure 7 shown, the TrackReferenceBox is included in the V-PCC timing metadata track, where the sample entry type is equal to 'pcsr', and the track_IDs[] in the TrackReferenceBox reference the track_group_id value of the V-PCC component track group. The reference_type in the TrackReferenceBox takes the value 'cdsc' and indicates that the timing metadata track contains content description information of the referenced V-PCC component track group. In this embodiment, the V-PCC spatial region timing metadata video track refers to the track group identifier of the V-PCC component track group, where the track_group_type of the V-PCC component track group is equal to 'pctg', and this V-PCC spatial region timing metadata video track indicates the corresponding spatial region information of the point cloud data that changes dynamically over time.

[0269] Some embodiments may preferably implement the following technical solutions.

[0270] 1. A method for processing point cloud data, including: determining a 3D region of the point cloud data and a 2D region of a point cloud track atlas frame, where one or more points in the 3D region are projected onto the 2D region of the point cloud track atlas frame; reconstructing the 3D region of the point cloud data according to the atlas frame data included in the 2D region and the video frame data of the corresponding point cloud component track. Refer to Solution 1 andFigure 1 Describe an example of the solution.

[0271] 2. The method according to Solution 1, wherein, based on an element in a point cloud track sample entry or sample, a 3D region of point cloud data and a 2D region of a point cloud track atlas frame are determined, and one or more points in the 3D region are projected onto the 2D region of the point cloud track atlas frame. Reference Figure 2 Describe an example of the solution.

[0272] 3. The method according to Solution 1 or Solution 2, comprising: identifying an atlas data box in a point cloud track according to a specific box type, wherein the atlas data box indicates that the 2D region contains atlas data for reconstructing the 3D region of the point cloud data.

[0273] 4. The method according to Solution 1 or Solution 2, comprising: identifying a 2D tile data box in a point cloud track according to a specific box type, wherein the 2D tile data box indicates that the 2D region contains 2D tile data of the 3D region of the point cloud data.

[0274] 5. The method according to Solution 1 or Solution 2, comprising: identifying a tile group data box in a point cloud track according to a specific box type, wherein the tile group data box indicates that the 2D region contains tile group data for reconstructing the 3D region of the point cloud data.

[0275] 6. The method according to Solution 1, wherein the determination comprises: determining a 3D region of the point cloud data based on an element in a timing metadata track; the timing metadata track contains a specific track reference to the point cloud track. Figure 3 And the related text describes some additional features and embodiments of the solution.

[0276] 7. The method according to Solution 1 or Solution 6, comprising: identifying a timing metadata track according to a specific sample entry type, the timing metadata track indicating that the 2D region contains one of the following information for reconstructing the 3D region of the point cloud data: atlas data, 2D tile data, tile group data.

[0277] 8. A method for processing point cloud data, comprising: determining a 3D region of the point cloud data and a point cloud component track group corresponding to the 3D region; and reconstructing the 3D region of the point cloud data based on video frame data from a point cloud component track in the point cloud component track group and corresponding point cloud track atlas frame data. Figure 4 Solutions 2 and the related text describe some embodiments and features of the solution.

[0278] 9. The method according to solution 8, wherein determining the 3D region includes: determining the 3D region of the point cloud data and the point cloud component track group corresponding to the 3D region based on the elements in the track group data box of the point cloud component track; wherein the point cloud component track group includes: an occupancy map track, a geometry track, and an attribute track.

[0279] 10. The method according to solution 8 or 9, including: identifying a track group data box in the point cloud component track according to a specific track group type, the track group data box indicating the 3D region of the point cloud data corresponding to the point cloud component track group; wherein the track group data boxes of the point cloud component tracks corresponding to the same 3D region of the point cloud data have the same track group identifier.

[0280] 11. The method according to solution 8 or 9, including: identifying a track group data box in the point cloud component 2D tile track according to a specific track group type, the track group data box indicating the 3D region of the point cloud data corresponding to the point cloud component 2D tile track group; wherein the track group data boxes of the point cloud component 2D tile tracks corresponding to the same 3D region of the point cloud data have the same track group identifier.

[0281] 12. The method according to solution 8 or 9, including: identifying a track group data box in the point cloud component tile group track according to a specific track group type, the track group data box indicating the 3D region of the point cloud data corresponding to the point cloud component tile group track group; wherein the track group data boxes of the point cloud component tile group tracks corresponding to the same 3D region of the point cloud data have the same track group identifier.

[0282] 13. The method according to solution 8, wherein the determination includes: determining the 3D region of the point cloud data and the point cloud component track group corresponding to the 3D region based on the elements in the timing metadata track; the timing metadata track contains a specific track reference to the point cloud component track group.

[0283] 14. The method according to solution 8 or solution 13, including: identifying the timing metadata track according to a specific sample entry type, the timing metadata track indicating the 3D region of the point cloud data corresponding to the point cloud component track group.

[0284] 15. The method according to any one of solutions 1 to 14, wherein the 3D region includes points projected onto a plurality of 2D regions of the point cloud track atlas frame.

[0285] 16. The method according to any one of Solutions 1 to 15, wherein the information of the 3D region of the point cloud data includes: 3D bounding box parameters, and the 3D bounding box parameters include: the x, y, and z coordinate offsets of the 3D region, and / or the width, height, and depth of the 3D region.

[0286] 17. The method according to any one of Solutions 1 - 16, wherein the 3D region of the point cloud data further includes one of the following information: the label of the point cloud data; the number of points of the point cloud data; or the number and type of point cloud attributes.

[0287] 18. The method according to any one of Solutions 1 to 17, wherein the information of the 2D region of the point cloud orbit atlas frame includes: 2D bounding box parameters, and the 2D bounding box parameters include: the x, y coordinate offsets of the 2D region, and / or the width and height of the 2D region.

[0288] 19. The method according to any one of Solutions 1 - 18, wherein the 2D region of the point cloud orbit atlas frame further includes one of the following information: layer information, description information, the number of atlases; 2D tile information or tile group information;

[0289] 20. A video processing device, comprising a processor configured to implement the method described in any previous solution.

[0290] 21. A computer - readable medium having code stored thereon for execution by a processor to perform the above - described method.

[0291] Figure 6 and Figure 7 shows an example of a file format for facilitating the point cloud data processing techniques described herein.

[0292] Figure 8 is a block diagram of an example of a device 800 for processing point cloud data. The device 800 includes a processor 802 programmed to implement the methods described in this application. The device 800 may also include dedicated hardware circuits for performing specific functions such as data decoding, accessing component tracks, etc. The device 800 may also include a memory that stores the executable code of the processor and / or 2D volume data and other data, including data conforming to various syntax elements described in this application.

[0293] In some embodiments, the 3D point cloud data encoder may be implemented to generate a bit - stream representation of the 3D point cloud by encoding 3D space information using the syntax and semantics described in this application.

[0294] The point cloud data encoding or decoding device can be implemented as part of a computer, a user device (such as a laptop, a tablet, or a gaming device).

[0295] The disclosed embodiments and other embodiments, modules, and functional operations described in this application can be implemented in digital electronic circuits, or in computer software, firmware, or hardware (including the structures disclosed in this application and their structural equivalents), or in a combination of one or more of them. The disclosed embodiments and other embodiments can be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded on a computer-readable medium for execution by, or to control the operation of, a data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a storage device, a composition affecting a machine-readable propagated signal, or a combination of one or more of them. The term "data processing device" encompasses all devices, apparatuses, and machines for processing data, which includes, for example, programmable processors, computers, or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program being discussed, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, for example, a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to a suitable receiver device.

[0296] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language (including compiled or interpreted languages), and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program does not necessarily correspond to a file in a file system. The program can be stored in a part of a file that holds other programs or data (for example, one or more scripts in a markup language document), stored in a single file dedicated to the program being discussed, or stored in multiple cooperating files (for example, files that store one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0297] The processes and logical flows described in this application can be executed by one or more programmable processors that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logical flows can also be executed with special-purpose logic circuitry, and the apparatus can also be implemented as special-purpose logic circuitry, e.g., an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0298] Processors suitable for executing computer programs include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data (e.g., magnetic disks, magneto-optical disks, or optical disks) to receive data from them, or to send data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include various forms of non-volatile memory, media, and memory devices, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special-purpose logic circuitry.

[0299] Although this patent application contains many details, these should not be construed as limiting the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features that are described in this patent application in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented separately in multiple embodiments or in any suitable sub-combination. Moreover, although the above features may be described as acting in certain combinations and even initially claimed as such, in some cases, one or more features from a claimed combination can be deleted from the combination, and the claimed combination may cover a sub-combination or a variant of a sub-combination.

[0300] Similarly, although operations are depicted in the drawings in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In addition, the separation of various system components in the embodiments described in this patent application should not be understood as requiring such separation in all embodiments.

[0301] Only some embodiments and examples are described, and other embodiments, enhancements, and variations can be made based on what is described and illustrated in this patent application.

Claims

1. A method for processing point cloud data, comprising: Determining a 3D region of the point cloud data; Determining a 2D region of a point cloud atlas frame, wherein one or more points in the 3D region of the point cloud data are projected onto the 2D region, wherein the 2D region is generated based on at least one of the following partitioning principles: defining the 2D region based on an atlas position on the point cloud atlas frame, defining the 2D region based on a tile position on the point cloud atlas frame, or defining the 2D region based on a tile group position on the point cloud atlas frame; Identifying a first box in the point cloud orbit according to a specific box type, wherein the first box indicates that the 2D region contains atlas data for reconstructing the 3D region of the point cloud data, or identifying a second box in the point cloud orbit according to a specific data box type, wherein the second box indicates that the 2D region contains 2D tile data of the 3D region of the point cloud data; Reconstructing the 3D region of the point cloud data according to the atlas frame data included in the 2D region and the video frame data of the corresponding point cloud component orbit.

2. The method according to claim 1, wherein Based on an element in a point cloud orbit sample entry or sample, determining the 3D region of the point cloud data and the 2D region of the point cloud atlas frame, and one or more points in the 3D region are projected onto the 2D region of the point cloud atlas frame.

3. The method according to claim 1, wherein The determining includes: Based on an element in a timing metadata orbit, determining the 3D region of the point cloud data; the timing metadata orbit contains a specific orbit reference to the point cloud orbit.

4. The method according to claim 1 or 3, comprising: Identifying the timing metadata orbit according to a specific sample entry type, the timing metadata orbit indicating that the 2D region contains one of the following information for reconstructing the 3D region of the point cloud data: Atlas data, 2D tile data, tile group data.

5. A method for processing point cloud data, comprising: Determining a 3D region of the point cloud data; Determining a group of point cloud component orbits corresponding to the 3D region; Identifying a first box in the point cloud component orbit according to a specific orbit group type, the first box indicating the 3D region of the point cloud data corresponding to the group of point cloud component orbits; wherein, the first boxes of the point cloud component orbits corresponding to the same 3D region of the point cloud data have the same orbit group identifier; and Reconstructing the 3D region of the point cloud data based on the video frame data of the point cloud component orbits in the group of point cloud component orbits and the corresponding point cloud atlas frame data, wherein the video frame data includes at least one of the following: decoded geometry video frame, decoded attribute video frame, decoded occupancy map video frame, block-to-atlas mapping information, or frame index in the output order.

6. The method according to claim 5, wherein, The determining the 3D region includes: Based on an element in the orbit group data box of the point cloud component orbit, determining the 3D region of the point cloud data and the group of point cloud component orbits corresponding to the 3D region; wherein, the group of point cloud component orbits includes: occupancy map orbit, geometry orbit, attribute orbit.

7. The method according to claim 5, wherein The determining includes: Determine the 3D region of the point cloud data and the point cloud component track group corresponding to the 3D region based on elements in a timing metadata track; the timing metadata track contains a specific track reference to the point cloud component track group.

8. The method according to claim 5 or 7, comprising: Identify the timing metadata track according to a specific sample entry type, the timing metadata track indicating the 3D region of the point cloud data corresponding to the point cloud component track group.

9. The method according to any one of claims 1 or 5, wherein The information of the 3D region of the point cloud data includes: 3D bounding box parameters, the 3D bounding box parameters including: the x, y, and z coordinate offsets of the 3D region, and / or the width, height, and depth of the 3D region.

10. The method according to claim 9, wherein, The information of the 2D region of the point cloud track atlas frame includes: 2D bounding box parameters, the 2D bounding box parameters including: the x and y coordinate offsets of the 2D region, and / or the width and height of the 2D region, the description information of the atlas, and 2D tile information.

11. A video processing apparatus, comprising a processor configured to implement the method according to any one of claims 1 to 10.

12. A computer-readable medium having code stored thereon for implementing the method according to any one of claims 1 to 10 by a processor.

Images