Information processing device and method

Abstract

The present disclosure relates to an information processing device and method with which any increase in reproduction processing load can be suppressed. According to the present disclosure: tile identification information that indicates the tile of a point cloud representing a three-dimensionally shaped object as a collection of points, the tile identification information corresponding to a data unit of a bit stream of the point cloud, is used to generate tile management information, which is information for managing a tile that corresponds to a sub-sample composed of one or a plurality of contiguous data units of a bit stream stored as sample in a file; and a file in which the bit stream and the tile management information are stored is generated. The present disclosure can be applied to, e.g., an information processing device or an information processing method.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that can suppress an increase in the load of playback processing. [Background technology]

[0002] Conventionally, a point cloud, which is a collection of points in three-dimensional space that simultaneously have position information and attribute information (color, reflection, etc.), is coded by separating it into geometry that indicates the three-dimensional shape and attributes that indicate the attribute information. An encoding technology called Geometry-based Point Cloud Compression (G-PCC) is currently being standardized in MPEG-I Part 9 (ISO / IEC 23090-9) (see, for example, Non-Patent Document 1).

[0003] There is also the International Organization for Standardization Base Media File Format (ISOBMFF), which is a file container specification of the Moving Picture Experts Group-4 (MPEG-4), an international standard technology for video compression (see, for example, Non-Patent Document 2).

[0004] Furthermore, with the aim of improving the efficiency of playback processing from local storage and network distribution of bitstreams coded with this G-PCC, a method for storing G-PCC bitstreams in ISOBMFF is currently being standardized in MPEG-I Part 18 (ISO / IEC 23090-18) (see, for example, Non-Patent Document 3).

[0005] The G-PCC bitstream can have a partial access structure that allows some points in the bitstream to be decoded and played back independently of others. In a point cloud with this partial access structure, a data unit that can be decoded and played back independently (independently accessible) is called a tile.

[0006] For example, a profile has been proposed that decodes only the portion of a point cloud within the field of view, or decodes the portion closer to the viewpoint at a higher resolution (see, for example, Non-Patent Document 4). By applying such a method, it is possible to suppress an increase in the processing of unnecessary information, and therefore to suppress an increase in the load of the playback process. This method is particularly useful for large-scale point clouds such as map data. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] "Information technology - MPEG-I (Coded Representation of Immersive Media) - Part 9: Geometry-based Point Cloud Compression", SO / IEC 23090-9:2020(E) [Non-patent document 2] "Information technology - Coding of audio-visual objects - Part 12:ISO base media file format", ISO / IEC 14496-12, 2015-02-20 [Non-patent document 3] Sejin Oh, Ryohei Takahashi, Youngkwon Lim, "WD of ISO / IEC 23090-18 Carriage of Geometry-based Point Cloud Compression Data", ISO / IEC JTC 1 / SC 29 / WG 11 N19286, 2020-06-05 [Non-patent document 4] Satoru Kuma, Ohji Nakagami, "[G-PCC] (New proposal) On scalability profile", ISO / IEC JTC1 / SC29 / WG11 MPEG2020 / m53292, April 2020 Summary of the Invention [Problem to be solved by the invention]

[0008] However, in the method described in Non-Patent Document 3, in the case of a G-PCC bitstream with such a partial access structure, each independently playable partial point cloud is stored in a different track. In other words, the granularity of partial access depends on the number of tracks.

[0009] Generally, the larger the point cloud, the more diverse partial accesses may be required. In other words, the method described in Non-Patent Document 3 requires more tracks. As the number of tracks increases, the file size may increase. Furthermore, as the number of tracks increases, the complexity of track management increases, which may increase the load of the playback process.

[0010] Therefore, it is possible to store multiple partial point clouds in one track. However, to realize partial access, it is necessary to extract the data units required for playback from the geometry data units and attribute data units that make up the G-PCC bitstream. To do this, it is necessary to understand the relationship between tiles and data units and, based on that relationship, identify the data unit corresponding to the tile to be played back.

[0011] However, in the method described in Non-Patent Document 3, information indicating the relationship between the tile and the data unit is stored only in the header of each data unit in the G-PCC bitstream. Therefore, in order to identify the data unit to extract, it is necessary to parse the G-PCC bitstream. In other words, it is necessary to parse unnecessary G-PCC bitstreams, which may increase the load of the playback process.

[0012] The present disclosure has been made in consideration of such circumstances, and aims to make it possible to suppress an increase in the load of the regeneration process. [Means for solving the problem]

[0013] An information processing device according to one aspect of the present technology includes: For each subsample, first flag information indicating whether the subsample is composed of a single or multiple consecutive data units that constitute the tile is generated as tile management information for managing the tile of the point cloud corresponding to the subsample, and if the first flag information is true, further, identification information of the tile corresponding to the subsample is generated. and a file generation unit that generates the file that stores the bitstream and the tile management information.

[0014] An information processing method according to one aspect of the present technology includes: The information processing device A bitstream of a point cloud that represents a 3D object as a set of points. For each subsample, first flag information indicating whether the subsample is composed of a single or multiple consecutive data units that constitute the tile is generated as tile management information for managing the tile of the point cloud corresponding to the subsample, and if the first flag information is true, further, identification information of the tile corresponding to the subsample is generated. and generating the file that stores the bitstream and the tile management information.

[0015] Another aspect of the present technology provides an information processing device for generating a bit stream of a point cloud that represents a three-dimensional object as a set of points. and a portion of the bitstream necessary for reproducing the desired tile based on first flag information, which is included in tile management information that manages tiles of the point cloud corresponding to a subsample of the tile, and which indicates whether the subsample is composed of a single or multiple consecutive data units that constitute the tile, and identification information of the tile corresponding to the subsample, which is further included in the tile management information when the first flag information is true. The information processing device includes an extraction unit that performs extraction.

[0016] An information processing method according to another aspect of the present technology includes: The information processing device A bitstream of a point cloud that represents a 3D object as a set of points and a portion of the bitstream necessary for reproducing the desired tile based on first flag information, which is included in tile management information that manages tiles of the point cloud corresponding to a subsample of the tile, and which indicates whether the subsample is composed of a single or multiple consecutive data units that constitute the tile, and identification information of the tile corresponding to the subsample, which is further included in the tile management information when the first flag information is true. It is an information processing method for extracting information.

[0017] In an information processing device and method according to one aspect of the present technology, a bit stream of a point cloud that represents a three-dimensional object as a set of points is generated. For each subsample, first flag information is generated as tile management information for managing the tile of the point cloud corresponding to the subsample, indicating whether the subsample is composed of a single or multiple consecutive data units that constitute a tile. If the first flag information is true, identification information of the tile corresponding to the subsample is further generated. is generated, and a file storing the bitstream and tile management information is generated.

[0018] In another aspect of the present technology, an information processing device and a method include: The part of the bitstream necessary for reproducing the desired tile is determined based on first flag information, which is included in tile management information that manages the tile of the point cloud corresponding to the subsample, and indicates whether the subsample is composed of a single or multiple consecutive data units that constitute a tile, and if the first flag information is true, the tile management information is further included in the tile management information to identify the tile corresponding to the subsample. It is extracted. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram illustrating an overview of G-PCC. [Figure 2] FIG. 10 is a diagram illustrating partial access. [Figure 3] A diagram showing an example of the structure of a G-PCC bitstream. [Figure 4] FIG. 10 is a diagram illustrating an example of tile inventory syntax. [Figure 5] FIG. 10 is a diagram illustrating an example of a file structure. [Figure 6] FIG. 10 is a diagram illustrating an example of a file structure. [Figure 7] FIG. 1 is a diagram illustrating scalable decoding. [Figure 8] FIG. 10 is a diagram illustrating signaling of tile identification information. [Figure 9] FIG. 10 is a diagram showing an example of a file structure in the case of a single track. [Figure 10] FIG. 10 is a diagram illustrating an example of signaling of tile identification information. [Figure 11] FIG. 10 is a diagram illustrating an example of a SubSampleInformationBox. [Figure 12] A figure showing an example of codec_specific_parameters. [Figure 13] FIG. 10 is a diagram illustrating an example of signaling of tile identification information. [Figure 14] A figure showing an example of codec_specific_parameters. [Figure 15] FIG. 10 is a diagram illustrating an example of signaling of tile identification information. [Figure 16] A figure showing an example of codec_specific_parameters. [Figure 17] FIG. 10 is a diagram illustrating an example of signaling of tile identification information using timed metadata. [Figure 18] FIG. 10 is a diagram showing an example of a file structure in the case of multi-track. [Figure 19] FIG. 10 is a diagram illustrating an example of signaling of tile identification information. [Figure 20] FIG. 10 is a diagram illustrating an example of signaling of tile identification information using timed metadata. [Figure 21] FIG. 10 is a diagram illustrating an example of the configuration of a Matryoshka media container. [Figure 22] FIG. 2 is a block diagram illustrating an example of the main configuration of a file generation device. [Figure 23] 10 is a flowchart illustrating an example of the flow of a file generation process. [Figure 24] FIG. 2 is a block diagram illustrating an example of the main configuration of a decoding device. [Figure 25] FIG. 2 is a block diagram showing an example of the main configuration of a playback processing unit. [Figure 26]10 is a flowchart showing an example of the flow of a playback process. [Figure 27] FIG. 1 is a block diagram illustrating an example of the main configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described in the following order. 1. Partial access of G-PCC bitstream 2. Signaling Tile Identity Information 3. First embodiment (file generation device) 4. Second embodiment (playback device) 5. Supplementary Notes

[0021] <1. Partial access of G-PCC bitstream> <References supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the contents described in the embodiments but also the contents described in the following non-patent documents that were publicly known at the time of filing, as well as the contents of other documents referenced in the following non-patent documents.

[0022] Non-patent document 1: (mentioned above) Non-patent document 2: (mentioned above) Non-patent document 3: (mentioned above) Non-patent document 4: (mentioned above) Non-patent document 5: https: / / www.matroska.org / index.html

[0023] In other words, the contents of the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements.

[0024] <Point Cloud> Previously, there was 3D data such as point clouds that represent three-dimensional structures using point position information and attribute information.

[0025] For example, in the case of a point cloud, a three-dimensional structure (a three-dimensional object) is represented as a set of a large number of points. A point cloud is composed of the position information of each point (also referred to as geometry) and the attribute information (also referred to as attribute). The attribute can include any information. For example, color information, reflectance information, normal information, etc. of each point may be included in the attribute. Thus, the point cloud has a relatively simple data structure and can represent any three-dimensional structure with sufficient accuracy by using a sufficiently large number of points.

[0026] <Overview of G-PCC> In Non-Patent Document 1, an encoding technique called Geometry-based Point Cloud Compression (G-PCC) is disclosed for encoding this point cloud separately into geometry and attributes. G-PCC is in the process of being standardized in MPEG-I Part 9 (ISO / IEC 23090-9).

[0027] For the compression of geometry, octree encoding as shown in FIG. 1 is applied. For example, octree encoding is a method of representing the presence or absence of points in each block in data represented by rectangular voxels as shown on the left side of FIG. 1 by an octree as shown on the right side of FIG. 1. In this method, as shown in FIG. 1, a block with points is represented as 1 and a block without points is represented as 0.

[0028] The encoded data (bitstream) generated by encoding geometry as described above is also referred to as a geometry bitstream.

[0029] Attribute compression is performed using techniques such as Predicting Weight Lifting, Region Adaptive Hierarchical Transform (RAHT), or Fix Weight Lifting. The coded data (bitstream) generated by coding attributes is also called an attribute bitstream. A bitstream that combines a geometry bitstream and an attribute bitstream is also called a G-PCC bitstream.

[0030] <tile> The G-PCC bitstream can have a partial access structure that allows some point bitstreams to be decoded and played back independently of others. In a point cloud with this partial access structure, tiles and slices are data units that can be decoded and played back independently (independently accessible).

[0031] As shown in A of FIG. 2, a bounding box 21 is set to contain a three-dimensional object 20. A tile 22 is a rectangular parallelepiped area within the bounding box 21. As shown in B of FIG. 2, a slice 24 is a set of points within the tile 23. Points may overlap between slices (i.e., one point may belong to multiple slices). A tile is made up of one or more slices (1 tile = Y slice(s)).

[0032] A point cloud at a certain time is called a point cloud frame. This frame is a data unit equivalent to a frame in a 2D video image. A point cloud frame is composed of one or more tiles (1 point cloud frame = X tile(s)).

[0033] <G-PCC bitstream with partial access structure> An example of the main structure of a G-PCC bitstream that encodes such a point cloud that allows partial access (an example of the Type-length-value bytestream format defined in Annex B of Non-Patent Document 1) is shown in Figure 3. In other words, the G-PCC bitstream shown in Figure 3 has a partial access structure, and a part of it can be extracted and decoded independently of the rest.

[0034] In Figure 3, each square represents one Type-length-value encapsulation structure (tlv_encapsulation()). As shown in Figure 3, the G-PCC bitstream has a Sequence Parameter Set (SPS), a Geometry Parameter Set (GPS), an Attribute Parameter Set (APS(s)), a Tile Inventory, a Geometry Data Unit, and an Attribute Data Unit.

[0035] A sequence parameter set is a parameter set that has parameters related to the entire sequence. A geometry parameter set is a parameter set that has parameters related to geometry. An attribute parameter set is a parameter set that has parameters related to attributes. There may be multiple geometry parameter sets and attribute parameter sets. The geometry parameter sets and attribute parameter sets may differ on a slice-by-slice basis (they can be set on a slice-by-slice basis).

[0036] The tile inventory manages information about tiles. For example, the tile inventory stores identification information, position information, size information, etc. for each tile. Figure 4 shows an example of the syntax of the tile inventory. As shown in Figure 4, the tile inventory stores, for each tile, tile identification information (tile_id), information about the tile's position and size (tile_bounding_box_offset_xyz, tile_bounding_box_size_xyz), etc. The tile inventory is variable on a frame-by-frame basis (it can be set on a frame-by-frame basis).

[0037] A data unit is a unit of data that can be extracted independently of others. A geometry data unit is a data unit of geometry. An attribute data unit is a data unit of an attribute. An attribute data unit is generated for each attribute included in an attribute.

[0038] A slice consists of one geometry data unit and zero or more attribute data units. A slice consists of a single or multiple consecutive data units in a G-PCC bitstream. Each data unit stores slice identification information (slice_id) that indicates the slice to which the data unit belongs. In other words, data units that belong to the same slice store the same slice identification information. In this way, geometry data units and attribute data units that belong to the same slice are linked together using the slice identification information.

[0039] A tile is composed of one or more consecutive slices in a G-PCC bitstream. Each geometry data unit stores tile identification information (tile_id) indicating the tile to which the slice to which the geometry data unit belongs belongs. In other words, geometry data units belonging to the same tile store the same tile identification information. In other words, slices belonging to the same tile are linked to each other using the tile identification information.

[0040] Furthermore, this tile identification information is managed in the tile inventory as described above, and each tile identification information is associated with information such as the position and size of the corresponding tile in three-dimensional space. In other words, when a desired point in three-dimensional space needs to be played back, the necessary data unit can be identified and extracted based on this tile identification information (and also slice identification information). Therefore, partial access can be achieved, and unnecessary information does not need to be decoded, thereby suppressing an increase in the load of the playback process.

[0041] <isobmff> Non-Patent Document 2 discloses ISOBMFF (International Organization for Standardization Base Media File Format), which is a file container specification of the international standard technology for video compression, MPEG-4 (Moving Picture Experts Group - 4).

[0042] <Storage of G-PCC Bitstream in ISOBMFF> Non-Patent Document 3 discloses a method of storing a G-PCC bitstream in ISOBMFF for the purpose of improving the efficiency of playback processing from local storage and network distribution of the bitstream encoded with this G-PCC. This method is under standardization work in MPEG-I Part 18 (ISO / IEC 23090-18).

[0043] FIG. 5 is a diagram showing an example of the file structure in that case. What stores the G-PCC bitstream in ISOBMFF is called a G-PCC file.

[0044] The sequence parameter set is stored in the GPCCDecoderConfigurationRecord of the G-PCC file. The GPCCDecoderConfigurationRecord may further include a geometry parameter set, an attribute parameter set, and a tile inventory according to the sample entry type.

[0045] The sample of the Media data box (Media) includes a geometry slice and an attribute slice corresponding to 1 point cloud frame. Furthermore, it may include a geometry parameter set, an attribute parameter set, and a tile inventory according to the sample entry type.

[0046] <Partial Access Structure of G-PCC File> A G-PCC file has a structure for accessing and decoding partial point clouds based on three-dimensional spatial information. The G-PCC file stores each partial point cloud in a different track. A partial point cloud is composed of one or more tiles. For example, as shown in FIG. 6, a point cloud frame 61 is composed of partial point cloud 61A, partial point cloud 61B, and partial point cloud 61C. In this case, partial point cloud 61A, partial point cloud 61B, and partial point cloud 61C are each stored in a different track (G-PCC track) of the G-PCC file. This structure allows a tile to be played to be selected by selecting the track to be played.

[0047] <Utilizing partial access> Non-Patent Document 4 discloses a profile that supports scalable decoding of point clouds. This profile enables, for example, decoding and rendering according to the viewpoint position during local playback of large-scale point cloud still images. For example, in FIG. 7, when looking from viewpoint 71 in line of sight 72, the area outside field of view 73 (white area in the figure) is not decoded, and only partial point clouds within field of view 73 are decoded and played back. Furthermore, partial point clouds in areas close to viewpoint 71 (dark gray area in the figure) are decoded and played back at a high LoD (high resolution), while partial point clouds in areas far from viewpoint 71 (light gray area in the figure) are decoded and played back at a low LoD (low resolution). This enables decoding and rendering processing according to the viewpoint position. This reduces the playback of unnecessary information, thereby suppressing an increase in the playback processing load.

[0048] Partial access within a track As described above, decoding and rendering partial point clouds according to the viewpoint position is useful, especially when playing back large-scale point clouds locally.

[0049] However, in the method described in Non-Patent Document 3, in the case of a G-PCC bitstream with such a partial access structure, each independently playable partial point cloud is stored in a different track. In other words, the granularity of partial access depends on the number of tracks.

[0050] Generally, the larger the point cloud, the more diverse partial accesses may be required. In other words, the method described in Non-Patent Document 3 requires more tracks. As the number of tracks increases, the file size may increase. Furthermore, as the number of tracks increases, the complexity of track management increases, which may increase the load of the playback process.

[0051] Therefore, it is possible to store multiple partial point clouds in one track. However, to realize partial access, it is necessary to extract the data units required for playback from the geometry data units and attribute data units that make up the G-PCC bitstream. To do this, it is necessary to understand the relationship between tiles and data units and, based on that relationship, identify the data unit corresponding to the tile to be played back.

[0052] However, in the method described in Non-Patent Document 3, information indicating the relationship between the tile and the data unit is stored only in the header of each data unit in the G-PCC bitstream. Therefore, in order to identify the data unit to extract, it is necessary to parse the G-PCC bitstream. In other words, it is necessary to parse unnecessary G-PCC bitstreams, which may increase the load of the playback process.

[0053] 2. Signaling of Tile Identification Information Therefore, tile identification information is stored in the G-PCC file, as shown in the top row of the table in Fig. 8. For example, subsamples are formed within a sample in the G-PCC file, and the tile identification information is stored in the G-PCC file as tile management information for managing tiles corresponding to each subsample.

[0054] For example, an information processing device may be provided with a tile management information generation unit that uses tile identification information indicating tiles of a point cloud corresponding to data units of a bit stream of the point cloud that represents a three-dimensional object as a collection of points to generate tile management information, which is information for managing tiles corresponding to sub-samples consisting of a single or multiple consecutive data units of a bit stream stored in a file as a sample, and a file generation unit that generates a file that stores the bit stream and tile management information.

[0055] For example, in an information processing method, tile identification information indicating tiles of a point cloud corresponding to data units of a bit stream of the point cloud, which represents a three-dimensional object as a collection of points, is used to generate tile management information, which is information for managing tiles corresponding to sub-samples consisting of a single or multiple consecutive data units of the bit stream stored in a file as a sample, and a file is generated to store the bit stream and tile management information.

[0056] Furthermore, for example, an information processing device may be provided with an extraction unit that extracts from a file the portion of the bitstream necessary to reproduce a desired tile, based on tile management information, which is information for managing tiles corresponding to subsamples stored in the file using tile identification information that indicates tiles of the point cloud corresponding to subsamples composed of a single or multiple consecutive data units of the bitstream, stored in the file together with the bitstream of a point cloud that represents a three-dimensional object as a collection of points.

[0057] For example, in an information processing method, a portion of a bitstream necessary to reproduce a desired tile is extracted from a file based on tile management information, which is information for managing tiles corresponding to subsamples stored in the file using tile identification information indicating tiles of the point cloud corresponding to subsamples composed of a single or multiple consecutive data units of the bitstream, which is stored in the file together with the bitstream of a point cloud that represents a three-dimensional object as a collection of points.

[0058] In this way, it is possible to extract and decode information necessary for playing back a desired tile based on the tile identification information managed by the tile management information, and generate presentation information. This reduces the amount of processing (parsing, etc.) of unnecessary information, thereby suppressing an increase in the load of the playback process.

[0059] Use cases for G-PCC bitstreams include encoding large-scale point cloud data, such as point cloud map data or virtual assets in film production (digitized versions of real film sets).

[0060] Such large point clouds are primarily intended for local playback. Because clients typically have limited cache size, they do not decode the entire G-PCC bitstream, but instead decode and render only the necessary regions.

[0061] In order to reduce the processing load during this repeated decoding and rendering process, it is expected that processes such as decoding and rendering only the partial point cloud of the area that is visible depending on the viewpoint position, or decoding and rendering partial point clouds of nearby areas at a high LoD (high resolution) and decoding and rendering partial point clouds of distant areas at a low LoD (low resolution) will be possible.

[0062] To decode and render only the partial point cloud of the region that is visible depending on the viewpoint, access to only a portion of the G-PCC bitstream is required.

[0063] Large-scale point clouds are primarily intended for local playback (playing back only a portion of the whole, rather than the whole). Therefore, as described above, by extracting and decoding the information required to play back the desired tile based on the tile identification information managed by the tile management information, and generating presentation information, it is possible to suppress an increase in the load on the client.

[0064] <2-1. Single Track> G-PCC files have a structure in which geometry and attributes are stored in one track (also called a single track encapsulation structure), and a structure in which geometry and attributes are stored in different tracks (also called a multi-track encapsulation structure). Here, we will explain how tile identification information is stored in a single track (Method 1), as shown in the second row from the top of the table in Figure 8.

[0065] Fig. 9 is a diagram showing an example of the main structure of a G-PCC file in the case of a single track. As shown in Fig. 9, in the case of a single track, both geometry data units and attribute data units can be stored in a sample. Note that a sample that stores G-PCC data is also called a G-PCC sample.

[0066] <2-1-1.Signaling by SubSampleInformationBox> The tile management information may be generated for each track of the file and may include a list of tile identification information corresponding to the subsamples stored in that track.

[0067] Furthermore, the G-PCC file may be an ISOBMFF file, and the tile management information may be stored in a box that stores information about subsamples in the moov box of the G-PCC file. For example, as shown in FIG. 10, the moov box of the G-PCC file contains a SubSampleInformationBox ('subs') defined in ISO / IEC 23090-18. As shown in the third row from the top of the table in FIG. 8, the tile management information (a list of tile identification information) may be stored in this SubSampleInformationBox (Method 1-1). The SubSampleInformationBox may also be stored in the moof box of the G-PCC file.

[0068] For example, when an already encoded G-PCC bitstream is stored in an ISOBMFF, a file generation device that generates the ISOBMFF parses the G-PCC bitstream to extract tile identification information, etc., and stores the extracted tile identification information, etc. in the SubSampleInformationBox as tile management information. Also, when encoding and filing are performed in a single process, the file generation device acquires the tile identification information, etc. from the encoder and stores the acquired tile identification information, etc. in the SubSampleInformationBox as tile management information. In this way, tile management information (a list of tile identification information) can be stored in the SubSampleInformationBox.

[0069] When playing back, the subsample (i.e., geometry data unit or attribute data unit) corresponding to the desired tile can be easily identified by referencing the tile management information in this SubSampleInformationBox. Therefore, it is possible to decode and play back only the data of the desired tile without increasing unnecessary parsing processes, etc. This makes it possible to suppress an increase in the load of the playback process.

[0070] An example of the syntax of this SubSampleInformationBox is shown in Fig. 11. As shown in Fig. 11, codec-specific parameters that can store any parameters are provided in SubSampleInformationBox. These codec-specific parameters may be extended to store tile management information (a list of tile identification information). By using an existing box to store tile management information (a list of tile identification information) in this way, compatibility with existing standards can be improved. This makes it possible to create files that can be processed by general-purpose encoders and decoders.

[0071] <2-1-1-1. Subsampling per tile> As shown in the fourth row from the top of the table in Fig. 8, the G-PCC data in a G-PCC sample may be subsampled for each tile (method 1-1-1). In the example of Fig. 10, the G-PCC data stored in the sample is subsampled for each tile. That is, a single or multiple consecutive data units of the bitstream, each consisting of geometry data units or attribute data units, or both, belonging to the same tile of the bitstream, stored in the sample, may be set as a subsample. Then, the tile management information may include information associating such subsamples with tile identification information corresponding to the geometry data units included in the subsample.

[0072] As mentioned above, a slice contains one geometry data unit and a tile is made up of one or more slices, so in the case of a single track, the subsample made up of this data unit can also be said to be made up of one or more consecutive data units containing one or more geometry data units.

[0073] As shown in FIG. 10, a single or a series of parameter sets and tile inventories are also set as subsamples.

[0074] In this configuration, as shown in FIG. 10, the tile management information described above stores, for each subsample, information (data_units_for_tile) indicating whether the subsample is a tile. If data_units_for_tile is false (e.g., 0), this indicates that the subsample is composed of a single or multiple consecutive parameter sets and a tile inventory. If data_units_for_tile is true (e.g., 1), this indicates that the subsample is composed of a single or multiple consecutive data units that constitute the same tile. For a subsample for which data_units_for_tile is true, tile identification information (tile_id) is also stored. The tile_id indicates the tile corresponding to the subsample (i.e., the tile to which the data units that constitute the subsample belong). In the example of FIG. 10, the data units that constitute a tile are grouped together as subsamples, and therefore the association between the geometry data unit and the attribute data unit is omitted in the tile management information (they are associated because they are included in the same subsample).

[0075] Fig. 12 is a diagram showing an example of the syntax of codec specific parameters. As shown in Fig. 12, in the codec specific parameters, data_units_for_tile and tile_id of a subsample are stored as tile management information. That is, in the tile management information, for each subsample, information (data_units_for_tile) indicating whether the subsample is a tile or not is stored, and if data_units_for_tile is true, tile identification information (tile_id) is also stored.

[0076] This configuration makes it easy to control whether to decode on a tile-by-tile basis, and also makes it easy to link geometry data with attribute data.

[0077] As shown in the example of FIG. 12, codec-specific parameters may be extended using flags. In this case, the contents of the codec-specific parameters can be switched depending on the value of flags. Therefore, tile management information (such as data_units_for_tile and tile_id) can be stored while leaving the existing parameters. This improves compatibility with existing standards. This makes it possible to create files that can be processed by general-purpose encoders and decoders.

[0078] <2-1-1-2. Subsampling per slice> Note that subsamples may be set for each slice. That is, as shown in the fifth row from the top of the table in FIG. 8, the G-PCC data in a G-PCC sample may be subsampled for each slice (method 1-1-2). In the example of FIG. 13, the G-PCC data stored in the sample is subsampled for each slice. That is, a single or multiple consecutive data units of a bitstream, each consisting of geometry data units or attribute data units, or both, belonging to the same slice of the bitstream, stored in the sample, may be set as a subsample. In this case, the tile identification information indicates the tile to which the slice corresponding to the subsample belongs. The tile management information may include information that associates, with such a subsample, tile identification information corresponding to the geometry data units included in the subsample.

[0079] As mentioned above, a slice contains one geometry data unit, so in the case of a single track, a sub-sample consisting of this data unit can also be said to be composed of a single or multiple consecutive data units containing one geometry data unit.

[0080] As shown in FIG. 13, a single or multiple consecutive parameter sets and tile inventories are also set as subsamples.

[0081] In this configuration, as shown in FIG. 13, the tile management information described above stores, for each subsample, information (data_units_for_slice) indicating whether the subsample is a slice. When data_units_for_slice is false (e.g., 0), the subsample is composed of a single or multiple consecutive parameter sets and a tile inventory. When data_units_for_slice is true (e.g., 1), the subsample is composed of a single or multiple consecutive data units that constitute the same slice. For a subsample for which data_units_for_slice is true, tile identification information (tile_id) is also stored. The tile_id indicates the tile to which the slice corresponding to the subsample belongs (i.e., the tile to which the data units that constitute the subsample belong). In the example of FIG. 13, the data units that constitute a slice are grouped together as subsamples, so the link between the geometry data unit and the attribute data unit is omitted in the tile management information (they are linked because they are included in the same subsample).

[0082] Fig. 14 is a diagram showing an example of the syntax of the codec specific parameters in this case. As shown in Fig. 14, in the codec specific parameters, the data_units_for_slice and tile_id of the subsample are stored as tile management information. That is, in the tile management information, for each subsample, information (data_units_for_slice) indicating whether the subsample is a slice is stored, and if data_units_for_slice is true, tile identification information (tile_id) is also stored.

[0083] This configuration makes it easy to control whether to decode on a slice-by-slice basis, and also makes it easy to link geometry data with attribute data.

[0084] In this case, too, codec-specific parameters may be extended using flags, as in the example shown in Figure 14. This improves compatibility with existing standards, making it possible to create files that can be processed by general-purpose encoders and decoders.

[0085] <2-1-1-3. Subsample per data unit> Note that subsamples may be set for each data unit. That is, as shown in the sixth row from the top of the table in FIG. 8, the G-PCC data in a G-PCC sample may be subsampled for each data unit (method 1-1-3). In the example of FIG. 15, the G-PCC data stored in a sample is subsampled for each data unit. That is, a single data unit of geometry or attribute of a bitstream stored in a sample may be set as a subsample. In this case, the tile identification information indicates the tile to which the subsample (data unit) belongs. The tile management information may include information that associates, for a subsample consisting of a data unit of geometry, tile identification information and slice identification information corresponding to the data unit of the geometry, and information that associates, for a subsample consisting of a data unit of an attribute, slice identification information corresponding to the data unit of the attribute. Note that the slice identification information is information that indicates a slice of a point cloud corresponding to a data unit of the bitstream.

[0086] As shown in FIG. 15, each parameter set and tile inventory is also set as a subsample.

[0087] In this configuration, in the tile management information described above, as shown in Fig. 15, for a subsample of a geometry data unit, a payload type, tile ID, and slice ID are stored. Also, for a subsample of an attribute data unit, a payload type and slice ID are stored. Furthermore, for the other subsamples, a payload type is stored.

[0088] The payload type indicates the type of data that makes up the subsample (for example, whether it is a geometry data unit, an attribute data unit, or other). The tile_id indicates the tile corresponding to the subsample (i.e., the tile to which the data units that make up the subsample belong). The slice id indicates the slice corresponding to the subsample (i.e., the slice to which the data units that make up the subsample belong). In this case, the geometry data unit and the attribute data unit are linked by the slice identification information.

[0089] FIG. 16 is a diagram showing an example of the syntax of codec specific parameters in this case. As shown in FIG. 16, in the codec specific parameters, as tile management information, payload type, tile_id, and geom_slice_id are stored for a subsample of a geometry data unit. geom_slice_id is slice identification information indicating the slice that the geometry data unit constitutes. Furthermore, payload type and attr_slice_id are stored for a subsample of an attribute data unit. attr_slice_id is slice identification information indicating the slice that the attribute data unit constitutes. Furthermore, payload type is stored for the other subsamples.

[0090] By adopting such a configuration, it is possible to easily control whether or not to decode in units of slices.

[0091] In this case, too, codec-specific parameters may be extended using flags, as in the example shown in Figure 16. This improves compatibility with existing standards, making it possible to create files that can be processed by general-purpose encoders and decoders.

[0092] <2-1-2.Signaling by SubSampleItemProperty> In the above description, SubSampleInformationBox is expanded to store tile management information (tile identification information), but instead of SubSampleInformationBox, SubSampleItemProperty may be expanded to store tile management information (tile identification information). The expansion method is the same as for SubSampleInformationBox described above. Storing tile management information in SubSampleItemProperty can also achieve the same effect for still images.

[0093] <2-1-3. Timed metadata signaling> As shown in the seventh row from the top of the table in Fig. 8, tile identification information may be stored in timed metadata (Method 1-2). As shown in Fig. 17, the timed metadata track is linked to the G-PCC track by a track reference ('gsli'). This method can be applied to each of the methods described above. In other words, the information stored in each method may be stored in timed metadata.

[0094] <2-2. Multi-track> This technology can also be applied when the G-PCC file has a multi-track (multi-track encapsulation structure). As shown in the eighth row from the top of the table in Fig. 8, the storage of tile identification information in the case of a multi-track will be explained (Method 2). In the case of a multi-track, the G-PCC file structure is as shown in Fig. 18. In other words, the geometry data unit and the attribute data unit are stored in different tracks. Therefore, as shown in Fig. 19, tile management information can be stored in each track.

[0095] In other words, in a G-PCC file, geometry data units and attribute data units may be stored in different tracks, and each track may store tile management information that manages tile identification information corresponding to the subsamples within that track.

[0096] The storage method for each track is the same as for a single track. Therefore, the same effects can be obtained in the case of a multi-track as in the case of a single track. Note that the methods described for the single track can also be applied to the multi-track.

[0097] For example, in the case of method 1-1-1, in each of the geometry track and attribute track, data_units_for_tile is stored as tile management information for each subsample in that track. For the subsamples that make up a tile, tile_id is also stored.

[0098] In the case of method 1-1-2, data_units_for_slice is stored as tile management information for each subsample in the geometry track and attribute track. For the subsamples that make up a slice, tile_id is also stored.

[0099] Furthermore, in the case of method 1-1-3, in the geometry track, the payload type is stored as tile management information for each subsample in the track. For subsamples of geometry data units, the tile_id and slice id are also stored. In the attribute track, the payload type is stored as tile management information for each subsample in the track. For subsamples of attribute data units, the slice id is also stored.

[0100] In a multi-track, data_units_for_tile and data_units_for_slice are true (e.g., 1) when the sub-samples are geometry data units in the case of a geometry track, and when the sub-samples are attribute data units that make up the same slice in the case of an attribute track.

[0101] As shown in Fig. 20, in the case of multi-track, tile identification information may be stored in timed metadata, just like in the case of single track. In the case of multi-track, the same effect as in the case of single track can be obtained.

[0102] <2-3. Matryoshka Media Container> Although the above has described an example in which ISOBMFF is used as the file format, the file in which the G-PCC bitstream is stored may be any file format other than ISOBMFF. For example, as shown in the bottom row of the table in Fig. 8, the G-PCC bitstream may be stored in a Matroska Media Container (Method 3). An example of the main configuration of a Matroska Media Container is shown in Fig. 21.

[0103] In this case, for example, the tile management information (tile identification information) may be stored as a newly defined element under the Track Entry element. Also, when the tile management information (tile identification information) is stored in timed metadata, the timed metadata may be stored in a Track entry different from the Track entry in which the G-PCC bitstream is stored.

[0104] 3. First Embodiment <File generation device> An encoding device will be described. The present technology (each method) described above can be applied to any device. Fig. 22 is a block diagram showing an example of the configuration of a file generation device, which is one aspect of an information processing device to which the present technology is applied. The file generation device 300 shown in Fig. 22 is a device that encodes point cloud data by applying G-PCC and stores the G-PCC bitstream generated by the encoding in ISOBMFF.

[0105] The file generation device 300 applies the above-described present technology and stores the G-PCC bitstream in the ISOBMFF so as to enable partial access. That is, the file generation device 300 stores the tile identification information of each subsample in the G-PCC file as tile management information.

[0106] Note that Fig. 22 shows the main processing units, data flows, etc., and does not necessarily include everything shown in Fig. 22. In other words, in file generation device 300, there may be processing units that are not shown as blocks in Fig. 22, and there may be processing or data flows that are not shown as arrows, etc. in Fig. 22.

[0107] 22 , the file generation device 300 includes an extraction unit 311, an encoding unit 312, a bitstream generation unit 313, a tile management information generation unit 314, and a file generation unit 315. The encoding unit 312 also includes a geometry encoding unit 321, an attribute encoding unit 322, and a metadata generation unit 323.

[0108] The extraction unit 311 extracts geometry data and attribute data from the point cloud data input to the file generation device 300. The extraction unit 311 supplies the extracted geometry data to a geometry encoding unit 321 of the encoding unit 312. The extraction unit 311 also supplies the extracted attribute data to an attribute encoding unit 322 of the encoding unit 312.

[0109] The encoding unit 312 encodes the point cloud data. The geometry encoding unit 321 encodes the geometry data supplied from the extraction unit 311 to generate a geometry bitstream. The geometry encoding unit 321 supplies the generated geometry bitstream to the metadata generation unit 323. The geometry encoding unit 321 also supplies the generated geometry bitstream to the attribute encoding unit 322.

[0110] The attribute encoding unit 322 encodes the attribute data supplied from the extraction unit 311 to generate an attribute bitstream. The attribute encoding unit 322 supplies the generated attribute bitstream to the metadata generation unit 323.

[0111] The metadata generation unit 323 generates metadata by referring to the supplied geometry bitstream and attribute bitstream, and supplies the generated metadata to the bitstream generation unit 313 together with the geometry bitstream and attribute bitstream.

[0112] The bitstream generator 313 multiplexes the supplied geometry bitstream, attribute bitstream, and metadata to generate a G-PCC bitstream. The bitstream generator 313 supplies the generated G-PCC bitstream to the tile management information generator 314.

[0113] The tile management information generator 314 applies the present technology described above in <2. Signaling of Tile Identification Information>, and uses the tile identification information indicating the tile of the point cloud corresponding to the data unit of the supplied G-PCC bitstream to generate tile management information, which is information for managing tiles corresponding to sub-samples consisting of one or multiple consecutive data units of the bitstream that are stored as samples in a file. The tile management information generator 314 supplies the tile management information to the file generator 315 together with the G-PCC bitstream.

[0114] The file generation unit 315 applies the present technology described above in <2. Signaling of Tile Identification Information> to generate a G-PCC file that stores the supplied G-PCC bitstream and tile management information (tile identification information). The file generation unit 315 outputs the G-PCC file generated as described above to the outside of the file generation device 300.

[0115] For example, when subsampling is performed for each tile, the tile management information generation unit 314 generates tile management information (a list of tile identification information) according to the syntax shown in Fig. 12. The file generation unit 315 stores the tile management information in the codec specific parameters of the SubSampleInformationBox.

[0116] Furthermore, when subsampling is performed for each slice, the tile management information generation unit 314 generates tile management information (a list of tile identification information) according to the syntax shown in Fig. 14. The file generation unit 315 stores the tile management information in the codec specific parameters of the SubSampleInformationBox.

[0117] Furthermore, when subsampling is performed for each data unit, the tile management information generation unit 314 generates tile management information (a list of tile identification information) according to the syntax shown in Fig. 16. The file generation unit 315 stores the tile management information in the codec specific parameters of the SubSampleInformationBox.

[0118] The file generation unit 315 can store the tile management information in SubSampleItemProperty or in timed metadata. Also, as described above in <2. Signaling of Tile Identification Information>, in the case of multi-track, the tile management information generation unit 314 can generate tile management information, and the file generation unit 315 can store the tile management information in a file.

[0119] By doing so, as described above in <2. Signaling of tile identification information>, it is possible to suppress an increase in the load of playback processing.

[0120] <File generation process flow> An example of the flow of the file generation process executed by this file generation device 300 will be described with reference to the flowchart of FIG.

[0121] When the file generation process starts, the extraction unit 311 of the file generation device 300 extracts geometry and attributes from the point cloud in step S301.

[0122] In step S302, the encoding unit 312 encodes the geometry and attributes extracted in step S301 to generate a geometry bitstream and an attribute bitstream, and further generates metadata for the encoded geometry and attribute bitstream.

[0123] In step S303, the bitstream generation unit 313 multiplexes the geometry bitstream, attribute bitstream, and metadata generated in step S302 to generate a G-PCC bitstream.

[0124] In step S304, the tile management information generation unit 314 applies the present technology described above in <2. Signaling of tile identification information> to generate tile management information that manages the tile identification information included in the G-PCC bitstream generated in step S303.

[0125] In step S305, the file generation unit 315 generates other information, and applies the above-described present technology to generate a G-PCC file that stores a G-PCC bitstream and tile management information.

[0126] When the process of step S305 ends, the file generation process ends.

[0127] As described above, in the file generation process, the file generation device 300 applies the present technology described in <2. Signaling of tile identification information> and stores the tile identification information in the G-PCC file. By doing so, it is possible to reduce the processing (decoding, etc.) of unnecessary information, and to suppress an increase in the load of the playback process.

[0128] 4. Second Embodiment <Playback device> Fig. 24 is a block diagram showing an example of the configuration of a playback device, which is one aspect of an information processing device to which the present technology is applied. The playback device 400 shown in Fig. 24 is a device that decodes a G-PCC file, constructs a point cloud, and performs rendering to generate presentation information. In this case, the playback device 400 applies the above-described present technology to extract information required to play a desired tile in the point cloud from the G-PCC file, and decodes and plays back the extracted information. In other words, the playback device 400 can decode and play back only a portion of the point cloud.

[0129] Note that Fig. 24 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, in playback device 400, there may be processing units that are not shown as blocks in Fig. 24, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 24.

[0130] 24, the playback device 400 has a control unit 401, a file acquisition unit 411, a playback processing unit 412, and a presentation processing unit 413. The playback processing unit 412 has a file processing unit 421, a decoding unit 422, and a presentation information generation unit 423.

[0131] The control unit 401 controls each processing unit in the playback device 400. The file acquisition unit 411 acquires a G-PCC file that stores the point cloud to be played back, and supplies it to (the file processing unit 421 of) the playback processing unit 412. The playback processing unit 412 performs processing related to the playback of the point cloud stored in the supplied G-PCC file.

[0132] The file processing unit 421 of the playback processing unit 412 acquires the G-PCC file supplied from the file acquisition unit 411 and extracts a bitstream from the G-PCC file. At this time, the file processing unit 421 applies the present technology described above in <2. Signaling of Tile Identification Information> to extract only the bitstream necessary for playing back the desired tile. The file processing unit 421 supplies the extracted bitstream to the decoding unit 422. The decoding unit 422 decodes the supplied bitstream and generates geometry and attribute data. The decoding unit 422 supplies the generated geometry and attribute data to the presentation information generation unit 423. The presentation information generation unit 423 constructs a point cloud using the supplied geometry and attribute data and generates presentation information for presenting (e.g., displaying) the point cloud. For example, the presentation information generation unit 423 performs rendering using the point cloud and generates a display image of the point cloud viewed from a predetermined viewpoint as presentation information. The presentation information generation unit 423 supplies the presentation information thus generated to the presentation processing unit 413 .

[0133] The presentation processing unit 413 performs processing to present the supplied presentation information. For example, the presentation processing unit 413 supplies the presentation information to a display device or the like external to the playback device 400, and causes the presentation information to be presented.

[0134] Fig. 25 is a block diagram showing an example of the main configuration of the playback processing unit 412. As shown in Fig. 25, the file processing unit 421 has a bitstream extraction unit 431. The decoding unit 422 has a geometry decoding unit 441 and an attribute decoding unit 442. The presentation information generation unit 423 has a point cloud construction unit 451 and a presentation processing unit 452.

[0135] The bitstream extraction unit 431 applies the technology described above in <2. Signaling of tile identification information>, refers to the tile management information contained in the supplied G-PCC file, and extracts the bitstreams required to play the desired tile from the G-PCC file (i.e., the geometry bitstream and attribute bitstream corresponding to the tile) based on the tile management information (the tile identification information contained in the tile management information).

[0136] For example, the bitstream extraction unit 431 identifies tile identification information corresponding to a desired tile based on information such as a tile inventory. Then, the bitstream extraction unit 431 identifies a subsample corresponding to the tile identification information corresponding to the desired tile by referring to tile management information stored in codec specific parameters of the SubSampleInformationBox. Then, the bitstream extraction unit 431 extracts the identified subsample bitstream.

[0137] For example, if the G-PCC data in a G-PCC sample is subsampled for each tile, the bitstream extraction unit 431 analyzes the tile management information stored in the codec specific parameters of the SubSampleInformationBox, etc., based on the syntax shown in Figure 12.

[0138] In addition, if the G-PCC data in the G-PCC sample is subsampled for each slice, the bitstream extraction unit 431 analyzes the tile management information stored in the codec specific parameters of the SubSampleInformationBox, etc., based on the syntax shown in Figure 14.

[0139] Furthermore, if the G-PCC data within the G-PCC sample is subsampled for each data unit, the bitstream extraction unit 431 analyzes the tile management information stored in the codec specific parameters of the SubSampleInformationBox, etc., based on the syntax shown in Figure 16.

[0140] If the tile management information is stored in SubSampleItemProperty, the bitstream extraction unit 431 refers to the SubSampleItemProperty. If the tile management information is stored in timed metadata, the bitstream extraction unit 431 refers to the timed metadata. The G-PCC file may be multi-track.

[0141] The bitstream extraction unit 431 supplies the extracted geometry bitstream to the geometry decoding unit 441. The bitstream extraction unit 431 also supplies the extracted attribute bitstream to the attribute decoding unit 442.

[0142] The geometry decoding unit 441 decodes the supplied geometry bitstream to generate geometry data. The geometry decoding unit 441 supplies the generated geometry data to the point cloud construction unit 451. The attribute decoding unit 442 decodes the supplied attribute bitstream to generate attribute data. The attribute decoding unit 442 supplies the generated attribute data to the point cloud construction unit 451.

[0143] The point cloud construction unit 451 constructs a point cloud using the supplied geometry and attribute data. That is, the point cloud construction unit 451 can construct desired tiles of the point cloud. The point cloud construction unit 451 supplies the constructed point cloud data to the presentation processing unit 452.

[0144] The presentation processing unit 452 generates presentation information using the supplied point cloud data. The presentation processing unit 452 supplies the generated presentation information to the presentation processing unit 413.

[0145] With this configuration, the playback device 400 can more easily extract, decode, construct, and present only desired tiles based on the tile management information (tile identification information) stored in the G-PCC file, without having to parse the entire bitstream, thereby reducing the load of playback processing.

[0146] <Recycling process flow> An example of the flow of the playback process executed by this playback device 400 will be described with reference to the flowchart of FIG.

[0147] When the playback process starts, the file acquisition unit 411 of the playback device 400 acquires the G-PCC file to be played back in step S401.

[0148] In step S402, the bitstream extraction unit 431 extracts parameter sets and data units necessary for decoding and displaying a desired tile based on the tile management information (tile identification information) stored in the G-PCC file acquired in step S401. That is, the bitstream extraction unit 431 applies the present technology described above in <2. Signaling of Tile Identification Information>, and extracts a geometry bitstream and attribute bitstream corresponding to the desired tile from the G-PCC file.

[0149] For example, the bitstream extraction unit 431 identifies and extracts a sequence parameter set, a geometry parameter set, an attribute parameter set, and a tile inventory based on the payload type stored in the SubSampleInformationBox of the G-PCC file. The bitstream extraction unit 431 determines a decoding method for each tile based on the tile position information indicated in the extracted tile inventory. The bitstream extraction unit 431 identifies and extracts the subsamples that make up the tile to be decoded (i.e., the geometry data units and attribute data units that make up the tile) based on the tile management information (tile identification information) stored in the SubSampleInformationBox of the G-PCC file.

[0150] In step S403, the geometry decoding unit 441 of the decoding unit 422 decodes the geometry bitstream extracted in step S402 to generate geometry data, and the attribute decoding unit 442 decodes the attribute bitstream extracted in step S402 to generate attribute data.

[0151] In step S404, the point cloud construction unit 451 constructs a point cloud using the geometry and attribute data generated in step S403. That is, the point cloud construction unit 451 can construct a desired tile (a part of the point cloud).

[0152] In step S405, the presentation processing unit 452 generates presentation information by performing rendering using the point cloud constructed in step S404, etc. The presentation processing unit 413 supplies the presentation information to an external device outside the playback device 400, where it is presented.

[0153] When the process of step S405 ends, the playback process ends.

[0154] As described above, in the playback process, the playback device 400 applies the present technology described in <2. Signaling of Tile Identification Information>, extracts information corresponding to a desired tile using the tile identification information stored in the G-PCC file, and plays back the extracted information. In this way, it is possible to reduce the processing (decoding, etc.) of unnecessary information, and to suppress an increase in the load of the playback process.

[0155] <5. Notes> <Computer> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, etc., that can execute various functions by installing various programs.

[0156] FIG. 27 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0157] In a computer 900 shown in FIG. 27, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0158] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0159] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0160] In a computer configured as above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0161] The program executed by the computer can be applied by recording it on removable media 921 such as package media, for example. In this case, the program can be installed in storage unit 913 via input / output interface 910 by inserting removable media 921 into drive 915.

[0162] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0163] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0164] <Applicable targets of this technology> While the above describes the application of this technology to the encoding and decoding of point cloud data, this technology is not limited to these examples and can be applied to the encoding and decoding of 3D data of any standard. In other words, as long as it does not conflict with the above-described technology, various processes such as encoding and decoding methods and specifications of various data such as 3D data and metadata are arbitrary. Furthermore, as long as it does not conflict with the above-described technology, some of the above-described processes and specifications may be omitted.

[0165] Furthermore, the present technology can be applied to any configuration, for example, various electronic devices.

[0166] Furthermore, for example, the present technology can also be implemented as a part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, or a set in which other functions are added to a unit (e.g., a video set).

[0167] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, AV (Audio Visual) equipment, a portable information processing terminal, or an IoT (Internet of Things) device.

[0168] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0169] <Fields and applications where this technology can be applied> Systems, devices, processing units, etc. to which the present technology is applied can be used in any field, such as transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. In addition, the applications thereof are also arbitrary.

[0170] For example, the present technology can be applied to systems and devices used to provide viewing content, etc. Furthermore, for example, the present technology can also be applied to systems and devices used for transportation, such as monitoring traffic conditions and controlling automatic driving. Furthermore, for example, the present technology can also be applied to systems and devices used for security. Furthermore, for example, the present technology can also be applied to systems and devices used for automatic control of machines, etc. Furthermore, for example, the present technology can also be applied to systems and devices used for agriculture and livestock farming. Furthermore, for example, the present technology can also be applied to systems and devices used to monitor natural conditions, such as volcanoes, forests, and oceans, and wildlife. Furthermore, for example, the present technology can also be applied to systems and devices used for sports.

[0171] <Other> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. In other words, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be assumed not only to include the identification information in the bit stream, but also to include difference information of the identification information relative to certain reference information in the bit stream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also difference information relative to the reference information.

[0172] Furthermore, various types of information (metadata, etc.) related to the coded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. In other words, data associated with each other may be combined into one piece of data or may be individual pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Also, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may refer to only a portion of the data, rather than the entire data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0173] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0174] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0175] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0176] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0177] Also, for example, each step of a single flowchart may be executed by one device, or may be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes may be executed by one device, or may be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0178] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0179] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0180] The present technology can also be configured as follows. (1) A tile management information generating unit that generates tile management information, which is information for managing tiles corresponding to sub-samples of a bit stream of a point cloud that represents a three-dimensional object as a set of points, using tile identification information indicating tiles of the point cloud that correspond to data units of the bit stream of the point cloud, the sub-samples being stored as samples in a file and consisting of a single or multiple consecutive data units of the bit stream; a file generation unit that generates the file that stores the bitstream and the tile management information; An information processing device comprising: (2) The tile management information is generated for each track of the file and includes a list of the tile identification information corresponding to the subsamples stored in the track. The information processing device described in (1). (3) The file is an ISOBMFF (International Organization for Standardization Base Media File Format) file, The tile management information is stored in a box that stores information about the subsamples in the moov box or moof box of the file. (2) An information processing device according to the present invention. (4) the sub-samples are composed of geometry data units or attribute data units, or both, that belong to the same tile of the bitstream; The tile management information includes information that associates the tile identification information corresponding to the data unit of the geometry included in the subsample with the subsample. (3) An information processing device according to the present invention. (5) the sub-samples are composed of geometry data units or attribute data units, or both, that belong to the same slice of the bitstream; The tile management information includes information that associates the tile identification information corresponding to the data unit of the geometry included in the subsample with the subsample. (3) An information processing device according to the present invention. (6) the sub-sample is composed of a single data unit of geometry or attribute of the bitstream; The tile management information includes: information associating, for the subsample consisting of the data unit of the geometry, the tile identification information corresponding to the data unit of the geometry and slice identification information indicating a slice of the point cloud corresponding to the data unit of the bitstream, which corresponds to the data unit of the geometry; information associating the slice identification information corresponding to the data unit of the attribute with the subsample consisting of the data unit of the attribute; Contains (3) An information processing device according to the present invention. (7) The tile management information is stored in timed metadata of the file. An information processing device according to any one of (3) to (6). (8) The file generation unit stores the data unit of geometry and the data unit of attributes in different tracks of the file; The tile management information generating unit generates the tile management information for each of the tracks. An information processing device according to any one of (1) to (7). (9) An encoding unit that encodes the point cloud data and generates the bit stream, The file generation unit generates the file that stores the bitstream generated by the encoding unit. An information processing device according to any one of (1) to (8). (10) Using tile identification information indicating tiles of a point cloud corresponding to data units of a bit stream of the point cloud, which represents a three-dimensional object as a set of points, generating tile management information, which is information for managing the tiles corresponding to sub-samples of the bit stream, each of which is composed of a single or consecutive multiple data units, stored as a sample in a file; Generate the file that stores the bitstream and the tile management information Information processing methods.

[0181] (11) An extracting unit that extracts from a file a portion of a bitstream necessary for reproducing a desired tile of the bitstream, based on tile management information that is information for managing a tile of the point cloud corresponding to a subsample configured by a single or multiple consecutive data units of the bitstream, stored in the file together with the bitstream of the point cloud that represents a three-dimensional object as a set of points. An information processing device comprising: (12) The tile management information is generated for each track of the file and includes a list of the tile identification information corresponding to the subsamples stored in the track; The extraction unit identifies the subsample corresponding to the desired tile based on the list, and extracts the identified subsample. (11) An information processing device according to (11). (13) The file is an ISOBMFF (International Organization for Standardization Base Media File Format) file, The extraction unit identifies the subsample corresponding to the desired tile based on the list of tile management information stored in a moov box or a moof box of the file, and extracts the identified subsample. (12) An information processing device according to (12). (14) The sub-samples are composed of the geometry data units or the attribute data units, or both, that belong to the same tile of the bitstream; The tile management information includes information that associates the tile identification information corresponding to the data unit of the geometry included in the subsample with the subsample. (13) An information processing device according to (13). (15) The sub-samples are composed of the geometry data units or the attribute data units, or both, that belong to the same slice of the bitstream; The tile management information includes information that associates the tile identification information corresponding to the data unit of the geometry included in the subsample with the subsample. (13) An information processing device according to (13). (16) The sub-sample is composed of a single data unit of geometry or attribute of the bitstream; The tile management information includes: information associating, for the subsample consisting of the data unit of the geometry, the tile identification information corresponding to the data unit of the geometry and slice identification information indicating a slice of the point cloud corresponding to the data unit of the bitstream, which corresponds to the data unit of the geometry; information associating the slice identification information corresponding to the data unit of the attribute with the subsample consisting of the data unit of the attribute; Contains (13) An information processing device according to (13). (17) The tile management information is stored in timed metadata of the file. An information processing device according to any one of (13) to (16). (18) In the file, the data units of geometry and the data units of attributes are stored in different tracks, The extractor extracts, from the file, the portion of the bitstream necessary for playing the desired tile based on the tile management information for each of the tracks. An information processing device according to any one of (11) to (17). (19) The image processing apparatus further includes a decoding unit that decodes the portion of the bitstream extracted by the extraction unit that is necessary for reproducing the desired tile. An information processing device according to any one of (11) to (18). (20) Extracting from a file a portion of a bitstream necessary for reproducing a desired tile of the bitstream based on tile management information, which is information for managing the tile corresponding to a subsample stored in the file using tile identification information indicating a tile of the point cloud corresponding to a subsample configured by a single or multiple consecutive data units of the bitstream, which is stored in the file together with the bitstream of a point cloud that represents a three-dimensional object as a set of points. Information processing methods. [Explanation of symbols]

[0182] 300 file generation device, 311 extraction unit, 312 encoding unit, 313 bitstream generation unit, 314 tile management information generation unit, 315 file generation unit, 321 geometry encoding unit, 322 attribute encoding unit, 323 metadata generation unit, 400 playback device, 401 control unit, 411 file acquisition unit, 412 playback processing unit, 413 presentation processing unit, 421 file processing unit, 422 decoding unit, 423 presentation information generation unit, 431 bitstream extraction unit, 441 geometry decoding unit, 442 attribute decoding unit, 451 point cloud construction unit, 452 presentation processing unit< / isobmff>

Claims

1. a tile management information generating unit that generates, for each subsample of a bit stream of a point cloud that represents a three-dimensional object as a set of points, first flag information indicating whether the subsample is composed of a single or multiple consecutive data units that constitute the tile as tile management information for managing the tile of the point cloud that corresponds to the subsample, and if the first flag information is true, further generates identification information of the tile that corresponds to the subsample; a file generation unit that generates a file that stores the bitstream and the tile management information; An information processing device comprising:

2. If the first flag information is true, the sub-sample is composed of the data unit; If the first flag information is false, the sub-sample is configured by one or more parameter sets and the inventory of the tile. The information processing device according to claim 1 .

3. The file is an ISOBMFF (International Organization for Standardization Base Media File Format) file, The tile management information is stored in the SubSampleInformationBox in the moov box or moof box of the file. The information processing device according to claim 1 .

4. The tile management information generation unit generates the tile management information when second flag information, which is predetermined flag information defined in the ISOBMFF, is true. The information processing device according to claim 3 .

5. The tile management information generation unit generates, when the second flag information is false and the payload type is an attribute, identification information of the attribute as the tile management information. The information processing device according to claim 4 .

6. When the first flag information is true, the tile management information generation unit further generates identification information of a slice of the point cloud corresponding to the subsample. The information processing device according to claim 1 .

7. The tile management information is stored in a timed metadata track of the file that is linked to a track that stores the bitstream by a track reference. The information processing device according to claim 1 .

8. the file generation unit stores the geometry data unit and the attribute data unit in different tracks of the file; The tile management information generating unit generates the tile management information for each of the tracks. The information processing device according to claim 1 .

9. An encoding unit that encodes the point cloud data and generates the bit stream, The file generation unit generates the file that stores the bitstream generated by the encoding unit. The information processing device according to claim 1 .

10. The information processing device For each subsample of a bitstream of a point cloud that represents a three-dimensional object as a set of points, generate first flag information indicating whether the subsample is composed of a single or multiple consecutive data units that constitute the tile as tile management information for managing the tile of the point cloud that corresponds to the subsample, and if the first flag information is true, further generate identification information for the tile that corresponds to the subsample; Generate a file that stores the bitstream and the tile management information Information processing methods.

11. an extraction unit that extracts a portion of the bitstream necessary for reproducing a desired tile, based on first flag information included in tile management information that manages tiles of a point cloud corresponding to a subsample of the bitstream of the point cloud, the subsample indicating whether the subsample is composed of a single or multiple consecutive data units that constitute the tile, and identification information of the tile corresponding to the subsample that is further included in the tile management information when the first flag information is true; An information processing device comprising:

12. If the first flag information is true, the sub-sample is composed of the data unit; If the first flag information is false, the sub-sample is configured by one or more parameter sets and the inventory of the tile. The information processing device according to claim 11.

13. the file storing the bitstream and the tile management information is an ISOBMFF (International Organization for Standardization Base Media File Format) file, The tile management information is stored in the SubSampleInformationBox in the moov box or moof box of the file. The information processing device according to claim 11.

14. When second flag information, which is predetermined flag information defined in the ISOBMFF, is true, the extraction unit extracts a portion necessary for playing the tile based on the first flag information and identification information of the tile. The information processing device according to claim 13.

15. When the second flag information is false and the payload type is an attribute, the extractor extracts a portion necessary for playing back the tile based on identification information of the attribute. The information processing device according to claim 14.

16. If the first flag information is true, the extractor further extracts a portion necessary for reproducing the tile based on identification information of a slice of the point cloud corresponding to the subsample. The information processing device according to claim 11.

17. The extraction unit extracts a portion necessary for playing the tile based on the first flag information and identification information of the tile included in the tile management information stored in a timed metadata track linked to a track storing the bitstream by a track reference of the file storing the bitstream. The information processing device according to claim 11.

18. the data units of geometry and the data units of attributes are stored in different tracks of a file storing the bitstream, The extraction unit extracts the portion necessary for playing the tile from each track based on the first flag information and the tile identification information included in the tile management information. The information processing device according to claim 11.

19. a decoding unit configured to decode the portion of the bitstream extracted by the extraction unit that is necessary to reproduce the desired tile; The information processing device according to claim 11.

20. The information processing device A portion of the bitstream necessary for reproducing a desired tile is extracted based on first flag information, which is included in tile management information that manages tiles of a point cloud corresponding to a subsample of the bitstream of the point cloud, and which indicates whether the subsample is composed of a single or multiple consecutive data units that constitute the tile, and identification information of the tile corresponding to the subsample, which is further included in the tile management information when the first flag information is true. Information processing methods.

Images