Information processing device, information processing method, reproduction processing device and reproduction processing method

Abstract

An information processing device, an information processing method, a reproduction processing device, and a reproduction processing method are provided that enable an audio-visual experience that meets the requirements of viewers. An encoding unit (121) generates a bit stream consisting of a data area including one or more geometric slices by encoding point cloud data into a hierarchical structure having a stage corresponding to a level of detail (LoD) representing a quality level. A metadata generation unit (125) generates partial slice specific information for specifying a correspondence between the LoD and a partial slice that is a partial area in the geometric slice corresponding to the LoD. A file generation unit (126) generates a file including the partial slice specific information.

Description

Technical Field

[0001] The present invention relates to an information processing device, an information processing method, a reproduction processing device and a reproduction processing method. Background Art

[0002] Moving Picture Experts Group 4 (MPEG-4) is known as an international standardized technology for video compression. In addition, the International Organization for Standardization (ISO) Base Media File Format (ISOBMFF) is a file container format for MPEG-4. In other words, video and audio data compressed using MPEG-4 can be stored in an ISOBMFF file for delivery.

[0003] In addition, a method of using a point cloud, which is a collection of points having both position information and attribute information such as color or reflection in a three-dimensional space, is used as a method of representing a three-dimensional space. The point cloud is associated with a coding technique called geometry-based point cloud compression (G-PCC), which encodes the geometry indicating a three-dimensional shape and the attributes indicating attribute information separately. The G-PCC is defined in MPEG-I Geometry-based Point Cloud Compression Part 9 (ISO / IEC 23090-9). The encoded geometry data is called a geometry bitstream, and the compressed attribute data is called an attribute bitstream. The decoded geometry data is necessary to decode the attribute bitstream.

[0004] Furthermore, for the purpose of performing a reproduction process from a local storage device of a G-PCC bitstream which is a bitstream encoded by using G-PCC and improving delivery efficiency through a network, a method of storing a G-PCC bitstream in an ISOBMFF file is considered.

[0005] Furthermore, scalable coding of level of detail (LoD) as a hierarchical coding of image resolution is supported in the codec layer as octrees for geometry and as a coding option (scalable fix weight lifting) for attributes. Thus, decoding of geometry bitstream and attribute bitstream can be interrupted at the desired LoD. Point clouds can be reconstructed and rendered at LoDs below the LoD when the G-PCC bitstream is fully decoded.

[0006] Reference List

[0007] Non-patent literature

[0008] Non-patent literature 1: ISO / IEC CD 23090-9:2019 Information technology. MPEG-I (Coded Representation of Immersive Media), Part 9: Geometry-based Point Cloud Compression, 2019-07-01. Summary of the invention

[0009] Technical issues

[0010] However, the proposed LoD scalable decoding technology requires the additional implementation of the LoD scalable decoding function for each decoder separately. LoD scalable decoding depends on the implementation of the decoder as described above. Therefore, in some cases, some reproduction processing devices cannot perform LoD scalable decoding. Therefore, it is difficult to provide a viewing experience that meets the requirements of viewers.

[0011] Therefore, the present disclosure provides an information processing device, an information processing method, a reproduction processing device, and a reproduction processing method that can provide a viewing experience that meets the requirements of viewers.

[0012] Solution to the problem

[0013] According to the present disclosure, an encoding unit is configured to encode point cloud data into a hierarchical structure having a stage corresponding to a level of detail (LoD) indicating a quality level to generate a bitstream consisting of a data region including one or more geometric slices. A metadata generation unit is configured to generate partial slice specific information for specifying a correspondence between the LoD and a partial slice as a partial region in the geometric slice corresponding to the LoD. A file generation unit is configured to generate a file including the partial slice specific information. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a diagram showing the structure of a G-PCC bitstream.

[0015] Figure 2 is a diagram showing an image of a point cloud frame of a G-PCC bitstream.

[0016] Figure 3 is a diagram showing an octree encoding of geometry.

[0017] Figure 4 is a diagram shown to describe LoD scalable coding.

[0018] Figure 5 is a diagram showing a processing example of the first method using LoD scalable decoding.

[0019] Figure 6 is a diagram showing a processing example of the second method using LoD scalable decoding.

[0020] Figure 7 is a diagram showing an exemplary system configuration of a delivery system.

[0021] Figure 8 It is a block diagram of a file generating device.

[0022] Fig. 9 is a diagram for describing access information and hierarchy information according to the first embodiment.

[0023] Fig.10 is a diagram showing a SubSampleInformationBox according to the first embodiment.

[0024] Fig.11 is a diagram showing an example of storing a SubSampleInformationBox according to the first embodiment.

[0025] Fig.12 is a block diagram of a client device.

[0026] Fig.13 is a block diagram showing details of a decoding processing unit.

[0027] Fig.14 is a flowchart of a file generation process executed by the file generation device according to the first embodiment.

[0028] Fig.15 This is a flowchart of the reproduction process executed by the client device according to the first embodiment.

[0029] Fig.16 is a diagram showing an example of codec_specific_parameters according to the modification (1) of the first embodiment.

[0030] Fig.17 It is a diagram for describing access information and hierarchy information according to a modification example (2-1) of the first embodiment.

[0031] Fig.18 : is a diagram showing an example of time-corresponding metadata according to the modification example (2-1) of the first embodiment.

[0032] Fig.19 : is a diagram showing a storage example of time-corresponding metadata according to the modification (2-1) of the first embodiment.

[0033] Fig. 20 is a diagram showing an example of codec specific parameters according to the modification example (2-1) of the first embodiment.

[0034] Fig.21 : is a diagram showing an example of time-corresponding metadata according to the modification example (2-2) of the first embodiment.

[0035] Fig. 22 is a diagram showing a G-PCC bitstream having a hierarchical structure slice.

[0036] Fig.23 : is a diagram showing an example of time-corresponding metadata according to the modification (3-2) of the first embodiment.

[0037] Fig.24 3 is a diagram showing extraction of partial slices according to the modification example (3-2) of the first embodiment.

[0038] Fig.25 : is a diagram showing an example of time-corresponding metadata according to a modification (3-3) of the first embodiment.

[0039] Fig.26 is a diagram showing an exemplary configuration of a sample group.

[0040] Fig. 27 is a diagram showing an example of UnitMapEntry according to the second embodiment.

[0041] Fig.28 is a diagram showing an example of a sample group according to the second embodiment.

[0042] Fig.29 is a diagram showing an example of a three-dimensional image of a point cloud in which LoD is different for each tile.

[0043] Fig.30 : is a diagram showing an example of time-corresponding metadata according to the modification example (1) of the second embodiment.

[0044] Fig.31 : is a diagram showing the correspondence relationship between sample groups and subsamples according to the modification example (2-1) of the second embodiment.

[0045] Fig.32 : is a diagram showing UnitMapEntry when a depth value is stored in a sample group according to the modification example (2-1) of the second embodiment.

[0046] Fig.33 : is a diagram showing a state of a sample group storing depth values ​​according to a modification (2-1) of the second embodiment.

[0047] Fig.34 : is a diagram showing time-corresponding metadata when a sample group is used according to the modification example (2-2) of the second embodiment.

[0048] Fig.35 : is a diagram showing the correspondence relationship between each layer and LoD according to the modification example (2-2) of the second embodiment.

[0049] Fig.36 is a diagram showing a method of storing a G-PCC still image in an ISOBMFF file.

[0050] Fig.37 is a diagram showing an example of SubSampleItemProperty according to the third embodiment.

[0051] Fig.38 FIG. 1 is a diagram showing a method of storing time-correlated metadata using a G-PCC still image.

[0052] Fig.39 : is a diagram showing an example of time-corresponding metadata according to the modification example (1) of the third embodiment.

[0053] Fig.40 is a diagram showing a method of storing partial slice specific information in the case where a G-PCC still image has a different type of LoD for each tile.

[0054] Fig.41 is a diagram showing an example of TileLoDDecodeIntemProperty according to the modification (2) of the third embodiment.

[0055] Fig.42 is a diagram showing the format of the Matroska media container.

[0056] Fig.43 is a diagram showing the hardware configuration of a computer. DETAILED DESCRIPTION

[0057] Embodiments of the information processing device, reproduction processing device, and information processing method disclosed herein are now described in detail with reference to the accompanying drawings. In addition, the present technology includes not only the scope described in the embodiments but also the scope described in the following non-patent literature known at the time of filing.

[0058] Non-patent document 1: (as mentioned above)

[0059] Non-patent document 2: ISO / IEC 14496-12:2015 Information technology, Coding of audio-visual object, Part 12: ISO base media file format, 2015-12.

[0060] Non-patent literature 3: ISO / IEC M49043, [G-PCC] CE12.15 report on LoD generation for spatial scalability version 2, 2019-07-06.

[0061] Non-patent document 4: ISO / IECN18607, Technologies Under Consideration for Carriage of Point Cloud Data, 2019-08-26.

[0062] Non-patent document 5: “Matroska Media Container”, [retrieved on June 15, 2020], Internet<URL:https: / / www.matroscka.org / > .

[0063] Non-Patent Document 6: “WebP Container Specification”, [retrieved on June 15, 2020], Internet<URL:https: / / developers.google.com / speed / webp / docs / riff_container> .

[0064] In other words, the statements disclosed in the above-mentioned non-documents are also the basis for judging the support requirements. For example, it is assumed that the terms used in the file structure disclosed in non-patent document 2 and non-patent document 4 to non-patent document 6 and the G-PCC bitstream structure disclosed in non-patent document 1 and non-patent document 3 are not directly defined in the implementation. Even in this case, they fall within the scope of the disclosure of the present technology and should be understood to meet the support requirements of the claims. Similarly, even if not directly described in the implementation, technical terms such as parsing, syntax, and semantics fall within the scope of the present technology and should be understood to meet the support requirements of the claims.

[0065] Furthermore, the present disclosure will be described in the order of the items listed below.

[0066] 1. First Implementation

[0067] 1.1 Modification Example (1) of First Embodiment

[0068] 1.2 Modification of the First Embodiment (2-1)

[0069] 1.3 Modification of the First Embodiment (2-2)

[0070] 1.4 Modification of the First Embodiment (2-3)

[0071] 1.5 Modification of the First Embodiment (3-1)

[0072] 1.6 Modification of the First Embodiment (3-2)

[0073] 1.7 Modification of the First Embodiment (3-3)

[0074] 2. Second Implementation

[0075] 2.1 Modification Example (1) of Second Embodiment

[0076] 2.2 Modification Example of Second Embodiment (2-1)

[0077] 2.3 Modification of the Second Embodiment (2-2)

[0078] 3. Third Implementation

[0079] 3.1 Modification Example (1) of the Third Embodiment

[0080] 3.2 Modification Example (2) of the Third Embodiment

[0081] 4. Fourth Implementation

[0082] [1. First embodiment]

[0083] (Configuration of System According to First Embodiment)

[0084] Figure 1 is a diagram showing the structure of a G-PCC bit stream. Figure 2 is a diagram showing an image of a point cloud frame of a G-PCC bitstream. Figure 1 The G-PCC bit stream 1 shown is a bit stream of a series of moving pictures, in which Figure 2 The point cloud frames 10 are shown arranged together.

[0085] like Figure 2 As shown, the point cloud frame 10 represents volume data formed by a plurality of points 15. In addition, the point cloud frame 10 includes a plurality of tiles 11. Tile 11 refers to one of the cuboids obtained by dividing the bounding box surrounding an entity of one point cloud frame 10 into cuboids having the same shape. Tile 11 is a point cloud outer frame of the points 15 included in the point cloud frame 10. Then, the tiles 11 each include a plurality of slices 12. Slices 12 are sets of points 15 included in the tile 11. Then, the slices 12 each include a geometric slice 13 and zero or more attribute slices 14. In addition, Figure 2An example of a case with attribute slices is shown. There is one attribute slice 14 for each attribute. In the case where the slice 12 is a collection of points having only position information but no attributes, no attribute slice 14 is provided. On the other hand, in the case where the slice 12 is a collection of points to which color or reflection is added, an attribute slice 14 is set for each attribute such as color or reflection.

[0086] Figure 3 The octree encoding shown is used to compress the geometry in the point cloud framework 10. Figure 3 is a diagram showing octree encoding of a geometric body. Octree encoding is a method of using an octree to represent the presence or absence of a point in each block in data represented as voxels. In octree encoding, Figure 6 As shown in the tree diagram, the value of the block with point 15 is represented by 1, and the value of the block without point 15 is represented by 0. Figure 6 The level of each octree in the tree diagram of is called depth. There is a limitation that the deeper the depth, the smaller the spatial area indicating the existence of point 15. Therefore, for geometry, the deeper the depth, the higher the image resolution and clarity. Then, the range representing the depth from the shallowest depth to the depth reaching a certain depth corresponds to LoD. In other words, for example, in Figure 3 In the case where there are layers of depth = 0 to 3 as shown, the region of depth = 0 corresponds to the region of LoD = 0, and the region where the layers of depth = 0 and depth = 1 are combined corresponds to the region of LoD = 1. The region where the regions of depth = 0 to 2 are combined corresponds to the region of LoD = 2, and the region where the regions of depth = 0 to 3 are combined corresponds to the region of LoD = 3. That is, it can be said that as the LoD increases, the resolution and clarity of the image are higher in stages.

[0087] In addition, attributes are compressed using prediction weight lifting coding, regional adaptive hierarchical transform (RAHT) coding, or fixed weight lifting coding. Then, the attribute is information about the attribute of the point, and is set corresponding to the geometry, so that the attribute set for the geometry on which octree coding is performed has a hierarchical structure similar to the geometry, and the image resolution and clarity are staged higher as the LoD increases.

[0088] Return to reference Figure 1 , the description continues. G-PCC bitstream 1 has Figure 1The structure shown. G-PCC bitstream 1 has a sequence parameter set (SPS), a geometry parameter set (GPS) and an attribute parameter set (APS). The SPS stores basic information for decoding each bitstream. The GPS stores parameters for decoding geometry. The APS stores basic information for decoding attributes. The GPS and APS store the identifier (Id) of their corresponding SPS, which is associated with the SPS. The GPS and APS may differ in the unit of geometry slice 13 and the unit of attribute slice 14.

[0089] Furthermore, the G-PCC bitstream 1 has a tile inventory, a geometry slice 13, and an attribute slice 14. The tile inventory includes the number of tiles 20 and the structure of the tiles 20. The tile inventory may be different for each point cloud framework 10.

[0090] In addition, the slice 12 is a combination of the geometric slice 13 and the attribute slice 14 as described above. The slice 12 is independently decodable and may have an overlap point 15 between different slices 12. In addition, the tile 11 is an outer frame including a plurality of slices 12. The tile 11 may have an overlap area between different tiles 11.

[0091] The geometry slice 13 stores gps_id, which is an identifier of the GPS, and is associated with the GPS through gps_id. In addition, the geometry slice 13 stores tile_id, which is an identifier of the tile 11 to which the slice 12 formed by the relevant geometry slice 13 belongs, and is associated with the tile stock through tile_id. In addition, the geometry slice 13 stores slice_id, which is an identifier of the slice 12 formed by the relevant geometry slice 13. A series of geometry slices 13 form a geometry bitstream.

[0092] The attribute slice 14 stores aps_id, which is the identification of the APS, and is associated with the APS through aps_id. In addition, the attribute slice 14 stores slice_id, which is the identifier of the slice 12 formed by the related attribute slice 14, and is associated with the geometry slice 13 through slice_id. A series of attribute slices 14 form an attribute bitstream.

[0093] In addition, the octree-coded geometry bitstream can be decoded at image resolutions up to a predetermined layer. In addition, as a coding option of the attribute, the codec layer also supports LoD scalable coding, which can be used to similarly perform decoding at image resolutions up to a predetermined layer. Figure 4 It is a diagram used to describe LoD scalable coding.

[0094] like Figure 4As shown, the geometry bitstream 21 and the attribute bitstream 22 have a structure layered according to the image resolution through LoD scalable coding. The geometry bitstream 21 and the attribute bitstream 22 each have a higher LoD from top to bottom toward the surface of the image, and increase the data density in the image to be displayed, thereby improving the image resolution.

[0095] Stopping the decoding of the geometry slice 13 and the attribute slice 14 in the middle of the data, i.e., in the middle of the hierarchy, using LoD scalable coding enables the point cloud framework 10 to be configured and presented using the LoD corresponding to the desired hierarchy. Hereinafter, decoding representing an image with high image resolution and high definition is referred to as decoding with high LoD, and decoding representing an image with low image resolution and low definition is referred to as decoding with low LoD. Then, decoding of a G-PCC bitstream in which the attribute bitstream is encoded using LoD scalable coding is referred to as LoD scalable decoding.

[0096] In other words, in the geometry bitstream 21, the partial geometry bitstream 23 corresponding to the low LoD portion is decoded with low LoD, generating an image with low image resolution. The partial geometry bitstream 23 is formed by a partial slice that is a part of each geometry slice 13. In addition, decoding with high LoD allows the entire geometry bitstream 21 to be decoded, for example, generating an image with high image resolution when fully decoded.

[0097] Similarly, in the attribute bitstream 22, a partial attribute bitstream 24 corresponding to the low LoD portion is decoded at a low LoD, generating an image with a low image resolution. The partial attribute bitstream 24 is formed of a partial slice that is a part of each attribute slice 14. In addition, decoding at a high LoD allows decoding of the entire attribute bitstream 22, for example, when decoding all slices 12 included in the G-PCC bitstream 1, generating an image with a high image resolution.

[0098] At this point, a description is given of a first method of decoding each slice 12 to a level of a desired LoD and a second method of additionally decoding data at the remaining high LoD position after the first method as a processing method of LoD scalable decoding.

[0099] Figure 5 is a diagram showing a processing example of a first method using LoD scalable decoding. In the first method, the decoder unconditionally decodes the SPS as a tile inventory in the G-PCC bitstream 1. Then, as shown in process P1, the decoder decodes a partial slice of the geometry slice 13 according to the desired LoD. Afterwards, as shown in process P2, the decoder skips the rest and starts decoding the next partial slice from the beginning of the next attribute slice 14.

[0100] Figure 6 is a diagram showing a processing example of the second method using LoD scalable decoding. In the case of the second method, the decoder performs decoding similar to that performed in the first method, jumps to the beginning of the remaining data as shown in processing P3, and decodes the remaining data as shown in processing P4. For example, after processing P1 of the first method is completed, parallel processing can be performed so that processing P3 of the second method is performed at the same timing as processing P2. In addition, after processing P1 and P2 of the first method are completed, parallel processing can also be performed so that processing P3 of the second method is performed at the same timing as processing P1 for attribute slice 14. In addition, processing P3 can also be performed after all slices have been processed. In addition, the second method is used for a situation where there are sufficient resources and time on the reproduction processing device side.

[0101] However, the LoD scalable decoding technology currently under consideration requires the additional implementation of the LoD scalable decoding function for each decoder individually. In other words, in the first method for LoD scalable decoding, the decoder needs to have a function of interrupting decoding at a desired LoD. In addition, in the second method for LoD scalable decoding, the decoder needs to have a function of storing a completion position of decoding with a low LoD completed earlier and restarting decoding with a high LoD from the completion position. This is because decoding is performed with a high LoD.

[0102] In the case where additional functions are implemented for each decoder to implement LoD scalable decoding as described above, LoD scalable decoding depends on the implementation of the decoder. Therefore, in some cases, some reproduction processing devices cannot perform LoD scalable decoding. Therefore, it is difficult to provide a viewing experience that meets the requirements of viewers.

[0103] For example, as a use case of G-PCC bitstream 1, in point cloud data, it is desired to decode an area near the viewpoint position and display it with a high LoD, and to decode a distant area and display it with a low LoD. The decoding and display in this case may also include progressive decoding for sequentially displaying reproducible images. In addition, as another use case, in film production, when using a virtual asset obtained by digitizing a real film set, it is desired to decode a preview with a low LoD, and decode an actual scene presentation with a high LoD.

[0104] Preferably, in order to realize such a use case, LoD scalable decoding of the entire G-PCC bitstream 1 is performed independently of the decoder or identification of the three-dimensional spatial position of the tile 11 of the G-PCC bitstream 1 and LoD scalable decoding of each tile 11 are performed. Therefore, the use of the information processing device and the reproduction processing device mentioned below allows the realization of LoD scalable decoding independent of the decoder.

[0105] Figure 7 5 is a diagram showing an exemplary system configuration of a delivery system. The delivery system 500 includes a file generation device 100, a client device 200, and a web server 300. The file generation device 100 corresponds to an information processing device, and the client device 200 corresponds to a reproduction processing device. The file generation device 100, the client device 200, and the web server 300 are connected to a network 400. Then, the file generation device 100, the client device 200, and the web server 300 can communicate with each other via the network 400. Although Figure 7 Each device is shown as being provided with only one of each, but the delivery system 500 may include a plurality of file generation devices 100 or a plurality of client devices 200 .

[0106] The file generating device 100 generates point cloud content, which is data for providing a three-dimensional image with a point cloud. The point cloud content includes a G-PCC bitstream 1 or metadata. The file generating device 100 uploads the generated point cloud content to the web server 300. In the present embodiment, a description is given of a configuration in which the web server 300 provides the point cloud content to the client device 200, but the delivery system 500 may adopt another configuration. For example, the file generating device 100 may include the function of the web server 300, store the generated point cloud content in its own device, and provide the point cloud content to the client device 200.

[0107] The web server 300 stores the point cloud content uploaded from the file generating device 100. Then, the web server 300 provides the point cloud content designated according to the request from the client device 200.

[0108] The client device 200 requests the web server 300 to transmit the point cloud content. Thereafter, the client device 200 acquires the point cloud content specified in the transmission request from the web server 300. Then, the client device 200 decodes the G-PCC bitstream 1 included in the point cloud content to generate an image, and causes a display device such as a monitor to display the image.

[0109] (Configuration of File Generating Device According to First Embodiment)

[0110] The file generating apparatus 100 will now be described. Figure 8 is a block diagram of a file generating device. Figure 8As shown, the file generation device 100 corresponding to the information processing device has an extraction unit 101, a file generation processing unit 102, a transmission unit 103, and a control unit 104. The control unit 104 performs control-related processing of the file generation processing unit 102. For example, the control unit 104 performs overall control such as operation timing of each component of the file generation processing unit 102.

[0111] The extraction unit 101 inputs the original data of the target point cloud content. The original data of the point cloud content includes control information and metadata for generating the point cloud data. The control information includes, for example, time control information about the point cloud data, information about the correspondence between geometric data and attribute data, and necessary parameters. The extraction unit 101 extracts geometric data and attribute data from the point cloud data included in the original data of the acquired point cloud content. Then, the extraction unit 101 outputs the geometric data and attribute data and the control information to the file generation processing unit 102.

[0112] The file generation processing unit 102 has an encoding unit 121 and a file generation unit 126. Furthermore, the encoding unit 121 includes a geometry encoder 122, an attribute encoder 123, and a bit stream generation unit 124.

[0113] The geometry encoder 122 acquires the geometry data and the control information related to the geometry output from the extraction unit 101. Then, the geometry encoder 122 performs encoding processing on the geometry data, and generates geometry slices in the point cloud content using the control information. In this case, the geometry encoder 122 performs octree encoding to generate geometry slices using LoD. In addition, the geometry encoder 122 outputs each geometry slice and information about each encoding parameter to the attribute encoder 123. Then, the geometry encoder 122 outputs each geometry slice, information about each encoding parameter, and the control information that form the geometry bitstream to the bitstream generation unit 124.

[0114] The attribute encoder 123 acquires the attribute data and control information related to the attribute output from the extraction unit 101. Then, the attribute encoder 123 uses the control information and the information input from the geometry encoder 122, and performs encoding processing on the attribute data to generate the attribute slice 14 in the point cloud content. In this case, the attribute encoder 123 generates information about the attribute slice having the LoD corresponding to the geometry slice subjected to octree encoding and information about the attribute slice indicating the corresponding relationship with the attribute data. Then, the attribute encoder 123 outputs a plurality of attribute slices forming the attribute bitstream, information about each attribute slice, and the control information to the bitstream generation unit 124.

[0115] The bitstream generation unit 124 inputs a plurality of geometry slices forming a geometry bitstream, information on each geometry slice, and control information from the geometry encoder 122. In addition, the bitstream generation unit 124 inputs a plurality of attribute slices forming a geometry bitstream and control information from the attribute encoder 123.

[0116] Then, the bitstream generation unit 124 combines the geometric slices and attribute slices that form the slices and arranges them in time order. In addition, the bitstream generation unit 124 generates SPS, GPS, APS, and tile inventory according to the respective arranged slices using control information. Subsequently, the bitstream generation unit 124 adds SPS, GPS, APS, and tile inventory to the data in which the geometric slices and attribute slices are arranged to generate a Figure 1 Thereafter, the bitstream generating unit 124 outputs the generated G-PCC bitstream and control information to the metadata generating unit 125.

[0117] The metadata generating unit 125 inputs the G-PCC bitstream and the control information from the bitstream generating unit 124. Then, the metadata generating unit 125 generates depth information which is hierarchy information indicating the LoD of each geometry slice and attribute slice forming the G-PCC bitstream and the partial slice corresponding to the LoD.

[0118] Then, the metadata generation unit 125 generates access information about access to the partial slices of the geometry slice and the attribute slice corresponding to each LoD. In addition, the metadata generation unit 125 generates partial slice specific information for associating the hierarchy information with the access information. The generation of the access information, the hierarchy information, and the partial slice specific information by the metadata generation unit 125 is now described in detail.

[0119] The metadata generation unit 125 determines that SPS, GPS, APS, and tile stock are stored in GPCCDecoderConfigurationRecord. However, the metadata generation unit 125 determines whether to include GPS, APS, and tile stock according to the type of sample entry.

[0120] Fig. 9 : is a diagram shown to describe access information and hierarchy information according to the first embodiment. Figure 8 The metadata generation unit 125 in the point cloud content sets one or more geometric slices and zero or more attribute slices constituting one frame in the point cloud content as one sample and stores them in the track. Fig. 9 As shown, the metadata generation unit 125 stores each geometric slice and attribute slice included in a point cloud framework in a sample 31.

[0121] Then, Figure 8 The metadata generation unit 125 in the embodiment sets subsamples for each geometry slice and attribute slice based on the depth information corresponding to the LoD, and generates partial slice specific information indicating the corresponding relationship with the LoD. Fig. 9 As shown, the metadata generation unit 125 divides each of the geometry slice and the attribute slice into segmented slices for each depth and sets each segmented slice as a subsample. Then, the metadata generation unit 125 generates partial slice specific information indicating the correspondence between the partial slice formed by combining the segmented slices corresponding to the required depth from the depth information corresponding to the LoD and the LoD.

[0122] In other words, the segmentation slices corresponding to each depth are difference areas that increase for each layer as the LoD is higher and the depth is deeper. Fig. 9 , subsamples #1, #2, #3, etc. correspond to respective segmented slices. For example, subsample #1 corresponds to a segmented slice corresponding to the shallowest depth in a geometric slice. In addition, subsample #2 corresponds to a segmented slice corresponding to the second shallowest depth in a geometric slice. In other words, subsample #2 corresponds to a segmented slice corresponding to the difference between the lowest LoD and the second lowest LoD. In addition, subsample #3 corresponds to a segmented slice corresponding to the deepest depth in a geometric slice. In other words, subsample #3 corresponds to a segmented slice corresponding to the difference between the deepest LoD and the second deepest LoD. In addition, subsample #4 corresponds to a segmented slice corresponding to the shallowest depth in an attribute slice. In addition, subsample #5 corresponds to a segmented slice corresponding to the second shallowest depth in an attribute slice. In addition, subsample #6 corresponds to a segmented slice corresponding to the deepest depth in an attribute slice.

[0123] Then, Figure 8 The metadata generation unit 125 in the POSITION_SCHEMA_01 sets the size-related information about each segmented slice to be used as information about accessing the partial slice corresponding to each LoD. This enables the client device 200 to access the segmented slice corresponding to the specified LoD or skip other segmented slices by using the size information about the segmented slices. Specifically, the partial slice is formed by segmented slices, so the partial slice corresponding to the specified LoD can be accessed by combining the set partial slice specific information with the size information about the segmented slices.

[0124] In addition, the metadata generation unit 125 sets a depth value when dividing the slice, and the depth value is information indicating the depth corresponding to each segmented slice. Fig. 9, the metadata generation unit 125 sets the depth value of subsample #1 to 0, the depth value of subsample #2 to 1, and the depth value of subsample #3 to 2. In addition, the metadata generation unit 125 sets the depth value of subsample #4 to 0, the depth value of subsample #5 to 1, and the depth value of subsample #6 to 2. LoD exists for each geometry slice and each attribute slice, so the depth value in this embodiment is assigned as a value starting from 0 from the head of each geometry slice and each attribute slice.

[0125] Then, the metadata generation unit 125 generates the metadata to be stored in Fig.10 The syntax 32 shown represents the data in the SubSampleInformationBox. Fig.10 is a diagram showing a SubSampleInformationBox according to the first embodiment. Fig.10 Each for loop with a variable i in corresponds to each subsample. Then, the metadata generation unit 125 represents the size of each subsample as subsample_size. In addition, the metadata generation unit 125 represents the depth value of the segmented slice represented by each subsample as codec_specific_parameters. codec_specific_parameters is subsample information defined for each encoding codec and is represented by syntax 33. Therefore, the metadata generation unit 125 stores partial slice specific information in SubSampleInformationBox, which is used to associate the size of the segmented slice as access information about the segmented slice with the depth value as hierarchical information. In other words, the metadata generation unit 125 stores the partial slice specific information in the subsample information. Thereafter, the metadata generation unit 125 outputs the G-PCC bitstream 1 to the file generation unit 126. In addition, the metadata generation unit 125 outputs information about each storage destination of SPS, GPS, APS, tile library, geometry slice 13, and attribute slice 14, and metadata including SubSampleInformationBox to the file generation unit 126.

[0126] In this regard, in the present embodiment, the metadata generation unit 125 stores the SPS, GPS, APS, and tile inventory in the GPCCDecoderConfigurationRecord, and stores the geometry slices 13 and the attribute slices 14 in the samples of one track. However, the storage method is not limited to the above description, for example, the metadata generation unit 125 may be configured to store all or part of the SPS, GPS, APS, and tile inventory in the sample. The determination of whether the GPS, APS, and tile inventory are stored in the GPCCDecoderConfigurationRecord or in the sample is performed according to the type of the sample entry. In addition, the metadata generation unit 125 may be configured to store the geometry slices 13 and the attribute slices 14 in the samples of a single track.

[0127] The file generation unit 126 inputs the G-PCC bitstream 1, the information about the storage destination of each data, and the data to be stored in the SubSampleInformationBox from the metadata generation unit 125. Then, the file generation unit 126 stores the acquired metadata in the ISOMBFF file. For example, the file generation unit 125 stores the partial slice correspondence information of each LoD in the subsample information about the ISOBMFF. Subsequently, the file generation unit 126 generates a G-PCC file in which the G-PCC bitstream 1 is stored in mdat, as shown in FIG. Fig.11 As shown, mdat is the actual data area of ​​the ISOBMFF file. Fig.11 is a diagram showing an example of storing a SubSampleInformationBox according to the first embodiment. Specifically, the file generation unit 126 stores the geometry slice 13 and the attribute slice 14 in the sample of one track. In addition, the file generation unit 126 stores the SPS, GPS, APS, and tile inventory in the GPCCDecoderConfigurationRecord. In addition, as Fig.11 As shown in the box 34 of , the file generation unit 126 stores the SubSampleInformationBox in the lower level of the movie box, in the lower level of the tack box, in the lower level of the media box, and in the lower level of the sample table box. Fig.11In the example, the movie box is represented as moov, the tack box is represented as trak, the media box is represented as media, and the sample table box is represented as stbl. In addition, the file generation unit 126 may store the SubSampleInformationBox in a lower level of the track fragment box in a lower level of the movie fragment box. Afterwards, the file generation unit 126 outputs the generated G-PCC file to the transmission unit 103.

[0128] The transmission unit 103 inputs the G-PCC file of the G-PCC bitstream 1 from the file generation unit 126. Then, the transmission unit 103 transmits the acquired G-PCC file of the G-PCC bitstream 1 to the web server 3 for uploading the point cloud content.

[0129] [Configuration of Client Device According to First Embodiment]

[0130] Fig.12 is a block diagram of a client device. Fig.12 As shown, the client device 200 corresponding to the reproduction processing device has a reproduction processing unit 201, a display unit 202, and a control unit 203. The control unit 203 controls the operation of each component of the reproduction processing unit 201. For example, the control unit 203 performs overall control on the operation timing of each component of the reproduction processing unit 201. The reproduction processing unit 201 has a file acquisition unit 211 and a decoding processing unit 212.

[0131] The file acquisition unit 211 sends a request for acquiring point cloud content data to the web server 300. Then, the file acquisition unit 211 acquires an ISOBMFF file from the web server 300, in which a G-PCC file of a G-PCC bitstream 1 of point cloud content is stored. Thereafter, the file acquisition unit 211 outputs the acquired ISOBMFF file to the decoding processing unit 212.

[0132] The decoding processing unit 212 has a file processing unit 221 , a decoding unit 222 , and a display information generating unit 223 . Fig.13 is a block diagram showing a specific configuration of a decoding processing unit.

[0133] The file processing unit 221 has a data extraction unit 231. The data extraction unit 231 obtains a G-PCC file in which the G-PCC bit stream 1 output from the file acquisition unit 211 is stored. Then, the data extraction unit 231 parses the obtained G-PCC file to extract metadata. Then, the data extraction unit 231 obtains partial slice specific information of each segmented slice from the SubSampleInformationBox included in the metadata. Then, the data extraction unit 231 obtains information corresponding to a subsample of the segmented slice corresponding to the LoD for which reproduction is specified from the partial slice specific information. Fig. 9 As shown in the process P3 in FIG. 1 , the data extraction unit 231 extracts the segmented slices included in the LoD area specified by the sample of the G-PCC file. Thereafter, the data extraction unit 231 obtains the size of the segmented slices included in the area equal to or greater than the specified LoD from the partial slice specific information. Then, as shown in FIG. Fig. 9 As shown in the process P4 in the above, the data extraction unit 231 skips the segmented slices of the area equal to or greater than the specified LoD, and moves to the head of the partial slice included in the area of ​​the next specified LoD. The file processing unit 221 repeats the above processes P3 and P4 to extract the partial slice corresponding to the specified LoD from the geometry slice 13 and the attribute slice 14. Therefore, the file processing unit 221 obtains the partial slice group corresponding to the specified LoD in the geometry bitstream and the attribute bitstream.

[0134] Then, the file processing unit 221 outputs the parameters related to the decoded partial slice group and the geometry included in the geometry bitstream corresponding to the specified LoD to the geometry decoder 232 of the decoding unit 222. In addition, the file processing unit 221 outputs the parameters related to the decoded partial slice group and the attributes included in the attribute bitstream corresponding to the specified LoD to the attribute decoder 233 of the decoding unit 222.

[0135] The geometry decoder 232 has a geometry decoder 232 and an attribute decoder 233 .

[0136] The geometry decoder 232 inputs the parameters related to the decoded partial slice group and the geometry included in the geometry bitstream corresponding to the specified LoD from the file processing unit 221. Then, the geometry decoder 232 decodes the acquired partial slice group by using the parameters related to the decoded geometry. Therefore, the geometry decoder 232 acquires the geometry data corresponding to the specified LoD. In this case, the geometry decoder 232 outputs the position information about the decoded partial slice to the attribute decoder 233. Thereafter, the geometry decoder 232 outputs the decoded partial slice group to the point cloud generation unit 234 of the display information generation unit 223.

[0137] In addition, the attribute decoder 233 inputs the parameters related to the decoding of the partial slice group and the attributes included in the attribute bitstream corresponding to the specified LoD from the file processing unit 221. Then, the attribute decoder 233 decodes the acquired partial slice group using the parameters related to the decoding attributes and the position information about the partial slice acquired from the geometry decoder 232. Therefore, the attribute decoder 233 acquires the data of the attribute corresponding to the specified LoD. Thereafter, the attribute decoder 233 outputs the decoded partial slice group to the point cloud generation unit 234 of the display information generation unit 223.

[0138] In this way, the geometry decoder 232 and the attribute decoder 233 obtain the partial slice group corresponding to the specified LoD from the file processing unit 221. Therefore, a point cloud corresponding to the specified LoD can be generated by performing normal decoding. In other words, the geometry decoder 232 and the attribute decoder 233 may not need to perform processing such as extracting a partial slice corresponding to the specified LoD or skipping an unused partial slice.

[0139] The display information generating unit 223 has a point cloud generating unit 234 and an image processing unit 235 .

[0140] The point cloud generation unit 234 inputs a partial slice group of the decoded geometry bitstream from the geometry decoder 232. In addition, the point cloud generation unit 234 inputs a partial slice group of the decoded attribute bitstream from the attribute decoder 233. Then, the point cloud generation unit 234 combines the geometry data corresponding to the specified LoD and the attribute data corresponding to the specified LoD to generate a point cloud corresponding to the specified LoD. Thereafter, the point cloud generation unit 234 outputs the generated point cloud to the image processing unit 235.

[0141] The image processing unit 235 inputs the point cloud data corresponding to the specified LoD from the point cloud generation unit 234. Then, the image processing unit 235 performs rendering using the acquired point cloud data to generate an image to be displayed. Thereafter, the image processing unit 235 outputs the display image to the display unit 202.

[0142] The display unit 202 has a display device such as a monitor. The display unit 202 inputs the display image generated by the image processing unit 235. Then, the display unit 202 causes the display device to display the acquired display image. This allows the display unit 202 to display the display image corresponding to the specified LoD on the display device.

[0143] [File Generation Program According to First Embodiment]

[0144] Now refer to Fig.14 The procedure of the file generation process executed by the file generation apparatus 100 according to the present embodiment will be described. Fig.14 is a flowchart of a file generation process executed by the file generation device according to the first embodiment.

[0145] The extraction unit 101 inputs raw data of point cloud content for displaying a three-dimensional image using a point cloud. Subsequently, the extraction unit 101 extracts geometry data and attribute data from the point cloud data included in the acquired raw data of the point cloud content. The geometry encoder 122 performs encoding processing on the geometry data to generate a geometry slice 13 with LoD. The attribute encoder 123 performs encoding processing on the attribute data by using the control information and position information input from the geometry encoder 122 to generate an attribute slice 14 with LoD. Therefore, the geometry encoder 122 and the attribute encoder 123 encode the point cloud data (step S101).

[0146] The bitstream generation unit 124 combines the encoded geometry slice 13 and attribute slice 14 to generate a G-PCC bitstream 1 (step S102 ).

[0147] The metadata generation unit 125 obtains the G-PCC bitstream 1 from the bitstream generation unit 124. Then, the metadata generation unit 125 determines to store the geometry slice 13 and the attribute slice 14 in one sample of a single track. In addition, the metadata generation unit 125 defines the segmented slices corresponding to each layer in the geometry slice 13 and the attribute slice 14 as subsamples. Then, the metadata generation unit 125 sets a depth value representing the depth corresponding to each segmented slice in each segmented slice. Then, the file generation unit 126 stores the depth value of each segmented slice in the SubSampleInformationBox as partial slice specific information (step S103).

[0148] In addition, the metadata generation unit 125 outputs metadata such as access information about accessing partial slices of the geometry slice and partial slices of the attribute slice corresponding to each LoD to the file generation unit 126. The file generation unit 126 generates a G-PCC file in which the G-PCC bitstream 1, the SubSampleInformationBox including the partial slice specific information, and the metadata are stored in the metadata area of ​​the ISOBMFF file (step S104). Thereafter, the file generation unit 126 outputs the generated G-PCC file to the transmission unit 106.

[0149] The transmission unit 106 inputs the G-PCC file of the point cloud content from the file generation unit 126. Then, the transmission unit 106 transmits the acquired G-PCC file to the web server 3 for uploading the point cloud content (step S105).

[0150] [Reproduction Processing Procedure According to the First Embodiment]

[0151] Now refer to Fig.15 The procedure of the reproduction process performed by the client device 200 according to the present embodiment is described. Fig.15 is a flowchart of a reproduction process executed by the client device according to the first embodiment.

[0152] The file acquisition unit 211 sends a request to acquire point cloud content to the web server 300, and acquires a G-PCC file of the designated point cloud content from the web server 300. Subsequently, the data extraction unit 231 of the file processing unit 221 parses the ISOBMFF file as the G-PCC file to extract metadata. Then, the data extraction unit 231 acquires partial slice specific information from the SubSampleInformationBox included in the metadata (step S201).

[0153] Subsequently, the data extraction unit 231 extracts a partial slice group for each of the geometry bitstream and the attribute bitstream for generating the point cloud of the specified LoD based on the partial slice specific information (step S202). Then, the data extraction unit 231 outputs the partial slice group of the geometry bitstream to the geometry decoder 232 of the decoding unit 222. In addition, the data extraction unit 231 outputs the partial slice group of the attribute bitstream to the attribute decoder 233 of the decoding unit 222.

[0154] The geometry decoder 232 and the attribute decoder 233 decode the extracted partial slice group of the geometry bitstream and the extracted partial slice group of the attribute bitstream, respectively (step S203). Therefore, the geometry decoder 232 and the attribute decoder 233 obtain the geometry data and the attribute data corresponding to the specified LoD, respectively.

[0155] The point cloud generation unit 234 of the display information generation unit 223 combines the decoded partial slice group of the geometry bitstream and the decoded partial slice group of the attribute bitstream to generate a point cloud (step S204 ). Thereafter, the point cloud generation unit 234 outputs the point cloud data to the image processing unit 235 .

[0156] The image processing unit 235 acquires the point cloud data from the point cloud generation unit 234. Then, the image processing unit 235 performs rendering on it to generate a display image. Then, the image processing unit 235 sends the generated display image to the display unit 202, and displays the point cloud as a three-dimensional image (step S205).

[0157] In this regard, in the present embodiment, the metadata generation unit 125 sets the segmented slices of the geometry slice and the attribute slice as subsamples, but the geometry slice and the attribute slice may be set as subsamples in other ways. In this case, by, for example, extending the byte length (32 bits) of codec_specific_parameters and storing the size of the segmented slice at each LoD for each subsample, it is possible to achieve an effect similar to the case where the metadata generation unit 125 defines the segmented slice as a subsample.

[0158] As described above, the file generation device according to the present embodiment sets the segmented slices of each layer corresponding to the LoD of the geometric slices and the attribute slices as subsamples. Then, the file generation device obtains the size of the segmented slices as access information to the segmented slices. In addition, the file generation device sets a depth value as information representing the depth corresponding to each segmented slice. Then, the file generation device generates a SubSampleInformationBox including partial slice specific information indicating the correspondence between the LoD and the partial slices formed by combining the segmented slices corresponding to the necessary depth. The client device obtains the partial slice specific information included in the SubSampleInformationBox from the ISOBMFF file of the G-PCC bitstream, and obtains the partial slice group corresponding to the specified LoD from the geometric slices and the attribute slices. Then, the decoder of the client device is able to obtain the geometric data and attribute data corresponding to the specified LoD by decoding the partial slice group corresponding to the LoD.

[0159] In this way, the client device can identify and extract the position of the partial slice for obtaining data corresponding to the desired LoD based on the partial slice specific information included in the ISOBMFF file of the G-PCC bitstream to input the position information into the decoder. This makes it possible to perform LoD scalable decoding without implementing an additional function of LoD scalable decoding in the decoder. Therefore, scalable decoding independent of the decoder can be performed, thereby providing a viewing experience that meets the viewer's request regardless of the type of client device.

[0160] [1.1 Modification example (1) of the first embodiment]

[0161] A modification (1) of the first embodiment will now be described. The file generation apparatus 100 according to the present modification provides an index to the client apparatus 200 so that the LoD is selectable according to the processing performance of the client apparatus 200 .

[0162] The metadata generation unit 125 of the file generation device 100 according to the present modification stores the number of decoding points, which is the number of points to be decoded for each LoD, in codec_specific_parameters in SubSampleInformationBox. Fig.16 The codec_specific_parameters shown in the syntax shown in 35. Fig.16 is a diagram showing an example of codec_specific_parameters according to the modification (1) of the first embodiment. In syntax 35, total_point indicates the number of decoding points.

[0163] In this regard, in the present modification, the metadata generation unit 125 uses the number of decoding points as an indicator, but the indicator is not limited to the number of decoding points. Other information may be used as long as the information is information indicating the load in the decoding process for each LoD. In addition, in the present modification, the metadata generation unit 125 stores the indicator in codec_specific_parameters, but the indicator may be stored in the high-level syntax (HLS) of the G-PCC bitstream 1 instead of the ISOBMFF file. For example, SPS, APS, GPS, tile inventory, geometry slice 13, or attribute slice 14 may be used as the HLS.

[0164] The file processing unit 221 of the client device 200 parses the ISOBMFF file of the point cloud content to obtain metadata. Then, the file processing unit 221 checks the codec_specific_parameters in the SubSampleInformationBox included in the metadata, and obtains the number of decoding points for each LoD. Subsequently, the file processing unit 221 selects an appropriate LoD according to the processing performance of its own device by using the number of decoding points for each LoD. Thereafter, the file processing unit 221 extracts a partial slice corresponding to the selected LoD from the geometric slice and the attribute slice, and outputs the extracted partial slice to the decoding unit 222.

[0165] As described above, the file generation device according to the present embodiment transmits an index indicating the load of the decoding process for each LoD to the client device. The client device selects the LoD to be decoded based on the provided index. This enables the client device to use more detailed criteria to determine the LoD to be decoded when decoding at a low LoD. Therefore, the client device can provide the user with a three-dimensional image with a more appropriate LoD, thereby providing a viewing experience that meets the viewer's request.

[0166] [1.2 Modification example of the first embodiment (2-1)]

[0167] A modification (2-1) of the first embodiment is now described. The file generation device 100 according to this modification is different from that of the first embodiment in that partial slice specific information for obtaining a desired LoD is stored in time-corresponding metadata.

[0168] Fig.17 is a diagram showing access information and hierarchy information for describing a modification example (2-1) according to the first embodiment. The metadata generation unit 125 of the file generation device 100 according to the present embodiment determines to store partial slice specific information in the time corresponding metadata 36. In this case, the metadata generation unit 125 sets the sample entry type of the time corresponding metadata 36 to LoDDecodeSample. In addition, the metadata generation unit 125 links the time corresponding metadata track with the track storing the G-PCC bitstream 1 using the track reference. In addition, for samples of the G-PCC bitstream 1 aligned with the decoding timestamp (DTS), the metadata generation unit 125 stores the decoding position information of each LoD in the time corresponding metadata. In other words, as Fig.17 As shown, samples of the geometry slices and attribute slices storing the G-PCC bitstream 1 and the time corresponding metadata are synchronized in time.

[0169] Fig.18 is a diagram showing exemplary time-corresponding metadata according to the modification example (2-1) of the first embodiment. For example, the metadata generating unit 125 generates a metadata having Fig.18 The time of the syntax shown corresponds to metadata 36. Fig.18 In the syntax of , unit_index represents an indicator used as identification information of a geometric slice and an attribute slice in the G-PCC bitstream 1. In addition, data_length_byte represents the numerical value of the portion in bytes of the size of the partial slice corresponding to each LoD. In addition, data_length_bits represents the numerical value of the portion in bytes of the size of the partial slice corresponding to each LoD. In other words, the value obtained by adding data_length_byte and data_length_bits corresponds to the size of the partial slice corresponding to each LoD. Here, the size of the partial slice corresponding to each LoD corresponds to the size of all data used to obtain the image corresponding to the LoD, that is, the sum of all data of the segmented slices corresponding to each layer included in the LoD. For example, as Fig.17As shown, in the case where layers D1 to D2 corresponding to the segmented slice exist in one geometry slice and the partial slice corresponding to LoD=1 includes data up to layer D2, the value obtained by adding data_length_byte and data_length_bits is a value indicating size Z1. In this case, the access information corresponds to unit_index of each of the geometry slice and the attribute slice and to the data length from the header of each of the geometry slice 13 and the attribute slice 14 corresponding to the value obtained by adding data_length_byte and data_length_bits.

[0170] Fig.19 2-1 is a diagram showing a storage example of time correspondence metadata according to the modification example (2-1) of the first embodiment. The file generating unit 126 inputs metadata including the time correspondence metadata 36 and the G-PCC bit stream 1 from the metadata generating unit 125 .

[0171] Then, the file generation unit 126 stores the G-PCC bitstream 1 in the sample. In addition, the file generation unit 126 stores the time-corresponding metadata 36 in the time-corresponding metadata track shown in the box 37. In addition, the file generation unit 126 stores the SubSampleInformationBox specified in the non-patent document 4 in the position indicated by the box 38. In this case, the subsample is set to a parameter set, a geometry slice, or an attribute slice. In addition, the codec specific parameter is defined as Fig. 20 Syntax 39 is shown. Fig. 20 is a diagram showing an example of codec specific parameters according to the modification example (2-1) of the first embodiment. Fig. 20 The information shown in the syntax 39 shown in makes it possible to identify the boundaries of the slice 12 and the type of the slice 12, i.e. whether it is a parameter set, a geometry slice or an attribute slice. In addition, the time corresponding metadata 36 stored in box 37 is linked to the subsample information stored in box 38 using a track reference.

[0172] The file processing unit 221 of the client device 200 parses the ISOBMFF file of the point cloud content to obtain the time correspondence metadata 36. Then, the file processing unit 221 obtains the decoding position of the partial slice corresponding to the specified LoD from the time correspondence metadata 36. Then, the file processing unit 221 extracts the partial slice corresponding to the specified LoD by obtaining data of the decoding time corresponding to the obtained decoding position from the geometric slice and the attribute slice. Thereafter, the file processing unit 221 outputs the extracted partial slice group to the decoding unit 222.

[0173] As described above, the file generation device according to the present modification example stores the partial slice specific information in the time corresponding metadata. Then, the client device specifies the decoding position of the partial slice corresponding to the specified LoD from the time corresponding metadata, and extracts the partial slice from the geometric slice and the attribute slice. In the first embodiment, when decoding the point cloud of the desired LoD, the client device reads all SubSampleInformation before decoding, and calculates the position of the partial slice to be decoded. On the other hand, the client device according to the present modification example is able to simultaneously set the decoding position of the partial slice corresponding to the desired LoD of each sample from the time corresponding metadata sample based on the decoding time of the sample of the G-PCC bitstream, thereby simplifying the process of extracting the partial slice. In addition, the client device is also able to obtain data of the partial slice that is only used to display the image corresponding to the specified LoD when delivered from the web server by referring to the time corresponding metadata.

[0174] [1.3 Modification of the first embodiment (2-2)]

[0175] Modification (2-2) of the first embodiment is now described. The file generation apparatus according to this embodiment stores an index for selecting LoD according to the processing performance of the client in the time-corresponding metadata, similarly to modification (1) of the first embodiment.

[0176] Fig.21 2 is a diagram showing exemplary time-corresponding metadata according to the modification (2-2) of the first embodiment. The metadata generation unit 125 according to the present modification generates time-corresponding metadata 36 in which information about total_point indicating the number of decoding points for each LoD shown in row 40 is stored.

[0177] In this regard, in the present modification, the metadata generation unit 125 uses the number of decoding points as an indicator, but the indicator is not limited to the number of decoding points. Other information may be used as long as it is information representing the load in the decoding process of each LoD. In this regard, in the present modification, the metadata generation unit 125 generates information to be stored in the ISOBMFF file as an indicator, but the indicator is not limited to the information to be stored in the ISOBMFF file, and an indicator to be stored in the HLS of the G-PCC bitstream 1 may also be generated. For example, SPS, APS, GPS, tile inventory, geometry slice 13, or attribute slice 14 may be used as the HLS.

[0178] The file processing unit 221 of the client device 200 parses the ISOBMFF file of the point cloud content to obtain metadata. Then, the file processing unit 221 obtains the number of decoded points for each LoD corresponding to each subsample included in the time-corresponding metadata 36. Subsequently, the file processing unit 221 selects an appropriate LoD according to the processing performance of its own device by using the number of decoded points for each LoD. Thereafter, the file processing unit 221 extracts a partial slice corresponding to the selected LoD from the geometric slice and the attribute slice, and outputs the extracted partial slice to the decoding unit 222.

[0179] As described above, the file generation device according to the present embodiment transmits an indicator indicating the load of the decoding process for each LoD to the client device using the time-corresponding metadata. The client device selects the LoD to be decoded based on the indicator provided by using the time-corresponding metadata. This enables the client device to determine the LoD to be decoded using a more detailed criterion when decoding at a low LoD. Therefore, in the case of notifying the LoD of each partial slice using the time-corresponding metadata, the client device can also provide the user with a three-dimensional image with a more appropriate LoD, thereby providing a viewing experience that meets the viewer's request.

[0180] [1.4 Modification example of the first embodiment (2-3)]

[0181] A modification (2-3) of the first embodiment is now described. The metadata generation unit 125 may further extend the partial slice specific information to store a unit type of the partial slice, which indicates whether each partial slice is a geometry slice 13 or an attribute slice 14. In addition, for the attribute slice 14, the metadata generation unit 125 may store the attribute type (attribute_type) in the partial slice specific information. The metadata generation unit 125 may allow the client device 2 side to determine whether to decode each attribute slice 14 according to the attribute.

[0182] Furthermore, the metadata generation unit 125 may store information associated with entries of the SubSampleInformationBox specified in Non-Patent Literature 4 instead of unit_index, data_length_byte, and data_length_bits stored in the time correspondence metadata 36. Furthermore, the metadata generation unit 125 may store byte positions in the time correspondence metadata 36 instead of unit_index.

[0183] In addition, the metadata generation unit 1125 may store the partial slice specific information in LodDecodeBox in which all types of decoding entries are newly defined, and store the LodDecodeBox in SampleEntry.

[0184] [1.5 Modification of the First Embodiment (3-1)]

[0185] A modification (3-1) of the first embodiment will now be described. A file generating apparatus 100 according to this modification is different from that of the first embodiment in that a G-PCC bit stream 1 having hierarchically structured slices is used.

[0186] Fig. 22 is a diagram showing a G-PCC bitstream having a hierarchical structure slice. Fig. 22 As shown, the hierarchical structure slice is a structure in which the slice 12 is divided into layers for each LoD layer. In other words, the segmented slices of the geometric slice and the attribute slice in the first embodiment are predefined in units of so-called layers, and the geometric slice and the attribute slice have a layer-by-layer hierarchical format structure. Each slice 12 includes a geometric layer 41 and an attribute layer 42 for each layer corresponding to each depth.

[0187] Even in the first embodiment, the geometry slice and the attribute slice have a hierarchical structure forming a partial slice, but each segmented slice is not defined as a predetermined unit. On the other hand, the geometry slice and the attribute slice according to the present modification have a layer as a predetermined unit therein. The geometry layer 41 and the attribute layer 42 are combined up to a specific layer, so that one geometry slice bitstream or one attribute bitstream corresponding to each LoD can be formed.

[0188] In the case where the G-PCC bitstream 1 has a hierarchical structure slice, decoding up to the desired LoD can be achieved by decoding in units of layers. Therefore, skipping is not performed in each of the geometric slice and the attribute slice. However, even in the hierarchical structure slice, since there are multiple slices 12 in the sample, data skipping processing is still performed. Data skipping in this case is performed layer by layer.

[0189] The metadata generation unit 125 sets each of the geometry layer and the attribute layer as a subsample. In addition, the metadata generation unit 125 sets a depth value as information indicating a depth corresponding to each layer. Then, the metadata generation unit 125 generates partial slice specific information indicating a correspondence relationship between a partial slice and a LoD according to the depth information corresponding to the layer.

[0190] The data extraction unit 231 of the client device 2 parses the ISOBMFF file of the point cloud content to obtain metadata. Then, the data extraction unit 231 specifies the geometry layer and attribute layer corresponding to the specified LoD from the partial slice specific information stored in the SubSampleInformationBox included in the metadata. Then, as shown in process P5, the data extraction unit 213 extracts the specified geometry layer and attribute layer, and as shown in process P6, uses the boundary of each layer obtained from the SubSampleInformationBox to jump to the next interlaced layer. The data extraction unit 231 repeats processes P5 and P6 to extract a partial slice group for generating a three-dimensional image of a point cloud corresponding to the specified LoD.

[0191] As described above, the file generation device according to the present modification generates a SubSampleInformationBox including partial slice specific information for extracting a desired LoD using a G-PCC bitstream with hierarchical structure slices. The client device uses the partial slice specific information included in the SubSampleInformationBox to extract a partial slice corresponding to the desired LoD from the G-PCC bitstream with hierarchical structure slices, and causes the decoder to decode it. This makes it possible to implement decoder-independent LoD scalable decoding even when a hierarchical structure is used, thereby providing a viewing experience that meets the viewer's request.

[0192] [1.6 Modification of the first embodiment (3-2)]

[0193] A modification (3-2) of the first embodiment is now described. The file generation apparatus 100 according to the present modification performs processing similar to that of the modification (2-1) of the first embodiment using the G-PCC bitstream 1 having hierarchically structured slices.

[0194] The metadata generation unit 125 determines that each partial slice specific information for specifying the geometry layer and the attribute layer is stored in the time correspondence metadata 36. In this case, the metadata generation unit 125 sets the sample entry type of the time correspondence metadata 36 to LoDDecodeSample. In addition, the metadata generation unit 125 links the time correspondence metadata track with the track storing the G-PCC bitstream 1 using the track reference. In addition, for the samples of the G-PCC bitstream 1 aligned with the decoding timestamp (DTS), the metadata generation unit 125 stores the decoding position information for each LoD in the time correspondence metadata. In other words, the samples and the time correspondence metadata storing the geometry layer and the attribute layer of the G-PCC bitstream 1 are synchronized in time.

[0195] Fig.233-2 is a diagram showing exemplary time-corresponding metadata according to the modification example (3-2) of the first embodiment. For example, the metadata generating unit 125 generates a metadata having Fig.23 The time corresponding metadata 43 of the syntax shown. The metadata generation unit 125 generates the metadata 43 by replacing the definition of unit_index with layer_index which is identification information about each of the geometry layer 41 and the attribute layer 42 - instead of Fig.18 The time corresponding metadata 43 is obtained by using the data_length_byte and data_length_bits in the time corresponding metadata 36 in the above. Defining the geometry layer 41 or the attribute layer 42 as a subsample enables the boundary of each layer to be identified by the default SubSampleInformationBox in the non-patent document 4. In addition, a method such as run-length encoding can be used to store the layer_index of the header layer and the number of subsequent layers that need to be decoded. It has the effect of reducing the number of bits when the layer_index is continuous.

[0196] The file processing unit 221 of the client device 200 parses the ISOBMFF file of the point cloud content to obtain the time correspondence metadata 43. Then, the file processing unit 221 obtains the decoded positions of the geometry layer and the attribute layer corresponding to the LoD specified by using the layer_index included in the time correspondence metadata 43 from the time correspondence metadata 36. Then, as shown in FIG. Fig.24 As shown, the file processing unit 221 extracts data of a decoding time corresponding to the acquired decoding position from the geometric slice and the attribute slice. Fig.24 3 is a diagram showing extraction of partial slices according to the modification example (3-2) of the first embodiment.

[0197] For example, in Fig.24 , the layer_index of each layer has L0 to L7, and the layer of depth = 0 has Layer_index = L0 and L1. In addition, the layer of depth = 1 has Layer_index = L2 and L3, and the layer of depth = 2 has Layer_index = L4 and L5. In the case of LoD = 1 specifying a hierarchy including depth = 0 and depth = 1, the file processing unit 221 specifies the decoding position of the layer of Layer_index = L0 to L3 from the time correspondence metadata 43. Then, the file processing unit 221 extracts data of the decoding time corresponding to the decoding position of the layer of Layer_index = L0 to L3 from the geometric slice and the attribute slice. Subsequently, the file processing unit 221 skips the layers of Layer_index = L4 and L5, and extracts data from the decoding position of the next layer of Layer_index = L6.

[0198] As described above, the file generation device according to the present modification example stores partial slice specific information in the time corresponding metadata when using a G-PCC bitstream with hierarchical structure slices. Then, the client device specifies the decoding position of the layer corresponding to the specified LoD from the time corresponding metadata, and extracts partial slices from the geometric slices and the attribute slices. Therefore, the client device according to the present modification example is able to simultaneously set the decoding position of the partial slice corresponding to the expected LoD of each sample from the time corresponding metadata sample based on the decoding time of the sample of the G-PCC bitstream, thereby simplifying the process of extracting the partial slice. In addition, the number of data bits can be further reduced by using layer_index instead of data_length_byte and data_length_bits.

[0199] [1.7 Modification of the First Embodiment (3-3)]

[0200] A modification (3-3) of the first embodiment will now be described. A file generation apparatus 100 according to this modification processes a G-PCC bitstream 1 in which slices 12 of a hierarchical structure and a normal structure are mixed. Fig.25 : is a diagram showing exemplary time-corresponding metadata according to the modification example (3-3) of the first embodiment.

[0201] The metadata generating unit 125 according to the present modification generates Fig.25 The time corresponding metadata 44 is shown. The time corresponding metadata 44 specifies the slice structure using layer_flag of line 45. For example, if the value of layer_flag is 0, it is a normal structure slice, and if the value is 1, it is a hierarchical structure slice. Then, data_length_bytes and data_length_bits are used for normal structure slices, and units_index is used for hierarchical structure slices. In this way, the metadata generation unit 125 defines partial slice specific information corresponding to the G-PCC bitstream 1 in which hierarchical structure slices and normal structure slices are mixed.

[0202] The file processing unit 221 of the client device 200 acquires the time-corresponding metadata 44 to extract a portion of the slices according to the slice structure.

[0203] As described above, even in the case where hierarchical structure slices and normal structure slices are mixed, LoD scalable decoding can be implemented independently of the decoder.

[0204] [2. Second Embodiment]

[0205] The second embodiment is now described. The file generation device according to this embodiment differs from the file generation device of the first embodiment in that it enables processing of each tile using a sample group and performs LoD scalable decoding of each tile. The file generation device according to this embodiment is also composed of Figure 8 In addition, the client device according to this embodiment is also composed of Fig.12 and Fig.13 Description of the functions of the respective components similar to those of the first embodiment will be omitted below.

[0206] The metadata generating unit 125 of the file generating apparatus 100 according to the present embodiment generates the metadata by using Fig.26 The illustrated configuration collects geometric slices and attribute slices of each tile 11 to generate a sample group. Fig.26 : is a diagram showing an exemplary configuration of a sample group. A sample group is a grouping structure in ISOBMFF that enables a plurality of decoding targets to be grouped together to process them as one target.

[0207] For example, similar to the first embodiment, the metadata generation unit 125 divides the geometry slice and the attribute slice into respective segment slices for each depth, and sets each segment slice as a subsample. In addition, the metadata generation unit 125 sets a depth value as information indicating a depth corresponding to each segment slice. Then, the metadata generation unit 125 stores the partial slice access information indicating the correspondence relationship between the partial slice and the LoD in the SubSampleInformationBox.

[0208] In addition, the metadata generation unit 125 expands the sample group ("unim") and uses Fig. 27 The UnitMapEntry of the syntax 50 shown is used to associate the subsample entry of the partial slice with the sample group to which the partial slice belongs. Fig. 27 is a diagram showing an example of UnitMapEntry according to the second embodiment. Therefore, the metadata generation unit 125 generates a sample group in which a geometry slice and an attribute slice constitute each tile 11. In the syntax 50, line 51 indicates access information in the SubSampleInformationBox of the subsample included in the sample group.

[0209] Fig.28 is a diagram showing an example of a sample group according to the second embodiment. Fig.28As shown, the respective geometry slices and attribute slices of unit_index=1 and 2 are included in one tile 11. The metadata generation unit 125 sets the tile 11 as a sample group with groupID=1, which is identification information. Then, the metadata generation unit 125 sets the subsamples to #1 to #8 in SubSampleInformationBox, where the subsamples correspond to the segmented slices of the geometry slices and attribute slices corresponding to the tile 11. In addition, the metadata generation unit 125 indicates which subsample is used in each sample group with groupID=1 through subs_entry of UnitMapEntry.

[0210] The client device 200 can change the LoD for each sample group. For example, the client device 200 receives an instruction to decode a sample group with a high LoD and to decode another sample group with a low LoD.

[0211] The data extraction unit 231 extracts segmented slices corresponding to the specified high LoD from the geometric slices and attribute slices included in the sample group specified for decoding with high LoD. In addition, the data extraction unit 231 extracts segmented slices corresponding to the specified low LoD from the geometric slices and attribute slices included in the sample group specified for decoding with low LoD. Then, for each sample group, the data extraction unit 231 outputs the extracted partial slice group corresponding to the high LoD and the extracted partial slice group corresponding to the low LoD to the decoding unit 222 for decoding.

[0212] Therefore, the file generation device 100 can generate Fig.29 The three-dimensional image of the point cloud is shown in which the LoD is different for each tile 11 . Fig.29 is a diagram showing an example of a three-dimensional image of a point cloud, where the LoD is different for each tile 11. Fig.29 In the example, in tiles 11A and 11B, data decoded with a high LoD including up to depth = 10 is displayed as a dense and detailed image. In addition, in the portion of tile 11 other than tiles 11A and 11B, data decoded with a low LoD including up to depth = 5 is displayed as a sparse and rough image.

[0213] In this regard, in the present embodiment, UnitMapEntry is used as metadata for tile access, and SubSampleInformationBox is used as metadata for LoD access. However, both types of information may be stored together in SubSampleInformationBox.

[0214] As described above, the file generation device according to the present embodiment collects the geometry slices and attribute slices included in the tile as a sample group for each tile. The client device inputs the LoD specified for each sample group, and displays a three-dimensional image of a point cloud decoded with a different type of LoD for each tile. Therefore, for example, the following three-dimensional image can be displayed: in this three-dimensional image, tiles near the viewpoint position are decoded with a high LoD and tiles far from the viewpoint are decoded with a low LoD, thereby providing a viewing experience that meets the request of the content creator or the viewer.

[0215] [2.1 Modification example (1) of the second embodiment]

[0216] A modification (1) of the second embodiment is now described. The file generation device 100 according to this modification is different from that of the second embodiment in that part of the slice specific information is stored in the time-corresponding metadata. Fig.30 : is a diagram showing exemplary time-corresponding metadata according to the modification example (1) of the second embodiment.

[0217] The metadata generating unit 125 according to the present modification generates Fig.30 The time-corresponding metadata 52 shown in FIG. 5 includes information about the tile 11 and partial slice information. In the time-corresponding metadata 52, row 53 indicates information about the tile 11. In addition, in the time-corresponding metadata 52, row 54 indicates partial slice information as in the modification (2-1) of the first embodiment. In other words, the partial slice corresponding to each tile 11 shown in row 53 is indicated by row 54.

[0218] The data extraction unit 231 of the client device 200 specifies a decoding position of a partial slice corresponding to a high LoD specified by using the time correspondence metadata for a geometry slice and an attribute slice included in a sample group specified to be decoded with a high LoD. In addition, the data extraction unit 231 specifies a decoding position of a partial slice corresponding to a low LoD specified by using the time correspondence metadata for a geometry slice and an attribute slice included in a sample group specified to be decoded with a low LoD. Then, the data extraction unit 231 extracts a partial slice group corresponding to the specified decoding position. Thereafter, for each sample group, the data extraction unit 231 outputs the extracted partial slice group corresponding to the high LoD and the extracted partial slice group corresponding to the low LoD to the decoding unit 222 for decoding.

[0219] Furthermore, in the present modification, the LoD is changed for each tile, but the metadata generating unit 125 may store the decoding position for each LoD with respect to the spatial information of each tile.

[0220] As described above, the file generation device according to the present modification example stores partial slice specific information about the geometric slices and attribute slices grouped into sample groups for each tile in the time-corresponding metadata. Then, the client device specifies the decoding position of the partial slice corresponding to the specified LoD from the time-corresponding metadata according to the LoD of each specified subsample, and extracts the partial slice from the geometric slice and the attribute slice. The client device according to the present modification example is able to simultaneously set the decoding position of the partial slice corresponding to the expected LoD of each sample from the time-corresponding metadata sample based on the decoding time of the sample of the G-PCC bitstream. Therefore, the client device according to the present modification example is able to display the LoD that changes for each tile while simplifying the process of extracting the partial slice.

[0221] [2.2 Modification example of the second embodiment (2-1)]

[0222] A modification (2-1) of the second embodiment will now be described. A file generation apparatus 100 according to this modification uses a G-PCC bit stream 1 having hierarchically structured slices. Fig.31 : is a diagram showing the correspondence relationship between sample groups and subsamples according to the modification example (2-1) of the second embodiment.

[0223] The metadata generation unit 125 according to the present modification defines each of the geometry layer and the attribute layer as a subsample. In addition, the metadata generation unit 125 sets a depth value as information indicating the depth corresponding to each layer. Then, the metadata generation unit 125 generates a SubSampleInformationBox including partial slice specific information indicating the correspondence relationship between each of the layers and the LoD. In addition, the metadata generation unit 125 uses the Fig. 27 The UnitMapEntry represented by syntax 50 in FIG. 1 associates each tile 11 with a SubSampleInformationBox having partial tile specific information. Thus, Fig.31 As shown, each sample group is associated with subsamples corresponding to the geometry layer 41 and the attribute layer 42.

[0224] The data extraction unit 231 of the client device 2 parses the ISOBMFF file of the point cloud content to obtain metadata. Then, the data extraction unit 231 specifies the geometry layer and attribute layer corresponding to the LoD of different types of LoD for each tile 11 from the partial slice specific information stored in the SubSampleInformationBox included in the metadata. Then, the data extraction unit 213 extracts the specified geometry layer and attribute layer.

[0225] Furthermore, although the above describes the case where the depth value is stored in the SubSampleInformationBox, Fig.32 As shown in syntax 55 in , the metadata generation unit 125 can store the depth value in the sample group without using the SubSampleInformationBox. Fig.32 : is a diagram showing UnitMapEntry when a depth value is stored in a sample group according to a modification example (2-1) of the second embodiment. Fig.33 is a diagram showing a state of a sample group storing depth values ​​according to a modification (2-1) of the second embodiment. Fig.33 As shown, the depth value of each layer is indicated by a subsample group with groupID=1, where groupID=1 is identification information corresponding to a specific tile 11 .

[0226] As described above, the file generation device according to the present modification generates a SubSampleInformationBox including partial slice specific information for extracting different LoDs for each layer using a G-PCC bitstream with hierarchical structure slices. The client device uses the partial slice specific information included in the SubSampleInformationBox to extract partial slices corresponding to different LoDs for each layer from the G-PCC bitstream with hierarchical structure slices, and causes the decoder to decode them. This makes it possible to display a three-dimensional image using a point cloud with different LoDs for each layer even when a hierarchical structure is used, thereby providing a viewing experience that meets the viewer's request.

[0227] [2.3 Modification of Second Embodiment (2-2)]

[0228] A modification (2-2) of the second embodiment will now be described. The file generation device 100 according to the present modification uses the G-PCC bitstream 1 having hierarchically structured slices and stores part of the slice specific information in the time-corresponding metadata.

[0229] The metadata generation unit 125 according to this modification generates time-corresponding metadata including information about the tile 11 and information about the geometry layer and the attribute layer. In this case, the time-corresponding metadata used in the modification (1) of the second embodiment Fig.30 The portion of line 54 in the time corresponding metadata 52 replaces the definition of unit_index with layer_index which is identification information about the geometry layer and attribute layer, instead of data_length_byte and data_length_bits.

[0230] The data extraction unit 231 of the client device 2 parses the ISOBMFF file of the point cloud content to obtain metadata. Then, the data extraction unit 231 specifies the geometry layer and attribute layer corresponding to the LoD of different types of LoD for each tile 11 from the partial slice specific information stored in the time-corresponding metadata. Then, the data extraction unit 213 extracts the specified geometry layer and attribute layer.

[0231] Furthermore, although this modification example uses Fig.30 However, in addition to the time corresponding metadata 52 in the metadata generation unit 125, the metadata generation unit 125 can also adopt the time corresponding metadata using the sample group. Fig.34 : is a diagram showing time-corresponding metadata when a sample group according to a modification example (2-2) of the second embodiment is used.

[0232] For example, the metadata generation unit 125 generates Fig.23 The metadata generation unit 125 sets the sample group of the tile 11 for each LoD based on these types of time-corresponding metadata 56 and 57. For example, the metadata generation unit 125 extends BrickGroupEntry and sets the definition as follows. The metadata generation unit 125 assumes that if the value of LoD stored in the time-corresponding metadata 56 is 0, the corresponding layer is a layer of LoD=0, and if the value of LoD is 1, the layer is a layer included in both LoD=0 and LoD=1.

[0233] Fig.35 : is a diagram showing the correspondence relationship between each layer and LoD according to the modification example (2-2) of the second embodiment. Fig.35 As shown, the temporal correspondence metadata 56 and 57 indicate, for example, that the geometry layers and attribute layers of unit_index=1 and 2 are layers of LoD=0 and 1, and are included in tile 11C. It is also shown that the geometry layers and attribute layers of unit_index=3 and 4 are layers of LoD=1 and are included in tile 11D.

[0234] As described above, the file generation device according to the present modification example generates time-corresponding metadata including partial slice-specific information for extracting different LoDs for each layer using a G-PCC bitstream with hierarchical structure slices. The client device uses the partial slice-specific information included in the time-corresponding metadata to extract the partial slices corresponding to the different LoDs for each layer from the G-PCC bitstream with hierarchical structure slices, and causes the decoder to decode them. This makes it possible to display a three-dimensional image using a point cloud with different LoDs for each layer from a G-PCC bitstream with hierarchical structure slices using the time-corresponding metadata, thereby providing a viewing experience that meets the viewer's request.

[0235] [3. Third embodiment]

[0236] The third embodiment is now described. The file generation device according to this embodiment is different from the file generation device of the first embodiment in that LoD scalable coding is performed on G-PCC still images. Figure 8 In addition, the client device according to this embodiment is also composed of Fig.12 and Fig.13 The block diagram in FIG. 1 is shown. The description of the functions of the respective components similar to those of the first embodiment is omitted below. The configuration according to the present embodiment described below is also applicable to the hierarchical structure slices.

[0237] Fig.36 is a diagram showing a method of storing a G-PCC still image in an ISOBMFF file. Fig.36 As shown, the metadata generation unit 125 of the file generation device 100 according to the present embodiment stores the G-PCC still image in the ISOBMFF file using MetaBox, similar to the case of storing the V-PCC still image defined in the non-patent document 4 in the ISOBMFF file. In this case, the metadata generation unit 125 causes parameter sets such as SPS, GPS, APS, and tile inventory to be included in ItemProperty("gpcC") arranged in the box 61.

[0238] Furthermore, the metadata generating unit 125 stores the partial slice specific information in SubSampleItemProperty arranged in box 61 . Fig.37 is a diagram showing an example of SubSampleItemProperty according to the third embodiment. The metadata generation unit 125 generates a SubSampleItemProperty having Fig.37 In this case, the metadata generation unit 125 generates a SubSampleItemProperty as shown in the syntax 62. Fig.37Codec_specific_parameters are defined as in syntax 63 in . Here, SubSampleItemProperty includes information similar to that of SubSampleInformationBox in the first embodiment.

[0239] The data extraction unit 231 of the client device 200 parses the acquired ISOBMFF file to extract metadata. Then, the data extraction unit 231 extracts a partial slice corresponding to the specified LoD from the bit stream of the G-CC still image by using the SubSampleItemProperty included in the metadata, and outputs the partial slice to the decoding unit 222.

[0240] In the decoding unit 222, the geometry decoder 232 decodes a partial slice of the geometry slice 13 in the G-CC still picture. In addition, in the decoding unit 222, the attribute decoder 233 decodes a partial slice of the attribute slice 14 in the G-CC still picture.

[0241] The point cloud generation unit 234 of the display information generation unit 223 generates a point cloud using the decoded geometry data and attribute data. The image processing unit 235 performs rendering, generates a three-dimensional still image of the point cloud, and displays it on the display unit 202.

[0242] As described above, the file generation device according to the present embodiment stores partial slice-specific information about the geometric slices and attribute slices of the G-PCC still image in SubSamapleItemProperty. The client device uses the SubSamapleItemProperty stored in the metadata obtained from the ISOBMFF file to extract the partial slices of the three-dimensional still image corresponding to the desired LoD, and causes the decoder to decode the extracted partial slices to display the three-dimensional still image of the point cloud. Therefore, the partial slices corresponding to the desired LoD included in the bitstream of the G-PCC still image can be decoded without relying on the decoder, providing a viewing experience that meets the viewer's request even for the G-PCC still image.

[0243] [3.1 Modification example (1) of the third embodiment]

[0244] The file generation device 100 according to this modification is different from the file generation device of the third embodiment in that, similar to the modification (2-1) of the first embodiment, the partial slice specific information for obtaining the desired LoD is stored in the time-corresponding metadata. The configuration according to this modification described below is also applicable to hierarchical structure slices. Fig.38FIG. 1 is a diagram showing a method of storing time-correlated metadata using a G-PCC still image.

[0245] The metadata generating unit 125 according to the present modification example Fig.38 In the block 64 shown in FIG. 1 , a LoDDecodeItemProperty including part of the slice specific information is newly defined and stored. For example, the metadata generation unit 125 generates the LoDDecodeItemProperty as follows: Fig.39 The time shown in the syntax corresponds to metadata 65. Fig.39 is a diagram showing exemplary time-corresponding metadata according to the modification (1) of the third embodiment. The LoDDecodeItemProperty includes Fig.18 The time correspondence metadata 36 of the syntax in the modification example (2-1) of the first embodiment shown is similar to information.

[0246] The data extraction unit 231 of the client device 200 parses the acquired ISOBMFF file to extract metadata. Then, the data extraction unit 231 specifies the decoding time corresponding to the LoD specified by using LoDDecodeItemProperty. Then, the data extraction unit 231 extracts a partial slice corresponding to the specified decoding time from the bit stream of the G-PCC still image.

[0247] As described above, the file generation device according to the present embodiment stores the partial slice specific information in the time corresponding metadata. The client device extracts the partial slice corresponding to the desired LoD by using the partial slice specific information stored in the time corresponding metadata. Therefore, the client device can simultaneously set the decoding position of the partial slice corresponding to the desired LoD from the time corresponding metadata sample based on the decoding time of the bit stream of the G-PCC still image, thereby simplifying the process of extracting the partial slice.

[0248] [3.2 Modification example (2) of the third embodiment]

[0249] The file generation device according to the present modification is different from the file generation device of the third embodiment in that the LoD is different for each tile. The configuration according to the present modification described below is also applicable to hierarchical structure slices. Fig.40 is a diagram showing a method of storing partial slice specific information in the case where a G-PCC still image has a different type of LoD for each tile.

[0250] The metadata generating unit 125 according to the present modification example Fig.40In the block 66 shown in FIG. 64 , a TileLoDDocodeItemProperty including part of the slice specific information is newly defined and stored. Fig.41 TileLoDDocodeItemProperty as shown in syntax 67. Fig.41 is a diagram showing an example of TileLoDDocodeItemProperty according to the modification example (2) of the third embodiment. Fig.30 Information similar to the time correspondence metadata 52 in the modification example (1) of the second embodiment shown.

[0251] The data extraction unit 231 of the client device 200 specifies a decoding position of a partial slice corresponding to a high LoD specified by using the time correspondence metadata of the geometric slice 13 and the attribute slice 14 included in the sample group specified to be decoded with high LoD in the G-PCC still image. In addition, the data extraction unit 231 specifies a decoding position of a partial slice corresponding to a low LoD specified by using the time correspondence metadata of the geometric slice 13 and the attribute slice 14 included in the sample group specified to be decoded with low LoD in the G-PCC still image. Then, the data extraction unit 231 extracts a partial slice group corresponding to the specified decoding position. Thereafter, the data extraction unit 231 outputs the extracted partial slice group corresponding to the high LoD and the extracted partial slice group corresponding to the low LoD to the decoding unit 222 for decoding for each sample group.

[0252] Furthermore, the metadata generation unit 125 may be further configured to acquire tile information by referring to a tile inventory included in ItemProperty ("gpcC").

[0253] As described above, the file generation device according to the present modification example stores partial slice specific information about the geometric slices and attribute slices grouped into sample groups for each tile of the G-PCC still image in the time-corresponding metadata. Then, the client device specifies the decoding position of the partial slice in the bitstream of the G-PCC still image corresponding to the specified LoD from the time-corresponding metadata according to the LoD of each specified subsample, and extracts the partial slice from the geometric slice and the attribute slice. The client device according to the present modification example is able to simultaneously set the decoding position of the partial slice corresponding to the desired LoD for each sample from the time-corresponding metadata sample based on the decoding time of the sample of the bitstream of the G-PCC still image. Therefore, the client device according to the present modification example is able to display the LoD that changes for each tile while simplifying the process of extracting the partial slice from the bitstream of the G-PCC still image.

[0254] [4. Fourth embodiment]

[0255] The above description gives the case of storing in ISOBMFF. However, even Fig.42 In case the Matroska media container (http: / / www.matroska.org / ) is used for transmission, metadata including the G-PCC bitstream 1 and some slice specific information may also be transmitted. Fig.42 is a diagram showing the format of the Matroska media container.

[0256] In this case, the file generation unit 126 stores information indicating the partial slices corresponding to each depth or information for associating the partial slice information with each tile 11 in a newly defined element in the track entry element. In addition, the file generation unit 126 stores time-corresponding metadata storing a decoding position for obtaining a desired LoD for each tile 11 or time-corresponding metadata storing a decoding position for obtaining a desired LoD in a track entry different from the track entry storing the G-PCC bitstream 1. In addition, for the bitstream of the G-PCC still image, the file generation unit 126 defines a new block using the extended file format of WebP and stores information similar to that shown in the third embodiment and its modified examples.

[0257] At this time, the file generation unit 126 sets the element type to binary and stores SceneDescriptionObjectMetadata() for the EBML data as binary data.

[0258] [Hardware Configuration]

[0259] Fig.43 is a hardware configuration diagram of a computer. The file generation device 100 and the client device 200 can be implemented as Fig.43 A computer 900 is shown. In the computer 900 , a processor 901 , a read-only memory (ROM) 902 , a random access memory (RAM) 903 , a hard disk drive 904 , and an input / output interface 905 are connected to one another via a bus 910 .

[0260] The input / output interface 905 is connected to, for example, an output device 906, an input device 907, a communication device 908, and a driver 909. The output device 906 is, for example, a display, a speaker, an output terminal, etc. The input device 907 is, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The driver 909 drives a removable recording medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0261] The communication device 908 is connected to an external network. The file generation device 100 and the client device 200 are connected to each other via the communication device 908. In addition, the file generation device 100 and the client device 200 are connected to the web server 300 via the communication device 908. The hard disk drive 904 is a nonvolatile storage device and an auxiliary storage device.

[0262] In the computer 900 configured as described above, the processor 901 loads the program stored in the hard disk drive 904 into the RAM 903 via the bus 910 and executes it, thereby performing a series of processing operations in the above-mentioned various embodiments and their modified examples. The RAM 903 also appropriately stores data and the like required for the processor 901 to perform various types of processing.

[0263] The program executed by the processor 901 is recordable and applicable to, for example, a removable recording medium 911 as a package medium, etc. In this case, by mounting the removable recording medium 911 on the drive 909, the program can be installed on the hard disk drive 904 via the input / output interface 905.

[0264] In addition, the program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, and digital satellite broadcasting. In this case, the program can be received through the communication device 908 and installed on the hard disk drive 904.

[0265] Alternatively, the program may be pre-installed on the hard disk drive 904 .

[0266] Although the above description gives the embodiments of the present disclosure, the technical scope of the present disclosure is not limited to the above embodiments, and various modifications and variations can be made without departing from the spirit and scope of the present disclosure. In addition, components covering different embodiments and modifications can be appropriately combined.

[0267] Furthermore, the effects described in this specification are merely illustrative and not restrictive, and other effects may be achieved.

[0268] Note that the present technology may also include the following configurations. (1)

[0270] An information processing device, comprising:

[0271] an encoding unit configured to encode the point cloud data into a hierarchical structure having stages corresponding to levels of detail (LoD) indicating a quality level to generate a bitstream consisting of a data region including one or more geometric slices;

[0272] a metadata generating unit configured to generate partial slice specific information for specifying a correspondence between the LoD and a partial slice that is a partial area in the geometric slice corresponding to the LoD; and

[0273] A file generating unit is configured to generate a file including the partial slice specific information. (2)

[0275] The information processing device according to (1),

[0276] wherein the metadata generation unit extracts depth information from the bitstream, the depth information indicating the depth of each layer constituting the LoD in the hierarchical structure, and

[0277] The metadata generation unit sets a portion of the geometric slice corresponding to the depth as a subsample. (3)

[0279] The information processing device according to (2),

[0280] The metadata generation unit further generates access information indicating access to each of the layers constituting the LoD. (4)

[0282] The information processing device according to (3),

[0283] The access information includes information related to the data size of the sub-sample. (5)

[0285] The information processing device according to (4),

[0286] The file generation unit stores the partial slice specific information and the access information in a subsample information box. (6)

[0288] The information processing device according to (5), wherein:

[0289] The metadata generation unit extracts information on the number of decoding points to be decoded for each of the layers as an amount of information, the number of decoding points serving as an index for selecting the depth according to processing performance of a reproduction processing device configured to reproduce three-dimensional content, and

[0290] The file generation unit stores the amount of information in the file. (7)

[0292] The information processing device according to (6), wherein:

[0293] The portion of the slice specific information includes the access information, and

[0294] The metadata generation unit stores the partial slice specific information in time-corresponding metadata. (8)

[0296] The information processing device according to (7),

[0297] The data size related information includes identification information of the bit stream and data length related information starting from the head of the bit stream. (9)

[0299] The information processing device according to (8), wherein:

[0300] The metadata generating unit extracts information on a number of decoding points to be decoded for each of the layers as information amount, the number of decoding points serving as an index for selecting the LoD according to processing performance of a reproduction processing device configured to reproduce the three-dimensional content, and

[0301] The file generation unit stores the amount of information in the file. (10)

[0303] The information processing device according to (3), wherein:

[0304] The bitstream also includes an attribute slice indicating additional attributes such as color and texture, and

[0305] The access information includes type information, and the type information indicates whether the subsample corresponds to the geometry slice or the attribute slice. (11)

[0307] The information processing device according to (3), wherein:

[0308] The point cloud data is composed of multiple blocks.

[0309] The bitstream also includes an attribute slice indicating additional attributes such as color and texture, and

[0310] The metadata generation unit generates, for each tile included in one frame of the bitstream, the portion of slice-specific information corresponding to the LoD among a plurality of slices corresponding to the tile. (12)

[0312] The information processing device according to (11),

[0313] The file generation unit associates the partial slice-specific information of each tile with a subsample information box including the partial slice-specific information. (13)

[0315] The information processing device according to (11),

[0316] The file generation unit stores the partial slice-specific information of each of the tiles in time-corresponding metadata for each LoD information indicating the quality level as the hierarchical information. (14)

[0318] The information processing device according to (11), wherein:

[0319] The encoding unit generates the bit stream having hierarchical units divided for each layer, and

[0320] The metadata generating unit generates the partial slice specific information corresponding to the slice included in each of the hierarchical units. (15)

[0322] The information processing device according to (1),

[0323] The file generation unit stores the bit stream without time information and part of the slice specific information in the file. (16)

[0325] An information processing device, comprising:

[0326] an encoding unit configured to encode the point cloud data into a hierarchical structure having stages corresponding to LoDs indicating quality levels to generate a bitstream and depth information, the bitstream being composed of a data region including one or more geometry slices and one or more attribute slices, the depth information indicating a depth of each layer constituting the LoD in the hierarchical structure;

[0327] a metadata generating unit configured to generate partial slice specific information and set the portion of the geometric slice corresponding to the depth as a subsample, wherein the partial slice specific information is used to specify a correspondence between the LoD and a partial slice that is a partial area of ​​the geometric slice corresponding to the LoD; and

[0328] A file generating unit is configured to generate a file including the partial slice specific information. (17)

[0330] An information processing method causes a computer to perform a process, the process comprising:

[0331] encoding the point cloud data into a hierarchical structure having a stage corresponding to a LoD indicating a quality level to generate a bitstream consisting of a data region including one or more geometric slices;

[0332] generating partial slice specific information for specifying a correspondence between the LoD and a partial slice that is a partial area in the geometric slice corresponding to the LoD; and

[0333] A file is generated including information specific to the partial slice. (18)

[0335] A reproduction processing device, comprising:

[0336] A file acquisition unit configured to receive a file including a bitstream and partial slice specific information, the bitstream being composed of a data region including one or more geometric slices by encoding point cloud data into a staged hierarchical structure corresponding to a LoD indicating a quality level, the partial slice specific information being used to specify a correspondence relationship between the LoD and a partial slice that is a partial region in the geometric slice corresponding to the LoD;

[0337] a data extraction unit configured to extract the portion of slice specific information from the file acquired by the file acquisition unit; and

[0338] A decoding unit is configured to decode the bitstream at the desired quality level based on the partial slice-specific information extracted by the data extraction unit. (19)

[0340] A reproduction processing method causes a computer to perform processing, the processing comprising:

[0341] receiving a file including a bitstream and partial slice specific information, the bitstream being composed of a data region including one or more geometric slices by encoding point cloud data into a staged hierarchical structure corresponding to a LoD indicating a quality level, the partial slice specific information being used to specify a correspondence relationship between the LoD and a partial slice being a partial region in the geometric slice corresponding to the LoD;

[0342] extracting the portion of slice-specific information from the acquired file; and

[0343] The bitstream is decoded at a desired quality level based on the extracted partial slice-specific information.

[0344] Reference numerals list

[0345] 1 G-PCC bitstream

[0346] 10 Point Cloud Framework

[0347] 11 tiles

[0348] 12 Slices

[0349] 13 Geometric Slices

[0350] 14 Attribute Slicing

[0351] 15 o'clock

[0352] 100 File generation device

[0353] 101 Extraction Unit

[0354] 102 File generation processing unit

[0355] 103 Transmission Unit

[0356] 104 Control Unit

[0357] 121 coding units

[0358] 122 Geometry Encoder

[0359] 123 Attribute Encoder

[0360] 124 Bitstream generation unit

[0361] 125 Metadata Generation Unit

[0362] 126 File Generation Unit

[0363] 200 Client Devices

[0364] 201 Reproduction processing unit

[0365] 202 Display unit

[0366] 203 Control Unit

[0367] 211 File Acquisition Unit

[0368] 212 Decoding Processing Unit

[0369] 221 File Processing Unit

[0370] 222 Decoding Unit

[0371] 223 Display information generation unit

[0372] 231 Data Extraction Unit

[0373] 232 Geometry Decoder

[0374] 233 Attribute Decoder

[0375] 234 Point cloud generation unit

[0376] 235 Image Processing Unit

[0377] 300 web server

[0378] 400 Network

[0379] 500 Delivery System

Claims

1. An information processing device, comprising: an encoding unit configured to encode the point cloud data into a hierarchical structure having a stage corresponding to a level of detail LoD indicating a quality level to generate a bitstream consisting of a data region including one or more geometric slices; A metadata generation unit, which is configured to: generating partial slice specific information, the partial slice specific information being used to specify a correspondence between the LoD and a partial slice that is a partial area in the geometric slice corresponding to the LoD; extracting depth information from the bitstream, the depth information indicating a depth of each layer constituting the LoD in the hierarchical structure; Setting a portion of the geometric slice corresponding to the depth as a subsample; and generating access information indicating access to each of the layers constituting the LoD, wherein the access information includes information related to a data size of the subsample; as well as The file generating unit is configured to generate an International Organization for Standardization Base Media File Format (ISOBMFF) file including the partial slice specific information, and store the partial slice specific information and the access information in a subsample information box.

2. The information processing device according to claim 1, wherein: The metadata generation unit extracts information on the number of decoding points to be decoded for each of the layers as an amount of information, the number of decoding points serving as an index for selecting the depth according to processing performance of a reproduction processing device configured to reproduce three-dimensional content, and The file generation unit stores the amount of information in the ISOBMFF file.

3. The information processing device according to claim 2, wherein: The portion of the slice specific information includes the access information, and The metadata generation unit stores the partial slice specific information in time-corresponding metadata.

4. The information processing device according to claim 3, in, The data size related information includes identification information of the bit stream and data length related information starting from the head of the bit stream.

5. The information processing device according to claim 4, wherein: The metadata generating unit extracts information on a number of decoding points to be decoded for each of the layers as information amount, the number of decoding points serving as an index for selecting the LoD according to processing performance of a reproduction processing device configured to reproduce the three-dimensional content, and The file generation unit stores the amount of information in the ISOBMFF file.

6. The information processing device according to claim 1, wherein: The bitstream further comprises an attribute slice indicating additional attributes of color and / or texture, and The access information includes type information, and the type information indicates whether the subsample corresponds to the geometry slice or the attribute slice.

7. The information processing device according to claim 1, wherein: The point cloud data is composed of multiple blocks. The bitstream further comprises an attribute slice indicating additional attributes of color and / or texture, and The metadata generation unit generates, for each tile included in one frame of the bitstream, the portion of slice-specific information corresponding to the LoD among a plurality of slices corresponding to the tile.

8. The information processing device according to claim 7, in, The file generation unit associates the partial slice specific information of each of the tiles with a subsample information box including the partial slice specific information.

9. The information processing device according to claim 7, in, The file generation unit stores the partial slice specific information of each of the tiles in the time-corresponding metadata for each LoD information indicating the quality level as the hierarchical information of the hierarchical structure.

10. The information processing device according to claim 7, wherein: The encoding unit generates the bit stream having hierarchical units divided for each layer, and The metadata generating unit generates the partial slice specific information corresponding to the slice included in each of the hierarchical units.

11. The information processing device according to claim 1, in, The file generation unit stores a bitstream without time information and part of the slice specific information in the ISOBMFF file.

12. An information processing device, comprising: an encoding unit configured to encode the point cloud data into a hierarchical structure having stages corresponding to a level of detail LoD indicating a quality level, so as to generate a bitstream and depth information, the bitstream being composed of a data region including one or more geometry slices and one or more attribute slices, the depth information indicating a depth of each layer constituting the LoD in the hierarchical structure; A metadata generation unit, which is configured to: generating partial slice specific information, the partial slice specific information being used to specify a correspondence between the LoD and a partial slice that is a partial area in the geometric slice corresponding to the LoD; extracting from the bitstream the depth information indicating the depth of each layer constituting the LoD in the hierarchical structure; Setting a portion of the geometric slice corresponding to the depth as a subsample; generating access information indicating access to each of the layers constituting the LoD, wherein the access information includes information related to a data size of the subsample; as well as The file generating unit is configured to generate an International Organization for Standardization Base Media File Format (ISOBMFF) file including the partial slice specific information, and store the partial slice specific information and the access information in a subsample information box.

13. An information processing method, causing a computer to perform a process, the process comprising: encoding the point cloud data into a hierarchical structure having a staged hierarchy corresponding to a level of detail LoD indicating a level of quality to generate a bitstream consisting of a data region including one or more geometric slices; generating partial slice specific information, the partial slice specific information being used to specify a correspondence between the LoD and a partial slice that is a partial area in the geometric slice corresponding to the LoD; extracting depth information from the bitstream, the depth information indicating a depth of each layer constituting the LoD in the hierarchical structure; Setting a portion of the geometric slice corresponding to the depth as a subsample; and generating access information indicating access to each of the layers constituting the LoD, wherein the access information includes information related to a data size of the subsample; Generate an ISOBMFF file including the specific information of the partial slices; as well as The partial slice specific information and the access information are stored in a subsample information box.

14. A reproduction processing device, comprising: A file acquisition unit is configured to receive an ISOBMFF file including a bitstream and partial slice specific information, wherein the bitstream is composed of a data region including one or more geometric slices by encoding point cloud data into a hierarchical structure having a stage corresponding to a level of detail LoD indicating a quality level, and the partial slice specific information is used to specify a correspondence between the LoD and a partial slice that is a partial region in the geometric slice corresponding to the LoD, wherein the bitstream includes depth information indicating a depth of each layer constituting the LoD in the hierarchical structure; a data extraction unit configured to extract the partial slice specific information from the ISOBMFF file acquired by the file acquisition unit; as well as a decoding unit configured to decode the bitstream at the desired quality level based on the partial slice specific information extracted by the data extraction unit, wherein the partial slice specific information and access information are stored in a subsample information box, and the access information includes data size related information of the subsample, wherein the access information indicates access to each of the layers constituting the LoD, and the subsample corresponds to a portion of the geometric slice corresponding to the depth.

15. A reproduction processing method, causing a computer to perform processing, the processing comprising: receiving an ISOBMFF file including a bitstream and partial slice specific information, wherein the bitstream is composed of a data region including one or more geometric slices by encoding point cloud data into a hierarchical structure having a stage corresponding to a level of detail LoD indicating a quality level, and the partial slice specific information is used to specify a correspondence relationship between the LoD and a partial slice that is a partial region of the geometric slice corresponding to the LoD, wherein the bitstream includes depth information indicating a depth of each layer constituting the LoD in the hierarchical structure; extracting the portion of slice-specific information from the acquired ISOBMFF file; as well as The bitstream is decoded at a desired quality level based on the extracted partial slice specific information, wherein the partial slice specific information and access information are stored in a subsample information box, and the access information includes data size related information of the subsample, wherein the access information indicates access to each of the layers constituting the LoD, and the subsample corresponds to a portion of the geometric slice corresponding to the depth.

Images