Information processing apparatus and method

Abstract

The present disclosure relates to an information processing device and method capable of mitigating a reduction in processing efficiency of decoding processing. A scene description representing a scene constituted by a 3D object is generated, and first reference information is set in the scene description, the first reference information pointing to a first buffer for storing sub-mesh association information for specifying data of a sub-mesh that is part of a mesh representing the 3D object. The sub-grid association information is stored in the first buffer, the sub-grid association information is acquired from the first buffer, data of the sub-grid is specified using the acquired sub-grid association information, and the grid is reconstructed. The present disclosure can be applied to, for example, an information processing device or an information processing method.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an information processing apparatus and method, and more particularly, to an information processing apparatus and method capable of suppressing reduction in processing efficiency of decoding processing. BACKGROUND

[0002] Conventionally, there is GL Transmission Format (glTF) (registered trademark) 2.0, which is a format for scene description for setting and rendering a three-dimensional (3D) object in a three-dimensional space (for example, see Non-Patent Literature 1). Further, in MPEG-I Scene Description (Moving Picture Experts Group (MPEG)-I Scene Description), a method of extending this glTF 2.0 and processing dynamic content in the time direction is proposed (for example, see Non-Patent Literature 2). This MPEG-I Scene Description supports Video-based Point Cloud Compression (V-PCC) as a point cloud encoding scheme as one of 3D data (for example, see Non-Patent Literature 3).

[0003] Meanwhile, as an encoding method of a mesh that is 3D data representing a three-dimensional structure of an object by connection with a vertex, there is Video-based Dynamic Mesh Coding (V-DMC) (for example, see Non-Patent Literatures 5 to 7).

[0004] In recent years, a method of storing a bitstream encoded by V-DMC (also referred to as a V-DMC bitstream) in International Standardization Organization Base Media File Format (ISOBMFF) or the like and distributing the bitstream is desired to be established. For example, it is conceivable to extend and standardize ISO / IEC 23090-10 as a distribution technology standard of V-PCC (for example, see Non-Patent Literature 8). In this case, the V-DMC bitstream is stored in a track of ISOBMFF.

[0005] Bibliographic List

[0006] Non-Patent Literature

[0007] Non-Patent Literature 1: Saurabh Bhatia, Patrick Cozzi, Alexey Knyazev, Tony Parisi, “Khronos glTF2.0”, https: / / github.com / KhronosGroup / glTF / tree / master / specification / 2.0, June 9, 2017 Non-Patent Literature 2: “Information technology - Coded representation of immersive media - Part 14: Scene Description”, ISO / IEC DIS 23090-14:2021(E), ISO / IEC JTC 1 / SC 29 / WG 03N00485, 137th MPEG Meeting, January 2022, Online Non-Patent Literature 3: “Potential improvements of ISO / IEC 23090-14 DAM 1 Support for immersive media codecs in scene description”, ISO / IEC JTC 1 / SC 29 / WG 03N00795, 2023-02-04

[0008] Non-patent literature 4: Khaled Mammou, Jungsun Kim, Alexis Tourapis, Dimitri Podborski, Krasimir Kolarov, "[V-CG] Apple's Dynamic Mesh Coding CfP Response", ISO / IEC JTC 1 / SC 29 / WG 7 m59281, April 2022 Non-patent literature 5: Jungsun Kim, Alexis Tourapis, Dimitri Podborski, Khaled Mammou, "VDMC support in the V3C framework", ISO / IEC JTC 1 / SC 29 / WG 7 m60748, October 2022 Non-patent literature 6: Alexis Tourapis, Jungsun Kim, Dimitri Podborski, Khaled Mammou, "Base mesh data substream format for VDMC", ISO / IEC JTC 1 / SC 29 / WG 7 m60362, July 2022 Non-patent literature 7: "WD 2.0 of V-DMC", ISO / IEC JTC 1 / SC 29 / WG 07 N0546, MPEG 141st Meeting, 2023-1

[0009] Non-patent literature 8: "Text of ISO / IEC FDIS 23090-10 Carriage of Visual Volumetric Video-based Coding Data", ISO / IEC JTC 1 / SC 29 / WG 03 N00241, 2021 / 8 / 20, ISO / IEC 23090-10:2022, 1st edition, 2022-05 SUMMARY

[0010] Problems to be solved by the Invention

[0011] According to the V-DMC coding, it is possible to transmit the non-registered mesh data at a higher compression rate than before, and it is expected that it will also be supported in the MPEG-I scene description as one of the coding techniques for 3D objects constituting a scene. In this case, it is assumed that the process for reconstructing 3D data from V-DMC data, such as subdivision of the base mesh, refinement of the vertex position by displacement vector, mapping of attributes, and the like, is performed in a higher processing efficiency in a presentation engine (PE) than in the case of a media access function (MAF).

[0012] However, the correspondence between the patch and the sub-mesh cannot be grasped in the PE, and it is difficult to reconstruct the 3D data from the V-DMC data in the PE. Thus, it is possible to reduce the processing efficiency of the decoding process.

[0013] The present disclosure is made in view of such circumstances, and an object thereof is to suppress reduction in processing efficiency of a decoding process.

[0014] Solution to the problem

[0015] An information processing apparatus according to an aspect of the present technology is an information processing apparatus including a scene description generation unit that generates a scene description that represents a scene composed of 3D objects, and sets first reference information to a first buffer that stores sub-mesh association information in the scene description, wherein the sub-mesh association information is information that specifies data of a sub-mesh that is a part of a mesh that represents a 3D object.

[0016] An information processing method according to an aspect of the present technology is an information processing method including generating a scene description that represents a scene composed of 3D objects, and setting first reference information to a first buffer that stores sub-mesh association information in the scene description, wherein the sub-mesh association information is information that specifies data of a sub-mesh that is a part of a mesh that represents a 3D object.

[0017] An information processing apparatus according to another aspect of the present technology is an information processing apparatus including a storage processing unit that stores sub-mesh association information that specifies data of a sub-mesh that is a part of a mesh that represents a 3D object in a first buffer specified by a scene description that represents a scene composed of 3D objects, and a reconstruction unit that acquires the sub-mesh association information from the first buffer, specifies data of the sub-mesh using the acquired sub-mesh association information, and reconstructs the mesh.

[0018] An information processing method according to another aspect of the present technology is an information processing method including storing sub-mesh association information that specifies data of a sub-mesh that is a part of a mesh that represents a 3D object in a first buffer specified by a scene description that represents a scene composed of 3D objects, and acquiring the sub-mesh association information from the first buffer, specifying data of the sub-mesh using the acquired sub-mesh association information, and reconstructing the mesh.

[0019] In an information processing apparatus and method according to an aspect of the present technology, a scene description that represents a scene composed of 3D objects is generated, and first reference information to a first buffer that stores sub-mesh association information is set in the scene description.

[0020] In an information processing apparatus and method according to another aspect of the present technology, sub-mesh association information for specifying data of a sub-mesh that is a part of a mesh representing a 3D object is stored in a first buffer specified by a scene description representing a scene composed of 3D objects, the sub-mesh association information is acquired from the first buffer, data of the sub-mesh is specified using the acquired sub-mesh association information, and the mesh is reconstructed. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a diagram showing a main configuration example of glTF 2.0.

[0022] Figure 2 is a diagram showing an example of glTF objects and reference relationships.

[0023] Figure 3 is a diagram showing a description example of a scene description.

[0024] Figure 4 is a diagram for describing a method of accessing binary data.

[0025] Figure 5 is a diagram showing a description example of a scene description.

[0026] Figure 6 is a diagram for describing an object extension method.

[0027] Figure 7 is a diagram showing a configuration of client processing.

[0028] Figure 8 is a diagram showing a configuration example for processing extensions of timing metadata.

[0029] Figure 9 is a diagram showing a description example of a scene description.

[0030] Figure 10 is a diagram showing an example of a point cloud reconstruction processing method in a case where V-PCC is applied.

[0031] Figure 11 is a diagram showing an example structure of a scene description in a case of reconstruction in MAF.

[0032] Figure 12 is a diagram showing an example structure of a scene description in a case of reconstruction in PE.

[0033] Figure 13 is a diagram showing an example of patch information in V-PCC.

[0034] Figure 14 is a diagram for explaining a mesh.

[0035] Figure 15 is a diagram for explaining V-DMC.

[0036] Figure 16 is a diagram showing an example of decoding processing of a V-DMC bitstream.

[0037] Figure 17 is a diagram for explaining a submesh.

[0038] Figure 18 is a diagram for explaining a submesh.

[0039] Figure 19 is a diagram for explaining a multi-track.

[0040] Figure 20 is a diagram showing an example of a method of supporting V-DMC data in a scene description.

[0041] Figure 21 is a diagram showing an example of an outline of decoding processing in a case where 3D data is reconstructed from V-DMC data in a MAF.

[0042] Figure 22 is a diagram showing an example of an outline of decoding processing in a case where 3D data is reconstructed from V-DMC data in a PE.

[0043] Figure 23 is a diagram showing an example of patch information in V-DMC encoding.

[0044] Figure 24 is a diagram showing an example of a scene description in a case where 3D data is reconstructed from V-DMC data in a PE.

[0045] Figure 25 is a diagram showing a description example of a scene description.

[0046] Figure 26 is a diagram for explaining an association between a patch and a submesh.

[0047] Figure 27 is a diagram showing an example of application of submesh association information.

[0048] Figure 28 is a diagram showing an example of a scene description in a case where submesh association information is applied.

[0049] Figure 29 is a diagram showing an example of submesh association information.

[0050] Figure 30 is a diagram showing an example of a scene description corresponding to submesh association information.

[0051] Figure 31 FIG. 1 is a diagram showing an example of sub-grid association information.

[0052] Figure 32 FIG. 2 is a diagram showing an example of a scenario description in a case where sub-grid association information is applied.

[0053] Figure 33 FIG. 3 is a diagram showing an example of sub-grid association information.

[0054] Figure 34 FIG. 4 is a diagram showing an example of a scenario description in a case where sub-grid association information is applied.

[0055] Figure 35 FIG. 5 is a diagram showing an example of sub-grid association information.

[0056] Figure 36 FIG. 6 is a diagram showing an example of an outline of a decoding process in a case where 3D data is reconstructed from V-DMC data in a PE.

[0057] Figure 37 FIG. 7 is a diagram showing an example of subdivision information.

[0058] Figure 38 FIG. 8 is a diagram showing an example of subdivision information.

[0059] Figure 39 FIG. 9 is a block diagram showing an example of a main configuration of a file generation apparatus.

[0060] Figure 40 FIG. 10 is a block diagram showing an example of a main configuration of a V-DMC data encoding unit.

[0061] Figure 41 FIG. 11 is a flowchart showing an example of a flow of a file generation process.

[0062] Figure 42 FIG. 12 is a flowchart showing an example of a flow of a V-DMC encoding process.

[0063] Figure 43 FIG. 13 is a flowchart showing an example of a flow of a file generation process.

[0064] Figure 44 FIG. 14 is a flowchart showing an example of a flow of a file generation process.

[0065] Figure 45 FIG. 15 is a flowchart showing an example of a flow of a file generation process.

[0066] Figure 46 FIG. 16 is a flowchart showing an example of a flow of a file generation process.

[0067] Figure 47 FIG. 17 is a flowchart showing an example of a flow of a file generation process.

[0068] Figure 48 is a flowchart showing an example of a flow of file generation processing.

[0069] Figure 49 is a block diagram showing an example of a main configuration of a reproduction apparatus.

[0070] Figure 50 is a block diagram showing an example of a main configuration of a MAF.

[0071] Figure 51 is a flowchart showing an example of a flow of reproduction processing.

[0072] Figure 52 is a flowchart showing an example of a flow of MAF processing.

[0073] Figure 53 is a block diagram showing an example of a main configuration of a reproduction apparatus.

[0074] Figure 54 is a block diagram showing an example of a main configuration of a MAF.

[0075] Figure 55 is a flowchart showing an example of a flow of reproduction processing.

[0076] Figure 56 is a flowchart showing an example of a flow of MAF processing.

[0077] Figure 57 is a block diagram showing an example of a main configuration of a MAF.

[0078] Figure 58 is a flowchart showing an example of a flow of MAF processing.

[0079] Figure 59 is a block diagram showing an example of a main configuration of a MAF.

[0080] Figure 60 is a flowchart showing an example of a flow of MAF processing.

[0081] Figure 61 is a block diagram showing an example of a main configuration of a reproduction apparatus.

[0082] Figure 62 is a block diagram showing an example of a main configuration of a MAF.

[0083] Figure 63 is a flowchart showing an example of a flow of reproduction processing.

[0084] Figure 64 is a flowchart showing an example of a flow of MAF processing.

[0085] Figure 65 is a block diagram illustrating a main configuration example of the MAF.

[0086] Figure 66 is a flowchart illustrating an example of the flow of the MAF processing.

[0087] Figure 67 is a block diagram illustrating a main configuration example of the MAF.

[0088] Figure 68 is a flowchart illustrating an example of the flow of the MAF processing.

[0089] Figure 69 is a block diagram illustrating a main configuration example of the computer. DETAILED DESCRIPTION

[0090] Hereinafter, modes for implementing the present disclosure (hereinafter, referred to as embodiments) will be described. Note that the description will be made in the following order.

[0091] 1. Literature and the like supporting technical contents and technical terms

[0092] 2. Scenario description and V-DMC

[0093] 3. V-DMC data distribution using scenario description

[0094] 4. First embodiment (file generation device)

[0095] 5. Second embodiment (reproduction device)

[0096] 6. Supplementary explanation

[0097] <1. Literature and the like supporting technical contents and technical terms>

[0098] The range disclosed in the present technology includes, in addition to the contents disclosed in the embodiments, the contents known at the time of filing described in the following non-patent literature and the like, the contents of other documents cited in the following non-patent literature and the like.

[0099] Non-Patent Literature 1: (as described above)

[0100] Non-Patent Literature 2: (as described above)

[0101] Non-Patent Literature 3: (as described above)

[0102] Non-Patent Literature 4: (as described above)

[0103] Non-Patent Literature 5: (as described above)

[0104] Non-Patent Literature 6: (as described above)

[0105] Non-Patent Literature 7: (as described above)

[0106] Non-Patent Literature 8: (as described above)

[0107] That is, the content described in the above Non-Patent Literature, the content of other documents cited in the above Non-Patent Literature, and the like are also bases for determining the support requirements.

[0108] <2. Scene Description and V-DMC>

[0109] gltf2.0

[0110] Conventionally, for example, as described in Non-Patent Literature 1, there is glTF (GL Transmission Format) (registered trademark) 2.0 as a format of a Scene Description for arranging and rendering a 3D (three-dimensional) object in a region (for example, a three-dimensional space). For example, as shown in Figure 1 glTF 2.0 includes a JSON format file (.glTF), a binary file (.bin), and an image file (.png,.jpg, etc.). The binary file stores binary data such as geometry and animation. The image file stores data such as a texture.

[0111] The JSON format file is a Scene Description file described in JavaScript (registered trademark) Object Notation (JSON). The Scene Description is metadata that describes (a description of) a scene of 3D content. The description of the Scene Description defines what kind of scene the scene is. The Scene Description file is a file that stores such a Scene Description.

[0112] The description of the JSON format file includes a list of pairs of a KEY and a VALUE. Examples of the format of the description will be described below.

[0113] "KEY":"VALUE"

[0114] The KEY includes a string. The VALUE includes a numerical value, a string, a true / false value, an array, an object, a null value, and the like.

[0115] In addition, a plurality of pairs of a KEY and a VALUE ("KEY":"VALUE") can be put together using {} (braces). The pairs put together in the braces are also called a JSON object. Examples of the format of the description will be described below.

[0116] "user":{"id":1,"name":"tanaka"}

[0117] In the case of this example, a JSON object in which the pair of "id": 1 and the pair of "name": "tanaka" are put together is defined as a value corresponding to the key (user).

[0118] Further, zero or more values can be made into an array by using []. The array is also called a JSON array. For example, a JSON object can be applied as an element of a JSON array. Examples of the format described will be described below.

[0119] test":["hoge","fuga","bar"]

[0120] "users":[{"id":1,"name":"tanaka"},{"id":2,"name":"yamada"},{"id":3,"name":"sato"}]

[0121] Figure 2 The glTF objects that can be described at the top of a JSON format file and the reference relationship they have are shown. Figure 2 The long circles in the tree structure shown indicate objects, and the arrows between the objects indicate the reference relationship. As Figure 2 As shown, objects such as "scene", "node", "mesh", "camera", "skin", "material", and "texture" are described at the top of the JSON format file.

[0122] Figure 3 A description example of such a JSON format file (scene description) is shown. Figure 3 A description example of a part of the top of the JSON format file 20 is shown. In the JSON format file 20, all of the top-level objects 21 used are described at the top. The top-level objects 21 are Figure 2 glTF objects shown. Further, in the JSON format file 20, as indicated by the arrows 22, the reference relationship between the objects is indicated. More specifically, the reference relationship is indicated by specifying the index of the element in the array of the object to be referenced by the property of the superior object.

[0123] Figure 4 is a diagram for describing a method of accessing binary data. As Figure 4 shown, the binary data is stored in a buffer object. That is, information for accessing the binary data in the buffer object (for example, a uniform resource identifier (URI), and the like) is indicated. In the JSON format file, as Figure 4As shown, a buffer object can be accessed from an object such as a mesh, a camera, and a skin, for example, via an accessor object and a bufferView object.

[0124] That is, in an object such as a mesh, a camera, or a skin, an accessor object to be referred to is specified. Figure 5 An example of a description of a mesh object in a JSON format file is shown. For example, as shown in Figure 5 In the mesh object, attributes of vertices such as NORMAL, POSITION, TANGENT, and TEXCORD_0 are defined as keys, and accessor objects to be referred to are specified as values of each attribute, as shown.

[0125] Next, an extension of an object of such a scene description will be described. Each object of glTF 2.0 can store a newly defined object in an extension object. Figure 6 An example of a description in a case where a newly defined object (ExtensionExample) is specified is shown. As shown in Figure 6 In a case where a newly defined extension is used, an extension object name (ExtensionExample in the example of Figure 6 is described in “extensionUsed” and “extensionRequired”. Thus, it is indicated that the extension is an extension that is used or an extension that is required for loading.

[0126] <CLIENT PROCESSING>

[0127] MPEG-I scene description is a standard for controlling reproduction of 6DoF visual using such a scene description conforming to glTF 2.0. Here, visual is visual information (information transmitted using visual) such as an image, and 6DoF visual indicates visual corresponding to movement (so-called free viewpoint) of six degrees of freedom (6DoF) of a viewer (receiver of visual information). That is, in MPEG-I scene description, reproduction of 6DoF visual (that is, reproduction of a visual scene) is controlled using a scene description representing a visual scene. Here, a visual scene indicates a scene related to visual. A visual scene includes visual objects arranged in a region (for example, a three-dimensional space). A visual object is an object existing in a region, and is visually constituted. That is, a reproduction device in MPEG-I scene description reproduces 6DoF visual and reconstructs a visual scene indicated by a scene description.

[0128] A scene description conforming to the MPEG-I scene description represents a visual scene including a high-definition visual object. Note that, in this specification, the scene description conforming to the MPEG-I scene description can also be referred to as an MPEG-I scene description.

[0129] Next, the processing of the client device in the MPEG-I scene description will be described. The client device acquires a scene description, acquires data of a 3D object based on the scene description, and generates a display image using the scene description and the data of the 3D object.

[0130] As described in Non-Patent Literature 2, in the client device, a rendering engine, a media access function, and the like perform processing. For example, as shown in Figure 7 the initial value of the scene description and information for updating the scene description (hereinafter, also referred to as update information), and generates the scene description at a processing target time. Then, the rendering engine 51 parses the scene description, and specifies media (moving images, audio, and the like) to be reproduced. Then, the rendering engine 51 requests a media access function 52 to acquire the media via a media access API (Application Program Interface). Further, the rendering engine 51 also performs setting of a pipeline process, designation of a buffer, and the like.

[0131] The media access function 52 acquires various data of the media requested by the rendering engine 51 from a cloud, a local storage, and the like. The media access function 52 provides the acquired various data (encoded data) of the media to a pipeline 53.

[0132] The pipeline 53 decodes the provided various data (encoded data) of the media by a pipeline process, and provides the decoding result to a buffer 54. The buffer 54 holds the provided various data of the media.

[0133] The rendering engine 51 performs rendering and the like using the various data of the media held in the buffer 54.

[0134] <Applications of timed media>

[0135] In recent years, for example, as described in Non-Patent Literature 2, in the MPEG-I scene description, glTF2.0 has been extended and timed media has been applied as 3D object content. Timed media is media data that changes in a time axis direction like a moving image in a two-dimensional image.

[0136] glTF is only applied to still image data as media data (3D object content). That is, glTF is not applied to media data of moving images. In the case where a 3D object is moved, an animation (a method of switching still images along a time axis) is applied.

[0137] In MPEG-I scene description, it has been studied to apply glTF 2.0, to apply a JSON format file as scene description, and to further extend glTF so that timed media (for example, video data) can be processed as media data. For example, in order to process timed media, the following extension is performed.

[0138] Figure 8 is a diagram for describing an extension for processing timed media. In Figure 8 In the example of

[0139] Further, as shown in Figure 8 , an MPEG texture video object (MPEG_texture_video) is provided as an extension object (extensions) of a texture object (texture). In the MPEG texture video object, information about an accessor corresponding to a buffer object to be accessed is stored. That is, the MPEG texture video object is an object that specifies an index of an accessor corresponding to a buffer in which texture media specified by the MPEG media object (MPEG_media) is decoded and stored.

[0140] Figure 9 is a diagram showing a description example of an MPEG media object (MPEG_media) and an MPEG texture video object (MPEG_texture_video) in a scene description for describing an extension for processing timed media. In Figure 9 In the case of the example of

[0141] "texture":[{"sampler":0,"source":1,"extensions":{"MPEG_texture_video":"accessor":2}}].

[0142] In addition, Figure 9 In the example case, lines 7 through 16 from the top are set to glTF extensions as described below. Then, as the value of the MPEG media object, various information related to the MPEG media object is stored, such as the MPEG media object's encoding and URI.

[0143] "MPEG_media":{

[0144] "media":[

[0145] {"name":"source_1","startTime":9.0,

[0146] "loop":"true","controls":"false",

[0147] "alternatives":[{"mimeType":"video / mp4;codecs=￥"avc1.42E01E￥"",

[0148] "uri":"video1.mp4",

[0149] "tracks":[{"track":""#track_ID=1"}]

[0150] }]

[0151] } ]

[0153] }

[0154] Additionally, each frame of data is decoded and stored sequentially in a buffer, but its position and other parameters fluctuate. Therefore, the scene description has a mechanism to store this fluctuation information so that the renderer can read the data. For example, ... Figure 8As shown, an MPEG buffer circular object (MPEG_buffer_circular) is provided as an extension object (extensions) of the buffer object (buffer). Information for dynamically storing data in the buffer object is stored in the MPEG buffer circular object. For example, information such as access information to the MPEG media and information indicating the number of frames is stored in the MPEG buffer circular object.

[0155] Further, as Figure 8 As shown, an MPEG accessor timed object (MPEG_accessor_timed) is provided as an extension object (extensions) of the accessor object (accessor). In this case, since the media data is a moving image, the buffer view object (bufferView) to be referred to in the time direction can change (the position can change). Therefore, information indicating the buffer view object to be referred to is stored in the MPEG accessor timed object. For example, the MPEG accessor timed object stores information indicating reference to the buffer view object (bufferView) in which a timed accessor information header is described. Note that the timed accessor information header is header information that stores information in, for example, a dynamically changed accessor object and buffer view object.

[0156] <V-PCC Support>

[0157] MPEG-I scene description supports video-based point cloud compression (V-PCC), which is a point cloud encoding scheme that is one of 3D data. As a method of reconstructing a point cloud, Non-Patent Literature 3 proposes a method of reconstruction in a media access function (MAF) (MAF reconstruction) and a method of reconstruction in a presentation engine (PE) (PE reconstruction) as shown in Figure 10

[0158] In the case of MAF reconstruction, a central processing unit (CPU) can be used to reconstruct the point cloud. In this case, a buffer stores the geometry data (position) and attribute data (color) of the reconstructed 3D data. That is, the reconstructed 3D data is provided to the PE. The scene description is as Figure 11 ​Configure it as shown. In the POSITION property of the attribute object in mesh.primitives, an accessor for the buffer that stores position information about points is specified. Similarly, in the COLOR property of the attribute object, an accessor for the buffer that stores color information about points is specified.

[0159] In the case of PE reconstruction, a graphics processing unit (GPU) can be used to reconstruct and render the point cloud. In this scenario, the pre-reconstruction V-PCC data (such as geometry, attributes, and occupied video data (decoded video) as well as atlas information (patch information)) is stored in a buffer. That is, the pre-reconstruction V-PCC data is provided to the PE. The scene description is as follows: Figure 12 Configure it as shown. Characteristics such as _MPEG_V3C_AVD, _MPEG_V3C_OVD_MAPS, _MPEG_V3C_GVD_MAPS, _MPEG_V3C_AD, and _MPEG_V3C_CONFIG are formed in the attribute object in mesh.primitives, and access to the buffer storing the V-PCC data before reconstruction is specified.

[0160] Figure 13 The image shows an example of atlas information. For example... Figure 13 As shown, the atlas information includes patch information related to the patches, and patches such as geometry, attributes, and enhancement maps can be identified by referring to the patch information. Therefore, patches for each video can be associated during reconstruction.

[0161] <Grid>

[0162] Meanwhile, as 3D data representing the three-dimensional structure of a three-dimensional object (an object with a three-dimensional shape), there exists a mesh that represents the three-dimensional shape of the object's surface by forming polygons from vertices and connections (also called edges).

[0163] like Figure 14 As shown in the upper left portion, in a mesh, a polygonal plane (polygon) is formed by vertices 71 and connections 72 that link vertices 71. In the following description, it is assumed that the polygon has a triangular shape. The surface of an object with a three-dimensional structure, i.e., the three-dimensional shape of the object, is represented by a polygon (also called a face). Note that, as Figure 14 The texture 73 shown in the upper right part can be attached (also called applied) to each face of the mesh.

[0164] The data of the mesh includes, for example Figure 14 information shown in the lower part of FIG. 7. Figure 14 The vertex information 74 shown in the first from the left of the lower part of FIG. 7 is information indicating a three-dimensional position (three-dimensional coordinates (X, Y, Z)) of each vertex 71 constituting the mesh. Figure 14 The connection information 75 shown in the second from the left of the lower part of FIG. 7 is information indicating each connection (edge) 72 constituting the mesh. Figure 14 The texture image 76 shown in the third from the left of the lower part of FIG. 7 is mapping information of the texture 73 attached to each face. Figure 14 The UV map 77 shown in the fourth from the left of the lower part of FIG. 7 is information indicating a correspondence relationship between the vertex 71 and the texture 73. In the UV map 77, the coordinates (UV coordinates) of each vertex 71 in the texture image 76 are shown.

[0165] Note that the mesh data can vary in the time direction like a moving image of 2D data. That is, the mesh data can have a structure in which frames of mesh data representing three-dimensional structures at different timings are continuous.

[0166] <v-dmc>

[0167] As an encoding method of such a mesh, for example, there is a video-based dynamic mesh coding (V-DMC) disclosed in Non-Patent Literature 4 to Non-Patent Literature 7.

[0168] In V-DMC, a mesh to be encoded (referred to as an original mesh in this specification) is represented by a base mesh which is coarser (i.e., rough) than the original mesh in terms of mesh resolution, and displacement vectors of partition points obtained by subdividing the base mesh, and the base mesh and the displacement vectors are encoded. For example, a dynamic mesh stream generated by a camera-capturing is to be encoded, which is (unregistered) mesh data having a structure varying every frame.

[0169] For example, assume that there is an original mesh as shown in the uppermost part of Figure 15 In Figure 15 , black dots indicate vertices, and lines connecting the black dots indicate connections (edges). As described above, a mesh originally forms a surface (polygon) by vertices and edges in space, but here, for the sake of description, the mesh is described as a group of vertices linearly (in series) connected in a plane.

[0170] By decimating some vertices of the original mesh, a rough (low-resolution) mesh as shown in the second part from the top of Figure 15 is formed. This is referred to as a base mesh.

[0171] By subdividing each polygon of this base mesh, vertices and edges are added as shown in the third part from the top of Figure 15 . For example, by such subdivision, vertices can be added in the number obtained by decimating the original mesh. That is, by subdividing the base mesh, a mesh having the same number of vertices as the original mesh is obtained. In this specification, the vertices added by such subdivision are also referred to as partition points.

[0172] However, when the vertices of the original mesh are decimated, the connections have been updated, and the partition points are formed on these updated connections (edges). Therefore, even if the number of vertices is made the same as that of the original mesh by subdivision, the positions of the vertices of the subdivided base mesh (the third part from the top in Figure 15 are different from those of the vertices of the original mesh (the uppermost part in Figure 15 ). In other words, as shown in the lowermost part of Figure 15 , ideally, the original mesh can be restored by moving (displacing) the positions of the vertices of the subdivided base mesh to the vertex positions of the original mesh. In this specification, the amount of such displacement (movement) of the vertices indicated in vector form is referred to as a displacement vector.

[0173] That is, ideally, the original mesh can be expressed as a base mesh and a displacement vector. By expressing the original mesh as a base mesh and a displacement vector in this way, the number of polygons (i.e., the number of vertices and the number of edges) is reduced. Therefore, by encoding the base mesh and the displacement vector instead of encoding the original mesh, a decrease in encoding efficiency (an increase in code amount) can be suppressed.

[0174] That is, in V-DMC, as shown in Figure 14 , V-DMC data including a base mesh, a displacement vector, attributes (textures), and atlas information is generated from 3D data such as mesh data, and the V-DMC data is encoded. The atlas information is information required to reconstruct a mesh, and includes, for example, information indicating a correspondence relationship between a base mesh, a displacement vector, and a texture. For example, the atlas information can include information associating patches of a mesh with patches of attributes, such as UV mapping 77. Figure 14

[0175] As described above, since mesh data can vary in the time direction, mesh data has a structure in the time direction (a consecutive frame structure). Therefore, in V-DMC, mesh data is divided and processed in accordance with the structure in the time direction. A processing unit in the time direction is called a sample. That is, encoding and decoding are performed using one frame or a plurality of consecutive frames of mesh data as one sample. Generally, one frame of mesh data is one sample.

[0176] A sample can be intra-encoded independently of other samples, or can be inter-encoded using other samples (other frames) as key samples (key frames).

[0177] When V-DMC data is encoded, a base mesh, a displacement vector, attributes (textures), and atlas information that constitute the V-DMC data are encoded.

[0178] The atlas information is encoded in a predetermined method. The displacement vector is packed into a frame image (also called a displacement map or a geometry map), and is encoded as a moving image (also called a displacement video or a geometry video) using an encoding scheme for moving images. The attributes (textures) are packed into a frame image (also called a texture image or an attribute map), and are encoded as a moving image (also called a texture video or an attribute video) by using an encoding scheme for moving images.

[0179] ​On the other hand, in the case of intra-frame encoding, the base mesh is encoded independently of other samples in accordance with a predetermined encoding scheme (e.g., Draco or the like), and in the case of inter-frame encoding, a difference from the key sample is derived and encoded. Note that in the case where there is no difference between the sample to be processed and the key sample (in the case where the sample to be processed is the same as the key sample), encoding can be skipped. That is, the base mesh is encoded as 3D data.

[0180] One bitstream (also referred to as a V-DMC bitstream) is generated by using each data generated in this way as a substream. Therefore, the V-DMC bitstream includes a substream of atlas information (atlas data substream), a substream of a base mesh (base mesh substream), a substream of geometry video (geometry video substream), and a substream of attribute video (attribute video substream).

[0181] The substream of atlas information has information for associating patches of a mesh (sub-mesh), patches of geometry (displacement video), and patches of attributes with each other. The substream of a base mesh has a static mesh (intra-frame) and motion information (inter-frame). After decoding, the base mesh is planarly partitioned (subdivided) (the number of faces is increased) based on the subdivision information indicated in the substream of atlas information. The substream of geometry video is a substream of displacement video and has displacement vectors. It has displacement vectors arranged in order of iteration count of subdivision. Each vertex is associated in decoding order. The substream of attribute video has UV texture. The substream of geometry video and the substream of attribute video encoded by an encoding scheme for moving images are also collectively referred to as video components.

[0182] Figure 16 is a diagram showing a configuration example of a decoder of a V-DMC bitstream. The V-DMC bitstream is divided into respective substreams of a base mesh, displacement vectors, attributes (texture), and atlas information by demultiplexing, and each substream is decoded by a decoding method corresponding to each encoding method. That is, the base mesh is decoded by a decoding scheme for 3D data. The displacement vectors (geometry video) and the attributes (attribute video) are decoded by a decoding scheme for moving images (i.e., 2D data). The atlas information is decoded in a predetermined scheme.

[0183] Then, as described above, the base mesh is subdivided based on the atlas information, the displacement vectors are applied to the vertices (that is, the vertices are displaced), and the texture is applied to the faces, thereby reconstructing the mesh (generating a recovered mesh).

[0184] <sub-mesh>

[0185] Incidentally, the grid (the base grid in the case of V-DMC) may include one or more sub-grids. A sub-grid is a unit of encoding processing in the spatial direction (region direction). That is, sub-grids can be encoded independently of each other. For example, if the grid 81 shown in A of Figure 17 is divided into sub-grid 81A and sub-grid 81B as shown in B of Figure 17 , the sub-grid 81A and the sub-grid 81B can be encoded independently. Thus, the sub-grid 81A and the sub-grid 81B can be decoded independently of each other. That is, only the sub-grid 81A can be decoded, or only the sub-grid 81B can be decoded. That is, only the desired part of the grid can be decoded, and an increase in the amount of unnecessary decoding processing can be suppressed. Therefore, an increase in the load and processing time of the decoding processing can be suppressed.

[0186] In addition, for example, intra-frame encoding can be performed on the sub-grid 81A, and inter-frame encoding can be performed on the sub-grid 81B. For example, as shown in A of Figure 18 , three types, i.e., I, P, and Skip, are prepared as the types (encoding types) of the encoding method for the sub-grid. I indicates intra-frame encoding. That is, the sub-grid specified with type I is encoded independently of other sub-grids. P indicates inter-frame encoding. That is, the difference (motion information) from other reference samples is encoded in the sub-grid specified with type P. Skip indicates skipping encoding. That is, the encoding of the sample is skipped, and other reference samples are repeated. Note that the reference sample is a sub-grid having the same sub-grid ID as the current sample.

[0187] As shown in B of Figure 19 , such an encoding type (smh_type) is managed for each sample (POC) of each sub-grid. Thus, in each sample, an encoding scheme can be set for each sub-grid. Note that the sub-grid ID (submesh_id) and the encoding type (smh_type) are managed in the sub-grid header.

[0188] <Distribution of V-DMC Bitstream>

[0189] In recent years, a method of storing such a V-DMC stream and distributing a V-DMC bitstream has been sought to be established, such as the International Standardized Organization Basic Media File Format (ISOBMFF). The ISOBMFF is a file container specification of the international standard technique "MPEG-4 (Moving Picture Experts Group-4)" of mobile image compression. For example, it can be assumed that ISO / IEC 23090-10, which is a distribution technique standard as V-PCC described in Non-Patent Literature 8, is extended, and a V-DMC bitstream is stored in the ISOBMFF. In this case, the V-DMC bitstream is stored in a track of the ISOBMFF. At this time, a method of storing the V-DMC bitstream in one track (single track) can be considered, or a method of storing the corresponding sub-streams of the atlas information, the base mesh, the displacement vector, and the attribute in different tracks (multi-track) can be considered, as shown in Figure 20

[0190] According to the V-DMC encoding as described above, non-registered mesh data can be transmitted at a higher compression rate than before, and it is expected that it will also be supported in the MPEG-I scene description as one of the encoding techniques for 3D objects that constitute a scene. However, the conventional MPEG-I scene description does not correspond to the V-DMC bitstream, and it is difficult to apply the MPEG-I scene description to the distribution of the V-DMC bitstream. In other words, in the distribution of 3D data to which the MPEG-I scene description is applied, it is difficult to apply the V-DMC encoding scheme as the encoding scheme of the 3D data. Therefore, there is a possibility that the encoding efficiency is reduced.

[0191] <3. V-DMC data distribution using scene description>

[0192] <Method 1>

[0193] Therefore, as shown in the uppermost part of the table of Figure 20 , a description for acquiring V-DMC data of 3D objects that constitute a scene is stored in a scene description (Method 1). That is, the V-DMC data can be handled in the scene description. By doing so, it is possible to suppress the reduction of the encoding efficiency.

[0194] <Method 1-1>

[0195] In the case where the MPEG-I scene description is applied to the distribution of the V-DMC data by applying Method 1, as shown in Figure 21 ​As shown from the second part from the top of the table, 3D data can be reconstructed from V-DMC data in MAF (Method 1-1). The process of reconstructing 3D data from V-DMC data can include subdivision of the base mesh, displacement of vertices by displacement (refinement), and mapping of attributes. That is, MAF reconstruction can be applied, and these processes can be performed in MAF. In the case of MAF reconstruction, these processes can be performed using a CPU. Figure 11 An outline of the process in the case of MAF reconstruction is shown. In this case, a buffer stores geometry data (position) and attribute data (color) of reconstructed 3D data. That is, the reconstructed 3D data is provided to the PE. Therefore, the scene description is associated with Figure 22 is configured as an example.

[0196] In this case, the first information processing apparatus that generates the scene description sets, in the scene description, reference information to a buffer that stores geometry of 3D data reconstructed using V-DMC data and reference information to a buffer that stores attributes of 3D data reconstructed using V-DMC data. The second information processing apparatus that decodes a V-DMC bitstream acquires the V-DMC bitstream and decodes it in MAF based on the scene description, reconstructs 3D data from V-DMC data, and stores geometry and attributes of the reconstructed 3D data in the buffer. The PE of the second information processing apparatus acquires geometry and attributes of the reconstructed 3D data from the buffer based on the scene description and performs rendering or the like.

[0197] In this way, MAF reconstruction can be implemented. That is, the first information processing apparatus and the second information processing apparatus can implement distribution of V-DMC data using a scene description, and can suppress a decrease in encoding efficiency in distribution of 3D data using a scene description.

[0198] <Method 1-2>

[0199] In the case where MPEG-I scene description is applied to distribution of V-DMC data by applying Method 1, a PC can reconstruct 3D data from V-DMC data. That is, PE reconstruction can be applied. For example, as Figure 23 As illustrated, subdivision of the base mesh can be performed in the MAF, and displacement of vertices by displacement vectors and mapping of attributes can be performed in the PE. In this case, the V-DMC data before reconstruction is stored in a buffer. More specifically, the subdivided base mesh, the displacement video, the attribute video, and the atlas information are each supplied to the PE via the buffer. The PE performs displacement of vertices and mapping of attributes by using the V-DMC data to reconstruct the 3D data. Furthermore, the PE can perform rendering of the 3D data and the like.

[0200] In the case of PE reconstruction, since the data is stored in the buffer, the execution timing of the processing of the PE can be controlled, and the processing of the PE can be processed by an arithmetic unit different from the processing of the MAF. Therefore, compared to the case of MAF reconstruction, it is possible to more efficiently perform the decoding processing. Furthermore, in the case of PE reconstruction, the processing of the PE can be performed using, for example, a GPU. A GPU can more efficiently perform parallel processing and pipelining processing than a CPU. Therefore, compared to the case of MAF reconstruction, it is possible to more efficiently perform the decoding processing by applying PE reconstruction.

[0201] Incidentally, the displacement vectors of the mesh are divided into a plurality of partial regions, and each of the partial regions is projected onto a two-dimensional plane. The "set of displacement vectors projected onto a two-dimensional plane" corresponding to the partial region is also referred to as a patch of displacement vectors. The displacement map (displacement video) is packed with the patches of displacement vectors. Similarly, the attributes of the mesh are divided into a plurality of partial regions, and each of the partial regions is projected onto a two-dimensional plane. The "set of attributes projected onto a two-dimensional plane" corresponding to the partial region is also referred to as a patch of attributes. The attribute map (attribute video) is packed with the patches of attributes.

[0202] Similar to the case of V-PCC, the atlas information includes patch information on the patches packed in the displacement video or the attribute video. For example, as illustrated in FIG. 6, the atlas information includes the number of patches, the size of each patch, and the position of each patch in the two-dimensional plane. Figure 24 As shown, the patch information can include identification information (submesh_id) of a submesh of the base mesh, information (gemety_2d_pos_x, gemety_2d_pos_y, gemety_2d_size_x, gemety_2d_size_y) indicating the position and size of the patch of the geometry (displacement vector), and information (attribute_2d_pos_x, attribute__2d_pos__y, attribute_2d_size_x, attribute_2d_size_y) indicating the position and size of the patch of the attribute. That is, the atlas information includes a correspondence relationship between each patch of the displacement vector and the attribute and the identification information of the submesh of the base mesh.

[0203] Figure 24 is a diagram showing a main configuration example of a scene description in the case of PE reconstruction. As shown in Figure 25 the MPEG_primitive_V3C object that processes V-DMC data is set to mesh.primitives, characteristics such as _MPEG_V3C_MESH, _MPEG_V3C_AVD, _MPEG_V3C_GVD, and _MPEG_V3C_AD are formed in the MPEG_primitive_V3C object, and an accessor to a buffer that stores the V-DMC data before reconstruction is designated. For example, in the _MPEG_V3C_MESH, an access to a buffer that stores the geometry (position) of each vertex of the base mesh and an access to a buffer that stores the normal vector (NORMAL) of each vertex are designated. In the _MPEG_V3C_AVD, an access to a buffer that stores the attribute video is designated. In the _MPEG_V3C_GVD, an access to a buffer that stores the displacement video is designated. In the _MPEG_V3C_AD, an access to a buffer that stores the atlas information is designated. Figure 26 a description example of a scene description is shown.

[0204] The regular decoder of the V-DMC data manages the submeshes of the base mesh as 3D data. That is, the decoder manages for each submesh of the base mesh which part of the data the identification information corresponds to. Therefore, the decoder can grasp the correspondence relationship between the data of the submesh of the base mesh and the patch of the displacement vector or the attribute based on the above-described atlas information. Therefore, the 3D data can be reconstructed from the V-DMC data (performing the displacement (refinement) of the vertex and the mapping of the attribute using the displacement vector).

[0205] However, when the PE reconstruction is applied as described above and the V-DMC data before the reconstruction is stored in the buffer (provided to the PE via the buffer), the PE cannot acquire information for managing the identification information of each sub-mesh even if the PE can acquire the data of the subdivided base mesh. That is, the PE cannot grasp the sub-meshes of the base mesh. Therefore, as shown in Figure 23 , even if the PE can grasp the correspondence between the "identification information of the sub-mesh of the base mesh" and "each patch of the displacement vector and the attribute" based on the patch information of the atlas information Figure 20 , the PE cannot grasp "to which part of the data of the base mesh does the identification information of the sub-mesh correspond". That is, the PE cannot grasp the correspondence between the "sub-mesh of the base mesh" and "patch of the displacement vector or the attribute". Therefore, in the PE, it is difficult to reconstruct the 3D data from the V-DMC data (perform the displacement (refinement) of the vertex and the mapping of the attribute using the displacement vector). Therefore, it is possible to reduce the processing efficiency of the decoding process.

[0206] Therefore, as shown in the third part from the top of the table in Figure 20 , the sub-mesh association information can be provided from the MAF to the PE, and the 3D data can be reconstructed from the V-DMC data using the sub-mesh association information in the PE (Method 1-2). The sub-mesh association information is information for specifying the data of the sub-mesh that is a part of the mesh representing the 3D object.

[0207] For example, the first information processing apparatus can include a scene description generation unit that generates a scene description representing a scene including a 3D object, and sets first reference information to a first buffer storing sub-mesh association information in the scene description.

[0208] Further, in the first information processing apparatus, a scene description representing a scene including a 3D object can be generated, and first reference information to a first buffer storing sub-mesh association information can be set in the scene description.

[0209] By doing so, the information processing apparatus that decodes the V-DMC bitstream using the scene description can achieve the PE reconstruction, and can perform the decoding process more efficiently compared to the case of the MAF reconstruction. That is, the first information processing apparatus can suppress the reduction of the processing efficiency of the decoding process.

[0210] Further, the second information processing apparatus can include a storage processing unit that stores, in a first buffer specified by a scene description representing a scene composed of 3D objects, submesh association information for specifying data of a submesh that is a part of a mesh representing the 3D objects, and a reconstruction unit that acquires the submesh association information from the first buffer, specifies data of the submesh using the acquired submesh association information, and reconstructs the mesh.

[0211] Further, in the second information processing apparatus, the submesh association information for specifying data of a submesh that is a part of a mesh representing 3D objects can be stored in a first buffer specified by a scene description representing a scene composed of the 3D objects, and the submesh association information can be acquired from the first buffer, data of the submesh can be specified using the acquired submesh association information, and the mesh can be reconstructed.

[0212] By so doing, the second information processing apparatus can grasp the correspondence between "a submesh of a base mesh" and "a patch of a displacement vector or an attribute" in the PE. That is, the PE reconstruction can be achieved, and the processing efficiency of the decoding process can be improved as compared with the case of the MAF reconstruction. That is, the first information processing apparatus and the second information processing apparatus can suppress the reduction in the processing efficiency of the decoding process.

[0213] <Method 1-2-1>

[0214] In the case of applying the method 1-2, for example, as shown in the fourth part from the top of the table, the reference information to the buffer storing the submesh association information can be newly defined (method 1-2-1). Figure 27

[0215] For example, the first buffer referred to in the first reference information set by the first information processing apparatus in the scene description can be a buffer different from the second buffer storing data of the mesh. That is, in the first information processing apparatus, the scene description generation unit can set "a buffer different from the second buffer storing data of the mesh" as "the first buffer storing the submesh association information", and set the first reference information to the first buffer in the scene description.

[0216] Further, in the second information processing apparatus, "the first buffer specified by the scene description" can be a buffer different from the second buffer storing data of the mesh. That is, the storage processing unit can store the submesh association information in the first buffer that is "a buffer different from the second buffer storing data of the mesh". Then, the reconstruction unit can acquire the submesh association information from the first buffer.

[0217] That is, as shown in the fourth part from the top of the table, the reference information to the buffer storing the submesh association information can be newly defined (method 1-2-1). Figure 28 ​As shown, a buffer (a buffer indicated in gray in the figure) is newly defined separately from the buffer storing the V-DMC data, and the sub-mesh association information for specifying information of a sub-mesh that is a part of a mesh representing a 3D object is provided from the MAF to the PE via the new buffer. Figure 28 is a diagram showing a configuration example of a scene description in this case. As Figure 29 shown, the _MPEG_V3C_SUBMESH_INFO property is set in the MPEG_primitive_V3C object of mesh.primitives, and an accessor to a buffer (buffer) storing the sub-mesh association information is specified. The first information processing apparatus generates a scene description having such a configuration. The MAF of the second information processing apparatus stores the sub-mesh association information in the buffer specified by the _MPEG_V3C_SUBMESH_INFO property based on the scene description. Then, the PE acquires the sub-mesh association information from the buffer specified by the _MPEG_V3C_SUBMESH_INFO property, and reconstructs the 3D data using the sub-mesh association information.

[0218] By doing so, the PE can grasp the correspondence between each sub-mesh included in the base mesh data and the identification information of the sub-mesh by the sub-mesh association information. Therefore, the PE can grasp the correspondence between the sub-meshes of the base mesh and the patches based on the sub-mesh association information and the patch information of the atlas information. That is, the PE can reconstruct the 3D data from the V-DMC data by using the identification information of the sub-mesh. That is, the PE reconstruction can be achieved, and the processing efficiency of the decoding process can be improved compared to the case of the MAF reconstruction. That is, the first information processing apparatus and the second information processing apparatus can suppress the decrease in the processing efficiency of the decoding process.

[0219] Note that, in the _MPEG_V3C_SUBMESH_INFO property, for each property of the base mesh, a timed accessor index of a buffer storing the sub-mesh association information can be specified. For example, as Figure 30 shown, properties such as "submesh_info_position", "submesh_info_normal", "submesh_info_tangent", "submesh_info_texcoord_n", "submesh_info_color_n", "submesh_info_joints_n", and "submesh_info_weights_n" can be set in the _MPEG_V3C_SUBMESH_INFO property, and as Figure 31 As illustrated, the index of the timing accessor can be specified for each property.

[0220] Further, in this case, the submesh association information can include boundary information indicating a boundary of a submesh in the data of the base mesh whose vertex of the mesh has been thinned. By indicating the boundary of the submesh, the PE is able to specify the data (of the location) of the submesh cell in the data of the base mesh.

[0221] For example, the boundary information can include information indicating the length of the data of the submesh and identification information of the submesh. For example, as illustrated in A of Figure 31 As illustrated in A of Figure 20 , the data of the base mesh is divided by each "length". The data in each section is the actual data of the submesh, and the "submesh_id" is assigned to each data. Since such boundary information is indicated in the submesh association information, the PE can easily grasp which part of the base mesh the data of the submesh corresponding to the submesh_id included in the patch information is.

[0222] <Method 1-2-2>

[0223] In the case of applying the method 1-2, for example, as illustrated in the fifth part from the top of the table of Figure 32 , the atlas data can be extended to store the submesh association information (method 1-2-2).

[0224] For example, the first buffer referred to in the first reference information set in the scene description by the first information processing apparatus can be configured to store the atlas information including the submesh association information. That is, in the first information processing apparatus, the scene description generation unit can set the buffer storing the atlas information as "the first buffer storing the submesh association information", and set the first reference information to the first buffer in the scene description.

[0225] Further, in the second information processing apparatus, "the first buffer specified by the scene description" can be configured to store the atlas information including the submesh association information. That is, the storage processing unit can store the atlas information including the submesh association information in the first buffer. Then, the re-creation unit can acquire the atlas information including the submesh association information from the first buffer.

[0226] That is, in this case, the scene description has, for example, as illustrated in Figure 33 The configuration shown, and the submesh association information is stored in the buffer as atlas information via an accessor specified by the MPEG_V3C_AD feature.

[0227] In this way, the PE can acquire the submesh association information as well as the atlas information. Therefore, the PE can grasp the correspondence between the submeshes and the patches of the base mesh on the basis of the atlas information including the submesh association information. That is, the PE can reconstruct the 3D data from the V-DMC data by using the identification information of the submeshes. That is, it is possible to realize the PE reconstruction, and it is possible to improve the processing efficiency of the decoding process as compared with the case of the MAF reconstruction. That is, the first information processing apparatus and the second information processing apparatus can suppress the reduction in the processing efficiency of the decoding process.

[0228] Note that, in this case, the atlas information can include the submesh association information associated with the patches. In the atlas information, since the submesh association information is indicated in association with each patch, the PE can more easily grasp the correspondence between the submeshes and the patches of the base mesh. In addition, since the submesh association information is indicated in association with the patches, the identification information of the submeshes becomes unnecessary. Therefore, it is possible to correspondingly suppress the increase in the amount of data.

[0229] Further, in this case, the submesh association information can include boundary information indicating a boundary of the submesh in the data of the base mesh in which the vertices of the mesh have been sparsified. By indicating the boundary of the submesh, the PE can specify the data (of the position) of the submesh unit in the data of the base mesh.

[0230] Further, the boundary information can include information indicating an offset and a length of the data of the submesh. For example, the atlas information can have patch information as shown in Figure 33 In the case of the example shown in Figure 20 The parameter "submesh_byte offset" and the parameter "submesh_byte length" are set for each patch. The "submesh_byte offset" is information indicating an offset of the boundary of the data of the submesh. The "submesh_byte length" is information indicating a length of the data of the submesh. In this way, by indicating the offset and the length of the data of each submesh in the data of the base mesh, the PE can specify the data (of the position) of the submesh unit in the data of the base mesh.

[0231] Note that, in a case where the above-described method 1-2-1 or method 1-2-2 is applied, the data format of MPEG_primitive_V3C / _MPEG_V3C_MESH is the same as that of the attribute / POSITION of a mesh / primitive provided by a conventional glTF. Thus, _MPEG_V3C_MESH is not defined by MPEG_primitive_V3C, but the attribute / POSITION of the mesh / primitive can be used at the time of PE reconstruction.

[0232] <Method 1-2-3>

[0233] In a case where the method 1-2 is applied, for example, as illustrated in a sixth part from the top of the table in FIG. 15, the base mesh can be extended to store the sub-mesh association information (method 1-2-3). Figure 34

[0234] For example, the first buffer referred to in the first reference information set in the scene description by the first information processing apparatus can be configured to store the base mesh including the sub-mesh association information. That is, in the first information processing apparatus, the scene description generation unit can set the buffer storing the base mesh as "the first buffer storing the sub-mesh association information", and set the first reference information to the first buffer in the scene description.

[0235] Further, in the second information processing apparatus, "the first buffer specified by the scene description" can be configured to store the base mesh including the sub-mesh association information. That is, the storage processing unit can store the base mesh including the sub-mesh association information in the first buffer. Then, the reconstruction unit can acquire the base mesh including the sub-mesh association information from the first buffer.

[0236] That is, in this case, the scene description has a configuration as illustrated in FIG. 16, and the sub-mesh association information is stored in the buffer as the atlas information via the accessor specified by the _MPEG_V3C_MESH attribute. Figure 34

[0237] In this way, the PE can acquire the sub-mesh association information as well as the base mesh. Thus, the PE can grasp the correspondence between the sub-meshes and the patches of the base mesh on the basis of the sub-mesh association information and the atlas information. That is, the PE can reconstruct the 3D data from the V-DMC data by using the identification information of the sub-mesh. That is, it is possible to achieve the PE reconstruction, and it is possible to improve the processing efficiency of the decoding process as compared with the case of the MAF reconstruction. That is, the first information processing apparatus and the second information processing apparatus can suppress the decrease in the processing efficiency of the decoding process.

[0238] ​​Note that in this case, the sub-mesh association information may include the sub-mesh identification information. Because the sub-mesh association information associated with the base mesh has identification information, the PE can associate it with the patch information of the atlas information using the identification information, and can more easily grasp the correspondence between the sub-mesh and patches of the base mesh.

[0239] For example, the type of a feature of the primitives / attributes included in the base mesh can be extended, an element of an accessor type can be added, and identification information of the submesh can be stored in it.

[0240] For example, the identification information of a submesh can be stored in the properties of the vertices of the base mesh. Figure 35 and Figure 36 As shown, the V3C_MESH_POSITION attribute set to the _MPEG_V3C_MESH attribute can specify a timed accessor to the buffer that stores the identification information of the sub-mesh associated with each vertex of the base mesh. By doing so, the identification information of the sub-mesh is associated with each vertex of the base mesh. That is, the sub-mesh association information can include the identification information of the sub-mesh associated with each vertex of the base mesh. In other words, the correspondence between each vertex of the base mesh and the sub-mesh is clearly indicated. Therefore, the PE can more easily grasp the correspondence between each vertex of the base mesh and the patch. For other attributes, the PE can identify the sub-mesh according to the data storage order based on the sub-mesh identification information of this V3C_MESH_POSITION.

[0241] Note that other properties, such as NORMAL, can be extended similarly. All properties of the basic mesh primitives / attributes can be extended.

[0242] <Method 1-2-4>

[0243] Although the above description describes subdivision of the base mesh in MAF during PE reconstruction, this process can be performed as follows: Figure 20 This is performed in the PE. Doing so further improves the processing efficiency of the decoding process. In this case, the base grid before subdivision is stored in a buffer (provided to the PE from the MAF). In a regular general-purpose decoder for V-DMC data, the subdivision method is managed, but in this method, the PE cannot obtain information and has difficulty understanding how to subdivide the base grid.

[0244] Therefore, when applying methods 1-2, for example, as Figure 37 As shown in the bottom row of the table, the subdivision information can be provided from the MAF to the PE, and the base mesh can be subdivided in the PE using the subdivision information (Method 1-2-4). The subdivision information is information related to the subdivision of the base mesh. The subdivision information specifies the subdivision method.

[0245] For example, in the first information processing apparatus, the scene description generation unit can further set, in the scene description, second reference information to a second buffer that stores subdivision information related to the subdivision of the base mesh whose vertices have been thinned.

[0246] Further, in the second information processing apparatus, the storage processing unit can further store, in the second buffer specified by the scene description, subdivision information related to the subdivision of the base mesh whose vertices have been thinned, and the reconstruction unit can acquire the subdivision information from the second buffer, subdivide the base mesh using the acquired subdivision information, and reconstruct the mesh using the subdivided base mesh.

[0247] In this way, the PE can acquire the subdivision information. Therefore, the PE can subdivide the base mesh based on the subdivision information. Thus, the processing efficiency of the decoding process can be further improved. That is, the first information processing apparatus and the second information processing apparatus can suppress a decrease in the processing efficiency of the decoding process.

[0248] Here, the second buffer that stores the subdivision information can be the same as or different from the first buffer that stores the submesh association information described above.

[0249] For example, the second buffer can be configured to store atlas information including the subdivision information. That is, the atlas data can be extended to store the subdivision information. In the first information processing apparatus, the scene description generation unit can set the buffer that stores the atlas information as the "second buffer that stores the subdivision information", and set, in the scene description, the second reference information to the second buffer. Further, in the second information processing apparatus, the "second buffer specified by the scene description" can be configured to store the atlas information including the subdivision information. That is, the storage processing unit can store the atlas information including the subdivision information in the second buffer. Then, the reconstruction unit can acquire the atlas information including the subdivision information from the second buffer.

[0250] In this way, the PE can acquire the subdivision information as well as the atlas information. Therefore, the PE can subdivide the base mesh based on the subdivision information. Thus, the second information processing apparatus can more efficiently perform the decoding process. That is, the first information processing apparatus and the second information processing apparatus can suppress a decrease in the processing efficiency of the decoding process.

[0251] Note that in this case, the atlas information can include subdivision information associated with the patch. The subdivision information is indicated in association with each patch in the atlas information, so that the PE can grasp the subdivision method of each sub-mesh of the base mesh.

[0252] The subdivision information can include any information as long as information that specifies the subdivision method is included. For example, the subdivision information can include information indicating whether a predetermined method is applied in the subdivision of the base mesh. The predetermined method can be any method. For example, information indicating whether a midpoint is applied can be included in the subdivision information. In addition, the subdivision information can include information indicating the number of iterations of the subdivision of the base mesh.

[0253] Figure 37 is a diagram showing an example of the atlas information. In the example of Figure 38 , patch information is extended, and fields of "subdivision_method" and "subdivision_iteration_count" are added. "subdivision_method" is a parameter indicating the subdivision method, and has a value of "0" or "1", as shown in Figure 20 . The value "0" indicates that the subdivision method is not specified (NONE), and the value "1" indicates that a midpoint (MIDPOINT) that repeatedly divides an edge between vertices into two equal parts is specified as the subdivision method. That is, "subdivision_method" indicates whether a midpoint is applied. "subdivision_iteration_count" is a parameter indicating the number of iterations (iteration count) of edge division by the midpoint.

[0254] Based on such subdivision information in the patch information, in the case where the value of "subdivision_method" is "1", the PE performs subdivision by applying a midpoint to the sub-mesh corresponding to the patch, and repeats edge division the number of times indicated by "subdivision_iteration_count". In this way, the PE can perform subdivision based on the subdivision information stored in the patch information of the atlas information.

[0255] <Combination>

[0256] Each of the above-described methods can be applied in combination with any other method, as long as there is no contradiction. Three or more methods can be applied in combination. For example, any two or more of the methods 1-2-1 to 1-2-4 can be applied in combination. Furthermore, the methods that can be combined can include not only those shown as "Method" in the table of Figure 39 , but also all of the elements described above. Furthermore, each of the above-described methods can be applied in combination with other methods not described above.

[0257] Note that, in this specification, as long as there is no contradiction, the description of the upper method is also applied to the lower method belonging to the method. For example, in the case of describing "a method 1 can be applied", a method 1-1 or a method 1-2 can be applied. In addition, any one or more of methods 1-2-1 to 1-2-4 can also be applied.

[0258] <4. First Embodiment>

[0259] <file generation apparatus>

[0260] The technology described above can be applied to any apparatus. Figure 39 is a block diagram illustrating an example of a configuration of a file generation apparatus that is one aspect of an information processing apparatus to which the present technology is applied. Figure 39 The file generation apparatus 300 illustrated is an apparatus that converts mesh data into V-DMC data, encodes the V-DMC data, and stores the V-DMC data in, for example, a file container such as ISOBMFF, thereby generating a content file for distribution. In addition, the file generation apparatus 300 generates a scene description for distribution.

[0261] Note that, Figure 39 The main parts of processing units, data flows, and the like are illustrated, and Figure 39 Those illustrated in Figure 39 The processing units illustrated as blocks in Figure 39 The processing or data flows illustrated as arrows and the like in

[0262] As illustrated in Figure 40 , the file generation apparatus 300 includes a control unit 311, a file generation processing unit 312, and an input unit 313. The control unit 311 performs processing related to control of the file generation processing unit 312. The file generation processing unit 312 performs processing related to generation of a content file and the like. The input unit 313 has an input device, receives information of an external input from, for example, a user or another apparatus, and provides the input information to the control unit 311. The input device can include, for example, a keyboard, a mouse, a touch panel, a physical switch, a physical button, an input terminal, and the like. The control unit 311 can perform processing based on the information provided from the input unit 313.

[0263] The file generation processing unit 312 includes a V-DMC data generation unit 331, a V-DMC data encoding unit 332, a file generation unit 333, a scene description generation unit 334, a storage unit 335, and a provision unit 336.

[0264] The V-DMC data generation unit 331 performs processing related to generation of V-DMC data. For example, the V-DMC data generation unit 331 can acquire mesh data indicating a 3D object provided from outside of the file generation apparatus 300. The V-DMC data generation unit 331 can convert the mesh data into V-DMC data. For example, the V-DMC data generation unit 331 can generate a base mesh by decimating the mesh. In addition, the V-DMC data generation unit 331 can generate a displacement vector corresponding to each vertex by using the base mesh and the original mesh. In addition, atlas information indicating a correspondence relationship among the base mesh, the displacement vector, and a texture can be generated. The V-DMC data generation unit 331 can provide the generated V-DMC data to the V-DMC data encoding unit 332.

[0265] The V-DMC data encoding unit 332 performs processing related to encoding of V-DMC data. For example, the V-DMC data encoding unit 332 can acquire V-DMC data provided from the V-DMC data generation unit 331. The V-DMC data encoding unit 332 can encode the V-DMC data to generate a V-DMC bitstream. The V-DMC data encoding unit 332 can provide the generated V-DMC bitstream to the file generation unit 333.

[0266] The file generation unit 333 performs processing related to generation of a content file. For example, the file generation unit 333 can acquire the V-DMC bitstream provided from the V-DMC data encoding unit 332. The file generation unit 333 can generate a content file storing the V-DMC bitstream. The content file can have any specification, and for example, can be an International Organization for Standardization Base Media File Format (ISOBMFF). The file generation unit 333 can provide the generated content file to the scene description generation unit 334. The file generation unit 333 can provide the generated content file to the storage unit 335.

[0267] The scene description generation unit 334 performs processing related to generation of a scene description. For example, the scene description generation unit 334 can acquire mesh data provided from outside of the file generation apparatus 300. The scene description generation unit 334 can acquire the content file provided from the file generation unit 333. The scene description generation unit 334 generates a scene description indicating a scene composed of 3D objects corresponding to the mesh data based on these information. This scene description corresponds to the V-DMC bitstream (i.e., V-DMC data). The scene description generation unit 334 generates a scene description generation unit file storing the scene description. The scene description generation unit 334 can provide the generated scene description file to the storage unit 335.

[0268] The storage unit 335 has a storage area, and performs processing relating to storage of information. For example, the storage unit 335 can acquire the content file provided from the file generation unit 333. The storage unit 335 can store the content file in the storage area. In addition, the storage unit 335 can acquire the scene description file provided from the scene description generation unit 334. The storage unit 335 can store the scene description file in the storage area.

[0269] The provision unit 336 performs processing relating to provision of information. For example, the provision unit 336 can acquire the scene description file or the content file stored in the storage unit 335. The provision unit 336 can provide the acquired scene description file or content file to the outside (e.g., a distribution server, a reproduction apparatus, or the like) of the file generation apparatus 300.

[0270] <V-DMC data encoding unit>

[0271] Figure 40 is a block diagram illustrating a main configuration example of the V-DMC data encoding unit 332. Note that, Figure 40 main parts of processing units, data flows, and the like are illustrated, and Figure 40 those illustrated in Figure 40 may not be all. That is, in the V-DMC data encoding unit 332, there can be processing units not illustrated as blocks in Figure 40 , or there can be processing or data flows not illustrated as arrows and the like in

[0272] As illustrated in Figure 21 , the V-DMC data encoding unit 332 includes an atlas information encoding unit 351, a base mesh encoding unit 352, a displacement vector correction unit 353, a displacement video encoding unit 354, a mesh reconstruction unit 355, an attribute map conversion unit 356, an attribute video encoding unit 357, and a multiplexing unit 358.

[0273] The atlas information encoding unit 351 performs processing relating to encoding of atlas information. For example, the atlas information encoding unit 351 can acquire the atlas information provided from the V-DMC data generation unit 331. Further, the atlas information encoding unit 351 can encode the acquired atlas information by a predetermined encoding scheme to generate encoded data of the atlas information. The atlas information encoding unit 351 can provide the generated encoded data of the atlas information to the multiplexing unit 358.

[0274] The base mesh encoding unit 352 performs processing relating to encoding of the base mesh. For example, the base mesh encoding unit 352 can acquire the base mesh provided from the V-DMC data generation unit 331. Further, the base mesh encoding unit 352 can quantize the acquired base mesh, encode the quantized base mesh by a predetermined encoding scheme (for example, Draco or the like), and generate encoded data of the base mesh. At this time, the base mesh encoding unit 352 can encode the base mesh in accordance with the atlas information. The base mesh encoding unit 352 can provide the generated encoded data of the base mesh to the displacement vector correction unit 353. Further, the base mesh encoding unit 352 can provide the generated encoded data of the base mesh to the multiplexing unit 358.

[0275] The displacement vector correction unit 353 performs processing relating to correction of the displacement vector. For example, the displacement vector correction unit 353 can acquire the base mesh and the displacement vector provided from the V-DMC data generation unit 331. Further, the displacement vector correction unit 353 can acquire the encoded data of the base mesh provided from the base mesh encoding unit 352. The displacement vector correction unit 353 can correct the displacement vector on the basis of these information. For example, the displacement vector correction unit 353 can decode and inverse-quantize the acquired encoded data of the base mesh, and can restore the base mesh (may generate the restored base mesh). The displacement vector correction unit 353 can subdivide each of the pre-encoding base mesh and the restored base mesh. The displacement vector correction unit 353 can compare the subdivided pre-encoding base mesh with the subdivided restored base mesh to obtain an encoding distortion. The displacement vector correction unit 353 can correct the displacement vector in accordance with the encoding distortion. The displacement vector correction unit 353 can provide the corrected displacement vector to the displacement video encoding unit 354. Further, the displacement vector correction unit 353 can provide the subdivided restored base mesh to the mesh reconstruction unit 355.

[0276] The displacement video encoding unit 354 performs processing relating to encoding of a displacement video. The displacement video is a moving image in which a displacement map is a frame image, and the displacement map is a two-dimensional region in which displacement vectors are packed. For example, the displacement video encoding unit 354 can acquire the displacement vectors provided from the displacement vector correction unit 353. The displacement video encoding unit 354 can generate a displacement map by performing wavelet transform on the displacement vectors, quantizing the displacement vectors, and packing the displacement vectors in a two-dimensional region. The displacement video encoding unit 354 can generate a displacement video using the displacement map as a frame image. The displacement video encoding unit 354 can convert the generated displacement video into a geometry video. The displacement video encoding unit 354 can encode the generated geometry video using a predetermined encoding scheme for a 2D moving image to generate encoded data of the geometry video. The displacement video encoding unit 354 can provide the generated encoded data of the geometry video to the multiplexing unit 358. Further, the displacement video encoding unit 354 can decode the generated encoded data of the geometry video to restore the geometry video. The displacement video encoding unit 354 can inverse-transform the geometry video into a displacement video (a reconstructed displacement video can be generated). The displacement video encoding unit 354 can extract a displacement map from the restored displacement video. The displacement video encoding unit 354 can unpack displacement vectors from the displacement map. The displacement video encoding unit 354 can inverse-quantize the displacement vectors. The displacement video encoding unit 354 can provide the inverse-quantized displacement vectors to the mesh reconstruction unit 355.

[0277] The mesh reconstruction unit 355 performs processing relating to mesh reconstruction. For example, the mesh reconstruction unit 355 can acquire the base mesh (the subdivided restored base mesh) provided from the displacement vector correction unit 353. Further, the mesh reconstruction unit 355 can acquire the displacement vectors provided from the displacement video encoding unit 354. The mesh reconstruction unit 355 can reconstruct a restored mesh using the base mesh and the displacement vectors. The mesh reconstruction unit 355 can provide the restored mesh to the attribute map conversion unit 356.

[0278] The attribute map conversion unit 356 performs processing related to conversion of an attribute map. For example, the attribute map conversion unit 356 can acquire the restored mesh provided from the mesh reconstruction unit 355. Further, the attribute map conversion unit 356 can acquire the atlas information provided from the V-DMC data generation unit 331. The attribute map conversion unit 356 can acquire the original mesh input to the file generation apparatus 300. The attribute map conversion unit 356 can acquire the attribute map provided from the V-DMC data generation unit 331. The attribute map conversion unit 356 can convert the acquired attribute map on the basis of the other information acquired. For example, the attribute map conversion unit 356 can convert the attribute map on the basis of the atlas information, the original mesh, or the like to correspond to the restored mesh. In other words, it can also be said that the attribute map conversion unit 356 generates a converted attribute map. Therefore, the attribute map conversion unit 356 can also be referred to as an attribute map generation unit. The attribute map conversion unit 356 can provide the converted attribute map to the attribute video encoding unit 357.

[0279] The attribute video encoding unit 357 performs processing related to encoding of an attribute video. For example, the attribute video encoding unit 357 can acquire the attribute map provided from the attribute map conversion unit 356. Further, the attribute video encoding unit 357 can generate an attribute video using the acquired attribute map as a frame image. Further, the attribute video encoding unit 357 can encode the generated attribute video in accordance with a predetermined encoding scheme for a 2D moving image to generate encoded data of the attribute video. The attribute video encoding unit 357 can provide the generated encoded data of the attribute video to the multiplexing unit 358.

[0280] The multiplexing unit 358 performs processing related to multiplexing of encoded data (substream). For example, the multiplexing unit 358 can acquire the encoded data of the atlas information provided from the atlas information encoding unit 351. Further, the multiplexing unit 358 can acquire the encoded data of the base mesh provided from the base mesh encoding unit 352. Further, the multiplexing unit 358 can acquire the encoded data of the geometry video provided from the displacement video encoding unit 354. Further, the multiplexing unit 358 can acquire the encoded data of the attribute video provided from the attribute video encoding unit 357. The multiplexing unit 358 can multiplex those encoded data as substreams to generate a V-DMC bitstream. Therefore, the multiplexing unit 358 can also be referred to as a bitstream generation unit. The multiplexing unit 358 can provide the generated V-DMC bitstream to the file generation unit 333.

[0281] The file generation apparatus 300 having the above-described configuration can be a first information processing apparatus, and can apply the various methods (the present technology) described above in <3. V-DMC data distribution using scenario description>.

[0282] <Flow 1 of file generation processing>

[0283] For example, the above-described method 1-1 can be applied to generate a scene description, in which 3D data is reconstructed from V-DMC data in the MAF. That is, as in the Figure 41 example of, the file generation device 300 can set, in the scene description, reference information for a buffer storing the geometry of the 3D data reconstructed using V-DMC data and reference information for a buffer storing the attributes of the 3D data reconstructed using V-DMC data.

[0284] Reference will be made to Figure 42 the flowchart of to describe an example of the flow of the file generation process executed by the file generation device 300 in this case.

[0285] In this case, when the file generation process starts, in step S301, the V-DMC data generation unit 331 of the file generation device 300 extracts a grid and generates V-DMC data.

[0286] In step S302, the V-DMC data encoding unit 332 performs V-DMC encoding processing to encode the V-DMC data and generate a V-DMC bitstream.

[0287] In step S303, the file generation unit 333 generates a content file and stores the V-DMC bitstream.

[0288] In step S304, the scene description generation unit 334 generates a scene description corresponding to the V-DMC bitstream (content file) and generates a scene description file storing the scene description.

[0289] In step S305, the scene description generation unit 334 sets, in the scene description, reference information for a buffer storing the geometry of the 3D data reconstructed using V-DMC data and reference information for a buffer storing the attributes of the 3D data reconstructed using V-DMC data.

[0290] In step S306, the storage unit 335 stores the generated scene description file and content file.

[0291] In step S307, the providing unit 336 provides the scene description file and content file to the outside of the file generation device 300 (e.g., a distribution server, a reproduction device, etc.).

[0292] When the processing in step S307 ends, the file generation process ends.

[0293] <Flowchart of V-DMC Encoding Processing>

[0294] Next, reference will be made to Figure 41 the flowchart of to describe in Figure 41 An example of the flow of the V-DMC encoding process performed in step S302 of the V-DMC encoding process of FIG. 31.

[0295] When the V-DMC encoding process starts, in step S321, the atlas information encoding unit 351 encodes the atlas information and generates encoded data of the atlas information.

[0296] In step S322, the base mesh encoding unit 352 encodes the base mesh and generates encoded data of the base mesh.

[0297] In step S323, the displacement vector correction unit 353 corrects the displacement vector.

[0298] In step S324, the displacement video encoding unit 354 converts a displacement video having a displacement map packed with the corrected displacement vector as a frame image into a geometry video, encodes the geometry video with an encoding scheme for moving images, and generates encoded data of the geometry video.

[0299] In step S325, the mesh reconstruction unit 355 reconstructs the mesh (generates a recovered mesh).

[0300] In step S326, the attribute map conversion unit 356 converts the attribute map.

[0301] In step S327, the attribute video encoding unit 357 encodes an attribute video having the attribute map as a frame image and generates encoded data of the attribute video.

[0302] In step S328, the multiplexing unit 358 multiplexes each of the encoded data of the atlas information, the encoded data of the bitstream, the encoded data of the geometry video, and the encoded data of the attribute video as a substream to generate a V-DMC bitstream.

[0303] When the process in step S328 ends, the V-DMC encoding process ends, and the process returns to Figure 27 .

[0304] By performing each of the processes as described above, the file generation apparatus 300 can realize MAF reconstruction. That is, the file generation apparatus 300 can realize distribution of V-DMC data using scene descriptions, and can suppress reduction in encoding efficiency in distribution of 3D data using scene descriptions.

[0305] <Flow 2 of file generation process>

[0306] Further, by applying the above-described method 1-2, the sub-mesh association information can be provided from the MAF to the PE, and a scene description can be generated so that the 3D data is reconstructed from the V-DMC data by using the sub-mesh association information in the PE. For example, the file generation apparatus 300 can generate a scene description representing a scene composed of 3D objects, and set first reference information to the first buffer storing the sub-mesh association information in the scene description.

[0307] In this way, the file generation apparatus 300 can realize the PE reconstruction, and can improve the processing efficiency of the decoding process compared to the case of the MAF reconstruction. That is, the file generation apparatus 300 can suppress the decrease in the processing efficiency of the decoding process.

[0308] In this case, for example, the above-described method 1-2-1 can be applied, and the file generation apparatus 300 can newly define the reference information to the buffer storing the sub-mesh association information in the scene description as in the example of Figure 28 or Figure 43 That is, the first buffer storing the sub-mesh association information can be a different buffer from the second buffer storing the data of the mesh.

[0309] An example of the flow of the file generation process performed by the file generation apparatus 300 in this case will be described with reference to the flowchart of Figure 41

[0310] In the file generation process in this case, each of the processes from step S341 to step S344 is performed similarly to each of the processes from step S301 to step S304. Figure 42 Figure 41 The V-DMC encoding process in step S342 is performed in a flow similar to the example of

[0311] In step S345, the scene description generation unit 334 sets the reference information to the buffer storing the base mesh, the reference information to the buffer storing the displacement video, the reference information to the buffer storing the attribute video, the reference information to the buffer storing the atlas information, and the reference information to the buffer storing the sub-mesh association information in the scene description.

[0312] Then, each of the processes of step S346 and step S347 is performed similarly to each of the processes of step S306 and step S307. Figure 32

[0313] When the process of step S347 ends, the file generation process ends.

[0314] ​​​By executing each of the processes as described above, the file generation apparatus 300 can implement PE reconstruction, and can improve the processing efficiency of the decoding process compared to the case of MAF reconstruction. That is, the file generation apparatus 300 can suppress a decrease in the processing efficiency of the decoding process.

[0315] <Flow 3 of file generation process>

[0316] In addition, the above-described method 1-2-2 can be applied, as in the example of Figure 44 In the example of the above-described method 1-2-2, the file generation apparatus 300 can store the subnet mesh association information by extending the atlas data in the scene description. That is, the first buffer that stores the subnet mesh association information can be configured to store the atlas information including the subnet mesh association information.

[0317] An example of the flow of the file generation process performed by the file generation apparatus 300 in this case will be described with reference to a flowchart of Figure 41

[0318] In the file generation process in this case, each of the processes from step S361 to step S364 is performed similarly to each of the processes from step S301 to step S304. Figure 42 The V-DMC encoding process in step S362 is performed in a flow similar to the example of Figure 41

[0319] In step S365, the scene description generation unit 334 sets, in the scene description, reference information to the buffer that stores the base mesh, reference information to the buffer that stores the displacement video, reference information to the buffer that stores the attribute video, and reference information to the buffer that stores the atlas information including the subnet mesh association information.

[0320] Then, each of the processes of step S366 and step S367 is performed similarly to each of the processes of step S306 and step S307. Figure 34

[0321] When the process of step S367 ends, the file generation process ends.

[0322] By executing each of the processes as described above, the file generation apparatus 300 can implement PE reconstruction, and can improve the processing efficiency of the decoding process compared to the case of MAF reconstruction. That is, the file generation apparatus 300 can suppress a decrease in the processing efficiency of the decoding process.

[0323] <Flow 4 of file generation process>

[0324] In addition, the above-described method 1-2-3 can be applied, and as in the example of Figure 45 ​​​In the example of FIG. 6, the file generation apparatus 300 stores the subgrid association information by extending the base grid in the scene description. That is, the file generation apparatus 300 can be configured to store the base grid including the subgrid association information.

[0325] An example of the flow of the file generation process performed by the file generation apparatus 300 in this case will be described with reference to the flowchart of FIG. 7. Figure 41

[0326] In the file generation process in this case, each of the processes from step S381 to step S384 is performed similarly to each of the processes of steps S301 to S304 of FIG. 6. Figure 42 The V-DMC encoding process in step S382 is performed in a similar flow to the example of FIG. 6. Figure 41

[0327] In step S385, the scene description generation unit 334 sets, in the scene description, reference information to the buffer storing the base grid including the subgrid association information, reference information to the buffer storing the displacement video, reference information to the buffer storing the attribute video, and reference information to the buffer storing the atlas information.

[0328] Then, each of the processes of steps S386 and S387 is performed similarly to each of the processes of steps S306 and S307 of FIG. 6. Figure 46

[0329] When the process of step S387 ends, the file generation process ends.

[0330] By performing each of the processes as described above, the file generation apparatus 300 can achieve the PE reconstruction, and can improve the processing efficiency of the decoding process compared to the case of the MAF reconstruction. That is, the file generation apparatus 300 can suppress the decrease in the processing efficiency of the decoding process.

[0331] <Flow of file generation process 5>

[0332] In addition, the above-described method 1-2-4 can be applied, the subdivision information can be provided from the MAF to the PE, and the base grid can be subdivided in the PE by using the subdivision information. That is, in the file generation apparatus 300, the scene description generation unit can set, in the scene description, second reference information to a second buffer storing subdivision information related to the subdivision of the base grid whose vertices have been thinned.

[0333] ​​​For example, Method 1-2-1 and Method 1-2-4 can be applied, and the file generation apparatus 300 can set, in the scene description, first reference information to the first buffer storing the sub-mesh association information and second reference information to a second buffer different from the first buffer and storing atlas information including the subdivision information.

[0334] An example of the flow of the file generation process performed by the file generation apparatus 300 in this case will be described with reference to the flowchart of FIG. 17. Figure 41

[0335] In the file generation process in this case, each process from Step S401 to Step S404 is performed similarly to each process of Steps S301 to S304 of Figure 42 Note that the V-DMC encoding process in Step S402 is performed in a flow similar to the example of Figure 41

[0336] In Step S405, the scene description generation unit 334 sets, in the scene description, reference information to the buffer storing the base mesh, reference information to the buffer storing the displacement video, reference information to the buffer storing the attribute video, reference information to the buffer storing the atlas information including the subdivision information, and reference information to the buffer storing the sub-mesh association information.

[0337] Then, each process of Step S406 and Step S407 is performed similarly to each process of Step S306 and Step S307 of Figure 47

[0338] When the process of Step S407 ends, the file generation process ends.

[0339] By performing each process as described above, the file generation apparatus 300 can achieve PE reconstruction, and can improve the processing efficiency of the decoding process compared to the case of MAF reconstruction. In addition, since the PE can acquire the subdivision information and subdivide the base mesh based on the subdivision information, the file generation apparatus 300 can further improve the processing efficiency of the decoding process compared to the case where only Method 1-2-1 is applied.

[0340] <Flow of File Generation Process 6>

[0341] In addition, Method 1-2-2 and Method 1-2-4 can be applied, and the file generation apparatus 300 can set, in the scene description, reference information to the buffer storing the atlas information including the sub-mesh association information and the subdivision information.

[0342] An example of the flow of the file generation process performed by the file generation apparatus 300 in this case will be described with reference to the flowchart of FIG. 17. Figure 41 ​​​​

[0343] In the file generation processing in this case, each processing from Step S421 to Step S424 is executed similarly to each processing of Steps S301 to S304 of Figure 42 Note that the V-DMC encoding processing in Step S422 is executed in a similar flow to the example of Figure 41

[0344] In Step S425, the scene description generation unit 334 sets, in the scene description, reference information to the buffer storing the base mesh, reference information to the buffer storing the displacement video, reference information to the buffer storing the attribute video, and reference information to the buffer storing the atlas information including the sub-mesh association information and the tessellation information.

[0345] Then, each processing of Step S426 and Step S427 is executed similarly to each processing of Steps S306 and S307 of Figure 48

[0346] When the processing of Step S427 ends, the file generation processing ends.

[0347] By executing each processing as described above, the file generation apparatus 300 can achieve the PE reconstruction, and can improve the processing efficiency of the decoding processing compared to the case of the MAF reconstruction. In addition, since the PE can acquire the tessellation information and tessellate the base mesh based on the tessellation information, the file generation apparatus 300 can further improve the processing efficiency of the decoding processing compared to the case where only the method 1-2-2 is applied.

[0348] <Flow 7 of file generation processing>

[0349] In addition, the method 1-2-3 and the method 1-2-4 can be applied, and the file generation apparatus 300 can set, in the scene description, first reference information to the first buffer storing the base mesh including the sub-mesh association information and second reference information to the second buffer storing the atlas information including the tessellation information.

[0350] An example of the flow of the file generation processing executed by the file generation apparatus 300 in this case will be described with reference to the flowchart of Figure 41

[0351] In the file generation processing in this case, each processing from Step S441 to Step S444 is executed similarly to each processing of Steps S301 to S304 of Figure 42 Note that the V-DMC encoding processing in Step S442 is executed in a similar flow to the example of Figure 41

[0352] ​​​​In step S445, the scene description generation unit 334 sets reference information in the scene description for the buffer storing the base grid including subgrid association information, the buffer storing displacement video, the buffer storing attribute video, and the buffer storing atlas information including subdivision information.

[0353] Then, with Figure 49 Each process of steps S306 and S307 is similarly executed as each process of steps S446 and S447.

[0354] When step S447 is completed, the file generation process ends.

[0355] By performing each of the processes described above, the file generation apparatus 300 can achieve PE reconstruction and improve the processing efficiency of the decoding process compared to the case of MAF reconstruction. Furthermore, since the PE can acquire subdivision information and subdivide the base mesh based on that information, the file generation apparatus 300 can further improve the processing efficiency of the decoding process compared to the case where only method 1-2-2 is applied.

[0356] <5. Second Implementation Method>

[0357] <Application of Method 1-1>

[0358] <Reproduction Device>

[0359] The above-described technology can be applied to any device. Figure 49 This is a block diagram illustrating an example configuration of a reproduction device as an aspect of an information processing apparatus to which this technology is applied. Figure 49 The reproduction apparatus 500 shown is a reproduction apparatus that performs reproduction processing of mesh data (V-DMC data). For example, the reproduction apparatus 500 acquires a scene description generated by the file generation apparatus 300, acquires a content file generated by the file generation apparatus 300 based on the scene description, decodes the V-DMC bitstream stored in the content file, and reproduces the V-DMC data. That is, the reproduction apparatus 500 uses the V-DMC data to reconstruct the mesh, performs rendering to generate a display image, and displays the display image.

[0360] Notice, Figure 49 The main components, such as the processing unit and data flow, are shown, and Figure 49 Those shown are not necessarily all. That is to say, in the reproduction device 500, there may be elements not shown in the diagram. Figure 49 The processing unit shown in the diagram is a block, or there may be units not shown in the diagram. Figure 49 The processing or data flow is shown as arrows, etc.

[0361] like Figure 21 The reproduction apparatus 500 includes a presentation engine (PE) 501, a media access function (MAF) 502, and a buffer 503, as shown. In addition, the PE 501 includes a scene description acquisition unit 511, a control unit 512, and a display processing unit 513.

[0362] The scene description acquisition unit 511 performs processing related to acquisition of a scene description. For example, the scene description acquisition unit 511 can acquire a scene description file generated by the file generation apparatus 300. The scene description acquisition unit 511 can provide the acquired scene description file to the control unit 512.

[0363] The PE 501 performs processing related to control using a scene description.

[0364] The control unit 512 performs processing related to control of decoding processing. For example, the control unit 512 can acquire a scene description file provided from the scene description acquisition unit 511. The control unit 512 can control the MAF 502 based on a scene description stored in the scene description file. Further, the control unit 512 can control the display processing unit 513 based on the scene description.

[0365] The MAF 502 performs processing related to a content file. For example, the MAF 502 can perform acquisition of a content file, decoding of a bitstream, and the like under the control of the control unit 512 of the PE 501. Further, the MAF 502 can store decoded data in the buffer 503.

[0366] The buffer 503 has a storage area and performs processing related to storage of information. For example, the buffer 503 can store data provided from the MAF in its own storage area. In addition, the buffer 503 can provide data stored in its own storage area to the PE. That is, the MAF 502 can provide data related to a content to the display processing unit 513 of the PE 501 via the buffer 503.

[0367] The display processing unit 513 performs processing related to display of a content. For example, the display processing unit 513 can acquire data related to a content from the buffer 503. The display processing unit 513 can generate display information of a content by using the acquired data. For example, the display processing unit 513 can generate display information by rendering 3D data or the like. Here, the display information indicates information to be displayed. For example, a display image (an image to be displayed) or the like can be included in the display information. The display processing unit 513 can provide the generated display information to a display apparatus (not shown) to be displayed.

[0368] The reproduction apparatus 500 with this configuration can be a second information processing apparatus, and various methods (the present technology) described above in <3. Usage scenario description of V-DMC data distribution> can be applied.

[0369] For example, the above-described method 1-1 can be applied so that the MAF 502 reconstructs 3D data from the V-DMC data. That is, the MAF 502 can store, in the buffer 503, the geometry of the 3D data to be reconstructed using the V-DMC data and the attributes of the 3D data to be reconstructed using the V-DMC data on the basis of the scene description. The PE 501 can acquire the geometry and the attributes of the reconstructed 3D data from the buffer 503, and generate display information using this data. That is, the reproduction apparatus 500 can perform MAF reconstruction as in the example of Figure 50

[0370] <maf>

[0371] Figure 50 A main configuration example of the MAF 502 in the case where the above-described method 1-1 is applied is shown. Note that, Figure 50 main parts of processing units, data streams, and the like are shown, and Figure 50 those shown in FIG. 5A are not necessarily all. That is, in the MAF 502, there can be processing units not shown as blocks in Figure 50 FIG. 5A, or there can be processing or data streams not shown as arrows and the like in Figure 50 FIG. 5A.

[0372] As shown in Figure 49 the case, the MAF 502 includes a content acquisition unit 531, a demultiplexing unit 532, a meshlet information decoding unit 533, a base mesh decoding unit 534, a geometry video decoding unit 535, an attribute video decoding unit 536, a subdivision unit 537, a displacement video decoding unit 538, a displacement vector application unit 539, and an attribute mapping unit 540.

[0373] The content acquisition unit 531 performs processing related to content acquisition. For example, the content acquisition unit 531 can acquire a designated content file under the control of the control unit 512 of the PE 501. Further, a V-DMC bitstream can be acquired from the content file. The content acquisition unit 531 can provide the V-DMC bitstream to the demultiplexing unit 532.

[0374] The demultiplexing unit 532 performs processing related to demultiplexing. For example, the demultiplexing unit 532 can acquire the V-DMC bitstream provided from the content acquisition unit 531. Further, the demultiplexing unit 532 can demultiplex the acquired V-DMC bitstream, and extract encoded data (substream) of meshlet information, encoded data (substream) of a base mesh, encoded data (substream) of a geometry video, and encoded data (substream) of an attribute video. Thus, the demultiplexing unit 532 can also be referred to as an acquisition unit of various types of information included in the V-DMC bitstream. The demultiplexing unit 532 can provide the extracted encoded data (substream) of meshlet information to the meshlet information decoding unit 533. Further, the demultiplexing unit 532 can provide the extracted encoded data (substream) of a base mesh to the base mesh decoding unit 534. Further, the demultiplexing unit 532 can provide the extracted encoded data (substream) of a geometry video to the geometry video decoding unit 535. Further, the demultiplexing unit 532 can provide the extracted encoded data (substream) of an attribute video to the attribute video decoding unit 536.

[0375] The atlas information decoding unit 533 performs processing related to decoding of the encoded data (sub-stream) of the atlas information. For example, the atlas information decoding unit 533 can acquire the encoded data (sub-stream) of the atlas information provided from the demultiplexing unit 532. Further, the atlas information decoding unit 533 can decode the acquired encoded data (sub-stream) of the atlas information to generate (restore) the atlas information. The atlas information decoding unit 533 can provide the atlas information to the subdivision unit 537, the displacement video decoding unit 538, the displacement vector application unit 539, and the attribute mapping unit 540.

[0376] The base mesh decoding unit 534 performs processing related to decoding of the encoded data (sub-stream) of the base mesh. For example, the base mesh decoding unit 534 can acquire the encoded data (sub-stream) of the base mesh provided from the demultiplexing unit 532. Further, the base mesh decoding unit 534 can decode the acquired encoded data (sub-stream) of the base mesh in accordance with a predetermined decoding scheme (e.g., Draco or the like), and generate (restore) information (e.g., a list of vertices, a list of triangles, or the like) related to the base mesh. The base mesh decoding unit 534 can also reconstruct the base mesh using the information related to the base mesh (a restored base mesh can be generated). The base mesh decoding unit 534 can provide the base mesh (the restored base mesh) to the subdivision unit 537.

[0377] The geometry video decoding unit 535 performs processing related to decoding of the encoded data (sub-stream) of the geometry video. For example, the geometry video decoding unit 535 can acquire the encoded data (sub-stream) of the geometry video provided from the demultiplexing unit 532. Further, the geometry video decoding unit 535 can decode the encoded data (sub-stream) of the geometry video in accordance with a predetermined decoding scheme for 2D moving images to generate (restore) the geometry video. The geometry video decoding unit 535 can provide the generated geometry video to the displacement video decoding unit 538.

[0378] The attribute video decoding unit 536 performs processing related to decoding of the encoded data (sub-stream) of the attribute video. For example, the attribute video decoding unit 536 can acquire the encoded data (sub-stream) of the attribute video provided from the demultiplexing unit 532. Further, the attribute video decoding unit 536 can decode the encoded data of the attribute video in accordance with a predetermined decoding scheme for 2D moving images, and generate (restore) the attribute video. The attribute video decoding unit 536 can provide the attribute video to the attribute mapping unit 540.

[0379] The subdivision unit 537 performs processing related to subdivision of the triangles of the base mesh (restored base mesh). For example, the subdivision unit 537 can acquire the base mesh (restored base mesh) provided from the base mesh decoding unit 534. The subdivision unit 537 can acquire the atlas information (e.g., a subdivision method, an identification information of a submesh (submesh_id), an identification information of a subpart (subpart_id), and the like) provided from the atlas information decoding unit 533. The subdivision unit 537 can subdivide the triangles of the base mesh (restored base mesh) based on the atlas information to generate a split point. The subdivision unit 537 can provide the subdivided base mesh (subdivided restored base mesh) to the displacement vector application unit 539.

[0380] The displacement video decoding unit 538 performs processing related to decoding (conversion) of the geometry video. For example, the displacement video decoding unit 538 can acquire the geometry video provided from the geometry video decoding unit 535. The displacement video decoding unit 538 can acquire the atlas information (e.g., a conversion method, a conversion parameter, 2D patch information, and the like) provided from the atlas information decoding unit 533. The displacement video decoding unit 538 can convert the geometry video to a displacement video based on the atlas information. The displacement video decoding unit 538 can provide the displacement video generated in this way to the displacement vector application unit 539.

[0381] The displacement vector application unit 539 performs processing related to application of the displacement vector to the subdivided base mesh (subdivided restored base mesh). For example, the displacement vector application unit 539 can acquire the subdivided base mesh (subdivided restored base mesh) provided from the subdivision unit 537. The displacement vector application unit 539 can acquire the atlas information (e.g., 2D patch information (2d patch information), subpart identification information (subpart_id), and the like) provided from the atlas information decoding unit 533. The displacement vector application unit 539 can acquire the displacement video provided from the displacement video decoding unit 538. The displacement vector application unit 539 can unpack the displacement vector from the displacement video. The displacement vector application unit 539 can apply the displacement vector to the vertices of the subdivided base mesh (subdivided restored base mesh). That is, the displacement vector application unit 539 can reconstruct a mesh (geometry) corresponding to the original mesh. In the present specification, the mesh (geometry) corresponding to the original mesh is also referred to as a decoded mesh. The displacement vector application unit 539 can provide the generated decoded mesh (geometry of the reconstructed 3D data) to the attribute mapping unit 540.

[0382] The attribute mapping unit 540 performs processing relating to mapping of attributes. For example, the attribute mapping unit 540 can acquire atlas information supplied from the atlas information decoding unit 533. The attribute mapping unit 540 can acquire attribute video supplied from the attribute video decoding unit 536. The attribute mapping unit 540 can unpack attributes from the attribute video. The attribute mapping unit 540 can acquire a decoded mesh (geometry of reconstructed 3D data) supplied from the displacement vector application unit 539. The attribute mapping unit 540 can map attributes to faces of the decoded mesh (geometry of reconstructed 3D data) on the basis of the atlas information. The attribute mapping unit 540 can store the decoded mesh (geometry of reconstructed 3D data) in the buffer 503-1. Further, the attribute mapping unit 540 can store attributes mapped to the decoded mesh (attributes of reconstructed 3D data) in the buffer 503-2.

[0383] The buffer 503-1 and the buffer 503-2 are different buffer areas formed in the buffer 503( Figure 51 ). That is, the geometry of reconstructed 3D data and the attributes are stored in different buffers. That is, the geometry of reconstructed 3D data and the attributes are supplied to the PE 501 via different buffers from each other.

[0384] With such a configuration, the reproduction apparatus 500 can realize MAF reconstruction. That is, the reproduction apparatus 500 can realize distribution of V-DMC data using scene description, and can suppress reduction in coding efficiency in distribution of 3D data using scene description.

[0385] <Flow 1 of reproduction processing>

[0386] An example of the flow of reproduction processing performed by the reproduction apparatus 500 having such a configuration will be described with reference to a flowchart of Figure 50 .

[0387] When the reproduction processing starts, in step S501, the scene description acquisition unit 511 of the reproduction apparatus 500 acquires a scene description.

[0388] In step S502, the MAF 502 performs MAF processing on the basis of the scene description, generates geometry and attributes of reconstructed 3D data, and stores the geometry and the attributes in a buffer.

[0389] In step S503, the display processing unit 513 acquires the geometry and the attributes of reconstructed 3D data from the buffer specified by the scene description. In the case of the example of Figure 52 , the display processing unit 513 acquires the geometry of reconstructed 3D data from the buffer 503-1, and acquires the attributes of reconstructed 3D data from the buffer 503-2.

[0390] In step S504, the display processing unit 513 generates display information related to the reconstructed 3D data using the acquired geometry and attributes, and causes the display device to display the display information. For example, the display processing unit 513 performs rendering or the like of the reconstructed 3D data and generates the display information.

[0391] When the processing of step S504 ends, the reproduction processing ends.

[0392] <MAF processing flow 2>

[0393] Next, an example of the flow of the MAF processing performed in step S502 of Figure 51 will be described with reference to a flowchart of Figure 51 .

[0394] When the MAF processing starts, in step S521, the content acquisition unit 531 acquires the content (V-DMC bitstream).

[0395] In step S522, the demultiplexing unit 532 demultiplexes the V-DMC bitstream to generate an atlas information substream, a base mesh substream, a geometry video substream, and an attribute video substream.

[0396] In step S523, the atlas information decoding unit 533 decodes the atlas information substream to generate (restore) the atlas information.

[0397] In step S524, the base mesh decoding unit 534 decodes the base mesh substream to generate (restore) the base mesh.

[0398] In step S525, the geometry video decoding unit 535 decodes the geometry video substream to generate (restore) the geometry video.

[0399] In step S526, the attribute video decoding unit 536 decodes the attribute video substream to generate (restore) the attribute video.

[0400] In step S527, the subdivision unit 537 subdivides the base mesh.

[0401] In step S528, the displacement video decoding unit 538 converts the geometry video into a displacement video.

[0402] In step S529, the displacement vector application unit 539 unpacks displacement vectors from the displacement video, applies the displacement vectors to the vertices of the subdivided base mesh, and generates a decoded mesh.

[0403] In step S530, the attribute mapping unit 540 unpacks attributes from the attribute video, and maps the attributes to the decoded mesh.

[0404] In step S531, the attribute mapping unit 540 stores the geometry and the attributes of the reconstructed 3D data in the buffers specified by the scene description. For example, the attribute mapping unit 540 stores the geometry of the reconstructed 3D data in the buffer 503-1, and stores the attributes of the reconstructed 3D data in the buffer 503-2.

[0405] When the processing of step S531 ends, the MAF processing ends, and the processing returns to Figure 22 .

[0406] By executing each processing as described above, the reproduction apparatus 500 can implement the MAF reconstruction. That is, the reproduction apparatus 500 can implement the distribution of the V-DMC data using the scene description, and can suppress the reduction of the encoding efficiency in the distribution of the 3D data using the scene description.

[0407] <Method 1-2>

[0408] <Reproduction apparatus>

[0409] For example, the above-described method 1-2 can be applied to the reproduction apparatus 500. That is, the reproduction apparatus 500 can perform the PE reconstruction as in the example of Figure 53 In this case, for example, as shown in Figure 49 , in addition to the configuration shown in Figure 50 , the PE 501 can include a displacement vector application unit 539 and an attribute mapping unit 540. The displacement vector application unit 539 and the attribute mapping unit 540 perform similar processing to the case of the example of Figure 54 . That is, the MAF 502 can perform the subdivision of the base mesh, and the PE 501 can perform the vertex displacement (refinement) and the mapping of the attributes by the displacement vector (displacement). In this case, the MAF 502 provides the subdivided base mesh to the PE 501 via the buffer 503.

[0410] In this way, compared to the case of the MAF reconstruction, the reproduction apparatus 500 can improve the processing efficiency of the decoding processing. That is, the reproduction apparatus 500 can suppress the reduction of the processing efficiency of the decoding processing.

[0411] <Method 1-2-1>

[0412] <maf>

[0413] In the reproduction apparatus 500 having such a configuration, for example, the above-described method 1-2-1 can be applied. That is, in the scene description, reference information to a buffer storing sub-mesh association information can be newly defined. In this case, the MAF 502 can have a configuration as illustrated in Figure 54 .

[0414] Note that, Figure 54 main parts of processing units, data flows, and the like are shown, and Figure 54 those shown in Figure 54 are not necessarily all. That is, in the MAF 502, there can be processing units not shown as blocks in Figure 54 , or there can be processing or data flows not shown as arrows and the like in .

[0415] As Figure 50 illustrated, in this case, the MAF 502 can include a content acquisition unit 531 to a displacement video decoding unit 538 in the configuration of the MAF 502 described above. Each of these processing units can perform similar processing to that in the example of Figure 50 . Figure 53

[0416] However, the atlas information decoding unit 533 can store the atlas information in a buffer 503-1. The subdivision unit 537 can store the subdivided base mesh in a buffer 503-2. The displacement video decoding unit 538 can store the displacement video in a buffer 503-3. The attribute video decoding unit 536 can store the attribute video in a buffer 503-4. The subdivision unit 537 can store the sub-mesh association information in a buffer 503-5.

[0417] The buffers from the buffer 503-1 to the buffer 503-5 are buffer areas different from each other formed in the buffer 503( Figure 53 ). That is, in the case of the present example, the sub-mesh association information is stored in a buffer different from the V-DMC data. Therefore, the displacement vector application unit 539 and the attribute mapping unit 540 of the PE 501( Figure 55 ) respectively acquire the V-DMC data and the sub-mesh association information from the buffer 503-1 to the buffer 503-5. In other words, the sub-mesh association information is provided to the PE 501 via a buffer different from the V-DMC data.

[0418] With such a configuration, the reproduction apparatus 500 can realize PE reconstruction. That is, compared to the case of MAF reconstruction, the reproduction apparatus 500 can improve the processing efficiency of the decoding process. That is, the reproduction apparatus 500 can suppress a decrease in the processing efficiency of the decoding process.

[0419] <Flow 2 of reproduction processing>

[0420] An example of the flow of the reproduction processing performed by the reproduction apparatus 500 having such a configuration will be described with reference to the flowchart of Figure 56

[0421] When the reproduction processing starts, in step S551, the scene description acquisition unit 511 of the reproduction apparatus 500 acquires the scene description.

[0422] In step S552, the MAF 502 performs MAF processing based on the scene description, generates V-DMC data and subnet mesh association information, and stores the V-DMC data and the subnet mesh association information in the buffer 503.

[0423] In step S553, the PE 501 acquires each of the subdivided base mesh, the displacement video, the attribute video, the atlas information, and the subnet mesh association information from the buffer specified by the scene description.

[0424] In step S554, the displacement vector application unit 539 of the PE 501 applies displacement vectors to the vertices of the subdivided base mesh based on the subnet mesh association information, and generates a decoded mesh. That is, the displacement vector application unit 539 generates the geometry of the reconstructed 3D data.

[0425] In step S555, the attribute mapping unit 540 of the PE 501 maps attributes to the decoded mesh based on the subnet mesh association information. That is, the attribute mapping unit 540 generates the attributes of the reconstructed 3D data.

[0426] In step S556, the display processing unit 513 of the PE 501 generates display information using the decoded mesh (the geometry of the reconstructed 3D data) and the attributes mapped to the decoded mesh (the attributes of the reconstructed 3D data), and displays the display information.

[0427] When the processing of step S556 ends, the reproduction processing ends.

[0428] <Flow 2 of MAF processing>

[0429] Next, an example of the flow of the MAF processing performed in step S552 of Figure 55 Figure 52

[0430] ​​​In the MAF processing in this case, each processing from Figure 55 Each processing from step S571 to step S578 is executed similarly to each processing of steps S521 to S528.

[0431] In step S579, the MAF 502 stores each of the subdivided base mesh, the displacement video, the attribute video, the atlas information, and the sub-mesh association information in the buffer specified by the scene description. That is, the subdivision unit 537 stores the subdivided base mesh in the buffer 503-2. The displacement video decoding unit 538 stores the displacement video in the buffer 503-3. The attribute video decoding unit 536 stores the attribute video in the buffer 503-4. The atlas information decoding unit 533 stores the atlas information in the buffer 503-1. The subdivision unit 537 stores the sub-mesh association information in the buffer 503-5.

[0432] When the processing of step S579 ends, the MAF processing ends, and the processing returns to Figure 53 .

[0433] By executing each processing as described above, the reproduction apparatus 500 can achieve PE reconstruction. That is, compared to the case of MAF reconstruction, the reproduction apparatus 500 can improve the processing efficiency of the decoding processing. That is, the reproduction apparatus 500 can suppress the decrease in the processing efficiency of the decoding processing.

[0434] <Application of Method 1-2-2>

[0435] <maf>

[0436] Further, for example, the method 1-2-2 can be applied to Figure 57 the reproduction apparatus 500. That is, the atlas data can be extended to store the submesh association information. In this case, the MAF 502 can have a configuration as shown in Figure 57 .

[0437] Note that Figure 57 the main parts of processing units, data flows, and the like are shown, and Figure 57 those shown in the Figure 57 are not necessarily all. That is, in the MAF 502, there can be processing units not shown as blocks in the Figure 57 , or there can be processing or data flows not shown as arrows and the like in the .

[0438] As shown in Figure 54 , in this case, the MAF 502 has a similar configuration to the MAF 502 described with reference to Figure 50 . Each of these processing units can perform processing similar to the case of the example of Figure 53 . However, in this case, the atlas information includes the submesh association information. Then, the atlas information decoding unit 533 can store the atlas information including the submesh association information in the buffer 503-1. The subdivision unit 537 can store the subdivided base mesh in the buffer 503-2. The displacement video decoding unit 538 can store the displacement video in the buffer 503-3. The attribute video decoding unit 536 can store the attribute video in the buffer 503-4.

[0439] The buffers from the buffer 503-1 to the buffer 503-4 are buffer areas different from each other formed in the buffer 503( Figure 53 ). That is, in the case of the present example, the submesh association information is stored in the buffer 503-1 together with the atlas information. Therefore, the displacement vector application unit 539 and the attribute mapping unit 540 of the PE 501( Figure 55 ) acquire the V-DMC data and the submesh association information from the buffer 503-1 to the buffer 503-4, respectively. In other words, the submesh association information is provided to the PE 501 via the same buffer as the atlas information.

[0440] With such a configuration, the reproduction apparatus 500 can achieve PE reconstruction. That is, compared to the case of MAF reconstruction, the reproduction apparatus 500 can improve the processing efficiency of the decoding process. That is, the reproduction apparatus 500 can suppress a decrease in the processing efficiency of the decoding process.

[0441] <MAF processing flow 3>

[0442] The reproduction processing performed by the reproduction apparatus 500 having this configuration is performed in a flow similar to that of the flowchart of Figure 58 will be described. The flow of the MAF processing performed in step S552 of Figure 55 will be described. The flow of the MAF processing performed in step S552 of Figure 52 will be described. The flow of the MAF processing performed in step S552 of

[0443] In the MAF processing in this case, each processing from step S601 to step S608 is performed similarly to each processing of steps S521 to S528 of Figure 55

[0444] In step S609, the MAF 502 stores each of the subdivided base mesh, the displacement video, the attribute video, and the atlas information including the sub-mesh association information in the buffer specified by the scene description. That is, the subdivision unit 537 stores the subdivided base mesh in the buffer 503-2. The displacement video decoding unit 538 stores the displacement video in the buffer 503-3. The attribute video decoding unit 536 stores the attribute video in the buffer 503-4. The atlas information decoding unit 533 stores the atlas information including the sub-mesh association information in the buffer 503-1.

[0445] When the processing of step S609 ends, the MAF processing ends, and the processing returns to Figure 53 .

[0446] By performing each processing as described above, the reproduction apparatus 500 can achieve the PE reconstruction. That is, compared to the case of the MAF reconstruction, the reproduction apparatus 500 can improve the processing efficiency of the decoding processing. That is, the reproduction apparatus 500 can suppress the decrease in the processing efficiency of the decoding processing.

[0447] <Method 1-2-3>

[0448] <maf>

[0449] Further, for example, the method 1-2-3 can be applied to Figure 59 the reproduction apparatus 500. That is, the base grid can be expanded to store the sub-grid association information. In this case, the MAF 502 can have a configuration as Figure 59 indicated.

[0450] In addition, Figure 59 main parts of processing units, data streams, and the like are shown, and Figure 59 those shown in the Figure 59 are not necessarily all. That is, in the MAF 502, there can be processing units not shown as blocks in the Figure 59 , or there can be processing or data streams not shown as arrows and the like in the .

[0451] As Figure 54 indicated, in this case, the MAF 502 has a similar configuration to the MAF 502 described with reference to Figure 50 . Each of these processing units can perform processing similar to that of the case of the example of Figure 53 . However, in this case, the subdivided base grid includes the sub-grid association information. Then, the subdivision unit 537 can store the subdivided base grid including the sub-grid association information in the buffer 503-2. The atlas information decoding unit 533 can store the atlas information in the buffer 503-1. The displacement video decoding unit 538 can store the displacement video in the buffer 503-3. The attribute video decoding unit 536 can store the attribute video in the buffer 503-4.

[0452] The buffers from the buffer 503-1 to the buffer 503-4 are buffer areas different from each other formed in the buffer 503( Figure 53 ). That is, in the case of the present example, the sub-grid association information is stored in the buffer 503-2 together with the subdivided base grid. Therefore, the displacement vector application unit 539 and the attribute mapping unit 540 of the PE 501( Figure 55 ) respectively acquire the V-DMC data and the sub-grid association information from the buffer 503-1 to the buffer 503-4. In other words, the sub-grid association information is provided to the PE 501 via the same buffer as the subdivided base grid.

[0453] With such a configuration, the reproduction apparatus 500 can achieve PE reconstruction. That is, compared to the case of MAF reconstruction, the reproduction apparatus 500 can improve the processing efficiency of the decoding process. That is, the reproduction apparatus 500 can suppress a decrease in the processing efficiency of the decoding process.

[0454] <MAF processing flow 4>

[0455] The reproduction processing performed by the reproduction apparatus 500 having such a configuration is performed in a flow similar to that of the flowchart of Figure 60 The flow of the MAF processing performed in step S552 of Figure 55 will be described with reference to the flowchart of Figure 52 the flow of the MAF processing performed in step S552 of

[0456] In the MAF processing in this case, each processing from step S631 to step S638 is performed similarly to each processing of steps S521 to S528 of Figure 55

[0457] In step S639, the MAF 502 stores each of the subdivided base mesh including the submesh association information, the displacement video, the attribute video, and the atlas information in the buffer specified by the scene description. That is, the subdivision unit 537 stores the subdivided base mesh including the submesh association information in the buffer 503-2. The displacement video decoding unit 538 stores the displacement video in the buffer 503-3. The attribute video decoding unit 536 stores the attribute video in the buffer 503-4. The atlas information decoding unit 533 stores the atlas information in the buffer 503-1.

[0458] When the processing of step S639 ends, the MAF processing ends, and the processing returns to Figure 36 .

[0459] By performing each processing as described above, the reproduction apparatus 500 can achieve the PE reconstruction. That is, compared to the case of the MAF reconstruction, the reproduction apparatus 500 can improve the processing efficiency of the decoding processing. That is, the reproduction apparatus 500 can suppress the decrease in the processing efficiency of the decoding processing.

[0460] <Method 1-2-4>

[0461] <Reproduction Apparatus>

[0462] For example, the above-described method 1-2-4 can be applied to the reproduction apparatus 500. That is, the reproduction apparatus 500 can perform the PE reconstruction as in the example of Figure 61 , and the PE can perform the subdivision of the base mesh. In this case, for example, as shown in Figure 53 , the PE 501 can include the subdivision unit 537 in addition to the configuration shown in Figure 50 . Also in this case, the subdivision unit 537 performs each processing from step S631 to step S638 similarly to each processing of steps S521 to S528 of Figure 62 ​an example similar processing. That is, the PE 501 can perform subdivision of a base mesh, displacement of vertices by a displacement, and mapping of attributes (refinement). In this case, the MAF 502 provides the base mesh (before being subdivided) and subdivision information to the PE 501 via the buffer 503.

[0463] By so doing, the reproduction apparatus 500 can further improve the processing efficiency of the decoding process. That is, the reproduction apparatus 500 can suppress a decrease in the processing efficiency of the decoding process.

[0464] <Method 1-2-1>

[0465] <maf>

[0466] In the reproduction apparatus 500 having such a configuration, for example, the method 1-2-1 and the method 1-2-4 can be applied. That is, the reference information to the buffer storing the sub-mesh association information can be newly defined in the scene description, and the subdivision information can be provided from the MAF to the PE, and the base mesh can be subdivided in the PE using the subdivision information. In this case, the MAF 502 can have a configuration as shown in Figure 62 .

[0467] Note that Figure 62 the main parts of the processing units, data flows, and the like are shown, and Figure 62 those shown in FIG. 12A are not necessarily all. That is, in the MAF 502, there can be processing units not shown as blocks in Figure 62 , or there can be processing or data flows not shown as arrows and the like in Figure 62 .

[0468] As shown in Figure 50 , in this case, the MAF 502 can include the content acquisition unit 531 to the attribute video decoding unit 536 and the displacement video decoding unit 538 in the configuration of the MAF 502 described with reference to Figure 50 . Each of these processing units can perform processing similar to the case of the example of Figure 61 .

[0469] However, the atlas information decoding unit 533 can store the atlas information including the subdivision information in the buffer 503-1. The base mesh decoding unit 534 can store the base mesh in the buffer 503-2. The displacement video decoding unit 538 can store the displacement video in the buffer 503-3. The attribute video decoding unit 536 can store the attribute video in the buffer 503-4. The base mesh decoding unit 534 can store the sub-mesh association information in the buffer 503-5.

[0470] The buffers from the buffer 503-1 to the buffer 503-5 are buffer areas different from each other formed in the buffer 503( Figure 61 ). That is, in the case of the present example, the sub-mesh association information is stored in a buffer different from the V-DMC data. Therefore, the subdivision unit 537, the displacement vector application unit 539, and the attribute mapping unit 540 of the PE 501( Figure 63 ) V-DMC data, subdivision information, and submesh association information are acquired from each of the buffers 503-1 to 503-5. In other words, the subdivision information is supplied to the PE 501 via the same buffer as the atlas information. In addition, the submesh association information is supplied to the PE 501 via a different buffer from the V-DMC data.

[0471] With such a configuration, the reproduction apparatus 500 is able to further improve the processing efficiency of the decoding process. That is, the reproduction apparatus 500 is able to suppress a decrease in the processing efficiency of the decoding process.

[0472] <Flow 3 of reproduction processing>

[0473] An example of the flow of the reproduction processing performed by the reproduction apparatus 500 having such a configuration will be described with reference to a flowchart of Figure 64

[0474] When the reproduction processing starts, in step S661, the scene description acquisition unit 511 of the reproduction apparatus 500 acquires a scene description.

[0475] In step S662, the MAF 502 performs MAF processing based on the scene description, and stores V-DMC data, subdivision information, and submesh association information in the buffer 503.

[0476] In step S663, the PE 501 acquires each of the base mesh, the displacement video, the attribute video, the atlas information including the subdivision information, and the submesh association information from the buffers specified by the scene description.

[0477] In step S664, the subdivision unit 537 of the PE 501 subdivides the base mesh based on the subdivision information.

[0478] In step S665, the displacement vector application unit 539 of the PE 501 applies displacement vectors to the vertices of the subdivided base mesh based on the submesh association information, and generates a decoded mesh. That is, the displacement vector application unit 539 generates a geometry of the reconstructed 3D data.

[0479] In step S666, the attribute mapping unit 540 of the PE 501 maps attributes to the decoded mesh based on the submesh association information. That is, the attribute mapping unit 540 generates attributes of the reconstructed 3D data.

[0480] In step S667, the display processing unit 513 of the PE 501 generates display information using the decoded mesh (the geometry of the reconstructed 3D data) and the attributes mapped to the decoded mesh (the attributes of the reconstructed 3D data), and displays the display information.

[0481] ​When the processing of step S667 ends, the reproduction processing ends.

[0482] <MAF processing flow 5>

[0483] Next, an example of the flow of the MAF processing performed in step S662 of Figure 63 will be described with reference to a flowchart. Figure 52

[0484] In the MAF processing in this case, each of the processes from step S701 to step S707 is performed similarly to each of the processes of steps S521 to S526 and the process of step S528 of Figure 63 .

[0485] In step S708, the MAF 502 stores each of the base mesh, the displacement video, the attribute video, the atlas information including the subdivision information, and the sub-mesh association information in the buffers specified by the scene description. That is, the base mesh decoding unit 534 stores the base mesh in the buffer 503-2. The displacement video decoding unit 538 stores the displacement video in the buffer 503-3. The attribute video decoding unit 536 stores the attribute video in the buffer 503-4. The atlas information decoding unit 533 stores the atlas information including the subdivision information in the buffer 503-1. The base mesh decoding unit 534 stores the sub-mesh association information in the buffer 503-5.

[0486] When the processing of step S708 ends, the MAF processing ends, and the processing returns to Figure 61 .

[0487] By performing each of the processes as described above, the reproduction apparatus 500 can further improve the processing efficiency of the decoding processing. That is, the reproduction apparatus 500 can suppress a decrease in the processing efficiency of the decoding processing.

[0488] <Method 1-2-2>

[0489] <maf>

[0490] Further, for example, the method 1-2-2 and the method 1-2-4 can be applied to the reproduction apparatus 500 in Figure 65 . That is, the atlas data can be expanded to store the submesh association information, and the subdivision information can be provided from the MAF to the PE to use the subdivision information to subdivide the base mesh in the PE. In this case, the MAF 502 can have a configuration as illustrated in Figure 65 .

[0491] Note that Figure 65 the main parts of processing units, data flows, and the like are illustrated, and Figure 65 those illustrated in Figure 65 are not necessarily all. That is, in the MAF 502, there can be processing units that are not illustrated as blocks in Figure 65 , or there can be processing or data flows that are not illustrated as arrows and the like in .

[0492] As illustrated in Figure 62 , in this case, the MAF 502 has a similar configuration to the MAF 502 described with reference to Figure 50 . Each of these processing units can perform processing similar to the case of the example of Figure 61 . However, in this case, the atlas information includes the submesh association information and the subdivision information. Then, the atlas information decoding unit 533 can store the atlas information including the submesh association information and the subdivision information in the buffer 503-1. The base mesh decoding unit 534 can store the base mesh in the buffer 503-2. The displacement video decoding unit 538 can store the displacement video in the buffer 503-3. The attribute video decoding unit 536 can store the attribute video in the buffer 503-4.

[0493] The buffers from the buffer 503-1 to the buffer 503-4 are buffer areas different from each other formed in the buffer 503( Figure 61 ). That is, in the case of the present example, the submesh association information and the subdivision information are stored in the buffer 503-1 together with the atlas information. Therefore, the subdivision unit 537, the displacement vector application unit 539, and the attribute mapping unit 540 of the PE 501( Figure 63 ) acquire the V-DMC data, the submesh association information, and the subdivision information from each of the buffer 503-1 to the buffer 503-4. In other words, the submesh association information and the subdivision information are provided to the PE 501 via the same buffer as the atlas information.

[0494] By such a configuration, the reproduction apparatus 500 can further improve the processing efficiency of the decoding process. That is, the reproduction apparatus 500 can suppress a decrease in the processing efficiency of the decoding process.

[0495] <MAF processing flow 6>

[0496] The reproduction processing performed by the reproduction apparatus 500 having such a configuration is performed in a flow similar to that of the flowchart of Figure 66 will be described with reference to the flowchart of Figure 63 will be described with reference to the flowchart of Figure 64 The example of the flow of the MAF processing performed in step S662 of

[0497] In the MAF processing in this case, each processing from step S731 to step S737 is performed similarly to each processing of steps S701 to S707 of Figure 63

[0498] In step S738, the MAF 502 stores each of the base mesh, the displacement video, the attribute video, and the atlas information including the sub-mesh association information and the tessellation information in the buffer specified by the scene description. That is, the base mesh decoding unit 534 stores the base mesh in the buffer 503-2. The displacement video decoding unit 538 stores the displacement video in the buffer 503-3. The attribute video decoding unit 536 stores the attribute video in the buffer 503-4. The atlas information decoding unit 533 stores the atlas information including the sub-mesh association information and the tessellation information in the buffer 503-1.

[0499] When the processing of step S738 ends, the MAF processing ends, and the processing returns to Figure 61 .

[0500] By performing each processing as described above, the reproduction apparatus 500 can further improve the processing efficiency of the decoding process. That is, the reproduction apparatus 500 can suppress a decrease in the processing efficiency of the decoding process.

[0501] <Method 1-2-3>

[0502] <maf>

[0503] Further, for example, the method 1-2-3 and the method 1-2-4 can be applied to the reproduction apparatus 500 in Figure 67 . That is, the base mesh can be expanded to store the sub-mesh association information, the subdivision information can be provided from the MAF to the PE, and the base mesh can be subdivided by using the subdivision information in the PE. In this case, the MAF 502 can have a configuration as illustrated in Figure 67 .

[0504] Note that Figure 67 the main parts of processing units, data flows, and the like are illustrated, and Figure 67 those illustrated in Figure 67 are not necessarily all. That is, in the MAF 502, there can be processing units that are not illustrated as blocks in Figure 67 , or there can be processing or data flows that are not illustrated as arrows and the like in .

[0505] As illustrated in Figure 62 , in this case, the MAF 502 has a similar configuration to the MAF 502 described with reference to Figure 50 . Each of these processing units can perform processing similar to the case of the example of Figure 61 . However, in this case, the base mesh includes the sub-mesh association information. Then, the base mesh decoding unit 534 can store the base mesh including the sub-mesh association information in the buffer 503-2. In addition, the atlas information includes the subdivision information. The atlas information decoding unit 533 can store the atlas information including the subdivision information in the buffer 503-1. The displacement video decoding unit 538 can store the displacement video in the buffer 503-3. The attribute video decoding unit 536 can store the attribute video in the buffer 503-4.

[0506] The buffers from the buffer 503-1 to the buffer 503-4 are buffer areas different from each other formed in the buffer 503( Figure 61 ). That is, in the case of the present example, the sub-mesh association information is stored in the buffer 503-2 together with the base mesh. Further, the subdivision information is stored in the buffer 503-1 together with the atlas information. Therefore, the subdivision unit 537, the displacement vector application unit 539, and the attribute mapping unit 540 of the PE 501( Figure 63 ) acquire the V-DMC data, the sub-mesh association information, and the subdivision information from each of the buffer 503-1 to the buffer 503-4. In other words, the sub-mesh association information is provided to the PE 501 via the same buffer as the base mesh. In addition, the subdivision information is provided to the PE 501 via the same buffer as the atlas information.

[0507] By such a configuration, the reproduction apparatus 500 can further improve the processing efficiency of the decoding process. That is, the reproduction apparatus 500 can suppress a decrease in the processing efficiency of the decoding process.

[0508] <MAF process flow 7>

[0509] The reproduction process performed by the reproduction apparatus 500 having such a configuration is performed in a flow similar to that of the flowchart of Figure 68 will be described with reference to the flowchart of Figure 63 will be described with reference to the flowchart of Figure 64 the flow of the MAF process performed in step S662 of

[0510] In the MAF process in this case, each of the processes from step S761 to step S767 is performed similarly to each of the processes of steps S701 to S707 of Figure 63

[0511] In step S768, the MAF 502 stores each of the subdivided base mesh including the submesh association information, the displacement video, the attribute video, and the atlas information in the buffer specified by the scene description. That is, the subdivision unit 537 stores the subdivided base mesh including the submesh association information in the buffer 503-2. The displacement video decoding unit 538 stores the displacement video in the buffer 503-3. The attribute video decoding unit 536 stores the attribute video in the buffer 503-4. The atlas information decoding unit 533 stores the atlas information in the buffer 503-1.

[0512] When the process of step S768 ends, the MAF process ends, and the process returns to Figure 69 .

[0513] By performing each of the processes as described above, the reproduction apparatus 500 can further improve the processing efficiency of the decoding process. That is, the reproduction apparatus 500 can suppress a decrease in the processing efficiency of the decoding process.

[0514] <6. SUPPLEMENTARY NOTE>

[0515] <COMPUTER>

[0516] The series of processes described above can be performed by hardware or can be performed by software. In the case where the series of processes is performed by software, a program included in the software is installed on a computer. Here, the computer includes, for example, a computer incorporating a dedicated hardware, a general-purpose personal computer capable of executing various functions by installing various programs, and the like.

[0517] Figure 69 ​is a block diagram showing an example of a hardware configuration of a computer that executes the above series of processes according to a program.

[0518] In ​ In the computer 900 shown in FIG. 1, a central processing unit (CPU) 901, a read only memory (ROM) 902, and a random access memory (RAM) 903 are connected to each other via a bus 904.

[0519] Further, an input / output interface 910 is also connected to the bus 904. An input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected to the input / output interface 910.

[0520] The input unit 911 can include, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, and the like. The output unit 912 can include, for example, a display, a speaker, an output terminal, and the like. The storage unit 913 can include, for example, a hard disk, a RAM disk, a non-volatile memory, and the like. The communication unit 914 can include, for example, a network interface and the like. The drive 915 can drive a removable medium 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0521] For example, in the computer configured as described above, the CPU 901 loads a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes the program, thereby executing the above series of processes. The RAM 903 also appropriately stores data and the like required by the CPU 901 to execute various types of processes.

[0522] For example, the program executed by the computer can be applied by being recorded in the removable medium 921 as a package medium or the like. In this case, when the removable medium 921 is attached to the drive 915, the program can be installed in the storage unit 913 via the input / output interface 910.

[0523] Further, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or a digital satellite broadcast. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0524] In addition, the program can be pre-installed in the ROM 902 or the storage unit 913.

[0525] <Objects to which the present technology is applied>

[0526] The present technology can be applied to any encoding / decoding scheme.

[0527] Further, the present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0528] Further, for example, the present technology can also be implemented as a part of a configuration of an apparatus, such as a processor (for example, a video processor) of a system large scale integration (LSI) or the like, a module (for example, a video module) using a plurality of processors or the like, a unit (for example, a video unit) using a plurality of modules or the like, or a collection (for example, a video collection) obtained by additionally adding other functions to the unit, or the like.

[0529] Further, for example, the present technology can also be applied to a network system including a plurality of apparatuses. For example, the present technology can be implemented as cloud computing shared and processed by a plurality of apparatuses in cooperation via a network. For example, the present technology can be implemented in a cloud service that provides a service related to an image (moving image) to any terminal such as a computer, an audio-visual (AV) apparatus, a portable information processing terminal, or an Internet of Things (IoT) apparatus.

[0530] Note that, in the present specification, a system refers to a collection of a plurality of components (apparatuses, modules (parts), or the like), and it is not important whether all the components are in the same housing. Thus, both a plurality of apparatuses stored in different housings and connected via a network and one apparatus in which a plurality of modules are stored in one housing are a system.

[0531] <Fields and applications to which the present technology is applicable>

[0532] A system, an apparatus, a processing unit, or the like to which the present technology is applied can be used in any field such as transportation, medical care, crime prevention, agriculture, animal husbandry, mining, beauty care, a factory, a household appliance, weather, and natural monitoring. Further, the application thereof is also arbitrary.

[0533] For example, the present technology can be applied to a system and an apparatus for providing content for appreciation or the like. Further, for example, the present technology can also be applied to a system and an apparatus for transportation such as traffic condition management and automatic driving control. Further, for example, the present technology can also be applied to a system and an apparatus for security. Further, for example, the present technology can be applied to a system and an apparatus for automatic control of a machine or the like. Further, for example, the present technology can also be applied to a system and an apparatus for use in agriculture and animal husbandry. Further, for example, the present technology can also be applied to a system and an apparatus for monitoring a natural condition such as a volcano, a forest, and a sea, a wild animal, or the like. Further, for example, the present technology can also be applied to a system and an apparatus for sports.

[0534] <Others>

[0535] Note that in this specification, a "flag" is information for identifying a plurality of states, and includes not only information for identifying two states of true (1) and false (0), but also information capable of identifying three or more states. Therefore, the value that can be taken by the "flag" can be, for example, binary or ternary or more of 1 / 0. That is, the number of bits that form the "flag" is an arbitrary number, and can be one bit or a plurality of bits. Further, assuming that the identification information (including the flag) includes not only the identification information thereof in a bit stream, but also difference information of the identification information with respect to certain reference information in the bit stream, in this specification, the "flag" and the "identification information" include not only the information thereof, but also the difference information with respect to the reference information.

[0536] Further, various information (for example, metadata) relating to encoded data (bit stream) can be transmitted or recorded in any form as long as the information is associated with the encoded data. Here, the term "associated" means, for example, that other data can be used (linked) when one data is processed. That is, data associated with each other can be collected as one data, or can be separate data. For example, information associated with encoded data (image) can be transmitted on a transmission path different from that of the encoded data (image). Further, for example, information associated with encoded data (image) can be recorded in a recording medium different from that of the encoded data (image) (or another recording area of the same recording medium). Note that the "associated" can not be the entire data, but a part of the data. For example, an image and information corresponding to the image can be associated with each other in any unit (for example, a plurality of frames, one frame, or a part within a frame).

[0537] Note that in this specification, terms such as "combine", "multiplex", "add", "incorporate", "include", "store", "put", "introduce", and "insert" mean to combine a plurality of objects into one, for example, to combine encoded data and metadata into one data, and mean one method of the above-described "associated".

[0538] Further, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present technology.

[0539] For example, a configuration described as one device (or processing unit) can be divided and configured as a plurality of devices (or processing units). Conversely, a configuration described as a plurality of devices (or processing units) can be collectively configured as one device (or processing unit). Furthermore, it goes without saying that a configuration other than the above-described configurations can be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, a part of the configuration of a certain device (or processing unit) can be included in the configuration of another device (or another processing unit).

[0540] Furthermore, for example, the above-described program can be executed in any device. In this case, it is only necessary that the device has the necessary functions (functional blocks and the like) and obtains the necessary information.

[0541] Furthermore, for example, each step of one flowchart can be executed by one device, or can be shared and executed by a plurality of devices. Furthermore, in the case where a plurality of processes are included in one step, the plurality of processes can be executed by one device, or can be shared and executed by a plurality of devices. In other words, the plurality of processes included in one step can also be executed as the processes of a plurality of steps. Conversely, the processes described as a plurality of steps can also be collectively executed as one step.

[0542] Furthermore, for example, in a program executed by a computer, the processes that describe the steps of the program can be executed in a time series in the order described in this specification, or can be executed individually at a desired timing such as when a call is made. That is, as long as there is no contradiction, the processes of the respective steps can be executed in an order different from the above-described order. Furthermore, the processes that describe the steps of the program can be executed in parallel with the processes of another program, or can be executed in combination with the processes of another program.

[0543] Furthermore, for example, as long as there is no contradiction, a plurality of technologies related to the present technology can be independently realized as a single entity. Of course, a plurality of arbitrary present technologies can be realized in combination. For example, part or all of the present technology described in any of the embodiments can be realized in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be realized together with another technology not described above.

[0544] Note that the present technology can also have the following configuration.

[0545] (1) An information processing apparatus including: a scene description generation unit that generates a scene description that represents a scene composed of 3D objects, and sets first reference information to a first buffer that stores sub-mesh association information in the scene description, wherein the sub-mesh association information is information that specifies data of a sub-mesh that is a part of a mesh that represents the 3D object.

[0546] (2) The information processing apparatus according to (1), wherein the first buffer is a different buffer from a second buffer storing data of the mesh.

[0547] (3) The information processing apparatus according to (2), wherein the sub-mesh association information includes boundary information indicating a boundary of the sub-mesh in data of a base mesh in which vertices of the mesh have been sparsified.

[0548] (4) The information processing apparatus according to (3), wherein the boundary information includes information indicating a length of data of the sub-mesh and identification information of the sub-mesh.

[0549] (5) The information processing apparatus according to any one of (1) to (4), wherein the first buffer is configured to store atlas information including the sub-mesh association information.

[0550] (6) The information processing apparatus according to (5), wherein the atlas information includes the sub-mesh association information associated with a patch.

[0551] (7) The information processing apparatus according to (6), wherein the sub-mesh association information includes boundary information indicating a boundary of the sub-mesh in data of a base mesh in which vertices of the mesh have been sparsified.

[0552] (8) The information processing apparatus according to (7), wherein the boundary information includes information indicating an offset and a length of data of the sub-mesh.

[0553] (9) The information processing apparatus according to any one of (1) to (8), wherein the first buffer is configured to store a base mesh including the sub-mesh association information.

[0554] (10) The information processing apparatus according to (9), wherein the sub-mesh association information includes identification information of the sub-mesh.

[0555] (11) The information processing apparatus according to (9) or (10), wherein the sub-mesh association information includes identification information of the sub-mesh associated with each vertex of the base mesh.

[0556] (12) The information processing apparatus according to any one of (1) to (11), wherein the scene description generation unit further sets, in the scene description, second reference information to a second buffer storing tessellation information related to tessellation of a base mesh in which vertices of the mesh have been sparsified.

[0557] (13) The information processing apparatus according to (12), wherein the second buffer is configured to store atlas information including the subdivision information.

[0558] (14) The information processing apparatus according to (13), wherein the atlas information includes the subdivision information associated with a patch.

[0559] (15) The information processing apparatus according to any one of (12) to (14), wherein the subdivision information includes information indicating whether a predetermined method is applied in subdivision of the base mesh.

[0560] (16) The information processing apparatus according to any one of (12) to (15), wherein the subdivision information includes information indicating a number of iterations of subdivision of the base mesh.

[0561] (17) An information processing method comprising: generating a scene description representing a scene composed of 3D objects; and setting first reference information to a first buffer storing sub-mesh association information in the scene description, wherein the sub-mesh association information is information for specifying data of a sub-mesh that is a part of a mesh representing the 3D object.

[0562] (21) An information processing apparatus comprising: a storage processing unit that stores sub-mesh association information for specifying data of a sub-mesh that is a part of a mesh representing a 3D object in a first buffer designated by a scene description representing a scene composed of the 3D object; and a reconstruction unit that acquires the sub-mesh association information from the first buffer, specifies data of the sub-mesh using the acquired sub-mesh association information, and reconstructs the mesh.

[0563] (22) The information processing apparatus according to (21), wherein the first buffer is a different buffer from a second buffer storing data of the mesh.

[0564] (23) The information processing apparatus according to (22), wherein the sub-mesh association information includes boundary information indicating a boundary of the sub-mesh in data of a base mesh in which a vertex of the mesh has been thinned.

[0565] (24) The information processing apparatus according to (23), wherein the boundary information includes information indicating a length of data of the sub-mesh and identification information of the sub-mesh.

[0566] (25) The information processing apparatus according to any one of (21) to (24), wherein the first buffer is configured to store atlas information including the sub-mesh association information.

[0567] (26) The information processing apparatus according to (25), wherein the atlas information includes the submesh association information associated with a patch.

[0568] (27) The information processing apparatus according to (26), wherein the submesh association information includes boundary information indicating a boundary of the submesh in data of a base mesh in which vertices of the mesh have been sparsified.

[0569] (28) The information processing apparatus according to (27), wherein the boundary information includes information indicating an offset and a length of data of the submesh.

[0570] (29) The information processing apparatus according to any one of (21) to (28), wherein the first buffer is configured to store a base mesh including the submesh association information.

[0571] (30) The information processing apparatus according to (29), wherein the submesh association information includes identification information of the submesh.

[0572] (31) The information processing apparatus according to (29) or (30), wherein the submesh association information includes identification information of the submesh associated with each vertex of the base mesh.

[0573] (32) The information processing apparatus according to (21) to (31), wherein the storage processing unit further stores, in a second buffer specified by the scene description, subdivision information about subdivision of a base mesh in which vertices of the mesh have been sparsified, and the reconstruction unit acquires the subdivision information from the second buffer, subdivides the base mesh using the acquired subdivision information, and reconstructs the mesh using the subdivided base mesh.

[0574] (33) The information processing apparatus according to (32), wherein the second buffer is configured to store atlas information including the subdivision information.

[0575] (34) The information processing apparatus according to (33), wherein the atlas information includes the subdivision information associated with a patch.

[0576] (35) The information processing apparatus according to any one of (32) to (34), wherein the subdivision information includes information indicating whether a predetermined method is applied in subdivision of the base mesh.

[0577] (36) The information processing apparatus according to any one of (32) to (35), wherein the subdivision information includes information indicating a number of iterations of subdivision of the base mesh.

[0578] (37) An information processing method, comprising: storing, in a first buffer specified by a scene description representing a scene composed of 3D objects, submesh association information for specifying data of a submesh that is a part of a mesh representing the 3D objects; and acquiring the submesh association information from the first buffer, specifying data of the submesh using the acquired submesh association information, and reconstructing the mesh.

[0579] List of reference signs

[0580] 300 file generation apparatus

[0581] 311 control unit

[0582] 312 file generation processing unit

[0583] 313 input unit

[0584] 331 V-DMC data generation unit

[0585] 332 V-DMC data encoding unit

[0586] 333 file generation unit

[0587] 334 scene description generation unit

[0588] 335 storage unit

[0589] 336 providing unit

[0590] 351 atlas information encoding unit

[0591] 352 base mesh encoding unit

[0592] 353 displacement vector correction unit

[0593] 354 displacement video encoding unit

[0594] 355 mesh reconstruction unit

[0595] 356 attribute map conversion unit

[0596] 357 attribute video encoding unit

[0597] 358 multiplexing unit

[0598] 500 reproduction apparatus

[0599] 501 PE

[0600] 502 MAF

[0601] 503 buffer

[0602] 511 scene description acquisition unit

[0603] 512 control unit

[0604] 513 display processing unit

[0605] 531 content acquisition unit

[0606] 532 demultiplexing unit

[0607] 533 atlas information decoding unit

[0608] 534 base mesh decoding unit

[0609] 535 geometry video decoding unit

[0610] 536 attribute video decoding unit

[0611] 537 tessellation unit

[0612] 538 displacement video decoding unit

[0613] 539 displacement vector application unit

[0614] 540 attribute mapping decoding unit

[0615] 900 computer< / maf> ​< / maf> ​< / maf> < / maf> ​< / maf> < / maf> < / maf> ​

Claims

1. An information processing apparatus comprising: a scene description generation unit that generates a scene description that represents a scene composed of 3D objects, and sets first reference information to a first buffer that stores sub-mesh association information in the scene description, wherein the sub-mesh association information is information that specifies data of a sub-mesh that is a part of a mesh that represents the 3D object.

2. The information processing apparatus according to claim 1, wherein the first buffer is a buffer that is different from a second buffer that stores data of the mesh.

3. The information processing apparatus according to claim 2, wherein the sub-mesh association information includes boundary information that indicates a boundary of the sub-mesh in data of a base mesh in which vertices of the mesh have been thinned.

4. The information processing apparatus according to claim 1, wherein the first buffer is configured to store atlas information that includes the sub-mesh association information.

5. The information processing apparatus according to claim 4, wherein the sub-mesh association information includes boundary information that indicates a boundary of the sub-mesh in data of a base mesh in which vertices of the mesh have been thinned.

6. The information processing apparatus according to claim 1, wherein the first buffer is configured to store a base mesh that includes the sub-mesh association information.

7. The information processing apparatus according to claim 6, wherein the sub-mesh association information includes identification information of the sub-mesh.

8. The information processing apparatus according to claim 1, wherein the scene description generation unit further sets second reference information to a second buffer that stores subdivision information related to subdivision of a base mesh in which vertices of the mesh have been thinned in the scene description.

9. The information processing apparatus according to claim 8, wherein the second buffer is configured to store atlas information that includes the subdivision information. 10.An information processing method comprising: generating a scene description that represents a scene composed of 3D objects; and setting first reference information to a first buffer that stores sub-mesh association information in the scene description, wherein the sub-mesh association information is information that specifies data of a sub-mesh that is a part of a mesh that represents the 3D object. 11.An information processing apparatus comprising: a storage processing unit that stores sub-mesh association information that specifies data of a sub-mesh that is a part of a mesh that represents 3D objects in a first buffer specified by a scene description that represents a scene composed of the 3D objects; and a reconstruction unit that acquires the sub-mesh association information from the first buffer, specifies data of the sub-mesh using the acquired sub-mesh association information, and reconstructs the mesh.

12. The information processing apparatus according to claim 11, wherein the first buffer is a buffer that is different from a second buffer that stores data of the mesh.

13. The information processing apparatus according to claim 12, wherein the sub-mesh association information includes boundary information that indicates a boundary of the sub-mesh in data of a base mesh in which vertices of the mesh have been thinned.

14. The information processing apparatus according to claim 11, wherein the first buffer is configured to store atlas information that includes the sub-mesh association information.

15. The information processing apparatus according to claim 14, wherein the sub-mesh association information includes boundary information that indicates a boundary of the sub-mesh in data of a base mesh in which vertices of the mesh have been thinned.

16. The information processing apparatus according to claim 11, wherein the first buffer is configured to store a base mesh that includes the sub-mesh association information.

17. The information processing apparatus according to claim 16, wherein the sub-mesh association information includes identification information of the sub-mesh. 18.The information processing apparatus according to claim 11, wherein, The storage processing unit also stores, in a second buffer specified by the scene description, subdivision information about subdivision of a base mesh whose vertices have been thinned out by the mesh, and The reconstruction unit acquires the subdivision information from the second buffer, subdivides the base mesh using the acquired subdivision information, and reconstructs the mesh using the subdivided base mesh.

19. The information processing apparatus according to claim 18, wherein The second buffer is configured to store atlas information including the subdivision information.

20. An information processing method, comprising: storing, in a first buffer specified by a scene description representing a scene composed of a 3D object, sub-mesh association information for specifying data of a sub-mesh that is a part of a mesh representing the 3D object; and acquiring the sub-mesh association information from the first buffer, specifying data of the sub-mesh using the acquired sub-mesh association information, and reconstructing the mesh.

Images