Information processing device and method

Abstract

The present disclosure relates to an information processing device and method with which it is possible to simultaneously use a plurality of video concepts for a single object using a scene description. In a 3D object video extension defined in a material layer of a scene description file, a component index having a different value for each video component is set for a video component included in a 3D object video. The video component is stored in a corresponding buffer on the basis of the component index, the video component is acquired from the buffer on the basis of the component index, and an image for display is generated using the acquired video component. The present disclosure can be applied, for example, to an information processing device, an information processing method, and the like.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and method, and particularly relates to an information processing apparatus and method that enable simultaneous use of a plurality of video components for one object using scene description.

Background Art

[0002] Conventionally, there has been glTF (The GL Transmission Format) (registered trademark) 2.0, which is a Scene Description format for arranging and rendering 3D (three-dimensional) objects in a three-dimensional space (see, for example, Non-Patent Document 1).

[0003] Furthermore, in MPEG (Moving Picture Experts Group)-I Scene Description, a method of extending glTF 2.0 to handle temporally dynamic content has been proposed (see, for example, Non-Patent Document 2).

[0004] Furthermore, rendering support for 3D object video composed of texture video and dynamic mesh has been proposed (see, for example, Non-Patent Document 3). Among 3D objects composed of such texture video and dynamic mesh, there has been a technique for encoding texture video in the VD texture (View Dependent Texture) format (see, for example, Patent Document 1 and Patent Document 2). In the case of the VD texture method, a higher-quality 3D object can be displayed compared to conventional texture video.

[0005] VD textures are texture videos that are mapped onto a mesh according to the viewpoint. In the VD texture method, multiple VD textures are prepared for a single dynamic mesh, each mapped from a different direction. Multiple VD textures can be mapped to a single mesh simultaneously.

[0006] In recent years, there has been a growing expectation for MPEG-I Scene Descriptions to support 3D data that allows for the simultaneous use of multiple video components for a single 3D object, similar to the VD texture method.

[0007] Furthermore, there was a method for separating accessors for each tile in the mesh.primitives.attribute layer of the scene description (see, for example, Non-Patent Document 4). Also, there was a method for associating multiple LoD textures with the material layer of the scene description so that the texture used can be switched according to the change in viewpoint position (see, for example, Non-Patent Document 5). In addition, the bitstream of the 3D object video as described above can be stored in, for example, ISOBMFF (International Organization for Standardization Base Media File Format) (see, for example, Non-Patent Document 6). [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] Saurabh Bhatia, Patrick Cozzi, Alexey Knyazev, Tony Parisi, "Khronos glTF2.0", https: / / github.com / KhronosGroup / glTF / tree / master / specification / 2.0, June 9, 2017 [Non-Patent Document 2] "Potential improvement of ISO / IEC DIS 23090-14 Scene Description for MPEG Media", ISO / IEC JTC 1 / SC 29 / WG 03 N0321, 2021-08-03 [Non-Patent Document 3] "Technologies under Consideration on Scene Description for MPEG Media", ISO / IEC JTC 1 / SC 29 / WG 03 N292, 2021-07-26 [Non-Patent Document 4] Imed Bouazizi, Thomas Stockhammer, "Attribute Registration for V3C", ISO / IEC JTC 1 / SC 29 / WG 03 m57411, 2021-07-07 [Non-Patent Document 5] Saurabh Bhatia, Gary Hsu, Adam Gritt, John Copic, Marc Appelsmeier, Dag Frommhold, "MSFT_lod", https: / / github.com / KhronosGroup / glTF / blob / main / extensions / 2.0 / Vendor / MSFT_lod / README.md [Non-Patent Document 6] "Information technology - JPEG 2000 image coding system - Part 12: ISO base media file format", ISO / IEC FDIS 14496-12:2018, 2020-12 [Patent Documents]

[0009] [Patent Document 1] International Publication No. 2021 / 079592 [Patent Document 2] International Publication No. 2021 / 193213 [Overview of the project] [Problems that the invention aims to solve]

[0010] However, conventional MPEG-I Scene Descriptions could not describe multiple video components that could be used simultaneously for a single 3D object within the scene description.

[0011] This disclosure is made in light of these circumstances and enables the simultaneous use of multiple video components for a single object using scene descriptions. [Means for solving the problem]

[0012] One aspect of this technology is an information processing device comprising: a file processing unit that stores video components included in a 3D object video in a buffer corresponding to the video component based on a component index; and a display image generation unit that retrieves the video component from the buffer based on the component index and generates a display image using the retrieved video component. Multiple video components can be used simultaneously in a single 3D object, and the component index is set for the video components included in the 3D object video in an extension for the 3D object video defined in the material layer of the scene description, and has a different value for each video component.

[0013] One aspect of this technology is an information processing method which involves storing video components included in a 3D object video in a buffer corresponding to the video component based on a component index, retrieving the video component from the buffer based on the component index, generating a display image using the retrieved video component, and allowing multiple video components to be used simultaneously in a single 3D object. The component index is set for the video components included in the 3D object video in an extension for the 3D object video defined in the material layer of the scene description, and has a different value for each video component.

[0014] The information processing device for another aspect of this technology includes a file generation unit that generates a scene description file and, in an extension for 3D object video defined in the material layer of the scene description file, sets a component index with a different value for each video component included in the 3D object video, and the information processing device can simultaneously use multiple video components in a single 3D object.

[0015] Another aspect of this technology involves generating a scene description file, and in an extension for 3D object video defined in the material layer of the scene description file, setting a component index with a different value for each video component included in the 3D object video, thereby enabling the simultaneous use of multiple video components in a single 3D object.

[0016] In an information processing apparatus and method according to one aspect of the present technology, based on a component index, a video component included in a 3D object video is stored in a buffer corresponding to the video component, the video component is acquired from the buffer based on the component index, and a display image is generated using the acquired video component.

[0017] In an information processing apparatus and method according to another aspect of the present technology, a scene description file is generated, and in an extension for 3D object video defined in a material layer of the scene description file, a component index having a different value for each video component is set for a video component included in the 3D object video. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] [Figure 1] This figure shows a main configuration example of glTF 2.0. [Figure 2] This figure shows an example of glTF objects and reference relationships. [Figure 3] This figure shows a description example of a scene description. [Figure 4] This figure is for explaining a method of accessing binary data. [Figure 5] This figure shows a description example of a scene description. [Figure 6] This figure is for explaining an object extension method. [Figure 7] This figure is for explaining a configuration of client processing. [Figure 8] This figure shows a configuration example of an extension for handling timed metadata. [Figure 9] This figure shows a description example of a scene description. [Figure 10] This figure shows an example of how a client processes dynamic meshes and texture videos. [Figure 11] This figure shows an example of object configuration in a scene description that supports dynamic meshes and textured video. [Figure 12] This figure shows an example of a scene description. [Figure 13] This is a diagram explaining VD textures. [Figure 14] This is a diagram explaining packed VD textures. [Figure 15] This figure shows an example of object configuration in a scene description that supports a tile structure. [Figure 16] This figure shows an example of a scene description that supports texture switching. [Figure 17] This figure shows an example of an ISOBMFF box structure. [Figure 18] This diagram illustrates an example of using scene descriptions to simultaneously utilize multiple video components for a single object. [Figure 19] This figure shows an example of object configuration in a scene description where multiple video components can be used simultaneously for a single object. [Figure 20] This figure shows an example of a scene description. [Figure 21] This figure shows an example of object configuration in a scene description that supports VD textures. [Figure 22] This figure shows an example of object configuration in a scene description that supports packed VD textures. [Figure 23] This figure shows an example of a scene description that stores camera parameters for a VD texture. [Figure 24] This figure shows an example of describing camera parameters. [Figure 25]This figure shows an example of a scene description that stores camera parameters for a VD texture, with an index set for each field. [Figure 26] This figure shows an example of a scene description that stores camera parameters and packing metadata for a packed VD texture. [Figure 27] This figure shows an example of describing camera parameters. [Figure 28] This figure shows an example of how to describe packing metadata. [Figure 29] This figure shows an example of a scene description that stores camera parameters and packing metadata for packed VD textures, each indexed for a specific VD texture. [Figure 30] This figure shows an example of describing camera parameters. [Figure 31] This figure shows an example of how to describe packing metadata. [Figure 32] This figure shows an example of describing camera parameters with an index set for each field. [Figure 33] This figure shows an example of a scene description. [Figure 34] This block shows the main configuration examples of a file generation device. [Figure 35] This is an example of the file generation process flow. [Figure 36] This block shows the main configuration examples for client devices. [Figure 37] This flowchart shows an example of the playback process. [Figure 38] This is a block diagram showing common computer configurations. [Modes for carrying out the invention]

[0019] The following describes the forms for implementing this disclosure (hereinafter referred to as embodiments). The explanation will be given in the following order. 1. Supporting literature and other resources for technical content and terminology. 2. MPEG-I Scene Description 3. Scene descriptions that support multiple video components 4. First Embodiment (File Generation Device) 5. Second Embodiment (Client Device) 6. Addendum

[0020] <1. Literature and other resources supporting the technical content and terminology> The scope disclosed in this technology includes not only the contents described in the embodiments, but also the contents described in the following non-patent and patent documents that were publicly known at the time of filing, as well as the contents of other documents referenced in the following non-patent and patent documents.

[0021] Non-patent document 1: (mentioned above) Non-patent document 2: (mentioned above) Non-patent document 3: (mentioned above) Non-patent document 4: (mentioned above) Non-patent document 5: (mentioned above) Non-patent document 6: (mentioned above) Patent Document 1: (mentioned above) Patent Document 2: (mentioned above)

[0022] In other words, the content described in the aforementioned non-patent and patent documents, as well as the content of other documents referenced in the aforementioned non-patent and patent documents, will also serve as a basis for determining the support requirement. For example, even if glTF2.0 and its extensions described in Non-Patent Documents 1 to 5, the terminology used in the 3D object configuration described in Patent Documents 1 and 2, and the file format structure of Non-Patent Document 6 are not directly defined in the detailed description of the invention, they will still be considered within the scope of this disclosure and satisfy the support requirement of the claims. Similarly, even if technical terms such as Parsing, Syntax, and Semantics are not directly defined in the detailed description of the invention, they will still be considered within the scope of this disclosure and satisfy the support requirement of the claims.

[0023] <2. MPEG-I Scene Description> <gltf2.0> Conventionally, as described in Non-Patent Document 1, for example, there was glTF (The GL Transmission Format) (registered trademark) 2.0, a format for arranging 3D (three-dimensional) objects in three-dimensional space. glTF 2.0 consists of a JSON format file (.glTF), a binary file (.bin), and an image file (.png, .jpg, etc.), as shown in Figure 1, for example. The binary file stores binary data such as geometry and animation. The image file stores data such as textures.

[0024] A JSON format file is a scene description file written in JSON (JavaScript® Object Notation). A scene description is metadata that describes a scene in 3D content. The description of this scene defines what kind of scene it is. A scene description file is a file that stores such a scene description. In this disclosure, a scene description file is also referred to as a scene description file.

[0025] A JSON format file consists of a list of key-value pairs. An example of its format is shown below. “KEY”:”VALUE”

[0026] The key consists of a string. The value consists of a number, string, boolean, array, object, or null, etc.

[0027] Furthermore, multiple key-value pairs ("KEY":"VALUE") can be grouped together using curly braces {}. This grouped structure is also called a JSON object. An example of its format is shown below. “user”:{"id":1, "name":"tanaka"}

[0028] In this example, a JSON object is defined that combines the pairs "id":1 and "name":"tanaka" as the values ​​corresponding to the key (user).

[0029] Furthermore, zero or more values ​​can be enclosed in square brackets [] to create an array. This array is also called a JSON array. For example, a JSON object can be used as an element of this JSON array. An example of its format is shown below. test:["hoge", "fuga", "bar"] "users":[{"id":1, "name":"tanaka"},{"id":2,"name":"yamada"},{"id":3, "name":"sato"}]

[0030] Figure 2 shows the glTF objects that can be placed at the top level of a JSON format file and the reference relationships they can have. In the tree structure shown in Figure 2, the elongated ovals represent objects, and the arrows between those objects indicate the reference relationships. As shown in Figure 2, objects such as "scene", "node", "mesh", "camera", "skin", "material", and "texture" are placed at the top level of a JSON format file.

[0031] An example of such a JSON format file (scene description) is shown in Figure 3. The JSON format file 20 in Figure 3 shows a partial example of the top level. In this JSON format file 20, all top-level objects 21 used are described at the very top. These top-level objects 21 are the glTF objects shown in Figure 2. In addition, the JSON format file 20 shows the reference relationships between objects, as indicated by the arrow 22. More specifically, these reference relationships are shown by specifying the index of the element in the array of the referenced object in the property of the parent object.

[0032] <How to access the data> Figure 4 illustrates how to access binary data. As shown in Figure 4, binary data is stored in a buffer object. In other words, the buffer object contains information for accessing the binary data (e.g., a URI (Uniform Resource Identifier)). In a JSON format file, as shown in Figure 4, objects such as meshes, cameras, and skins can access their buffer objects via accessor objects and bufferView objects.

[0033] In other words, for objects such as meshes, cameras, and skins, the accessor object to be referenced is specified. Figure 5 shows an example of how a mesh object is described in a JSON format file. For example, as shown in Figure 5, in a mesh object, vertex attributes such as NORMAL, POSITION, TANGENT, and TEXCORD_0 are defined as keys, and for each attribute, the accessor object to be referenced is specified as the value.

[0034] <Object Extension> Next, we will explain how to extend such scene description objects. Each object in glTF2.0 can store newly defined objects within an extension object. Figure 6 shows an example of how to specify a newly defined object (ExtensionExample). As shown in Figure 6, when using a newly defined extension, the extension object name (ExtensionExample in the example in Figure 6) is written in “extensionUsed” and “extensionRequired”. This indicates that this extension is either an extension that is used or an extension that is required for loading.

[0035] <Client Processing> Next, we will explain the processing performed by the client device in MPEG (Moving Picture Experts Group)-I Scene Description. The client device obtains the scene description, retrieves 3D object data based on that scene description, and generates a display image using that scene description and 3D object data.

[0036] As described in Non-Patent Document 2, in a client device, a presentation engine, a media access function, and the like perform processing. For example, as shown in FIG. 7, a presentation engine 51 of a client device 50 acquires an initial value of a scene description and information for updating the scene description (hereinafter also referred to as update information), and generates a scene description for a processing target time. Then, the presentation engine 51 parses the scene description and identifies a medium (video, audio, or the like) to be reproduced. Then, the presentation engine 51 requests a media access function 52 to acquire the medium via a media access API (Media Access API (Application Program Interface)). The presentation engine 51 also performs setting of pipeline processing, specification of a buffer, and the like.

[0037] The media access function 52 acquires various data of the medium requested from the presentation engine 51 from a cloud, a local storage, or the like. The media access function 52 supplies various data (encoded data) of the acquired medium to a pipeline 53.

[0038] The pipeline 53 decodes various data (encoded data) of the supplied medium by pipeline processing, and supplies a decoding result to a buffer 54. The buffer 54 holds various data of the supplied medium.

[0039] The presentation engine 51 performs rendering or the like using various data of the medium held in the buffer 54.

[0040] <Application of Timed Media> In recent years, as shown in Non-Patent Document 2, for example, the application of timed media as 3D object content by extending glTF 2.0 in MPEG-I Scene Descriptions has been considered. Timed media is media data that changes in the direction of time, such as moving images in two-dimensional images. In this specification, changing in the direction of time is also referred to as "dynamic," and not changing in the direction of time is also referred to as "static."

[0041] glTF could only handle static data as media data (3D object content). In other words, glTF did not support dynamic media data. To make 3D objects dynamic, animation (a method of switching static media data over time) was used.

[0042] In MPEG-I Scene Description, glTF 2.0 is applied, JSON format files are used as scene descriptions, and further extensions to glTF are being considered to allow the handling of timed media (e.g., video data) as media data. The following extensions are being considered to handle timed media, for example:

[0043] Figure 8 illustrates an extension for handling timed media. In the example in Figure 8, the MPEG (Moving Picture Experts Group) media object (MPEG_media) is a glTF extension that specifies dynamic MPEG media attributes such as uri, track, renderingRate, and startTime.

[0044] Furthermore, as shown in Figure 8, an MPEG texture video object (MPEG_texture_video) is provided as an extension object of the texture object. This MPEG texture video object stores information about the accessor corresponding to the buffer object being accessed. In other words, the MPEG texture video object is an object that specifies the index of the accessor corresponding to the buffer in which the texture media specified by the MPEG media object (MPEG_media) is decoded and stored. A texture object that has an MPEG texture video object is called a video component.

[0045] Figure 9 shows an example of how to describe an MPEG media object (MPEG_media) and an MPEG texture video object (MPEG_texture_video) in a scene description to illustrate extensions for handling timed media. In the example in Figure 9, the second line from the top shows that an MPEG texture video object (MPEG_texture_video) is set as an extension object (extensions) for a texture object (texture), as shown below. The value of that MPEG video texture object is specified as the index of the accessor (in this example, "2").

[0046] "texture":[{"sampler":0, "source":1, "extensions":{"MPEG_texture_video ":"accessor":2}}],

[0047] Furthermore, in the example shown in Figure 9, the MPEG media object (MPEG_media) is set as a glTF extension object (extensions) from the 7th to the 16th row from the top, as shown below. The value of that MPEG media object stores various information about the MPEG media object, such as its encoding and URI.

[0048] "MPEG_media":{ "media":[ {"name":"source_1", "renderingRate":30.0, "startTime":9.0, "timeOffset":0.0, "loop":"true", "controls":"false", "alternatives":[{"mimeType":"video / mp4;codecs=\"avc1.42E01E\"", "uri":"video1.mp4", "tracks":[{"track":""#track_ID=1"}] }] } ] }

[0049] Furthermore, each frame data is decoded and sequentially stored in a buffer, but since its position and other properties change, the scene description includes a mechanism to store this changing information so that the renderer can read the data. For example, as shown in Figure 8, an MPEG buffer circular object (MPEG_buffer_circular) is provided as an extension object (extension) of the buffer object (buffer). This MPEG buffer circular object stores information for dynamically storing data within the buffer object. For example, information such as the data length of the buffer header (bufferHeader) and the number of frames is stored in this MPEG buffer circular object. The buffer header, for example, stores information such as the index, the timestamp of the frame data to be stored, and the data length.

[0050] Furthermore, as shown in Figure 8, an MPEG accessor timed object (MPEG_timed_accessor) is provided as an extension object (extension) of the accessor object (accessor). In this case, since the media data is video, the buffer view object (bufferView) referenced in the time direction may change (its position may fluctuate). Therefore, information indicating the referenced buffer view object is stored in this MPEG accessor timed object. For example, the MPEG accessor timed object stores information indicating a reference to the buffer view object (bufferView) in which a timedAccessor information header is described. The timedAccessor information header is, for example, header information that stores information within the dynamically changing accessor object and buffer view object.

[0051] <Dynamic Mesh Pipeline> Non-patent document 3 discloses the processing of dynamic meshes and texture videos in a client. In the case of a dynamic mesh, for example, as shown in the thick rectangle in Figure 10, the vertex information (position) of the mesh, texture coordinate information indicating the coordinate system of the texture, and vertex indices are stored individually in buffers. These data may be stored in the buffers after encoding or without encoding. A texture video is a dynamic texture that is applied to the dynamic mesh (also referred to as mapping in this specification). The texture video is encoded using a 2D image encoding scheme (e.g., HEVC (High Efficiency Video Coding)), decoded using that encoding scheme, and stored in a buffer different from the dynamic mesh.

[0052] <Dynamic Mesh Texture Settings> Non-patent document 3 discloses a scene description that supports such texture videos and dynamic meshes. Figure 11 shows an example of the main configuration of objects in such a scene description. As shown in Figure 11, the dynamic mesh is stored in a buffer corresponding to the accessor specified in the primitives attribute layer of the scene description. In contrast, the texture video is stored in a buffer corresponding to the accessor specified in the primitives material layer of the scene description. Thus, there is a one-to-one correspondence between dynamic meshes and texture videos. In other words, as shown in the example description in Figure 12, the material layer of the scene description stores information for one texture video.

[0053] <VDテクスチャ> Patent Document 1 discloses a View-Dependent Texture (VD texture) format as a data format for meshes (textures), which maps textures corresponding to the viewpoint to the mesh.

[0054] Conventional textures (also referred to herein as VI textures (View Independent Textures)) consist of textures that map to the entire mesh information M, as shown on the left of Figure 13. Therefore, in the VI texture format, all textures constituting the VI texture are mapped to the mesh information M, regardless of the viewpoint. In other words, the texture is mapped to the entire mesh information M regardless of the position or direction of the viewpoint (whether or not it is used for rendering). As shown on the left of Figure 13, during encoding, the VI texture data is formed as a UV texture map stored (also referred to as packing) in a 2D plane of the UV coordinate system for each chunk (also referred to as a patch). In other words, in this case, one mesh information M and one UV texture map are associated and encoded (i.e., there is a one-to-one correspondence between the mesh and the texture).

[0055] In contrast, a VD texture is a texture that is mapped to a mesh according to the viewpoint. A VD texture is composed of captured images (camera footage) of a 3D object taken from a certain viewpoint, as shown on the right in Figure 13. During mapping, the captured image (texture) is mapped to the mesh from the direction of that viewpoint. In other words, a VD texture contains the texture of the part of the mesh that is visible from that viewpoint. Therefore, generally, one VD texture cannot cover the texture of the entire mesh information M, so multiple VD textures mapped from different directions are associated with one mesh information M, as shown on the right in Figure 13. Then, the necessary VD texture (for example, a VD texture close to the rendering viewpoint position) is selected according to the position and direction of the viewpoint during rendering and mapped to the mesh information M.

[0056] Generally, when comparing the VD texture format and the VI texture format, the VD texture format has a higher rendering (mapping) processing load. However, the encoding load on the encoder side is lower for the VD texture format. Also, the encoding capacity is higher for the VD texture format. However, the VD texture format is more suitable for human perception characteristics, which tend to prioritize texture resolution and fidelity over surface details, and the subjective image quality of the rendered display image is higher.

[0057] Furthermore, multiple VD textures can be mapped to a single mesh simultaneously. In this specification, "simultaneously mapped" means that the state in which multiple VD textures are mapped to a single mesh exists at least once. The timing of mapping each of the multiple VD textures may be different from each other.

[0058] <Packed DVD Textures> Patent Document 2 discloses a data format called Packed VD texture, which packs multiple VD textures into a single image and transmits it to a client. For example, in Figure 14, VD textures 61 to 66, which are composed of images taken from different viewpoints, are packed into a two-dimensional image 60. In other words, in the case of this Packed VD texture, multiple VD textures are encoded as a single image. The decoder decodes the bitstream to obtain the image, extracts the desired VD texture from the image (also called unpacking), and maps it to a mesh.

[0059] In the packed VD texture format, VD textures are generally packed at a lower resolution, resulting in a lower subjective image quality compared to the standard VD texture format. However, even with this packed VD texture format, multiple unpacked VD textures can be mapped simultaneously to a single mesh.

[0060] Furthermore, packed VD textures and VD textures may be used in combination. For example, the encoder may encode multiple VD textures and packed VD textures for a single mesh. In that case, the decoder may map both the VD texture and the unpacked VD texture of the packed VD texture to a single mesh.

[0061] <Simultaneous use of multiple video components> In recent years, there has been a growing expectation for MPEG-I Scene Descriptions to support 3D data that allows the simultaneous use of multiple video components for a single 3D object, such as the VD texture format and packed VD texture format. In this specification, "simultaneous use" means that at least one instance exists where multiple video components are being used for a single 3D object. The timing of the use of each of the video components may differ from one another. Furthermore, "to use" means to perform some processing using the video component. For example, "mapping a texture to a mesh" can be considered "using a texture."

[0062] <Tiles> Incidentally, Non-Patent Document 4 discloses a method for associating meshes with buffers on a tile-by-tile basis in a scene description, for example, as shown in Figure 15. In the example in Figure 15, an extended object is set for each tile in the primitives attribute layer, and each is associated with a different accessor. Therefore, in this method, the mesh data is stored in a different buffer for each tile. Thus, the presentation engine (PE) can select and reconstruct the desired tiles.

[0063] <Texture switching> Furthermore, Non-Patent Document 5 discloses a method for associating multiple textures with a primitive material layer, as shown in Figure 16, for example. In the example of Figure 16, in the scene description 80, an extended object "MSFT_lod" is set as shown in the solid rectangle 82, and the textures are hierarchically structured (Level of Detail) based on resolution, etc., as shown in the solid rectangles 84 and 86. Therefore, the base color information texture (baseColorTexture) has multiple Levels of Detail (Levels of Detail) as shown in the dotted rectangles 81, 83, and 85. Each Level of Detail is assigned a different index. In other words, the texture of each Level of Detail is associated with a different accessor. In other words, the texture of each Level of Detail is stored in a different buffer.

[0064] The decoder's presentation engine (PE) selects one of its multiple Levels of Display (LoDs) and maps it. For example, the presentation engine selects one LoD depending on the viewpoint position. In this way, the presentation engine can switch the LoD of the texture it maps to, for example, in response to changes in viewpoint position.

[0065] <isobmff> Encoded data for 3D object videos, including dynamic meshes and texture videos (e.g., VD textures and packed VD textures), may be stored in a file container such as ISOBMFF (International Organization for Standardization Base Media File Format), as shown in Figure 17. ISOBMFF is disclosed in Non-Patent Literature 6.

[0066] <Support for multiple video components that can be used simultaneously> However, conventional MPEG-I Scene Descriptions could not describe multiple video components that could be used simultaneously for a single 3D object within the scene description.

[0067] For example, in the method described in Non-Patent Document 4, it was possible to associate dynamic meshes with different accessors for each tile in the primitives attribute layer, but it was not possible to associate video components with different accessors in the material layer. Also, in the method described in Non-Patent Document 5, it was not possible to use multiple video components simultaneously.

[0068] <3. Scene description that supports multiple video components> <Method 1> Therefore, as shown in the top row of the table in Figure 18, an extension object for 3D object video is defined in the material layer of the scene description (SD), and an index is set for each video component in that extension object (Method 1).

[0069] For example, an information processing device (e.g., a file generation device) generates a scene description file, and in the extension for 3D object video defined in the material layer of that scene description file, it includes a file generation unit that sets a component index with a different value for each video component included in the 3D object video. Note that multiple video components can be used simultaneously in a single 3D object.

[0070] Furthermore, in the information processing method (for example, the file generation method), a scene description file is generated, and in the extension for 3D object video defined in the material layer of that scene description file, a component index with a different value is set for each video component included in that 3D object video. Note that multiple video components can be used simultaneously in a single 3D object.

[0071] For example, an information processing device (e.g., a playback device) includes a file processing unit that stores video components included in a 3D object video in a buffer corresponding to that video component based on a component index, and a display image generation unit that retrieves the video component from the buffer based on the component index and generates a display image using the retrieved video component. Multiple video components can be used simultaneously in a single 3D object. Furthermore, the component index is set for the video components included in the 3D object video in the extension for 3D object videos defined in the material layer of the scene description, and each video component has a different value.

[0072] Furthermore, in the information processing method (e.g., playback method), based on the component index, the video components included in the 3D object video are stored in the buffer corresponding to that video component, the video component is retrieved from the buffer based on the component index, and the display image is generated using the retrieved video component. Note that multiple video components can be used simultaneously in a single 3D object. The component index is set for the video components included in the 3D object video in the extension for 3D object video defined in the material layer of the scene description, and each video component has a different value.

[0073] By describing the scene description in this way, multiple video components can be associated with different accessors. Therefore, the media access function (MAF) of the playback device can store these video components in different buffers based on the scene description. This allows the playback device's presentation engine to select the desired video components and map them to a single object. In other words, multiple video components can be used simultaneously for a single object using the scene description.

[0074] <Method 1-1> Furthermore, when Method 1 is applied, an index may be set for each VD texture or packed VD texture, as shown in the second row from the top of the table in Figure 18 (Method 1-1). In other words, VD textures or packed VD textures may be applied as video components.

[0075] For example, in the case of VD textures, "using" means "mapping the VD texture." In other words, "mapping multiple VD textures to one mesh" is equivalent to "using multiple video components simultaneously." Similarly, in the case of packed VD textures, "using" means "mapping multiple VD textures that have been unpacked from a packed VD texture." In other words, "mapping multiple VD textures that have been unpacked from a packed VD texture to one mesh" is equivalent to "using multiple video components simultaneously."

[0076] For example, in the information processing device described above (e.g., a file generation device), the video component may be a VD texture which is an image captured from a predetermined viewpoint of a 3D object. Alternatively, the file generation unit may set a component index for each of the multiple VD textures included in the 3D object video within its extension.

[0077] Furthermore, in the aforementioned information processing device (e.g., a file generation device), the video component may be a packed VD texture, which is a VD texture containing multiple VD textures, each of which is an image of a 3D object captured from a predetermined viewpoint. Additionally, the file generation unit may set a component index for the packed VD texture in its extension.

[0078] For example, in the information processing device described above (e.g., a playback device), the video component may be a VD texture, which is an image captured from a predetermined viewpoint of a 3D object. Alternatively, the file processing unit may store multiple VD textures with different viewpoints in their respective buffers based on the component index. A display image generation unit may also obtain a desired VD texture from the buffer based on the component index. The display image generation unit may then map the obtained VD texture to a 3D object (mesh). Finally, the display image generation unit may generate a display image using the 3D data (the mesh to which the VD texture has been mapped).

[0079] Furthermore, in the above-described information processing device (e.g., a playback device), the video component may be a packed VD texture, which is a VD texture containing multiple VD textures, each of which is an image captured from a predetermined viewpoint of a 3D object. Alternatively, the file processing unit may store the packed VD texture in a buffer corresponding to that packed VD texture based on its component index. A display image generation unit may also retrieve the packed VD texture from its buffer based on its component index. The display image generation unit may then unpack a desired VD texture from the retrieved packed VD texture. The display image generation unit may then map the unpacked VD texture to a 3D object (mesh). Finally, the display image generation unit may generate a display image using the 3D data (the mesh to which the VD texture is mapped).

[0080] For example, the object configuration of the scene description may be as shown in the example in Figure 19. That is, an extension object "MPEG_video_components" for 3D object video may be provided in the primitives' material layer, and each VD texture and packed VD texture may be associated with a different accessor in that MPEG_video_components. Alternatively, the MPEG_video_components may be provided in the primitives themselves, rather than in the primitives' material layer.

[0081] To that end, the scene description may be written as shown in the example in Figure 20. In other words, an index (also called a component index) may be set for each VD texture and packed VD texture. As mentioned above, a component index is an index that has a different value for each video component. In other words, in the examples in Figures 19 and 20, the component index has a different value for each VD texture and packed VD texture. In the example in Figure 20, in material layer 111, as shown in the square frame 112, an index of "1" ("index":1) is set for one VD texture ("vdTexture") and an index of "2" ("index":2) is set for one packed VD texture ("packedVdTexture"). If there are multiple VD textures, a component index with a different value will be set for each VD texture.

[0082] By describing the scene description in this way, each VD texture and packed VD texture can be associated with a different accessor, as shown in the example in Figure 19. Therefore, the media access function (MAF) of the playback device can store each VD texture and packed VD texture in a different buffer based on its scene description. This allows the playback device's presentation engine to select the desired VD texture or packed VD texture and map it to the mesh. In other words, multiple VD textures and packed VD textures can be mapped to a single mesh simultaneously using the scene description.

[0083] Furthermore, a video component may consist only of VD textures and may not include packed VD textures. In other words, multiple VD textures associated with a single mesh may be applied as multiple video components. In that case, the object configuration in the scene description may be as shown in the example in Figure 21. That is, an extension object "MPEG_video_components" for 3D object video may be provided in the primitives' material layer, and each VD texture in that MPEG_video_components may be associated with a different accessor. Alternatively, MPEG_video_components may be provided in the primitives themselves, rather than in the primitives' material layer.

[0084] Furthermore, a video component may consist only of packed VD textures and may not include unpacked VD textures. In other words, a packed VD texture, in which multiple VD textures associated with a single mesh are packed, may be applied as multiple video components. In that case, the object configuration in the scene description may be as shown in the example in Figure 22. That is, an extension object "MPEG_video_components" for 3D object video may be provided in the primitives' material layer, and the packed VD textures may be associated with accessors in that MPEG_video_components. Alternatively, MPEG_video_components may be provided in the primitives themselves, rather than in the primitives' material layer.

[0085] <Method 1-2> Furthermore, when Method 1 is applied, an index may be set for each video component (geometry, attribute, occupancy) of the V-PCC (Video-based Point Cloud Compression), as shown in the third row from the top of the table in Figure 18 (Method 1-2). In other words, the geometry, attributes, and occupancy map of the V-PCC may be applied as video components.

[0086] A point cloud is 3D data that represents the 3D shape of an object using a large number of points. The data of a point cloud consists of the geometry and attributes of each point. The geometry indicates the position of the point in three-dimensional space (e.g., 3D coordinates). The attributes indicate the attribute information assigned to that point. This attribute information can be anything; for example, it may include color information, reflectivity information, normal information, etc.

[0087] V-PCC is one method for encoding point cloud data. In V-PCC, geometry and attributes are packed into a 2D plane patch by patch and encoded as a 2D image (using an encoding scheme for 2D images). V-PCC allows for the addition of occupancy maps to these geometries and attributes. An occupancy map is map information that indicates the extent to which geometry (and attributes) exist in a 2D image. Occupancy maps are generated for the 2D image in which geometry is packed (also referred to herein as a geometry map) and the 2D image in which attributes are packed (also referred to herein as an attribute map), and these occupancy maps are encoded using an encoding scheme for 2D images, similar to geometry maps and attribute maps. In other words, in the case of V-PCC, the geometry map, attribute map, and occupancy map each constitute a video component. The encoded data for the point cloud includes encoded data for the geometry map, encoded data for the attribute map, and encoded data for the occupancy map.

[0088] During decoding, the encoded data of the geometry map, attribute map, and occupancy map are decoded using encoding schemes for 2D images. Then, based on the occupancy map, the geometry (each patch) is unpacked from the geometry map, and the attributes (each patch) are unpacked from the attribute map. Finally, the point cloud is reconstructed using the unpacked geometry and attributes.

[0089] In other words, in the case of V-PCC, "using" means "reconstructing the point cloud." This reconstruction of the point cloud is done using geometry and attributes, as described above. Furthermore, an occupancy map is used to unpack that geometry and attributes. In other words, "the point cloud is reconstructed using a geometry map, attribute map, and occupancy map" can be said to mean "the geometry map, attribute map, and occupancy map are used simultaneously."

[0090] In recent years, when applying such V-PCC, it has been considered to treat the video components (geometry (geometry map), attributes (attribute map), and occupancy map) as video components and associate them with accessors in the material layer within the scene description. In this case, similar to the VD textures and packed VD textures mentioned above, a component index may be set for the video components, and each video component may be associated with a different accessor. In other words, different indices may be set for each of the geometry (geometry map), attributes (attribute map), and occupancy map, and each may be associated with a different accessor.

[0091] For example, in the information processing device described above (e.g., a file generation device), the video component may be a geometry map in which the geometry of the point cloud is packed, an attribute map in which the attributes of the point cloud are packed, and an occupancy map corresponding to the geometry map and the attribute map. Alternatively, the file generation unit may set component indices with different values ​​for each of the geometry map, attribute map, and occupancy map in its extension.

[0092] For example, in the information processing device (e.g., a playback device) described above, the video component may consist of a geometry map in which the geometry of the point cloud is packed, an attribute map in which the attributes of the point cloud are packed, and an occupancy map corresponding to the geometry map and the attribute map. Alternatively, the file processing unit may store the geometry map, attribute map, and occupancy map in their respective buffers based on the component index. A display image generation unit may also retrieve the geometry map, attribute map, and occupancy map from its buffers based on its component index. The display image generation unit may then reconstruct the point cloud using the retrieved geometry map, attribute map, and occupancy map. Finally, the display image generation unit may generate a display image using the reconstructed point cloud.

[0093] <Method 1-3> If Method 1 is applied and the VD texture is applied as a video component, the camera parameters of the VD texture may be stored as shown in the fourth row from the top of the table in Figure 18 (Method 1-3).

[0094] As mentioned above, a VD texture may be composed of captured images taken from a certain viewpoint. Camera parameters, which are parameters related to the camera used to generate these captured images, i.e., camera parameters corresponding to the VD texture, may be stored in the scene description. These camera parameters may be used, for example, when mapping the VD texture to a mesh.

[0095] For example, in the information processing device described above (e.g., a file generation device), the file generation unit may further store camera parameters corresponding to the VD texture in its extension for 3D object video.

[0096] For example, in the information processing device described above (e.g., a playback device), the file processing unit may further store camera parameters corresponding to the VD texture in a buffer corresponding to that VD texture. Alternatively, the display image generation unit may further obtain camera parameters corresponding to a desired VD texture from the buffer based on the component index. The display image generation unit may then use the obtained camera parameters to map the VD texture to a 3D object.

[0097] For example, as shown in Figure 23, the "MPEG_video_components" extension object for 3D object video defined in the material layer has camera parameters ("cameraParam") corresponding to the VD texture set. In other words, the camera parameters of this VD texture are stored in a buffer.

[0098] Figure 24 shows an example of how to describe these camera parameters in a scene description. As in this example, camera parameters may include parameters such as cam_id, intrinsic_param, Rs, Ts, and distortion. cam_id is a camera identifier used to identify the camera. intrinsic_param indicates the intrinsic parameters of the camera. Rs and Ts indicate the extrinsic parameters of the camera. For example, Rs indicates the pose of the camera (viewpoint), and Ts indicates the position of the camera (viewpoint). Distortion is a parameter that indicates the output vector of the distortion coefficient. Of course, the contents of the parameters included in the camera parameters stored in the scene description are arbitrary. For example, parameters other than those in the example above may be included, or some of the parameters in the example above may be omitted.

[0099] The referenced accessor has MPEG_accessor_timed. Setting immutable=TRUE allows access to static data extracted from MPEG media and stored in a buffer.

[0100] As shown in Figure 23, each VD texture has an accessor index assigned to its camera parameters ("cameraParam"). In other words, the camera parameters are stored in different buffers for each VD texture.

[0101] In this way, by storing the camera parameters corresponding to the VD texture (and the camera that generated it) in the scene description, the playback device can more easily map the VD texture to the mesh based on those camera parameters.

[0102] <Method 1-3-1> Furthermore, when applying Method 1-3, an index may be set for each field of the camera parameters, as shown in the fifth row from the top of the table in Figure 18 (Method 1-3-1). In other words, an index (also called a field index) with a different value for each field of the camera parameters may be set.

[0103] For example, in the information processing device described above (e.g., a file generation device), the file generation unit may further set a field index with a different value for each field of the camera parameters in its extension.

[0104] For example, in the information processing device described above (e.g., a playback device), the file processing unit may store each field of the camera parameters in its corresponding buffer based on the field index. Furthermore, the display image generation unit may retrieve each field of the camera parameters corresponding to a desired VD texture from the buffer based on the field index. The display image generation unit may then use the retrieved camera parameters to map the VD texture to a 3D object. Note that the field index is set for each field of the camera parameters in its extension and has a different value for each field.

[0105] In the example in Figure 25, the camera identifier `cameraId` is assigned an index value of "100". The internal parameter `intrinsicParam` is assigned an index value of "101". The external parameter `Rs` is assigned an index value of "102". The external parameter `Ts` is assigned an index value of "103". The output vector `distortion`, representing the distortion coefficient, is assigned an index value of "104". In other words, in the scene description, different index values ​​are assigned to each field of the camera parameters. This allows the media access function of the playback device to store these camera parameters in separate buffers for each field, based on this scene description. Therefore, the presentation engine of the playback device can easily obtain the desired field of the camera parameters by selecting a buffer based on this scene description.

[0106] <Method 1-4> Alternatively, if Method 1 is applied and a packed VD texture is applied as a video component, the camera parameters and packing metadata of the packed VD texture may be stored as shown in the sixth row from the top of the table in Figure 18 (Method 1-4).

[0107] As mentioned above, a packed VD texture contains multiple VD textures. Camera parameters, which are parameters related to the imaging (camera) used to generate each VD texture (image), that is, camera parameters corresponding to each VD texture packed into the packed VD texture, may be stored in the scene description. These camera parameters may be used, for example, when mapping the unpacked VD textures to a mesh.

[0108] Additionally, packing metadata indicating where the VD texture is packed within the packed VD texture may be stored in the scene description. This packing metadata may be used, for example, to unpack the VD texture from the packed VD texture.

[0109] For example, in the information processing device described above (e.g., a file generation device), the file generation unit may further store camera parameters and packing metadata corresponding to the packed VD texture in its extension.

[0110] For example, in the information processing device (e.g., playback device) described above, the file processing unit may further store camera parameters and packing metadata corresponding to the packed VD texture in a buffer corresponding to the packed VD texture. Alternatively, the display image generation unit may further obtain camera parameters and packing metadata corresponding to the packed VD texture from the buffer based on the component index. The display image generation unit may then unpack the desired VD texture from the packed VD texture based on the obtained packing metadata. The display image generation unit may then map the VD texture to a 3D object using the obtained camera parameters.

[0111] For example, as shown in Figure 26, the "MPEG_video_components" extension object for 3D object video defined in the material layer has camera parameters ("packedCameraParam") and packing metadata ("packedMeta") corresponding to the packed VD texture set. In other words, the camera parameters and packing metadata of this packed VD texture are stored in a buffer.

[0112] Figure 27 shows an example of how to describe these camera parameters in a scene description. In this case as well, the content of the parameters included in the camera parameters is arbitrary. For example, parameters such as cam_id, intrinsic_param, Rs, Ts, and distortion may be included. However, since a packed VD texture contains multiple VD textures, the camera parameters corresponding to a packed VD texture include the camera parameters for each of those multiple VD textures. Therefore, in the example in Figure 27, unlike the example in Figure 24, a for loop is used to show each parameter for all cameras (VD textures). Note that the parameter camera_num indicates the number of cameras (VD textures).

[0113] Figure 28 shows an example of how packing metadata is described in a scene description. Packing metadata includes information such as where each VD texture is packed within the packed VD texture. In other words, similar to the camera parameters shown in Figure 27, the packing metadata is written using a for loop so that each parameter is shown for all cameras (VD textures). The parameter camera_num indicates the number of cameras (VD textures).

[0114] The content of the parameters included in the packing metadata is arbitrary. For example, it may include parameters such as cam_id, src_width[i], src_height[i], src_top[i], src_left[i], transform_type[i], dst_width[i], dst_height[i], dst_top[i], and dst_left[i]. cam_id is a camera identifier used to identify the camera. src_width[i] is a parameter indicating the width (horizontal length) of the corresponding VD texture. src_height[i] is a parameter indicating the height (vertical length) of the corresponding VD texture. src_top[i] is a parameter indicating the position of the top edge of the corresponding VD texture. src_left[i] is a parameter indicating the position of the left edge of the corresponding VD texture. These parameters represent the values ​​after unpacking from the packed VD texture. Furthermore, transform_type[i] is a parameter that indicates the placement information (rotation, etc.) of the packed VD texture.

[0115] dst_width[i] is a parameter that indicates the width (horizontal length) of the corresponding VD texture. dst_height[i] is a parameter that indicates the height (vertical length) of the corresponding VD texture. dst_top[i] is a parameter that indicates the position of the top edge of the corresponding VD texture. dst_left[i] is a parameter that indicates the position of the left edge of the corresponding VD texture. These parameters represent the values ​​when packed into a packed VD texture.

[0116] In other words, the packing metadata indicates which parts of the VD texture are stored where within the packed VD texture. Of course, the parameters included in the packing metadata stored in the scene description are arbitrary. For example, it may include parameters other than those in the example above, or some of the parameters in the example above may be omitted.

[0117] Note that camera parameters and packing metadata are assigned different accessor indices. In the example in Figure 26, the camera parameter ("packedCameraParam") is assigned an index value of "200," and the packing metadata ("packedMeta") is assigned an index value of "201." In other words, the camera parameter and packing metadata are stored in different buffers. The camera parameter is stored in a static buffer, and the packing metadata is stored in a circular buffer. Also, as shown in Figure 27, the camera parameters for all VD textures packed into a packed VD texture are stored together in a single buffer. Similarly, as shown in Figure 28, the packing metadata for all VD textures packed into a packed VD texture is stored together in a single buffer.

[0118] In this way, by storing packing metadata corresponding to the packed VD texture in the scene description, the playback device can more easily unpack the VD texture based on that packing metadata. Furthermore, by storing camera parameters corresponding to the packed VD texture in the scene description, the playback device can more easily map the VD texture to the mesh based on those camera parameters.

[0119] <Method 1-4-1> Furthermore, when applying Method 1-4, an index may be set for each VD texture packed into the packed VD texture, as shown in the seventh row from the top of the table in Figure 18 (Method 1-4-1). In other words, for each camera parameter and packing metadata packed into the packed VD texture, an index (also called a VD texture index) with a different value for each corresponding VD texture may be set.

[0120] For example, in the information processing device described above (e.g., a file generation device), the file generation unit may further set a VD texture index with a different value for each VD texture for each camera parameter and packing metadata packed into the packed VD texture, in its extension.

[0121] For example, in the information processing device (e.g., playback device) described above, the file processing unit may store the camera parameters and packing metadata of each VD texture packed into a packed VD texture in corresponding buffers, based on the VD texture index. The display image generation unit may further obtain the camera parameters and packing metadata corresponding to a desired VD texture from the buffer based on its VD texture index. The display image generation unit may then unpack the desired VD texture from the packed VD texture based on the obtained packing metadata. The display image generation unit may then map the VD texture to a 3D object using the obtained camera parameters. The VD texture index is set for the camera parameters and packing metadata of each VD texture in the extension described above, and has a different value for each corresponding VD texture.

[0122] In the example in Figure 29, the camera parameter ("packedCameraParam_1") corresponding to the first VD texture packed in the packed VD texture has an index value of "100". Similarly, the camera parameter ("packedCameraParam_2") corresponding to the second VD texture packed in the packed VD texture has an index value of "200". In other words, a VD texture index is set for each camera parameter.

[0123] In this example, as mentioned above, the camera parameters are separated for each VD texture. Therefore, the method of describing the camera parameters is independent for each VD texture, as shown in Figure 30, for example. As a result, a for loop using camera_num, as in the example in Figure 27, is unnecessary. Also, since names that can identify the corresponding VD texture are used as names for the camera parameters, such as "packedCameraParam_1" and "packedCameraParam_2", cam_id is also unnecessary.

[0124] In this way, the media access function of the playback device can store these camera parameters in separate buffers for each corresponding VD texture based on this scene description. Therefore, the presentation engine of the playback device can easily obtain the camera parameters for the desired VD texture by selecting a buffer based on this scene description.

[0125] Similarly, the packing metadata ("packedMeta_1") corresponding to the first VD texture packed into the packed VD texture has an index value of "101". Also, the packing metadata ("packedMeta_2") corresponding to the second VD texture packed into the packed VD texture has an index value of "201". In other words, a VD texture index is set for each packing metadata.

[0126] In this example, as mentioned above, the packing metadata is separated for each VD texture. Therefore, the method of describing the packing metadata is independent for each VD texture, as shown in Figure 31, for example. As a result, a for loop using camera_num, as in the example in Figure 28, is unnecessary. Also, since names that can identify the corresponding VD texture are used as camera parameter names, such as "packedMeta_1" and "packedMeta_2", cam_id is also unnecessary.

[0127] In this way, the media access function of the playback device can store this packing metadata in separate buffers for each corresponding VD texture based on this scene description. Therefore, the presentation engine of the playback device can easily retrieve the packing metadata for the desired VD texture by selecting a buffer based on this scene description.

[0128] <Method 1-4-2> Furthermore, when applying Method 1-4, an index may be set for each field of camera parameters and packing metadata, as shown in the eighth row from the top of the table in Figure 18 (Method 1-4-2). In other words, for each field of camera parameters and packing metadata, an index (also called a field index) with a different value for each field may be set, similar to the example in Figure 25.

[0129] For example, in the information processing device described above (e.g., a file generation device), the file generation unit may further set a field index with a different value for each field of camera parameters and packing metadata in its extension.

[0130] For example, in the information processing device described above (e.g., a playback device), the file processing unit may store each field of camera parameters and packing metadata in corresponding buffers based on the field index. Furthermore, the display image generation unit may retrieve the fields of camera parameters and packing metadata corresponding to the desired VD texture from the buffers based on the field index. The display image generation unit may then unpack the desired VD texture from the packed VD texture based on the retrieved packing metadata. Finally, the display image generation unit may map the VD texture to a 3D object using the retrieved camera parameters. Note that the field index is set for the fields of camera parameters and packing metadata in the extension, and each field has a different value.

[0131] In this way, the media access function of the playback device can store these camera parameters and packing metadata in separate buffers for each field, based on this scene description. Therefore, the presentation engine of the playback device can easily retrieve the desired fields of camera parameters and packing metadata by selecting the buffer based on this scene description.

[0132] Furthermore, methods 1-4-1 and 1-4-2 described above may be applied in combination. In other words, different index values ​​may be assigned to each corresponding VD texture and field for the camera parameters and packing metadata.

[0133] In the example in Figure 32, the index value "101" is set for the internal parameter "instrinsicParam_1" included in the camera parameters corresponding to the VD texture of the first eye, which is packed into the packed VD texture. Similarly, the index value "102" is set for the external parameter "Rs_1" corresponding to the VD texture of the first eye. Similarly, the index value "103" is set for the external parameter "Ts_1" corresponding to the VD texture of the first eye. Similarly, the index value "104" is set for the distortion coefficient output vector "distortion_1" corresponding to the VD texture of the first eye.

[0134] Additionally, the index value "201" is set for "src_video_1", which is included in the packing metadata corresponding to the first eye's VD texture packed into the packed VD texture, and contains information about the VD texture in its unpacked state from the packed VD texture. Similarly, the index value "202" is set for "transformType_1", which is the placement information of the packed VD texture corresponding to that first eye's VD texture. Similarly, the index value "203" is set for "dst_video_1", which contains information about the VD texture in its packed state within the packed VD texture corresponding to that first eye's VD texture.

[0135] In this way, the media access function of the playback device can store these camera parameters and packing metadata in separate buffers, one for each corresponding VD texture and one for each field, based on this scene description. Therefore, the presentation engine of the playback device can easily obtain the desired fields of camera parameters and packing metadata corresponding to the desired VD texture by selecting a buffer based on this scene description.

[0136] In the example shown in Figure 32, the accessors for each parameter type of VD texture captured by the same camera are paired by name (index_X). For example, the presentation engine of the playback device can obtain the camera parameters and packing metadata corresponding to the first VD texture packed into the packed VD texture by obtaining the field with the name "XXX_1" (where XXX is any string). In other words, the presentation engine of the playback device can easily obtain the camera parameters and packing metadata for each corresponding VD texture.

[0137] <Method 1-5> Alternatively, if Method 1 is applied and a packed VD texture is applied as a video component, the Media Access Function (MAF) may unpack the VD texture from the packed VD texture and store it in a buffer, as shown in the ninth row from the top of the table in Figure 18 (Method 1-5).

[0138] In this case, the file generator does not set the packed VD texture object in the material layer of the scene description. The media access function of the playback device generates a VD texture from the packed VD texture and stores that VD texture in the circular buffer referenced by the VD texture object in the material layer of the scene description. The presentation engine (PE) can then process it in the same way as when the VD texture is applied as a video component.

[0139] For example, if the scene description is configured to reference the same packed VD texture from multiple VD textures, the media access function of the playback device may perform the processing as described above. In other words, the media access function generates a VD texture from the packed VD texture and stores that VD texture in the circular buffer referenced by the VD texture object in the material layer of the scene description.

[0140] <Method 1-6> Alternatively, if Method 1 is applied and a packed VD texture is applied as a video component, the Media Access Function (MAF) may unpack the VD texture from the packed VD texture, map that VD texture to a 3D object (mesh), generate a VI texture (UV texture map), and store it in a buffer, as shown in the bottom row of the table in Figure 18 (Method 1-6).

[0141] In this case, the file generator does not set a packed VD texture object or a VD texture object in the material layer of the scene description. The media access function of the playback device generates a VD texture from the packed VD texture, maps that VD texture to a 3D object (mesh), generates a VI texture (UV texture map), and stores that UV texture map in the circular buffer referenced by the "MPEG_media" object. The presentation engine (PE) should process it in the same way as when a VI texture (UV texture map) is applied as a video component.

[0142] For example, if the material layer of the scene description is configured to reference a VD texture or packed VD texture from an object other than a VD texture, the media access function of the playback device may be configured to perform the processing as described above. The fact that an object other than a VD texture is configured to reference a VD texture or packed VD texture may also be indicated by the track.codecs of the "MPEG_media" object. For example, the scheme_type of the ISOBMFF that stores the VD texture may be set to 'vdte', and the scheme_type of the ISOBMFF that stores the packed VD texture may be set to 'pvdt'. In this case, codecs=resv.vdte.*** indicates that a VD texture is stored, and codecs=resv.pvdt.*** indicates that a packed VD texture is stored.

[0143] For example, if the scene description is set as shown in Figure 33, then codecs=resv.vdte.***, and the VD texture is stored in "vd_tex.mp4". In other words, the VD texture is referenced from an object other than the VD texture, "MPEG_media". Therefore, the media access function of the playback device processes as described above. That is, the media access function generates a VD texture from the packed VD texture, maps that VD texture to a 3D object (mesh), generates a VI texture (UV texture map), and stores that UV texture map in the circular buffer referenced by the "MPEG_media" object.

[0144] Each of the methods described above may be combined with other methods as appropriate.

[0145] <4. First Embodiment> <File Generator> The technology described above can be applied to any device. Figure 34 is a block diagram showing an example of the configuration of a file generation device, which is one embodiment of an information processing device to which this technology is applied. The file generation device 300 shown in Figure 34 is a device that encodes 3D object content (e.g., 3D data such as a point cloud) and stores it in a file container such as ISOBMFF. The file generation device 300 also generates a scene description file for the 3D object content.

[0146] Note that Figure 34 shows the main components such as the processing unit and data flow, and does not necessarily represent everything. In other words, the file generation device 300 may have processing units that are not shown as blocks in Figure 34, or processes and data flows that are not shown as arrows or other symbols in Figure 34.

[0147] As shown in Figure 34, the file generation device 300 has a control unit 301 and a file generation processing unit 302. The control unit 301 controls the file generation processing unit 302. The file generation processing unit 302 is controlled by the control unit 301 and performs processing related to file generation. For example, the file generation processing unit 302 may acquire data of 3D object content to be stored in the file. Alternatively, the file generation processing unit 302 may generate a content file by storing the acquired 3D object content data in a file container. The file generation processing unit 302 may also generate a scene description corresponding to the 3D object content and store it in a scene description file. The file generation processing unit 302 may output the generated file to an external location. For example, the file generation processing unit 302 may upload the generated file to a distribution server or the like.

[0148] The file generation processing unit 302 includes an input unit 311, a preprocessing unit 312, an encoding unit 313, a file generation unit 314, a recording unit 315, and an output unit 316.

[0149] The input unit 311 performs processing related to the acquisition of 3D object content data (3D data representing the three-dimensional structure of an object). For example, the input unit 311 may acquire a 3D object video (dynamic 3D object) containing multiple video components from outside the file generation device 300 as 3D object content data. For example, the video components may be VD textures, packed VD textures, or both. For example, the input unit 311 may acquire a dynamic mesh and multiple VD texture videos (dynamic VD textures). Alternatively, the input unit 311 may acquire a dynamic mesh, multiple VD texture videos, and packed VD texture videos (dynamic packed VD textures). The video components may also be V-PCC video components (geometry, attributes, occupancy maps). Of course, the video components can be any data that can be used simultaneously in multiples within a single 3D object, and are not limited to these examples. The input unit 311 may supply the acquired 3D object content data to the preprocessing unit 312.

[0150] The preprocessing unit 312 performs preprocessing on the 3D object content data before encoding. For example, the preprocessing unit 312 may acquire the 3D object content data supplied from the input unit 311. The preprocessing unit 312 may also acquire information necessary for generating a scene description from the acquired 3D object content data. The preprocessing unit 312 may also supply the acquired information to the file generation unit 314. The preprocessing unit 312 may also supply the 3D object content data to the encoding unit 313.

[0151] The encoding unit 313 performs processing related to the encoding of 3D object content data. For example, the encoding unit 313 may acquire 3D object content data supplied from the preprocessing unit 312. Alternatively, the encoding unit 313 may encode the acquired 3D object content data and generate the encoded data.

[0152] For example, if a dynamic mesh and multiple VD texture videos are supplied to the encoding unit 313, each may be encoded separately. In this case, the encoding unit 313 encodes the VD texture videos using an encoding method for 2D images. If a packed VD texture video is supplied, the encoding unit 313 may encode that packed VD texture video. In this case, the encoding unit 313 encodes the packed VD texture video using an encoding method for 2D images.

[0153] The encoding unit 313 may also supply the encoded data of the generated 3D object content to the file generation unit 314.

[0154] The file generation unit 314 performs processing related to the generation of files, etc. For example, the file generation unit 314 may acquire encoded data of 3D object content supplied from the encoding unit 313. The file generation unit 314 may also acquire information supplied from the preprocessing unit 312. Furthermore, the file generation unit 314 may generate a file container (content file) that stores the encoded data of 3D object content supplied from the encoding unit 313. The specifications of this content file (file container) are arbitrary, and it can be any file as long as it can store the encoded data of 3D object content. For example, it may be ISOBMFF.

[0155] Furthermore, the file generation unit 314 may use the information supplied from the preprocessing unit 312 to generate a scene description corresponding to the encoded data of the 3D object content. The file generation unit 314 may then generate a scene description file and store the generated scene description. The file generation unit 314 may also supply the generated files (such as ISOBMFF or the scene description file) to the recording unit 315.

[0156] The recording unit 315 has any recording medium, such as a hard disk or semiconductor memory, and performs processing related to data recording. For example, the recording unit 315 may record files supplied from the file generation unit 314 onto its recording medium. Alternatively, the recording unit 315 may read files recorded on the recording medium and supply them to the output unit 316 in accordance with a request from the control unit 301 or the output unit 316, or at a predetermined timing.

[0157] The output unit 316 may acquire files, etc., supplied from the recording unit 315 and output those files, etc., to an external device (for example, a distribution server or playback device) outside the file generation device 300.

[0158] In a file generation device 300 with the above configuration, the above-described technology may be applied in <3. Scene description corresponding to multiple video components>.

[0159] For example, by applying Method 1, the file generation unit 314 may generate a scene description file, and in the extension for 3D object video defined in the material layer of that scene description file, a component index with a different value may be set for each video component included in the 3D object video. Note that multiple such video components can be used simultaneously in a single 3D object.

[0160] Alternatively, when Method 1 is applicable, Method 1-1 may be applied, and the video component may be a VD texture which is an image captured from a predetermined viewpoint of a 3D object. The file generation unit 314 may then set a component index for each of the multiple VD textures included in the 3D object video in the extension for 3D object video described above.

[0161] Furthermore, if Method 1 is applied and a VD texture is applied as a component, Method 1-3 may be applied, and the file generation unit 314 may further store the camera parameters corresponding to the VD texture in the extension for 3D object video described above.

[0162] Furthermore, when Method 1-3 is applied, Method 1-3-1 may be applied, and the file generation unit 314 may further set a field index with a different value for each field of the camera parameters in the extension for 3D object video described above.

[0163] Furthermore, when Method 1 is applicable, Method 1-1 may be applied to make the video component a packed VD texture, which is a pack of multiple VD textures, each of which is an image of a 3D object captured from a predetermined viewpoint. The file generation unit 314 may then set a component index for the packed VD texture in the extension for 3D object video described above.

[0164] Furthermore, if Method 1 is applied and a packed VD texture is applied as a component, Method 1-4 may be applied, and the file generation unit 314 may further store camera parameters and packing metadata corresponding to the packed VD texture in the extension for 3D object video described above.

[0165] Furthermore, when Method 1-4 is applied, Method 1-4-1 may be applied, and the file generation unit 314 may further set a VD texture index with a different value for each corresponding VD texture for the camera parameters and packing metadata of each VD texture packed into the packed VD texture in the extension for 3D object video described above.

[0166] Furthermore, when Method 1-4 is applied, Method 1-4-2 may be applied, and the file generation unit 314 may further set a field index with a different value for each field in the camera parameter and packing metadata fields of the 3D object video extension described above.

[0167] Alternatively, when Method 1 is applicable, Method 1-2 may be applied to make the video component a geometry map in which the geometry of the point cloud is packed, an attribute map in which the attributes of the point cloud are packed, and an occupancy map corresponding to the geometry map and the attribute map. The file generation unit 314 may then set component indices for the geometry map, attribute map, and occupancy map, each having different values ​​from each other, in the extension for 3D object video described above.

[0168] Of course, other aspects of this technique described above may also be applied in <3. Scene Descriptions Supporting Multiple Video Components>. Furthermore, multiple aspects of this technique may be combined and applied as appropriate.

[0169] The file generation device 300, with this configuration, can generate a scene description that associates multiple video components, which can be used simultaneously for a single object, with different accessors. Therefore, the playback device can use the scene description to simultaneously utilize multiple video components for a single object.

[0170] <File generation process flow> An example of the file generation process flow performed by a file generation device 300 with this configuration using the method 1 described above will be explained with reference to the flowchart in Figure 35.

[0171] When the file generation process starts, the input unit 311 of the file generation device 300 acquires a 3D object video (dynamic 3D data) in step S301. For example, the input unit 311 acquires a 3D object video containing multiple video components as this 3D data.

[0172] In step S302, the preprocessing unit 312 performs preprocessing on the 3D object video. For example, the preprocessing unit 312 obtains from the 3D object video information used to generate a scene description, which is spatial arrangement information for placing one or more 3D objects in 3D space.

[0173] In step S303, the file generation unit 314 and the encoding unit 313 encode the 3D object video acquired in step S301 and generate the encoded data.

[0174] In step S304, the file generation unit 314 generates a content file (e.g., ISOBMFF) to store the encoded data.

[0175] In step S305, the file generation unit 314 generates a scene description file. This scene description file stores a scene description in which the 3D objects represented by the 3D object video acquired in step S301 are placed in 3D space.

[0176] In step S306, the file generation unit 314 sets reference information to each buffer in the scene description where the data constituting the 3D object video is stored. This 3D object video consists of a dynamic mesh, multiple video components, and metadata, etc. The file generation unit 314 sets reference information to each buffer in the scene description so that this configuration data (dynamic mesh, each video component, metadata) is stored in different buffers. For example, the file generation unit 314 may apply Method 1 and set a component index with a different value for each video component in the extension for the 3D object video defined in the material layer of the scene description file. Note that multiple video components can be used simultaneously in a single 3D object.

[0177] In step S307, the recording unit 315 records the generated scene description file and content file onto the recording medium. The output unit 316 reads the scene description file and content file from the recording medium and outputs the read files to the outside of the file generation device 300 at a predetermined timing. For example, the output unit 316 may transmit (upload) the scene description file and content file to other devices such as a distribution server or playback device via a communication medium such as a network. Alternatively, the output unit 316 may record the scene description file and content file onto an external recording medium such as removable media. In that case, the output files may be supplied to other devices (such as a distribution server or playback device) via the external recording medium.

[0178] Once step S307 is completed, the file generation process ends.

[0179] By performing the file generation process in this manner, the file generation device 300 can generate a scene description that associates multiple video components, which can be used simultaneously for a single object, with different accessors. Therefore, the playback device can use the scene description to simultaneously utilize multiple video components for a single object.

[0180] Alternatively, by applying Method 1-1, the video component may be a VD texture, which is an image captured from a predetermined viewpoint of a 3D object. Then, in step S306, the file generation unit 314 may set a component index for each of the multiple VD textures included in the 3D object video in the extension for 3D object video described above.

[0181] Furthermore, if method 1-1 is applicable, method 1-3 may be applied, and in step S306, the file generation unit 314 may further store the camera parameters corresponding to the VD texture in the extension for 3D object video described above.

[0182] Furthermore, if method 1-3 is applicable, method 1-3-1 may be applied, and in step S306, the file generation unit 314 may further set a field index with a different value for each field of the camera parameters in the extension for 3D object video described above.

[0183] Alternatively, by applying Method 1-1, the video component may be a packed VD texture in which multiple VD textures, which are captured images of a 3D object taken from a predetermined viewpoint, are packed together. Then, in step S306, the file generation unit 314 may set a component index for the packed VD texture in the extension for 3D object video described above.

[0184] Furthermore, if method 1-1 is applicable, method 1-4 may be applied, and in step S306, the file generation unit 314 may further store camera parameters and packing metadata corresponding to the packed VD texture in the extension for 3D object video described above.

[0185] Furthermore, if method 1-4 is applicable, method 1-4-1 may be applied, and in step S306, the file generation unit 314 may further set a VD texture index with a different value for each corresponding VD texture for the camera parameters and packing metadata of each VD texture packed into the packed VD texture in the extension for 3D object video described above.

[0186] Furthermore, if Method 1-4 is applicable, Method 1-4-2 may be applied, and in step S306, the file generation unit 314 may further set a field index with a different value for each field of the camera parameters and packing metadata in the extension for 3D object video described above.

[0187] Alternatively, by applying Method 1-2, the video component may be a geometry map in which the geometry of the point cloud is packed, an attribute map in which the attributes of the point cloud are packed, and an occupancy map corresponding to the geometry map and the attribute map. Then, in step S306, the file generation unit 314 may set component indices for the geometry map, attribute map, and occupancy map, each having different values ​​from each other, in the extension for 3D object video described above.

[0188] Of course, other aspects of this technique described above may also be applied in <3. Scene Descriptions Supporting Multiple Video Components>. Furthermore, multiple aspects of this technique may be combined and applied as appropriate.

[0189] <5. Second Embodiment> <Client device> Figure 36 is a block diagram showing an example of the configuration of a client device, which is one embodiment of an information processing device to which this technology is applied. The client device 400 shown in Figure 36 is a playback device that performs playback processing of 3D object content based on a scene description. For example, the client device 400 plays back 3D object data stored in a content file generated by the file generation device 300. In doing so, the client device 400 performs processing related to the playback based on the scene description.

[0190] Note that Figure 36 shows the main components such as the processing unit and data flow, and does not necessarily represent an exhaustive list of all components. In other words, the client device 400 may have processing units that are not shown as blocks in Figure 36, or processing and data flows that are not shown as arrows or other symbols in Figure 36.

[0191] As shown in Figure 36, the client device 400 has a control unit 401 and a playback processing unit 402. The control unit 401 performs processing related to the control of the playback processing unit 402. The playback processing unit 402 performs processing related to the playback of 3D object data.

[0192] The playback processing unit 402 includes a file acquisition unit 411, a file processing unit 412, a decoding unit 413, a display information generation unit 414, a display unit 415, and a display control unit 416.

[0193] The file acquisition unit 411 performs processing related to file acquisition. For example, the file acquisition unit 411 may acquire files supplied from outside the client device 400, such as a distribution server or a file generation device 300. The file acquisition unit 411 may also acquire files stored in local storage (not shown). For example, the file acquisition unit 411 may acquire a scene description file. The file acquisition unit 411 may also acquire a content file. For example, the file acquisition unit 411 may supply the acquired file to the file processing unit 412. The file acquisition unit 411 may perform processing related to the acquisition of this file in accordance with the control of the file processing unit 412. For example, the file acquisition unit 411 may acquire a file requested by the file processing unit 412 from an external source or local storage and supply it to the file processing unit 412.

[0194] The file processing unit 412 performs processing related to files and the like. For example, the file processing unit 412 may have the functionality of the media access function (MAF) 52 as described with reference to Figure 7, and may perform the processing described as the processing performed by the media access function 52.

[0195] For example, the file processing unit 412 may control the file acquisition unit 411 in accordance with the control of the display information generation unit 414 to acquire a scene description file from outside the client device 400 or from local storage, etc. The file processing unit 412 may also supply the scene description file to the display information generation unit 414. Furthermore, the file processing unit 412 may control the file acquisition unit 411 in accordance with the control of the display information generation unit 414 to acquire data stored in a content file outside the client device 400 or from local storage, etc. (for example, encoded data of a 3D object video, etc.).

[0196] Alternatively, the file processing unit 412 may supply the encoded data, such as the 3D object video, to the decoding unit 413 for decoding and store the obtained data in a buffer (corresponding to buffer 54 in Figure 7) not shown. In other words, it can be said that the file processing unit 412 effectively stores this data in a buffer (corresponding to buffer 54 in Figure 7) not shown.

[0197] Furthermore, the file processing unit 412 may supply information related to image display (e.g., scene description) to the display control unit 416.

[0198] The decoding unit 413 performs decoding-related processing. For example, the decoding unit 413 may have the function of pipeline 53 (decoder) as described with reference to Figure 7, and may perform the processing described as the processing performed by pipeline 53 (decoder).

[0199] For example, the decoding unit 413 may decode encoded data such as 3D object video supplied from the file processing unit 412 in accordance with the control of the file processing unit 412 (media access function 52). Alternatively, the decoding unit 413 may store the data obtained by the decoding (video components, etc.) in a buffer (corresponding to buffer 54 in Figure 7) in accordance with the control of the file processing unit 412 (media access function 52). In this case, the decoding unit 413 stores the data obtained by the decoding (video components, etc.) in a buffer associated with that data by the scene description.

[0200] The display information generation unit 414 performs processing related to the generation of display images in accordance with the control of the display control unit 416. Therefore, the display information generation unit 414 can also be called a display image generation unit. For example, the display information generation unit 414 may have the functions of the presentation engine (PE) 51, as described with reference to Figure 7, and may perform the processing described as the processing performed by the presentation engine 51.

[0201] For example, the display information generation unit 414 may control the file processing unit 412 (media access function 52) to obtain a scene description file. Alternatively, the display information generation unit 414 may control the file processing unit 412 (media access function 52) based on the scene description stored in the scene description file to obtain desired data (for example, encoded data for a 3D object video) to be stored in the content file.

[0202] Furthermore, the display information generation unit 414 may acquire desired data from an unillustrated buffer (corresponding to buffer 54 in Figure 7) based on the scene description. The display information generation unit 414 may also reconstruct a 3D object video (3D data) using the acquired data. The display information generation unit 414 may also perform rendering using the 3D object video to generate a display image. This display image is a 2D image of the 3D object as seen from a specified viewpoint. The display information generation unit 414 may also supply the generated display information to the display unit 415 for display.

[0203] The display unit 415 has a display device and performs processing related to image display. For example, the display unit 415 may acquire display information supplied from the display information generation unit 414 and display it using its display device.

[0204] The display control unit 416 performs processing related to the control of image display. For example, the display control unit 416 may acquire information such as a scene description supplied from the file processing unit 412. The display control unit 416 may also control the display information generation unit 414 based on that information.

[0205] In a client device 400 configured as described above, the above-described technology may be applied in <3. Scene Description for Multiple Video Components>.

[0206] For example, by applying Method 1, the file processing unit 412 may store the video components included in the 3D object video in a buffer corresponding to the video component, based on the component index. Alternatively, the display information generation unit 414 may retrieve the video components from the buffer based on the component index and generate a display image using the retrieved video components.

[0207] Alternatively, when Method 1 is applicable, Method 1-1 may be applied to make the video component a VD texture, which is an image captured from a predetermined viewpoint of a 3D object. The file processing unit 412 may then store multiple VD textures with different viewpoints in their respective buffers based on the component index. The display information generation unit 414 may also obtain a desired VD texture from the buffer based on its component index. The display information generation unit 414 may then map the obtained VD texture to a 3D object (mesh). The display information generation unit 414 may then generate a display image using the 3D data (the mesh to which the VD texture is mapped).

[0208] Furthermore, when Method 1 is applied and a VD texture is applied as a component, Method 1-3 may be applied, and the file processing unit 412 may further store the camera parameters corresponding to the VD texture in the buffer corresponding to that VD texture. The display information generation unit 414 may further obtain the camera parameters corresponding to the desired VD texture from the buffer based on the component index. The display information generation unit 414 may then use the obtained camera parameters to map the VD texture to a 3D object.

[0209] Furthermore, when Method 1-3 is applied, Method 1-3-1 may be applied, and the file processing unit 412 may store each field of the camera parameters in its corresponding buffer based on the field index. The display information generation unit 414 may also further obtain each field of the camera parameters corresponding to the desired VD texture from the buffer based on its field index. The display information generation unit 414 may then map the VD texture to a 3D object using the obtained camera parameters. The field index is set for the fields of the camera parameters in the 3D object video extension described above, and each field has a different value.

[0210] Furthermore, when Method 1 is applicable, Method 1-1 may be applied to make the video component a packed VD texture, which is a VD texture in which multiple VD textures, which are captured images of a 3D object taken from a predetermined viewpoint, are packed together. Alternatively, the file processing unit 412 may store the packed VD texture in a buffer corresponding to that packed VD texture based on the component index. Alternatively, the display information generation unit 414 may obtain the packed VD texture from the buffer based on its component index. The display information generation unit 414 may then unpack the desired VD texture from the obtained packed VD texture. The display information generation unit 414 may then map the unpacked VD texture to a 3D object. The display information generation unit 414 may then generate a display image using the 3D data (mesh to which the VD texture is mapped).

[0211] Furthermore, when Method 1 is applied and a packed VD texture is applied as a component, Method 1-4 may be applied, and the file processing unit 412 may further store the camera parameters and packing metadata corresponding to the packed VD texture in the buffer corresponding to the packed VD texture. The display information generation unit 414 may further obtain the camera parameters and packing metadata corresponding to the packed VD texture from the buffer based on the component index. The display information generation unit 414 may then unpack the desired VD texture from the packed VD texture based on the obtained packing metadata. The display information generation unit 414 may then map the VD texture to a 3D object using the obtained camera parameters.

[0212] Furthermore, when Method 1-4 is applied, Method 1-4-1 may be applied, and the file processing unit 412 may store the camera parameters and packing metadata of each VD texture packed into the packed VD texture in the corresponding buffer, based on the VD texture index. The display information generation unit 414 may also obtain the camera parameters and packing metadata corresponding to the desired VD texture from the buffer based on its VD texture index. The display information generation unit 414 may then unpack the desired VD texture from the packed VD texture based on the obtained packing metadata. The display information generation unit 414 may then map the VD texture to a 3D object using the obtained camera parameters. The VD texture index is set for the camera parameters and packing metadata of each VD texture in the 3D object video extension described above, and has a different value for each corresponding VD texture.

[0213] Furthermore, when Method 1-4 is applied, Method 1-4-2 may be applied, and the file processing unit 412 may store each field of camera parameters and packing metadata in its corresponding buffer based on the field index. The display information generation unit 414 may also further obtain each field of camera parameters and packing metadata corresponding to the desired VD texture from the buffer based on its field index. The display information generation unit 414 may then unpack the desired VD texture from the packed VD texture based on the obtained packing metadata. The display information generation unit 414 may also map the VD texture to a 3D object using the obtained camera parameters. The field index is set for the fields of camera parameters and packing metadata in the 3D object video extension described above, and each field has a different value.

[0214] Alternatively, when Method 1 is applicable, Method 1-2 may be applied to make the video component a geometry map in which the geometry of the point cloud is packed, an attribute map in which the attributes of the point cloud are packed, and an occupancy map corresponding to the geometry map and the attribute map. Alternatively, the file processing unit 412 may store the geometry map, attribute map, and occupancy map in their respective buffers based on the component index. Alternatively, the display information generation unit 414 may retrieve the geometry map, attribute map, and occupancy map from the buffers based on its component index. The display information generation unit 414 may then reconstruct the point cloud using the retrieved geometry map, attribute map, and occupancy map. The display information generation unit 414 may then generate a display image using the reconstructed point cloud.

[0215] Of course, other aspects of this technique described above may also be applied in <3. Scene Descriptions Supporting Multiple Video Components>. Furthermore, multiple aspects of this technique may be combined and applied as appropriate.

[0216] With this configuration, the client device 400 can store multiple video components that can be used simultaneously for a single object in different buffers, based on the scene description. Therefore, the client device 400 can use multiple video components simultaneously for a single object using its scene description.

[0217] <Playback Process Flow> An example of the playback process flow performed by a client device 400 with this configuration using the method 1 described above will be explained with reference to the flowchart in Figure 37.

[0218] When playback processing begins, the file acquisition unit 411 of the client device 400 acquires the scene description file in step S401.

[0219] In step S402, the display information generation unit 414 analyzes the scene description file and obtains reference information to buffers in which each piece of data constituting the 3D object video (for example, each video component) is stored.

[0220] In step S403, the display information generation unit 414 controls the file processing unit 412 according to its scene description file to acquire encoded data of the 3D object video. The file processing unit 412 controls the file acquisition unit 411 according to the control of the display information generation unit 414 to acquire encoded data of the 3D object video. The file acquisition unit 411 acquires encoded data of the 3D object video stored in a content file such as an external or local storage of the client device 400 according to the control of the file processing unit 412.

[0221] In step S404, the decoding unit 413 decodes the encoded data of the 3D object video in accordance with the control of the file processing unit 412.

[0222] In step S405, the file processing unit 412 controls the decoding unit 413 to store the data decoded in step S404 (data constituting the 3D object video) in a buffer (not shown, corresponding to buffer 54 in Figure 7) according to the scene description. The decoding unit 413 stores the data decoded in step S404 (data constituting the 3D object video) in the buffer associated with that data in the scene description, according to the control of the file processing unit 412. For example, the file processing unit 412 (the decoding unit 413 controlled by the file processing unit 412) may apply Method 1 and store each video component included in the 3D object video in the buffer corresponding to that video component, based on the component index.

[0223] In step S406, the display information generation unit 414 reads (acquires) the configuration data (video components, etc.) of the 3D object video from its buffer according to the scene description.

[0224] In step S407, the display information generation unit 414 reconstructs the 3D object video using its configuration data (video components, etc.).

[0225] In step S408, the display information generation unit 414 reconstructs the scene from the 3D object video according to the scene description and generates a display image.

[0226] For example, the display information generation unit 414 may apply Method 1 and, in step S406, acquire video components from its buffer based on the component index. Alternatively, the display information generation unit 414 may apply Method 1 and, in steps S407 and S408, generate a display image using the acquired video components.

[0227] In step S409, the display unit 415 displays its display image.

[0228] When step S409 is completed, the playback process ends.

[0229] By performing the playback process in this manner, the client device 400 can store multiple video components that can be used simultaneously for a single object in separate buffers, based on the scene description. Therefore, the client device 400 can use multiple video components simultaneously for a single object using its scene description.

[0230] Alternatively, by applying Method 1-1, the video component may be a VD texture, which is an image captured from a predetermined viewpoint of a 3D object. Then, in step S405, the file processing unit 412 (or the decoding unit 413 controlled by the file processing unit 412) may store multiple VD textures with different viewpoints in their respective buffers based on the component index. In step S406, the display information generation unit 414 may obtain a desired VD texture from the buffer based on its component index. Then, in step S407, the display information generation unit 414 may map the obtained VD texture to a 3D object (mesh). Then, in step S408, the display information generation unit 414 may generate a display image using the 3D data (the mesh to which the VD texture is mapped).

[0231] Furthermore, if method 1-1 is applicable, method 1-3 may be applied, and in step S405, the file processing unit 412 (or the decoding unit 413 controlled by the file processing unit 412) may further store the camera parameters corresponding to the VD texture in the buffer corresponding to that VD texture. Also, in step S406, the display information generation unit 414 may further obtain the camera parameters corresponding to the desired VD texture from the buffer based on the component index. Then, in step S407, the display information generation unit 414 may map the VD texture to a 3D object using the obtained camera parameters.

[0232] Furthermore, when method 1-3 is applicable, method 1-3-1 may be applied, and in step S405, the file processing unit 412 (or the decoding unit 413 controlled by the file processing unit 412) may store each field of the camera parameters in its corresponding buffer based on the field index. In step S406, the display information generation unit 414 may further obtain each field of the camera parameters corresponding to the desired VD texture from the buffer based on its field index. Then, in step S407, the display information generation unit 414 may map the VD texture to a 3D object using the obtained camera parameters. The field index is set for the fields of the camera parameters in the 3D object video extension described above, and each field has a different value.

[0233] Alternatively, by applying Method 1-1, the video component may be a packed VD texture in which multiple VD textures, which are captured images of a 3D object from a predetermined viewpoint, are packed together. In step S405, the file processing unit 412 (or the decoding unit 413 controlled by the file processing unit 412) may store the packed VD texture in a buffer corresponding to that packed VD texture based on the component index. In step S406, the display information generation unit 414 may obtain the packed VD texture from the buffer based on its component index. The display information generation unit 414 may then unpack the desired VD texture from the obtained packed VD texture. In step S407, the display information generation unit 414 may map the unpacked VD texture to the 3D object. In step S408, the display information generation unit 414 may generate a display image using the 3D data (the mesh to which the VD texture is mapped).

[0234] Furthermore, if method 1-1 is applicable, method 1-4 may be applied, and in step S405, the file processing unit 412 (or the decoding unit 413 controlled by the file processing unit 412) may further store the camera parameters and packing metadata corresponding to the packed VD texture in the buffer corresponding to the packed VD texture. Also, in step S406, the display information generation unit 414 may further obtain the camera parameters and packing metadata corresponding to the packed VD texture from the buffer based on the component index. The display information generation unit 414 may then unpack the desired VD texture from the packed VD texture based on the obtained packing metadata. Finally, in step S407, the display information generation unit 414 may map the VD texture to a 3D object using the obtained camera parameters.

[0235] Furthermore, when method 1-4 is applicable, method 1-4-1 may be applied, and in step S405, the file processing unit 412 (or the decoding unit 413 controlled by the file processing unit 412) may store the respective camera parameters and packing metadata of the VD textures packed into the packed VD texture in the corresponding buffers, based on the VD texture index. In step S406, the display information generation unit 414 may further obtain the camera parameters and packing metadata corresponding to the desired VD texture from the buffer based on its VD texture index. The display information generation unit 414 may then unpack the desired VD texture from the packed VD texture based on the obtained packing metadata. In step S407, the display information generation unit 414 may map the VD texture to a 3D object using the obtained camera parameters. The VD texture index is set for the camera parameters and packing metadata for each VD texture in the 3D object video extension described above, and has a different value for each corresponding VD texture.

[0236] Furthermore, if method 1-4 is applicable, method 1-4-2 may be applied, and in step S405, the file processing unit 412 (or the decoding unit 413 controlled by the file processing unit 412) may store each field of camera parameters and packing metadata in the corresponding buffer based on the field index. In step S406, the display information generation unit 414 may further obtain each field of camera parameters and packing metadata corresponding to the desired VD texture from the buffer based on its field index. The display information generation unit 414 may then unpack the desired VD texture from the packed VD texture based on the obtained packing metadata. In step S407, the display information generation unit 414 may map the VD texture to a 3D object using the obtained camera parameters. The field index is set for the fields of camera parameters and packing metadata in the 3D object video extension described above, and each field has a different value.

[0237] Alternatively, by applying Method 1-2, the video component may be a geometry map in which the geometry of the point cloud is packed, an attribute map in which the attributes of the point cloud are packed, and an occupancy map corresponding to the geometry map and the attribute map. In step S405, the file processing unit 412 (or the decoding unit 413 controlled by the file processing unit 412) may store the geometry map, attribute map, and occupancy map in their respective buffers based on the component index. In step S406, the display information generation unit 414 may retrieve the geometry map, attribute map, and occupancy map from the buffers based on its component index. In step S407, the display information generation unit 414 may reconstruct the point cloud using the retrieved geometry map, attribute map, and occupancy map. In step S408, the display information generation unit 414 may generate a display image using the reconstructed point cloud.

[0238] Of course, other aspects of this technique described above may also be applied in <3. Scene Descriptions Supporting Multiple Video Components>. Furthermore, multiple aspects of this technique may be combined and applied as appropriate.

[0239] <6. Addendum> <Combinations> Each of the above-described examples of this technology may be applied in combination with other examples as appropriate, provided that no contradictions arise. Furthermore, each of the above-described examples of this technology may be applied in combination with other technologies not described above.

[0240] <Computer> The series of processes described above can be executed by either hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the computer includes a computer incorporated in dedicated hardware, and a device capable of executing various functions by installing various programs, such as a general-purpose personal computer, for example.

[0241] FIG. 38 is a block diagram showing a configuration example of computer hardware that executes the series of processes described above by means of a program.

[0242] In a computer 900 shown in FIG. 38, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are mutually connected via a bus 904.

[0243] An input / output interface 910 is also connected to the bus 904. An input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected to the input / output interface 910.

[0244] The input unit 911 is composed of, for example, a keyboard, a mouse, a microphone, a touch panel, input terminals, and the like. The output unit 912 is composed of, for example, a display, a speaker, output terminals, and the like. The storage unit 913 is composed of, for example, a hard disk, a RAM disk, a non-volatile memory, and the like. The communication unit 914 is composed of, for example, a network interface. The drive 915 drives a removable medium 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0245] In a computer configured as described above, the CPU 901 loads, for example, a program stored in the memory unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes it, thereby performing the series of processes described above. The RAM 903 also stores data necessary for the CPU 901 to perform various processes as appropriate.

[0246] The program to be executed by the computer can be recorded and applied, for example, on removable media 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable media 921 into the drive 915.

[0247] Furthermore, this program can also be provided via wired or wireless transmission media such as a local area network, the internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0248] Additionally, this program can be pre-installed on ROM902 or memory unit913.

[0249] <Applicable subjects of this technology> This technology can be applied to any encoding and decoding scheme.

[0250] Furthermore, this technology can be applied to any configuration. For example, it can be applied to various electronic devices.

[0251] Furthermore, this technology can also be implemented as part of a device, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set with additional functions added to a unit (e.g., a video set).

[0252] Furthermore, this technology can also be applied to network systems composed of multiple devices. For example, this technology may be implemented as cloud computing, where multiple devices share and collaborate on processing via a network. For example, this technology may be implemented in a cloud service that provides image (video) related services to any terminal such as computers, AV (Audio Visual) equipment, portable information processing terminals, and IoT (Internet of Things) devices.

[0253] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules within a single enclosure, are both considered systems.

[0254] <Fields and applications where this technology can be applied> Systems, devices, and processing units incorporating this technology can be used in any field, such as transportation, healthcare, security, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. Furthermore, their applications are entirely arbitrary.

[0255] For example, this technology can be applied to systems and devices used to provide entertainment content. Furthermore, for example, this technology can be applied to systems and devices used for traffic management, such as traffic condition monitoring and automated driving control. In addition, for example, this technology can be applied to systems and devices used for security. Furthermore, for example, this technology can be applied to systems and devices used for automatic control of machinery, etc. Furthermore, for example, this technology can be applied to systems and devices used for agriculture and livestock farming. Furthermore, for example, this technology can be applied to systems and devices that monitor natural conditions such as volcanoes, forests, and oceans, as well as wildlife. Furthermore, for example, this technology can be applied to systems and devices used for sports.

[0256] <Other> In this specification, "flag" refers to information used to identify multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the values ​​that this "flag" can take are, for example, two values, 1 / 0, or three or more values. In other words, the number of bits that constitute this "flag" is arbitrary, and can be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in the form of the identification information itself in the bitstream, but also in the form of differential information of the identification information relative to a certain reference information in the bitstream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also differential information relative to the reference information.

[0257] Furthermore, various types of information (metadata, etc.) related to encoded data (bitstream) may be transmitted or recorded in any form, as long as they are associated with the encoded data. Here, the term "associate" means, for example, making it possible to use (link) one piece of data when processing the other. In other words, associated data may be combined into a single piece of data, or they may be individual pieces of data. For example, information associated with encoded data (image) may be transmitted on a different transmission path than the encoded data (image). Also, for example, information associated with encoded data (image) may be recorded on a different recording medium (or a different recording area on the same recording medium) than the encoded data (image). Note that this "association" may not apply to the entire data, but only to a part of it. For example, an image and the information corresponding to that image may be associated with each other in any unit, such as multiple frames, a single frame, or a part within a frame.

[0258] In this specification, terms such as "combine," "multiplex," "add," "integrate," "include," "store," "insert," "insert," and "place" mean combining multiple things into one, such as combining encoded data and metadata into a single data, and represent one method of "associating" as described above.

[0259] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the gist of this technology.

[0260] For example, the configuration described as a single device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as a single device (or processing unit). Furthermore, it is also possible to add configurations other than those described above to the configuration of each device (or each processing unit). In addition, if the overall system configuration and operation are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0261] Furthermore, for example, the program described above may be executed on any device. In that case, the device should have the necessary functions (such as functional blocks) and be able to obtain the necessary information.

[0262] Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0263] Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs.

[0264] Furthermore, for example, unless a contradiction arises, a plurality of techniques related to the present technology can each be implemented independently as a single unit. Of course, any plurality of the present technologies can also be implemented in combination. For example, part or all of the present technology described in any one embodiment can also be implemented in combination with part or all of the present technology described in other embodiments. Furthermore, part or all of any of the above-described present technologies can also be implemented in combination with other technologies not described above.

[0265] Note that the present technology can also adopt the following configurations. (1) a file processing unit that stores, based on a component index, a video component included in a 3D object video into a buffer corresponding to the video component; a display image generating unit that acquires the video component from the buffer based on the component index, and generates a display image using the acquired video component; and comprising: a plurality of the video components can be used simultaneously in one 3D object, the component index is set for the video component included in the 3D object video in an extension for the 3D object video defined in a material layer of a scene description, and has a different value for each video component An information processing apparatus. (2) the video component is a VD texture that is a captured image obtained by capturing the 3D object from a predetermined viewpoint, the file processing unit stores, based on the component index, the plurality of VD textures having mutually different viewpoints into the respectively corresponding buffers, the display image generating unit acquires a desired VD texture from the buffer based on the component index, maps the acquired VD texture onto the 3D object, and generates the display image (1) The information processing device described above. (3) The file processing unit further stores the camera parameters corresponding to the VD texture in the buffer corresponding to the VD texture, The display image generation unit further obtains the camera parameters corresponding to the desired VD texture from the buffer based on the component index, and maps the VD texture to the 3D object using the obtained camera parameters. (2) The information processing device described in (2). (4) The file processing unit stores each field of the camera parameter in the corresponding buffer based on the field index. The display image generation unit further obtains each field of the camera parameters corresponding to the desired VD texture from the buffer based on the field index, and maps the VD texture to the 3D object using the obtained camera parameters. The field index is set for the field of the camera parameter in the extension, and has a different value for each field. (3) The information processing device described above. (5) The video component is a packed VD texture in which multiple VD textures, which are captured images of the 3D object taken from a predetermined viewpoint, are packed together. The file processing unit stores the packed VD texture in the buffer corresponding to the packed VD texture based on the component index. The display image generation unit retrieves the packed VD texture from the buffer based on the component index, unpacks the desired VD texture from the retrieved packed VD texture, maps the unpacked VD texture to the 3D object, and generates the display image. (1) to (4) the information processing device described in any of these. (6) The file processing unit further stores the camera parameters and packing metadata corresponding to the packed VD texture in the buffer corresponding to the packed VD texture, The display image generation unit further obtains the camera parameters and packing metadata corresponding to the packed VD texture from the buffer based on the component index, unpacks the desired VD texture from the packed VD texture based on the obtained packing metadata, and maps the VD texture to the 3D object using the obtained camera parameters. (5) The information processing device described above. (7) The file processing unit stores the respective camera parameters and packing metadata of the VD textures packed into the packed VD textures, based on the VD texture index, in the corresponding buffers. The display image generation unit further obtains the camera parameters and packing metadata corresponding to the desired VD texture from the buffer based on the VD texture index, unpacks the desired VD texture from the packed VD texture based on the obtained packing metadata, and maps the VD texture to the 3D object using the obtained camera parameters. The VD texture index is set in the extension for the camera parameters and packing metadata for each VD texture, and has a different value for each corresponding VD texture. (6) The information processing device described above. (8) The file processing unit stores each of the camera parameters and the packing metadata fields in the corresponding buffer based on the field index. The display image generation unit further obtains the camera parameters and packing metadata fields corresponding to the desired VD texture from the buffer based on the field index, unpacks the desired VD texture from the packed VD texture based on the obtained packing metadata, and maps the VD texture to the 3D object using the obtained camera parameters. The field index is set in the extension for the fields of the camera parameters and the packing metadata, and has a different value for each field. (6) The information processing device described above. (9) The video component is a geometry map in which the geometry of the point cloud is packed, an attribute map in which the attributes of the point cloud are packed, and an occupancy map corresponding to the geometry map and the attribute map. The file processing unit stores the geometry map, the attribute map, and the occupancy map in the corresponding buffers based on the component index. The display image generation unit retrieves the geometry map, attribute map, and occupancy map from the buffer based on the component index, reconstructs the point cloud using the retrieved geometry map, attribute map, and occupancy map, and generates the display image. (1) The information processing device described above. (10) Based on the component index, the video components included in the 3D object video are stored in the buffer corresponding to the video component, Based on the component index, the video component is retrieved from the buffer, and the retrieved video component is used to generate a display image. Multiple video components can be used simultaneously in a single 3D object. The component index is set for the video components included in the 3D object video in the extension for the 3D object video defined in the material layer of the scene description, and has a different value for each video component. Information processing methods.

[0266] (11) A file generation unit that generates a scene description file and, in an extension for a 3D object video defined in the material layer of the scene description file, sets a component index with a different value for each video component included in the 3D object video, Multiple video components can be used simultaneously within a single 3D object. Information processing device. (12) The video component is a VD texture which is an image captured from a predetermined viewpoint of the 3D object, The file generation unit sets the component index for each of the multiple VD textures included in the 3D object video in the extension. (11) The information processing device described above. (13) The file generation unit further stores the camera parameters corresponding to the VD texture in the extension. (12) The information processing device described above. (14) The file generation unit further sets a field index in the extension for each field of the camera parameters, with a different value for each field. (13) The information processing device described above. (15) The video component is a packed VD texture in which multiple VD textures, which are captured images of the 3D object taken from a predetermined viewpoint, are packed together. The file generation unit sets the component index for the packed VD texture in the extension. (11) to (14) the information processing device described in any of these. (16) The file generation unit further stores camera parameters and packing metadata corresponding to the packed VD texture in the extension. (15) The information processing device described above. (17) The file generation unit further sets a VD texture index in the extension that has a different value for each VD texture for each camera parameter and packing metadata packed in the packed VD texture. (16) The information processing device described above. (18) The file generation unit further sets a field index in the extension that has a different value for each of the camera parameter and packing metadata fields. (16) The information processing device described above. (19) The video component is a geometry map in which the geometry of the point cloud is packed, an attribute map in which the attributes of the point cloud are packed, and an occupancy map corresponding to the geometry map and the attribute map. The file generation unit sets the component index for each of the geometry map, attribute map, and occupancy map in the extension, with values ​​that are different from each other. (11) The information processing device described above. (20) Generate a scene description file, and in the extension for 3D object video defined in the material layer of the scene description file, set a component index with a different value for each video component for the video components included in the 3D object video, Multiple video components can be used simultaneously within a single 3D object. Information processing methods. [Explanation of Symbols]

[0267] 300 File generation device, 301 Control unit, 302 File generation processing unit, 311 Input unit, 312 Preprocessing unit, 313 Encoding unit, 314 File generation unit, 315 Recording unit, 316 Output unit, 400 Client device, 401 Control unit, 402 Client processing unit, 411 File acquisition unit, 412 File processing unit, 413 Decoding unit, 414 Display information generation unit, 415 Display unit, 416 Display control unit< / isobmff>

Claims

1. A file processing unit that stores video components included in a 3D object video in buffers corresponding to the video components, based on the component index, A display image generation unit that acquires the video component from the buffer based on the component index and generates a display image using the acquired video component. Equipped with, Multiple video components can be used simultaneously in a single 3D object. The component index is set for the video components included in the 3D object video in the extension for the 3D object video defined in the material layer of the scene description, and has a different value for each video component. Information processing device.

2. The video component is a VD texture which is an image captured from a predetermined viewpoint of the 3D object. The file processing unit stores, based on the component index, a plurality of VD textures with different viewpoints in the corresponding buffers. The display image generation unit retrieves the desired VD texture from the buffer based on the component index, maps the retrieved VD texture to the 3D object, and generates the display image. The information processing apparatus according to claim 1.

3. The file processing unit further stores the camera parameters corresponding to the VD texture in the buffer corresponding to the VD texture. The display image generation unit further obtains the camera parameters corresponding to the desired VD texture from the buffer based on the component index, and maps the VD texture to the 3D object using the obtained camera parameters. The information processing apparatus according to claim 2.

4. The file processing unit stores each field of the camera parameter in the corresponding buffer based on the field index. The display image generation unit further obtains each field of the camera parameters corresponding to the desired VD texture from the buffer based on the field index, and maps the VD texture to the 3D object using the obtained camera parameters. The field index is set for the field of the camera parameter in the extension, and has a different value for each field. The information processing apparatus according to claim 3.

5. The video component is a packed VD texture in which multiple VD textures, which are captured images of the 3D object taken from a predetermined viewpoint, are packed together. The file processing unit stores the packed VD texture in the buffer corresponding to the packed VD texture based on the component index. The display image generation unit retrieves the packed VD texture from the buffer based on the component index, unpacks the desired VD texture from the retrieved packed VD texture, maps the unpacked VD texture to the 3D object, and generates the display image. The information processing apparatus according to claim 1.

6. The file processing unit further stores the camera parameters and packing metadata corresponding to the packed VD texture in the buffer corresponding to the packed VD texture. The display image generation unit further obtains the camera parameters and packing metadata corresponding to the packed VD texture from the buffer based on the component index, unpacks the desired VD texture from the packed VD texture based on the obtained packing metadata, and maps the VD texture to the 3D object using the obtained camera parameters. The information processing apparatus according to claim 5.

7. The file processing unit stores the respective camera parameters and packing metadata of the VD textures packed into the packed VD texture, based on the VD texture index, in the corresponding buffers. The display image generation unit further obtains the camera parameters and packing metadata corresponding to the desired VD texture from the buffer based on the VD texture index, unpacks the desired VD texture from the packed VD texture based on the obtained packing metadata, and maps the VD texture to the 3D object using the obtained camera parameters. The VD texture index is set in the extension for the camera parameters and packing metadata for each VD texture, and has a different value for each corresponding VD texture. The information processing apparatus according to claim 6.

8. The file processing unit stores each field of the camera parameters and the packing metadata in the corresponding buffer based on the field index. The display image generation unit further obtains the camera parameters and packing metadata fields corresponding to the desired VD texture from the buffer based on the field index, unpacks the desired VD texture from the packed VD texture based on the obtained packing metadata, and maps the VD texture to the 3D object using the obtained camera parameters. The field index is set in the extension for the fields of the camera parameters and the packing metadata, and has a different value for each field. The information processing apparatus according to claim 6.

9. The video component includes a geometry map in which the geometry of the point cloud is packed, an attribute map in which the attributes of the point cloud are packed, and an occupancy map corresponding to the geometry map and the attribute map. The file processing unit stores the geometry map, the attribute map, and the occupancy map in the corresponding buffers based on the component index. The display image generation unit retrieves the geometry map, attribute map, and occupancy map from the buffer based on the component index, reconstructs the point cloud using the retrieved geometry map, attribute map, and occupancy map, and generates the display image. The information processing apparatus according to claim 1.

10. Based on the component index, the video components included in the 3D object video are stored in the buffer corresponding to the video component. Based on the component index, the video component is retrieved from the buffer, and the retrieved video component is used to generate a display image. Multiple video components can be used simultaneously in a single 3D object. The component index is set for the video components included in the 3D object video in the extension for the 3D object video defined in the material layer of the scene description, and has a different value for each video component. Information processing methods.

11. The system includes a file generation unit that generates a scene description file and, in an extension for 3D object video defined in the material layer of the scene description file, sets a component index with a different value for each video component included in the 3D object video. Multiple video components can be used simultaneously within a single 3D object. Information processing device.

12. The video component is a VD texture which is an image captured from a predetermined viewpoint of the 3D object. The file generation unit sets the component index for each of the multiple VD textures included in the 3D object video in the extension. The information processing apparatus according to claim 11.

13. The file generation unit further stores the camera parameters corresponding to the VD texture in the extension. The information processing apparatus according to claim 12.

14. The file generation unit further sets a field index in the extension that has a different value for each field of the camera parameters. The information processing apparatus according to claim 13.

15. The video component is a packed VD texture in which multiple VD textures, which are captured images of the 3D object taken from a predetermined viewpoint, are packed together. The file generation unit sets the component index for the packed VD texture in the extension. The information processing apparatus according to claim 11.

16. The file generation unit further stores camera parameters and packing metadata corresponding to the packed VD texture in the extension. The information processing apparatus according to claim 15.

17. The file generation unit further sets, in the extension, a VD texture index with a different value for each VD texture corresponding to the camera parameters and packing metadata of each VD texture packed into the packed VD texture. The information processing apparatus according to claim 16.

18. The file generation unit further sets a field index in the extension that has a different value for each of the camera parameter and packing metadata fields. The information processing apparatus according to claim 16.

19. The video component includes a geometry map in which the geometry of the point cloud is packed, an attribute map in which the attributes of the point cloud are packed, and an occupancy map corresponding to the geometry map and the attribute map. The file generation unit sets the component index for each of the geometry map, attribute map, and occupancy map in the extension, with values ​​that are different from each other. The information processing apparatus according to claim 11.

20. A scene description file is generated, and in the extension for 3D object video defined in the material layer of the scene description file, a component index with a different value is set for each video component included in the 3D object video. Multiple video components can be used simultaneously within a single 3D object. Information processing methods.

Images