Information processing device and method

Abstract

The present disclosure relates to an information processing device and a method which make it possible to suppress an increase in a reproduction processing load. Timed metadata identification information indicating that metadata of a linked external file changes in the time direction is stored in an MPEG_media extension; timed metadata access information linking a camera object to the metadata is stored in the camera object; and a scene description file describing a 3D object content scene is generated. In addition, timed metadata that changes in the time direction is acquired on the basis of the timed metadata identification information and the timed metadata access information stored in the scene description file, and a 3D object content display image is generated on the basis of the acquired timed metadata. The present disclosure may be applied, for example, to an information processing device, an information processing method, or the like.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that can suppress an increase in the load of playback processing. [Background technology]

[0002] Conventionally, there is the GL Transmission Format (glTF) 2.0 (registered trademark), which is a format for placing 3D (three-dimensional) objects in a three-dimensional space (see, for example, Non-Patent Document 1). In recent years, in the Moving Picture Experts Group (MPEG)-I Scene Description, extending glTF 2.0 to apply content that changes in the time direction has been considered (see, for example, Non-Patent Document 2). For example, a method has been proposed for handling timed texture media that has been coded and stored in the International Organization for Standardization Base Media File Format (ISOBMFF) or the like as texture data (see, for example, Non-Patent Document 3). Furthermore, studies are underway to handle not only content but also metadata associated with the content that changes over time (see, for example, Non-Patent Document 4). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Saurabh Bhatia, Patrick Cozzi, Alexey Knyazev, Tony Parisi, "Khronos glTF2.0", https: / / github.com / KhronosGroup / glTF / tree / master / specification / 2.0, June 9, 2017 [Non-patent document 2] Lukasz Kondrad, Imed Bouazizi, "Technologies under Considerations on Scene Description for MPEG Media", ISO / IEC JTC1 / SC29 / WG11 MPEG2020 / N19290, April 2020 [Non-patent document 3] "Information technology. Coding of audio-visual objects. Part 12", ISO / IEC 14496-12, 2015-02-20 [Non-patent document 4] Shuichi Aoki and Yuma Wakahara, "(36.1) Proposed extensions of glTF2 for supporting MPEG media", ISO / IEC JTC1 / SC29 / WG11 MPEG2020 / M53397r1, April 2020 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional methods, when time-varying metadata is handled in a scene description, all of the time-converted metadata is described in the scene description. When playing back 3D object content, the time-converted metadata and other timed media have different synchronization mechanisms, so they must be synchronized. This requires cumbersome work and can increase the load on the playback process of 3D object content.

[0005] The present disclosure has been made in view of such circumstances, and aims to make it possible to suppress an increase in the load of the regeneration process. [Means for solving the problem]

[0006] According to one aspect of the present technology, an information processing device stores timed metadata identification information, which indicates that metadata of a linked external file changes in a time direction, in an MPEG_media extension, and links a camera object to the metadata. timed The information processing device includes a file generation unit that stores metadata access information in the camera object and generates a scene description file that describes a scene of 3D object content.

[0007] According to one aspect of the present technology, an information processing method includes storing timed metadata identification information, which indicates that metadata of a linked external file changes in the time direction, in an MPEG_media extension, and linking a camera object to the metadata. timed This is an information processing method for generating a scene description file that describes a scene of 3D object content, and stores metadata access information in the camera object.

[0008] An information processing device according to another aspect of the present technology is an information processing device that includes an acquisition unit that acquires timed metadata that changes in the time direction based on timed metadata identification information stored in an MPEG_media extension of a scene description file that describes a scene of 3D object content and timed metadata access information stored in a camera object of the scene description file, and a generation unit that generates a display image of the 3D object content based on the timed metadata acquired by the acquisition unit.

[0009] An information processing method according to another aspect of the present technology is an information processing method for acquiring timed metadata that changes in the time direction based on timed metadata identification information stored in an MPEG_media extension of a scene description file that describes a scene of 3D object content and timed metadata access information stored in a camera object of the scene description file, and for generating a display image of the 3D object content based on the acquired timed metadata.

[0010] In the information processing device and method according to one aspect of the present technology, timed metadata identification information indicating that the metadata of the linked external file changes in the time direction is stored in an MPEG_media extension, and the camera object is linked to that metadata. timed A scene description file is generated that describes the scene of the 3D object content, storing metadata access information for the camera object.

[0011] In an information processing device and method according to another aspect of the present technology, timed metadata that changes in the time direction is acquired based on timed metadata identification information stored in the MPEG_media extension of a scene description file that describes a scene of 3D object content and timed metadata access information stored in a camera object of that scene description file, and a display image of the 3D object content is generated based on the acquired timed metadata. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of the main configuration of glTF2.0. [Figure 2] FIG. 1 is a diagram illustrating an example of glTF objects and reference relationships. [Figure 3] FIG. 10 is a diagram illustrating an example of a description of a JSON format file. [Figure 4] FIG. 10 is a diagram illustrating a method for accessing binary data. [Figure 5] FIG. 10 is a diagram illustrating an example of a description of a JSON format file. [Figure 6] FIG. 10 is a diagram illustrating the relationship between a buffer object, a buffer view object, and an accessor object. [Figure 7] 10A and 10B are diagrams illustrating examples of descriptions of a buffer object, a buffer view object, and an accessor object. [Figure 8] FIG. 1 is a diagram illustrating an extension method for glTF2.0. [Figure 9] FIG. 10 is a diagram illustrating an extension for handling timed media. [Figure 10] FIG. 10 is a diagram illustrating an extension for handling timed media. [Figure 11] FIG. 10 is a diagram illustrating an extension for handling timed media. [Figure 12] FIG. 10 is a diagram illustrating an extension for handling timed media. [Figure 13] FIG. 2 is a diagram illustrating a camera object. [Figure 14] FIG. 10 is a diagram illustrating an example of a camera object that undergoes time conversion. [Figure 15] FIG. 10 is a diagram illustrating an example of a camera object that undergoes time conversion. [Figure 16] FIG. 10 is a diagram illustrating a method for transmitting timed metadata. [Figure 17] FIG. 10 is a diagram illustrating the association of camera objects for time conversion. [Figure 18] FIG. 10 is a diagram showing an example of a description of an MPEG_media object in a scene description. [Figure 19] FIG. 10 is a diagram showing an example of a description of camera extensions in a scene description. [Figure 20]FIG. 10 is a diagram illustrating an example of the semantics of parameters applied in camera extensions. [Figure 21] FIG. 10 is a diagram illustrating an example of a method for accessing timed metadata via an accessor object. [Figure 22] FIG. 10 is a diagram illustrating an example of timed metadata. [Figure 23] FIG. 10 is a diagram showing an example of a description of camera extensions in a scene description. [Figure 24] FIG. 10 is a diagram showing an example of a description of camera extensions in a scene description. [Figure 25] FIG. 10 is a diagram showing an example of multiple camera objects. [Figure 26] FIG. 10 is a diagram showing an example of linking multiple pieces of timed metadata to a scene description. [Figure 27] FIG. 10 is a diagram showing an example of linking multiple pieces of timed metadata to a scene description. [Figure 28] FIG. 2 is a block diagram illustrating an example of the main configuration of a file generation device. [Figure 29] 10 is a flowchart illustrating an example of the flow of a file generation process. [Figure 30] FIG. 2 is a block diagram illustrating an example of the main configuration of a client device. [Figure 31] 10 is a flowchart illustrating an example of the flow of a client process. [Figure 32] 10 is a flowchart illustrating an example of the flow of a client process. [Figure 33] 10 is a flowchart illustrating an example of the flow of a client process. [Figure 34] FIG. 1 is a block diagram illustrating an example of the main configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described in the following order. 1.MPEG-I Scene Description 2. Transmission of timed metadata, etc. 3. First embodiment (file generation device) 4. Second embodiment (client device) 5. Additional Notes

[0014] <1.MPEG-I Scene Description> <References supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the contents described in the embodiments but also the contents described in the following non-patent documents that were publicly known at the time of filing, as well as the contents of other documents referenced in the following non-patent documents.

[0015] Non-patent document 1: (mentioned above) Non-patent document 2: (mentioned above) Non-patent document 3: (mentioned above) Non-patent document 4: (mentioned above)

[0016] In other words, the contents of the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements.

[0017] <gltf2.0> Conventionally, as described in Non-Patent Document 1, for example, there is glTF (The GL Transmission Format) (registered trademark) 2.0, which is a format for placing 3D (three-dimensional) objects in a three-dimensional space. As shown in FIG. 1, for example, glTF 2.0 is composed of a JSON format file (.glTF), a binary file (.bin), and an image file (.png, .jpg, etc.). The binary file stores binary data such as geometry and animation. The image file stores data such as texture.

[0018] A JSON format file is a scene description file written in JSON (JavaScript (registered trademark) Object Notation). A scene description is metadata that describes (a description of) a scene of 3D content. The description of this scene description defines what kind of scene it is. A scene description file is a file that stores such a scene description. In this disclosure, a scene description file is also referred to as a scene description file.

[0019] The description of the JSON format file consists of a list of key and value pairs. An example of the format is shown below. “KEY”:”VALUE”

[0020] Keys consist of strings, and values consist of numbers, strings, booleans, arrays, objects, or null.

[0021] Additionally, multiple key-value pairs ("KEY":"VALUE") can be grouped together using {} (curly braces). This grouping is also called a JSON object. An example of the format is shown below. “user”:{"id":1, "name":"tanaka"}

[0022] In this example, a JSON object that combines the pair "id":1 and the pair "name":"tanaka" is defined as the value corresponding to the key (user).

[0023] You can also create an array of zero or more values using square brackets ([]). This array is also called a JSON array. For example, a JSON object can be applied as an element of this JSON array. An example of the format is shown below. test":["hoge", "fuga", "bar"] "users":[{"id":1, "name":"tanaka"},{"id":2,"name":"yamada"},{"id":3, "name":"sato"}]

[0024] Figure 2 shows the glTF objects that can be written at the top level of a JSON format file and the reference relationships they can have. The long circles in the tree structure shown in Figure 2 represent objects, and the arrows between those objects show the reference relationships. As shown in Figure 2, objects such as "scene", "node", "mesh", "camera", "skin", "material", and "texture" are written at the top level of a JSON format file.

[0025] An example of such a JSON format file (scene description) is shown in FIG. 3. The JSON format file 20 in FIG. 3 shows an example of a portion of the top-level description. In this JSON format file 20, all top-level objects 21 used are described at the top level. These top-level objects 21 are the glTF objects shown in FIG. 2. Furthermore, in the JSON format file 20, reference relationships between objects are shown as indicated by arrows 22. More specifically, the reference relationships are shown by specifying the index of an element in the array of the referencing object in the property of the higher-level object.

[0026] Fig. 4 is a diagram illustrating a method for accessing binary data. As shown in Fig. 4, binary data is stored in a buffer object. That is, information for accessing the binary data (for example, a uniform resource identifier (URI)) is indicated in the buffer object. In a JSON format file, as shown in Fig. 4, objects such as a mesh, camera, and skin can access the buffer object via an accessor object and a bufferView object.

[0027] That is, for objects such as mesh, camera, and skin, the accessor object to be referenced is specified. An example of a mesh object description in a JSON format file is shown in Figure 5. For example, as shown in Figure 5, in a mesh object, vertex attributes such as NORMAL, POSITION, TANGENT, and TEXCORD_0 are defined as keys, and for each attribute, the accessor object to be referenced is specified as a value.

[0028] The relationship between buffer objects, buffer view objects, and accessor objects is shown in Figure 6. An example of how these objects are written in a JSON format file is shown in Figure 7.

[0029] 6, buffer object 41 is an object that stores information (such as a URI) for accessing binary data, which is actual data, and information indicating the data length (for example, byte length) of that binary data. A in FIG. 7 shows an example of the description of buffer object 41. "bytelength":102040" shown in A in FIG. 7 indicates that the byte length of buffer object 41 is 102040 bytes, as shown in FIG. 6. Furthermore, "uri":"duck.bin" shown in A in FIG. 7 indicates that the URI of buffer object 41 is "duck.bin", as shown in FIG. 6.

[0030] 6, the buffer view object 42 is an object that stores information about a subset area of binary data specified in the buffer object 41 (i.e., information about a partial area of the buffer object 41). B of Fig. 7 shows an example of description of the buffer view object 42. As shown in Fig. 6 and B of Fig. 7, the buffer view object 42 stores information such as identification information of the buffer object 41 to which the buffer view object 42 belongs, an offset (e.g., a byte offset) indicating the position of the buffer view object 42 within the buffer object 41, and a length (e.g., a byte length) indicating the data length (e.g., a byte length) of the buffer view object 42.

[0031] As shown in B of Fig. 7, when there are multiple buffer view objects, information is written for each buffer view object (i.e., for each subset area). For example, information such as "buffer":0, "bytelength":25272, and "byteOffset":0 shown at the top of B of Fig. 7 is information for the first buffer view object 42 (bufferView[0]) shown in the buffer object 41 in Fig. 6. Furthermore, information such as "buffer":0, "bytelength":76768, and "byteOffset":25272 shown at the bottom of B of Fig. 7 is information for the second buffer view object 42 (bufferView[1]) shown in the buffer object 41 in Fig. 6.

[0032] "buffer":0" of the first buffer view object 42 (bufferView[0]) shown in B of FIG. 7 indicates that the identification information of the buffer object 41 to which the buffer view object 42 (bufferView[0]) belongs is "0" (Buffer[0]), as shown in FIG. 6. Also, "bytelength":25272" indicates that the byte length of the buffer view object 42 (bufferView[0]) is 25272 bytes. Furthermore, "byteOffset":0" indicates that the byte offset of the buffer view object 42 (bufferView[0]) is 0 bytes.

[0033] "buffer":0" of the second buffer view object 42 (bufferView[1]) shown in B of FIG. 7 indicates that the identification information of the buffer object 41 to which the buffer view object 42 (bufferView[0]) belongs is "0" (Buffer[0]), as shown in FIG. 6. Also, "bytelength":76768" indicates that the byte length of the buffer view object 42 (bufferView[0]) is 76768 bytes. Furthermore, "byteOffset":25272" indicates that the byte offset of the buffer view object 42 (bufferView[0]) is 25272 bytes.

[0034] 6, the accessor object 43 is an object that stores information about how to interpret data in the buffer view object 42. C in Fig. 7 shows an example of the description of the accessor object 43. As shown in Figs. 6 and 7C, the accessor object 43 stores information such as the identification information of the buffer view object 42 to which the accessor object 43 belongs, the offset (e.g., byte offset) indicating the position of the buffer view object 42 within the buffer object 41, the component type of the buffer view object 42, the number of data items stored in the buffer view object 42, and the type of data items stored in the buffer view object 42. This information is described for each buffer view object.

[0035] In the example of C in FIG. 7, information such as "bufferView":0, "byteOffset":0, "componentType":5126, "count":2106," and "type":"VEC3" is shown. "bufferView":0" indicates that the identification information of the buffer view object 42 to which the accessor object 43 belongs is "0" (bufferView[0]), as shown in FIG. 6. Furthermore, "byteOffset":0" indicates that the byte offset of the buffer view object 42 (bufferView[0]) is 0 bytes. Furthermore, "componentType":5126" indicates that the component type is FLOAT type (OpenGL macro constant). Furthermore, "count":2106" indicates that 2106 pieces of data are stored in the buffer view object 42 (bufferView[0]). Furthermore, "type":"VEC3" indicates that the data (type) stored in the buffer view object 42 (bufferView[0]) is a three-dimensional vector.

[0036] All accesses to data other than images are defined by reference to this accessor object 43 (by specifying the accessor index).

[0037] In glTF 2.0, an object can be extended to store newly defined objects within an extension object. Figure 8 is a diagram for explaining the extension method of glTF 2.0. The description example shown in Figure 8 shows a description example when storing a newly defined object (CompareDataExtension) in the extension object of a scene object. In the example shown in Figure 8, a new object (CompareDataExtension) is stored in the extension object (extensions) of the scene object ("scenes"). When extending an object in this way, the name of the new object is defined in "extensionUsed" and "extensionRequired". This indicates that the new object is an object required for loading.

[0038] <Application of Timed media> In recent years, for example, as shown in Non-Patent Document 2, in MPEG (Moving Picture Experts Group)-I Scene Description, it has been studied to extend glTF 2.0 and apply timed media as 3D object content. Timed media is media data that changes in the time axis direction, like moving images in two-dimensional images.

[0039] glTF was only applicable to still image data as media data (3D object content). That is, glTF did not support media data of moving images. When moving a 3D object, animation (a method of switching still images along the time axis) was applied.

[0040] For MPEG-I Scene Description, glTF 2.0 will be applied, JSON format files will be used as scene descriptions, and further, extensions to glTF are being considered to enable handling of timed media (e.g., video data) as media data. For example, the following extensions will be made to handle timed media:

[0041] Figure 9 is a diagram explaining extensions for handling timed media. As shown in Figure 9, an MPEG media object (MPEG_media) that manages actual data such as video data is provided as an extension object (extension) of the glTF object. In other words, information about the actual data such as video data is stored in the MPEG media object.

[0042] 9, an MPEG video texture object (MPEG_video_texture) is provided as an extension object (extension) of the texture object (texture). The MPEG video texture object stores accessor information corresponding to the buffer object to be accessed. That is, the MPEG video texture object stores accessor information corresponding to the buffer object in which the texture media specified by the MPEG media object (MPEG_media) is decoded and stored.

[0043] 10 is a diagram showing an example of the description of an MPEG media object (MPEG_media) and an MPEG video texture object (MPEG_video_texture) in a scene description to explain extensions for handling timed media. In the example of FIG. 10, an MPEG video texture object (MPEG_video_texture) is set as an extension object (extensions) of a texture object (texture) in the second line from the top, as shown below. The accessor index ("2" in this example) is then specified as the value of that MPEG video texture object.

[0044] "texture":[{"sampler":0, "source":1, "extensions":{"MPEG_video_texture":"accessor":2}}],

[0045] 10, an MPEG media object (MPEG_media) is set as a glTF extension object (extensions) on lines 7 to 16 from the top, as shown below: The value of the MPEG media object stores various information about the MPEG media object, such as the encoding and URI of the MPEG media object.

[0046] "MPEG_media":{ "media":[ {"name":"source_1", "renderingRate":30.0, "startTime":9.0, "timeOffset":0.0, "loop":"true", "controls":"false", "alternatives":[{"mimeType":"video / mp4;codecs=\"avc1.42E01E\"", "uri":"video1.mp4", "tracks":[{"track":""#track_ID=1"}] }] } ] }

[0047] 9, an MPEG circular buffer object (MPEG_circular_buffer) is provided as an extension object of the buffer object. The MPEG circular buffer object stores information for dynamically storing data in the buffer object. For example, information indicating the data length of the buffer header and information indicating the number of frames are stored in the MPEG circular buffer object. The buffer header stores information such as an index, a timestamp and data length of the frame data to be stored, etc.

[0048] 9, an MPEG timed accessor object (MPEG_timed_accessor) is provided as an extension object of the accessor object. In this case, since the media data is video, the buffer view object (bufferView) referenced in the time direction may change (its position may fluctuate). Therefore, information indicating the buffer view object to be referenced is stored in this MPEG timed accessor object. For example, the MPEG timed accessor object stores information indicating a reference to the buffer view object (bufferView) in which the timed accessor information header (timedAccessor information header) is written. The timed accessor information header is header information that stores, for example, dynamically changing accessor objects and information in the buffer view object.

[0049] 11 is a diagram showing an example of the description of an MPEG circular buffer object (MPEG_circular_buffer) and an MPEG timed accessor object (MPEG_timed_accessor) in a scene description to explain extensions for handling timed media. In the example of Fig. 11, the MPEG timed accessor object (MPEG_timed_accessor) is set as an extension object (extensions) of the accessor object (accessors) in the fifth line from the top, as shown below. Then, parameters such as the index of the buffer view object ("1" in this example), the update rate (updataRate), and immutable information (immutable) and their values are specified as the value of the MPEG timed accessor object.

[0050] "MPEG_timed_accessor":{"bufferView":1, "updateRate":25.0, "immutable":1,"}

[0051] 11, an MPEG circular buffer object (MPEG_circular_buffer) is set as an extension object (extensions) of the buffer object (buffer) on the 13th line from the top, as shown below: Then, parameters such as the buffer frame count (count), header length (headerLength), and update rate (updataRate) and their values are specified as values of the MPEG circular buffer object.

[0052] "MPEG_circular_buffer":{"count":5, "headerLength":12, "updateRate":25.0}

[0053] Fig. 12 is a diagram for explaining extensions for handling timed media, showing examples of the relationship between an MPEG timed accessor object and an MPEG circular buffer object, and an accessor object, a buffer view object, and a buffer object.

[0054] As described above, the MPEG circular buffer object of the buffer object stores information necessary for storing time-varying data in the buffer area indicated by the buffer object, such as the buffer frame count (count), header length (headerLength), update rate (updataRate), etc. Furthermore, the buffer header (bufferHeader), which is the header of the buffer area, stores parameters such as the index (idex), timestamp (timestamp), data length (length), etc.

[0055] As described above, the MPEG timed accessor object of the accessor object stores information about the buffer view object it references, such as the index (bufferView) of the buffer view object, the update rate (updataRate), immutable information, etc. Also, this MPEG timed accessor object stores information about the buffer view object in which the timed accessor information header it references is stored. The timed accessor information header can store a timestamp delta (timestamp_delta), update data of the accessor object, update data of the buffer view object, etc.

[0056] <Camera object> In glTF2.0, there is a camera object. A camera object is an object that cuts out a part of a three-dimensional space and displays it as a two-dimensional image, and stores information such as the angle of view and perspective. As shown in A of FIG. 13, a camera object 81 is linked to a node object. Therefore, the position and orientation in space are determined by the information stored in the properties of that node object. An example of the properties of a node object is shown in B of FIG. 13.

[0057] <Time-varying camera object signals> Non-Patent Document 4 proposes changing the position, direction, angle of view, and other information of a camera object over time. This allows content creators to specify how to present a time-varying three-dimensional space. One method of achieving this is to define an extension 82 for a camera object 81, and to describe parameters that change over time, as shown in A of Figure 14.

[0058] An example of the description of camera object 81 in the scene description in this case is shown in B of Fig. 14. As shown in B of Fig. 14, in this case, an MPEG timed camera object (MPEG_timed_camera) that changes in the time direction is set in extension 82 of camera object 81, and the MPEG timed camera object expresses the changes in various camera-related parameters in the time direction.

[0059] In this MPEG timed camera object, for example, values of parameters such as those shown in FIG. 15 (e.g., frame_rate, frame_number, camera_position, camera_orientation, aspectRatio, yfov, zfar, znear, etc.) are described. frame_rate is a parameter indicating the frame rate of the camera. frame_number is a parameter indicating the frame number of the camera. camera_position is a parameter indicating the position of the camera. camera_orientation is a parameter indicating the orientation of the camera. aspectRatio is a parameter indicating the aspect ratio of the camera. yfov is a parameter indicating the vertical field of view (radians). zfar is a parameter indicating the distance from the camera to the rear clipping plane. znear is a parameter indicating the distance from the camera to the front clipping plane.

[0060] <Increasing complexity of processing> When handling metadata that changes over time in a scene description, such as information about a camera object whose parameter values change over time, the method of writing all of that metadata into the scene description as described above could lead to complicated processing.

[0061] For example, the synchronization mechanisms for the time-direction transformation metadata described in such a scene description and other timed media stored in an external file for the scene description and changing in the time direction are different. Therefore, when playing 3D object content, it was necessary to synchronize them. This meant that the playback process became complicated and the load on the playback process could increase.

[0062] Furthermore, when editing metadata that changes in the time direction (such as the above-mentioned MPEG timed camera object) described in the scene description, even if only a portion of the metadata is to be edited, it is necessary to read the entire metadata and perform the editing process as if it were the entire metadata. This means that the editing process becomes complicated and the load of the editing process may increase.

[0063] <2. Transmission of timed metadata, etc.> Therefore, as shown in the top row of the table in Fig. 16, metadata that changes over time is stored in timed metadata, which is data separate from the scene description. In the present disclosure, timed metadata related to the camera is also referred to as timed camera metadata.

[0064] By doing so, the timed metadata has the same structure as other timed media, and the mechanism for synchronizing the timed metadata is also the same as other timed media. Therefore, simply by acquiring the timed metadata information and the corresponding other timed media information at the same time, it is possible to easily synchronize the timed metadata with other timed media. This prevents the playback process from becoming too complicated and the playback process load from increasing.

[0065] Furthermore, since the timed metadata is configured as an external file of the scene description, it is easy to edit only a part of the timed metadata, which means that the editing process is not complicated and the load of the editing process is not increased.

[0066] <Method 1> In order to link such timed metadata configured as an external file to a scene description file, timed metadata identification information, which is identification information indicating that it is timed metadata (metadata that changes in the time direction), may be stored in an MPEG media object (MPEG_media object) (Method 1), as shown in the second row from the top of the table in Fig. 16. Also, timed metadata access information, which is information for accessing the timed metadata, may be stored in an extension of the camera object (Method 1).

[0067] For example, in an information processing method, timed metadata identification information indicating that the metadata of the linked external file changes in the time direction is stored in an MPEG media object (MPEG_media extension), and timed metadata access information linking a camera object to that metadata is stored in the camera object, and a scene description file describing a scene of 3D object content is generated.

[0068] For example, an information processing device may include a file generation unit that stores timed metadata identification information (timed metadata identification information) indicating that the metadata of a linked external file changes in the time direction in an MPEG media object (MPEG_media extension), stores timed metadata access information (timed metadata access information) that links a camera object to that metadata in the camera object, and generates a scene description file that describes a scene of 3D object content.

[0069] For example, in an information processing method, timed metadata that changes in the time direction is obtained based on timed media identification information (timed metadata identification information) stored in an MPEG media object (MPEG_media extension) of a scene description file that describes a scene of 3D object content and timed media access information (timed metadata access information) stored in a camera object of the scene description file, and a display image of the 3D object content is generated based on the obtained timed metadata.

[0070] For example, an information processing device may include an acquisition unit that acquires timed metadata that changes in the time direction based on timed media identification information (timed metadata identification information) stored in an MPEG media object (MPEG_media extension) of a scene description file that describes a scene of 3D object content and timed media access information (timed metadata access information) stored in a camera object of the scene description file, and a generation unit that generates a display image of the 3D object content based on the timed metadata acquired by the acquisition unit.

[0071] In this way, timed metadata can be linked to scene descriptions. Therefore, a device that plays back 3D object content can easily access the timed metadata based on the timed metadata identification information and the timed metadata access information. Therefore, as described above, a device that plays back 3D object content can prevent the playback process and editing process from becoming complicated and the load of these processes from increasing.

[0072] 17 is a diagram illustrating the linking of camera objects to be time-converted. As shown in Fig. 17, a scene description file 101 has a configuration as shown in the dotted line box. As described above with reference to Fig. 9 etc., a texture object can access an MPEG media object (i.e., timed texture data outside the scene description file) via an accessor object, a buffer view object, and a buffer object as shown in the solid line box 102.

[0073] Similarly, the camera object 81 can access MPEG media objects 121 (i.e., timed metadata outside of the scene description file) via an accessor object 111, a buffer view object 112, and a buffer object 113, as shown in solid box 103.

[0074] At this time, information about the external file is stored in an MPEG media object (MPEG_media object) to link the scene description file with the external file. Timed media identification information is stored in this MPEG media object (MPEG_media). This timed media identification information is identification information that indicates that the information in the external file (or a track within that file) is timed metadata for a camera object (camera).

[0075] For example, this timed media identification information may be stored in the mimeType of the alternatives array of the MPEG media object (MPEG_media extension) of the scene description file. When playing back 3D object content, timed metadata may be acquired based on the timed media identification information stored in the mimeType of the alternatives array of the MPEG media object (MPEG_media extension) of the scene description file.

[0076] Figure 18 is a diagram showing an example of the description of an MPEG media object (MPEG_media extension) in a scene description file. As shown in the eighth line from the top of Figure 18, in the mimeType of the alternative array, a sample entry (e.g., 'camp') of the timed metadata track is set in codecs. Note that the type indicating the timed metadata is arbitrary and is not limited to the above example (camp). Since a recommended view port (which stores metadata) specified in MPEG-I part 10 is used as the timed metadata of the camera object, the type of the sample entry indicating the recommended view port is stored, which indicates how the data should be interpreted.

[0077] Also, as shown in Figure 17, in order to indirectly link the camera object 81 to timed metadata (in the sequence of accessor object (accessor) 111 → buffer view object (bufferView) 112 → buffer object (buffer) 113 → MPEG media object (MPEG_media) 121), an MPEG timed camera object (MPEG_timed_camera) 122 is defined in the camera object 81, and the index of the accessor object 111 to the buffer object 113 in which the timed metadata is stored is referenced.

[0078] That is, the timed metadata access information may be stored in an MPEG timed camera object (MPEG_timed_camera extension) of the camera object of the scene description file. When playing back 3D object content, the timed metadata may be acquired based on the timed metadata access information stored in the MPEG timed camera object (MPEG_timed_camera extension) of the camera object of the scene description file.

[0079] 19 is a diagram showing an example of a description of a camera object (cameras) in a scene description file. As shown in lines 11 to 16 from the top of Fig. 19, an MPEG timed camera object 122 is defined.

[0080] 19, in this timed metadata access information, a plurality of parameters may be set, and for each of these parameters, an accessor object corresponding to an area of the buffer object to be referenced may be specified using an accessor index. Then, when playing back 3D object content, timed metadata may be acquired based on such timed metadata access information.

[0081] For example, as shown in FIG. 19, parameters such as POSISION and ORIENTATION may be defined in this timed metadata access information. The semantics of these parameters are shown in FIG. 20. As shown in FIG. 20, POSITION is a parameter that indicates the position of the camera. More specifically, a reference to an accessor that makes available time data of the camera position (data that indicates the position of the camera at each time) is provided as the value of this POSITION. ORIENTATION is a parameter that indicates the orientation of the camera. More specifically, a reference to an accessor that makes available time data of the camera orientation (data that indicates the orientation of the camera at each time) is provided as the value of this ORIENTATION.

[0082] That is, the parameters defined as the timed metadata access information may include a POSITION that indicates the position of the camera and an ORIENTATION that indicates the orientation of the camera. When playing back 3D object content, the timed metadata may be acquired based on the timed metadata access information.

[0083] In addition, in this timed metadata access information, PERSPECTIVE may be further defined as shown in Fig. 19. As shown in Fig. 20, PERSPECTIVE is a parameter indicating the camera's angle of view and perspective information. More specifically, a reference to an accessor that makes available time data of the camera's angle of view and perspective information (data indicating the camera's angle of view and perspective information at each time) is provided as the value of this PERSPECTIVE.

[0084] Note that the parameters defined in the timed metadata access information are arbitrary, and parameters other than the above examples may be defined.

[0085] For example, in FIG. 19, the value of POSITION is set to "0." This specifies the 0th element of the array in the accessor object. A in FIG. 21 is a diagram showing an example of how an accessor object is described in a scene description file. In the example of A in FIG. 21, the accessor object is configured as an array including elements 141, 142, and 143. In the example of FIG. 19, POSITION is linked to element 141 in A in FIG. 21. This element 141 stores information about area 153 of buffer view object 152 of buffer object 1151 shown in B in FIG. 21. In other words, POSITION is linked to this area 153.

[0086] Similarly, in Figure 19, the value of ORIENTATION is set to "1." This specifies the first element of the array in the accessor object. That is, ORIENTATION is associated with element 142 in Figure 21A, i.e., region 154 of buffer view object 152 of buffer object 1151 shown in Figure 21B.

[0087] Similarly, in Figure 19, the value of PERSPECTIVE is set to "2." This specifies the second element of the array in the accessor object. That is, PERSPECTIVE is associated with element 143 in Figure 21A, i.e., region 155 of buffer view object 152 of buffer object 1151 shown in Figure 21B.

[0088] As shown in A of Fig. 22, samples 161 of MP4 data delivered as timed metadata are aligned in the time direction. An example of the syntax of this sample is shown in B of Fig. 22. As shown in B of Fig. 22, parameter values such as position, orientation, and perspective are set in each sample. In other words, the values of these parameters can change in the time direction (for each sample).

[0089] <Method 1-1> As shown in the third row from the top of the table in Fig. 16, a single piece of timed camera metadata may be linked to a scene description (method 1-1). For example, in the case of Fig. 19, one MPEG timed camera object is defined in the camera object. In this way, a single piece of timed camera metadata can be linked to a scene description.

[0090] <Method 1-1-1> Furthermore, as shown in the fourth row from the top of the table in Fig. 16, the values of the above-mentioned multiple parameters may be collectively specified as one vector (Method 1-1-1). For example, the values of the above-mentioned three parameters, POSITION, ORIENTATION, and PERSPECTIVE, may be collectively specified as an 11-dimensional vector.

[0091] For example, the timed metadata access information may specify an accessor object corresponding to an area of a buffer object referenced by multiple parameters using a single vector, and when playing back 3D object content, the timed metadata may be acquired based on such timed metadata access information (i.e., using the single vector described above).

[0092] FIG. 23A shows an example of how a camera object and an accessor object are described in a scene description file. In the example shown in FIG. 23A, SAMPLEDATA is defined instead of POSITION, ORIENTATION, and PERSPECTIVE. This SAMPLEDATA is a parameter that collectively specifies the values of POSITION, ORIENTATION, and PERSPECTIVE. FIG. 23B shows the semantics of SAMPLEDATA. For example, if the value of POSITION is represented by a three-dimensional vector, the value of ORIENTATION is represented by a four-dimensional vector, and the value of PERSPECTIVE is represented by a four-dimensional vector, then the value of SAMPLEDATA is represented by an 11-dimensional vector combining these. In other words, the value of SAMPLEDATA includes the values of POSITION, ORIENTATION, and PERSPECTIVE, and the values of POSITION, ORIENTATION, and PERSPECTIVE can be derived from the value of SAMPLEDATA. Note that the 11-dimensional vector is not defined as a type, so it may be defined in the accessor object as shown in FIG. 23A.

[0093] <Method 1-1-2> Alternatively, the values of the above-mentioned multiple parameters may be collectively specified as an array (Method 1-1-2), as shown in the fifth row from the top of the table in Fig. 16. For example, the values of the above-mentioned three parameters, POSITION, ORIENTATION, and PERSPECTIVE, may be specified using an array whose elements are the values of these parameters.

[0094] For example, the timed metadata access information may specify an accessor object corresponding to an area of a buffer object referenced by multiple parameters using an array whose elements are accessor indices corresponding to each parameter. When playing back 3D object content, timed metadata may be acquired based on such timed metadata access information (i.e., using the above-mentioned array).

[0095] FIG. 24A shows an example of how a camera object and an accessor object are described in a scene description file. In the example shown in FIG. 24A, SAMPLEDATA is defined instead of POSITION, ORIENTATION, and PERSPECTIVE. This SAMPLEDATA is a parameter whose value is an array whose elements are the values of the above-mentioned POSITION, ORIENTATION, and PERSPECTIVE. FIG. 24B shows the semantics of this SAMPLEDATA. That is, the values of POSITION, ORIENTATION, and PERSPECTIVE are provided as elements of this SAMPLEDATA array. In other words, in this case too, the value of SAMPLEDATA includes the values of POSITION, ORIENTATION, and PERSPECTIVE, and the values of POSITION, ORIENTATION, and PERSPECTIVE can be derived from the value of SAMPLEDATA. Note that because a vector array is not defined as a type, it may be defined in the accessor object as shown in FIG. 24A.

[0096] <Method 1-2> As shown in the sixth row from the top of the table in Fig. 16, multiple pieces of timed camera metadata may be linked to a scene description (method 1-2). For example, as shown in Fig. 25, multiple cameras (camera 201 and camera 204) may be set in a three-dimensional space 200, and the timed camera metadata of each camera may be linked to a scene description. In the example of Fig. 25, camera 201 photographs 3D object 203 while moving as indicated by dotted arrow 202. Camera 204 photographs 3D object 206 while moving as indicated by dotted arrow 205.

[0097] In such a case, if the method of describing timed metadata in the scene description is applied, the timed metadata of cameras not selected by the user will also be transmitted (acquired) together with the scene description.

[0098] By linking the timed camera metadata of multiple cameras to the scene description in this way, when generating a display image, the viewpoint of the display image can be selected from multiple candidates. In other words, it is possible to transmit (acquire) only the necessary data (timed metadata of the cameras selected by the user).

[0099] The method for selecting the camera (timed camera metadata) is arbitrary. For example, the camera may be selected by a user or may be selected based on the hardware performance of the device that performs the playback process.

[0100] <Method 1-2-1> In this way, when linking timed camera metadata from multiple cameras to a scene description, it may be possible to specify multiple MPEG media objects in the timed metadata access information, as shown in the seventh row from the top of the table in Figure 16, for example.

[0101] For example, the timed metadata access information may be configured to specify multiple accessor objects corresponding to the area of the buffer object to be referenced. When playing back 3D object content, the timed metadata may be acquired based on the timed metadata access information (i.e., using one of the multiple specified accessor objects).

[0102] A of Fig. 26 is a diagram showing an example of description of a camera object (MPEG timed camera object) in this case. B of Fig. 26 is a diagram showing an example of description of an MPEG media object in this case. As shown in B of Fig. 26, MPEG media object 221 and MPEG media object 222 are set in the MPEG media object. MPEG media object 221 and MPEG media object 222 specify different timed camera metadata from each other. In other words, in this case, multiple MPEG media objects are prepared.

[0103] On the other hand, as shown in A of Fig. 26, two MPEG timed camera objects are set in the extension of the camera object. The first MPEG timed camera object specifies MPEG media object 221, and the second MPEG timed camera object specifies MPEG media object 222. In other words, the camera object is (indirectly) linked to multiple MPEG media objects.

[0104] In other words, the MPEG timed camera object is defined so that it can be handled as an array, and multiple MPEG media objects can be specified. This allows each media to be allocated to a buffer, and the accessor to each is referenced from the MPEG timed camera object (MPEG_timed_camera object). Each media object can specify a different timed metadata file, or it can specify a different track of the same timed metadata file.

[0105] In this way, timed camera metadata from multiple cameras can be linked to a scene description.

[0106] <Method 1-2-2> In addition, when linking timed camera metadata from multiple cameras to a scene description, an MPEG media object may be able to store multiple camera metadata, as shown in the bottom row of the table in Figure 16, for example.

[0107] For example, an MPEG media object (MPEG_media extension) may store information about multiple pieces of timed metadata that change in the time direction as different elements of an alternative array, and when playing back 3D object content, the timed metadata may be retrieved based on such timed metadata access information.

[0108] A of Fig. 27 is a diagram showing an example of how a camera object (MPEG timed camera object) is described in this case. B of Fig. 27 is a diagram showing an example of how an MPEG media object is described in this case. As shown in B of Fig. 27, timed camera metadata 231 and timed camera metadata 232 are set in the MPEG media object using an alternative arrangement.

[0109] On the other hand, as shown in A of FIG. 27, one MPEG timed camera object is set in the extension of the camera object.

[0110] In other words, only one MPEG timed camera object (MPEG_timed_camera object) is stored. Also, an MPEG media object (MPEG_media) may have one media object, within which multiple alternatives objects may be specified. To identify the contents of the alternatives array (alternatives), a name property may be added to the alternative object as an identifying label. This alternative array may specify different timed metadata files, or may specify different tracks of the same timed metadata file.

[0111] In this way, timed camera metadata from multiple cameras can be linked to a scene description.

[0112] 3. First Embodiment <File generation device> The techniques of the present technology described above can be applied to any device. Fig. 28 is a block diagram showing an example of the configuration of a file generation device, which is one aspect of an information processing device to which the present technology is applied. The file generation device 300 shown in Fig. 28 is a device that generates information for distributing 3D object content. For example, the file generation device 300 generates a 3D object content file to be distributed and generates a scene description file (scene description) for the 3D object content.

[0113] Note that Fig. 28 shows the main processing units, data flows, etc., and is not necessarily all that is shown in Fig. 28. In other words, in file generation device 300, there may be processing units that are not shown as blocks in Fig. 28, and there may be processing or data flows that are not shown as arrows, etc. in Fig. 28.

[0114] 28 , file generation device 300 has a control unit 301 and a file generation processing unit 302. Control unit 301 controls file generation processing unit 302. File generation processing unit 302 is controlled by control unit 301 to perform processing related to file generation. For example, file generation processing unit 302 generates a 3D object content file to be distributed. Furthermore, file generation processing unit 302 generates a scene description file corresponding to the 3D object content file. File generation processing unit 302 outputs the generated file to the outside of file generation device 300.

[0115] The file generation processing unit 302 includes an input unit 311 , a preprocessing unit 312 , an encoding unit 313 , a file generation unit 314 , a recording unit 315 , and an output unit 316 .

[0116] The input unit 311 acquires 3D object content and supplies it to the preprocessing unit 312. The preprocessing unit 312 extracts information necessary for file generation from the 3D object content. The preprocessing unit 312 supplies the extracted information to the file generation unit 314. The preprocessing unit 312 also supplies the 3D object content to the encoding unit 313.

[0117] The encoding unit 313 encodes the 3D object content supplied from the preprocessing unit 312 to generate encoded data (bit stream). The encoding unit 313 supplies the generated encoded data of the 3D object content to the file generation unit 314.

[0118] The file generation unit 314 acquires the coded data of the 3D object content supplied from the coding unit 313. The file generation unit 314 also acquires the information supplied from the pre-processing unit 312.

[0119] The file generation unit 314 generates a 3D object content file that stores the encoded data of the acquired 3D object content. The file generation unit 314 generates the file by appropriately using the information supplied from the pre-processing unit 312. For example, the file generation unit 314 stores the information supplied from the pre-processing unit 312 in the 3D object content file.

[0120] The file generation unit 314 also generates a scene description file corresponding to the generated 3D object content file. At this time, the file generation unit 314 generates the scene description file by applying the present technology described above in <2. Transmission of timed metadata, etc.>. The file generation unit 314 may apply any one or more of the techniques of the present technology described above in <2. Transmission of timed metadata, etc.>.

[0121] For example, the file generation unit 314 generates a scene description file that describes a scene of the 3D object content, storing timed metadata identification information indicating that the metadata of the linked external file changes in the time direction in an MPEG media object (MPEG_media extension), and storing timed metadata access information in the camera object that links the camera object to that metadata.

[0122] At this time, the file generation unit 314 may store the timed metadata identification information in the mimeType of the alternative array of the MPEG media object (MPEG_media extension) of the scene description file.

[0123] The file generator 314 may also store the timed metadata access information in an MPEG timed camera object (MPEG_timed_camera extension) of the camera object in the scene description file.

[0124] Furthermore, the file generation unit 314 may use an accessor index to specify an accessor object corresponding to an area of the buffer object to be referenced for each of a plurality of parameters in the timed metadata access information.

[0125] In addition, the file generation unit 314 may use an accessor index to specify an accessor object corresponding to the area of the buffer object to be referenced for each of multiple parameters: POSITION, which indicates the position of the camera object; ORIENTATION, which indicates the orientation of the camera object; and PERSPECTIVE, which indicates the angle of view and perspective information of the camera object.

[0126] Furthermore, the file generation unit 314 may use one vector to specify the accessor objects corresponding to the areas of the buffer object referenced by multiple parameters in the timed metadata access information.

[0127] Furthermore, the file generation unit 314 may specify an accessor object corresponding to an area of a buffer object referenced by a plurality of parameters in the timed metadata access information by using an array whose elements are accessor indices corresponding to the respective parameters.

[0128] Furthermore, the file generation unit 314 may specify, in the timed metadata access information, a plurality of accessor objects corresponding to the area of the buffer object to be referenced.

[0129] Furthermore, the file generation unit 314 may store information about multiple pieces of timed metadata that change in the time direction as different elements of an alternative array in an MPEG media object (MPEG_media extension).

[0130] File generation unit 314 supplies the generated file to recording unit 315. Recording unit 315 has an arbitrary recording medium such as a hard disk or semiconductor memory, and records the file supplied from file generation unit 314 on that recording medium. Furthermore, recording unit 315 reads out the file recorded on the recording medium in accordance with a request from control unit 301 or output unit 316, or at a predetermined timing, and supplies the file to output unit 316.

[0131] The output unit 316 acquires the file supplied from the recording unit 315, and outputs the file to an external device outside the file generation device 300 (for example, a distribution server, a playback device, etc.).

[0132] With this configuration, the file generation device 300 can link timed metadata to scene descriptions. Therefore, a device that plays back 3D object content can easily access the timed metadata based on the timed metadata identification information and the timed metadata access information. Therefore, as described above, a device that plays back 3D object content can prevent the playback process and editing process from becoming complicated and the load of those processes from increasing.

[0133] <File generation process flow> An example of the flow of file generation processing executed by file generation device 300 in FIG. 28 will be described with reference to the flowchart in FIG.

[0134] When the file generation process is started, in step S301, the input unit 311 of the file generation device 300 acquires 3D object content, which is data of a 3D object.

[0135] In step S302, the preprocessing unit 312 extracts information to be stored in a file from the 3D object content obtained in step S301.

[0136] In step S303, the preprocessing unit 312 generates timed camera metadata based on the 3D object content.

[0137] In step S304, the preprocessing unit 312 generates timed metadata identification information and timed metadata access information based on the 3D object content.

[0138] In step S305, the encoding unit 313 encodes the 3D object content and the timed camera metadata.

[0139] In step S306, the file generation unit 314 generates a scene description file including timed metadata identification information and timed metadata access information. The file generation unit 314 also generates a file storing encoded data of 3D object content. The file generation unit 314 also generates a file storing encoded data of timed camera metadata. In this case, the file generation unit 314 generates the scene description file by applying the present technology described above in <2. Transmission of timed metadata, etc.>. The file generation unit 314 may apply any one or more of the techniques of the present technology described above in <2. Transmission of timed metadata, etc.>.

[0140] For example, the file generation unit 314 generates a scene description file that describes a scene of the 3D object content, storing timed metadata identification information indicating that the metadata of the linked external file changes in the time direction in an MPEG media object (MPEG_media extension), and storing timed metadata access information in the camera object that links the camera object to that metadata.

[0141] At this time, the file generation unit 314 may store the timed metadata identification information in the mimeType of the alternative array of the MPEG media object (MPEG_media extension) of the scene description file.

[0142] The file generator 314 may also store the timed metadata access information in an MPEG timed camera object (MPEG_timed_camera extension) of the camera object in the scene description file.

[0143] Furthermore, the file generation unit 314 may use an accessor index to specify an accessor object corresponding to an area of the buffer object to be referenced for each of a plurality of parameters in the timed metadata access information.

[0144] In addition, the file generation unit 314 may use an accessor index to specify an accessor object corresponding to the area of the buffer object to be referenced for each of multiple parameters: POSITION, which indicates the position of the camera object; ORIENTATION, which indicates the orientation of the camera object; and PERSPECTIVE, which indicates the angle of view and perspective information of the camera object.

[0145] Furthermore, the file generation unit 314 may use one vector to specify the accessor objects corresponding to the areas of the buffer object referenced by multiple parameters in the timed metadata access information.

[0146] Furthermore, the file generation unit 314 may specify an accessor object corresponding to an area of a buffer object referenced by a plurality of parameters in the timed metadata access information by using an array whose elements are accessor indices corresponding to the respective parameters.

[0147] Furthermore, the file generation unit 314 may specify, in the timed metadata access information, a plurality of accessor objects corresponding to the area of the buffer object to be referenced.

[0148] Furthermore, the file generation unit 314 may store information about multiple pieces of timed metadata that change in the time direction as different elements of an alternative array in an MPEG media object (MPEG_media extension).

[0149] In step S307, the recording unit 315 records the file generated in step S306 on a recording medium.

[0150] In step S308, output unit 316 reads the file recorded in step S307 from the recording medium, and outputs the read file to the outside of file generation device 300 at a predetermined timing.

[0151] When the process of step S308 ends, the file generation process ends.

[0152] By performing each process as described above, the file generation device 300 can link the timed metadata to the scene description. Therefore, a device that plays back 3D object content can easily access the timed metadata based on the timed metadata identification information and the timed metadata access information. Therefore, as described above, a device that plays back 3D object content can prevent the playback process and editing process from becoming complicated and the load of these processes from increasing.

[0153] 4. Second Embodiment <Client device> Fig. 30 is a block diagram showing an example of the configuration of a client device, which is one aspect of an information processing device to which the present technology is applied. The client device 400 shown in Fig. 30 is a playback device that performs playback processing of 3D object content based on a scene description file. For example, the client device 400 plays back 3D object content that is stored in a 3D object content file generated by the file generation device 300. For example, the client device 400 decodes encoded data of the 3D object content stored in the 3D object content file based on the scene description file, renders the generated 3D object content, and generates and displays a display image.

[0154] Note that Fig. 30 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, in client device 400, there may be processing units that are not shown as blocks in Fig. 30, and there may be processing or data flows that are not shown as arrows, etc. in Fig. 30.

[0155] As shown in FIG. 30 , the client device 400 has a control unit 401 and a playback processing unit 402. The control unit 401 performs processing related to controlling the playback processing unit 402. The playback processing unit 402 performs processing related to playing back 3D object content stored in a 3D object content file. For example, the playback processing unit 402 is controlled by the control unit 401 to acquire a scene description file from a distribution server (not shown) or the like. Based on the scene description file, the playback processing unit 402 acquires a 3D object content file that stores the 3D object content to be played back. Then, the playback processing unit 402 performs playback processing of the 3D object content stored in the acquired 3D object content file.

[0156] The playback processing unit 402 includes a file acquisition unit 411 , a file processing unit 412 , a decoding unit 413 , a display information generation unit 414 , a display unit 415 , and a display control unit 416 .

[0157] The file acquisition unit 411 acquires a scene description file supplied from outside the client device 400, such as a distribution server or the file generation device 300. The file acquisition unit 411 supplies the acquired scene description file to the file processing unit 412.

[0158] The file acquisition unit 411 also acquires a 3D object content file that stores the 3D object content to be played back, under the control of the file processing unit 412 using the scene description file. The file acquisition unit 411 supplies the acquired 3D object content file to the file processing unit 412.

[0159] The file processing unit 412 acquires the scene description file supplied from the file acquisition unit 411. The file processing unit 412 selects 3D object content to be played back based on the acquired scene description file. The file processing unit 412 then controls the file acquisition unit 411 to acquire a 3D object content file that stores the selected 3D object content.

[0160] In such processing, the file processing unit 412 applies the present technology described above in <2. Transmission of timed metadata, etc.>. The file processing unit 412 may apply any one or more of the techniques of the present technology described above in <2. Transmission of timed metadata, etc.>.

[0161] For example, the file processing unit 412 acquires timed metadata that changes in the time direction based on timed metadata identification information stored in an MPEG media object (MPEG_media extension) of a scene description file that describes a scene of the 3D object content and timed metadata access information stored in a camera object of the scene description file, and generates a display image of the 3D object content based on the acquired timed metadata.

[0162] At this time, the file processing unit 412 may acquire the timed metadata based on the timed metadata identification information stored in the mimeType of the alternative array of the MPEG media object (MPEG_media extension) of the scene description file.

[0163] Furthermore, the file processing unit 412 may acquire timed metadata based on timed metadata access information stored in the MPEG timed camera object (MPEG_timed_camera extension) of the camera object of the scene description file.

[0164] Furthermore, the file processing unit 412 may acquire timed metadata based on timed metadata access information that specifies, for each of a plurality of parameters, an accessor object corresponding to the area of the buffer object being referenced using an accessor index.

[0165] In addition, the file processing unit 412 may acquire timed metadata based on timed metadata access information that specifies, using an accessor index, an accessor object corresponding to the area of the buffer object to be referenced for each of multiple parameters: POSITION, which indicates the position of the camera object; ORIENTATION, which indicates the orientation of the camera object; and PERSPECTIVE, which indicates the angle of view and perspective information of the camera object.

[0166] Furthermore, the file processing unit 412 may acquire timed metadata based on timed metadata access information that specifies, by one vector, an accessor object corresponding to an area of a buffer object referenced by multiple parameters.

[0167] In addition, the file processing unit 412 may acquire timed metadata based on timed metadata access information that specifies an accessor object corresponding to an area of a buffer object referenced by multiple parameters using an array whose elements are accessor indices corresponding to each parameter.

[0168] Furthermore, the file processing unit 412 may acquire timed metadata based on timed metadata access information that specifies multiple accessor objects corresponding to the area of the buffer object to be referenced.

[0169] In addition, the file processing unit 412 may acquire timed metadata using an MPEG media object (MPEG_media extension) that stores information about multiple pieces of timed metadata that change in the time direction as different elements of an alternative array.

[0170] The file processing unit 412 supplies the 3D object content file acquired as described above to the decoding unit 413 and the display control unit 416.

[0171] The decoding unit 413 decodes the coded data of the 3D object content stored in the 3D object content file supplied from the file processing unit 412. In other words, the decoding unit 413 decodes the coded data of the 3D object content selected by the file processing unit 412. The decoding unit 413 supplies the 3D object content obtained by the decoding to the display information generation unit 414.

[0172] The display information generation unit 414 acquires the 3D object content supplied from the decoding unit 413. The display information generation unit 414 also acquires control information supplied from the display control unit 416. Then, the display information generation unit 414 generates a display image or the like from the acquired 3D object content in accordance with the control information. The display information generation unit 414 supplies the generated display image or the like to the display unit 415.

[0173] The display unit 415 has a display device and displays the display image supplied from the display information generation unit 414 using the display device.

[0174] The display control unit 416 acquires information supplied from the file processing unit 412. Based on the information, the display control unit 416 controls the display information generation unit 414. The display control unit 416 supplies control information to the display information generation unit 414, thereby controlling the display of the display image.

[0175] With this configuration, the client device 400 can easily access the timed metadata based on the timed metadata identification information and the timed metadata access information. Therefore, as described above, the client device 400 can prevent the playback process and editing process from becoming complicated and can prevent an increase in the load of these processes.

[0176] <Client processing flow 1> An example of the flow of client processing executed by client device 400 in Fig. 30 will be described with reference to the flowchart in Fig. 31. The flowchart shown in Fig. 31 illustrates an example of the flow of client processing in the case of method 1-1 (including method 1-1-1 and method 1-1-2).

[0177] When the client process starts, the file acquisition unit 411 of the client device 400 acquires a scene description file in step S401.

[0178] In step S402, the file processing unit 412 analyzes the scene description file acquired in step S401, acquires timed camera metadata associated with the MPEG timed camera based on the timed metadata identification information and the timed metadata access information, and conceptually stores the data in a buffer.

[0179] For example, the file processing unit 412 acquires timed metadata that changes in the time direction based on timed metadata identification information stored in an MPEG media object (MPEG_media extension) of a scene description file that describes a scene of the 3D object content and timed metadata access information stored in a camera object of the scene description file, and generates a display image of the 3D object content based on the acquired timed metadata.

[0180] At this time, the file processing unit 412 may acquire the timed metadata based on the timed metadata identification information stored in the mimeType of the alternative array of the MPEG media object (MPEG_media extension) of the scene description file.

[0181] Furthermore, the file processing unit 412 may acquire timed metadata based on timed metadata access information stored in the MPEG timed camera object (MPEG_timed_camera extension) of the camera object of the scene description file.

[0182] Furthermore, the file processing unit 412 may acquire timed metadata based on timed metadata access information that specifies, for each of a plurality of parameters, an accessor object corresponding to the area of the buffer object being referenced using an accessor index.

[0183] In addition, the file processing unit 412 may acquire timed metadata based on timed metadata access information that specifies, using an accessor index, an accessor object corresponding to the area of the buffer object to be referenced for each of multiple parameters: POSITION, which indicates the position of the camera object; ORIENTATION, which indicates the orientation of the camera object; and PERSPECTIVE, which indicates the angle of view and perspective information of the camera object.

[0184] Furthermore, the file processing unit 412 may acquire timed metadata based on timed metadata access information that specifies, by one vector, an accessor object corresponding to an area of a buffer object referenced by multiple parameters.

[0185] In addition, the file processing unit 412 may acquire timed metadata based on timed metadata access information that specifies an accessor object corresponding to an area of a buffer object referenced by multiple parameters using an array whose elements are accessor indices corresponding to each parameter.

[0186] Furthermore, the file processing unit 412 may acquire timed metadata based on timed metadata access information that specifies multiple accessor objects corresponding to the area of the buffer object to be referenced.

[0187] In addition, the file processing unit 412 may acquire timed metadata using an MPEG media object (MPEG_media extension) that stores information about multiple pieces of timed metadata that change in the time direction as different elements of an alternative array.

[0188] In step S403, the file acquisition unit 411 acquires other media.

[0189] In step S404, the file processing unit 412 reads out the data conceptually stored in the buffer by the accessor in the MPEG timed camera in synchronization with other media. The decoding unit 413 decodes the read encoded data of the 3D object content, etc.

[0190] In step S405, the display information generation unit 414 places other media in the scene and generates a display image based on the timed camera information.

[0191] In step S406, the display unit 415 displays the display image.

[0192] When the process of step S406 ends, the client process ends.

[0193] By executing each process as described above, the client device 400 can easily access the timed metadata based on the timed metadata identification information and the timed metadata access information. Therefore, as described above, the client device 400 can prevent the playback process and editing process from becoming complicated and can prevent an increase in the load of these processes.

[0194] <Client processing flow 2> An example of the flow of client processing executed by client device 400 in Fig. 30 will be described with reference to the flowchart in Fig. 32. The flowchart shown in Fig. 32 shows an example of the flow of client processing in the case of method 1-2-1.

[0195] When the client process starts, the file acquisition unit 411 of the client device 400 acquires a scene description file in step S441.

[0196] In step S442, the file processing unit 412 analyzes the timed metadata identification information and the timed metadata access information of the scene description file acquired in step S441.

[0197] Then, the file processing unit 412 acquires the timed metadata based on the timed metadata identification information and the timed metadata access information. The timed metadata identification information may be stored in the mimeType of the alternative array of the MPEG media object (MPEG_media extension) of the scene description file. The timed metadata access information may be stored in the MPEG timed camera object (MPEG_timed_camera extension) of the camera object of the scene description file.

[0198] In this case, the file processing unit 412 can acquire the timed metadata based on the timed metadata access information that specifies a plurality of accessor objects corresponding to the area of the buffer object to be referenced. That is, the process is executed as follows.

[0199] In step S443, the file processing unit 412 determines whether there are multiple MPEG timed camera objects based on the analysis result of step S442. That is, the file processing unit 412 determines whether the timed metadata access information specifies multiple accessor objects. If it is determined that there are multiple MPEG timed camera objects (that is, the timed metadata access information specifies multiple accessor objects), as in the example of A in Fig. 26, the processing proceeds to step S444.

[0200] In step S444, file processing unit 412 presents the names of MPEG-timed cameras to the user for selection. When the processing of step S444 ends, the processing proceeds to step S445. Also, if it is determined in step S443 that there is a single MPEG-timed camera, the processing proceeds to step S445.

[0201] In step S445, the file acquisition unit 411 acquires timed metadata associated with the MPEG timed camera selected by the user in step S444, or timed metadata associated with a single MPEG timed camera, and conceptually stores the data in a buffer.

[0202] In this case, the file processing unit 412 may acquire the timed metadata based on timed metadata access information that specifies, for each of the multiple parameters, an accessor object corresponding to the area of the buffer object being referenced using an accessor index.

[0203] In addition, the file processing unit 412 may acquire timed metadata based on timed metadata access information that specifies, using an accessor index, an accessor object corresponding to the area of the buffer object to be referenced for each of multiple parameters: POSITION, which indicates the position of the camera object; ORIENTATION, which indicates the orientation of the camera object; and PERSPECTIVE, which indicates the angle of view and perspective information of the camera object.

[0204] Furthermore, the file processing unit 412 may acquire timed metadata based on timed metadata access information that specifies, by one vector, an accessor object corresponding to an area of a buffer object referenced by multiple parameters.

[0205] In addition, the file processing unit 412 may acquire timed metadata based on timed metadata access information that specifies an accessor object corresponding to an area of a buffer object referenced by multiple parameters using an array whose elements are accessor indices corresponding to each parameter.

[0206] The processes of steps S446 to S449 are executed in the same manner as the processes of steps S403 to S406 in Fig. 31. When the process of step S449 ends, the client process ends.

[0207] By executing each process as described above, the client device 400 can easily access the timed metadata based on the timed metadata identification information and the timed metadata access information. Therefore, as described above, the client device 400 can prevent the playback process and editing process from becoming complicated and can prevent an increase in the load of these processes.

[0208] <Client processing flow 3> An example of the flow of client processing executed by client device 400 in Fig. 30 will be described with reference to the flowchart in Fig. 33. The flowchart shown in Fig. 33 shows an example of the flow of client processing in the case of method 1-2-2.

[0209] When the client process starts, the file acquisition unit 411 of the client device 400 acquires a scene description file in step S481.

[0210] In step S482, the file processing unit 412 analyzes the timed metadata identification information and the timed metadata access information of the scene description file acquired in step S481.

[0211] Then, the file processing unit 412 acquires the timed metadata based on the timed metadata identification information and the timed metadata access information. The timed metadata identification information may be stored in the mimeType of the alternative array of the MPEG media object (MPEG_media extension) of the scene description file. The timed metadata access information may be stored in the MPEG timed camera object (MPEG_timed_camera extension) of the camera object of the scene description file.

[0212] In this case, the file processing unit 412 may acquire the timed metadata using an MPEG media object (MPEG_media extension) that stores information about multiple pieces of timed metadata that change in the time direction as different elements of an alternative array. That is, the process is performed as follows.

[0213] In step S483, the file processing unit 412 determines whether there are multiple timed camera media alternative objects based on the analysis result of step S482. That is, the file processing unit 412 determines whether the MPEG media object stores information about multiple timed metadata as different elements of the alternative array. If it is determined that there are multiple timed camera metadata specified in the MPEG media object, as in the example of B in Fig. 27, the processing proceeds to step S484.

[0214] In step S484, the file processing unit 412 presents the name of each element of the alternative array to the user for selection. That is, the file processing unit 412 prompts the user to select timed metadata (timed camera metadata). When the processing of step S484 ends, the processing proceeds to step S485. Also, if it is determined in step S483 that there is a single media alternative object for the timed camera, the processing proceeds to step S485.

[0215] In step S485, the file acquisition unit 411 acquires the timed metadata selected by the user in step S484 or a single piece of timed metadata, and conceptually stores the data in a buffer.

[0216] In this case, the file processing unit 412 may acquire the timed metadata based on timed metadata access information that specifies, for each of the multiple parameters, an accessor object corresponding to the area of the buffer object being referenced using an accessor index.

[0217] In addition, the file processing unit 412 may acquire timed metadata based on timed metadata access information that specifies, using an accessor index, an accessor object corresponding to the area of the buffer object to be referenced for each of multiple parameters: POSITION, which indicates the position of the camera object; ORIENTATION, which indicates the orientation of the camera object; and PERSPECTIVE, which indicates the angle of view and perspective information of the camera object.

[0218] Furthermore, the file processing unit 412 may acquire timed metadata based on timed metadata access information that specifies, by one vector, an accessor object corresponding to an area of a buffer object referenced by multiple parameters.

[0219] In addition, the file processing unit 412 may acquire timed metadata based on timed metadata access information that specifies an accessor object corresponding to an area of a buffer object referenced by multiple parameters using an array whose elements are accessor indices corresponding to each parameter.

[0220] The processes of steps S486 to S489 are executed in the same manner as the processes of steps S403 to S406 in Fig. 31. When the process of step S489 ends, the client process ends.

[0221] By executing each process as described above, the client device 400 can easily access the timed metadata based on the timed metadata identification information and the timed metadata access information. Therefore, as described above, the client device 400 can prevent the playback process and editing process from becoming complicated and can prevent an increase in the load of these processes.

[0222] <5. Notes> <Computer> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, etc., that can execute various functions by installing various programs.

[0223] FIG. 34 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0224] In a computer 900 shown in FIG. 34, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0225] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0226] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0227] In a computer configured as above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0228] The program executed by the computer can be applied by recording it on removable media 921 such as package media, for example. In this case, the program can be installed in storage unit 913 via input / output interface 910 by inserting removable media 921 into drive 915.

[0229] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0230] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0231] <Applicable targets of this technology> This technology can be applied to any image encoding / decoding method.

[0232] Furthermore, the present technology can be applied to any configuration, for example, various electronic devices.

[0233] Furthermore, for example, the present technology can also be implemented as a part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, or a set in which other functions are added to a unit (e.g., a video set).

[0234] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, AV (Audio Visual) equipment, a portable information processing terminal, or an IoT (Internet of Things) device.

[0235] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0236] <Fields and applications where this technology can be applied> Systems, devices, processing units, etc. to which the present technology is applied can be used in any field, such as transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. In addition, the applications thereof are also arbitrary.

[0237] For example, the present technology can be applied to systems and devices used to provide viewing content, etc. Furthermore, for example, the present technology can also be applied to systems and devices used for transportation, such as monitoring traffic conditions and controlling automatic driving. Furthermore, for example, the present technology can also be applied to systems and devices used for security. Furthermore, for example, the present technology can also be applied to systems and devices used for automatic control of machines, etc. Furthermore, for example, the present technology can also be applied to systems and devices used for agriculture and livestock farming. Furthermore, for example, the present technology can also be applied to systems and devices used to monitor natural conditions, such as volcanoes, forests, and oceans, and wildlife. Furthermore, for example, the present technology can also be applied to systems and devices used for sports.

[0238] <Other> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. In other words, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be assumed not only to include the identification information in the bit stream, but also to include difference information of the identification information relative to certain reference information in the bit stream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also difference information relative to the reference information.

[0239] Furthermore, various types of information (metadata, etc.) related to the coded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. In other words, data associated with each other may be combined into one piece of data or may be individual pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Also, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of the same recording medium). Note that this "association" may refer to only a portion of the data, rather than the entire data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0240] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0241] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0242] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0243] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0244] Also, for example, each step of a single flowchart may be executed by one device, or may be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes may be executed by one device, or may be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0245] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0246] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0247] The present technology can also be configured as follows. (1) A file generation unit that generates a scene description file that describes a scene of 3D object content, storing timed metadata identification information indicating that metadata of a linked external file changes in the time direction in an MPEG_media extension and storing timed metadata access information that links a camera object to the metadata in the camera object. An information processing device comprising: (2) The scene description file stores the timed metadata identification information in the mimeType of the alternatives array of the MPEG_media extension. An information processing device according to (1). (3) The scene description file stores the timed metadata access information in the MPEG_timed_camera extension of the camera object. An information processing device according to (1) or (2). (4) The timed metadata access information specifies, for each of a plurality of parameters, an accessor object corresponding to the area of the buffer object to be referenced, using an accessor index. (3) An information processing device according to the present invention. (5) The plurality of parameters include a POSITION indicating the position of the camera object, an ORIENTATION indicating the orientation of the camera object, and a PERSPECTIVE indicating the angle of view and perspective information of the camera object. (4) An information processing device according to the present invention. (6) The timed metadata access information specifies, by a single vector, an accessor object corresponding to an area of a buffer object referenced by multiple parameters. An information processing device according to (4) or (5). (7) The timed metadata access information specifies an accessor object corresponding to an area of a buffer object referenced by a plurality of parameters by an array having the accessor index corresponding to each parameter as an element. An information processing device according to (4) or (5). (8) The timed metadata access information specifies a plurality of accessor objects corresponding to the area of the buffer object to be referenced. An information processing device according to any one of (3) to (7). (9) The MPEG_media extension stores information about multiple timed metadata that change in the time direction as different elements of an alternative array. An information processing device according to any one of (1) to (7). (10) A scene description file is generated that describes a scene of the 3D object content, storing timed metadata identification information, which indicates that the metadata of the linked external file changes in the time direction, in the MPEG_media extension, and storing timed metadata access information, which links the camera object to the metadata, in the camera object. Information processing methods.

[0248] (11) An acquisition unit that acquires timed metadata that changes in the time direction based on timed metadata identification information stored in the MPEG_media extension of a scene description file that describes a scene of 3D object content and timed metadata access information stored in the camera object of the scene description file; a generation unit that generates a display image of the 3D object content based on the timed metadata acquired by the acquisition unit; An information processing device comprising: (12) The acquisition unit acquires the timed metadata based on the timed metadata identification information stored in the mimeType of the alternatives array of the MPEG_media extension of the scene description file. (11) An information processing device according to (11). (13) The acquisition unit acquires the timed metadata based on the timed metadata access information stored in the MPEG_timed_camera extension of the camera object in the scene description file. The information processing device according to (11) or (12). (14) The acquisition unit acquires the timed metadata based on the timed metadata access information that specifies, for each of a plurality of parameters, an accessor object corresponding to an area of the buffer object to be referenced using an accessor index. (13) An information processing device according to (13). (15) The acquisition unit acquires the timed metadata based on the timed metadata access information that specifies, using an accessor index, an accessor object corresponding to an area of a buffer object to be referenced for each of POSITION indicating the position of the camera object, ORIENTATION indicating the orientation of the camera object, and PERSPECTIVE indicating the angle of view and perspective information of the camera object. (14) An information processing device according to (14). (16) The acquisition unit acquires the timed metadata based on the timed metadata access information that specifies, by one vector, an accessor object corresponding to an area of a buffer object referenced by a plurality of parameters. The information processing device according to (14) or (15). (17) The acquisition unit acquires the timed metadata based on the timed metadata access information, which specifies an accessor object corresponding to an area of a buffer object referenced by a plurality of parameters by an array having the accessor index corresponding to each parameter as an element. The information processing device according to (14) or (15). (18) The acquisition unit acquires the timed metadata based on the timed metadata access information that specifies a plurality of accessor objects corresponding to an area of the buffer object to be referenced. An information processing device according to any one of (13) to (17). (19) The acquisition unit acquires any one of a plurality of pieces of information about timed metadata that change in the time direction and are stored as different elements of an alternative array in the MPEG_media extension. An information processing device according to any one of (11) to (17). (20) acquiring timed metadata that changes in the time direction based on timed metadata identification information stored in the MPEG_media extension of a scene description file that describes a scene of the 3D object content and timed metadata access information stored in the camera object of the scene description file; generating a display image of the 3D object content based on the acquired timed metadata; Information processing methods. [Explanation of symbols]

[0249] 300 file generation device, 301 control unit, 302 file generation processing unit, 311 input unit, 312 preprocessing unit, 313 encoding unit, 314 file generation unit, 315 recording unit, 316 output unit, 400 client device, 401 control unit, 402 client processing unit, 411 file acquisition unit, 412 file processing unit, 413 decoding unit, 414 display information generation unit, 415 display unit, 416 display control unit

Claims

1. a file generation unit that generates a scene description file that describes a scene of 3D object content, the file generation unit storing timed metadata identification information, which indicates that metadata of a linked external file changes in the time direction, in an MPEG_media extension, and storing timed metadata access information, which links a camera object to the metadata, in the camera object; An information processing device comprising:

2. The scene description file stores the timed metadata identification information in the mimeType of the alternatives array of the MPEG_media extension. The information processing device according to claim 1 .

3. The scene description file stores the timed metadata access information in the MPEG_timed_camera extension of the camera object. The information processing device according to claim 1 .

4. The timed metadata access information specifies, for each of a plurality of parameters, an accessor object corresponding to the area of the buffer object to be referenced, using an accessor index. The information processing device according to claim 3 .

5. The plurality of parameters include a POSITION that indicates the position of the camera object, an ORIENTATION that indicates the orientation of the camera object, and a PERSPECTIVE that indicates the angle of view and perspective information of the camera object. The information processing device according to claim 4 .

6. The timed metadata access information specifies, by a single vector, an accessor object corresponding to an area of a buffer object referenced by multiple parameters. The information processing device according to claim 4 .

7. The timed metadata access information specifies an accessor object corresponding to an area of a buffer object referenced by a plurality of parameters by an array having the accessor index corresponding to each parameter as an element. The information processing device according to claim 4 .

8. The timed metadata access information specifies a plurality of accessor objects corresponding to the area of the buffer object to be referenced. The information processing device according to claim 3 .

9. The MPEG_media extension stores information about multiple timed metadata that change in the time direction as different elements of an alternative array. The information processing device according to claim 1 .

10. A scene description file is generated that describes a scene of the 3D object content, storing timed metadata identification information, which indicates that the metadata of the linked external file changes in the time direction, in an MPEG_media extension, and storing timed metadata access information, which links the camera object to the metadata, in the camera object. Information processing methods.

11. an acquisition unit that acquires timed metadata that changes in the time direction based on timed metadata identification information stored in an MPEG_media extension of a scene description file that describes a scene of 3D object content and timed metadata access information stored in a camera object of the scene description file; a generation unit that generates a display image of the 3D object content based on the timed metadata acquired by the acquisition unit; An information processing device comprising:

12. The acquisition unit acquires the timed metadata based on the timed metadata identification information stored in the mimeType of the alternatives array of the MPEG_media extension of the scene description file. The information processing device according to claim 11.

13. The acquisition unit acquires the timed metadata based on the timed metadata access information stored in the MPEG_timed_camera extension of the camera object in the scene description file. The information processing device according to claim 11.

14. The acquisition unit acquires the timed metadata based on the timed metadata access information that specifies, for each of a plurality of parameters, an accessor object corresponding to an area of the buffer object to be referenced using an accessor index. The information processing device according to claim 13.

15. The acquisition unit acquires the timed metadata based on the timed metadata access information, which specifies, using an accessor index, an accessor object corresponding to an area of a buffer object to be referenced for each of POSITION, which indicates the position of the camera object, ORIENTATION, which indicates the orientation of the camera object, and PERSPECTIVE, which indicates the angle of view and perspective information of the camera object. The information processing device according to claim 14.

16. The acquisition unit acquires the timed metadata based on the timed metadata access information that specifies, by one vector, an accessor object corresponding to an area of a buffer object referenced by a plurality of parameters. The information processing device according to claim 14.

17. The acquisition unit acquires the timed metadata based on the timed metadata access information, which specifies an accessor object corresponding to an area of a buffer object referenced by a plurality of parameters by an array having the accessor index corresponding to each parameter as an element. The information processing device according to claim 14.

18. The acquisition unit acquires the timed metadata based on the timed metadata access information that specifies a plurality of accessor objects corresponding to the area of the buffer object to be referenced. The information processing device according to claim 13.

19. The acquisition unit acquires any one of a plurality of pieces of information relating to timed metadata that changes in the time direction and that are stored as different elements of an alternative array in the MPEG_media extension. The information processing device according to claim 11.

20. acquire timed metadata that changes in the time direction based on timed metadata identification information stored in an MPEG_media extension of a scene description file that describes a scene of the 3D object content and timed metadata access information stored in a camera object of the scene description file; generating a display image of the 3D object content based on the acquired timed metadata; Information processing methods.

Images