Information processing apparatus and method

Abstract

The present disclosure relates to an information processing device and method capable of interactively processing media through scene description. Scene descriptions are provided in which there are descriptions specifying that interactive media is to be interactively played and descriptions specifying execution conditions and processing content of the interactive play. Then, in a case where an execution condition of the interactive playback specified in the scene description is satisfied, the interactive media specified to be interactively played in the scene description is interactively played according to a processing content of the interactive playback specified in the scene description. The present disclosure can be applied to an information processing device, an information processing method, or the like.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and method, and more particularly, to an information processing apparatus and method capable of interactively processing media through scene description. Background Art

[0002] Conventionally, there has been GL Transmission Format (glTF) (registered trademark) 2.0, which is a format for scene description for arranging and rendering three-dimensional (3D) objects in a three-dimensional space (for example, see Non-Patent Literature 1).

[0003] Furthermore, in Moving Picture Experts Group (MPEG)-I scene description, a method of extending glTF2.0 and processing dynamic content in the time direction is proposed (for example, see Non-Patent Document 2).

[0004] In addition, technical research and considerations for coping with interactive processing in MPEG-I scene description have been developed (for example, see non-patent document 3). In addition, technical research and considerations for processing tactile media in MPEG-I scene description have been carried out in parallel with the standardization of compression and transmission technology for tactile media (for example, see non-patent document 4).

[0005] Then, standardization of tactile media compression and transmission technology was developed, which compresses tactile information as a component of 2D video content, 3DoF video content, or 6DoF video content in addition to compressing audio media and video media (for example, see Non-Patent Document 5).

[0006] Reference List

[0007] Non-patent literature

[0008] Non-Patent Literature 1: Saurabh Bhatia, Patrick Cozzi, Alexey Knyazev, Tony Parisi, “Khronos glTF2.0” https: / / github.com / KhronosGroup / glTF / tree / master / specification / 2.0, June 9, 2017

[0009] Non-patent document 2: “Text of ISO / IEC CD 23090-14 Scene Description for MPEGMedia”, ISO / IEC JTC 1 / SC 29 / WG 3N00485, October 12, 2021

[0010] Non-patent document 3: “Exploration Experiments for the MPEG-I SceneDescription”, ISO / IEC JTC 1 / SC 29 / WG 3N 0540, 2022 / 4 / 29

[0011] Non-Patent Document 4: Quentin Galvane, Eric Vezzoli, Yeshwant Muthusamy, Philippe Guillotel, Titouan Rabu, “Haptic Support extensions for MPEG-I SD” ISO / IEC JTC1 / SC 29 / WG 3m59268, April 2022

[0012] Non-patent document 5: “Considerations for a WD on Haptics”, ISO / IEC JTC 1 / SC 29 / WG 2N 0208, April 29, 2022 Summary of the invention

[0013] Technical issues

[0014] In the interactivity framework described in non-patent document 3, a trigger is defined as an interaction generation condition, and an action is defined as an interaction to be generated. In addition, a behavior that enumerates multiple triggers and multiple actions is defined. For example, an MPEG media (MPEG_media) that is a target of interactive processing is played in such a framework.

[0015] However, in the MPEG-I scene description described in Non-Patent Document 2, the definition of MPEG media is only startTime and autoPlay. Therefore, it is difficult to perform interactive processing on MPEG media.

[0016] The present disclosure is proposed in view of this situation, and enables media to be interactively processed through scene description.

[0017] Solution to the problem

[0018] An information processing device according to one aspect of the present technology is an information processing device, which includes: a playback unit, which interactively plays the interactive media specified in the scene description to be interactively played according to the processing content of the interactive playback specified in the scene description when the execution condition of the interactive playback specified in the scene description is met, wherein the interactive playback is a playback method of playing the media by interactive processing, and the interactive processing is an interactive processing of executing the processing content specified in the scene description when the execution condition specified in the scene description is met.

[0019] An information processing method according to one aspect of the present technology is an information processing method, which includes: interactively playing the interactive media specified in the scene description to be interactively played according to the processing content of the interactive playback specified in the scene description when the execution conditions of the interactive playback specified in the scene description are met, wherein the interactive playback is a playback method of playing the media by interactive processing, and the interactive processing is an interactive processing of executing the processing content specified in the scene description when the execution conditions specified in the scene description are met.

[0020] According to another aspect of the present technology, an information processing device is an information processing device, which includes: a providing unit, which provides a scene description, in which there is a description specifying that interactive media is to be played interactively and a description specifying execution conditions and processing contents of the interactive playback, wherein interactive playback is a playback method of playing media by interactive processing, and interactive processing is interactive processing of executing processing contents specified in the scene description when the execution conditions specified in the scene description are met.

[0021] According to another aspect of the present technology, an information processing method is an information processing method, which includes: providing a scene description, in which there is a description specifying that interactive media is to be played interactively and a description specifying execution conditions and processing contents of the interactive playback, wherein interactive playback is a playback method of playing media by interactive processing, and interactive processing is interactive processing of executing processing contents specified in the scene description when the execution conditions specified in the scene description are met.

[0022] In the information processing apparatus and method according to various aspects of the present technology, when the execution conditions of the interactive playback specified in the scene description are met, the interactive media specified to be interactively played in the scene description is interactively played according to the processing content of the interactive playback specified in the scene description.

[0023] In an information processing device and method according to another aspect of the present technology, a scenario description is provided in which there is a description specifying that interactive media is to be interactively played and a description specifying execution conditions and processing contents of the interactive play. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] [ Figure 1 ]A diagram showing an example of the main configuration of glTF2.0.

[0025] [ Figure 2 ] A diagram showing an example of glTF objects and reference relationships.

[0026] [ Figure 3 ]A diagram showing a description example of a scene description.

[0027] [ Figure 4 ]Diagram describing methods for accessing binary data.

[0028] [ Figure 5 ]A diagram showing a description example of a scene description.

[0029] [ Figure 6 ] Diagram describing the relationship between buffer objects, buffer view objects, and accessor objects.

[0030] [ Figure 7 ] A diagram showing an example description of a buffer object, a buffer view object, and an accessor object.

[0031] [ Figure 8 ]A diagram describing an example configuration of objects in a scene description.

[0032] [ Fig. 9 ]A diagram showing a description example of a scene description.

[0033] [ Fig.10 ]A diagram describing the extension methods of an object.

[0034] [ Fig.11 ]Diagram describing the configuration of client processing.

[0035] [ Fig.12 ]A diagram showing an example configuration of an extension for processing timed metadata.

[0036] [ Fig.13 ]A diagram showing a description example of a scene description.

[0037] [ Fig.14 ]A diagram showing a description example of a scene description.

[0038] [ Fig.15 ]A diagram showing an example configuration of an extension for processing timed metadata.

[0039] [ Fig.16 ]A diagram showing an example of the main configuration of a client.

[0040] [ Fig.17 ]A flowchart describing an example of the process of client processing.

[0041] [ Fig.18 ]A diagram describing the definition of MPEG media.

[0042] [ Fig.19 ]A diagram describing the definition of MPEG media.

[0043] [ Fig. 20 ]A diagram describing the definition of MPEG media.

[0044] [ Fig.21 ]A diagram describing the definition of MPEG media.

[0045] [ Fig. 22 ] Diagram describing the MPEG-I scene description interactive framework.

[0046] [ Fig.23 ]Diagram describing an overview of the interactive functionality.

[0047] [ Fig.24 ]Diagram describing the architecture of triggers and actions.

[0048] [ Fig.25 ]A diagram describing the definition of a trigger.

[0049] [ Fig.26 ]A diagram describing the definition of an operation.

[0050] [ Fig. 27 ]A diagram describing the definition of an operation.

[0051] [ Fig.28 ]A diagram describing the definition of a behavior.

[0052] [ Fig.29 ]Diagram of an example of the extended structure of MPEG_scene_interaction.

[0053] [ Fig.30 ]A diagram showing an example of a processing model.

[0054] [ Fig.31 ]A diagram showing an example of a processing model.

[0055] [ Fig.32 ] Diagram of an extended example of scene description for processing tactile media.

[0056] [ Fig.33 ] A diagram showing an example of MPEG haptics.

[0057] [ Fig.34 ] A diagram describing an overview of encoding of tactile media.

[0058] [ Fig.35 ]A diagram showing an example configuration of a binary header.

[0059] [ Fig.36 ] A diagram showing an example of the semantics of haptic file metadata.

[0060] [ Fig.37 ]A diagram showing an example of the semantics of avatar metadata.

[0061] [ Fig.38 ]A diagram showing an example of semantics-aware metadata.

[0062] [ Fig.39 ]A diagram showing an example of the semantics of reference device metadata.

[0063] [ Fig.40 ]A diagram showing an example of the semantics of a track header.

[0064] [ Fig.41 ]A diagram showing an example of semantics with headers.

[0065] [ Fig.42 ] A diagram showing examples of the semantics of transient and curved bands. [ Fig.43 ]A diagram showing an example of the semantics of a band body.

[0066] [ Fig.44 ]A diagram showing a main configuration example of a general data model.

[0067] [ Fig.45 ]A diagram showing an example of device definition.

[0068] [ Fig.46 ] A diagram showing an example of device definitions stored in a track.

[0069] [ Fig.47 ]A diagram showing an example with definitions.

[0070] [ Fig.48 ]A diagram showing an example of effect definition.

[0071] [ Fig.49 ] A diagram showing an example of media item expansion in a scene description. [ Fig.50 ] A diagram showing an expanded example of a media item in a scene description. [ Fig.51 ]A diagram showing an expanded example of interactive media.

[0072] [ Fig.52]A diagram showing an expanded example of interactive media.

[0073] [ Fig.53 ]A diagram showing an expanded example of interactive media.

[0074] [ Fig.54 ]A diagram showing an expanded example of interactive media.

[0075] [ Fig.55 ]A diagram showing an expanded example of interactive media.

[0076] [ Fig.56 ]A diagram showing an expanded example of interactive media.

[0077] [ Fig.57 ]A diagram showing an expanded example of interactive media.

[0078] [ Fig.58 ]A diagram showing an expanded example of interactive media.

[0079] [ Fig.59 ]A diagram showing an expanded example of an action type.

[0080] [ Fig.60 ]A diagram showing an expanded example of parameters used for setting.

[0081] [ Fig.61 ]A diagram showing an expanded example of an action type.

[0082] [ Fig.62 ]A diagram showing an expanded example of parameters used for preliminary processing.

[0083] [ Fig.63 ]A diagram showing an extended example of a behavior definition.

[0084] [ Fig.64 ]A diagram showing an extended example of MPEG_scene_interactivity.

[0085] [ Fig.65 ]A diagram showing an extended example of a behavior definition.

[0086] [ Fig.66 ]A block diagram showing a main configuration example of a file generating device.

[0087] [ Fig.67 ]A flowchart showing an example of the flow of file generation processing.

[0088] [ Fig.68 ]A flowchart showing an example of the flow of file generation processing.

[0089] [ Fig.69]A flowchart showing an example of the flow of file generation processing.

[0090] [ Fig.70 ]A flowchart showing an example of the flow of file generation processing.

[0091] [ Fig.71 ]A flowchart showing an example of the flow of file generation processing.

[0092] [ Fig.72 ]A flowchart showing an example of the flow of file generation processing.

[0093] [ Fig.73 ]A flowchart showing an example of the flow of file generation processing.

[0094] [ Fig.74 ]A flowchart showing an example of the flow of file generation processing.

[0095] [ Fig.75 ]A block diagram showing a main configuration example of a client device.

[0096] [ Fig.76 ]A flowchart showing an example of the flow of playback processing.

[0097] [ Fig.77 ]A flowchart showing an example of the process of 3D object content playback processing.

[0098] [ Fig.78 ]A flowchart showing an example of the process of interactive playback.

[0099] [ Fig.79 ]A flowchart showing an example of the process of interactive playback.

[0100] [ Fig.80 ]A flowchart showing an example of the process of interactive playback.

[0101] [ Fig.81 ]A flowchart showing an example of the process of interactive playback.

[0102] [ Fig.82 ]A flowchart showing an example of the process of interactive playback.

[0103] [ Fig.83 ]A block diagram showing an example of the main configuration of a computer. DETAILED DESCRIPTION

[0104] Hereinafter, a mode for carrying out the present disclosure (hereinafter referred to as an embodiment) will be described. It should be noted that the description will be given in the following order.

[0105] 1. Documents supporting technical content and technical terminology, etc.

[0106] 2. Interactive playback control

[0107] 3. Media Definition Extension for Interactive Playback

[0108] 4. First Embodiment (File Generating Device)

[0109] 5. Second Embodiment (Client Device)

[0110] 6. Appendix

[0111] <1. Documents supporting technical content and technical terminology, etc.>

[0112] The scope of the present technical disclosure includes not only the contents described in the embodiments but also the contents described in the following non-patent documents and the like which are known on the filing date of the present application and the like, and the contents of other documents and the like cited in the following non-patent documents and the like.

[0113] Non-patent document 1: (as mentioned above)

[0114] Non-patent document 2: (as mentioned above)

[0115] Non-patent document 3: (as mentioned above)

[0116] Non-patent document 4: (as mentioned above)

[0117] Non-Patent Document 5: (as mentioned above)

[0118] Non-Patent Literature 6: Yeshwant Muthusamy, Chris Ullrich, Manuel Cruz, “Everything You Wanted to Know About Haptics”, March 21, 2022

[0119] That is, the contents described in the above non-patent documents, the contents of other documents cited in the above non-patent documents, etc. are also the basis for determining the support requirements. For example, it is assumed that even if the syntax and terms of glTF2.0, its extensions, etc. described in the above non-patent documents are not directly defined in the present disclosure, they fall within the scope of the present disclosure and meet the support requirements of the scope of the claims. It is also assumed that, for example, even if technical terms such as parsing, syntax, and semantics are not directly defined in the present disclosure, they fall within the scope of the present disclosure and meet the support requirements of the scope of the claims.

[0120] <2. Interactive playback control>

[0121] <glTF2.0>

[0122] Conventionally, for example, as described in Non-Patent Document 1, there is GL Transmission Format (glTF) (registered trademark) 2.0 as a format for arranging three-dimensional (3D) objects in a three-dimensional space. Figure 1 As shown, glTF2.0 consists of JSON format files (.glTF), binary files (.bin), and image files (.png, .jpg, etc.). Binary files store binary data such as geometry and animations. Image files store data such as textures.

[0123] The JSON format file is a scene description file described by JavaScript (registered trademark) Object Notation (JSON). The scene description refers to metadata that describes (explains) the scene of the 3D content. By describing the scene description, it is defined what the scene is like. The scene description file is a file that stores such a scene description. In the present disclosure, the scene description file will also be referred to as a scene description file.

[0124] The description of the JSON format file consists of a pair of enumerations of keys (KEY) and values ​​(VALUE). An example of its format is shown below.

[0125] "KEY": "VALUE"

[0126] Keys are strings. Values ​​are numbers, strings, true / false values, arrays, objects, null, etc.

[0127] In addition, multiple (key / value) pairs ("KEY":"VALUE") can be combined with {} (curly braces). The combination of these curly braces is also called a JSON object. An example of its form is shown below.

[0128] "user":{"id":1,"name":"tanaka"}

[0129] In the case of this example, as a value corresponding to the key (user), a JSON object combining a pair of "id":1 and a pair of "name":"tanaka" is defined.

[0130] In addition, zero or more values ​​can be turned into an array using [] (square brackets). This array is also called a JSON array. For example, as an element of a JSON array, a JSON object can also be applied. An example of its form is shown below.

[0131] test":["hoge","fuga","bar"]

[0132] "users":[{"id":1,"name":"tanaka"},{"id":2,"name":"yamada"},{"id":3,"name":"sato"}]

[0133] The glTF objects that can be described at the top level of a JSON file and the reference relationships that the glTF objects may have are as follows: Figure 2 shown. Figure 2 The long circles of the tree structure shown indicate objects, and each arrow between objects indicates a reference relationship. Figure 2 As shown, objects such as "scene", "node", "mesh", "camera", "skin", "material", and "texture" are described at the top level of the JSON format file.

[0134] Figure 3 An example description of such a JSON format file (scene description) is shown in FIG. Figure 3 The JSON format file 20 in FIG. 1 shows an example of a description of a portion of the top layer. In the JSON format file 20, all the top-level objects 21 to be used are described at the top level. These top-level objects 21 are Figure 2 In addition, in the JSON format file 20, the reference relationship between objects is shown by arrows 22. More specifically, by specifying the index of the array element of the object to be referenced in the properties of the top-level object, its reference relationship can be shown.

[0135] Figure 4 is a diagram that describes how binary data is accessed. Figure 4 As shown, binary data is stored in a buffer object. That is, information (such as a uniform resource identifier (URI)) for accessing binary data in a buffer object is shown. In a JSON format file, such as Figure 4 As shown, for example, from objects such as meshes, cameras, and skins, their buffer objects can be accessed via accessor objects and buffer view objects (bufferView objects).

[0136] That is, in an object such as a mesh, camera, or skin, specify the accessor object to refer to. An example of a description of a mesh object (mesh) in a JSON format file is as follows Figure 5 For example, Figure 5 As shown, in the mesh object, attributes of the vertices (such as NORMAL, POSITION, TANGENT, and TEXCORD_0) are defined as keys, and the accessor objects to be referenced are specified as the values ​​of the corresponding attributes.

[0137] The relationship between buffer objects, buffer view objects and accessor objects is as follows Figure 6 In addition, the description examples of these objects in the JSON format file are as follows Figure 7 shown.

[0138] exist Figure 6 In the example, the buffer object 41 is an object that stores information (eg, URI) for accessing binary data as real data and information indicating the data length (eg, byte length) of the binary data. Figure 7 A shows a description example of the buffer object 41 thereof. Figure 7 The "bytelength": 102040" shown in A indicates that Figure 6 As shown, the byte length of the buffer object 41 is 102040 bytes. In addition, Figure 7 The "uri":"duck.bin" shown in A indicates that Figure 6 As shown, the URI of the buffer object 41 is "duck.bin".

[0139] exist Figure 6 , the buffer view object 42 is an object that stores information about a subset area of ​​binary data specified in the buffer object 41 (ie, information about an area of ​​a part of the buffer object 41). Figure 7 FIG. B shows an example description of the buffer view object 42. Figure 6 and Figure 7 As shown in B, the buffer view object 42 stores information, such as identification information of the buffer object 41 to which the buffer view object 42 belongs, an offset (e.g., byte offset) indicating the position of the buffer view object 42 in the buffer object 41, and a length (e.g., byte length) indicating the data length (e.g., byte length) of the buffer view object 42.

[0140] In the presence of Figure 7 In the case of multiple buffer view objects shown in B, information is described for each buffer view object (ie, each subset area). For example, Figure 7 Information such as ""buffer":0", ""bytelength":25272", and ""byteOffset":0" shown on the upper side of B is about Figure 6 The information of the first buffer view object 42 (bufferView[0]) shown in the buffer object 41 in FIG. Figure 7 The information such as ""buffer":0", ""bytelength":76768", and ""byteOffset":25272" shown on the lower side of B is Figure 6The information of the second buffer view object 42 (bufferView[1]) shown in the buffer object 41 in FIG.

[0141] Figure 7 The "buffer":0" of the first buffer view object 42 (bufferView[0]) shown in FIG. Figure 6 As shown, the identification information of the buffer object 41 to which the buffer view object 42 (bufferView[0]) belongs is "0" (Buffer[0]). In addition, ""bytelength": 25272" indicates that the byte length of the buffer view object 42 (bufferView[0]) is 25272 bytes. In addition, ""byteOffset": 0" indicates that the byte offset of the buffer view object 42 (bufferView[0]) is 0 bytes.

[0142] Figure 7 The "buffer":0" of the second buffer view object 42 (bufferView[1]) shown in FIG. Figure 6 As shown, the identification information of the buffer object 41 to which the buffer view object 42 (bufferView[0]) belongs is "0" (Buffer[0]). In addition, ""bytelength": 76768" indicates that the byte length of the buffer view object 42 (bufferView[0]) is 76768 bytes. In addition, ""byteOffset": 25272" indicates that the byte offset of the buffer view object 42 (bufferView[0]) is 25272 bytes.

[0143] exist Figure 6 , the accessor object 43 is an object that stores information about an interpretation method of data of the buffer view object 42. Figure 7 C shows an example of description of the accessor object 43. Figure 6 and Figure 7 As shown in C, the accessor object 43 stores information such as identification information of the buffer view object 42 to which the accessor object 43 belongs, an offset (such as a byte offset) indicating the position of the buffer view object 42 in the buffer object 41, the component type of the buffer view object 42, the number of data stored in the buffer view object 42, and the type of data stored in the buffer view object 42. This information will be described for each buffer view object.

[0144] exist Figure 7In the C example, information such as ""bufferView":0", ""byteOffset":0", ""componentType":5126", ""count":2106", and ""type":"VEC3"" is shown. ""bufferView":0" indicates that Figure 6 As shown, the identification information of the buffer view object 42 to which the accessor object 43 belongs is "0" (bufferView[0]). In addition, ""byteOffset": 0" indicates that the byte offset of the buffer view object 42 (bufferView[0]) is 0 bytes. In addition, ""componentType": 5126" indicates that the component type is FLOAT type (OpenGL macro constant). In addition, ""count": 2106" indicates that the number of data stored in the buffer view object 42 (bufferView[0]) is 2106. In addition, ""type": "VEC3"" indicates that the data (type) stored in the buffer view object 42 (bufferView[0]) is a three-dimensional vector.

[0145] Access to data other than images is defined by reference to accessor objects 43 (by specifying the accessor's index).

[0146] Next, in the scene description (JSON format file) according to this glTF2.0, a method of specifying a 3D object of a point cloud will be described. A point cloud is a 3D content that represents a three-dimensional structure (an object of a three-dimensional shape) as a collection of multiple points. Point cloud data consists of position information (also called geometry) and attribute information (also called attributes) of each point. Attributes can include any information. For example, color information, reflectivity information, normal information, etc. of each point can be included in the attributes. Therefore, a point cloud has a relatively simple data structure, and by using a sufficient number of points, any three-dimensional structure can be expressed with sufficient accuracy.

[0147] In the case where the point cloud does not change in direction over time (also known as static), 3D objects are specified by using glTF 2.0 mesh.primitives objects. Figure 8 is a diagram showing a configuration example of objects in a scene description in a case where a point cloud is static. Fig. 9 is a diagram showing a description example of a scene description.

[0148] like Fig. 9 As shown, the mode of the primitive object is specified as 0, indicating that the data is to be processed as points of a point cloud. Figure 8 and Fig. 9As shown, in the position attribute (POSITION attribute) of the attribute object in mesh.primitives, the accessor to the buffer storing the position information of the point (Point) is specified. Similarly, in the color attribute (COLOR attribute) of the attribute object, the accessor to the buffer storing the color information of the point (Point) is specified. The buffer and the buffer view (bufferView) can be one (data can be stored in a single file).

[0149] Next, we will describe the extension of the scene description object. Each object in glTF 2.0 can store newly defined objects in the extension object. Fig.10 2 shows a description example in the case of defining a newly defined object (ExtensionExample). Fig.10 As shown, when using a newly defined extension, its extension object name (in Fig.10 The example of ExtensionExample is described in "extensionUsed" and "extensionRequired". Accordingly, it indicates that the extension is the extension to be used, or the extension is the extension required to be loaded.

[0150] <Client Processing>

[0151] Next, the processing of the client device in the MPEG-I scene description will be described.The client device acquires the scene description, acquires the data of the 3D object based on the scene description, and generates a display image by using the scene description and the data on the 3D object.

[0152] As described in Non-Patent Document 2, in the client device, the presentation engine, the media access function, etc. perform processing. Fig.11 As shown, the rendering engine 51 of the client device 50 obtains the initial value of the scene description and the information for updating the scene description (hereinafter also referred to as update information), and generates the scene description of the processing target time. Then, the rendering engine 51 analyzes the scene description and identifies the media to be played (dynamic images, sounds, etc.). Then, the rendering engine 51 requests the media access function 52 to obtain the media via the media access application program interface (API). In addition, the rendering engine 51 also performs setting pipeline processing, specifying buffers, etc.

[0153] The media access function 52 acquires various types of data of media requested by the presentation engine 51 from the cloud, a local storage device, etc. The media access function 52 supplies the acquired various types of data of media (encoded data) to the pipeline 53 .

[0154] The pipeline 53 decodes various types of data (encoded data) of the supplied media through pipeline processing, and supplies the decoded results to the buffer (Buffer) 54. The buffer 54 retains various types of data of the supplied media.

[0155] The presentation engine 51 performs rendering and the like by using various types of data of the media retained in the buffer 54 .

[0156] <Application of Timed Media>

[0157] In recent years, for example, as shown in Non-Patent Document 2, it has been considered to extend glTF2.0 in MPEG-I scene description and apply timed media as 3D object content. Timed media refers to media data that changes in the time axis direction, like a moving image in a two-dimensional image.

[0158] glTF is applicable only to still image data as media data (3D object content). That is, glTF is not applicable to media data of moving images. In the case of 3D object movement, animation (a method of switching still images along the time axis) is applied.

[0159] In MPEG-I scene description, it is considered to apply glTF2.0, apply JSON format files as scene description, and extend glTF so that timed media (such as video data) can be processed as media data. For example, in order to process timed media, the following extensions are performed.

[0160] Fig.12 is a diagram that describes an extension for handling timed media. Fig.12 In the example of , the MPEG media object (MPEG_media) is an extension of glTF, and is an object that specifies attributes of MPEG media such as video data, such as uri, track, renderingRate, and startTime.

[0161] In addition, if Fig.12 As shown, the MPEG texture video object (MPEG_texture_video) is provided as an extension object (extensions) of the texture object (texture). In the MPEG texture video object, information of an accessor corresponding to a buffer object to be accessed is stored. That is, the MPEG texture video object is an object that specifies an index of an accessor corresponding to a buffer in which the texture media specified by the MPEG media object (MPEG_media) is decoded and stored.

[0162] Fig.13: is a diagram showing an example of description of an MPEG media object (MPEG_media) and an MPEG texture video object (MPEG_texture_video) in a scene description for describing an extension that handles timed media. Fig.13 In the case of the example, in the second row from the top, an MPEG texture video object (MPEG_texture_video) has been set as an extension object (extensions) of a texture object (texture), as described below. Then, the index of the accessor ("2" in this example) has been assigned the value of the MPEG texture video object.

[0163] "texture":[{"sampler":0,"source":1,"extensions":{"MPEG_texture_vide0":"accessor":2}}],

[0164] In addition, Fig.13 In the example of , in the seventh to sixteenth lines from the top, as an extension object (extensions) of glTF, an MPEG media object (MPEG_media) is set as described below. Then, as a value of the MPEG media object, various types of information about the MPEG media object, such as the encoding and URI of the MPEG media object, are stored.

[0165]

[0166] Furthermore, although the individual pieces of frame data are decoded and stored sequentially in the buffer, their positions etc. may change, so a system is provided to store the changing information in the scene description so that the renderer can read out the data. Fig.12 As shown, an MPEG buffer circular object (MPEG_buffer_circular) is provided as an extension object (extensions) of a buffer object (buffer). In the MPEG buffer circular object, information for dynamically storing data in the buffer object is stored. For example, information such as information indicating the data length of a buffer header (bufferHeader) and information indicating the number of frames is stored in the MPEG buffer circular object. It should be noted that the buffer header stores information such as an index, a timestamp, and a data length of the frame data to be stored.

[0167] In addition, if Fig.12As shown, the MPEG accessor timing object (MPEG_timed_accessor) is provided as an extension object (extensions) of the accessor object (accessor). In this case, the media data is a moving image, so the buffer view object (bufferView) to be referenced can change in the time direction (the position can change). In view of this, information indicating the buffer view object to be referenced is stored in the MPEG accessor timing object. For example, in the MPEG accessor timing object, an information indicating a reference to the buffer view object (bufferView) is stored, in which a timed accessor information header (timedAccessor information header) is described. It should be noted that the timed accessor information header is header information that stores dynamically changing information, such as in the accessor object and the buffer view object.

[0168] Fig.14 : is a diagram showing an example of description of an MPEG buffer circular object (MPEG_buffer_circular) and an MPEG accessor timed object (MPEG_accessor_timed) in a scene description for describing an extension for handling timed media. Fig.14 In the case of the example of , in the fifth line from the top, the MPEG accessor timing object (MPEG_accessor_timed) has been set as the extension object (extensions) of the accessor object (accessor), as described below. Then, as the value of the MPEG accessor timing object, the parameters of the buffer view object (such as the index (in this example, "1"), the update rate (updataRate) and the immutable information (immutable)) and their values ​​are specified.

[0169] "MPEG_accessor_timed":{"bufferView":1,"updateRate":25.0,"immutable":1,")

[0170] In addition, Fig.14 In the case of the example in , in the thirteenth line from the top, the MPEG buffer circular object (MPEG_buffer_circular) has been set as an extension object (extensions) of the buffer object (buffer), as described below. Then, parameters such as the buffer frame count (count), header length (headerLength) and update rate (updataRate) and their values ​​are specified as the values ​​of the MPEG buffer circular object.

[0171] "MPEG_buffer_circular": {"count": 5, "headerLength": 12, "updateRate": 25.0}

[0172] Fig.15 is a diagram used to describe an extension for processing timed media. Fig.15 Examples of MPEG accessor timing objects and MPEG buffer loop objects and the relationship between accessor objects, buffer view objects, and buffer objects are shown.

[0173] In the MPEG buffer loop object of the buffer object, information required for storing data that changes over time in the buffer area indicated by the buffer object, such as a buffer frame count (count), a header length (headerLength), and an update rate (updataRate) as described above, is stored. In addition, in the buffer header (bufferHeader) that is a header of the buffer area, parameters such as an index (index), a timestamp (timestamp), and a data length (length) are stored.

[0174] In the MPEG accessor timing object of the accessor object, information about the buffer view object to be referenced, such as the index (bufferView), update rate (updataRate) and immutable information (immutable) of the buffer view object is stored. In addition, in the MPEG accessor timing object, information about the buffer view object is stored, in which the timing accessor information header to be referenced is stored. In the timing accessor information header, timestamp delta (timestamp_delta), update data of the accessor object, update data of the buffer view object, etc. can be stored.

[0175] <Client side processing during use of MPEG_texture_video>

[0176] A scene description is spatial arrangement information for arranging one or more 3D objects in a 3D space. The scene description enables its content to be updated along the time axis. That is, the scene description enables the arrangement of 3D objects to be updated over time. The client processing performed in the client device will then be described.

[0177] Fig.16 shows a main configuration example related to client processing of a client device, Fig.17 is a flowchart showing an example of the flow of client processing. Fig.16As shown, the client device includes a rendering engine (hereinafter also referred to as PE) 51, a media access function (hereinafter also referred to as MAF) 52, a pipeline 53, and a buffer 54. The rendering engine (PE) 51 includes a glTF analysis unit 63 and a rendering processing unit 64.

[0178] The presentation engine (PE) 51 causes the media access function 52 to acquire media, acquires data via the buffer 54, and performs processing related to display, etc. Specifically, for example, the processing is performed according to the following flow.

[0179] When client processing is started, the glTF analysis unit 63 of the rendering engine (PE) 51 Fig.17 PE processing is started as in the example of , and in step S21, an SD (glTF) file 62 as a scene description file is acquired, and the scene description is parsed.

[0180] In step S22, the glTF analysis unit 63 checks the media bound to the 3D object (texture), the buffer storing the processed media, and the accessor. In step S23, the glTF analysis unit 63 notifies the media access function 52 of this information as a file acquisition request.

[0181] Media Access Function (MAF) 52 Fig.17 The MAF process is started as in the example of , and the notification is acquired in step S11. In step S12, the media access function 52 acquires the media (3D object file (mp4)) based on the notification.

[0182] In step S13 , the media access function 52 decodes the acquired media (3D object file (mp4)). In step S14 , the media access function 52 stores the data of the media acquired by decoding in the buffer 54 based on the notification from the presentation engine (PE) 51 .

[0183] In step S24, the rendering processing unit 64 of the presentation engine 51 reads out (acquires) data at appropriate timing from the buffer 54. In step S25, the rendering processing unit 64 performs rendering by generating an image to be displayed using the acquired data.

[0184] The media access function 52 repeats the processing in steps S13 and S14, thereby performing such processing for each time (each frame). In addition, the rendering processing unit 64 of the presentation engine 51 repeats the processing in steps S24 and S25, thereby performing such processing for each time (each frame). When the processing for all frames ends, the media access function 52 terminates the MAF processing, and the presentation engine 51 terminates the PE processing. That is, the client processing ends.

[0185] <Definition of MPEG_media>

[0186] Non - Patent Document 2 also describes an extension of MPEG media (MPEG_media). The extension of MPEG media is provided in the form of an array of media items to be referenced in the scene description. An example of the definition of the items to be used in the media array of MPEG media is as Fig.18 and Fig.19 shown. For example, startTime indicates the time to start rendering the media at the specified time. This value is specified in seconds. In the case of a texture with a limited time, it is necessary to render a static image until the startTime of the texture is reached. When startTime is "0", it means the presentation time of the current scene. Autoplay specifies to start playing immediately after the media is ready. AutoplayGroup is a function that can specify autoplay by group. Loop specifies loop playback. Controls specifies to display the user interface related to playing the media. Alternatives indicates alternative ways of the same media (for example, a different video codec from the one used). It should be noted that for a media item, there must be either startTime or autoplay.

[0187] Fig. 20 An extended example of the alternative array is shown. In the alternative array, the items defined in Fig. 20 are used. Fig.21 An extended example of the track array is shown. In the track, the items defined in Fig.21 are used.

[0188] <MPEG - I Scene Description Interactivity Framework>

[0189] In Non - Patent Document 3, the MPEG - I scene description interactivity framework is described. In this framework, the interactive processing of media is defined. Interactive processing refers to an interactive type of processing in which media processing (actions) are executed and triggered when specific execution conditions are met. In this framework, such execution conditions (triggers) and processing content (actions) are provided in the scene description (MPEG - I scene description). That is, Non - Patent Document 3 describes a framework for controlling the interactive processing of media through scene description.

[0190] As Fig. 22 shown, at the scene level, the glTF extension function of MPEG interactivity, namely MPEG_scene_interactivity, is introduced. The extension of MPEG_scene_interactivity at the scene level adopts a semantic approach based on the definitions of behavior, trigger, and action. Fig.23An example of semantics is shown.

[0191] Behaviors define what type of interaction is allowed when executing a dedicated virtual object corresponding to a glTF node. Behaviors have the function of associating one or more triggers with one or more actions. Triggers define execution conditions that should be met before executing an action. In other words, triggers indicate execution conditions for interactive processing. Actions define how operations affect the scene. In other words, actions indicate the processing content of interactive processing. Behaviors express interactive processing (what processing is executed under what conditions) by associating triggers with actions.

[0192] An example of a trigger and action architecture is Fig.24 When a trigger is provided to a node, an action is sent back. That is, a trigger is an event that triggers some type of interactivity, and an action is an interactive feedback to the trigger. For example, when a collision is detected between 3D objects in a (virtual) three-dimensional space, feedback (interactive action) such as animation, sound or haptics (such as vibration) is sent back.

[0193] Fig.25 An example of the semantics of this trigger is shown. Fig.25 As shown, the types of execution conditions are defined, and details about each type (various items) are defined. For example, the trigger types include VISIBIRITY, PROXIMITY, USER_INPUT, TIMED, COLLIDER, etc. VISIBIRITY is a trigger activated by the field of view (perspective). PROXIMITY is a trigger activated by the distance between the virtual scene and the avatar. USER_INPUT is a trigger activated by user interaction (such as gestures). TIMED is a trigger activated by timed media at a specific time. COLLIDER is a trigger activated by a collision between objects in the scene.

[0194] also, Fig.26 and Fig. 27 An example of action semantics is shown in Fig.26 and 27As shown, the types of processing content are defined, and details (various items) are defined for each type. For example, the action type includes ACTIVATE, TRANSFORM, ANIMATE, CONTROL_MEDIA, PLACE_AT, MANIPULATE, SET_MATERIAL, etc. ACTIVATE is an operation related to the activation of a node by an application. TRANSFORM is an operation related to the application of a transformation matrix applied to a node. ANIMATE is an operation related to the playback operation of an animation (such as Play, Pause, Resume, and Stop). CONTROL_MEDIA is an operation related to the playback operation of a media (such as Play, Pause, Resume, and Stop). PLACE_AT is an operation related to arranging a node at a specified position. MANIPULATE is an action about operating a node (such as tracking, moving, translating, rotating, and scaling transformations) through a user's pointing device. SET_MATERIAL is an action related to setting a material for a node.

[0195] also, Fig.28 An example of the semantics of a behavior is shown in . In a behavior, a combination of triggers and actions is provided. Each of the triggers and actions can be single, or can be multiple. That is, it can be executed under multiple conditions, (and / or) multiple operations can be executed (sequentially / simultaneously). Priority indicates the priority when multiple behaviors are enabled at the same time. TriggersControl is tag information indicating the combination conditions (AND / OR) of multiple triggers (trigger array). ActionsControl is tag information indicating the combination mode (whether to execute sequentially, whether to execute in parallel, etc.) of multiple actions (action array). InterruptAction is an action that is executed while the behavior is "still in progress" when the scene is updated (SceneUpdate).

[0196] An example of an extended structure of MPEG_scene_interaction is Fig.29 As shown. Fig.29 In the example, User Inputs, MPEG_media_collision, MPEG_avatar, and MPEG_recommended_viewport can be used as trigger targets. In addition, MPEG_media, MPEG_haptic, MPEG_audio_spatial, and MPEG_material_haptic can be used as action targets.

[0197] At execution time, the application repeats the defined operations and Fig.30 The program shown checks the implementation of the relative triggers. First, when the trigger of the defined behavior is activated, the corresponding action is also activated. From the activation until the defined action is completed, the behavior waits for the "in progress" state. In the case of multiple behaviors affecting the same node at the same time, the behavior with the highest priority is processed with respect to this node. Other behaviors are not processed at the same time. Such processing is repeated.

[0198] In addition, when a new scene description is provided, the application Fig.31 The program execution processing shown. In the case where a relative action is being executed when the scene is updated, the behavior is considered to be in progress. In the case where a unique association of trigger and action is described even after the scene is updated, the behavior is considered to be "still being defined". In the case of a behavior "not yet defined", its interrupt action is executed. When all interrupt operations (if any) are completed, the application deletes the old scene data and the new data is considered to be the same as the updated scene description.

[0199] <Tactile Media>

[0200] By the way, for example, as described in non-patent document 4, in addition to audio media and video media (i.e., 2D video content and 3 degrees of freedom (DoF) / 6 degrees of freedom video content components), standardization work on encoding and transmission technology of tactile information (also referred to as tactile media) has begun. For example, tactile media is information that expresses a virtual sense by using vibration. Tactile media is used in association with 3D data, for example, 3D data is information that expresses a 3D space. For example, 3D data includes content that expresses the three-dimensional shape of a 3D object arranged in a three-dimensional space (e.g., a mesh and a point cloud), video content and audio content (e.g., video and audio 6DoF content) deployed to the 3D space, and the like.

[0201] It should be noted that the media associated with 3D data can be any information and is not limited to such tactile media. For example, images, sounds, etc. can all be included in such media. The media associated with 3D data (such as images, sounds, and vibrations) include synchronous media and interactive media. Synchronous media is played synchronously with the progress (change) of the scene (the state of the three-dimensional space) in the time direction, and interactive media is played when a predetermined condition (such as the user's operation) is met in the scene (i.e., played in association with a predetermined event). Tactile media as synchronous media is also called synchronous tactile media. In addition, tactile media of interactive media is also called interactive tactile media. For example, in the case of wind blowing or 3D object movement, synchronous tactile media is vibration that occurs according to its state (expressing the changing state of its scene), etc. For example, in the case of the user's avatar touching the 3D object, in the case of the avatar moving the 3D object, or in the case of the avatar hitting the 3D object, interactive tactile media is vibration that occurs to express its feeling, etc. Of course, these are all examples of tactile media, and tactile media is not limited to these examples.

[0202] Furthermore, the media associated with 3D data are media that does not change in the time direction and media that can change.

[0203] For example, "the media can change in the time direction" may include: the media can change the playback content (action) in the time direction. For example, "the media can change the playback content in the time direction" may include moving images and long-time audio information and vibration information. In addition, for example, "the media can change the playback content in the time direction" may include: media that is played only in a predetermined time zone and media that plays content according to time (for example, media that switches the image to be displayed, the sound to be played, any vibration, etc. according to time), etc. In addition, for example, "the media can change in the time direction" may include: the media can change the bound playback conditions (events) in the time direction. For example, "the media can change the bound playback conditions in the time direction" may include: the media can change the content of events such as touch, press or press in the time direction. In addition, for example, "the media can change the bound playback conditions in the time direction" may include: the media can change the location where the event occurs in the time direction. For example, it can include media that is played when the right side of the object is touched at time T1 and media that is played when the left side of the object is touched at time T2. Of course, as long as it is media that changes in the time direction, it can be any media, and it is not limited to these examples. On the other hand, for example, "media that does not change in the time direction" may include: media whose playback content (action) does not change in the time direction (media whose actions are the same at any time). In addition, for example, "media that does not change in the time direction" may include: media whose bound playback conditions (events) do not change in the time direction (media whose content of the event and the location where the event occurs are the same at any time). In this specification, the fact that the media can change in the time direction is also referred to as "dynamic". For example, media that can change in the time direction (timed media) is also referred to as dynamic media. For example, tactile media that can change in the time direction is also referred to as dynamic tactile media. In addition, the fact that it cannot change in the time direction is also referred to as "static". For example, media that does not change in the time direction is also referred to as static media. For example, tactile media that does not change in the time direction is also referred to as static tactile media.

[0204] <Scene Description Extensions for Tactile Media>

[0205] In addition, as described in Non-Patent Document 4, technical research and consideration on processing tactile media in MPEG-I scene description has started simultaneously with the standardization of coding and transmission technology of tactile media. Fig.32 Four gLTF extensions are shown: MPEG_haptic, MPEG_material_haptic, MPEG_avatar, and MPEG_interaction, to support tactile media in scene description.

[0206] MPEG_haptic is information (e.g., link information) about tactile media data (also referred to as tactile data) to be referenced from the scene description. This tactile data exists as independent data, just like data about sound, image, etc. In addition, this tactile data can be encoded (can be encoded data).

[0207] MPEG_material_haptic, which is a mesh / material extension of a defined 3D object, defines tactile material information (e.g., at which position of a 3D object (mesh) is the tactile media associated). Information about static tactile media is defined in the material information. In addition, information for accessing MPEG_haptic (e.g., link information) may also be defined in the tactile material information.

[0208] MPEG_avatar defines the 3D shape (avatar) of a user moving in three-dimensional space. MPEG_interaction lists the conditions (what the user can do) and possible actions (how the object reacts) that the avatar (user) can perform. For example, MPEG_interaction defines the interaction (i.e. events) that occurs between a user (MPEG_avatar) and a 3D object, and the actions that occur as a result (e.g. vibration occurs when the user touches a 3D object).

[0209] For example, when an avatar defined in MPEG_avatar causes an interaction (event) defined in MPEG_interaction, static tactile media (for example, rendering vibrations output by a vibration device) depending on the position where the interaction is generated, etc. is generated and played according to the material information in MPEG_materal_haptics to take an action corresponding to the interaction. MPEG_material_haptic is static information associated with texture information of the scene description. Otherwise, tactile data to be referenced by the MPEG_haptic shown in MPEG_materal_haptics is read out, and dynamic tactile media is generated and played. That is, MPEG_haptic is activated from the media_control of the action by the trigger (execution condition) of the interaction. An example of the semantics of MPEG_haptic is as follows: Fig.33 shown.

[0210] In non-patent document 3, a coding method for such tactile media is proposed. In this method, by using Fig.34 The architecture shown on the upper side encodes the tactile signal (wav) and the tactile signal description (ivs, ahap), and generates the exchange format (gmap) and the distribution format (mpg). Fig.34The table at the bottom shows a configuration example of the distribution format. As shown in the table, the bit stream of the tactile media is composed of a binary header and a binary body. In the binary header, the properties of the encoded data (tactile stream) of the tactile media, the rendering device, the information about the encoding method, etc. are stored. In addition, in the binary body, the encoded data (tactile stream) of the tactile media is stored.

[0211] For example, Fig.35 As shown, the binary header includes tactile file metadata, avatar metadata, perception metadata, reference device metadata and track header, and has the following Fig.35 The hierarchical structure shown.

[0212] The haptic file metadata includes information about the haptic media. Examples of the semantics of the haptic file metadata are as follows: Fig.36 As shown in FIG. 4 , avatar metadata includes information about an avatar. Examples of semantics of avatar metadata are as follows: Fig.37 As shown in Figure 1. Perception metadata includes information about how an item behaves. Examples of the semantics of this perception metadata are Fig.38 The reference device metadata includes information about the reference device (which device is moved and how it is moved). Examples of the semantics of the reference device metadata are as follows: Fig.39 The track header includes a track in which the binary data of the project is stored and information about playing the binary data. An example of the semantics of the track header is as follows: Fig.40 shown.

[0213] The binary body includes a band header, an instantaneous band body, a curved band body, and a wave band body. Examples of the semantics of the band header are as follows: Fig.41 In addition, examples of the semantics of transient bands and curved bands are shown in Fig.42 shown.

[0214] The band body is encoded using any of the vector band body, quantized band body, and small band body. Examples of these semantics are Fig.43 shown.

[0215] like Fig.44 As shown, the data structure regarding the tactile media having such information generally has a hierarchical structure.

[0216] like Fig.45 As shown, a reference device is defined by an ID, a name and an entity location. Optionally, specific properties can be specified for each device.

[0217] like Fig.46 As shown, a track is defined by an ID, description, body part (body_part), blending weights, gain values, and a list of haptic bands. Various additional properties may also be specified, such as a reference device ID, desired sampling frequency, and sample count.

[0218] like Fig.47 As shown, a haptic band is defined by the type of band, encoding mode, interpolation function, window length, frequency range, and a list of haptic effects.

[0219] like Fig.48 As shown, an effect is defined by its position (timestamp), its phase, its signal type and a list of keyframes.

[0220] <Interactive playback of media>

[0221] As described above, in the interactivity framework described in non-patent document 3, a trigger is defined as an interaction generation condition, and an action is defined as an interaction to be generated. In addition, a behavior that enumerates multiple triggers and multiple actions is defined. For example, in such a framework, there already exists a method for playing MPEG media (MPEG_media) as a target of interactive processing. In this specification, a method for playing media as interactive processing is also referred to as interactive playback. That is, in this framework, interactive playback of MPEG media is also set as a target.

[0222] However, in the MPEG-I scene description described in Non-Patent Document 2, only startTime and autoPlay are defined as the above-mentioned MPEG media. Therefore, it is difficult to interactively play the MPEG media from the scene description by using the interactivity framework described in Non-Patent Document 3.

[0223] <3. Media Definition Extension for Interactive Playback>

[0224] <Method 1>

[0225] In view of this, if Fig.49 As shown in the top row of the table of , a media item that specifies media processing performed through interaction is broadly defined (Method 1).

[0226] For example, the first information processing device is configured to include a playback unit that interactively plays the interactive media specified to be interactively played in the scene description according to the processing content of the interactive playback specified in the scene description when the execution conditions of the interactive playback specified in the scene description are met.

[0227] For example, in a first information processing method executed by a first information processing device, when the execution conditions of the interactive playback specified in the scene description are met, the interactive media specified in the scene description is interactively played according to the processing content of the interactive playback specified in the scene description.

[0228] It should be noted that interactive playback is a way of playing media as interactive processing. Moreover, interactive processing is an interactive processing of executing the processing content specified in the scene description when the execution conditions specified in the scene description are met. In addition, interactive media refers to media to be played interactively. Such interactive media can be any type of media. For example, interactive media can be visual information such as images (dynamic images), can be auditory information such as sounds, can be tactile information such as vibrations, and can be other information.

[0229] By broadly defining the media in this way, the first information processing device is based on interactively processing the media according to the scene description. That is, the first information processing device can correspond to all media operations that occur in the scene description. Therefore, by using, for example, the interactivity framework described in non-patent document 3, the first information processing device is able to interactively play the MPEG media in the scene description.

[0230] Furthermore, the second information processing apparatus is configured to include a providing unit that provides a scenario description in which there are a description for specifying that the interactive media is to be interactively played and a description for specifying execution conditions and processing contents of the interactive play.

[0231] For example, in a second information processing method executed by a second information processing apparatus, a scene description is provided in which there is a description for specifying that interactive media is to be played interactively and a description for specifying execution conditions and processing contents of the interactive playback.

[0232] It should be noted that interactive playback is a method of playing media as interactive processing. Moreover, interactive processing is an interactive processing of executing the processing content specified in the scene description when the execution conditions specified in the scene description are met. In addition, interactive media refers to media to be played interactively. Such interactive media can be any type of media. For example, interactive media can be visual information such as images (moving images), can be auditory information such as sounds, can be tactile information such as vibrations, and can be other information.

[0233] By broadly defining the media in this manner and the second information processing device providing such a scene description, the device to which the scene description is to be provided (e.g., the first information processing device) can interactively process the media based on the scene description. That is, the device to which the scene description is to be provided can correspond to all media operations occurring in the scene description. For example, the device to which the scene description is to be provided can interactively play the MPEG media in the scene description by using, for example, the interactivity framework described in non-patent document 3.

[0234] It should be noted that the method of specifying interactive media to be interactively played in a scene description may be any method. An example thereof will be described below.

[0235] <Extended Example 1>

[0236] The items used in the media array for MPEG media can be broadly defined, and the media can be specified to be played interactively.

[0237] For example, in the scene description, any one of a start time designation, an automatic play designation, or an interactive play designation may exist as a description about the media referenced from the scene description, and the interactive media may be specified to be interactively played by using an interactive play designation whose value is true. The providing unit of the second information processing device may provide such a scene description. Based on such a scene description, the playing unit of the first information processing device may interactively play the interactive media according to the processing content of the interactive play when the execution conditions of the interactive play are met. It should be noted that the start time designation is a description that specifies the start time of the playback of the media. The automatic play designation is a description that specifies whether to start playing the media immediately after the media is ready. The interactive play designation is a description that specifies whether to play the media interactively.

[0238] Fig.50 An example thereof is shown. As shown in this example, "interactiveplay" can be added as an item of the media array. The interactiveplay specifies that the media starts playing when the execution condition of the interactive process is met in the scene description interactivity framework. Any of startTime, autoplay or interactiveplay can exist as a description (media item) about the media referenced from the scene description.

[0239] That is, in this case, by describing the interactive play of the media in the scene description, it is specified that the media is to be played interactively (i.e., interactive media). Therefore, through such a description, the second information processing device (e.g., client device) that plays the content can interactively play the interactive media.

[0240] <Extended Example 2>

[0241] In addition, interactive playback of media can be enabled without automatically playing the media.

[0242] For example, in a scene description, as a description about the media to be referenced according to the scene description, in a case where the value of the automatic play designation is false, the interactive play designation can exist, and interactive media can be interactively played by using an automatic play designation with a false value and an interactive play designation with a true value. The providing unit of the second information processing device can provide such a scene description. Based on the scene description, the playback unit of the first information processing device can interactively play the interactive media according to the processing content of the interactive playback when the execution conditions of the interactive playback are met. It should be noted that the automatic play designation is a description that specifies whether to start playing the media immediately after the media is ready. The interactive play designation is a description that specifies whether to play the media interactively.

[0243] Fig.51 As shown in this example, in the case of autoplay=true, automatic play (i.e., a play method that starts playing the media immediately after the media is ready) can be applied, and in the case of autoplay=false, interactive play can be enabled.

[0244] In addition, in a state where the media is not automatically played, you can enable automatic playback specified by autoplay to play the media, or play the media by a playback method other than playing at a specified time specified by startTime (other methods). Then, you can enable interactive playback to apply it as one of the other methods.

[0245] For example, in a scene description, as a description about the media to be referenced according to the scene description, other method designations can exist when the value of the automatic play designation is false, and interactive media can be interactively played by using an automatic play designation with a false value and an interactive play designation with a true value. The providing unit of the second information processing device can provide such a scene description. Based on such a scene description, the playback unit of the first information processing device can interactively play the interactive media according to the processing content of the interactive playback when the execution conditions of the interactive playback are met. It should be noted that the automatic play designation is a description of whether to start playing the media immediately after the media is ready. Other method designations are descriptions that specify whether to apply playback methods other than specified time playback and automatic playback. Specified time playback is a playback method that starts playing the media at a specified time. Automatic playback is a playback method that starts playing the media immediately after the media is ready. Interactive playback designation is one of the other method designations, and is a description that specifies whether to play the media interactively.

[0246] Fig.52An example thereof is shown. As shown in the note, in this example, in the case of autoplay = true, automatic play (a play method that starts playing immediately after the media is ready) can be applied, and in the case of autoplay = false, another method can be applied. Thus, in this case, the media can be interactively played by applying interactiveplay.

[0247] That is, in the case of this example, if autoplay = false, a playback method other than interactiveplay can be applied (the applicable playback method is not limited to interactiveplay). Here, the applicable playback method can be any playback method. For example, as an item of the media array, "interruptplay" can be added, and interruptplay specifies that media processing is started when an interrupt event (although it is an object in the scene description, it is controlled outside the scene description) occurs. For example, in the case of the media description interruptplay, when another application controls an object in the scene as a metaverse space, an event that affects the scene is generated, and the interruption is notified from the external application to the PE of the scene description, so that the media can be played in the scene.

[0248] <Extended Example 3>

[0249] Furthermore, in a state where the media is not automatically played, interactive playing of the media may be considered.

[0250] For example, in the scene description, as a description about the media to be referenced according to the scene description, in the case where the value of the autoplay designation is false, it can be considered that the media has been designated to be played with interactive processing, and it can be designated to interactively play the interactive media by using the autoplay designation with a value of false. The providing unit of the second information processing device can provide such a scene description. Based on such a scene description, the playback unit of the first information processing device can interactively play the interactive media according to the processing content of the interactive playback when the execution conditions of the interactive playback are met. It should be noted that the autoplay designation is a description that specifies whether to start playing the media immediately after the media is ready.

[0251] Fig.53An example thereof is shown. In the case of this example, although interactiveplay is not additionally defined, if autoplay=false, the media is (considered to be) played interactively. That is, the media is regarded as interactive media and interactive play is applied. That is, even if interactiveplay is not described for the media, the processing content defined by the action is executed when the execution condition defined by the trigger is met. By defining autoplay in this way, the media can be played interactively without adding a new item (interactiveplay) (while suppressing the addition of parts that are changed from the conventional method). Therefore, the media can be played interactively more easily.

[0252] It should be noted that if Fig.54 As shown, such a definition can be performed in a comment instead of as a definition of autoplay. By defining it in a comment, there is no need to change the definition of autoplay. Also, in this case, Fig.52 As shown, when the media is not automatically played, it can be considered that the media is played by the automatic play specified by autoplay, or by a play method (other method) other than the specified time play specified by startTime. In other words, the play method considered to be applied can also include a play method other than interactive play. For example, the above-mentioned interruptplay can be considered to be applied.

[0253] <Extended Example 4>

[0254] In addition, interactive playback of media may be enabled in situations where a specified start time for playback of the media is inappropriate.

[0255] For example, in a scene description, as a description about the media to be referenced according to the scene description, in the case where the start time designation is a negative value, an interactive play designation can exist, and interactive playback of the interactive media can be designated by using a start time designation of a negative value and an interactive play designation of a true value. The providing unit of the second information processing device can provide such a scene description. Based on such a scene description, the playback unit of the first information processing device can interactively play the interactive media according to the processing content of the interactive playback when the execution conditions of the interactive playback are met. It should be noted that the start time designation is a description of the playback start time of the media. The interactive playback designation is a description that specifies whether the media is to be played interactively.

[0256] Fig.55An example is shown. StartTime is an item that specifies the start time of media playback, so its value is zero or positive (zero or positive is a normal value). In other words, a negative value is not suitable as the value of startTime (the time cannot be specified correctly). Therefore, the media cannot be played normally. In view of this, in this case, application interactive playback can be enabled. That is, Fig.55 As shown in the example, when startTime is not a negative value (zero or positive), the specified time play (a play method that starts playing the media at a specified time) is applied, and when startTime is a negative value, the application interactive play (i.e. interactive play) can be enabled.

[0257] In addition, if the specified media playback start time is not appropriate, you can enable automatic playback specified by autoplay to play the media, or a playback method other than playing at a specified time specified by startTime (other methods) to play the media. Then, you can enable interactive playback to apply it as one of the other methods.

[0258] For example, in a scene description, as a description about the media to be referenced according to the scene description, other method designations can exist when the start time designation is a negative value, and interactive media can be interactively played by using a start time designation of a negative value and an interactive play designation of a true value. The providing unit of the second information processing device can provide such a scene description. Based on such a scene description, the playback unit of the first information processing device can interactively play the interactive media according to the processing content of the interactive playback when the execution conditions of the interactive playback are met. It should be noted that the start time designation is a description of the playback start time of the specified media. Other method designations are descriptions that specify whether to apply playback methods other than specified time playback and automatic playback. Specified time playback is a playback method that starts playing media at a specified time. Automatic playback is a playback method that starts playing media immediately after the media is ready. Interactive playback designation is one of the other method designations, and is a description that specifies whether to play media interactively.

[0259] Fig.56 An example thereof is shown. As shown in the comments, in this example, when startTime is zero or a positive value, a specified time play (a play method that starts playing the media at a specified time) can be applied, and when startTime is a negative value (such as "-1"), another method can be applied. Then, in this case, the media can be interactively played by applying interactiveplay.

[0260] That is, in the case of this example, if startTime is a negative value, a playback method other than interactive play can be applied (the applicable playback method is not limited to interactive play). Here, the applicable playback method can be any playback method. For example, as in the above case, interrupt play can be applied.

[0261] <Extended Example 5>

[0262] Furthermore, in the case where the start time for specifying the playback of the media is inappropriate, it may be considered that the media is to be played interactively.

[0263] For example, in a scene description, as a description about the media to be referenced according to the scene description, in the case where the start time is specified as a negative value, it can be considered that the media has been specified to be played with interactive processing, and it can be specified to interactively play the interactive media by using the start time specification as a negative value. The providing unit of the second information processing device can provide such a scene description. Based on such a scene description, the playback unit of the first information processing device can interactively play the interactive media according to the processing content of the interactive playback when the execution conditions of the interactive playback are met. It should be noted that the start time specification is a description of the start time of the playback of the specified media.

[0264] Fig.57 An example thereof is shown. In the case of this example, although interactiveplay is not additionally defined, if startTime is a negative value, the media is (considered to be) interactively played. That is, the media is regarded as interactive media and interactive play is applied. That is, even if interactiveplay is not described for the media, the processing content defined by the action is executed when the execution condition defined by the trigger is met. By defining startTime in this way, the media can be interactively played without adding a new item (interactiveplay) (while suppressing the addition of parts changed from the conventional method). Therefore, the media can be interactively played more easily.

[0265] It should be noted that if Fig.58 As shown in , this definition can be performed in a comment instead of as a definition of startTime. By defining it in a comment, there is no need to change the definition of startTime. Also, in this case, Fig.56As in the example in , when the start time of the specified media playback is not appropriate, you can consider playing the media by the automatic playback specified by autoplay, or playing the media by a playback method (other method) other than the specified time playback specified by startTime. That is, the playback method to be applied may also include a playback method other than interactiveplay. For example, you can consider applying the above-mentioned interruptplay.

[0266] <Method 2>

[0267] The played media is output by a certain output device. For example, tactile data (tactile information) played as an action is rendered and output by a tactile device. In addition, image data (visual information) played as an action is displayed as an image by a display device such as a display. In addition, audio data (auditory information) played as an action is output as sound by an audio output device such as a speaker. Such an output device needs to perform settings, such as powering on and setting the operating mode, so that the information can be output. Such settings require a certain amount of time. Therefore, it is necessary to perform settings at an appropriate time (according to the properties of the device) before performing the action so that the information is output without delay at the time of performing the action.

[0268] However, as described above, in the framework described in Non-Patent Document 3 (MPEG-I Scene Description Interactivity Framework), since such setting of the device is not defined in the action, it is difficult to execute such setting of the device as interactive processing.

[0269] In view of this, if Fig.49 As shown in the second row from the top of the table, the action type for the setting of the specified device can be widely defined (method 2).

[0270] For example, in the scenario description, as the type of processing content of the interactive process, the setting of the device for outputting the interactive media can be specified. The providing unit of the second information processing device can provide such a scenario description. When the execution condition of the interactive process described in such a scenario description is met, the playback unit of the first information processing device can set the device according to the processing content of the interactive process.

[0271] For example, Fig.59 As shown, SETUP_DEVICE can be additionally defined in the type of action (processing content). SETUP_DEVICE is an operation related to the setting of a device. When this type is specified, the process related to the setting of the device will be executed as an interactive process (based on a trigger).

[0272] In addition, in the scene description, the execution condition of the device setting can be set to be satisfied before the execution condition of the interactive play. The providing unit of the second information processing device can provide such a scene description. The playing unit of the first information processing device can set the device according to such an execution condition.

[0273] As described above, in general, it is ideal to perform the settings of the device before playing the media. In view of this, the execution conditions of the settings of the device can be set to be satisfied before the execution conditions of interactively playing the media. For example, in the case where the execution condition is a temperature condition and the interactive media is played when the target temperature is equal to or higher than 30°C, it can be set to perform the settings of the device when the target temperature is equal to or higher than 25°C. In addition, in the case where the execution condition is a distance condition and the interactive media is played when the object approaches to within 1 meter, it can be set to perform the settings of the device when the object approaches to within 5 meters. Of course, both the execution conditions of media processing and the execution conditions of device settings can be any conditions, or can be conditions other than these examples.

[0274] Before the execution condition of the media processing is actually met or before the action is generated, the setting is set to execute the action for the appropriate execution condition (trigger), so that the interactive media can be output without delay through the device setting. Therefore, the degradation of the user experience quality can be suppressed. In addition, compared with the case where the device setting is performed at a predetermined timing in advance (for example, when the scene is initialized), the total power consumption of the device during this time can be reduced.

[0275] It should be noted that the device to be set can be any device. In addition, the content of the setting can also be any content. For example, a device driver (amplifier, fan, compressor, etc.) can be a setting target device. In addition, the power-on process of the device driver can be performed as a setting. In addition, a thermal presentation device can be a setting target device. In addition, in the thermal presentation device, temperature setting processes such as preheating processes and precooling processes can be performed as settings. In addition, a tactile presentation device can be set as a setting target device. In addition, in the tactile presentation device, air pressure setting processes such as slightly inflated air can be performed as settings.

[0276] You can specify the target devices for which the setting is enabled and the contents of the setting.

[0277] For example, in the scene description, the set processing target can be specified by using the index of the device. The providing unit of the second information processing device can provide such a scene description. The playback unit of the first information processing device can set the device specified as the processing target in such a scene description.

[0278] In addition, in the scene description, it can be specified that the device is capable of outputting interactive media as the set content. The providing unit of the second information processing device can provide such a scene description. The playing unit of the first information processing device can set the device so that the playing of the interactive media is enabled according to the specification of the processing content of the interactive process described in such a scene description.

[0279] For example, Fig.59 As shown, in the case where SETUP_DEVICE is specified as the type of action, device and deviceControl can be set as options. Device is an item that specifies the device that is the setting target. The device to be set can be specified by using the index of the device. DeviceControl is an item that specifies the content of the setting. That is, through this item, how to set can be specified. For example, by specifying SETUP as deviceControl, the device is set to enable playing interactive media (the device is enabled to perform the action immediately).

[0280] In this way, the desired content can be set up as an interactive process associated with the desired device.

[0281] It should be noted that setup can be performed outside of device initialization. For example, you can use deviceControl to specify when and how setup should be performed.

[0282] For example, in the scene description, it can be specified that the initialization device is set to the setting content. The providing unit of the second information processing device can provide such a scene description. The playback unit of the first information processing device can set the initialization device according to the specification of the processing content of the interactive process described in such a scene description.

[0283] In addition, in the scene description, the processing content and the processing mode to be executed by the device can be specified as the setting content to be updated. The providing unit of the second information processing device can provide such a scene description. The playback unit of the first information processing device can set the device to update the processing content and the processing mode to be executed by the device according to the specification of the processing content of the interactive processing described in such a scene description.

[0284] exist Fig.60 In the example shown, INITIAL_SETUP and CHANGE_SETUP can be provided as deviceControl. INITIAL_SETUP specifies settings that enable the initialization device to perform actions immediately. CHANGE_SETUP specifies settings that enable the device to perform a desired operation among the multiple types of operations that can be performed, or settings that switch the operating mode to a desired mode.

[0285] By applying options in this way, more detailed control (more specific specification of operations) can be achieved.

[0286] <Method 3>

[0287] For example, in the case where the interactive media that is the target of interactive playback exists in other devices (e.g., a server such as a CDN), it is necessary to obtain the interactive media before interactive playback. For interactive playback without delay, it is necessary to obtain the interactive media at an appropriate timing before playback. In addition, depending on the configuration and function of the output device, there may be a case where the media data needs to be set to the output device.

[0288] However, as described above, in the framework described in non-patent document 3 (MPEG-I Scene Description Interactivity Framework), since such processing before playback (also called preliminary processing) is not defined in the action, it is difficult to perform such preliminary processing as interactive processing.

[0289] In view of this, if Fig.49 As shown in the third row from the top of the table, the action type for pre-processing the media for playback can be widely defined (method 3).

[0290] For example, in the scene description, as the type of processing content of the interactive process, it can be specified that preprocessing is performed on the interactive media before the interactive media is interactively played. The providing unit of the second information processing device can provide such a scene description. The first information processing device can also include a preliminary processing unit, which performs preliminary processing according to the processing content of the interactive process when the execution condition of the interactive process is met in such a scene description.

[0291] For example, Fig.61 As shown, MEDIA_PRE_PROCESS can be additionally defined in the type of action (processing content). MEDIA_PRE_PROCESS is an operation related to preliminary processing of media (processing performed before media playback). When this type is specified, the preliminary processing will be performed as an interactive process (based on a trigger).

[0292] You can enable specifying the target media for preliminary processing and the content for preliminary processing.

[0293] For example, in the scene description, the processing target of the preliminary processing can be specified by using the index of the media. The providing unit of the second information processing device can provide such a scene description. The preliminary processing unit of the first information processing device can perform preliminary processing on the interactive media specified as the processing target in such a scene description.

[0294] For example, Fig.61 As shown, in the case where MEDIA_PRE_PROCESS is specified as the type of action, media can be set as an option. media is an item that specifies the media that is the target of preliminary processing. The media to be preliminarily processed can be specified by using the index of the media.

[0295] In addition, when MEDIA_PRE_PROCESS is specified as the type of action, mediaProcess can be set as an option. mediaProcess is an item that specifies the content of preliminary processing. That is, through this item, the type of processing to be performed is specified.

[0296] In this way, for the desired media, preliminary processing of the desired content can be performed as interactive processing.

[0297] The mediaProcess can be any value.

[0298] For example, in the scene description, as the content of preliminary processing, it can be specified to obtain interactive media. The providing unit of the second information processing device can provide such a scene description. As preliminary processing, the preliminary processing unit of the first information processing device can obtain the encoded data of the interactive media according to the description of the preliminary processing content in such a scene description.

[0299] For example, Fig.62 As shown, MEDIA_FETCH can be provided as mediaProcess. MEDIA_FETCH specifies the process of obtaining media locally at the terminal as a preliminary process. That is, as a preliminary process, by setting this MEDIA_FETCH as mediaProcess in the scene description, media can be obtained from a server or the like.

[0300] In addition, in the scene description, as the content of preliminary processing, the acquisition and decoding of interactive media can be specified. The providing unit of the second information processing device can provide such a scene description. As preliminary processing, the preliminary processing unit of the first information processing device can acquire and decode the encoded data of the interactive media according to the specification of the preliminary processing content in such scene description.

[0301] For example, Fig.62As shown, MEDIA_DECODE can be provided as mediaProcess. MEDIA_DECODE specifies the process of decoding the media as preliminary processing. That is, as preliminary processing, by setting MEDIA_DECODE as mediaProcess in the scene description, the encoded data of the media can be obtained from the server, etc., the encoded data can be decoded, and data about the media can be generated.

[0302] In addition, in the scene description, as the content of preliminary processing, the acquisition, decoding and conversion of interactive media can be specified. The providing unit of the second information processing device can provide such a scene description. As preliminary processing, the preliminary processing unit of the first information processing device can acquire and decode the encoded data of the interactive media according to the specification of the preliminary processing content in such scene description, and convert the decoded interactive media according to the properties of the device that outputs the interactive media.

[0303] For example, Fig.62 As shown, MEDIA_TRANSRATION can be provided as mediaProcess. MEDIA_TRANSRATION specifies the process of converting the decoded media according to the device properties as a preliminary process. That is, as a preliminary process, in the scene description, by setting MEDIA_TRANSRATION to mediaProcess, the encoded data of the media can be obtained, and the encoded data can be decoded to generate media data. Then, the generated media data can be converted according to the properties of the device that outputs the media.

[0304] In addition, in the scene description, as the content of preliminary processing, the setting of the interactive media can be specified. The providing unit of the second information processing device can provide such a scene description. As preliminary processing, the preliminary processing unit of the first information processing device can set the interactive media to the memory of the device that outputs the interactive media according to the specification of the preliminary processing content in such scene description.

[0305] For example, Fig.62 As shown, MEDIA_DATA_SET can be provided as mediaProcess. MEDIA_DATA_SET specifies the process of setting media to (the memory of) the output device as a preliminary process. That is, as a preliminary process, by setting this MEDIA_DATA_SET as mediaProcess in the scene description, the memory of the output device can be set to the media. That is, the device can be enabled to output (play) the media.

[0306] In addition, in the scene description, as the content of preliminary processing, deletion of interactive media can be specified. The providing unit of the second information processing device can provide such a scene description. As preliminary processing, the preliminary processing unit of the first information processing device can delete the interactive media set to the memory of the device that outputs the interactive media according to the specification of the content of preliminary processing in such scene description.

[0307] For example, Fig.62 As shown, MEDIA_DATA_RELEASE can be provided as mediaProcess. As a preliminary process, MEDIA_DATA_RELEASE specifies the process of releasing (deleting) the media set to the memory of the output device. That is, as a preliminary process, by setting this MEDIA_DATA_RELEASE as mediaProcess in the scene description, the media can be released (deleted) from the memory of the output device.

[0308] By applying the option in this way, more detailed control (more specific operating specifications) can be achieved.

[0309] <Method 4>

[0310] The settings described in <Method 2> and the preliminary processing described in <Method 3> can be performed before rendering the scene. That is, in the case where such processing is set as interactive processing, in the scene description, there are behaviors that are executed when rendering the scene and behaviors that are executed before rendering. Therefore, since it is necessary to check for behaviors that are not executed, unnecessary processing may be added.

[0311] In view of this, if Fig.49 As shown in the fourth row from the top of the table, objects that specify behaviors for pre-processing media for playback can be widely defined (Method 4).

[0312] For example, in the scenario description, a behavior that specifies the execution conditions and processing contents of preliminary processing performed on the interactive media before the interactive media is interactively played may be described. The providing unit of the second information processing device may provide such a scenario description. The first information processing device may further include a preliminary processing unit that performs preliminary processing according to the behavior described in such a scenario description.

[0313] For example, Fig.63As shown, the behavior of preliminary processing "Pre_behavior" can be provided as an object of behavior. Pre_behavior is a behavior for defining preliminary processing, and binds the trigger of preliminary processing with the operation of preliminary processing. Here, preliminary processing refers to processing performed before rendering the scene. The processing content can be any processing content, and includes not only the preliminary processing mentioned above, but also the setting device. For example, setting the initialization device can also be included in the preliminary processing.

[0314] By defining the "Pre_behavior" of the behavior, it can be clearly indicated that it is necessary to perform processing (behavior) related to the scene description before performing the rendering operation of the scene described. It should be noted that this Pre_behavior can be applied to each example of the above-mentioned method 2 and method 3.

[0315] By specifying that the behavior is a behavior for preliminary processing in this way, the playback device can easily grasp that the behavior is a behavior for preliminary processing. Therefore, for example, it is possible to prevent the behavior for preliminary processing and analysis from being referred to when rendering the scene, and to suppress the increase in the load of the playback process.

[0316] Note that triggers and actions for providing preliminary processing may be provided instead of providing the behavior for preliminary processing.

[0317] For example, in the scenario description, an execution condition for specifying the execution of preliminary processing for interactive media before interactively playing the interactive media and the processing content specified as the preliminary processing may be described. The providing unit of the second information processing device may provide such a scenario description. The first information processing device may further include a preliminary processing unit that performs preliminary processing according to the processing content of the preliminary processing when the execution condition for the preliminary processing described in the scenario description is met.

[0318] For example, Fig.64 As shown, "Pre-triggers" and "Pre-Actions" can be provided. Pre-triggers are triggers for preliminary processing and specify the execution conditions of preliminary processing. In other words, the execution conditions designated as Pre-triggers are the execution conditions of preliminary processing. Pre-Actions specify the processing content (action) of preliminary processing. In other words, the processing content designated as Pre-Actions is the processing content of preliminary processing.

[0319] By specifying the trigger and action as preliminary processing in this way, it is easy to identify whether it is preliminary processing or processing performed when rendering the scene in the scene description. Therefore, API processing becomes easy. It should be noted that Pre-triggers and Pre-Actions can be applied to each example of the above-mentioned method 2 and method 3.

[0320] <Method 5>

[0321] In the case where the triggers and actions of behaviors are controlled as scenes (i.e., in the case where interactive processing is controlled by using scene descriptions), there may be cases where the operation desired by the content author cannot be obtained depending on the operation content of the action. For example, in the case of device operation requiring low latency, processing delays may reduce the quality of user experience under the control of scene descriptions.

[0322] In view of this, if Fig.49 As shown in the fourth row from the top of the table in , an object of a behavior can be specified, and the object of the behavior specifies switching control from the scene description to the application (method 5). That is, the interactive media can be interactively played under the control of the application outside the scene description. Then, control can be performed in the scene description. For example, such control functions can be added to the behavior.

[0323] For example, in the scene description, a behavior specified for controlling the interactive process in the application can be described. The providing unit of the second information processing device can provide such a scene description. The playing unit of the first information processing device can enable the application to control the interactive process according to the behavior described in such a scene description.

[0324] For example, Fig.65 As shown, "Application_control" can be provided as an object of behavior. The Application_control specifies that when the trigger and operation of the behavior are activated, the application will process the behavior. That is, the interactive processing indicated by the behavior is not controlled by the scene description. For example, when Application_control is set, the execution condition (trigger) is sensed, the media is set to the device that executes the processing content (operation), and the action operation is performed. Through this function, it can be indicated that the content author's intention to perform processing outside the scene description is effective.

[0325] For example, assume that the trigger (execution condition) of the first behavior is that a finger approaches the trigger of a pistol-type controller exceeding a threshold, and the action (processing content) is tactile feedback. In addition, in this behavior, assume that Application_control has been set to true (=1). In addition, assume that the trigger (execution condition) of the second behavior is that the trigger of the pistol-type controller is pulled with a finger, and the action (processing content) is vibration feedback expressing the state that the pistol has been fired. In addition, assume that in this behavior, Application_control has been set to false (=0).

[0326] Since the first behavior is set to Application_control = 1, it is out of the control of the scene description after being executed once, and when the trigger occurs, the interactive media is input to the tactile device. Therefore, ultra-low-latency tactile feedback can be achieved. That is, the first behavior is not controlled by the scene description. On the other hand, since the second behavior is feedback that does not require ultra-low latency, the trigger and action are controlled by the scene description. In this way, suitable playback for the content author's intention can be performed.

[0327] Note that the "Application_control" can be defined as an action type instead of a behavior type.

[0328] <Combination>

[0329] The above methods 1 to 5 may be applied alone or in appropriate combination. In addition, these methods may be applied in combination with methods other than the above methods.

[0330] <4. First Embodiment>

[0331] <File creation device>

[0332] The present technology described above can be applied to any device. Fig.66 : is a block diagram showing an example of the configuration of a file generating device as one aspect of an information processing device to which the present technology is applied. Fig.66 The file generating device 300 shown is a device for encoding 3D object content (eg, 3D data such as point cloud) associated with interactive media such as tactile media and storing the file. In addition, the file generating device 300 generates a scene description file of the 3D object content.

[0333] It should be noted that Fig.66 shows the main elements such as processing units and data flow, and can be found in Fig.66 That is, in the file generating device 300, there may be Fig.66 The processing units are shown as blocks and may exist without Fig.66 The processing and flow of data are shown as arrows etc.

[0334] like Fig.66 As shown, the file generation apparatus 300 includes a control unit 301 and a file generation processing unit 302. The control unit 301 controls the file generation processing unit 302. The file generation processing unit 302 is controlled by the control unit 301 to perform processing related to the generation of a file.

[0335] The file generation processing unit 302 includes an input unit 311 , a preprocessing unit 312 , an encoding unit 313 , a file generation unit 314 , a storage unit 315 , and an output unit 316 .

[0336] The input unit 311 performs processing related to acquiring data provided from outside the file generating device 300. Therefore, it can also be said that the input unit 311 is an acquisition unit. The input unit 311 includes an SD input unit 321, a 3D input unit 322, and an IM input unit 323. The SD input unit 321 acquires scene configuration data (data for scene description generation) provided to the file generating device 300. The SD input unit 321 generates a scene description by using the acquired scene configuration data. Therefore, it can also be said that the SD input unit 321 is a generation unit (or SD generation unit) that generates a scene description. It should be noted that the scene description can be provided from outside the file generating device 300. In this case, the SD input unit 321 only needs to acquire the scene description and skip (omit) the scene description generation. Therefore, it can also be said that the SD input unit 321 is an acquisition unit (or SD acquisition unit) that acquires scene configuration data or scene description. The SD input unit 321 provides the acquired or generated scene description to the SD preprocessing unit 331 of the preprocessing unit 312. The 3D input unit 322 acquires 3D data provided to the file generating device 300. The 3D input unit 322 provides the acquired 3D data to the 3D preprocessing unit 332 of the preprocessing unit 312. Therefore, it can also be said that the 3D input unit 322 is an acquisition unit (or 3D acquisition unit) that acquires 3D data. The IM input unit 323 acquires data of interactive media (also referred to as IM data) provided to the file generating device 300. The IM input unit 323 provides the acquired IM data to the IM preprocessing unit 333 of the preprocessing unit 312. Therefore, it can also be said that the IM input unit 323 is an acquisition unit (or IM acquisition unit) that acquires IM data.

[0337] The preprocessing unit 312 performs processing related to preprocessing performed before encoding on the data provided from the input unit 311. The preprocessing unit 312 includes an SD preprocessing unit 331, a 3D preprocessing unit 332, and an IM preprocessing unit 333. The SD preprocessing unit 331 acquires information required to generate a file (also referred to as an SD file) storing the scene description, for example, based on the scene description provided from the SD input unit 321, and provides the information to the SD file generating unit 351 of the file generating unit 314. In addition, the SD preprocessing unit 331 stores information about the 3D data provided from the 3D preprocessing unit 332, information about the IM data provided from the IM preprocessing unit 333, and information about alternative media data in the scene description. At this time, the SD preprocessing unit 331 can generate new information based on the provided information, and store the new information in the scene description. Furthermore, the SD preprocessing unit 331 supplies the scene description to the SD encoding unit 341 of the encoding unit 313. The 3D preprocessing unit 332 acquires information required to generate a file storing 3D data (also referred to as a 3D file) based on, for example, the 3D data supplied from the 3D input unit 322, and supplies the file to the 3D file generating unit 352 of the file generating unit 314. Furthermore, the 3D preprocessing unit 332 extracts information to be stored in the scene description and information for generating the information to be stored in the scene description from the 3D data, and supplies them as information about the 3D data to the SD preprocessing unit 331. Furthermore, the 3D preprocessing unit 332 supplies the 3D data to the 3D encoding unit 342 of the encoding unit 313. The IM preprocessing unit 333 acquires information required to generate a file storing IM data (also referred to as an IM file) based on, for example, the IM data supplied from the IM input unit 323, and supplies the information to the IM file generating unit 353 of the file generating unit 314. In addition, the IM preprocessing unit 333 extracts information to be stored in the scene description and information for generating the information to be stored in the scene description from the IM data, and supplies them as information about the IM data to the SD preprocessing unit 331. In addition, the IM preprocessing unit 333 supplies the IM data to the IM encoding unit 343 of the encoding unit 313.

[0338] The encoding unit 313 performs processing related to encoding the data supplied from the preprocessing unit 312. The encoding unit 313 includes an SD encoding unit 341, a 3D encoding unit 342, and an IM encoding unit 343. The SD encoding unit 341 encodes the scene description supplied from the SD preprocessing unit 331, and supplies the encoded data to the SD file generating unit 351 of the file generating unit 314. The 3D encoding unit 342 encodes the 3D data supplied from the 3D preprocessing unit 332, and supplies the encoded data to the 3D file generating unit 352 of the file generating unit 314. The IM encoding unit 343 encodes the IM data supplied from the IM preprocessing unit 333, and supplies the encoded data to the IM file generating unit 353 of the file generating unit 314.

[0339] The file generation unit 314 performs processing related to the generation of files, etc. The file generation unit 314 includes an SD file generation unit 351, a 3D file generation unit 352, and an IM file generation unit 353. The SD file generation unit 351 generates an SD file storing a scene description based on information provided from the SD preprocessing unit 331 and the SD encoding unit 341. The SD file generation unit 351 provides the SD file to the SD storage unit 361 of the storage unit 315. The 3D file generation unit 352 generates a 3D file storing encoded data of 3D data based on information provided from the 3D preprocessing unit 332 and the 3D encoding unit 342. The 3D file generation unit 352 provides the 3D file to the 3D storage unit 362 of the storage unit 315. The IM file generation unit 353 generates an IM file storing encoded data of IM data based on information provided from the IM preprocessing unit 333 and the IM encoding unit 343. The IM file generation unit 353 provides the IM file to the IM storage unit 363 of the storage unit 315.

[0340] The storage unit 315 includes any storage medium (e.g., a hard disk or a semiconductor memory), and performs processing related to data storage. The storage unit 315 includes an SD storage unit 361, a 3D storage unit 362, and an IM storage unit 363. The SD storage unit 361 stores the SD file provided from the SD file generation unit 351. In addition, the SD storage unit 361 provides the SD file to the SD output unit 371 according to a request from the SD output unit 371 of the output unit 316 or the like or at a predetermined timing. The 3D storage unit 362 stores the 3D file provided from the 3D file generation unit 352. In addition, the 3D storage unit 362 provides the 3D file to the 3D output unit 372 according to a request from the 3D output unit 372 of the output unit 316 or the like or at a predetermined timing. The IM storage unit 363 stores the IM file provided from the IM file generation unit 353. In addition, the IM storage unit 363 provides the IM file to the IM output unit 373 according to a request from the IM output unit 373 of the output unit 316 or the like or at a predetermined timing.

[0341] The output unit 316 acquires a file or the like provided from the storage unit 315, and outputs the file or the like to the outside of the file generating device 300 (e.g., a distribution server or a playback device). The output unit 316 includes an SD output unit 371, a 3D output unit 372, and an IM output unit 373. The SD output unit 371 acquires an SD file read out from the SD storage unit 361, and outputs the SD file to the outside of the file generating device 300. The 3D output unit 372 acquires a 3D file read out from the 3D storage unit 362, and outputs the 3D file to the outside of the file generating device 300. The IM output unit 373 acquires an IM file read out from the IM storage unit 363, and outputs the IM file to the outside of the file generating device 300.

[0342] That is, it can also be said that the output unit 316 is a providing unit that provides files to other devices. It can also be said that the SD output unit 371 is a providing unit that provides SD files to other devices (or an SD file providing unit). It can also be said that the 3D output unit 372 is a providing unit that provides 3D files to other devices (or a 3D file providing unit). In addition, it can also be said that the IM output unit 373 is a providing unit that provides IM files to other devices (or an IM file providing unit).

[0343] In the file generation device 300 having the above configuration, the above second information processing device is adopted, and in <3. Interactive playback media definition extension>, the above present technology can be applied. For example, any one or more of the above methods 1 to 5 can be applied to the file generation device 300.

[0344] By doing so, the media can be interactively processed through scene descriptions.

[0345] <File Generation Process Flow 1>

[0346] Next, the flow of the file generation process performed by the file generation device 300 will be described. As described above, the above method 1 can be applied to the file generation device 300. Fig.67 An example of the flow of the file generation processing in this case is described with reference to the flowchart in FIG.

[0347] When the file generation process has started, in step S301, the SD input unit 321 acquires a scene description. It should be noted that the SD input unit 321 can acquire scene configuration data and generate a scene description by using the scene configuration data. The 3D input unit 322 acquires 3D object content (3D data). The IM input unit 323 acquires interactive media (IM data).

[0348] In step S302, the SD preprocessing unit 331 performs preprocessing on the scene description and extracts information required to generate the SD file from the scene description. The 3D preprocessing unit 332 performs preprocessing on the 3D data and extracts information required to generate the 3D file from the 3D data. The IM preprocessing unit 333 performs preprocessing on the IM data and extracts information required to generate the IM file from the IM data.

[0349] In step S303, the 3D preprocessing unit 332 generates information about 3D data. In addition, the IM preprocessing unit 333 generates information about IM data. The SD preprocessing unit 331 generates a description of the correlation between the 3D object content (3D data) and the interactive media (IM data) based on such information, etc., and stores the description in the scene description.

[0350] In step S304, the SD pre-processing unit 331 generates a description of one or more behaviors including the playback conditions (trigger) of the interactive media (IM data) and the playback processing contents (actions) thereof, and stores the description in the scene description.

[0351] In step S305, the SD preprocessing unit 331 generates a description indicating that it is interactive media (IM data) to be interactively played, and stores the description in the scene description.

[0352] In step S306, the SD encoding unit 341 encodes the scene description to generate encoded data of the scene description. The 3D encoding unit 342 encodes the 3D object content (3D data) to generate encoded data of the 3D data. The IM encoding unit 343 encodes the interactive media (IM data) to generate encoded data of the IM data.

[0353] In step S307, the SD file generation unit 351 generates an SD file storing the encoded data of the scene description. The 3D file generation unit 352 generates a 3D file storing the encoded data of the 3D data. The IM file generation unit 353 generates an IM file storing the encoded data of the IM data.

[0354] In step S308, the SD storage unit 361 stores SD files, the 3D storage unit 362 stores 3D files, and the IM storage unit 363 stores IM files.

[0355] In step S309, the SD output unit 371 outputs an SD file and provides the SD file to other devices (e.g., client devices). The 3D output unit 372 outputs a 3D file and provides the 3D file to other devices (e.g., client devices). The IM output unit 373 outputs an IM file and provides the IM file to other devices (e.g., client devices).

[0356] When the processing in step S309 ends, the file generation processing ends.

[0357] By performing the corresponding processing in this way, the file generation device 300 can apply the method 1 of the present technology. Therefore, the media can be interactively processed through the scene description.

[0358] <File Generation Process Flow 2>

[0359] In addition, the above-mentioned method 1 and method 2 can be applied to the file generating device 300. Fig.68 An example of the flow of the file generation processing in this case is described with reference to the flowchart in FIG.

[0360] When the file generation process has started, the same as steps S301 to S304 ( Fig.67 ) and execute the corresponding processing in steps S321 to S324 in the same manner.

[0361] In step S325 , the SD preprocessing unit 331 generates a description of one or more behaviors including the setting conditions (triggers) of the device and the setting contents (actions) thereof, and stores the description in the scene description.

[0362] and step S305 to step S309 ( Fig.67 ) and similarly execute the corresponding processing in steps S326 to S330.

[0363] When the processing in step S330 ends, the file generation processing ends.

[0364] By performing the corresponding processing in this way, the file generation device 300 can apply the method 1 and the method 2 of the present technology. Therefore, the media can be interactively processed by the scene description.

[0365] <File Generation Process Flow 3>

[0366] In addition, the above-mentioned method 1 and method 3 can be applied to the file generating device 300. Fig.69 An example of the flow of the file generation processing in this case is described with reference to the flowchart in FIG.

[0367] When the file generation process has started, the same as steps S301 to S304 ( Fig.67 ) and similarly execute the corresponding processing in steps S351 to S354.

[0368] In step S355, the SD pre-processing unit 331 generates a description of one or more behaviors including preliminary processing execution conditions (triggers) of interactive media and preliminary processing contents (actions) thereof, and stores the description in the scene description.

[0369] and step S305 to step S309 ( Fig.67 ) and similarly execute the corresponding processing in steps S356 to S360.

[0370] When the processing in step S360 ends, the file generation processing ends.

[0371] By performing the corresponding processing in this way, the file generation device 300 can apply the method 1 and the method 3 of the present technology. Therefore, the media can be interactively processed through the scene description.

[0372] <File Generation Process Flow 4>

[0373] In addition, the above-mentioned methods 1 to 3 can be applied to the file generating device 300. Fig.70 An example of the flow of the file generation processing in this case is described with reference to the flowchart in FIG.

[0374] When the file generation process has started, the same as steps S321 to S325 ( Fig.68 ) and similarly execute the corresponding processing in steps S371 to S375.

[0375] and step S355 to step S360 ( Fig.69 ) and similarly execute the corresponding processing in steps S376 to S381.

[0376] When the processing in step S381 ends, the file generation processing ends.

[0377] By performing the corresponding processing in this way, the file generation device 300 can apply the method 1 to the method 3 of the present technology. Therefore, the media can be interactively processed through the scene description.

[0378] <File Generation Process Flow 5>

[0379] In addition, the above-mentioned method 1, method 2 and method 4 can be applied to the file generating device 300. Fig.71 An example of the flow of the file generation processing in this case is described with reference to the flowchart in FIG.

[0380] When the file generation process has started, the same as steps S371 to S375 ( Fig.70 ) and similarly executes the corresponding processing in steps S401 to S405.

[0381] In step S406 , the SD pre-processing unit 331 generates a description of one or more actions that are specified to be preliminary processed, and stores the description in the scene description.

[0382] and steps S377 to S381 ( Fig.70 ) and similarly execute the corresponding processing in steps S407 to S411.

[0383] When the processing in step S411 ends, the file generation processing ends.

[0384] By performing the corresponding processing in this way, the file generation device 300 can apply the method 1, method 2, and method 4 of the present technology. Therefore, the media can be interactively processed through the scene description.

[0385] <File Generation Process Flow 6>

[0386] In addition, the above-mentioned method 1, method 3 and method 4 can be applied to the file generating device 300. Fig.72 An example of the flow of the file generation processing in this case is described with reference to the flowchart in FIG.

[0387] When the file generation process has started, the same as steps S401 to S404 ( Fig.71 ) and similarly execute the corresponding processing in steps S431 to S434.

[0388] In step S435, the SD pre-processing unit 331 generates a description of one or more behaviors including preliminary processing execution conditions (triggers) of interactive media and preliminary processing contents (actions) thereof, and stores the description in the scene description.

[0389] and step S406 to step S411 ( Fig.71 ) and similarly execute the corresponding processing in steps S436 to S441.

[0390] When the processing in step S441 ends, the file generation processing ends.

[0391] By performing the corresponding processing in this way, the file generation device 300 can apply the method 1, method 3, and method 4 of the present technology. Therefore, the media can be interactively processed through the scene description.

[0392] <File Generation Process Flow 7>

[0393] In addition, the above-mentioned methods 1 to 4 can be applied to the file generating device 300. Fig.73 An example of the flow of the file generation processing in this case is described with reference to the flowchart in FIG.

[0394] When the file generation process has started, the same as steps S431 to S434 ( Fig.72 ) and execute the corresponding processing in steps S461 to S464 in the same manner.

[0395] In step S465, the SD preprocessing unit 331 generates a description of one or more behaviors including the setting conditions (triggers) and setting contents (actions) of the device, and stores the description in the scene description.

[0396] and step S435 to step S441 ( Fig.72 ) and execute the corresponding processing in steps S466 to S472 in the same manner.

[0397] When the processing in step S472 ends, the file generation processing ends.

[0398] By performing the corresponding processing in this way, the file generation device 300 can apply the methods 1 to 4 of the present technology. Therefore, the media can be interactively processed through the scene description.

[0399] <File Generation Process Flow 8>

[0400] In addition, the above-mentioned method 1 and method 5 can be applied to the file generating device 300. Fig.74 An example of the flow of the file generation processing in this case is described with reference to the flowchart in FIG.

[0401] When the file generation process has started, the same as steps S301 to S304 ( Fig.67 ) and similarly execute the corresponding processing in steps S491 to S494.

[0402] In step S495, the SD pre-processing unit 331 determines whether there is a behavior desired to be controlled by the application. In the case Application_control has been set to true and it is determined that there is a behavior desired to be controlled by the application, the process proceeds to step S496.

[0403] In step S496 , the SD pre-processing unit 331 generates a description of one or more behaviors specified to be controlled by the application, and stores the description in the scene description.

[0404] When the processing in step S496 ends, the processing proceeds to step S497. In addition, in the case where it is determined in step S495 that there is no behavior desired to be controlled by the application, the processing in step S496 is skipped and the processing proceeds to step S497.

[0405] and step S305 to step S309 ( Fig.67 ) and similarly execute the corresponding processing in steps S497 to S501.

[0406] When the processing in step S501 ends, the file generation processing ends.

[0407] By performing the corresponding processing in this way, the file generation device 300 can apply the method 1 and the method 5 of the present technology. Therefore, the media can be interactively processed through the scene description.

[0408] The combination of the methods of the present technology described above are examples, and the combination of the methods of the present technology may have any mode and is not limited to these examples.

[0409] <5. Second Embodiment>

[0410] <Client Device>

[0411] The present technology described above can be applied to any device. Fig.75 : is a block diagram showing an example of a configuration of a client device as one aspect of an information processing device to which the present technology is applied. Fig.75 The client device 700 shown is a playback device that performs playback processing of 3D data or IM data associated with 3D data based on scene description. For example, the client device 700 obtains a file generated by the file generation device 300 and plays the 3D data or IM data stored in the file.

[0412] It should be noted that Fig.75 shows the main elements such as processing units and data flow, and can provide Fig.75 That is, in the client device 700, there may be Fig.75 The processing units are shown as blocks and may exist without Fig.75 The processing and flow of data are shown as arrows etc.

[0413] like Fig.75 As shown, the client device 700 includes a control unit 701 and a client processing unit 702. The control unit 701 performs processing related to the control of the client processing unit 702. The client processing unit 702 performs processing related to the playback of 3D data or IM data.

[0414] The client processing unit 702 includes an acquisition unit 711 , a file processing unit 712 , a decoding unit 713 , an SD analysis unit 714 , an output control unit 715 , and an output unit 716 .

[0415] The acquisition unit 711 performs processing related to acquiring data provided from a distribution server, the file generation device 300, etc. to the client device 700. For example, the acquisition unit 711 includes an SD acquisition unit 721, a 3D acquisition unit 722, and an IM acquisition unit 723. The SD acquisition unit 721 acquires an SD file provided from outside the client device 700, and provides the SD file to an SD file processing unit 731 of the file processing unit 712. The 3D acquisition unit 722 acquires a 3D file provided from outside the client device 700, and provides the 3D file to a 3D file processing unit 732 of the file processing unit 712. The IM acquisition unit 723 acquires an IM file provided from outside the client device 700, and provides the IM file to an IM file processing unit 733 of the file processing unit 712.

[0416] The file processing unit 712 performs processing related to the file acquired by the acquisition unit 711. For example, the file processing unit 712 can extract data stored in the file. The file processing unit 712 includes an SD file processing unit 731, a 3D file processing unit 732, and an IM file processing unit 733. The SD file processing unit 731 acquires the SD file provided from the SD acquisition unit 721, extracts the encoded data of the scene description from the SD file, and provides the encoded data to the SD decoding unit 741. The 3D file processing unit 732 acquires the 3D file provided from the 3D acquisition unit 722, extracts the encoded data of the 3D data from the 3D file, and provides the encoded data to the 3D decoding unit 742. The IM file processing unit 733 acquires the IM file provided from the IM acquisition unit 723, extracts the encoded data of the IM data from the IM file, and provides the encoded data to the IM decoding unit 743.

[0417] The decoding unit 713 performs processing related to decoding the encoded data supplied from the file processing unit 712. The decoding unit 713 includes an SD decoding unit 741, a 3D decoding unit 742, and an IM decoding unit 743. The SD decoding unit 741 decodes the encoded data of the scene description supplied from the SD file processing unit 731 to generate (restore) the scene description, and supplies the decoded data to the SD analysis unit 714. The 3D decoding unit 742 decodes the encoded data of the 3D data supplied from the 3D file processing unit 732 to generate (restore) the 3D data, and supplies the decoded data to the 3D output control unit 752. The IM decoding unit 743 decodes the encoded data of the IM data supplied from the IM file processing unit 733 to generate (restore) the IM data, and supplies the decoded data to the IM output control unit 753.

[0418] The SD analysis unit 714 performs processing related to the analysis of the scene description. For example, the SD analysis unit 714 acquires the scene description provided from the SD decoding unit 741 and analyzes the scene description. In addition, the SD analysis unit 714 provides the analysis result or the information derived or acquired based on the analysis result to the acquisition unit 711 and the output control unit 715, and controls the acquisition of information and the playback of the content. That is, the acquisition unit 711 (3D acquisition unit 722 and IM acquisition unit 723) and the output control unit 715 (3D output control unit 752 and IM output control unit 753) perform processing under the control of the SD analysis unit 714. Therefore, it can also be said that the SD analysis unit 714 is a control unit (or an acquisition control unit or a playback control unit).

[0419] The output control unit 715 performs processing related to output control of 3D data, IM data, etc. For example, the output control unit 715 is able to perform processing such as rendering by using the 3D data or the IM data. The output control unit 715 includes a 3D output control unit 752 and an IM output control unit 753. The 3D output control unit 752 performs rendering, etc. by using the 3D data provided from the 3D decoding unit 742 to generate information to be output (e.g., an image), and provides the information to the 3D output unit 762 of the output unit 716. The IM output control unit 753 performs rendering, etc. by using the IM data provided from the IM decoding unit 743 to generate information to be output (e.g., vibration information), and provides the information to the IM output unit 763 of the output unit 716.

[0420] It can also be said that the output control unit 715 plays 3D data and IM data. That is, it can also be said that the output control unit 715 is a playback unit that plays such data. It can also be said that the 3D output control unit 752 plays 3D data. That is, it can also be said that the 3D output control unit 752 is a playback unit (or 3D playback unit) that plays 3D data. It can also be said that the IM output control unit 753 plays IM data. That is, it can also be said that the IM output control unit 753 is a playback unit (or IM playback unit) that plays IM data.

[0421] It should be noted that, in the case where there are behaviors related to preliminary processing in the scene description, the IM acquisition unit 723, the IM file processing unit 733, the IM decoding unit 743, and the IM output control unit 753 (which may also include the IM output unit 763) perform preliminary processing related to interactive media according to the description. Therefore, it can also be said that these processing units are preliminary processing units.

[0422] The output unit 716 includes a display device, an audio output device, a tactile device (e.g., a vibration device), etc., and performs processing related to the output of information (image display, audio output, tactile media output (e.g., vibration output), etc.). The output unit 716 includes a 3D output unit 762 and an IM output unit 763. The 3D output unit 762 includes, for example, an image display device such as a display, an audio output device such as a speaker, etc., and outputs information to be output of 3D data provided from the 3D output control unit 752 by using these devices (e.g., displaying an image and outputting audio information). The IM output unit 763 includes, for example, an output device for tactile media such as a vibration device and interactive type media, and outputs information to be output of media data provided from the IM output control unit 753 (e.g., vibration information) by using the output device.

[0423] In the client device 700 having the above configuration, the above first information processing device is adopted, and in <3. Media definition extension for interactive playback>, the above present technology can be applied. For example, any one or more of the above methods 1 to 5 can be applied to the client device 700.

[0424] By doing so, the media can be interactively processed through scene descriptions.

[0425] <Playback Processing Flow>

[0426] Will refer to Fig.76 An example of the flow of playback processing performed by the client device 700 is described with reference to the flowchart in FIG.

[0427] When the playback process has started, the SD acquisition unit 721 acquires the SD file in step S701.

[0428] In step S702, the SD file processing unit 731 extracts the encoded data of the scene description stored in the SD file.

[0429] In step S703, the SD decoding unit 741 decodes the encoded data of the extracted scene description to generate (restore) the scene description.

[0430] In step S704 , the SD analysis unit 714 analyzes the scene description.

[0431] In step S705 , the 3D acquisition unit 722 to the 3D output unit 762 start 3D object content playback processing based on the analyzed scene description.

[0432] In step S706, in the case where interactive media indicated to be interactively played is included in the scene description, the IM acquisition unit 723 to the IM output unit 763 start interactive media playback processing.

[0433] When the processing in step S706 ends, the playback processing ends.

[0434] <Flow of 3D object content playback processing>

[0435] Next, we will refer to Fig.77 The flowchart in Fig.76 An example of the flow of the 3D object content playback process performed in step S705.

[0436] When the 3D object content playback process has started, the 3D acquisition unit 722 acquires the 3D file in step S721. In step S722, the 3D file processing unit 732 extracts the encoded data of the 3D object content (3D data) from the 3D file. In step S723, the 3D decoding unit 742 decodes the encoded data of the 3D object content (3D data). In step S724, the 3D output control unit 752 reconfigures the 3D object content (3D data), renders it, and generates a display image. In step S725, the 3D output unit 762 displays the display image.

[0437] In step S726, the control unit 701 determines whether to terminate the 3D object content playback process. In the case where it is determined not to terminate it, the process returns to step S721, and such a process and the process after such a process are repeated. In the case where it is determined to terminate the 3D object content playback process, the 3D object content playback process ends, and the process returns to step S721. Fig.76 .

[0438] <Interactive Media Playback Processing Flow 1>

[0439] Next, we will describe Fig.76 The interactive media playback process performed in step S706 of FIG. 1 is described above. As described above, the method 1 can be applied to the client device 700. Fig.78 An example of the flow of the interactive media playback processing in this case is described with reference to the flowchart in FIG.

[0440] When the interactive media playback process has started, in step S741, the IM acquisition unit 723 determines whether the execution condition (trigger) of the interactive playback has been met based on the analysis result of the scene description by the SD analysis unit 714. In the case where the execution condition of the interactive playback described in the scene description has been met, the process proceeds to step S742.

[0441] In step S742, the IM acquisition unit 723 acquires an IM file from outside the client device 700 according to the description of the processing content (action) and the like of the interactive play.

[0442] In step S743, the IM file processing unit 733 extracts the encoded data of the interactive media (IM data) from the IM file based on the description of the processing content (action) and the like of the interactive play.

[0443] In step S744, the IM decoding unit 743 decodes the encoded data according to the description of the processing content (action) of the interactive play and the like to generate (restore) the interactive media (IM data).

[0444] In step S745 , the IM output control unit 753 sets the interactive device of the IM output unit 763 .

[0445] In step S746, based on the description of the processing content (action) of the interactive playback, etc., the IM output control unit 753 plays the generated interactive media (IM data), generates output information, and causes the output information to be output from the interactive device of the IM output unit 763.

[0446] When the processing in step S746 ends, the processing proceeds to step S747. In addition, in the case where it is determined in step S741 that the execution condition of interactive play (trigger) is not satisfied, the processing in steps S741 to S746 is skipped (omitted) and the processing proceeds to step S747.

[0447] In step S747, the control unit 701 determines whether to terminate the interactive media playback process. In the case where it is determined that it is not to be terminated, the process returns to step S741, and such a process and the process after such a process are repeated. In addition, in the case where it is determined in step S747 that the interactive media playback process is terminated, the interactive media playback process ends, and the process returns to step S741. Fig.76 .

[0448] By performing the corresponding processing in this way, the client device 700 can apply the method 1 of the present technology. Therefore, the media can be interactively processed through the scene description.

[0449] <Interactive Media Playback Processing Flow 2>

[0450] In addition, the above-mentioned method 1 and method 2 can be applied to the client device 700. Fig.79 An example of the flow of the interactive media playback processing in this case is described with reference to the flowchart in FIG.

[0451] When the interactive media playback process has started, in step S761, the IM output control unit 753 determines whether the set execution condition (trigger) of the device has been satisfied. In the case where it is determined that it has been satisfied, the process proceeds to step S762.

[0452] In step S762, the IM output control unit 753 sets the interactive device of the IM output unit 763 according to the description of the processing content (action) and the like of the setting of the device.

[0453] When the setting of the device is completed, the process proceeds to step S763. In addition, in the event that it is determined in step S761 that the execution condition (trigger) of the setting of the device is not satisfied, the process in step S762 is skipped (omitted) and the process proceeds to step S763.

[0454] With Fig.78 The corresponding processing in steps S763 to S767 is performed in a similar manner to the corresponding processing in steps S741 to S744 and step S746. When the processing in step S767 ends, the processing proceeds to step S768. In addition, in the case where it is determined in step S763 that the execution condition of the interactive play (trigger) is not satisfied, the processing in steps S764 to S767 is skipped (omitted), and the processing proceeds to step S768.

[0455] In step S768, the control unit 701 determines whether to terminate the interactive media playback process. In the case where it is determined that it is not to be terminated, the process returns to step S761, and such a process and the process after such a process are repeated. In addition, in the case where it is determined in step S768 that the interactive media playback process is terminated, the interactive media playback process ends, and the process returns to step S761. Fig.76 .

[0456] By performing the corresponding processing in this way, the client device 700 can apply the method 1 and the method 2 of the present technology. Therefore, the media can be interactively processed through the scene description.

[0457] <Interactive Media Playback Process 3>

[0458] In addition, the above-mentioned method 1 and method 3 can be applied to the client device 700. Fig.80 An example of the flow of the interactive media playback processing in this case is described with reference to the flowchart in FIG.

[0459] When the interactive media playback process starts, in step S781, the IM acquisition unit 723, the IM file processing unit 733, the IM decoding unit 743 and the IM output control unit 753 (which may include the IM output unit 763) determine whether the execution condition (trigger) of the preliminary process has been met. In the case where it is determined that it has been met, the process proceeds to step S782.

[0460] In step S782, as preliminary processing, the IM acquisition unit 723 acquires an IM file according to the description of the processing contents (actions) of the preliminary processing, etc. It should be noted that when the processing contents of the preliminary processing do not include acquisition of an IM file, the processing is skipped (omitted).

[0461] In step S783, as a preliminary process, the IM file processing unit 733 extracts the encoded data of the interactive media (IM data) from the IM file according to the description of the processing contents (actions) of the preliminary process, etc. It should be noted that, in the case where the processing contents of the preliminary process do not include extraction of the encoded data of the interactive media (IM data), this process is skipped (omitted).

[0462] In step S784, as a preliminary process, the IM decoding unit 743 decodes the encoded data to generate (restore) the interactive media (IM data) according to the description of the processing content (action) etc. of the preliminary process. It should be noted that in the case where the processing content of the preliminary process does not include decoding of the encoded data of the interactive media (IM data), the process is skipped (omitted).

[0463] In step S785, the IM acquisition unit 723, the IM file processing unit 733, the IM decoding unit 743, and the IM output control unit 753 (which may include the IM output unit 763) perform other processing as preliminary processing according to the description of the processing content (action) of the preliminary processing, etc. For example, in the case where the preliminary processing includes setting, releasing, etc. of IM data, such processing is also performed. Of course, other processing can be performed. It should be noted that in the case where the processing content of the preliminary processing does not include other processing, the processing is skipped (omitted).

[0464] When the processing in step S785 ends, the processing proceeds to step S786. In addition, in the case where it is determined in step S781 that the execution condition (trigger) of the preliminary processing is not satisfied, the processing of steps S782 to S785 is skipped (omitted) and the processing proceeds to step S786.

[0465] With Fig.78 The corresponding processing in steps S786 to S788 is performed in a similar manner to the corresponding processing in steps S741, S745, and S746. Then, when the processing in step S788 ends, the processing proceeds to step S789. In addition, in the case where it is determined in step S786 that the execution condition of the interactive play (trigger) is not satisfied, the processing in steps S787 and S788 is skipped (omitted), and the processing proceeds to step S789.

[0466] Then, in step S789, the control unit 701 determines whether to terminate the interactive media playback process. In the case where it is determined that it is not to be terminated, the process returns to step S781, and such process and the process after such process are repeated. In addition, in the case where it is determined in step S789 that the interactive media playback process is terminated, the interactive media playback process ends, and the process returns to step S781. Fig.76 .

[0467] By performing the corresponding processing in this way, the client device 700 can apply the method 1 and the method 3 of the present technology. Therefore, the media can be interactively processed through the scene description.

[0468] <Interactive Media Playback Process 4>

[0469] In addition, the above-mentioned methods 1 to 3 can be applied to the client device 700. Fig.81 An example of the flow of the interactive media playback processing in this case is described with reference to the flowchart in FIG.

[0470] When the interactive media playback process has begun, Fig.79The corresponding processing in step S801 and step S802 is performed in a similar manner to the corresponding processing in step S761 and step S762.

[0471] In addition, Fig.80 The corresponding processing in steps S803 to S807 is performed in a similar manner to the corresponding processing in steps S781 to S785.

[0472] When the processing in step S807 ends, the processing proceeds to step S808. In addition, in the case where it is determined in step S803 that the execution condition of the preliminary processing is not satisfied, the corresponding processing in steps S804 to S807 is skipped (omitted) and the processing proceeds to step S808.

[0473] With Fig.80 The corresponding processing in step S808 and step S809 is performed in a similar manner to the corresponding processing in step S786 and step S788. When the processing in step S788 ends, the processing proceeds to step S810. In addition, in the case where it is determined in step S808 that the execution condition of the interactive play (trigger) is not satisfied, the processing in step S809 is skipped (omitted), and the processing proceeds to step S810.

[0474] Then, in step S810, the control unit 701 determines whether to terminate the interactive media playback process. In the case where it is determined not to terminate it, the process returns to step S801, and such process and the process after such process are repeated. In addition, in the case where it is determined in step S810 that the interactive media playback process is terminated, the interactive media playback process ends, and the process returns to step S802. Fig.76 .

[0475] By performing the corresponding processing in this way, the client device 700 can apply methods 1 to 3 of the present technology. Therefore, media can be interactively processed through scene description.

[0476] <Interactive Media Playback Processing Flow 5>

[0477] In addition, the above-mentioned method 1 and method 5 can be applied to the client device 700. Fig.82 An example of the flow of the interactive media playback processing in this case is described with reference to the flowchart in FIG.

[0478] When the interactive media playback process has started, in step S831, the IM output control unit 753 transfers control of the behavior specified to be controlled by the application to the application based on the description of the scenario description.

[0479] Then, in step S832, the IM acquisition unit 723, the IM file processing unit 733, the IM decoding unit 743, the IM output control unit 753, and the IM output unit 763 perform operations such as referring to Fig.78 The interactive media playback process interactively plays the interactive media based on the scene description without specifying that it is an application-controlled behavior.

[0480] When the processing in step S832 ends, the interactive media playback processing ends, and the processing returns to Fig.76 .

[0481] By performing the corresponding processing in this way, the client device 700 can apply the method 1 and the method 5 of the present technology. Therefore, the media can be interactively processed through the scene description.

[0482] It should be noted that in step S832, for example, the following steps may be performed: Fig.79 In addition, the interactive media playback process described in the embodiment of the present invention can be performed as described in the reference Fig.80 In addition, the interactive media playback process described in the embodiment of the present invention can be performed as described in the reference Fig.81 The interactive media playback process described. That is, method 2, method 3, etc. can be applied.

[0483] <6. Appendix>

[0484] <Combination>

[0485] The corresponding examples (corresponding methods) of the present technology may be applied in combination with other examples (other methods) as appropriate, as long as no contradiction occurs. In addition, the corresponding examples of the present technology may be applied in combination with technologies other than the above-mentioned technologies.

[0486] <Computer>

[0487] The above series of processing can be performed by hardware or by software. If the series of processing is performed by software, the program configuring the software is installed in the computer. Here, the computer includes a computer combined in dedicated hardware, such as a general-purpose personal computer capable of performing various functions by installing various programs, etc.

[0488] Fig.83 : is a block diagram showing a configuration example of hardware of a computer that executes the above-described series of processes according to a program.

[0489] exist Fig.83 In the illustrated computer 900 , a central processing unit (CPU) 901 , a read only memory (ROM) 902 , and a random access memory (RAM) 903 are connected to one another via a bus 904 .

[0490] An input / output interface 910 is also connected to the bus 904. An input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected to the input / output interface 910.

[0491] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 912 includes, for example, a display, a speaker, and an output terminal. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a nonvolatile memory. The communication unit 914 includes, for example, a network interface. The drive 915 drives a removable medium 921 such as a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0492] In the computer configured in this way, the CPU 901 loads a program stored in the storage unit 913, for example, into the RAM 903 via the input / output interface 910 and the bus 904, and executes the program. In this way, the above-mentioned series of processing is performed. In the RAM 903, data and the like required for the CPU 901 to perform various types of processing are also appropriately stored.

[0493] The program executed by the computer may be provided, recorded on the removable medium 921 as a package medium, for example. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by mounting the removable medium 921 on the drive 915 .

[0494] In addition, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, and digital satellite broadcasting. In this case, the program can be received by the communication unit 914 and can be installed in the storage unit 913.

[0495] Otherwise, the program may be installed in the ROM 902 or the storage unit 913 in advance.

[0496] <Targets of application of this technology>

[0497] The present technique can be applied to any encoding and decoding method.

[0498] In addition, the present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0499] In addition, for example, the present technology can also be implemented as some configurations of devices such as a processor used as a system large-scale integration (LSI) (e.g., a video processor), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a collection obtained by still adding other additional functions to the unit (e.g., a video collection).

[0500] In addition, for example, the present technology can also be applied to a network system composed of multiple devices. For example, the present technology can be implemented as cloud computing shared and jointly processed by multiple devices via a network. For example, for any terminal such as a computer, an audio-visual (AV) device, a portable information processing terminal, or an Internet of Things (IoT) device, the present technology can be implemented in a cloud service that provides services related to images (moving images).

[0501] It should be noted that in this specification, a system means a group of multiple components (devices, modules (components), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices that have been stored in separate housings and connected via a network and a single device having a housing in which multiple modules have been stored are both systems.

[0502] <Fields and applications where this technology can be applied>

[0503] Systems, devices, processing units, etc. to which the present technology is applied can be used in any fields, such as transportation, medical care, crime prevention, agriculture, animal husbandry, mining, beauty care, factories, home electronics, weather and nature monitoring. In addition, they are used for any application.

[0504] For example, the present technology can be applied to systems and devices provided for providing content to be viewed, etc. In addition, for example, the present technology can also be applied to systems and devices provided for traffic such as monitoring traffic conditions and self-driving control. In addition, for example, the present technology can also be applied to systems and devices provided for safety. In addition, for example, the present technology can also be applied to systems and devices provided for automatic control on machines, etc. In addition, for example, the present technology can also be applied to systems and devices provided for agriculture and animal husbandry. In addition, for example, the present technology can also be applied to systems and devices that monitor the conditions of the natural world such as volcanoes, forests and oceans, wild animals, etc. In addition, for example, the present technology can also be applied to systems and devices provided for sports.

[0505] <Others>

[0506] It should be noted that in the present specification, "flag" refers to information used to identify multiple states, and includes not only information used when identifying two states of true (1) or false (0), but also information that can identify three or more states. Therefore, the value that this "flag" can take can be a binary value such as 1 / 0, or it can be three or more values. That is, the number of bits that constitute this "flag" is arbitrary, and can be 1 bit, or can be multiple bits. In addition, with respect to identification information (also including flags), it is assumed that not only the identification information is included in the bit stream, but also the difference information of the identification information relative to some reference information is also included in the bit stream. Therefore, in the present specification, with respect to "flag" and "identification information", "flag" and "identification information" include not only the information, but also the difference information relative to the reference information.

[0507] In addition, various types of information (metadata, etc.) about the coded data (bitstream) can be transmitted or recorded in any form, as long as the information is associated with the coded data. Here, the term "associated" means that, for example, a piece of data can be used (can be linked) when processing other data. That is, data associated with each other can be combined into a single data, or can be discrete data fragments. For example, information associated with the coded data (image) can be transmitted on a communication channel different from the communication channel of the coded data (image). In addition, for example, information associated with the coded data (image) can be recorded on a recording medium different from the recording medium of the coded data (image) (or recorded in another recording area of ​​the same recording medium). It should be noted that the "associated" can be not only the entire data, but also a part of the data. For example, an image and information corresponding to the image can be associated with each other in any unit such as multiple frames, a frame, or a part of a frame.

[0508] It should be noted that in this specification, the terms "synthesize", "multiplex", "add", "integrate", "include", "store", "input", "put into", "insert" and the like mean combining multiple objects into one, for example, combining encoded data and metadata into a single data segment, and mean a method of the above-mentioned "association".

[0509] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications may be made without departing from the gist of the present technology.

[0510] For example, a configuration described as a single device (or processing unit) may be divided to constitute it as a plurality of devices (or processing units). Conversely, the configuration described above as a plurality of devices (or processing units) may be combined and configured as a single device (or processing unit). In addition, of course, configurations other than the above configurations may be added to the configuration of each device (or corresponding processing unit). In addition, some configurations of certain devices (or processing units) may be included in the configurations of other devices (or other processing units) as long as the configuration and operation are substantially the same as the entire system.

[0511] In addition, for example, the above-mentioned program can be executed in any device. In this case, it is sufficient to enable the device to have necessary functions (functional blocks, etc.) and to be able to obtain necessary information.

[0512] In addition, for example, a single device may be enabled to execute the corresponding steps of a single flowchart so that multiple devices share and execute them. In addition, in the case where a single step includes multiple processes, the multiple processes may be executed by a single device and may be shared and executed by multiple devices. In other words, multiple processes included in a single step may be executed as processes of multiple steps. Conversely, processes described as multiple steps may be combined and executed as a single step.

[0513] In addition, for example, in the case of a program executed by a computer, the processing of the steps describing the program may be performed in chronological order according to the sequence described herein, or may be performed in parallel or individually at necessary timings (e.g., when a call is made). That is, the processing in the corresponding steps may be performed in an order different from the above order as long as no contradiction is generated. In addition, the processing of the steps describing the program may be performed in parallel with the processing of another program, or may be performed in combination with the processing of another program.

[0514] In addition, for example, as long as no contradiction occurs, multiple technologies related to the present technology can be implemented separately and independently. Of course, any multiple technologies can also be implemented in combination. For example, some or all of the technologies described in any embodiment can also be implemented in combination with some or all of the technologies described in other embodiments. In addition, part or all of any of the technologies described above can also be implemented simultaneously with other technologies not described above.

[0515] It should be noted that the present technology can also adopt the following configurations.

[0516] (1) An information processing device comprising:

[0517] A playback unit, when an execution condition of interactive playback specified in the scene description is met, the playback unit interactively plays the interactive media specified in the scene description to be interactively played according to the processing content of interactive playback specified in the scene description, wherein:

[0518] The interactive playback is a method of playing media as an interactive process, and

[0519] The interactive processing is an interactive type processing that executes the processing content specified in the scenario description when the execution condition specified in the scenario description is satisfied.

[0520] (2) The information processing device according to (1), wherein:

[0521] In the scenario description,

[0522] As a description about the media to be referred to according to the scene description, there is any one of a start time designation, an automatic play designation, or an interactive play designation, and

[0523] specifying that the interactive media is to be played interactively by using the interactive play designation having a value of true,

[0524] When the execution condition of the interactive play is met, the play unit interactively plays the interactive media according to the processing content of the interactive play,

[0525] The start time specification is a description of specifying the start time of playing the media.

[0526] The autoplay specification is a description that specifies whether to start playing the media once the media is ready, and

[0527] The interactive play specification is a description that specifies whether to play the media interactively.

[0528] (3) The information processing device according to (1), wherein:

[0529] In the scenario description,

[0530] As a description about the media to be referred to according to the scene description, the interactive play designation can exist when the value of the automatic play designation is false, and

[0531] specifying that the interactive media is to be interactively played by using the autoplay designation having a value of false and the interactiveplay designation having a value of true,

[0532] When the execution condition of the interactive play is met, the play unit interactively plays the interactive media according to the processing content of the interactive play,

[0533] The autoplay specification is a description that specifies whether to start playing the media once the media is ready, and

[0534] The interactive play specification is a description that specifies whether to play the media interactively.

[0535] (4) The information processing device according to (1), wherein:

[0536] In the scenario description,

[0537] As a description about the media to be referenced according to the scene description, other method designation can exist when the value of the autoplay designation is false, and

[0538] The interactive media is to be interactively played by specifying that the interactive media is to be interactively played using an autoplay designation with a value of false and an interactiveplay designation with a value of true,

[0539] When the execution condition of the interactive play is met, the play unit interactively plays the interactive media according to the processing content of the interactive play,

[0540] The automatic play designation is a description that designates whether to start playing the media immediately once the media is ready.

[0541] The other method designation is a description of whether to apply a playback method other than designated time playback and automatic playback.

[0542] The specified time playback is a method of starting to play the media at a specified time.

[0543] The autoplay is a play method that starts playing the media once the media is ready, and

[0544] The interactive play designation is one of the other method designations, and is a description that designates whether to interactively play the media.

[0545] (5) The information processing device according to (1), wherein:

[0546] In the scenario description,

[0547] As a description of the media to be referred to according to the scene description, in the case where the value of the automatic play designation is false, it is considered that the media has been designated to be played as the interactive process,

[0548] By specifying that the interactive media is to be played interactively using an autoplay designation with a value of false,

[0549] When the execution condition of the interactive play is met, the play unit interactively plays the interactive media according to the processing content of the interactive play, and

[0550] The automatic play designation is a description that designates whether to start playing the media immediately once the media is ready.

[0551] (6) The information processing device according to (1), wherein:

[0552] In the scenario description,

[0553] As a description of the media to be referenced according to the scene description, the interactive play designation can exist in the case where a start time is designated as a negative value, and

[0554] specifying that the interactive media is to be played interactively by using the start time designation being a negative value and the interactive play designation being true,

[0555] When the execution condition of the interactive play is met, the play unit interactively plays the interactive media according to the processing content of the interactive play,

[0556] The start time specification is a description that specifies the start time of playing the media, and

[0557] The interactive play specification is a description that specifies whether the media is to be played interactively.

[0558] (7) The information processing device according to (1), wherein:

[0559] In the scenario description,

[0560] As a description about the media referenced by the scene description, other method designation can exist in the case where the start time is designated as a negative value, and

[0561] By using the start time designation being the negative value and the interactive play designation being true, specifying that the interactive media is to be played interactively,

[0562] When the execution condition of the interactive play is met, the play unit interactively plays the interactive media according to the processing content of the interactive play,

[0563] The start time specification is a description of specifying the start time of playing the media.

[0564] The other method designation is a description that designates whether to apply a playback method other than the specified time playback and the automatic playback.

[0565] The specified time playback is a method of starting to play the media at a specified time.

[0566] The autoplay is a play method that starts playing the media once the media is ready, and

[0567] The interactive play specification is one of the other method specifications, and is a description that specifies whether the media is to be played interactively.

[0568] (8) The information processing device according to (1), wherein:

[0569] In the scenario description,

[0570] As a description of the media referenced by the scene description, in the case where the start time is specified as a negative value, it is considered that the media is specified to be played as the interactive process, and

[0571] specifying that the interactive media is to be played interactively by specifying the start time as a negative value,

[0572] When the execution condition of the interactive play is met, the play unit interactively plays the interactive media according to the processing content of the interactive play, and

[0573] The start time specification is a description that specifies the start time of playing the media.

[0574] (9) The information processing device according to any one of (1) to (8), wherein:

[0575] In the scenario description, the setting of the device that outputs the interactive media has been specified as the type of processing content of the interactive process, and

[0576] When an execution condition of the interactive process is satisfied, the playback unit sets the device according to processing content of the interactive process.

[0577] (10) The information processing device according to (9), wherein:

[0578] In the scenario description, the execution condition for setting the device has been set to be satisfied before the execution condition for the interactive play.

[0579] (11) The information processing device according to (9) or (10), wherein:

[0580] In the scene description, a processing target of the setting is specified by using an index of the device, and

[0581] The playback unit sets a device designated as the processing target.

[0582] (12) The information processing device according to any one of (9) to (11), wherein:

[0583] In the scene description, as the content of the setting, it is specified that the device is capable of outputting the interactive media, and

[0584] The playback unit configures the device so as to play the interactive media according to the specification of the content.

[0585] (13) The information processing device according to any one of (9) to (11), wherein:

[0586] In the scenario description, as the content of the setting, the device to be initialized by the setting is specified, and

[0587] The playback unit initializes the device according to the specified settings of the content.

[0588] (14) The information processing device according to any one of (9) to (11), wherein:

[0589] In the scenario description, as the contents of the setting, the contents of processing and the processing mode to be performed by the device are specified, and

[0590] The playback unit sets the device to update the processing content and processing mode to be executed by the device according to the designation of the content.

[0591] (15) The information processing device according to any one of (1) to (14), wherein:

[0592] In the scenario description, as the type of processing content of the interactive processing, it has been specified that preliminary processing is performed on the interactive media before interactively playing the interactive media, and the information processing apparatus further includes:

[0593] A preliminary processing unit, when an execution condition of the interactive process is satisfied, executes the preliminary process according to the processing content of the interactive process.

[0594] (16) The information processing device according to (15), wherein:

[0595] In the scene description, a processing target of the preliminary processing is specified by using an index of the media, and

[0596] The preliminary processing unit performs the preliminary processing on the interactive media designated as the processing target.

[0597] (17) The information processing device according to (15), wherein:

[0598] In the scene description, as the content of the preliminary processing, the acquisition of the interactive media is specified, and

[0599] The preliminary processing unit obtains the encoded data of the interactive media as the preliminary processing according to the designation of the content.

[0600] (18) The information processing device according to (15), wherein:

[0601] In the scene description, as the content of the preliminary processing, the acquisition and decoding of the interactive media are specified, and

[0602] The preliminary processing unit acquires and decodes the encoded data of the interactive media according to the specification of the content as the preliminary processing.

[0603] (19) The information processing device according to (15), wherein:

[0604] In the scene description, as the content of the preliminary processing, the acquisition, decoding and conversion of the interactive media are specified, and

[0605] The preliminary processing unit acquires and decodes the encoded data of the interactive media as the preliminary processing, and converts the decoded interactive media based on the properties of a device that outputs the interactive media according to the specification of the contents.

[0606] (20) The information processing device according to (15), wherein:

[0607] In the scene description, as the content of the preliminary processing, setting the interactive media is specified, and

[0608] The preliminary processing unit sets the interactive media to a memory of a device that outputs the interactive media according to the designation of the content as the preliminary processing.

[0609] (21) The information processing device according to (15), wherein:

[0610] In the scene description, as the content of the preliminary processing, deletion of the interactive media is specified, and

[0611] The preliminary processing unit deletes the interactive media set to the memory of the device outputting the interactive media as the preliminary processed content according to the designation of the content.

[0612] (22) The information processing device according to any one of (1) to (21), wherein:

[0613] In the scenario description, an action of specifying execution conditions and processing contents for displaying preliminary processing performed on the interactive media before interactively playing the interactive media is described, and the information processing apparatus further includes:

[0614] A preliminary processing unit is configured to perform the preliminary processing according to the behavior.

[0615] (23) The information processing device according to any one of (1) to (21), wherein:

[0616] In the scenario description, the execution conditions and processing contents of the preliminary processing to be performed on the interactive media before the interactive media is interactively played are described, and the information processing device further includes:

[0617] A preliminary processing unit executes the preliminary processing according to processing contents of the preliminary processing when a condition for executing the preliminary processing is satisfied.

[0618] (24) The information processing device according to any one of (1) to (23), wherein:

[0619] In the scenario description, behaviors specified for controlling the interactive process in the application are described, and

[0620] The playback unit enables the application to control the interactive process according to the behavior.

[0621] (25) An information processing method comprising:

[0622] In the case where the execution condition of the interactive play specified in the scene description is met, the interactive media specified in the scene description to be interactively played is interactively played according to the processing content of the interactive play specified in the scene description, wherein:

[0623] The interactive playback is a method of playing media as an interactive process, and

[0624] The interactive processing is an interactive type processing that executes the processing content specified in the scenario description when the execution condition specified in the scenario description is satisfied.

[0625] (31) An information processing device comprising:

[0626] A providing unit, wherein the providing unit provides a scene description, in which there is a description specifying that an interactive media is to be interactively played and a description specifying execution conditions and processing contents of the interactive playback, wherein:

[0627] The interactive playback is a method of playing media as an interactive process, and

[0628] The interactive processing is an interactive type processing that executes the processing content specified in the scenario description when the execution condition specified in the scenario description is satisfied.

[0629] (32) The information processing device according to (31), wherein:

[0630] In the scenario description,

[0631] As a description about the media to be referred to according to the scene description, there is any one of a start time designation, an automatic play designation, or an interactive play designation, and

[0632] The interactive media is to be interactively played, designated by the interactive play designation being true,

[0633] The start time specification is a description of specifying the start time of playing the media.

[0634] The autoplay specification is a description that specifies whether to start playing the media once the media is ready, and

[0635] The interactive play specification is a description that specifies whether the media is to be played interactively.

[0636] (33) The information processing device according to (31), wherein:

[0637] In the scenario description,

[0638] As a description of the media referenced by the scene description, the interactive play designation can exist when the value of the automatic play designation is false, and

[0639] The interactive media is designated to be interactively played by an autoplay designation of false and an interactiveplay designation of true, and

[0640] The automatic play designation is a description that specifies whether to start playing the media immediately after the media is ready, and

[0641] The interactive play specification is a description that specifies whether to play the media interactively.

[0642] (34) The information processing device according to (31), wherein:

[0643] In the scenario description,

[0644] As a description of the media referenced from the scene description, other method designations can exist when the value of the autoplay designation is false, and

[0645] The interactive media is to be interactively played by specifying that the autoplay designation is false and the interactiveplay designation is true,

[0646] The automatic play specification is a description that specifies whether to start playing the media immediately after the media is ready.

[0647] The other method designation is a description of whether to apply a playback method other than designated time playback and automatic playback.

[0648] The specified time playback is a method of starting to play the media at a specified time.

[0649] The automatic play is a play method that starts playing the media immediately after the media is ready, and

[0650] The interactive play designation is one of the other method designations, and is a description that designates whether to interactively play the media.

[0651] (35) The information processing device according to (31), wherein:

[0652] In the scenario description,

[0653] In the event that the value of the automatic play designation for the media description to be referenced according to the scene description is false, the media is deemed to be designated to be interactively played, and

[0654] The interactive media is to be interactively played by specifying an autoplay designation with a value of false, and

[0655] The automatic play designation is a description that designates whether to start playing the media once the media is ready.

[0656] (36) The information processing device according to (31), wherein:

[0657] In the scenario description,

[0658] In the case where a negative value is specified for the start time of a media description to be referenced according to the scene description, it is considered that the media is specified to be played interactively, and

[0659] The interactive media is specified to be played interactively by specifying the start time as a negative value, and

[0660] The start time specification is a description that specifies the start time of playing the media.

[0661] The interactive play specification is a description that specifies whether to play the media interactively.

[0662] (37) The information processing device according to (31), wherein:

[0663] In the scenario description,

[0664] As a description about the media referenced by the scene description, other method designation can exist in the case where the start time is designated as a negative value, and

[0665] By specifying a negative start time and a true interactive play specification, the interactive media is specified to be played interactively.

[0666] The start time specification is a description of specifying the start time of playing the media.

[0667] The other method designation is a description of whether to apply a playback method other than designated time playback and automatic playback.

[0668] The specified time playback is a method of starting to play the media at a specified time.

[0669] The autoplay is a play method that starts playing once the media is ready, and

[0670] The interactive play designation is one of the other method designations, and is a description that designates whether to interactively play the media.

[0671] (38) The information processing device according to (31), wherein:

[0672] In the scenario description,

[0673] In the case where a negative value is specified for the start time of a media description to be referenced according to the scene description, it is considered that the media is specified to be played interactively, and

[0674] The interactive media is specified to be played interactively by specifying the start time as a negative value, and

[0675] The start time specification is a description that specifies the start time of playing the media.

[0676] (39) The information processing device according to any one of (31) to (38), wherein:

[0677] In the scenario description, as the type of processing contents of the interactive process regarding the interactive media, the setting of a device that outputs the interactive media is specified.

[0678] (40) The information processing device according to (39), wherein:

[0679] In the scenario description, the execution condition regarding the setting of the device has been set to be satisfied before the execution condition regarding the interactive play of the media.

[0680] (41) The information processing device according to (39) or (40), wherein:

[0681] In the scene description, a processing target of the setting is specified by using an index of the device.

[0682] (42) The information processing device according to any one of (39) to (41), wherein:

[0683] In the scene description, as the content of the setting, it is specified that the device is capable of outputting the interactive media.

[0684] (43) The information processing device according to any one of (39) to (41), wherein

[0685] In the scene description, as the content of the setting, the device initialized by the setting is specified.

[0686] (44) The information processing device according to any one of (39) to (41), wherein:

[0687] In the scenario description, as the contents of the settings, the contents of processing and the processing mode to be executed by the device are specified.

[0688] (45) The information processing device according to any one of (1) to (44), wherein:

[0689] In the scene description, as the type of processing contents regarding the interactive processing of the interactive media, it is specified that preliminary processing is performed on the interactive media before the interactive media is interactively played.

[0690] (46) The information processing device according to (45), wherein:

[0691] In the scene description, a processing target of the preliminary processing is specified by using an index of the media.

[0692] (47) The information processing device according to (45), wherein:

[0693] In the scene description, as the content of the preliminary processing, the acquisition of the interactive media is specified.

[0694] (48) The information processing device according to (45), wherein:

[0695] In the scene description, as the content of preliminary processing, the acquisition and decoding of interactive media are specified.

[0696] (49) The information processing device according to (45), wherein

[0697] In the scene description, as the content of the preliminary processing, the acquisition, decoding and conversion of the interactive media are specified.

[0698] (50) The information processing device according to (45), wherein:

[0699] In the scenario description, as the content of the preliminary processing, setting the interactive media to a memory of a device that outputs the interactive media is specified.

[0700] (51) The information processing device according to (45), wherein:

[0701] In the scene description, as the content of the preliminary processing, deletion of the interactive media is specified.

[0702] (52) The information processing device according to any one of (31) to (51), wherein:

[0703] In the scenario description, an action for specifying execution conditions and processing contents for displaying preliminary processing performed on the interactive media before interactively playing the interactive media is described.

[0704] (53) The information processing device according to any one of (31) to (51), wherein:

[0705] In the scenario description, execution conditions of preliminary processing to be performed on the interactive media before the interactive media is interactively played and processing contents of the preliminary processing are described.

[0706] (54) The information processing device according to any one of (31) to (53), wherein:

[0707] In the scenario description, behaviors specified for controlling the interactive process in the application are described.

[0708] (55) An information processing method comprising:

[0709] A scenario description is provided, in which there is a description specifying that the interactive media is to be interactively played and a description specifying execution conditions and processing contents of the interactive playing, wherein:

[0710] The interactive playback is a method of playing media as an interactive process, and

[0711] The interactive processing is an interactive type processing that executes the processing content specified in the scenario description when the execution condition specified in the scenario description is satisfied.

[0712] [reference numerals list]

[0713] 300 File generation device

[0714] 301 Control Unit

[0715] 302 File Generation Processing Unit

[0716] 311 Input Unit

[0717] 312 Preprocessing Unit

[0718] 313 coding units

[0719] 314 File Generation Unit

[0720] 315 Storage Unit

[0721] 316 Output Unit

[0722] 321 SD Input Unit

[0723] 322 3D input unit

[0724] 323 IM input unit

[0725] 331 SD pre-processing unit

[0726] 332 3D preprocessing unit

[0727] 333 IM Preprocessing Unit

[0728] 341 SD Encoding Unit

[0729] 342 3D encoding unit

[0730] 343 IM coding unit

[0731] 351 SD file generation unit

[0732] 352 3D file generation unit

[0733] 353 IM file generation unit

[0734] 361 SD storage unit

[0735] 362 3D storage unit

[0736] 363 IM storage unit

[0737] 371 SD output unit

[0738] 372 3D output unit

[0739] 373 IM output unit

[0740] 700 Client Devices

[0741] 701 Control Unit

[0742] 702 Client Processing Unit

[0743] 711 Get Unit

[0744] 712 File Processing Unit

[0745] 713 Decoding Unit

[0746] 714 SD Analysis Unit

[0747] 715 Output Control Unit

[0748] 716 Output Unit

[0749] 721 SD acquisition unit

[0750] 722 3D acquisition unit

[0751] 723 IM acquisition unit

[0752] 731 SD file processing unit

[0753] 732 3D file processing unit

[0754] 733 IM file processing unit

[0755] 741 SD Decoder Unit

[0756] 742 3D decoding unit

[0757] 743 IM Decoding Unit

[0758] 752 3D Output Control Unit

[0759] 753 IM Output Control Unit

[0760] 762 3D output unit

[0761] 763 IM Output Unit

[0762] 900 Computer

Claims

1. An information processing device, comprising: A playback unit, when an execution condition of interactive playback specified in the scene description is met, the playback unit interactively plays the interactive media specified in the scene description to be interactively played according to the processing content of interactive playback specified in the scene description, wherein: The interactive playback is a method of playing media as an interactive process, and The interactive processing is an interactive type processing that executes the processing content specified in the scenario description when the execution condition specified in the scenario description is satisfied.

2. The information processing device according to claim 1, wherein: In the scenario description, As a description about the media to be referred to according to the scene description, there is any one of a start time designation, an automatic play designation, or an interactive play designation, and specifying that the interactive media is to be played interactively by using the interactive play designation having a value of true, When the execution condition of the interactive play is met, the play unit interactively plays the interactive media according to the processing content of the interactive play, The start time specification is a description of specifying the start time of playing the media. The autoplay specification is a description that specifies whether to start playing the media once the media is ready, and The interactive play specification is a description that specifies whether to play the media interactively.

3. The information processing device according to claim 1, wherein: In the scenario description, As a description about the media to be referred to according to the scene description, the interactive play designation can exist when the value of the automatic play designation is false, and specifying that the interactive media is to be interactively played by using the autoplay designation having a value of false and the interactiveplay designation having a value of true, When the execution condition of the interactive play is met, the play unit interactively plays the interactive media according to the processing content of the interactive play, The autoplay specification is a description that specifies whether to start playing the media once the media is ready, and The interactive play specification is a description that specifies whether to play the media interactively.

4. The information processing device according to claim 1, wherein: In the scenario description, As a description about the media to be referenced according to the scene description, other method designation can exist when the value of the autoplay designation is false, and The interactive media is to be interactively played by specifying that the interactive media is to be interactively played using an autoplay designation with a value of false and an interactiveplay designation with a value of true, When the execution condition of the interactive play is met, the play unit interactively plays the interactive media according to the processing content of the interactive play, The automatic play designation is a description that designates whether to start playing the media once the media is ready. The other method designation is a description of whether to apply a playback method other than designated time playback and automatic playback. The designated time play is a play method that starts playing the media at a designated time, and the automatic play is a play method that starts playing the media once the media is ready, and The interactive play designation is one of the other method designations, and is a description that designates whether to interactively play the media.

5. The information processing device according to claim 1, wherein: In the scenario description, As a description of the media to be referred to according to the scene description, in the case where the value of the automatic play designation is false, it is considered that the media has been designated to be played as the interactive process, By specifying that the interactive media is to be played interactively using an autoplay designation with a value of false, When the execution condition of the interactive play is met, the play unit interactively plays the interactive media according to the processing content of the interactive play, and The automatic play designation is a description that designates whether to start playing the media once the media is ready.

6. The information processing device according to claim 1, wherein: In the scenario description, As a description of the media to be referenced according to the scene description, the interactive play designation can exist in the case where a start time is designated as a negative value, and specifying that the interactive media is to be played interactively by using the start time designation being a negative value and the interactive play designation being true, When the execution condition of the interactive play is met, the play unit interactively plays the interactive media according to the processing content of the interactive play, The start time specification is a description that specifies the start time of playing the media, and The interactive play specification is a description that specifies whether the media is to be played interactively.

7. The information processing device according to claim 1, wherein: In the scenario description, As a description about the media to be referenced according to the scene description, other method designation can exist in the case where the start time is designated as a negative value, and By using the start time designation being the negative value and the interactive play designation being true, specifying that the interactive media is to be played interactively, When the execution condition of the interactive play is met, the play unit interactively plays the interactive media according to the processing content of the interactive play, The start time specification is a description of specifying the start time of playing the media. The other method designation is a description that designates whether to apply a playback method other than the specified time playback and the automatic playback. The designated time play is a play method that starts playing the media at a designated time, and the automatic play is a play method that starts playing the media once the media is ready, and The interactive play specification is one of the other method specifications, and is a description that specifies whether the media is to be played interactively.

8. The information processing device according to claim 1, wherein: In the scenario description, As a description of the media to be referenced according to the scene description, in the case where the start time is specified as a negative value, it is considered that the media is specified to be played as the interactive process, and specifying that the interactive media is to be played interactively by specifying the start time as a negative value, When the execution condition of the interactive play is met, the play unit interactively plays the interactive media according to the processing content of the interactive play, and The start time specification is a description that specifies the start time of playing the media.

9. The information processing device according to claim 1, wherein: In the scenario description, the setting of the device that outputs the interactive media has been specified as the type of processing content of the interactive process, and When the execution condition of the interactive process is satisfied, the playback unit sets the device according to the processing content of the interactive process.

10. The information processing device according to claim 9, wherein: In the scenario description, the execution condition for setting the device has been set to be satisfied before the execution condition for the interactive play.

11. The information processing device according to claim 1, wherein: In the scenario description, as the type of processing content of the interactive processing, it has been specified that preliminary processing is performed on the interactive media before interactively playing the interactive media, and the information processing apparatus further includes: A preliminary processing unit, when an execution condition of the interactive process is satisfied, executes the preliminary process according to the processing content of the interactive process.

12. The information processing device according to claim 1, wherein: In the scenario description, an action of specifying execution conditions and processing contents for displaying preliminary processing performed on the interactive media before interactively playing the interactive media is described, and the information processing apparatus further includes: A preliminary processing unit is configured to perform the preliminary processing according to the behavior.

13. The information processing device according to claim 1, wherein: In the scenario description, behaviors specified for controlling the interactive process in the application are described, and The playback unit enables the application to control the interactive process according to the behavior.

14. An information processing method, comprising: In the case where the execution condition of the interactive play specified in the scene description is met, the interactive media specified in the scene description to be interactively played is interactively played according to the processing content of the interactive play specified in the scene description, wherein: The interactive playback is a method of playing media as an interactive process, and The interactive processing is an interactive type processing that executes the processing content specified in the scenario description when the execution condition specified in the scenario description is satisfied.

15. An information processing device, comprising: A providing unit, wherein the providing unit provides a scene description, in which there is a description specifying that an interactive media is to be interactively played and a description specifying execution conditions and processing contents of the interactive playback, wherein: The interactive playback is a method of playing media as an interactive process, and The interactive processing is an interactive type processing that executes the processing content specified in the scenario description when the execution condition specified in the scenario description is satisfied.

16. The information processing device according to claim 15, wherein: In the scenario description, As a description about the media to be referred to according to the scene description, there is any one of a start time designation, an automatic play designation, or an interactive play designation, and The interactive media is to be interactively played, designated by the interactive play designation being true, The start time specification is a description of specifying the start time of playing the media. The autoplay specification is a description that specifies whether to start playing the media once the media is ready, and The interactive play specification is a description that specifies whether the media is to be played interactively.

17. The information processing device according to claim 15, wherein: In the scenario description, In the event that the value of the automatic play designation for the media description to be referenced according to the scene description is false, the media is deemed to be designated to be interactively played, and The interactive media is to be interactively played by specifying an autoplay designation with a value of false, and The automatic play designation is a description that designates whether to start playing the media once the media is ready.

18. The information processing device according to claim 15, wherein: In the scenario description, In the case where a negative value is specified for the start time of a media description to be referenced according to the scene description, it is considered that the media is specified to be played interactively, and specifying that the interactive media is to be played interactively by using the start time specification as the negative value, and The start time specification is a description that specifies the start time of playing the media.

19. The information processing device according to claim 15, wherein: In the scenario description, behaviors specified for controlling the interactive process in the application are described.

20. An information processing method, comprising: A scenario description is provided, in which there is a description specifying that the interactive media is to be interactively played and a description specifying execution conditions and processing contents of the interactive playing, wherein: The interactive playback is a method of playing media as an interactive process, and The interactive processing is an interactive type processing that executes the processing content specified in the scenario description when the execution condition specified in the scenario description is satisfied.

Images