Information processing device and method

Abstract

The present disclosure relates to an information processing device and method with which it is possible to suppress the occurrence of a sub-media output delay due to the delay of start-up of a device. At a timing with which is taken into account the startup time of a sub-media output device that outputs a sub-media to be reproduced in conjunction with reproduction of a main medium that is the independent medium of content, device startup directive information for directing startup of the sub-media output device is supplied to the sub-media output device or a control device for controlling the sub-media output device. The sub-media output device is started up on the basis of the device startup directive information supplied at a timing with which the startup time of the sub-media output device is taken into account. The present disclosure can be applied to, for example, information processing devices, electronic apparatuses, and information processing methods and programs.

Description

Information processing device and method

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that can suppress the occurrence of a delay in sub-media output due to a delay in device startup.

[0002] Conventionally, there is the glTF (The GL Transmission Format) (registered trademark) 2.0, which is a scene description format for arranging and rendering 3D (three-dimensional) objects in a three-dimensional space (see, for example, Non-Patent Document 1). Also, in the MPEG-I scene description (MPEG (Moving Picture Experts Group)-I Scene Description), a method has been proposed in which glTF 2.0 is extended to handle dynamic content in the time direction (see, for example, Non-Patent Document 2).

[0003] In recent years, in content that handles such scenes, it has become possible to play sub-media in conjunction with the main media, which is the main media that expresses the scene.

[0004] For example, in 3D scene content, it is expected that content will be introduced to the market in the future that uses viewpoint images that represent the 3D space as the main media, and haptic media that presents haptics in 3D space synchronized with the 3D scene, or haptic media that presents haptics through interaction as the sub-media.MPEG standardization is progressing with the standardization of haptic compression methods, haptic rendering in 3D space, and 3D scene technologies that realize haptic rendering through interaction (see, for example, Non-Patent Documents 3 and 4).

[0005] Rendering a scene in content requires a high processing load, so one approach is to render it on the server, encode the 2D video from the user's viewpoint, and send it to the device. In this case, although the processing load for haptics itself is lower than that of video, 3D scene processing is required for haptic rendering in 3D space and haptic rendering through interaction. Therefore, it is reasonable to process haptics on the server as well. For example, to present haptics by touching a haptic device through interaction within a scene, the server can render the haptics on the server side, just like the video, and then transmit the converted signal data to the device. The device then synchronizes the presentation of the video and haptics.

[0006] In other words, when main media and sub-media are played back in conjunction with each other, the terminal device acquires the sub-media supplied from the server at an appropriate timing, starts up a device based on the sub-media, and after the startup is complete, causes the device to output the sub-media at an appropriate timing that matches the output of the main media.

[0007] Saurabh Bhatia, Patrick Cozzi, Alexey Knyazev, Tony Parisi, "Khronos glTF2.0", https: / / github.com / KhronosGroup / glTF / tree / master / specification / 2.0, June 9, 2017"Information technology - Coded representation of immersive media - Part 14: Scene Description", ISO / IEC DIS 23090-14:2021(E), ISO / IEC JTC 1 / SC 29 / WG 03 N00485, 137th MPEG meeting, January 2022, online"Revised text of ISO / IEC 23090-14 DAM 2 Support for haptics, augmented reality, avatars, interactivity and lighting", WG3N01025, 144th MPEG meeting, October 2023, Hannover"Draft Text of FDIS ISO / IEC 23090-31 MPEG Haptics Coding Phase 1", WG7N00765, 144th MPEG meeting, October 2023, Hannover

[0008] However, some devices, such as those that output haptic media, require a relatively long time to start up. Therefore, when using such devices that require a relatively long time to start up, the above-described method may result in delays in device startup relative to the appropriate timing for outputting the secondary media. This may result in delays in the output of the secondary media relative to the timing of interaction events or synchronization, potentially reducing the quality of the content and the user experience.

[0009] The present disclosure has been made in consideration of such circumstances, and makes it possible to suppress the occurrence of a delay in sub-media output due to a delay in device startup.

[0010] An information processing device according to one aspect of the present technology is an information processing device that includes a device startup control unit that controls the startup of a sub-media output device by supplying device startup instruction information that instructs the startup of a sub-media output device to the sub-media output device or a control device that controls the sub-media output device, at a timing that takes into account the startup time of the sub-media output device that outputs sub-media that is played in conjunction with the playback of main media, which is the main media of the content.

[0011] An information processing method according to one aspect of the present technology is an information processing method that includes controlling the startup of a sub-media output device, which outputs a sub-media that is played in conjunction with the playback of a main media, which is the subject media of the content, by supplying device startup instruction information that instructs the startup of the sub-media output device to the sub-media output device or a control device that controls the sub-media output device, at a timing that takes into account the startup time of the sub-media output device.

[0012] Another aspect of the information processing device of the present technology is an information processing device that includes a sub-media output device control unit that starts up a sub-media output device based on device start-up instruction information that is supplied at a timing that takes into account the start-up time of the sub-media output device that outputs sub-media that is played in conjunction with the playback of the main media, which is the main media of the content.

[0013] Another aspect of the information processing method of the present technology is an information processing method that includes starting up a sub-media output device based on device start-up instruction information that is supplied at a timing that takes into account the start-up time of the sub-media output device, which outputs sub-media that is played in conjunction with the playback of main media, which is the main media of the content.

[0014] In an information processing device and method according to one aspect of the present technology, the startup of a sub-media output device that outputs a sub-media that is played in conjunction with the playback of a main media, which is the subject media of the content, is controlled by supplying device startup instruction information that instructs the startup of the sub-media output device to the sub-media output device or a control device that controls the sub-media output device at a timing that takes into account the startup time of the sub-media output device.

[0015] In another aspect of the information processing device and method of the present technology, the sub-media output device is started up based on device start-up instruction information supplied at a timing that takes into account the start-up time of the sub-media output device that outputs sub-media that is played in conjunction with the playback of the main media, which is the main media of the content.

[0016] 1 is a block diagram showing an example of the main configuration of a content playback system; FIG. 2 is a block diagram showing an example of the main configuration of hardware of a content providing server and a terminal device; FIG. 3 is a functional block diagram showing an example of functions of the content providing server; FIG. 4 is a functional block diagram showing an example of functions of the terminal device; FIG. 5 is a flowchart explaining an example of the flow of server processing; FIG. 6 is a flowchart explaining an example of the flow of terminal processing; FIG. 7 is a functional block diagram showing an example of functions of the content providing server; FIG. 8 is a functional block diagram showing an example of functions of the terminal device; FIG. 9 is a flowchart explaining an example of the flow of server processing; FIG. 10 is a functional block diagram showing an example of functions of the content providing server; FIG. 11 is a flowchart explaining an example of the flow of server processing; FIG. 12 is a functional block diagram showing an example of functions of the content providing server; FIG. 13 is a flowchart explaining an example of the flow of server processing; FIG. 14 is a diagram showing an example of the probability of an interaction event occurring; FIG. 15 is a functional block diagram showing an example of functions of the content providing server; FIG. 16 is a flowchart explaining an example of the flow of server processing; FIG. 17 is a functional block diagram showing an example of functions of the content providing server; 1 is a flowchart illustrating an example of the flow of terminal processing; FIG. 2 is a functional block diagram illustrating an example of functions possessed by a content providing server; FIG. 3 is a functional block diagram illustrating an example of functions possessed by a terminal device; FIG. 4 is a flowchart illustrating an example of the flow of server processing; FIG. 5 is a functional block diagram illustrating an example of functions possessed by a content providing server; FIG. 6 is a flowchart illustrating an example of the flow of server processing; FIG. 7 is a functional block diagram illustrating an example of functions possessed by a content providing server; FIG. 8 is a flowchart illustrating an example of the flow of server processing; FIG. 9 is a functional block diagram illustrating an example of functions possessed by a content providing server;

[0017] Below, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. Literature, etc. supporting technical content and technical terminology 2. Playback of sub-media 3. First embodiment (interactive playback of sub-media in 3D scenes) 4. Second embodiment (prediction of interaction events in 3D scenes) 5. Third embodiment (synchronized playback of sub-media in 3D scenes) 6. Fourth embodiment (interactive playback of sub-media in 2D scenes) 7. Fifth embodiment (prediction of interaction events in 2D scenes) 8. Sixth embodiment (synchronized playback of sub-media in 2D scenes) 9. Supplementary notes

[0018] <1. Literature, etc. supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the content described in the embodiments, but also the content described in the following non-patent documents, etc. that were publicly known at the time of filing, and the content of other documents referenced in the following non-patent documents.

[0019] Non-patent document 1: (described above) Non-patent document 2: (described above) Non-patent document 3: (described above) Non-patent document 4: (described above)

[0020] In other words, the contents of the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements.

[0021] <2. Sub-media playback> <Contents that handle scenes>

[0022] Conventionally, there is glTF (The GL Transmission Format) (registered trademark) 2.0, which is a scene description format for arranging and rendering 3D (three-dimensional) objects in a three-dimensional space, as described in Non-Patent Document 1. Furthermore, as described in Non-Patent Document 2, a method has been proposed for handling dynamic content in the time direction by extending glTF 2.0 in MPEG-I scene description (MPEG (Moving Picture Experts Group)-I Scene Description).

[0023] In recent years, in content that handles such scenes, it has become possible to play sub-media in conjunction with the main media, which is the main media that expresses the scene.

[0024] For example, in the future, it is expected that 3D scene content will be introduced to the market in which viewpoint images that express the 3D space are used as the main media, and haptic media that presents haptics in 3D space synchronized with the 3D scene, or haptic media that presents haptics through interaction, are used as sub-media. For example, as described in Non-Patent Documents 3 and 4, MPEG standardization is progressing on standardization of haptic compression methods, haptic rendering in 3D space, and 3D scene technology that realizes haptic rendering through interaction.

[0025] Rendering a scene in content requires a high processing load, so one approach is to render it on the server, encode the 2D video from the user's viewpoint, and send it to the device. In this case, although the processing load for haptics itself is lower than that of video, 3D scene processing is required for haptic rendering in 3D space and haptic rendering through interaction. Therefore, it is reasonable to process haptics on the server as well. For example, to present haptics by touching a haptic device through interaction within a scene, the server can render the haptics on the server side, just like the video, and then transmit the converted signal data to the device. The device then synchronizes the presentation of the video and haptics.

[0026] In other words, when main media and sub-media are played back in conjunction with each other, the terminal device acquires the sub-media supplied from the server at an appropriate timing, starts up a device based on the sub-media, and after the startup is complete, causes the device to output the sub-media at an appropriate timing that matches the output of the main media.

[0027] However, some devices, such as those that output haptic media, require a relatively long time to start up. Therefore, when using such devices that require a relatively long time to start up, the above-described method may result in delays in device startup relative to the appropriate timing for outputting the secondary media. This may result in delays in the output of the secondary media relative to the timing of interaction events or synchronization, potentially reducing the quality of the content and the user experience.

[0028] Therefore, in order to prevent a delay in the output of the sub-media due to a delay in starting up the device, an instruction to start up the device that outputs the sub-media is supplied at a sufficiently early timing.

[0029] <Content Playback System> Fig. 1 is a block diagram showing an example of the configuration of a content playback system, which is one aspect of an information processing system to which the present technology is applied. The content playback system 100 shown in Fig. 1 is a system that plays main media and sub-media, which are subjective media that express scenes in content that handles scenes. In this content playback system 100, the sub-media are media that are played in conjunction with the playback of the main media.

[0030] 1, the content playback system 100 includes a content providing server 111, a terminal device 112, an input device 121, a main media output device 122, and a sub-media output device 123. The content providing server 111 and the terminal device 112 are communicably connected to each other via a network 110. The content providing server 111 executes a service that applies content to clients. The content providing server 111 may have any configuration, and may be composed of multiple servers, or may be composed of a virtual server whose physical configuration is not specified, such as a so-called cloud server.

[0031] The terminal device 112 functions as a client of the content providing server 111 and receives services, i.e., content, provided by the content providing server 111. The terminal device 112 is also connected to an input device 121, a main media output device 122, and a sub-media output device 123, and controls them to receive inputs such as instructions from a user or the like and to output the main media and sub-media of the provided content. The terminal device 112 may be a dedicated device such as a game console, or a general-purpose device such as a personal computer. The terminal device 112 may also be composed of multiple devices or may be composed of software.

[0032] The network 110 is a communication network that serves as a communication medium between devices. The network 110 may be a wired communication network, a wireless communication network, or both. For example, the network 110 may be a wired local area network (LAN), a wireless LAN, a public telephone line network, a wide area communication network for wireless mobile devices such as a 4G line or a 5G line, the Internet, or a combination thereof. The network 110 may be a single communication network or multiple communication networks. For example, the network 110 may be partially or entirely composed of communication cables of a predetermined standard, such as a USB (Universal Serial Bus) (registered trademark) cable or an HDMI (High-Definition Multimedia Interface) (registered trademark) cable.

[0033] The input device 121 is a device that accepts information input from the outside using some kind of sensor, such as an acceleration sensor, a force sensor, a vibration sensor, a heat sensor, an optical sensor, an electrical sensor, a magnetic sensor, or a chemical sensor. The input device 121 may be capable of accepting any type of information. Furthermore, the input device 121 may have multiple types of sensors and be capable of accepting multiple types of information. Furthermore, the input device 121 may be configured in any manner. For example, the input device 121 may be configured as an independent device such as a touch panel, a keyboard, a switch, a button, a controller, a mouse, a microphone, a camera, or the like, or as a sensor module incorporated into some device such as an output device. Furthermore, the input device 121 may not only be configured physically, but also as software such as a GUI (Graphical User Interface). Using such functions, the input device 121 accepts, for example, user instructions and supplies the accepted information to the terminal device 112.

[0034] The main media output device 122 has a function of outputting main media, such as a monitor or speaker. Details of the main media will be described later, but the main media output device 122 may basically be able to output any type of information. Furthermore, the main media output device 122 may be able to output multiple types of information. With such a function, the main media output device 122 outputs main media supplied from the terminal device 112, for example. Furthermore, the main media output device 122 may supply information regarding its own characteristics to the terminal device 112, for example.

[0035] The sub-media output device 123 has a function of outputting sub-media, such as a monitor or speaker. Details of the sub-media will be described later; however, the sub-media output device 123 may basically be capable of outputting any type of information. The sub-media output device 123 may also be a device that is driven based on information supplied as sub-media. For example, the sub-media output device 123 may be lighting equipment such as a spotlight, laser light, or mirror ball. The sub-media output device 123 may also be a device or system that launches fireworks, flames, confetti, tape, water, air, or the like. The sub-media output device 123 may also be a device or system that drives stage equipment, such as moving a stage set or opening and closing a blackout curtain. The sub-media output device 123 may also be mechanical equipment such as a robot, arm, or drone. The sub-media output device 123 may also be a device or system that performs magnetic force control or power control.

[0036] The secondary media output device 123 may also be capable of outputting multiple types of information. Furthermore, the secondary media output device 123 may be multiple types of driving devices or systems. With such functions, the secondary media output device 123 may, for example, output secondary media supplied from the terminal device 112 or drive based on the secondary media. Furthermore, the secondary media output device 123 may, for example, supply information about its own characteristics to the terminal device 112.

[0037] In Figure 1, one network 110, one content providing server 111, one terminal device 112, one input device 121, one main media output device 122, and one sub-media output device 123 are shown, but the content playback system 100 may have any number of these components; for example, there may be multiple components of some or all of these components, and the numbers of these components do not have to be the same. In other words, the content playback system 100 can have any number of networks 110, content providing servers 111, terminal devices 112, input devices 121, main media output devices 122, and sub-media output devices 123.

[0038] 1, the input device 121, the main media output device 122, and the sub-media output device 123 are shown as independent blocks, but this block configuration does not indicate the physical configuration of the devices. The input device 121, the main media output device 122, and the sub-media output device 123 may be physically configured in any manner. For example, part or all of the configuration of each of the input device 121, the main media output device 122, and the sub-media output device 123 may be integrated with part or all of the configuration of another block. For example, a monitor and speaker of a head-mounted display may be included in the main media output device 122, and an acceleration sensor and microphone of the head-mounted display may be included in the input device 121.

[0039] 2 is a block diagram showing an example of the hardware configuration of the content providing server 111 and the terminal device 112. For example, the content providing server 111 and the terminal device 112 may each have hardware configured as shown in FIG.

[0040] As shown in FIG. 2, in the content providing server 111 (or the terminal device 112), a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, and a RAM (Random Access Memory) 203 are interconnected via a bus 204.

[0041] An input / output interface 210 is also connected to the bus 204. An input unit 211, an output unit 212, a storage unit 213, a communication unit 214, and a drive 215 are connected to the input / output interface 210.

[0042] The input unit 211 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 212 includes, for example, a display, a speaker, and an output terminal. The storage unit 213 includes, for example, a hard disk, a RAM disk, and a non-volatile memory. The communication unit 214 includes, for example, a network interface. The drive 215 drives removable media 221 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0043] In the content providing server 111 (or terminal device 112) configured as described above, the CPU 201 performs a series of processes described below by, for example, loading a program stored in the storage unit 213 into the RAM 203 via the input / output interface 210 and the bus 204 and executing the program. The RAM 203 also stores data necessary for the CPU 201 to execute various processes as appropriate.

[0044] The program executed by the content providing server 111 (or the terminal device 112) can be applied by recording it on, for example, a removable medium 221 such as a package medium. In this case, by inserting the removable medium 221 into the drive 215, the program can be installed in the storage unit 213 via the input / output interface 210.

[0045] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 214 and installed in the storage unit 213.

[0046] Alternatively, this program can be installed in advance in the ROM 202 or the storage unit 213 .

[0047] 3. First Embodiment Content Providing Server Fig. 3 is a functional block diagram that schematically illustrates an example of functions possessed by the content providing server 111. The content providing server 111 realizes the functions shown in Fig. 3 using the hardware in Fig. 2. For example, the content providing server 111 realizes the functions shown in Fig. 3 by the CPU 201 executing the program as described above. Furthermore, information is exchanged between the content providing server 111 and the terminal device 112, for example, by using communication with the terminal device 112 via the network 110 via the communication unit 214. In Fig. 3, a description of such hardware configuration is omitted.

[0048] As shown in FIG. 3, the content providing server 111 has, as functional blocks, a scene processing unit 311, a media processing unit 312, a main media supply unit 313, and a sub-media supply unit 314.

[0049] The scene processing unit 311 performs processing related to a content scene. Note that, in this first embodiment, the content scene is assumed to be a scene in 3D space (hereinafter also referred to as a 3D scene). That is, the scene processing unit 311 performs processing related to the content 3D scene, such as constructing a 3D scene. For example, the scene processing unit 311 may acquire operation information supplied from the terminal device 112. This operation information is information indicating an operation performed by a user or the like on the input device 121. For example, this operation information may include instructions input by the user or the like. The scene processing unit 311 may also acquire viewing position direction information supplied from the terminal device 112. This viewing position direction information is information indicating the position and direction of the viewpoint of the user viewing the 3D scene constructed by the scene processing unit 311. This viewing position direction information is information received by the input device 121. For example, this viewing position direction information may include information detected by an acceleration sensor or the like provided in the head-mounted display.

[0050] The scene processing unit 311 may also supply information about the 3D scene that it has constructed to the main media supply unit 313 .

[0051] The scene processing unit 311 may also acquire device startup time information supplied from the terminal device 112. This device startup time information is information indicating the time required for the startup of the sub-media output device 123. In this specification, "start-up" of a device refers to the device entering a state in which it is able to output media (i.e., the state change until it enters a state in which it is able to output media). Any specific processing or state change may be included in this "start-up." For example, this "start-up" may include turning on the power of the device, initializing it, setting parameters, stabilizing its output, restarting it, etc. The process of a device entering a state in which it is able to output media is also referred to as "starting up" the device.

[0052] The scene processing unit 311 may also acquire device characteristic information supplied from the terminal device 112. This device characteristic information is information relating to the characteristics of the secondary media output device 123 (output characteristics of the secondary media).

[0053] The scene processing unit 311 may also supply information about the 3D scene that it has constructed to the sub-media supply unit 314. For example, the scene processing unit 311 may supply sub-media data 333 selected by the interaction processing unit 321 to the sub-media supply unit 314. The scene processing unit 311 may also supply device characteristic information of the sub-media output device 123 used to output the sub-media to the sub-media supply unit 314.

[0054] The scene processing unit 311 may also acquire data necessary for constructing a scene from the media processing unit 312. For example, the scene processing unit 311 may acquire 3D scene data 331, main media-related data 332, sub-media data 333, and the like from the media processing unit 312.

[0055] The scene processing unit 311 may also use this information to construct a 3D scene. For example, the scene processing unit 311 may construct a 3D scene indicated by the 3D scene data using the main media-related data 332 and the sub-media data 333.

[0056] In this first embodiment, the sub-media is interactively played back when a predetermined event occurs in the 3D scene. Hereinafter, an event that triggers such sub-media playback is also referred to as an interaction event. Event-based playback is also referred to as interactive playback. The content of this interactive playback may be anything. For example, when objects collide in a 3D scene (corresponding to an interaction event), a collision sound (corresponding to the sub-media) may be output from a speaker (corresponding to the sub-media output device). Furthermore, when an avatar touches an object in a 3D scene (corresponding to an interaction event), the tactile sensation (corresponding to the sub-media) may be expressed in a hand glove (corresponding to the sub-media output device) worn by the user.

[0057] For such interactive playback, the scene processing unit 311 has an interaction processing unit 321 and a device startup control unit 322. The interaction processing unit 321 executes processing related to such interactive playback. For example, the interaction processing unit 321 may detect interaction events, such as collision detection of objects in a scene.

[0058] The interaction processing unit 321 may also execute processing related to the playback of sub-media corresponding to the detected interaction event. For example, the interaction processing unit 321 may select sub-media (sub-media data 333) to be played back in response to the detected interaction event. The interaction processing unit 321 may also select a sub-media output device 123 to be used to output the selected sub-media. The scene processing unit 311 may supply the information determined by the interaction processing unit 321 to the sub-media supply unit 314. For example, the scene processing unit 311 may supply device startup time information of the selected sub-media output device 123 to the device startup control unit 322 as information about the selected sub-media output device 123.

[0059] Furthermore, the interaction processing unit 321 may control the startup of the selected sub-media output device 123 via the device startup control unit 322 .

[0060] The device startup control unit 322 executes processing related to control of startup of the sub-media output device 123. For example, the device startup control unit 322 may supply device startup instruction information that instructs the terminal apparatus 112 to start up the sub-media output device 123 selected by the interaction processing unit 321. This device startup instruction information is information that instructs the startup of the sub-media output device 123. In other words, the device startup control unit 322 controls the startup of the sub-media output device 123 by supplying device startup instruction information that instructs the sub-media output device 123 selected by the interaction processing unit 321 to start up to the terminal apparatus 112 that controls the driving of the sub-media output device 123. Note that if the sub-media output device 123 has a communication function with the content providing server 111, the device startup control unit 322 may supply this device startup instruction information to the sub-media output device 123.

[0061] The media processing unit 312 stores information related to media and performs media-related processing. For example, the media processing unit 312 may store 3D scene data 331, main media-related data 332, and sub-media data 333, and supply them to the scene processing unit 311 as appropriate.

[0062] The 3D scene data 331 is data that represents a 3D scene of the content. The 3D scene data 331 may be, for example, a scene description written in a predetermined format. The main media related data 332 is data used to generate main media that represents the 3D scene. For example, the main media related data 332 may be timed 3D data that can change in the time direction. The sub-media data 333 is data of the sub-media. For example, the sub-media data 333 may be timed haptic data that can change in the time direction. Note that these data may be coded. In other words, these data may be coded data.

[0063] The main media supply unit 313 executes processing related to the main media, which is the primary media of the content. For example, the main media supply unit 313 may acquire data necessary for generating the main media, such as main media-related data 332, from the scene processing unit 311. The main media supply unit 313 may use the acquired data to generate main media that represents a 3D scene constructed in the scene processing unit 311. For example, the main media supply unit 313 may generate main media of 2D data obtained by rendering the 3D scene onto a 2D plane.

[0064] This main media may be any type of information. For example, the main media may be visual information. For example, the main media may be image data. This image data may be 2D data or 3D data. Furthermore, the main media may be auditory information. For example, the main media may be audio data. For example, this audio data may be monaural data, stereo data, or 3D data. Of course, these are just examples, and the main media is not limited to these examples. However, since the main media is the primary medium for expressing content, it is generally often video or audio.

[0065] Furthermore, the main media supply unit 313 may supply the generated main media to the terminal device 112 that controls the main media output device 122. If the main media output device 122 has a function of communicating with the content providing server 111, the main media supply unit 313 may supply the main media to the main media output device 122.

[0066] The sub-media supply unit 314 executes processing related to the sub-media that is played back in conjunction with the playback of the main media. For example, the sub-media supply unit 314 may acquire data necessary for generating the sub-media, such as the sub-media data 333, from the scene processing unit 311. The sub-media supply unit 314 may use this data to generate the sub-media specified by the interaction processing unit 321. For example, the sub-media supply unit 314 may decode the sub-media data 333, which is encoded data.

[0067] This sub-media may be any type of information. For example, the sub-media may be tactile (haptic) information. For example, the sub-media may be force (pressure) information that represents vibration, wind, etc. The sub-media may also be thermal (cold) information that represents heat, etc. The sub-media may also be taste information that represents flavor, etc. The sub-media may also be olfactory information that represents scent, etc. Of course, the sub-media may also be visual information such as image data (2D data, 3D data, etc.). The sub-media may also be auditory information such as audio data (monaural, stereo, 3D, etc.).

[0068] The sub-media may also be information intended for physical (facility) control. For example, the sub-media may be control information for controlling an actuator. The controlled object and the control content may be any as long as they function in conjunction with the main media. For example, the sub-media may control lighting equipment. For example, the sub-media may control the color, brightness, direction, etc. of spotlights or laser light. The sub-media may also control the rotation of a mirror ball. The sub-media may also control stage production equipment. For example, the sub-media may control the output (timing, direction, output amount, etc.) of fireworks, flames, confetti, tape, water, air, etc. The sub-media may also control stage equipment. For example, the sub-media may control the movement of a stage set or the raising and lowering of a blackout curtain. The sub-media may also control mechanical devices. For example, the sub-media may control the drive of a robot or arm, the movement of a drone, etc. The sub-media may also be used for power control or magnetic force control.

[0069] Of course, these are just examples, and sub-media are not limited to these examples. However, sub-media are media that are played in conjunction with main media, and generally, it is desirable for them to have effects that supplement the effects of the main media (for example, visual effects, auditory effects, etc.).

[0070] Also, in this example, the sub-media supply unit 314 may execute processing related to characteristic conversion of the sub-media. For example, the sub-media supply unit 314 may have a device characteristic conversion processing unit 341. The sub-media data 333 is generated so as to conform to reference characteristics, which are the standard characteristics of the sub-media output device 123. The device characteristic conversion processing unit 341 converts the characteristics of the generated sub-media to match the device characteristics of the sub-media output device 123 to be used. The device characteristic conversion processing unit 341 may acquire device characteristic information of this sub-media output device 123 from the scene processing unit 311, and convert the characteristics of the sub-media based on the device characteristic information (i.e., to match the device characteristics indicated by the device characteristic information).

[0071] The sub-media supply unit 314 may supply the sub-media to the terminal device 112 that controls the sub-media output device 123. If the sub-media output device 123 has a communication function with the content providing server 111, the sub-media supply unit 314 may supply the sub-media to the sub-media output device 123.

[0072] <Terminal Device> Fig. 4 is a functional block diagram that schematically shows an example of the functions of the terminal device 112. The terminal device 112 realizes the functions shown in Fig. 4 using the hardware of Fig. 2. For example, the terminal device 112 realizes the functions shown in Fig. 4 by the CPU 201 executing the program as described above. Furthermore, information is exchanged between the terminal device 112 and the content providing server 111 by, for example, communication with the content providing server 111 via the network 110 by the communication unit 214. In Fig. 4, a description of such hardware configuration is omitted.

[0073] As shown in FIG. 4, the terminal device 112 has, as functional blocks, an operation information supply unit 411, a viewing position direction information supply unit 412, a main media output device control unit 413, a sub-media output device control unit 414, a device start-up time information supply unit 415, and a device characteristic information supply unit 416.

[0074] The operation information supply unit 411 acquires operation information from the input device 121 and supplies it to the content providing server 111. This operation information is information indicating an operation performed by a user or the like on the input device 121. For example, this operation information may include an instruction or the like input by a user or the like.

[0075] The viewing position and direction information supply unit 412 acquires viewing position and direction information from the input device 121 and supplies it to the content providing server 111. This viewing position and direction information is information that indicates the position and direction of the viewpoint of the user viewing the 3D scene constructed by the scene processing unit 311. This viewing position and direction information is information that is accepted by the input device 121. For example, this viewing position and direction information may include information detected by an acceleration sensor or the like provided in the head-mounted display.

[0076] The main media output device control unit 413 executes processing related to the control of the main media output device 122. For example, the main media output device control unit 413 may acquire main media supplied from the content providing server 111. The main media output device control unit 413 may supply control information for controlling the main media output device 122 to the main media output device 122 based on the acquired main media. The main media output device control unit 413 may supply the acquired main media to the main media output device 122.

[0077] The sub-media output device control unit 414 executes processing related to the control of the sub-media output device 123. For example, the sub-media output device control unit 414 may acquire device start-up instruction information supplied from the content providing server 111. The sub-media output device control unit 414 may supply control information for starting up the sub-media output device 123 to the sub-media output device 123 based on the device start-up instruction information. The sub-media output device control unit 414 may also acquire sub-media supplied from the content providing server 111. The sub-media output device control unit 414 may supply control information for controlling the sub-media output device 123 to the sub-media output device 123 based on the acquired sub-media. The sub-media output device control unit 414 may supply the acquired sub-media to the sub-media output device 123.

[0078] The device startup time information supply unit 415 acquires the device startup time information supplied from the secondary media output device 123 and supplies it to the content providing server 111. This device startup time information is information indicating the time required for the secondary media output device 123 to start up.

[0079] The device characteristic information supply unit 416 acquires the device characteristic information supplied from the secondary media output device 123 and supplies it to the content providing server 111. This device characteristic information is information relating to the characteristics of the secondary media output device 123 (output characteristics of the secondary media).

[0080] <Supply of Device Start-Up Instruction Information> As described above, the secondary media is interactively played back in conjunction with the playback of the main media in such a content playback system 100. Therefore, the secondary media output device 123 needs to output the secondary media at an appropriate timing according to the occurrence timing of an interaction event.

[0081] Furthermore, as described above, since the sub-media is played back as an auxiliary to the main media, it is generally not output continuously but rather intermittently. For example, if the sound of an object colliding is used as the sub-media, that sub-media will only be played back immediately after the object collides. In such cases, the sub-media output device 123 may not always be in a state where it can output the playback device. When outputting the sub-media, the sub-media output device 123 may be powered on, initial settings may be performed, and output may begin.

[0082] However, as described above, a variety of information can be used as secondary media. Therefore, various devices can be the secondary media output device 123. For this reason, there may be cases where the time required for the secondary media output device 123 to start up (start-up time) is longer than that required for output devices for images, audio, etc., which are often used as main media.

[0083] In this way, when the startup time of the sub-media output device 123 is long, if the sub-media output device control unit 414 of the terminal device 112 acquires the sub-media and then controls and starts up the sub-media output device 123, there is a risk that the output of the sub-media will not arrive in time for its appropriate output timing. In other words, there is a risk that the output of the sub-media will be delayed. In that case, for example, an inappropriate gap may occur between the output timing of the sub-media and the occurrence timing of an interaction event, which may reduce the quality of the user experience. In other words, there is a risk that the quality of the content provided by the content providing server 111 will be reduced.

[0084] Therefore, in order to prevent a delay in the secondary media output due to a delay in device startup, the device startup control unit 322 supplies device startup instruction information to the secondary media output device control unit 414 at a sufficiently early timing. Also, the secondary media output device control unit 414 acquires the device startup instruction information at a sufficiently early timing and starts up the secondary media output device 123.

[0085] For example, the content providing server 111 (first information processing device) is provided with a device startup control unit 322 that controls the startup of the sub-media output device 123 by supplying device startup instruction information that instructs the startup of the sub-media output device 123 to the sub-media output device 123 or a control device (sub-media output device control unit 414 of the terminal device 112) that controls the sub-media output device 123 at a timing that takes into account the startup time of the sub-media output device 123 that outputs the sub-media that is played in conjunction with the playback of the main media, which is the main media of the content. In addition, the content providing server 111 (first information processing device) supplies device start-up instruction information instructing the start-up of the sub-media output device 123 to the sub-media output device 123 or the control device (sub-media output device control unit 414 of the terminal device 112) that controls the sub-media output device 123 at a timing that takes into account the start-up time of the sub-media output device 123, which outputs the sub-media that is played in conjunction with the playback of the main media, which is the main media of the content, thereby controlling the start-up of the sub-media output device 123. Furthermore, the first program supplies device startup instruction information instructing the startup of the sub-media output device 123 to the sub-media output device 123 or the control device (sub-media output device control unit 414 of the terminal device 112) that controls the sub-media output device 123 at a timing that takes into consideration the startup time of the sub-media output device 123, which outputs the sub-media that is played in conjunction with the playback of the main media, which is the subject media of the content, thereby causing the computer to execute processing that includes controlling the startup of the sub-media output device 123.

[0086] In this way, the terminal device 112 (the secondary media output device control unit 414 thereof) can start up the secondary media output device 123 in time for the secondary media output timing based on the device start-up instruction information. Therefore, the content providing server 111 can suppress the occurrence of secondary media output delays due to delays in device start-up.

[0087] For example, the terminal device 112 (second information processing device) may include a sub-media output device control unit (sub-media output device control unit 414) that starts up the sub-media output device 123 based on device start-up instruction information supplied at a timing that takes into account the start-up time of the sub-media output device 123, which outputs the sub-media played in conjunction with the playback of the main media, which is the primary media of the content. The second information processing method may also include the terminal device 112 (second information processing device) starting up the sub-media output device 123 based on device start-up instruction information supplied at a timing that takes into account the start-up time of the sub-media output device 123, which outputs the sub-media played in conjunction with the playback of the main media, which is the primary media of the content. The second program may also cause a computer to execute processing that includes starting up the sub-media output device 123 based on device start-up instruction information supplied at a timing that takes into account the start-up time of the sub-media output device 123, which outputs the sub-media played in conjunction with the playback of the main media, which is the primary media of the content.

[0088] In this way, the terminal device 112 (the secondary media output device control unit 414 thereof) can start up the secondary media output device 123 in time for the secondary media output timing based on the device start-up instruction information. Therefore, the terminal device 112 can suppress the occurrence of a secondary media output delay due to a delay in device start-up.

[0089] <Flow of Server Processing> An example of the flow of server processing executed by the content providing server 111 in this case will be described with reference to the flowchart of FIG.

[0090] When the server processing starts, the scene processing unit 311 acquires the 3D scene data 331 in step S101.

[0091] In step S102 , the scene processing unit 311 acquires device startup time information and device characteristic information of the secondary media output device 123 .

[0092] In step S103, the scene processing unit 311 acquires operation information and viewing position direction information.

[0093] The processes from step S104 to step S107 and the processes from step S108 to step S113 are executed in parallel.

[0094] In step S104 , the scene processing unit 311 acquires the main media related data 332 .

[0095] In step S105 , the scene processing unit 311 constructs a 3D scene based on the 3D scene data 331 .

[0096] In step S106, the main media supply unit 313 generates main media corresponding to the 3D scene using the main media related data 332 and the like.

[0097] In step S107, the main media supply unit 313 supplies the generated main media to the terminal device 112, which is a client of the content providing server 111. When the process of step S107 ends, the process proceeds to step S114.

[0098] In step S108, the interaction processing unit 321 analyzes the interaction in the 3D scene.

[0099] In step S109, the interaction processing unit 321 determines whether the analysis results satisfy the interaction condition. That is, the interaction processing unit 321 determines whether an interaction event has occurred. If it is determined that the interaction condition has been satisfied (that is, an interaction event has occurred), the process proceeds to step S110.

[0100] In step S110, the device startup control unit 322 supplies device startup instruction information for the secondary media output device at a timing that takes the startup time into consideration.

[0101] In step S111, the media sub-supply unit 314 generates media sub-content corresponding to the 3D scene and the interaction event that occurred therein.

[0102] In step S112 , the device characteristics conversion processing unit 341 converts the device characteristics of the sub-media based on the device characteristics information of the sub-media output device 123 so as to match the device characteristics of the sub-media output device 123 .

[0103] In step S113, the sub-media supply unit 314 supplies the sub-media to the terminal device 112. When the processing of step S113 ends, the processing proceeds to step S114. Also, if it is determined in step S109 that the interaction condition is not satisfied (i.e., an interaction event has not occurred), the processing proceeds to step S114.

[0104] In step S114, the scene processing unit 311 determines whether or not to end the server processing. If the content is being played back and it is determined not to end the server processing, the process returns to step S103, and the subsequent processes are executed. Also, if the content playback has ended and it is determined to end the server processing in step S114, the server processing ends.

[0105] By executing each process in this manner, the content providing server 111 can suppress the occurrence of a delay in sub-media output due to a delay in device startup.

[0106] In the first information processing apparatus, the "timing taking into consideration the startup time of the secondary media output device" may be a timing that is a predetermined time earlier than the timing at which the secondary media output device outputs the secondary media. In other words, in step S110, the device startup control unit 322 may supply the device startup instruction information earlier by a predetermined fixed time.

[0107] In the first information processing apparatus, the "timing taking into consideration the startup time of the secondary media output device" may be timing based on the startup time of the secondary media output device 123 indicated by the device startup time information. That is, in step S110, the device startup control unit 322 may set the timing for supplying the device startup instruction information based on the device startup time information.

[0108] The first information processing device may further include a sub-media supply unit 314 that supplies sub-media corresponding to a scene of the content to the sub-media output device or the control device. The "timing taking into account the startup time of the sub-media output device" may be a timing before the sub-media is supplied. In other words, the device startup control unit 322 may execute the process of step S110 before the process of step S113.

[0109] Alternatively, the sub-media supply unit 314 may convert the device characteristics of the sub-media to match the characteristics of the sub-media output device, and supply the converted sub-media. The "timing taking into account the startup time of the sub-media output device" may be a timing before the device characteristics of the sub-media are converted. In other words, the device startup control unit 322 may execute the process of step S110 before the process of step S112.

[0110] Alternatively, the sub-media supply unit 314 may generate sub-media corresponding to the scene and supply the generated sub-media. The "timing taking into consideration the startup time of the sub-media output device" may be a timing before the sub-media is generated. In other words, the device startup control unit 322 may execute the process of step S110 before the process of step S111.

[0111] The first information processing apparatus may further include an interaction processing unit that analyzes an interaction event in a scene. The "timing that takes into account the startup time of the secondary media output device" may be a timing before the secondary media is supplied based on the occurrence of the interaction event. In other words, the device startup control unit 322 may execute the process of step S110 after the process of step S109.

[0112] The first information processing apparatus may further include a main media supply unit 313 that supplies main media corresponding to a scene of content to a main media output device 122 that outputs the main media or to a control device (the terminal apparatus 112 (main media output device control unit 413 thereof)) that controls the main media output device 122. The "timing that takes into account the startup time of the sub-media output device" may be a timing before the main media is supplied. In other words, the device startup control unit 322 may execute the process of step S110 before the process of step S107.

[0113] Alternatively, the main media supply unit 313 may generate main media corresponding to the scene and supply the generated main media. The "timing taking into consideration the startup time of the secondary media output device" may be a timing before the main media is generated. In other words, the device startup control unit 322 may execute the process of step S110 before the process of step S106.

[0114] The main media and sub-media may be media corresponding to a 3D scene of the content, that is, the scene processing unit 311 may construct a 3D scene in step S105.

[0115] <Flow of Terminal Processing> An example of the flow of terminal processing executed by the terminal device 112 in this case will be described with reference to the flowchart of FIG.

[0116] When terminal processing is started, in step S131, the device start-up time information supply unit 415 supplies the device start-up time information of the secondary media output device 123 to the content providing server 111. The device characteristic information supply unit 416 supplies the device characteristic information of the secondary media output device 123 to the content providing server 111.

[0117] The processes from step S132 to step S135 and the processes from step S136 to step S139 are executed in parallel.

[0118] In step S132, the operation information supply unit 411 supplies the operation information acquired from the input device 121 to the content providing server 111. The viewing position / direction information supply unit 412 supplies the viewing position / direction information acquired from the input device 121 to the content providing server 111.

[0119] In step S133 , the main media output device control unit 413 acquires the main media supplied from the content providing server 111 .

[0120] In step S134, the main media output device control unit 413 corrects the main media in accordance with the user direction.

[0121] In step S135, the main media output device control unit 413 supplies and outputs the main media to the main media output device 122. When the process of step S135 ends, the process proceeds to step S140.

[0122] Also, in step S136, the secondary media output device control unit 414 determines whether or not it has acquired device startup instruction information supplied from the content providing server 111. If it is determined that it has been acquired, the process proceeds to step S137.

[0123] In step S137, the secondary media output device control section 414 starts up the secondary media output device 123 in time for the device usage timing in accordance with the device start-up instruction information.

[0124] In step S138 , the sub-media output device control unit 414 acquires the sub-media supplied from the content providing server 111 .

[0125] In step S139, the sub-media output device control unit 414 supplies the sub-media to the sub-media output device 123 and outputs it. When the processing of step S139 ends, the processing proceeds to step S140. Also, if it is determined in step S136 that device startup instruction information has not been acquired, the processing proceeds to step S140.

[0126] In step S140, the secondary media output device control unit 414 determines whether or not to end the terminal processing. If content is being played and it is determined not to end the terminal processing, the process returns to step S132 and step S136, and the subsequent processes are executed. Also, if it is determined in step S140 that content playback has ended and the terminal processing is to end, the terminal processing ends.

[0127] By executing each process in this manner, the terminal device 112 can suppress the occurrence of a delay in sub-media output due to a delay in device startup.

[0128] In the second information processing apparatus, the "timing that takes into consideration the startup time of the secondary media output device 123" may be a timing that is a predetermined time earlier than the timing at which the secondary media output device 123 outputs the secondary media. In other words, in step S136, the secondary media output device control unit 414 may acquire the device startup instruction information earlier by a predetermined fixed time.

[0129] The second information processing apparatus may further include a device startup time information supply unit 415 that supplies device startup time information indicating the startup time of the secondary media output device 123. The "timing taking into consideration the startup time of the secondary media output device 123" may be timing based on the startup time of the secondary media output device 123 indicated by the device startup time information. That is, in step S136, the secondary media output device control unit 414 may acquire device startup instruction information that is supplied at the timing set based on the device startup time information.

[0130] The secondary media output device control unit 414 may further acquire the supplied secondary media and output the acquired secondary media to the secondary media output device. The "timing taking into account the startup time of the secondary media output device 123" may be a timing before the secondary media is acquired. In other words, the secondary media output device control unit 414 may execute the process of step S136 before the process of step S138.

[0131] The second information processing device may further include a main media output device control unit that acquires main media corresponding to a scene of the content and causes the main media output device to output the acquired main media. The "timing that takes into account the startup time of the sub-media output device 123" may be a timing before the main media is acquired. In other words, the sub-media output device control unit 414 may execute the process of step S136 before the process of step S133.

[0132] The above-mentioned main media and sub-media may be media corresponding to 3D scenes of the content.

[0133] <Content Providing Server> In the above, it has been explained that the device characteristic conversion of the sub-media is performed in the content providing server 111, but this device characteristic conversion may also be performed in the terminal device 112. This case will be described below.

[0134] Fig. 7 is a functional block diagram showing an example of functions of the content providing server 111. As shown in Fig. 7, in this case, the sub-media supply unit 314 does not have a device characteristics conversion processing unit 341. In other words, the sub-media supply unit 314 supplies the generated sub-media to the terminal device 112 without converting the device characteristics (leaving it in correspondence with the characteristics of the reference device).

[0135] In this case, it is not necessary to supply device characteristic information because no conversion of device characteristics is performed in the content providing server 111. Therefore, the above-described process of transmitting device characteristic information from the terminal device 112 to the content providing server 111 is omitted.

[0136] Other than that, it is the same as the example above.

[0137] <Terminal Device> FIG. 8 is a functional block diagram that schematically illustrates an example of the functions of the terminal device 112 in this case. As shown in FIG. 8, in this case, the sub-media output device control unit 414 has a device characteristics conversion processing unit 341. That is, the sub-media output device control unit 414 may perform processing related to sub-media characteristic conversion, similar to the sub-media supply unit 314 ( FIG. 3 ) in the example described above. The device characteristics conversion processing unit 341 may convert the characteristics of the sub-media supplied from the content providing server 111 based on the device characteristic information of the sub-media output device 123 (i.e., to match the device characteristics indicated by the device characteristic information), similar to the example described above. The sub-media output device control unit 414 may supply the sub-media to the sub-media output device 123.

[0138] In this case, the content providing server 111 does not convert the device characteristics and there is no need to supply device characteristic information, so the above-described process of transmitting device characteristic information from the terminal device 112 to the content providing server 111 is omitted. That is, in this case, the device characteristic information supply unit 416 may be omitted.

[0139] Other than that, it is the same as the example above.

[0140] That is, in this case, as in the example described above, in order to suppress the occurrence of a delay in secondary media output due to a delay in device startup, the device startup control unit 322 supplies device startup instruction information to the secondary media output device control unit 414 at a sufficiently early timing. Also, the secondary media output device control unit 414 acquires the device startup instruction information at a sufficiently early timing and starts up the secondary media output device 123.

[0141] In this way, the terminal device 112 (the sub-media output device control unit 414 thereof) can start up the sub-media output device 123 in time for the sub-media output timing based on the device start-up instruction information. Therefore, the content providing server 111 and the terminal device 112 can suppress the occurrence of a sub-media output delay due to a delay in device start-up.

[0142] <Flow of Server Processing> An example of the flow of server processing executed by the content providing server 111 in this case will be described with reference to the flowchart of FIG.

[0143] When the server processing starts, the scene processing unit 311 acquires the 3D scene data 331 in step S151.

[0144] In step S152 , the scene processing unit 311 acquires device startup time information of the secondary media output device 123 .

[0145] In step S153, the scene processing unit 311 acquires operation information and viewing position direction information.

[0146] The processes from step S154 to step S157 and the processes from step S158 to step S162 are executed in parallel.

[0147] In step S154 , the scene processing unit 311 acquires the main media related data 332 .

[0148] In step S155 , the scene processing unit 311 constructs a 3D scene based on the 3D scene data 331 .

[0149] In step S156, the main media supply unit 313 generates main media corresponding to the 3D scene using the main media related data 332 and the like.

[0150] In step S157, the main media supply unit 313 supplies the generated main media to the terminal device 112, which is a client of the content providing server 111. When the process of step S157 ends, the process proceeds to step S163.

[0151] In addition, in step S158, the interaction processing unit 321 analyzes the interaction in the 3D scene.

[0152] In step S159, the interaction processing unit 321 determines whether the interaction condition is satisfied based on the analysis result. That is, the interaction processing unit 321 determines whether an interaction event has occurred. If it is determined that the interaction condition is satisfied (that is, an interaction event has occurred), the process proceeds to step S160.

[0153] In step S160, the device startup control unit 322 supplies device startup instruction information for the secondary media output device at a timing that takes the startup time into consideration.

[0154] In step S161, the media sub-supply unit 314 generates media sub-content corresponding to the 3D scene and the interaction event that occurred therein.

[0155] In step S162, the sub-media supply unit 314 supplies the sub-media to the terminal device 112. When the processing of step S162 ends, the processing proceeds to step S163. Also, if it is determined in step S159 that the interaction condition is not satisfied (i.e., an interaction event has not occurred), the processing proceeds to step S163.

[0156] In step S163, the scene processing unit 311 determines whether or not to end the server processing. If it is determined that the content is being played back and the server processing is not to be ended, the process returns to step S153, and the subsequent processes are executed. Also, if it is determined in step S163 that the content playback has ended and the server processing is to be ended, the server processing is ended.

[0157] By executing each process in this manner, the content providing server 111 can suppress the occurrence of a delay in sub-media output due to a delay in device startup.

[0158] <Flow of Terminal Processing> An example of the flow of terminal processing executed by the terminal device 112 in this case will be described with reference to the flowchart of FIG.

[0159] When the terminal processing is started, in step S181, the device start-up time information supply unit 415 supplies the device start-up time information of the secondary media output device 123 to the content providing server 111.

[0160] The processes from step S182 to step S185 and the processes from step S186 to step S190 are executed in parallel.

[0161] In step S182, the operation information supply unit 411 supplies the operation information acquired from the input device 121 to the content providing server 111. The viewing position / direction information supply unit 412 supplies the viewing position / direction information acquired from the input device 121 to the content providing server 111.

[0162] In step S183 , the main media output device control unit 413 acquires the main media supplied from the content providing server 111 .

[0163] In step S184, the main media output device control unit 413 corrects the main media in accordance with the user direction.

[0164] In step S185, the main media output device control unit 413 supplies the main media to the main media output device 122 to output it. When the process of step S185 ends, the process proceeds to step S191.

[0165] Also, in step S186, the secondary media output device control unit 414 determines whether or not it has acquired device startup instruction information supplied from the content providing server 111. If it is determined that it has been acquired, the processing proceeds to step S187.

[0166] In step S187, the secondary media output device control section 414 starts up the secondary media output device 123 in time for the device usage timing in accordance with the device start-up instruction information.

[0167] In step S 188 , the sub-media output device control unit 414 acquires the sub-media supplied from the content providing server 111 .

[0168] In step S189 , the device characteristics conversion processing unit 341 converts the device characteristics of the sub-media to match the device characteristics of the sub-media output device 123 based on the device characteristics information of the sub-media output device 123 .

[0169] In step S190, the sub-media output device control unit 414 supplies the sub-media to the sub-media output device 123 and outputs it. When the processing of step S190 ends, the processing proceeds to step S191. Also, if it is determined in step S186 that device startup instruction information has not been acquired, the processing proceeds to step S191.

[0170] In step S191, the secondary media output device control unit 414 determines whether or not to end the terminal processing. If content is being played and it is determined not to end the terminal processing, the process returns to step S182 and step S186, and the subsequent processes are executed. Also, if it is determined in step S191 that content playback has ended and the terminal processing is to end, the terminal processing ends.

[0171] By executing each process in this manner, the terminal device 112 can suppress the occurrence of a delay in sub-media output due to a delay in device startup.

[0172] The sub-media output device control unit 414 may convert the device characteristics of the acquired sub-media to match the characteristics of the sub-media output device 123, and output the converted sub-media to the sub-media output device 123. The "timing taking into account the startup time of the sub-media output device" may be a timing before the device characteristics of the sub-media are converted. In other words, the sub-media output device control unit 414 may execute the process of step S186 before the process of step S189.

[0173] <4. Second embodiment> <Prediction of event occurrence> Furthermore, if the start-up time of the haptic device takes longer than the delay in server processing, a further presentation delay occurs, and there is a risk that a problem will occur in which the video and haptics cannot be presented synchronously due to the time delay between the delay-compensated video presentation and the haptic presentation that cannot be delay-compensated.

[0174] Therefore, a 3D scene may be analyzed to predict an interaction with a haptic device, and information for activating the haptic device may be sent to the terminal before the interaction occurs.

[0175] 11 is a functional block diagram schematically illustrating an example of functions of the content providing server 111 according to the second embodiment. As shown in FIG. 11 , in this case, the scene processing unit 311 further includes an interaction prediction unit 521.

[0176] The interaction prediction unit 521 executes processing related to the prediction of an interaction event. For example, the interaction prediction unit 521 may predict the occurrence of the interaction event in a scene. That is, the interaction prediction unit 521 predicts whether or not an interaction event will occur in the scene in the future, and supplies the prediction result to the interaction processing unit 321. The interaction processing unit 321 controls the device launch control unit 322 based on the prediction result. That is, the device launch control unit 322 may supply device launch instruction information to the terminal device 112, instructing the launch of the secondary media output device 123 that outputs secondary media corresponding to the interaction event predicted by the interaction prediction unit 521.

[0177] Other than that, it is the same as the example in FIG. 3 (first embodiment).

[0178] In this way, the content providing server 111 (first information processing device) may further include an interaction prediction unit 521 that predicts the occurrence of an interaction event in a scene. In this case, the "timing taking into consideration the startup time of the secondary media output device" may be a timing based on the occurrence timing of the predicted interaction event.

[0179] In this way, the content providing server 111 can supply the device start-up instruction information at an earlier timing. Therefore, the terminal device 112 (the secondary media output device control unit 414 thereof) can start up the secondary media output device 123 at an earlier timing based on the device start-up instruction information. Therefore, the content providing server 111 can further suppress the occurrence of secondary media output delays due to delays in device start-up.

[0180] <Terminal Device> In this case, the functional block configuration of the terminal device 112 may be the same as that in the first embodiment (FIG. 4).

[0181] <Flow of Server Processing> An example of the flow of server processing executed by the content providing server 111 in this case will be described with reference to the flowchart of FIG.

[0182] In this case, the processes of steps S211 to S217 are executed in the same manner as the processes of steps S101 to S107 in FIG.

[0183] In step S218, the interaction prediction unit 521 predicts and analyzes interactions in the 3D scene.

[0184] In step S219, the interaction prediction unit 521 determines whether the analysis results satisfy an interaction prediction condition. That is, the interaction prediction unit 521 determines whether the occurrence of an interaction event is predicted. If it is determined that the interaction prediction condition is satisfied (that is, the occurrence of an interaction event is predicted), the process proceeds to step S220.

[0185] The processes of steps S220 to S224 are executed in the same manner as the processes of steps S110 to S114 in FIG.

[0186] If it is determined in step S224 that the playback of the content has ended and the server processing is to be ended, the server processing ends.

[0187] By executing each process in this manner, the content providing server 111 can further reduce the occurrence of delays in sub-media output due to delays in device startup.

[0188] <Flow of Terminal Processing> The terminal processing in this case may be executed in the same manner as in the first embodiment (FIG. 6).

[0189] <Content Providing Server> In the second embodiment, similarly to the first embodiment, the device characteristic conversion of the sub-media may be executed in the terminal device 112 .

[0190] In this case, the content providing server 111 has the functional blocks shown in Fig. 13. That is, the content providing server 111 in this case has an interaction prediction unit 521 in addition to the configuration in the example of Fig. 7. This interaction prediction unit 521 is as described above with reference to Fig. 11.

[0191] Therefore, the content providing server 111 can supply the device start-up instruction information at an earlier timing. Therefore, the terminal device 112 (the secondary media output device control unit 414 thereof) can start up the secondary media output device 123 at an earlier timing based on the device start-up instruction information. Therefore, the content providing server 111 can further suppress the occurrence of secondary media output delays due to delays in device start-up.

[0192] <Terminal Device> In this case, the functional block configuration of the terminal device 112 may be the same as that in the first embodiment (FIG. 8).

[0193] <Flow of Server Processing> An example of the flow of server processing executed by the content providing server 111 in this case will be described with reference to the flowchart of FIG.

[0194] In this case, the processes of steps S241 to S247 are executed in the same manner as the processes of steps S151 to S157 in FIG.

[0195] In step S248, the interaction prediction unit 521 predicts and analyzes interactions in the 3D scene.

[0196] In step S249, the interaction prediction unit 521 determines whether the interaction prediction condition is satisfied based on the analysis result. That is, the interaction prediction unit 521 determines whether the occurrence of an interaction event is predicted. If it is determined that the interaction prediction condition is satisfied (that is, the occurrence of an interaction event is predicted), the process proceeds to step S250.

[0197] The processes of steps S250 to S253 are executed in the same manner as the processes of steps S160 to S163 in FIG.

[0198] If it is determined in step S253 that the playback of the content has ended and the server processing is to be ended, the server processing ends.

[0199] By executing each process in this manner, the content providing server 111 can further reduce the occurrence of delays in sub-media output due to delays in device startup.

[0200] <Flow of Terminal Processing> The terminal processing in this case may be executed in the same manner as in the first embodiment (FIG. 10).

[0201] <Predicting Occurrence Probability> When predicting the occurrence of an interaction event, the probability of a future interaction event may be predicted, and the presence or absence of an interaction event may be predicted based on the predicted probability. For example, as shown in the example of FIG. 15 , the probability of an interaction event may be predicted for each future time period, and if this probability is equal to or greater than a predetermined threshold, an interaction event may be predicted to occur. In the example of FIG. 15 , if the device delay time is 0.6 seconds and the threshold is 90%, the probability of an interaction occurring at time t is calculated based on the 3D scene and user information, and an action is executed at time t when the probability calculated as 90% after t + 0.6 seconds has passed.

[0202] For example, in the first information processing device, the interaction prediction unit 521 may derive the occurrence probability of an interaction event and predict the occurrence of an interaction event based on the derived occurrence probability.

[0203] In this way, the occurrence of an interaction event can be predicted more accurately.

[0204] 5. Third Embodiment Synchronized Playback of Sub-Media In the first and second embodiments, the sub-media has been described as being interactively played back, but the playback method of the sub-media may be synchronized playback, in which the sub-media is played back in synchronization with the playback of the main media. In this third embodiment, a case in which the sub-media is played back in synchronization will be described. In other words, the sub-media is assumed to be synchronized media.

[0205] 16 is a functional block diagram showing an example of functions of the content providing server 111 according to the third embodiment. As shown in FIG. 16, in this case, the scene processing unit 311 has a synchronization-advance processing unit 541 instead of the interaction processing unit 321.

[0206] The synchronization advance processing unit 541 executes processing related to synchronized playback of the sub-media. In this case, the synchronization advance processing unit 541 may analyze the processing prior to the actual synchronized playback of the sub-media and detect the future playback timing of the sub-media. Based on the results of this advance analysis, the device startup control unit 322 may supply device startup instruction information to the terminal device 112 that instructs the startup of the sub-media output device 123 that will output the sub-media to be played in the future.

[0207] Other than that, it is the same as the example in FIG. 3 (first embodiment).

[0208] In this way, the content providing server 111 (first information processing device) may further include a synchronization advance processing unit that analyzes in advance the synchronous playback of the secondary media with the main media in a scene. The "timing taking into consideration the startup time of the secondary media output device" may be a timing before the secondary media to be synchronously played is supplied.

[0209] In this way, the content providing server 111 can supply the device start-up instruction information at an earlier timing. Therefore, the terminal device 112 (the secondary media output device control unit 414 thereof) can start up the secondary media output device 123 at an earlier timing based on the device start-up instruction information. Therefore, the content providing server 111 can further suppress the occurrence of secondary media output delays due to delays in device start-up.

[0210] <Terminal Device> In this case, the functional block configuration of the terminal device 112 may be the same as that in the first embodiment (FIG. 4).

[0211] <Flow of Server Processing> An example of the flow of server processing executed by the content providing server 111 in this case will be described with reference to the flowchart of FIG.

[0212] In this case, the processes of steps S271 to S277 are executed in the same manner as the processes of steps S101 to S107 in FIG.

[0213] In step S278, the synchronization-advance processing unit 541 analyzes the synchronized playback of the sub-media in the 3D scene.

[0214] In step S279, the synchronization advance processing unit 541 determines, based on the analysis result, whether or not there is a synchronized playback timing for the sub-media in the future. If it is determined that there is a synchronized playback timing, the process proceeds to step S280.

[0215] The processes of steps S280 to S284 are executed in the same manner as the processes of steps S110 to S114 in FIG.

[0216] If it is determined in step S284 that the playback of the content has ended and the server processing is to be ended, the server processing ends.

[0217] By executing each process in this manner, the content providing server 111 can further reduce the occurrence of delays in sub-media output due to delays in device startup.

[0218] <Flow of Terminal Processing> The terminal processing in this case may be executed in the same manner as in the first embodiment (FIG. 6).

[0219] <Content Providing Server> In the third embodiment, similarly to the first embodiment, the terminal device 112 may also perform device characteristic conversion of the sub-media.

[0220] In this case, the content providing server 111 has the functional blocks shown in Fig. 18. That is, the content providing server 111 in this case has a synchronization-first processing unit 541 instead of the interaction processing unit 321 in the configuration of the example in Fig. 7. The synchronization-first processing unit 541 is as described above with reference to Fig. 16.

[0221] Therefore, the content providing server 111 can supply the device start-up instruction information at an earlier timing. Therefore, the terminal device 112 (the secondary media output device control unit 414 thereof) can start up the secondary media output device 123 at an earlier timing based on the device start-up instruction information. Therefore, the content providing server 111 can further suppress the occurrence of secondary media output delays due to delays in device start-up.

[0222] <Terminal Device> In this case, the functional block configuration of the terminal device 112 may be the same as that in the first embodiment (FIG. 8).

[0223] <Flow of Server Processing> An example of the flow of server processing executed by the content providing server 111 in this case will be described with reference to the flowchart of FIG.

[0224] In this case, the processes of steps S301 to S307 are executed in the same manner as the processes of steps S151 to S157 in FIG.

[0225] In step S308, the synchronization advance processing unit 541 analyzes the synchronized playback of the sub-media in the 3D scene.

[0226] In step S309, the synchronization advance processing unit 541 determines, based on the analysis result, whether or not there is a synchronized playback timing for the sub-media in the future. If it is determined that there is a synchronized playback timing, the process proceeds to step S310.

[0227] The processes of steps S310 to S313 are executed in the same manner as the processes of steps S160 to S163 in FIG.

[0228] If it is determined in step S313 that the playback of the content has ended and the server processing is to be ended, the server processing ends.

[0229] By executing each process in this manner, the content providing server 111 can further reduce the occurrence of delays in sub-media output due to delays in device startup.

[0230] <Flow of Terminal Processing> The terminal processing in this case may be executed in the same manner as in the first embodiment (FIG. 10).

[0231] 6. Fourth Embodiment Interactive Playback of 2D Scene Sub-Media Note that, in the first to third embodiments, the scene processing unit 311 has been described as constructing a 3D scene, but the content scene may be 2D (flat or curved). In the following embodiments, a case will be described in which the content scene is a 2D scene.

[0232] <Content Providing Server> Fig. 20 is a functional block diagram showing an example of functions of the content providing server 111 according to the fourth embodiment. In this case, a scene processing unit 311 constructs a 2D scene.

[0233] Therefore, in this case, the media processing unit 312 having information about the 3D scene, as shown in the configuration example of FIG. 3, is omitted. Furthermore, since this is a 2D scene, information about the user's viewing position and direction is unnecessary, and thus acquisition of this information is omitted. Furthermore, in this case, the main media corresponding to the 2D scene is stored in the main media supply unit 313. Therefore, the main media supply unit 313 simply supplies the main media to the terminal device 112. However, if the main media in a 2D scene needs to be updated, the scene processing unit 311 acquires the main media from the main media supply unit 313, updates it, and returns the updated main media to the main media supply unit 313. The main media supply unit 313 then supplies the updated main media to the terminal device 112. In this case, the sub-media supply unit 314 corresponds to the 2D scene, is generated based on interaction events occurring in the 2D scene, and is supplied to the terminal device 112.

[0234] Other than that, it is the same as the example in FIG. 3 (first embodiment).

[0235] That is, in the first information processing device, the main medium and the sub-medium may be media corresponding to 2D scenes of the content.

[0236] In this way, even if the content scene is a 2D scene, the content providing server 111 can suppress the occurrence of a delay in sub-media output due to a delay in device startup, just as in the case of a 3D scene.

[0237] 21 is a functional block diagram showing an example of functions of the terminal device 112 according to the fourth embodiment. In this case, the main media and sub-media supplied to the terminal device 112 are media corresponding to 2D scenes of the content. Therefore, in this case, the viewing position direction information supply unit 412 shown in FIG. 4 is omitted.

[0238] Other than that, it is the same as the example in FIG. 4 (first embodiment).

[0239] That is, in the second information processing device, the main medium and the sub-medium may be media corresponding to 2D scenes of the content.

[0240] In this way, even if the content scene is a 2D scene, the terminal device 112 can suppress the occurrence of a delay in sub-media output due to a delay in device startup, just as in the case of a 3D scene.

[0241] <Flow of Server Processing> An example of the flow of server processing executed by the content providing server 111 in this case will be described with reference to the flowchart of FIG.

[0242] When the server processing starts, the scene processing unit 311 acquires device start-up time information and device characteristic information of the secondary media output device 123 in step S331.

[0243] In step S332, the scene processing unit 311 acquires operation information.

[0244] The processes from step S333 to step S335 and the processes from step S336 to step S341 are executed in parallel.

[0245] In step S333, the scene processing unit 311 acquires the main media.

[0246] In step S334, the scene processing unit 311 constructs a 2D scene in accordance with the operation information, and updates the main media as necessary.

[0247] In step S335, the main media supply unit 313 supplies the main media to the terminal device 112, which is a client of the content providing server 111. When the process of step S335 ends, the process proceeds to step S342.

[0248] In addition, in step S336, the interaction processing unit 321 analyzes the interaction in the 2D scene.

[0249] In step S337, the interaction processing unit 321 determines whether the analysis results satisfy the interaction condition. That is, the interaction processing unit 321 determines whether an interaction event has occurred. If it is determined that the interaction condition has been satisfied (that is, an interaction event has occurred), the process proceeds to step S338.

[0250] In step S338, the device startup control unit 322 supplies device startup instruction information for the secondary media output device at a timing that takes the startup time into consideration.

[0251] In step S339, the sub-media supply unit 314 generates sub-media corresponding to the 2D scene and the interaction event that occurred therein.

[0252] In step S 340 , the device characteristics conversion processing unit 341 converts the device characteristics of the sub-media to match the device characteristics of the sub-media output device 123 based on the device characteristics information of the sub-media output device 123 .

[0253] In step S341, the sub-media supply unit 314 supplies the sub-media to the terminal device 112. When the processing of step S341 ends, the processing proceeds to step S342. Also, if it is determined in step S337 that the interaction condition is not satisfied (i.e., an interaction event has not occurred), the processing proceeds to step S342.

[0254] In step S342, the scene processing unit 311 determines whether or not to end the server processing. If content is being played back and it is determined not to end the server processing, the process returns to step S332, and the subsequent processes are executed. Also, if content playback has ended and it is determined in step S342 that the server processing should be ended, the server processing ends.

[0255] By executing each process in this manner, the content providing server 111 can suppress the occurrence of a delay in sub-media output due to a delay in device startup.

[0256] <Flow of Terminal Processing> An example of the flow of terminal processing executed by the terminal device 112 in this case will be described with reference to the flowchart of FIG.

[0257] When terminal processing is started, in step S361, the device start-up time information supply unit 415 supplies the device start-up time information of the secondary media output device 123 to the content providing server 111. The device characteristic information supply unit 416 supplies the device characteristic information of the secondary media output device 123 to the content providing server 111.

[0258] The processes from step S362 to step S364 and the processes from step S365 to step S368 are executed in parallel.

[0259] In step S 362 , the operation information supply unit 411 supplies the operation information acquired from the input device 121 to the content providing server 111 .

[0260] In step S363 , the main media output device control unit 413 acquires the main media supplied from the content providing server 111 .

[0261] In step S364, the main media output device control unit 413 supplies and outputs the main media to the main media output device 122. When the process of step S364 ends, the process proceeds to step S369.

[0262] Also, in step S365, the secondary media output device control section 414 determines whether or not it has acquired device startup instruction information supplied from the content providing server 111. If it is determined that it has been acquired, the processing proceeds to step S366.

[0263] In step S366, the secondary media output device control section 414 starts up the secondary media output device 123 in time for the device usage timing in accordance with the device start-up instruction information.

[0264] In step S 367 , the secondary media output device control unit 414 acquires the secondary media supplied from the content providing server 111 .

[0265] In step S368, the sub-media output device control unit 414 supplies the sub-media to the sub-media output device 123 and outputs it. When the processing of step S368 ends, the processing proceeds to step S369. Also, if it is determined in step S365 that device startup instruction information has not been acquired, the processing proceeds to step S369.

[0266] In step S369, the secondary media output device control unit 414 determines whether or not to end the terminal processing. If content is being played and it is determined not to end the terminal processing, the process returns to step S362 and step S365, and the subsequent processes are executed. Also, if it is determined in step S369 that content playback has ended and the terminal processing is to end, the terminal processing ends.

[0267] By executing each process in this manner, the terminal device 112 can suppress the occurrence of a delay in sub-media output due to a delay in device startup.

[0268] <Content Providing Server> In the fourth embodiment, too, the device characteristics conversion may be performed in the terminal device 112. This case will be described below.

[0269] Fig. 24 is a functional block diagram showing an example of functions possessed by the content providing server 111. As shown in Fig. 24, in this case, the sub-media supply unit 314 does not have a device characteristics conversion processing unit 341. In other words, the sub-media supply unit 314 supplies the generated sub-media to the terminal device 112 without converting the device characteristics (leaving it in correspondence with the characteristics of the reference device).

[0270] In this case, it is not necessary to supply device characteristic information because no conversion of device characteristics is performed in the content providing server 111. Therefore, the process of transmitting device characteristic information from the terminal device 112 to the content providing server 111 described above in the fourth embodiment is omitted.

[0271] Other than that, this fourth embodiment is the same as the example described above (FIG. 20).

[0272] <Terminal Device> FIG. 25 is a functional block diagram illustrating an example of the functions of the terminal device 112 in this case. As shown in FIG. 25, in this case, the sub-media output device control unit 414 has a device characteristics conversion processing unit 341. That is, the sub-media output device control unit 414 may perform processing related to sub-media characteristics conversion, similar to the sub-media supply unit 314 ( FIG. 20 ) in the example described above in this fourth embodiment. As in the example described above, the device characteristics conversion processing unit 341 may convert the characteristics of the sub-media supplied from the content providing server 111 based on the device characteristics information of the sub-media output device 123 (i.e., to match the device characteristics indicated by the device characteristics information). The sub-media output device control unit 414 may supply the sub-media to the sub-media output device 123.

[0273] In this case, the content providing server 111 does not convert the device characteristics and there is no need to supply device characteristic information, so the process of transmitting device characteristic information from the terminal device 112 to the content providing server 111 described above in the fourth embodiment is omitted. That is, in this case, the device characteristic information supply unit 416 may be omitted.

[0274] Other than that, this fourth embodiment is the same as the example described above (FIG. 21).

[0275] In other words, in this case as well, the content providing server 111 and the terminal device 112 can suppress the occurrence of a delay in sub-media output due to a delay in device startup.

[0276] <Flow of Server Processing> An example of the flow of server processing executed by the content providing server 111 in this case will be described with reference to the flowchart of FIG.

[0277] When the server processing starts, the scene processing unit 311 acquires device startup time information of the secondary media output device 123 in step S381.

[0278] In step S382, the scene processing unit 311 acquires operation information.

[0279] The processes from step S383 to step S385 and the processes from step S386 to step S390 are executed in parallel.

[0280] In step S383, the scene processing unit 311 acquires the main media.

[0281] In step S384, the scene processing unit 311 constructs a 2D scene in accordance with the operation information, and updates the main media as necessary.

[0282] In step S385, the main media supply unit 313 supplies the main media to the terminal device 112, which is a client of the content providing server 111. When the process of step S385 ends, the process proceeds to step S391.

[0283] In addition, in step S386, the interaction processing unit 321 analyzes the interaction in the 2D scene.

[0284] In step S387, the interaction processing unit 321 determines whether the analysis results satisfy the interaction condition. That is, the interaction processing unit 321 determines whether an interaction event has occurred. If it is determined that the interaction condition has been satisfied (that is, an interaction event has occurred), the process proceeds to step S388.

[0285] In step S388, the device startup control unit 322 supplies device startup instruction information for the secondary media output device at a timing that takes the startup time into consideration.

[0286] In step S389, the sub-media supply unit 314 generates sub-media corresponding to the 2D scene and the interaction event that occurred there.

[0287] In step S390, the sub-media supply unit 314 supplies the sub-media to the terminal device 112. When the processing of step S390 ends, the processing proceeds to step S391. Also, if it is determined in step S387 that the interaction condition is not satisfied (i.e., an interaction event has not occurred), the processing proceeds to step S391.

[0288] In step S391, the scene processing unit 311 determines whether or not to end the server processing. If content is being played back and it is determined not to end the server processing, the process returns to step S382, and the subsequent processes are executed. Also, if content playback has ended and it is determined in step S391 that the server processing should be ended, the server processing ends.

[0289] By executing each process in this manner, the content providing server 111 can suppress the occurrence of a delay in sub-media output due to a delay in device startup.

[0290] <Flow of Terminal Processing> An example of the flow of terminal processing executed by the terminal device 112 in this case will be described with reference to the flowchart of FIG.

[0291] When the terminal processing is started, in step S 411 , the device start-up time information supply unit 415 supplies the device start-up time information of the secondary media output device 123 to the content providing server 111 .

[0292] The processes from step S412 to step S414 and the processes from step S415 to step S419 are executed in parallel.

[0293] In step S412 , the operation information supply unit 411 supplies the operation information acquired from the input device 121 to the content providing server 111 .

[0294] In step S 413 , the main media output device control unit 413 acquires the main media supplied from the content providing server 111 .

[0295] In step S414, the main media output device control unit 413 supplies the main media to the main media output device 122 for output. When the process of step S414 ends, the process proceeds to step S420.

[0296] Also, in step S415, the secondary media output device control unit 414 determines whether or not it has acquired device startup instruction information supplied from the content providing server 111. If it is determined that it has been acquired, the process proceeds to step S416.

[0297] In step S416, the secondary media output device control unit 414 starts up the secondary media output device 123 in time for the device usage timing in accordance with the device start-up instruction information.

[0298] In step S417 , the sub-media output device control unit 414 acquires sub-media corresponding to the 2D scene supplied from the content providing server 111 .

[0299] In step S 418 , the device characteristics conversion processing unit 341 converts the sub-media to match the device characteristics of the sub-media output device 123 .

[0300] In step S419, the sub-media output device control unit 414 supplies the sub-media to the sub-media output device 123 and outputs it. When the processing of step S419 ends, the processing proceeds to step S420. Also, if it is determined in step S415 that device startup instruction information has not been acquired, the processing proceeds to step S420.

[0301] In step S420, the secondary media output device control unit 414 determines whether or not to end the terminal processing. If content is being played and it is determined not to end the terminal processing, the process returns to step S412 and step S415, and the subsequent processes are executed. Also, if it is determined in step S420 that content playback has ended and the terminal processing is to end, the terminal processing ends.

[0302] By executing each process in this manner, the terminal device 112 can suppress the occurrence of a delay in sub-media output due to a delay in device startup.

[0303] <7. Fifth embodiment> <Prediction of event occurrence> In the case of a 2D scene, as in the second embodiment, an interaction with a haptic device may be predicted, and information to activate the haptic device may be sent to the terminal before the interaction occurs.

[0304] <Content Providing Server> FIG. 28 is a functional block diagram schematically illustrating an example of functions possessed by the content providing server 111 according to the fifth embodiment. As shown in FIG. 28, in this case, compared to the example of FIG. 20, the scene processing unit 311 further includes an interaction prediction unit 521. This interaction prediction unit 521 is similar to the example of FIG. 11. That is, the interaction prediction unit 521 predicts whether or not an interaction event will occur in the future in a 2D scene and supplies the prediction result to the interaction processing unit 321. The interaction processing unit 321 controls the device startup control unit 322 based on the prediction result. That is, the device startup control unit 322 may supply device startup instruction information to the terminal device 112, instructing the startup of the secondary media output device 123 that outputs the secondary media corresponding to the interaction event predicted by the interaction prediction unit 521.

[0305] Other than that, it is the same as the example in FIG. 20 (fourth embodiment).

[0306] In this way, the content providing server 111 can supply the device start-up instruction information at an earlier timing. Therefore, the terminal device 112 (the secondary media output device control unit 414 thereof) can start up the secondary media output device 123 at an earlier timing based on the device start-up instruction information. Therefore, the content providing server 111 can further suppress the occurrence of secondary media output delays due to delays in device start-up.

[0307] <Terminal Device> In this case, the functional block configuration of the terminal device 112 may be the same as that in the first embodiment (FIG. 21).

[0308] <Flow of Server Processing> An example of the flow of server processing executed by the content providing server 111 in this case will be described with reference to the flowchart of FIG.

[0309] In this case, the processes of steps S441 to S445 are executed in the same manner as the processes of steps S331 to S335 in FIG.

[0310] In step S446, the interaction prediction unit 521 predicts and analyzes interactions in the 2D scene.

[0311] In step S447, the interaction prediction unit 521 determines whether the interaction prediction condition is satisfied based on the analysis result. That is, the interaction prediction unit 521 determines whether the occurrence of an interaction event is predicted. If it is determined that the interaction prediction condition is satisfied (that is, the occurrence of an interaction event is predicted), the process proceeds to step S448.

[0312] The processes of steps S448 to S452 are executed in the same manner as the processes of steps S338 to S342 in FIG.

[0313] If it is determined in step S342 that the playback of the content has ended and the server processing is to be ended, the server processing ends.

[0314] By executing each process in this manner, the content providing server 111 can further reduce the occurrence of delays in sub-media output due to delays in device startup.

[0315] <Flow of Terminal Processing> The terminal processing in this case may be executed in the same manner as in the fourth embodiment (FIG. 23).

[0316] <Content Providing Server> In the fifth embodiment, similarly to the fourth embodiment, the device characteristic conversion of the sub-media may be executed in the terminal device 112 .

[0317] In this case, the content providing server 111 has the functional blocks shown in Fig. 30. That is, the content providing server 111 in this case has an interaction prediction unit 521 in addition to the configuration of the example in Fig. 24. This interaction prediction unit 521 is as described above with reference to Fig. 28.

[0318] Therefore, the content providing server 111 can supply the device start-up instruction information at an earlier timing. Therefore, the terminal device 112 (the secondary media output device control unit 414 thereof) can start up the secondary media output device 123 at an earlier timing based on the device start-up instruction information. Therefore, the content providing server 111 can further suppress the occurrence of secondary media output delays due to delays in device start-up.

[0319] <Terminal Device> In this case, the functional block configuration of the terminal device 112 may be the same as that in the fourth embodiment (FIG. 25).

[0320] <Flow of Server Processing> An example of the flow of server processing executed by the content providing server 111 in this case will be described with reference to the flowchart of FIG.

[0321] In this case, the processes of steps S471 to S475 are executed in the same manner as the processes of steps S381 to S385 in FIG.

[0322] In step S476, the interaction prediction unit 521 predicts and analyzes interactions in the 2D scene.

[0323] In step S477, the interaction prediction unit 521 determines whether the interaction prediction condition is satisfied based on the analysis result. That is, the interaction prediction unit 521 determines whether the occurrence of an interaction event is predicted. If it is determined that the interaction prediction condition is satisfied (that is, the occurrence of an interaction event is predicted), the process proceeds to step S478.

[0324] The processes of steps S478 to S481 are executed in the same manner as the processes of steps S388 to S391 in FIG.

[0325] If it is determined in step S481 that the playback of the content has ended and the server processing is to be ended, the server processing ends.

[0326] By executing each process in this manner, the content providing server 111 can further reduce the occurrence of delays in sub-media output due to delays in device startup.

[0327] <Flow of Terminal Processing> The terminal processing in this case may be executed in the same manner as in the first embodiment (FIG. 27).

[0328] <Prediction of Occurrence Probability> In the case of 2D scenes, as in the case of 3D scenes, when predicting the occurrence of an interaction event, the probability of a future interaction event occurring may be predicted, and the presence or absence of an interaction event may be predicted based on that occurrence probability.

[0329] In this way, the occurrence of an interaction event can be predicted more accurately.

[0330] 8. Sixth Embodiment Synchronized Playback of Sub-Media In the fourth and fifth embodiments, the sub-media was described as being interactively played back, but in the case of a 2D scene, the playback method of the sub-media may also be synchronized playback, in which the sub-media is played back in synchronization with the playback of the main media. In this sixth embodiment, a case in which the sub-media is played back in synchronization will be described. In other words, the sub-media is assumed to be synchronized media.

[0331] <Content Providing Server> Fig. 32 is a functional block diagram showing an example of functions of the content providing server 111 according to the sixth embodiment. As shown in Fig. 32, in this case, the scene processing unit 311 has a synchronization-advance processing unit 541 instead of the interaction processing unit 321.

[0332] The synchronization advance processing unit 541 is the same as the example described in the third embodiment. The synchronization advance processing unit 541 executes processing related to synchronized playback of the sub-media. In this case, the synchronization advance processing unit 541 may analyze the processing prior to the actual synchronized playback of the sub-media, and detect the future playback timing of the sub-media. Based on the results of this advance analysis, the device startup control unit 322 may supply device startup instruction information to the terminal device 112, instructing the startup of the sub-media output device 123 that will output the sub-media to be played in the future.

[0333] Other than that, it is the same as the example in FIG. 20 (fourth embodiment).

[0334] In this way, the content providing server 111 can supply the device start-up instruction information at an earlier timing. Therefore, the terminal device 112 (the secondary media output device control unit 414 thereof) can start up the secondary media output device 123 at an earlier timing based on the device start-up instruction information. Therefore, the content providing server 111 can further suppress the occurrence of secondary media output delays due to delays in device start-up.

[0335] <Terminal Device> In this case, the functional block configuration of the terminal device 112 may be the same as that in the fourth embodiment (FIG. 21).

[0336] <Flow of Server Processing> An example of the flow of server processing executed by the content providing server 111 in this case will be described with reference to the flowchart of FIG.

[0337] In this case, the processes of steps S501 to S505 are executed in the same manner as the processes of steps S331 to S335 in FIG.

[0338] In step S506, the synchronization advance processing unit 541 analyzes the synchronized playback of the sub-media in the 2D scene.

[0339] In step S507, the synchronization advance processing unit 541 determines, based on the analysis result, whether or not there is a synchronized playback timing for the sub-media in the future. If it is determined that there is a synchronized playback timing, the process proceeds to step S508.

[0340] The processes of steps S508 to S512 are executed in the same manner as the processes of steps S338 to S342 in FIG.

[0341] If it is determined in step S284 that the playback of the content has ended and the server processing is to be ended, the server processing ends.

[0342] By executing each process in this manner, the content providing server 111 can further reduce the occurrence of delays in sub-media output due to delays in device startup.

[0343] <Flow of Terminal Processing> The terminal processing in this case may be executed in the same manner as in the fourth embodiment (FIG. 23).

[0344] <Content Providing Server> In the sixth embodiment, similarly to the fourth embodiment, the device characteristic conversion of the sub-media may be executed in the terminal device 112 .

[0345] In this case, the content providing server 111 has the functional blocks shown in Fig. 34. That is, the content providing server 111 in this case has a synchronization-first processing unit 541 instead of the interaction processing unit 321 in the configuration of the example in Fig. 24. The synchronization-first processing unit 541 is as described above with reference to Fig. 32.

[0346] Therefore, the content providing server 111 can supply the device start-up instruction information at an earlier timing. Therefore, the terminal device 112 (the secondary media output device control unit 414 thereof) can start up the secondary media output device 123 at an earlier timing based on the device start-up instruction information. Therefore, the content providing server 111 can further suppress the occurrence of secondary media output delays due to delays in device start-up.

[0347] <Terminal Device> In this case, the functional block configuration of the terminal device 112 may be the same as that in the fourth embodiment (FIG. 25).

[0348] <Flow of Server Processing> An example of the flow of server processing executed by the content providing server 111 in this case will be described with reference to the flowchart of FIG.

[0349] In this case, the processes of steps S531 to S535 are executed in the same manner as the processes of steps S381 to S385 in FIG.

[0350] In step S536, the synchronization advance processing unit 541 analyzes the synchronized playback of the sub-media in the 2D scene.

[0351] In step S537, the synchronization advance processing unit 541 determines, based on the analysis result, whether or not there is a synchronized playback timing for the sub-media in the future. If it is determined that there is a synchronized playback timing, the process proceeds to step S538.

[0352] The processes of steps S538 to S541 are executed in the same manner as the processes of steps S388 to S391 in FIG.

[0353] If it is determined in step S541 that the playback of the content has ended and the server processing is to be ended, the server processing ends.

[0354] By executing each process in this manner, the content providing server 111 can further reduce the occurrence of delays in sub-media output due to delays in device startup.

[0355] <Flow of Terminal Processing> The terminal processing in this case may be executed in the same manner as in the fourth embodiment (FIG. 27).

[0356] 9. Supplementary Notes Computer The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term computer includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0357] 2 (the content providing server 111 or the terminal device 112), the CPU 201 loads a program stored in the storage unit 213, for example, into the RAM 203 via the input / output interface 210 and the bus 204, and executes the program, thereby performing the above-described series of processes. The RAM 203 also stores data necessary for the CPU 201 to execute various processes, as appropriate.

[0358] The program executed by the computer can be applied by recording it on, for example, removable media 221 such as package media. In this case, by inserting the removable media 221 into the drive 215, the program can be installed in the storage unit 213 via the input / output interface 210.

[0359] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 214 and installed in the storage unit 213.

[0360] Alternatively, this program can be installed in advance in the ROM 202 or the storage unit 213 .

[0361] When the series of processes is executed by hardware, the functional blocks of the content providing server 111 and the terminal device 112 described above may be realized as hardware.

[0362] <Application of the Present Technology> The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0363] Furthermore, for example, the present technology can be implemented as a part of a device, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, or a set (e.g., a video set) in which other functions are further added to a unit. In particular, the functional blocks of the content providing server 111 and the terminal device 112 described above in each figure may each be configured by an independent module or processor, or one or more modules or processors may be configured to perform the processing of multiple units.

[0364] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0365] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0366] <Fields and uses to which this technology can be applied> Systems, devices, processing units, etc. to which this technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, the uses thereof are also arbitrary.

[0367] <Others> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. That is, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in a bitstream, but also in a bitstream that includes differential information of the identification information relative to certain reference information. Therefore, in this specification, "flag" and "identification information" encompass not only the information itself, but also differential information relative to the reference information.

[0368] Furthermore, various types of information (e.g., metadata) related to the coded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. That is, the associated pieces of data may be combined into one piece of data or may be separate pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Furthermore, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may refer not to the entire data, but to only part of the data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0369] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0370] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0371] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0372] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

[0373] Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0374] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0375] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0376] The present technology can also be configured as follows. (1) An information processing device including: a device startup control unit that controls startup of a secondary media output device, the secondary media output device being played in conjunction with playback of a main media that is the subject media of content, by supplying device startup instruction information to the secondary media output device or a control device that controls the secondary media output device, the device startup instruction information instructing startup of the secondary media output device at timing that takes into account a startup time of the secondary media output device. (2) The information processing device described in (1), wherein the timing that takes into account the startup time of the secondary media output device is timing that is a predetermined time before the timing at which the secondary media output device outputs the secondary media. (3) The information processing device described in (1) or (2), wherein the timing that takes into account the startup time of the secondary media output device is timing based on the startup time of the secondary media output device indicated by device startup time information. (4) The information processing device according to any one of (1) to (3), further comprising a sub-media supply unit that supplies the sub-media corresponding to a scene of the content to the sub-media output device or the control device, wherein the timing taking into account the startup time of the sub-media output device is a timing before the sub-media is supplied. (5) The information processing device according to (4), wherein the sub-media supply unit is configured to convert device characteristics of the sub-media to match characteristics of the sub-media output device and supply the converted sub-media, wherein the timing taking into account the startup time of the sub-media output device is a timing before the device characteristics of the sub-media are converted. (6) The information processing device according to (4) or (5), wherein the sub-media supply unit is configured to generate the sub-media corresponding to the scene and supply the generated sub-media, wherein the timing taking into account the startup time of the sub-media output device is a timing before the sub-media is generated.(7) The information processing device according to any one of (4) to (6), further comprising an interaction processing unit that analyzes an interaction event in the scene, wherein the timing taking into account the startup time of the secondary media output device is a timing before the secondary media is supplied based on the occurrence of the interaction event. (8) The information processing device according to (7), further comprising an interaction prediction unit that predicts the occurrence of the interaction event in the scene, wherein the timing taking into account the startup time of the secondary media output device is a timing based on the predicted occurrence timing of the interaction event. (9) The information processing device according to (8), wherein the interaction prediction unit derives a probability of the interaction event occurring and predicts the occurrence of the interaction event based on the derived occurrence probability. (10) The information processing device according to any one of (4) to (9), further comprising a synchronization advance processing unit that analyzes in advance the synchronous playback of the secondary media with the main media in the scene, wherein the timing taking into account the startup time of the secondary media output device is a timing before the secondary media to be synchronously played is supplied. (11) The information processing device according to any one of (1) to (10), further comprising a main media supply unit that supplies the main media corresponding to a scene of the content to a main media output device that outputs the main media or a control device that controls the main media output device, wherein the timing taking into account a startup time of the sub-media output device is a timing before the main media is supplied. (12) The information processing device according to (11), wherein the main media supply unit is configured to generate the main media corresponding to the scene and supply the generated main media, wherein the timing taking into account a startup time of the sub-media output device is a timing before the main media is generated.(13) The information processing device according to any one of (1) to (12), wherein the main media and the secondary media are media corresponding to 3D scenes of the content. (14) The information processing device according to any one of (1) to (13), wherein the main media and the secondary media are media corresponding to 2D scenes of the content. (15) An information processing method including: controlling startup of a secondary media output device by supplying device startup instruction information to the secondary media output device or a control device controlling the secondary media output device, at timing taking into account startup time of the secondary media output device that outputs secondary media played in conjunction with playback of the main media, which is the primary media of the content. (16) A program for causing a computer to execute processing including: controlling startup of the secondary media output device by supplying device startup instruction information to the secondary media output device or a control device controlling the secondary media output device, at timing taking into account startup time of the secondary media output device that outputs secondary media played in conjunction with playback of the main media, which is the primary media of the content.

[0377] (21) An information processing device comprising: a sub-media output device control unit that starts up a sub-media output device based on device start-up instruction information that is supplied at a timing that takes into account a start-up time of the sub-media output device that outputs sub-media that is played in conjunction with the playback of main media, which is the subject media of the content. (22) The information processing device according to (21), wherein the timing that takes into account the start-up time of the sub-media output device is a timing that is a predetermined time before the timing at which the sub-media output device outputs the sub-media. (23) The information processing device according to (21) or (22), further comprising: a device start-up time information supply unit that supplies device start-up time information indicating the start-up time of the sub-media output device, wherein the timing that takes into account the start-up time of the sub-media output device is a timing based on the start-up time of the sub-media output device indicated by the device start-up time information. (24) The information processing device according to any of (21) to (23), wherein the sub-media output device control unit further acquires the supplied sub-media and causes the acquired sub-media to be output by the sub-media output device, wherein the timing that takes into account the start-up time of the sub-media output device is a timing before the sub-media is acquired. (25) The information processing device according to (24), wherein the secondary media output device control unit is configured to convert the acquired device characteristics of the secondary media to match the characteristics of the secondary media output device and output the converted secondary media to the secondary media output device, and the timing taking into account the startup time of the secondary media output device is a timing before the device characteristics of the secondary media are converted.(26) The information processing device according to any one of (21) to (25), further comprising a main media output device control unit that acquires the main media corresponding to a scene of the content and causes a main media output device to output the acquired main media, wherein the timing considering the startup time of the sub-media output device is a timing before the main media is acquired. (27) The information processing device according to any one of (21) to (26), wherein the main media and the sub-media are media corresponding to a 3D scene of the content. (28) The information processing device according to any one of (21) to (27), wherein the main media and the sub-media are media corresponding to a 2D scene of the content. (29) An information processing method including: starting up the sub-media output device based on device startup instruction information supplied at a timing considering the startup time of a sub-media output device that outputs sub-media played in conjunction with playback of the main media, which is the main media of the content. (30) A program for causing a computer to execute a process including starting up a secondary media output device that outputs secondary media that is played in conjunction with the playback of a main media, which is the subject media of the content, based on device start-up instruction information that is supplied at a timing that takes into account the start-up time of the secondary media output device.

[0378] 100 Content playback system, 110 Network, 111 Content providing server, 112 Terminal device, 121 Input device, 122 Main media output device, 123 Sub-media output device, 311 Scene processing unit, 312 Media processing unit, 313 Main media supply unit, 314 Sub-media supply unit, 321 Interaction processing unit, 322 Device startup unit, 331 3D scene data, 332 Main media related data, 333 Sub-media data, 341 Device characteristics conversion processing unit, 411 Operation information supply unit, 412 Viewing position direction information supply unit, 413 Main media output device control unit, 414 Sub-media output device control unit, 415 Device startup time information supply unit, 416 Device characteristics information supply unit, 521 Interaction prediction unit, 541 Synchronization advance processing unit

Claims

1. An information processing device having a device startup control unit that controls the startup of a secondary media output device by supplying device startup instruction information that instructs the startup of a secondary media output device to the secondary media output device or a control device that controls the secondary media output device, at a timing that takes into account the startup time of the secondary media output device that outputs secondary media that is played in conjunction with the playback of main media, which is the main media of the content.

2. The information processing device according to claim 1, wherein the timing taking into consideration the startup time of the secondary media output device is a timing that is a predetermined time before the timing at which the secondary media output device outputs the secondary media.

3. The information processing device according to claim 1, wherein the timing taking into consideration the startup time of the secondary media output device is timing based on the startup time of the secondary media output device indicated by device startup time information.

4. An information processing device as described in claim 1, further comprising a sub-media supply unit that supplies the sub-media corresponding to a scene of the content to the sub-media output device or the control device, and the timing taking into account the start-up time of the sub-media output device is a timing before the sub-media is supplied.

5. The information processing device of claim 4, wherein the sub-media supply unit is configured to convert the device characteristics of the sub-media to match the characteristics of the sub-media output device and supply the converted sub-media, and the timing taking into account the startup time of the sub-media output device is a timing before the device characteristics of the sub-media are converted.

6. The information processing device of claim 4, wherein the sub-media supply unit is configured to generate the sub-media corresponding to the scene and supply the generated sub-media, and the timing taking into account the startup time of the sub-media output device is a timing before the sub-media is generated.

7. An information processing device as described in claim 4, further comprising an interaction processing unit that analyzes an interaction event in the scene, and the timing taking into account the startup time of the sub-media output device is a timing before the sub-media is supplied based on the occurrence of the interaction event.

8. The information processing device according to claim 7, further comprising an interaction prediction unit that predicts the occurrence of the interaction event in the scene, and the timing taking into account the startup time of the secondary media output device is based on the predicted occurrence timing of the interaction event.

9. The information processing device according to claim 8, wherein the interaction prediction unit derives an occurrence probability of the interaction event, and predicts the occurrence of the interaction event based on the derived occurrence probability.

10. An information processing device as described in claim 4, further comprising a synchronization advance processing unit that analyzes in advance the synchronized playback of the sub-media with the main media in the scene, and the timing taking into account the startup time of the sub-media output device is a timing before the sub-media to be synchronized and played is supplied.

11. An information processing device as described in claim 1, further comprising a main media supply unit that supplies the main media corresponding to a scene of the content to a main media output device that outputs the main media or a control device that controls the main media output device, and the timing taking into account the startup time of the sub-media output device is a timing before the main media is supplied.

12. The information processing device described in claim 11, wherein the main media supply unit is configured to generate the main media corresponding to the scene and supply the generated main media, and the timing taking into account the startup time of the sub-media output device is a timing before the main media is generated.

13. The information processing device according to claim 1, wherein the main media and the sub-media are media corresponding to 3D scenes of the content.

14. The information processing device according to claim 1, wherein the main media and the sub-media are media corresponding to 2D scenes of the content.

15. An information processing method including controlling the startup of a sub-media output device that outputs a sub-media that is played in conjunction with the playback of a main media, which is the main media of the content, by supplying device startup instruction information that instructs the startup of the sub-media output device to the sub-media output device or a control device that controls the sub-media output device, at a timing that takes into account the startup time of the sub-media output device.

16. An information processing device having a sub-media output device control unit that starts up a sub-media output device based on device start-up instruction information supplied at a timing that takes into account the start-up time of the sub-media output device, which outputs sub-media that is played in conjunction with the playback of the main media, which is the main media of the content.

17. The information processing device according to claim 16, wherein the timing taking into consideration the startup time of the secondary media output device is a timing that is a predetermined time before the timing at which the secondary media output device outputs the secondary media.

18. An information processing device as described in claim 16, further comprising a device startup time information supply unit that supplies device startup time information indicating the startup time of the secondary media output device, and the timing taking into account the startup time of the secondary media output device is timing based on the startup time of the secondary media output device indicated by the device startup time information.

19. The information processing device of claim 16, wherein the sub-media output device control unit further acquires the supplied sub-media and causes the acquired sub-media to be output by the sub-media output device, and the timing taking into account the start-up time of the sub-media output device is a timing before the sub-media is acquired.

20. An information processing method including starting up a sub-media output device that outputs a sub-media that is played in conjunction with the playback of a main media, which is the main media of the content, based on device start-up instruction information supplied at a timing that takes into account the start-up time of the sub-media output device.

Images