Information processing device, information processing method, viewing system

Abstract

This enables a more immersive content viewing experience. [Solution] The control unit of the information processing device performs a determination process to determine which effects correspond to the scenes in the visual content and target the effect targets, based on content information including scene attribute information for some or all of the scenes in the visual content and object information including scene attribute information for objects that are real in the viewing space and are targeted for the effect, and a control process to execute the effect in accordance with the corresponding scene in the visual content based on the determination of the determination process.

Description

Technical Field

[0001] The present technology relates to a viewing system, an information processing apparatus included in the viewing system, and an information processing method thereof.

Background Art

[0002] Various forms of viewing systems using different devices exist for playing back moving image and still image content. For example, systems using a monitor device, a screen, or a large panel as a display device, and systems using a head-mounted display (hereinafter referred to as "HMD") are known. In particular, viewing systems using an HMD are suitable for enhancing the user's sense of presence and immersion in the world of the content. Patent Document 1 listed below discloses a technology for superimposing and displaying an effect image matching a scene on a content playback image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Incidentally, some viewing systems that use an HMD are configured to allow the user to visually recognize the surrounding viewing space in addition to the content playback image. This is a so-called video see-through scheme, which allows the user to visually recognize the surroundings while viewing the content playback image. Note that video see-through includes schemes in which an image of the surroundings captured by a camera attached to the HMD is combined with a content image for display, and schemes in which the transmittance of the HMD screen is controlled to allow the user to actually see the surroundings.

[0005] Even when viewing content while looking at the actual viewing space using such a video see-through scheme, a method for further enhancing the sense of immersion is desired. Therefore, this disclosure aims to provide users with a highly immersive experience in a system that allows them to view content using a video see-through method. [Means for solving the problem]

[0006] The information processing device relating to this technology includes a control unit that performs a determination process to determine an effect that corresponds to a scene in the visual content and targets the target object, based on content information including scene attribute information for some or all scenes in the visual content and object information including scene attribute information for an object that is the target of the effect among the objects that actually exist in the viewing space; and a control process that executes the effect in accordance with the corresponding scene in the visual content based on the determination of the determination process. This allows effects using real-world objects in the viewing space to be executed in accordance with the scene while viewing visual content. [Brief explanation of the drawing]

[0007] [Figure 1] This is an explanatory diagram of an example system configuration of an embodiment of this technology. [Figure 2] This is an explanatory diagram of HMDs and AR glasses. [Figure 3] This is an explanatory diagram of the view that the user sees in the embodiment. [Figure 4] This is an explanatory diagram of another system configuration example of the embodiment. [Figure 5] This is an explanatory diagram of the system components of the embodiment. [Figure 6] This is a block diagram of the HMD according to the embodiment. [Figure 7] This is a block diagram of the functions of the image processing unit and control unit of the HMD according to the embodiment. [Figure 8] This is an explanatory diagram of the content information definition table of the embodiment. [Figure 9] This is an explanatory diagram of the object information definition table of the embodiment. [Figure 10]This is a flowchart of basic processing of the embodiment. [Figure 11] This is an explanatory diagram of a definition table in the first example of the first embodiment. [Figure 12] This is an explanatory diagram of the effect of the first example of the first embodiment. [Figure 13] This is an explanatory diagram of a definition table in the second example of the first embodiment. [Figure 14] This is an explanatory diagram of the effect of the second example of the first embodiment. [Figure 15] This is a flowchart of a processing example of the first embodiment. [Figure 16] This is a flowchart of information acquisition processing of the first embodiment. [Figure 17] This is a flowchart of effect determination processing of the first embodiment. [Figure 18] This is an explanatory diagram of a definition table in the example of the second embodiment. [Figure 19] This is an explanatory diagram of the effect of the second embodiment. [Figure 20] This is an explanatory diagram of the effect of the second embodiment. [Figure 21] This is a flowchart of a processing example of the second embodiment. [Figure 22] This is a flowchart of a processing example of the second embodiment. [Figure 23] This is a flowchart of a processing example related to object information of the third embodiment. [Figure 24] This is an explanatory diagram of the effect of the fourth embodiment. [Figure 25] This is an explanatory diagram of the effect of the fourth embodiment. [Figure 26] This is a flowchart of a processing example of the fourth embodiment. [Figure 27] This is a flowchart of a processing example of an introduction effect of the fourth embodiment. [Description of Embodiments]

[0008] The embodiments will be described below in the following order. <1. Viewing System> <2. HMD Configuration> <3. Effect processing using content information and object information> <4. First Embodiment> <5. Second Embodiment> <6. Third Embodiment> <7. Fourth Embodiment> <8. Summary and Variations>

[0009] In this disclosure, "image" includes both video and still images. While the embodiments use viewing of content playback images as an example, the images are primarily intended to be videos. The images to be viewed may include audio, or they may be images only. In this embodiment, the content to be viewed is referred to as "visual content." Visual content is content that plays at least images, such as videos or still images, and may also include audio playback. Visual content is assumed to be content with a fixed storyline, such as movies, dramas, and music videos, but it may also be visual content with user interaction elements, such as games, VR (Virtual Reality), and AR (Augmented Reality).

[0010] <1. Viewing System> Figure 1 shows an example of the configuration of the viewing system according to the embodiment. The figure shows the HMD1 and the effect target object 20. As an example of the effect target object 20, the controlled device 2 and the target object 3 are shown.

[0011] An HMD is a device that a user wears on their head to view images, as shown in Figure 2. This HMD1 is a display device, but as shown in Figure 1, it incorporates a processor as a control unit 10 that performs calculation and control processing, and can be considered as an information processing device that performs effect control in the embodiment. An example of the configuration of the HMD1 including the control unit 10 will be described later in Figure 6.

[0012] In this embodiment, we will describe an example where HMD1 is used as the display device, but it is also possible to use AR glasses 5, shown in Figure 2, as the display device. AR glasses 5 are a display device that recognizes objects in real space and overlays virtual information onto them.

[0013] In Figure 1, Controlled Device 2 refers to a device that actually exists in the viewing space, can communicate with the control unit 10 in the HMD1, and can be controlled by the HMD1. For example, it refers to a device equipped with communication capabilities with external devices, such as IoT (Internet of Things) home appliances, that can perform some operation through control from an external device. Typically, Controlled Device 2 is assumed to be a home appliance that exists in the viewing space where the user views visual content. The viewing space is such as the user's room at home, and examples of Controlled Device 2 include lighting fixtures, lighting systems, curtain opening / closing systems, air conditioners, and fans installed in the room. Devices that can be controlled by the HMD1 and that are registered in the object information definition table described later as items to be used for effects are considered Controlled Device 2.

[0014] Target object 3 refers to an object that actually exists in the viewing space but whose movement cannot be controlled by the control unit 10 within the HMD1. Assuming a user's room at home, there are many objects whose movement cannot be controlled by the control unit 10. For example, vases, ornaments, tables, chairs, walls, wall decorations, etc. In Figure 1, among these objects, those that are registered in the object information definition table described later as objects that the HMD1 will use for effects are designated as target object 3.

[0015] In the viewing system shown in Figure 1, the user wearing the HMD1 will see a display image 40 as shown in Figure 3, which will be a video see-through. For example, a relatively large content image 41 is displayed in the center of the display image 40 screen. Surrounding the content image 41, a viewing space image 42, which represents the actual scenery of the viewing space, is displayed. For example, the HMD1 is equipped with an imaging device, and the HMD1 can display an image of the scenery in front of the user as a viewing space image 42 surrounding the content image 41.

[0016] In the example shown in Figure 3, the viewing space image 42 shows the controlled devices 2: a lighting fixture 21, curtains 22 opened and closed by a curtain opening / closing system, an air conditioner 23, and a fan 24. However, this is shown as an example of a viewing space, and the controlled devices 2 themselves do not necessarily need to be visible to the user. It is sufficient that the user can perceive changes in the viewing space through the operation of the controlled devices 2. For example, devices that change the brightness, room temperature, airflow, etc., of the viewing space would be considered controlled devices 2.

[0017] In the example in Figure 3, the vase 31 and window 32 are shown as target objects 3. These vase 31 and window 32 are objects that the user can see as part of the viewing space image 42. In other words, any object that can be seen as part of the viewing space image 42 is considered a target object 3. In the case of HMD1, the viewing space image 42 is determined by the orientation of the user's face, so objects that can be seen by turning to the right rear, left rear, behind, or above the user also fall under the category of objects that can be seen as part of the viewing space image 42 and can therefore be considered target objects 3.

[0018] The HMD1 displays visual content to the user as such a display image 40, and also performs effect control determined according to the scene of the visual content. For example, the control unit 10 of the HMD1 performs effect control such as controlling the controlled device 2, the lighting device 21, to change the brightness of the room, or controlling the curtain opening and closing system to open and close the curtains 22.

[0019] Furthermore, the control unit 10 of the HMD1 can apply image effects to the vase 31 in the captured image that appears in the viewing space image 42, for target objects 3 whose movement cannot be controlled, such as the vase 31. Although the controlled device 2 and the target object 3 are described separately as the effect target object 20, if the controlled device 2 appears in the viewing space image 42 as shown in Figure 3, it can also be treated as the target object 3 and used as the target of the image effect.

[0020] As described above, "using" the effect target object 20 for an effect includes causing the controlled device 2, which is the effect target object 20, to perform some action, or making the image of the target object 3 or the controlled device 2, which is the effect target object 20, the target of an image effect.

[0021] Figure 4 shows another example of the viewing system configuration, in which the HMD1 and the effect control device 4, which has a control unit 10, are configured as separate components. In this case, the HMD1 is used as a display device, and various effect controls are performed by a separate effect control device 4. This effect control device 4 can also be considered as an information processing device that performs the effect controls of the embodiment.

[0022] Figure 5 shows the components that perform effect control in the viewing system shown in Figure 1 or Figure 4. The components in Figure 5 are functions that are implemented by program in the control unit 10 in Figures 1 and 4. Figure 5 shows the information input unit 100, the judgment processing unit 200, the effect control unit 300, and the information management unit 400.

[0023] The information input unit 100 refers to a function that accepts information about the visual content viewed by the user and information about the user's viewing space as input. For example, it is envisioned to have a function that inputs information about the content through communication with an external device that is the source of the visual content, or through communication with an external server or external database via a network. It is also envisioned to have a function that obtains information about the viewing space as an image captured by an imaging device, or through various sensors such as an illuminance sensor or a temperature sensor. Furthermore, it is envisioned to have a function that inputs information about the controlled device 2, such as its type or operating function, as viewing space information through communication with the controlled device 2, such as an IoT home appliance. In other words, the information input unit 100 refers to a function that inputs information necessary for the control processing of this embodiment.

[0024] The information management unit 400 is a function that manages the information necessary for the judgment process related to effects, and includes an object information definition unit 401 and a content information definition unit 402. The object information definition unit 401 is a function that stores and reads information in the object information definition table 60, which will be described later in Figure 9. The object information definition table 60 stores object information, including scene attribute information, for objects that are real objects in the viewing space and have been designated as effect targets 20. In particular, the object information definition table 60 defines information about what kind of effect should be executed for the effect targets 20, under what conditions and in what scenes.

[0025] The content information definition unit 402 is a function that stores and retrieves information in the content information definition table 50, which will be described later in Figure 8. The content information definition table 50 registers scene attribute information for some or all of the scenes within the visual content.

[0026] The determination processing unit 200 has the function of determining whether or not effects can be executed according to the scene of the visual content that the user is viewing, and determining what kind of effects should be executed according to the scene. For example, the determination processing unit 200 makes a determination regarding effects for the visual content that the user is about to view or is currently viewing, using the scene information defined in the content information definition unit 402 and the information of the effect target object 20 defined in the object information definition unit 401. In other words, the determination processing unit 200 performs a determination process to determine whether an effect will be executed, and what type of effect to execute, corresponding to the scene of the visual content to be viewed, using the effect target object 20 as the processing target, based on the content information definition table 50 and the object information definition table 60. In this case, the determination processing unit 200 also determines which effect to execute by checking whether the effect target object 20 satisfies the effect generation conditions shown in the object information definition table 60.

[0027] The effect control unit 300 is a function that controls the execution of effects based on the determination of the determination processing unit 200. For example, the effect control unit 300 controls each part so that effect processing of images and sounds superimposed on the target object 3, which is a real object, on the display image 40 on the HMD1, and interactions such as dimming and blowing air using the controlled device 2 are executed as effects.

[0028] <2. HMD Configuration> For example, assuming the configuration shown in Figure 1, Figure 6 shows an example configuration of an HMD1 equipped with a control unit 10 having the functions shown in Figure 5.

[0029] Figure 6 shows the configuration of the HMD1, including a control unit 10, an image processing unit 11, an imaging unit 12, a sensor unit 13, a communication unit 14, a storage unit 15, a display unit 16, and an audio output unit 17. Note that HMD1 may have other configurations, or it does not need to have all of them.

[0030] The communication unit 14 comprehensively represents the device that performs communication with external devices. For example, it represents the components that perform network communication, wired communication, short-range wireless communication, etc. In this embodiment, the communication unit 14 communicates with devices such as a video player that serves as a content source, or a server device via a public network, and receives the visual content itself or information related to the visual content. The communication unit 14 transmits the received image signal V1 and accompanying audio signal A1 of the visual content to the image processing unit 11. The visual content signal may be received by the communication unit 14 as streaming data, or the communication unit 14 may download it, store it in the storage unit 15 or the like, and then read it out and supply it to the image processing unit 11. Alternatively, the HMD1 may also function as a media playback unit and play back the content signal from a storage medium such as a memory card.

[0031] The communication unit 14 also performs the process of transmitting control signals from the control unit 10 to an external device. For example, the communication unit 14 transmits the control signals generated by the control unit 10 to the controlled device 2. Furthermore, through communication with the controlled device 2 by the communication unit 14, the control unit 10 can also recognize the presence and operating functions of the controlled device 2 in the surrounding area.

[0032] The imaging unit 12 includes various lenses and an image sensor that constitute the incident optical system. The image sensor includes a light-receiving element, a gain processing circuit for the signal received by the light-receiving element and converted photoelectrically, and an A / D (Analog / Digital) conversion circuit for converting it into digital data.

[0033] The light-receiving element in an image sensor is configured as, for example, a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type. The image sensor performs processes such as CDS (Correlated Double Sampling) and AGC (Automatic Gain Control) on the electrical signal obtained by photoelectric conversion of the received light, and further performs A / D conversion processing. The captured image signal V2 as digital data is then transmitted to the image processing unit 11. The imaging unit 12 also has a microphone, which picks up ambient sound to obtain an audio signal A2 and transmits it to the image processing unit 11.

[0034] The image processing unit 11 primarily performs image processing for content viewing. Specifically, it performs necessary signal processing on the image signal V1 and audio signal A1 of the visual content, supplies the processed image signal V3 to the display unit 16, and supplies the audio signal A3 to the audio output unit 17. This enables image display on the display unit 16 and audio output on the audio output unit 17. The image processing unit 11 also performs development processing on the captured image signal V2 and combines it with the content image signal V1. The image signal V3, which is the result of combining the image signal V1 and the captured image signal V2, is supplied to the display unit 16, making it possible to display a video see-through image as shown in Figure 3. The image processing unit 11 can also perform effect processing on the captured image signal V2.

[0035] The display unit 16 consists of a display such as a liquid crystal display (LCD) or an electroluminescent (EL) panel and its display driving circuit.

[0036] The audio output unit 17 is formed, for example, as an earphone that can be worn on the user's ear in a headset that constitutes the HMD1.

[0037] The sensor unit 13 comprehensively represents the various sensors equipped in the HMD1. For example, it is equipped with an IMU (inertial measurement unit), which can detect angular velocity using three-axis angular velocity (gyro) sensors for pitch, yaw, and roll, and acceleration using an acceleration sensor. The sensor unit 13 can also be equipped with an illuminance sensor to detect the brightness of the viewing space. Furthermore, by equipping the sensor unit 13 with a position sensor, it becomes possible to determine the room used as the viewing space from the position information. In addition, by equipping the sensor unit 13 with a distance measuring sensor, it becomes possible to detect the distance from the HMD1 to an object. Various types of information detected by the sensor unit 13, such as illuminance information, IMU data, position information, and distance information, are supplied to the control unit 10.

[0038] The memory unit 15 stores information used by the processor, which acts as the control unit 10, for processing. This memory unit 15 includes, for example, ROM (Read Only Memory) 15a, RAM (Random Access Memory) 15b, and non-volatile memory 15c such as flash memory. The memory unit 15 may be a memory area built into the microcomputer chip acting as the control unit 10, or it may be composed of a separate memory chip.

[0039] RAM15b is used as a work area for various data processing operations of the control unit 10, and is used for temporary storage of data, programs, etc. ROM 15a and non-volatile memory 15c are used by the control unit 10 to store the OS (Operating System) for controlling each part, application programs for various operations, and various types of information. In this embodiment, the storage unit 15, for example, the non-volatile memory 15c, stores the object information definition table 60 and the content information definition table 50.

[0040] The control unit 10 is composed of a processor such as a CPU. The control unit 10 may be a GPU (Graphics Processing Unit), a GPGPU (General-purpose computing on graphics processing units), an AI (artificial intelligence) processor, or it may be composed of multiple processors.

[0041] The control unit 10 controls the HMD1 by executing programs stored in the ROM 15a and non-volatile memory 15c of the storage unit 15. Specifically, it controls processes such as display output, audio output, and communication processing. In addition, the control unit 10, through a software program, provides the functions described in Figure 5 to perform effect control using the effect target object 20 in accordance with the content display.

[0042] Figure 7 shows the image processing unit 11, the control unit 10, and the communication unit 14, focusing on effect control. The control unit 10 includes the information input unit 100, the determination processing unit 200, the effect control unit 300, and the information management unit 400, as explained in Figure 5.

[0043] The image processing unit 11 includes a content signal processing unit 11a, an imaging signal processing unit 11b, an effect processing unit 11c, and a synthesis processing unit 11d.

[0044] The content signal processing unit 11a receives, for example, the image signal V1 and audio signal A1 of the visual content received by the communication unit 14, performs decoding, image quality adjustment, etc., and supplies the result to the synthesis processing unit 11d. The imaging signal processing unit 11b performs development processing on the captured image signal V2 from the imaging unit 12, and also performs adjustment processing on the audio signal A1, and supplies it to the effect processing unit 11c. The effects processing unit 11c performs image effects on the image of the target object 3 in the captured image signal V2. The effects processing unit 11c may also perform sound effects on the audio signal A2 or perform speech synthesis.

[0045] The synthesis processing unit 11d synthesizes the image signal V1 and the captured image signal V2 to generate an image signal V3 as a video see-through image. The synthesis processing unit 11d basically outputs only audio signal A1 as audio signal A3 when content is being viewed, but in cases such as when audio effects are added or when ambient sounds are played, it may synthesize audio signal A2 with audio signal A1 to generate audio signal A3.

[0046] Through the processing of the image processing unit 11 as described above, the display unit 16 displays the display image 40 as a video see-through image, and image effects can be applied to the viewing space image 42. Furthermore, the audio output unit 17 can output ambient sounds and effect sounds in addition to the content sound.

[0047] The functions of the control unit 10 are described below. The information input unit 100 can acquire content information from the communication unit 14, and can also acquire information about the surrounding viewing space by inputting the captured image signal V2. The information input unit 100 can acquire various information about the visual content to be viewed by communicating with a content source, external server, or external database, for example, via the communication unit 14. For example, when streaming or downloading visual content, the information input unit 100 requests content information from an external server, etc., and inputs the content information. Furthermore, the information input unit 100 can communicate with surrounding IoT devices via the communication unit 14 and input information from IoT devices to confirm the presence of IoT devices and verify the operation of IoT devices. In addition, the information input unit 100 can also determine objects present in the surroundings by performing object recognition processing using the captured image signal V2.

[0048] The information management unit 400, using the content information definition unit 402 in Figure 5, generates or updates a content information definition table 50 for each content title based on the content information acquired by the information input unit 100, and stores it in the storage unit 15. The information management unit 400 also reads the content information definition table 50 from the storage unit 15 for the determination processing of the determination processing unit 200. Furthermore, the information management unit 400, using the object information definition unit 401 in Figure 5, generates or updates an object information definition table 60 for the object that will be the effect target object 20 based on the viewing space information acquired by the information input unit 100, and stores it in the storage unit 15. The information management unit 400 also reads the object information definition table 60 from the storage unit 15 for the determination processing of the determination processing unit 200.

[0049] The determination processing unit 200 performs effect determination by referring to the object information definition table 60 and the content information definition table 50, and based on the determination, executes the effect or obtains the result of the determination of which effect to execute.

[0050] The effect control unit 300 performs specific effect control based on the determination made by the determination processing unit 200. For example, the effect control unit 300 sends a control signal C1 to the effect processing unit 11c to instruct the image to be affected (the image of the target object 3) and the effect content, and to execute the image effect processing. The effect control unit 300 also sends a control signal C2 to the controlled device 2 via the communication unit 14 to instruct the controlled device 2 to perform an action as an effect, and to execute that action.

[0051] In this embodiment, the HMD1 configured as shown in Figures 6 and 7 performs effect processing using the effect target object 20 in accordance with the scene when viewing visual content. Although Figure 6 shows an example of the configuration of HMD1, for example, if we consider the system in Figure 4, it is also conceivable that the configuration within the area enclosed by the dashed line LX in Figure 6, namely the control unit 10, image processing unit 11, communication unit 14, and storage unit 15, is provided by the effect control device 4, and HMD1 is provided by the imaging unit 12, sensor unit 13, display unit 16, and audio output unit 17. Furthermore, an example is also conceivable in which the configuration within the area enclosed by the dashed line LY in Figure 6, namely the control unit 10, communication unit 14, and storage unit 15, is provided by the effect control device 4, and the HMD1 is equipped with an image processing unit 11, imaging unit 12, sensor unit 13, display unit 16, and audio output unit 17.

[0052] <3. Effect processing using content information and object information> This section describes an example of effect processing performed by the HMD1 configured as shown in Figures 6 and 7, by referring to Content Information Definition Table 50 and Object Information Definition Table 60.

[0053] Figure 8 shows a specific example of the content information definition table 50. The content information definition unit 402 of the information management unit 400, which is shown as a function of the control unit 10, stores the content information acquired for the visual content to be viewed in the storage unit 15 and manages it as the content information definition table 50.

[0054] Content Information Definition Table 50 is a table that defines what scene attributes a visual content intended for viewing has in different scenes.

[0055] Content Information Definition Table 50 defines content attributes, scenes, and scene attributes for each content title of visual content. In Figure 8, for content title CA, which is an action movie, the content attribute is set to "Action," and the content of scenes from timecode "XX:XX" and scenes from timecode "YY:YY" are stored. Scene attributes such as "Explosion" and "Morning" are also stored for each scene. Furthermore, for content title CB, which is a romance film, the content attribute is set to "Romance," the content of the scene starting from timecode "ZZ:ZZ" is stored, and scene attributes such as "Rain" are also stored.

[0056] The scene attributes defined here refer to the experience within the visual content, as well as attributes that can be reproduced in the user's viewing space through image effects, sound effects, lighting changes, etc., such as changes in time or season, weather, etc. Note that while the timecodes for each scene in the visual content are shown as "XX:XX," they are actually expected to be structured as "hours:minutes:seconds:frames."

[0057] Figure 9 shows a specific example of the object information definition table 60. The object information definition table 60 is a table that defines what kind of effect an object detected in the user's viewing space, such as furniture or home appliances, will be used for, under what scene attributes and conditions it is met. The object information definition unit 401 of the information management unit 400, which is shown as a function of the control unit 10, stores object information about the effect target objects 20 present in the viewing space in the storage unit 15 and manages it as the object information definition table 60.

[0058] The object information definition table 60 defines room and scene attributes for each effect target object 20. A room is information stored to correspond to the effect target objects 20 present in each room, for example, when a user uses multiple rooms as a viewing space.

[0059] Scene attributes indicate the type of scene in which the effect target object 20 will be used for the effect. One or more scene attributes can be set for a single effect target object 20. For example, if "window and curtain opening / closing system" is registered as the effect target object 20, the scene attributes "morning," "rain," and "explosion" are stored as shown in the figure. Also, for the lighting equipment designated as the effect target object 20, "morning" and "rain" are stored. For example, the window and curtain opening / closing system are grouped together as a single effect target 20, but they may also be registered as separate effect targets 20.

[0060] Furthermore, one or more effect generation conditions are stored corresponding to the scene attributes. Effect generation conditions are information that indicates the conditions for effect activation. Examples of effect generation conditions include weather and the existence and positional relationship of the effect target object 20 registered in the object information definition table 60. In addition to these, a variety of other factors can also be effect generation conditions, such as temperature, humidity, ambient sound level, number of viewers, viewer position, and the positional relationship between the viewer position and the effect target object 20.

[0061] Furthermore, for each effect generation condition, the content of the effect that can be activated when that condition is met is also stored. The content of the effect includes the actions to be performed by the controlled device 2, which is the target object 20 of the effect, and the specific content of the image effect applied to the image of the target object 3.

[0062] In Figure 9, the "window and curtain opening / closing system" and "lighting equipment," which are controlled devices 2, are used as examples of the effect target object 20. However, target objects 3, such as "vases," may also be registered in the object information definition table 60. Similarly, in the case of target object 3, the room type, scene attributes, one or more effect generation conditions, and effect content are stored in the object information definition table 60.

[0063] Figure 10 illustrates the outline of the effect control processing of the control unit 10 using the content information definition table 50 and object information definition table 60 described above. The control unit 10 performs a determination process in steps S1 to S5 of Figure 10, and then performs a control process for effect execution in step S6.

[0064] In step S1, the control unit 10 acquires content information about the visual content that the user is viewing or is viewing, as well as object information in the surrounding viewing space. Based on the acquired content information and object information, the control unit 10 generates or updates the content information definition table 50 and the object information definition table 60.

[0065] In step S2, the control unit 10 processes the information stored in the content information definition table 50 for the current visual content and determines the scene. In step S3, the control unit 10 processes the information stored in the object information definition table 60 regarding the viewing space, for example, the room being viewed, and determines whether the object 20 is an effect target. In step S4, the control unit 10 checks the effect generation conditions for the effect target object 20 and determines whether the current situation satisfies the conditions.

[0066] Based on these processes, in step S5, the control unit 10 selects an effect target 20 to be used for the effect for each scene of the visual content, based on matching scene attributes, and determines the effect content to be executed according to the effect generation conditions. For example, the control unit 10 selects an effect target object 20 from the object information definition table 60 that has the same scene attributes as defined in the content information definition table 50. Then, for the selected one or more effect target objects 20, it determines the effect generation conditions and determines the effect content suitable for execution.

[0067] In step S6, the control unit 10 performs effect control so that the effect it has determined to execute for that scene is actually activated, according to the timing of the scene. This allows users to experience effects that utilize controlled devices 2 and target objects 3, in accordance with the scene, while viewing visual content—that is, effects outside of the content image.

[0068] <4. First Embodiment> The following describes the processing of the first embodiment, citing the first and second examples of specific effects.

[0069] As the first example, we will explain an example of applying a sunlight effect to a morning scene in a certain visual content. Figure 11 shows a portion of the content information definition table 50 and the object information definition table 60. In the content information definition table 50, it is stated that the visual content in this case is, for example, an action movie, with the content attribute set to "Action," and that the scene from the timecode "YY:YY" and the scene attribute "Morning" are stored. In the object information definition table 60, it is assumed that the existence of "window and curtain opening / closing system" and "lighting equipment" was recorded as effect target objects 20 that include the scene attribute "morning".

[0070] In this case, the control unit 10 performs effect processing in the following steps (P1) to (P3). (P1) The control unit 10 obtains information about the visual content to be viewed from the content information definition table 50 and obtains the scene attribute "morning" for the defined scene.

[0071] (P2) The control unit 10 determines from the object information definition table 60 whether the effect target object 20, which has the acquired scene attribute "morning" defined, satisfies the effect generation conditions. Then it obtains one of the following determination results P2a, P2b, or P2c.

[0072] • Judgment result P2a: If the time of viewing is morning or noon and the weather is sunny, the control unit 10 determines that the effect to be executed is "Slowly open the curtains and use the sunlight coming in through the window to recreate the feeling of light entering," which is the effect content stored in the object information definition table 60. • Judgment result P2b: If the weather is not sunny but the presence of lighting equipment can be determined in the object information definition table 60, the control unit 10 determines that the effect to be executed is "Gradually brighten with lighting equipment to recreate the morning light," which is the effect content stored in the object information definition table 60. • Judgment result P2c: If the weather is not sunny and the lighting equipment is not defined in the object information definition table 60, the control unit 10 determines that the effect to be executed is "superimpose an effect image that looks like the morning sun is shining through the windowpane," which is the effect content stored in the object information definition table 60.

[0073] (P3) The control unit 10 controls the execution of the effect determined by the time code of the visual content being viewed, according to the timing of the scene for which the determined effect occurred, and determines that the effect is one of the above determination results P2a, P2b, or P2c. Figure 12 illustrates the effect. Figure 12 shows the display image 40 that the user wearing the HMD1 is viewing, as explained in Figure 3.

[0074] If the result is P2a, the control unit 10 controls the curtain opening and closing system to slowly open the curtain 22, as shown as effect EF2a in Figure 12. As a result, the user wearing the HMD1 and watching will see, along with the content image 41, a viewing space image 42 in which the amount of light entering from the window 32 due to sunlight gradually increases.

[0075] If the judgment result is P2b, the control unit 10 closes the curtain 22 and controls the lighting equipment 21 to gradually increase its light intensity, as shown as effect EF2b. As a result, the user wearing the HMD1 and watching content will feel the morning light streaming in as they view the content image 41 and the viewing space image 42 shows the ambient light intensity increasing.

[0076] If the judgment result is P2c, the control unit 10 controls the curtain 22 to be open, as shown as effect EF2c. The control unit 10 also controls the image processing unit 11 so that an effect image of sunlight shining in is superimposed on the image of the window 32. This allows the user wearing the HMD1 to see a viewing space image 42 that depicts morning light shining in while viewing the content image 41.

[0077] Furthermore, for a given scene in Content Information Definition Table 50, there may be no effect target object 20 that contains the same scene attributes as the scene attribute. Also, even if an effect target object 20 containing the same scene attributes as the scene attribute exists, the effect generation conditions may not be met. For example, this may occur when none of the judgment results P2a, P2b, or P2c are obtained. In these cases, the effect is determined not to be executed for that scene, and procedure (P3) is not performed at the timing of that scene. This point is also true for the examples described later.

[0078] As a second example, we will explain an example of applying a rain effect to a rain scene in a certain visual content. Figure 13 shows a portion of Content Information Definition Table 50 and Object Information Definition Table 60. In Content Information Definition Table 50, the content attribute of the visual content in this case is set to "Romance," and the scene starting from timecode "ZZ:ZZ" and the scene attribute "Rain" are stored. In the object information definition table 60, it is assumed that the existence of "window and curtain opening / closing system" and "lighting equipment" was recorded as effect target objects 20 that include the scene attribute "rain".

[0079] In this case, the control unit 10 performs effect processing in the following steps (P11) to (P13). (P11) The control unit 10 obtains information about the visual content to be viewed from the content information definition table 50 and obtains the scene attribute "rain" for the defined scene.

[0080] (P12) The control unit 10 determines from the object information definition table 60 whether the effect target object 20, which has the acquired scene attribute "rain" defined, satisfies the effect generation conditions. Then, for example, it obtains the following determination results P12a, P12b, and P12c.

[0081] • Judgment result P12a: If the weather is rainy, the control unit 10 determines that it will execute the effect "Open the curtains and enhance external sound intake," which is the effect content stored in the object information definition table 60. • Judgment result P12b: If the weather is not rainy but the presence of lighting equipment can be determined in the object information definition table 60, the control unit 10 determines that the effect to be executed is "Turn off the room lights and add a rain sound effect," which is the effect content stored in the object information definition table 60. · Judgment result P12c: If the weather is not rainy and the lighting equipment is not defined in the object information definition table 60, the control unit 10 determines that the effect to be executed is the effect described in the object information definition table 60, which is "apply a dark filter to the entire room and superimpose an effect image of rain falling onto the window glass".

[0082] (P13) The control unit 10 controls the execution of the determined effect based on the time code of the visual content being viewed, in accordance with the timing of the scene in which the determined effect occurs. Figure 14 illustrates the effect. Figure 14 shows the display image 40 viewed by the user wearing the HMD1.

[0083] If the judgment result is P12a, the control unit 10 controls the curtain opening and closing system to open the curtain 22, as shown as effect EF12a in Figure 12. This enhances the influx of outside sounds, in this case the sound of rain. As a result, the user wearing the HMD1 and watching the video perceives the sound of rain more strongly, and also makes it possible to see the actual rain through the window 32.

[0084] If the judgment result is P12b, the control unit 10 closes the curtain 22 and controls the lighting equipment 21 to reduce the light intensity, as shown in the effect EF12b. Furthermore, the control unit 10 instructs the image processing unit 11 to perform sound effect processing to add the sound of rain to the content audio. As a result, the user wearing the HMD1 and watching the content will feel the decrease in ambient light while looking at the content image 41, and will also feel the rain through the effect sound.

[0085] If the judgment result is P12c, the control unit 10 controls the curtain 22 to be open, as shown as effect EF12c. The control unit 10 also controls the image processing unit 11 so that image processing is performed on the viewing space image 42 so that a dark filter is applied to the entire room, and image processing is performed so that an effect image of rain falling on the image of the window 32 is superimposed. This allows the user wearing the HMD1 to see the viewing space image 42 which depicts rainy weather while viewing the content image 41.

[0086] As shown in the first and second examples above, when the weather in the content being viewed differs from that in the real world, an effect can be implemented that allows users to experience the weather of the content in the real world as well, thereby extending the world of the content into the real world and enhancing user immersion. Examples of processing performed by the control unit 10 to achieve these effects are explained in Figures 15 to 17. The flowcharts shown are examples of processing performed by the control unit 10 according to the function shown in Figure 5. It is assumed that the object information definition table 60 is stored in the storage unit 15.

[0087] Figure 15 shows the processing flow from before viewing visual content to after viewing ends. In step S101, the control unit 10 determines the viewing space. For example, the control unit 10 determines which room the viewing space is and what kind of items are present in the viewing space. For example, the type of room and the items present in the room can be determined by communication with IoT devices or by image recognition from images captured by the imaging unit 12. The type of room may also be determined using the location information of the sensor unit 13. Through these processes, the control unit 10 determines which room is the viewing space and recognizes the effect target objects 20 present in the viewing space. Based on this, the control unit 10 determines which effect target objects 20 are available for effect use from among the effect target objects 20 stored in the object information definition table 60 stored in the storage unit 15. Alternatively, a separate object information definition table 60 may be stored for each room, and the control unit 10 may select the appropriate object information definition table 60 according to the room it has determined.

[0088] In step S102, the control unit 10 performs a process to acquire information about the visual content that the user is about to view. In particular, the control unit 10 acquires the information necessary for the content information definition table 50.

[0089] An example of the process in step S102 is shown in Figure 16. In step S121, the control unit 10 obtains content information about the visual content to be viewed, for example, by communicating with an external server. The content information includes content attribute information and scene information stored in the content information definition table 50. Scene information here refers to information that includes at least the time code and scene attribute information of the scene in the content information definition table 50, and is an item in one row of the content information definition table 50. The control unit 10 checks the information of one or more scenes from the received content information, which will be items stored in the content information definition table 50.

[0090] In step S122, it is determined whether the content information definition table 50 for the visual content in question, that is, the content title that the user is about to view, is already stored in the storage unit 15.

[0091] If the content is not already stored, the control unit 10 proceeds to step S128 and stores a new content information definition table 50 for the current content title. For example, the control unit 10 generates a content information definition table 50 by combining the content attribute information and one or more scene information included in the content information acquired in step S121, and stores this in the storage unit 15.

[0092] If the content information definition table 50 for the current content title is already stored in the storage unit 15, the control unit 10 checks the scene information using loop processing LP1. Loop processing LP1 is repeated for the number of times equal to the number of scene information entries included in the content information acquired in step S121.

[0093] In step S123, the control unit 10 selects one scene from the content information acquired to be processed. In step S124, the control unit 10 determines whether the scene information is already stored as a single row item in the content information definition table 50. If it is already stored, the control unit 10 does not perform any processing. On the other hand, if the scene information is not stored in the content information definition table 50, the control unit 10 proceeds to step S125 and adds it to the content information definition table 50 as new scene information for the content title.

[0094] Once the above processing has been performed on all received scene information, the loop process LP1 is terminated. This allows new scene information, namely the scene's timecode and scene attribute information, to be added to the content information definition table 50, even if the content title has already been stored.

[0095] In step S103 of Figure 15, the control unit 10 performs an effect determination process. This is the process in which the control unit 10 determines which scenes in the visual content to be viewed will have an effect applied, and what the effect will be. An example of the process in step S103 is shown in Figure 17.

[0096] In the process shown in Figure 17, the control unit 10 repeats the loop process LP2 for the number of scenes in the content information definition table 50 corresponding to the visual content, that is, the number of scenes in which the effect is executed. In this loop processing LP2, the control unit 10 selects one scene information to be processed from the content information definition table 50 in step S131. That is, it selects the time code and scene attributes of one scene to be processed. Then it performs loop processing LP3 with respect to the selected scene information. Loop processing LP3 is repeated for the number of rows in the object information definition table 60.

[0097] In step S132, the control unit 10 selects one piece of object information, that is, one row of information from the object information definition table 60, as the target for processing. The information in one row here refers to a set of information consisting of scene attributes, effect generation conditions, and effect content. For example, Figure 9 shows that the object information definition table 60 stores seven rows of object information, indicated as rows L1 to L7. In other words, in step S132, the control unit 10 extracts one set of effect generation conditions and effect content, and one corresponding scene attribute.

[0098] In step S133, the control unit 10 determines whether the scene attributes of the selected object information match the scene attributes of the scene information processed in step S131. If they do not match, the loop process LP3 returns to step S132, and another row of object information is selected from the object information definition table 60.

[0099] If the scene attribute matches, the control unit 10 proceeds to step S134 to determine the effect generation conditions for that object information. In other words, the control unit 10 determines whether the current situation satisfies the effect generation conditions. For example, the control unit 10 determines whether the current weather, the presence of the effect target object 20, the temperature, ambient sound, brightness, etc., satisfy the effect generation conditions. To make this determination, the control unit 10 may refer to detection information from the sensor unit 13. It can also acquire weather information, etc., through communication via the communication unit 14. Furthermore, it can acquire information about the viewing space from the image captured by the imaging unit 12.

[0100] In step S135, the control unit 10 determines whether the effect generation conditions are met. If they are not met, the loop process LP3 returns to step S132, and another row of object information is selected from the object information definition table 60.

[0101] If the effect generation conditions are met, the control unit 10 proceeds to step S136, determines that the effect indicated as the effect content in the object information on one line is the effect to be executed during the viewing of the visual content, and sets the effect execution data. For example, the scene's time code and the effect control content are associated to form a single effect execution data.

[0102] Once the above loop process LP3 has been executed for all rows in the object information definition table 60, the effect settings for one scene are complete. Therefore, the effect content that matches the scene attributes and satisfies the effect generation conditions is set as the effect to be executed. Effect execution data is set according to the scene attributes and effect generation conditions, so one or more effect execution data may be set for a single scene. Also, there may be cases where no effect execution data is set for a scene if there is no object information that matches the scene attributes or if there is no object information that satisfies the effect generation conditions.

[0103] Once the effect settings for one scene are complete, the control unit 10 returns to step S131 as loop processing LP2 and selects other scene information recorded in the content information definition table 50 as the next processing target. Then the control unit 10 executes loop processing LP3 for the selected scene information.

[0104] Through the process shown in Figure 17, one or more effect execution data are set for each scene information stored in the content information definition table 50 for the visual content being viewed. However, there may be scene information for which no effect execution data can be set.

[0105] The control unit 10 executes the process shown in Figure 17 as step S103 in Figure 15, and then waits for the start of viewing the visual content in step S104. In the example in Figure 15, steps S102 and S103 are performed before content viewing begins, but they may also be performed during content viewing. At the very least, the effect execution data should be set in time for the scene in which the effect is to be executed.

[0106] The control unit 10, which is waiting for viewing to begin, proceeds from step S104 to step S105 in response to a trigger such as user operation, and starts displaying the visual content and outputting sound on the HMD1. This allows the user to start watching the visual content, such as a movie.

[0107] In step S106, the control unit 10 determines that viewing of the visual content has ended. Until it is determined that viewing has ended, the control unit 10 determines the effect timing in step S107. The control unit 10 compares the time code in the set effect execution data with the time code of the visual content being viewed to determine whether the effect timing, that is, the timing to perform effect control, has been reached.

[0108] Then, when the timing for activating a certain effect execution data is reached, the control unit 10 proceeds to step S108 and performs effect execution control. In other words, the control unit 10 executes the control set for the corresponding effect execution data. For example, it executes control to the controlled device 2, or image effect control instructing the image processing unit 11. As a result, effects like those shown in Figures 12 and 14 are executed.

[0109] The control unit 10 continues the processes in steps S107 and S108 until it determines in step S106 that viewing has ended, such as through user operation or the end of playback of the visual content.

[0110] According to this first embodiment of processing, when a user views visual content, one or more effects matching a specific scene are executed using the effect target object 20 at the timing of that scene. The user can experience greater immersion not only from the content image 41 but also from the surrounding effect target objects 20.

[0111] <5. Second Embodiment> An example of effect processing in the second embodiment will be described. The second embodiment is an example in which priority is set for the effects to be executed, and how to handle cases where an effect cannot be executed is also added.

[0112] As a third example, let's consider an example where an explosion effect is displayed in an explosion scene. Figure 18 shows a portion of Content Information Definition Table 50 and Object Information Definition Table 60. In Content Information Definition Table 50, the content attribute of the visual content in this case is set to "Action," and the scene from timecode "XX:XX" and the scene attribute "Explosion" are stored. In the object information definition table 60, it is assumed that the existence of "vase," "window and curtain opening / closing system," and "electric fan" was recorded as effect target objects 20 that include the scene attribute "explosion."

[0113] In this case, the control unit 10 performs effect processing in the following steps (P21) to (P23). (P21) The control unit 10 obtains information about the visual content to be viewed from the content information definition table 50 and obtains the scene attribute "explosion" for the defined scene.

[0114] (P22) The control unit 10 determines from the object information definition table 60 whether the acquired scene attribute "explosion" is defined for the effect target object 20 and whether the effect generation conditions are met. Then, for example, the following determination results P22a, P22b, P22c, P22d, P22e, P22f, and P22g are obtained.

[0115] ·Judgment result P22a: If a vase is present as the closest breakable object to the user, and a fan is also present, the control unit 10 determines that the effect to be executed is the one described in the object information definition table 60, which is to "superimpose an effect image of a vase breaking" and "generate a sudden gust of wind at the timing of the explosion". · Judgment result P22b: If a window is the closest breakable object to the user and a fan is also present, the control unit 10 determines that the effect to be executed is the one described in the object information definition table 60, which is to "superimpose an effect video of a window pane shattering" and "generate a sudden gust of wind at the timing of the explosion". · Judgment result P22c: If a vase is the closest breakable object to the user and there is no electric fan, the control unit 10 determines that it will execute the effect "superimpose an effect video of a vase breaking," which is the effect content stored in the object information definition table 60. • Judgment result P22d: If a window is the closest breakable object to the user and there is no fan, the control unit 10 determines that it will execute the effect "superimpose an effect video of a window pane shattering," which is the effect content stored in the object information definition table 60. • Judgment result P22e: If there are no breakable objects near or far from the user, and a fan is present, the control unit 10 determines that the effect to be executed is "generate a sudden gust of wind at the time of the explosion," which is the effect content stored in the object information definition table 60. • Judgment result P22f: If no effect target object 20 matching the scene attribute "explosion" is stored in the object information definition table 60, the user will be asked to walk around the real-world viewing space and search for any objects that can be defined with the scene attribute "explosion," such as windowpanes or vases. • Judgment Result P22g: If the effect target object 20 that matches the scene attribute "explosion" is not stored in the object information definition table 60 and cannot be found even after searching the room, the system will prompt the user to place an object that can be defined by the scene attribute "explosion" so that the "explosion" effect can be displayed.

[0116] (P23) The control unit 10 controls the execution of the determined effect based on the time code of the visual content being viewed, etc., when the timing of the scene for the determined effect has arrived. Figures 19 and 20 illustrate how the effect works. Figures 19 and 20 show the display image 40 that the user wearing the HMD1 is viewing.

[0117] If the result is P22a, the control unit 10 controls the image processing unit 11 to perform image processing that superimposes an effect image of the vase 31 breaking, as shown as effect EF22a in Figure 19. The control unit 10 also controls the operation of the fan 24, which is an IoT appliance, to generate wind. This allows the user wearing the HMD 1 and watching the content to experience explosions and gusts of wind other than the content image 41.

[0118] If the result is P22b, the control unit 10 controls the image processing unit 11 to perform image processing that superimposes an effect image of the glass of the window 32 shattering, as shown as effect EF22b. The control unit 10 also controls the fan 24 to generate wind. This allows the user wearing the HMD1 to experience explosions and gusts of wind other than the content image 41.

[0119] If the judgment result is P22c, the control unit 10 controls the image processing unit 11 to perform image processing that superimposes an effect image of a vase 31 breaking, as shown as effect EF22c. This allows the user wearing the HMD1 and watching to experience an explosion other than the content image 41.

[0120] If the result is P22d, the control unit 10 controls the image processing unit 11 to perform image processing that superimposes an effect image of the glass of the window 32 shattering, as shown as effect EF22d in Figure 20. This allows the user wearing the HMD1 and watching to experience an explosion other than the content image 41.

[0121] If the result is P22e, the control unit 10 controls the fan 24 to generate wind, as indicated by the effect EF22e. This causes the user wearing the HMD1 and watching to experience a gust of wind.

[0122] If the judgment result is P22f, the effect target object 20 stored in the object information definition table 60 does not exist, so the effect cannot be executed as is. In this case, instead of at the scene timing, for example, immediately after obtaining the judgment result P22f, a search request is made to the user by displaying a message 35 or by voice guidance, as shown as guide GD22f in Figure 20. For example, the user is asked to actually walk around the surroundings, i.e., the room that is the viewing space, or to look around. As a result, the control unit 10 performs object recognition processing from the image captured by the imaging unit 12 and searches for an object that can define the scene attribute "explosion". For example, if a breakable object such as a vase 31 or a window 32 is found, it is registered in the object information definition table 60. After that, the judgment (P22) above is performed again to obtain the judgment result P22d or judgment result P22e. When it is the scene timing, effect control is performed according to the judgment result P22d or judgment result P22e.

[0123] If the judgment result is P22g, the effect cannot be executed because there is no target object 20 in the current viewing space. Therefore, for example, immediately after obtaining the judgment result P22g, or after the content viewing has finished, a message 36 is displayed to the user, or an audio guidance is given, as shown as guide GD22g in Figure 20, recommending that the user bring in an item. For example, the user is presented with suggestions to purchase an appropriate item as the target object 20, bring it into the room, or place it in an appropriate location. By doing so, the possibility of being able to perform an effect corresponding to the scene attribute "explosion" when viewing the same or different visual content next time is increased.

[0124] The above judgment results P22a to P22d indicate that, in determining which effect to execute according to the scene attribute "explosion," priority is set according to the effect generation conditions in Figure 18. For example, if there are both a vase 31 and a window 32 as effect targets 20, the use of the effect on the vase 31, which is closer to the user, will be prioritized over the effect on the window 32. This increases the likelihood of executing an effect that utilizes an object that is easily within the user's field of view, i.e., within the displayed image 40, and is likely to be in the viewing space image 42. Furthermore, if the judgment result is P22f to P22g, it means that the effect target object 20 does not currently exist, and the user will be prompted to take action to address this.

[0125] An example of the processing of the control unit 10 in the second embodiment described above will be explained in Figures 21 and 22. In this example, the control unit 10 performs the processing shown in Figures 15 and 16, similar to the first embodiment, but performs the processing shown in Figures 21 and 22 instead of the processing shown in Figure 17. In Figure 21, the same step numbers are used for the same processing as in Figure 17 to avoid redundant explanations.

[0126] The loop processes LP2 and LP3 in Figure 21 are roughly the same as the process described in Figure 17. From the content information definition table 50, one scene information to be processed is selected, and the loop process LP3 is repeated for the number of rows in the object information definition table 60 for the selected scene information to determine the effect to be executed for that scene and generate effect execution data.

[0127] Here, the object information definition table 60 may store conditions for setting priority in the effect generation conditions. For example, the condition in Figure 18 is "There are no breakable objects closer to the user than this object." When such effect generation conditions are set, priority determination for effect usage is made for multiple effect target objects 20. For example, in a situation where there is a window 32 and a vase 31, the vase 31, which is closer to the user, is selected preferentially and effect execution data is generated. The effect for window 32 is rejected in step S135 because the effect generation conditions are not met.

[0128] As a result, in step S136, a determination is made that corresponds to one of the above determination results P22a to P22e, and effect execution data for the scene is set. Note that there are cases where effect execution data cannot be set.

[0129] In Figure 21, after completing loop processing LP2, step S138 checks for the existence of scenes for which effect execution data has not been set. In other words, these are scenes for which, among the scenes whose scene information is stored in content information definition table 50, there are no effect target objects 20 with common scene attributes, or even if there are effect target objects 20 with common scene attributes, the effect generation conditions are not met, and therefore effect execution data could not be set. In other words, this is the case where the aforementioned judgment result P22f is obtained for such a scene.

[0130] If there is a scene for which no effect execution data has been set, the control unit 10 proceeds to the process shown in Figure 22, as indicated by "ct1". In step S141, the control unit 10 processes a search request and object recognition for the user. For example, as shown as guide GD22f in Figure 20, the control unit 10 has the image processing unit 11 execute actions such as displaying message 35 and outputting sound to request a search from the user. At the same time, the control unit 10 receives the captured image signal V2 from the imaging unit 12 and performs object recognition processing to determine whether or not a suitable object exists as the target object 20 for the effect.

[0131] For example, the control unit 10 performs such object recognition processing for a predetermined time, and in step S142, the process branches depending on whether or not an object that can be stored in the object information definition table 60 as the effect target object 20 has been found.

[0132] If the target object 20 cannot be found, the above-mentioned determination result P22g will be obtained. In that case, the control unit 10 proceeds to step S150 and controls the image processing unit 11 to execute a message 36 or voice guidance recommending the bringing in of an appropriate item, as shown as guide GD22g in Figure 20. In this case, the process in step S150 is performed before viewing the content, but it may also be performed after viewing the content.

[0133] If an effect target object 20 is found in step S141, the control unit 10 proceeds to step S143 and adds the found effect target object 20 to the object information definition table 60. For example, the control unit 10 identifies the object found this time as an effect target object 20 for the scene attribute for which effect execution data could not be set this time, generates effect generation conditions and effect content, and registers them in the object information definition table 60.

[0134] The control unit 10 then repeats the loop process LP4 for the number of lines in the content information definition table 50 for which effect execution data has not been set for scenes. In this loop process LP4, the control unit 10 selects one scene information for which effect execution data has not been set as the target for processing in step S144. Then, it performs loop process LP5 on the selected scene information. Loop process LP5 is repeated for the number of rows added in step S143 in the object information definition table 60.

[0135] In step S145, the control unit 10 selects one of the newly added object information items in the object information definition table 60 as the target for processing. That is, information consisting of a set of scene attributes, effect generation conditions, and effect content.

[0136] In step S146, the control unit 10 determines whether the scene attributes of the selected object information match the scene attributes of the scene information processed in step S144. If they do not match, the loop process LP5 returns to step S145, and the control unit selects the object information of the other row newly added to the object information definition table 60.

[0137] If the scene attribute matches, the control unit 10 proceeds to step S147 and determines the effect generation conditions for that object information. In step S148, the control unit 10 determines whether the effect generation conditions are met. If they are not met, the loop process LP5 returns to step S145, and the control unit selects the other newly added row of object information from the object information definition table 60.

[0138] If the effect generation conditions are met, the control unit 10 proceeds to step S149 and sets the effect execution data for the effect indicated as the effect content in the object information on one line, as an effect to be executed in accordance with the scene selected in step S144.

[0139] After executing the above loop process LP3 for all rows newly added to the object information definition table 60, the additional effect settings for the one scene where effect execution data was not set are completed. This provides an opportunity to set effect execution data for the scene where it was not set in the process shown in Figure 21.

[0140] Once the effect settings for one scene are complete, the control unit 10 returns to step S144 as loop processing LP4 and selects other scene information for which effect execution data has not yet been set as the next processing target. Then the control unit 10 executes loop processing LP5 for the selected scene information.

[0141] Even after performing the object search and additional effect execution data setting process as described above, there may still be scenes where effect execution data cannot be set. In that case, the control unit 10 proceeds from step S151 to step S150 and controls the image processing unit 11 to execute a message 36 or voice guidance recommending the bringing in of an appropriate item, as described as guide GD22g.

[0142] After the process shown in Figure 22, the control unit 10 returns to the process shown in Figure 21, as indicated by "ct2", completes the process in step S103 in Figure 15, and proceeds to step S104.

[0143] In the second embodiment described above, a more appropriate effect can be selected by setting a priority for the effects to be executed. Furthermore, if an object matching the scene attributes does not exist in the user's viewing space, the user is prompted to register the object, leading to a more immersive experience.

[0144] <6. Third Embodiment> A third embodiment is illustrated in Figure 23. This is an example of viewing space determination performed as step S101 in Figure 15.

[0145] The control unit 10 performs the process shown in Figure 23 during step S101 in Figure 15. In step S301, the control unit 10 determines whether the location where the user is about to view content is the same location where visual content has been viewed in the past.

[0146] For example, the control unit 10 manages the object information definition table 60 by associating it with the room's location information. The object information definition table 60, which stores the effect target objects 20, is created corresponding to the room that will be the viewing space. The location information obtained by the sensor unit 13 is associated with this object information definition table 60 so that an object information definition table 60 is created for each room. The control unit 10 performs this management, and in step S101, it only needs to determine whether the current location corresponds to any of the location information in the stored object information definition table 60.

[0147] If a location that has been viewed in the past, i.e., the object information definition table 60 corresponding to that location, is stored, the control unit 10 proceeds to step S302. In step S302, the control unit 10 performs a process to acquire information about surrounding objects in the current viewing space. For example, the control unit 10 receives the captured image signal V2 from the imaging unit 12 and performs object recognition processing to determine whether or not there is an object suitable as the effect target object 20. In this case, the control unit 10 may process search requests from the user, such as asking them to walk around the room or look around, so that the imaging unit 12 can efficiently capture images of the surroundings. Furthermore, the control unit 10 may communicate with IoT devices to confirm their presence and whether they can operate as controlled devices 2.

[0148] The control unit 10 then repeats the loop process LP6 for the number of times corresponding to the number of object information units acquired. In this loop process LP6, in step S303, the control unit 10 selects one of the objects it has found that could be the effect target object 20 as the object to be processed. Then, in step S304, the control unit 10 determines whether the selected object is already stored in the object information definition table 60 corresponding to its location. If it is already stored, the loop process LP6 returns to step S303. If the object is not already stored, the control unit 10 proceeds to step S305 and registers the object as an effect target object 20 in the object information definition table 60.

[0149] This loop processing LP6 makes it possible to register any changes in the objects placed in a certain location (such as a room) as new target objects 20 for the effect, thereby increasing the likelihood of the effect being executed.

[0150] The control unit 10 may also perform a process to delete the effect target object 20 from the object information definition table 60 if the effect target object 20 that was stored in the object information definition table 60 is no longer present. This allows the object information definition table 60 to be updated so that effects can be executed according to the current conditions of the room, etc.

[0151] If the current location is not a location that has been viewed in the past, that is, if the object information definition table 60 corresponding to that location is not stored, the control unit 10 proceeds to step S310.

[0152] In step S310, the control unit 10 processes a search request and an object recognition process for the user. This is similar to the process in step S141 in Figure 22. Furthermore, the control unit 10 may attempt to communicate with the IoT device to confirm the existence of the device or system that will be the controlled device 2.

[0153] In step S311, the control unit 10 branches the process depending on whether or not it has found an object that could be used for the effect, that is, an object that could become the effect target 20.

[0154] If the controlled device 2 or target object 3 is found to be an object that could be the target object 20 of the effect, the control unit 10 repeats the loop process LP7 for the number of objects found.

[0155] In this loop processing LP7, in step S312, the control unit 10 selects one of the objects it has found to be a potential effect target 20 as the object to be processed. Then, in step S313, the control unit 10 stores the selected object as the effect target 20 in a new object information definition table 60.

[0156] This loop processing LP7 creates an object information definition table 60 corresponding to a specific location (such as a room) when the content is viewed for the first time in that location, and stores it in the memory unit 15. This makes it possible to execute effects using the effect target object 20 even when viewing in a new location.

[0157] <7. Fourth Embodiment> As a fourth embodiment, we will describe an example in which, when viewing visual content begins, an introductory image selected according to the attributes of the visual content to be viewed is displayed according to the position of objects in the viewing space. For example, an appearance animation incorporating the content elements is displayed within the display image 40 before viewing the visual content begins.

[0158] Figure 24 shows an example of an installation image. Let's assume the visual content to be viewed is a science fiction film set in space. In this case, within the displayed image 40, a UFO (Unidentified Flying Object) 37 appears from window 32 as shown in effect EF30-1 in Figure 24, and then, as shown in effect EF30-2, the UFO 37 moves above the content image 41 in front. Then, as shown in effect EF30-3, the UFO 37 emits light that creates a screen, and as shown in effect EF30-4, the UFO 37 is sucked into the screen. This screen becomes the content image 41, and the movie begins.

[0159] Figure 25 shows another example. Let's assume the visual content to be viewed is an action movie. In this case, within the displayed image 40, a bomb 38 is thrown through window 32 as shown in effect EF40-1 in Figure 25, and then the bomb 38 explodes as shown in effect EF40-2. Then, as shown in effect EF40-3, a scene is displayed in which a screen appears amidst the smoke of the explosion, and as shown in effect EF40-4, the smoke clears and the screen appears. This screen becomes the content image 41, and the movie begins.

[0160] For example, as shown in Figures 24 and 25, introductory images tailored to the content being viewed are displayed in accordance with the room layout, naturally guiding the user from the real world into the world of the content.

[0161] An example of processing by the control unit 10 is explained in Figures 26 and 27. Figure 26 shows an example in which step S160 is added to the process in Figure 15. Since the other processes are the same as in Figure 15, the same step numbers are used and their explanations are omitted.

[0162] In step S104, the control unit 10 waits for the start of viewing the visual content. When it determines that it is time to start viewing based on user operation or a predetermined trigger, it performs an introduction effect processing in step S160.

[0163] The introduction effect processing is shown in Figure 27. In step S161, the control unit 10 determines the content type. For example, it determines the genre of the visual content, such as science fiction, action, romance, adventure, comedy, musical, etc. This can be determined from the content information acquired in step S101.

[0164] In step S162, the control unit 10 selects an introductory effect according to the content type. For example, the memory unit 15 stores multiple introductory effect scenarios corresponding to the genre of content, such as movies. For example, the introductory effect in Figure 26 for space science fiction movies and Figure 27 for action movies are also scenarios such as introductory effects. The control unit 10 selects which scenario of introductory effect to execute this time.

[0165] In step S163, the control unit 10 controls the execution of the introduction effect based on the selected scenario. In other words, the control unit 10 controls the image processing unit 11 so that the introduction image is displayed within the HMD1's display image 40 by the image synthesis effect. This allows the user to see introductory images such as those shown in Figure 26 and Figure 27. The control unit 10 then proceeds to step S105 in Figure 26, initiating playback of the visual content. The user will then watch the main movie or other content following the introductory image.

[0166] <8. Summary and Variations> According to the above embodiments, the following effects can be obtained.

[0167] In the embodiment, HMD1 or effect control device 4 were given as examples of information processing devices. The control unit 10, which is a processor mounted on these information processing devices, performs at least a determination process and a control process. As a determination process, the control unit 10 makes a determination regarding an effect that corresponds to a scene in the visual content and targets the effect object 20, based on content information including scene attributes for some or all of the scenes in the visual content and object information including scene attributes for the object that is the target of the effect among the objects that actually exist in the viewing space. In this case, the control unit 10 uses the content information definition table 50 as content information and the object information definition table 60 as object information. Furthermore, the control unit 10, as part of its control processing, executes effects in accordance with the corresponding visual content scene based on the determination process. This allows for effects that utilize the user's surrounding environment and are adapted to the scene, for example, when viewing content with video see-through using an HMD1. For instance, by reflecting the weather, lighting, and experiences within the content onto the real world, a highly immersive viewing experience becomes possible.

[0168] In this embodiment, the HMD1 is a device equipped with a display unit 16 that allows the user to view the viewing space along with the playback image of the visual content. For example, the display unit 16 of the HMD1 displays the viewing space image 42 captured by the imaging unit 12, that is, the image of the scene in front of the user, together with the content image 41. This allows the user to visually perceive the effects produced by the effect target objects 20 present in the viewing space. Alternatively, instead of using an image captured by the imaging unit 12, the configuration may allow the user to see the area in front of them along with the content image 41 by transmitting direct light to a certain extent. In this case as well, the user can perceive the effect produced by the controlled device 2.

[0169] In this embodiment, the HMD1 is an example that includes an image processing unit 11 that receives a playback image of visual content (image signal V1) and an image of the viewing space captured by the imaging unit 12 (captured image signal V2), and is capable of performing effect processing on the captured image and synthesis processing to combine the captured image and the playback image. For example, the image processing unit 11 can perform effect processing on the captured image signal V2 of the viewing space captured by the imaging unit 12 by the effect processing unit 11c, thereby executing an image effect on the object to be affected 20. In this case, the captured image signal V2 of the viewing space and the image signal V1 of the content image 41 are combined and supplied to the display unit 16, allowing the user to see the viewing space with the image effect applied to the object to be affected 20.

[0170] In this embodiment, the control unit 10 selects an effect target object 20 from the object information definition table 60 that matches the scene attributes of the scene shown in the content information definition table 50, and determines to execute an effect that processes the selected effect target object 20. Because scene attributes are registered in the content information definition table 50 and the object information definition table 60, the matching between the scene and the effect target object 20 can be determined based on the commonality of the scene attributes. As a result, the control unit 10 can select an effect target object 20 that is appropriate for the content of the scene and target it for effect processing.

[0171] In this embodiment, the object information definition table 60 includes information on the effect generation conditions for the effect target object 20 and the effect content corresponding to the effect generation conditions. The control unit 10 then determines the effect content by checking whether the effect generation conditions are met for the effect target object 20 included in the object information definition table 60. Since the effect generation conditions and effect content for the effect target object 20 are registered in the object information definition table 60, the control unit 10 can determine whether or not the effect can be effectively executed and what the effect content will be. This allows the control unit 10 to execute an effect appropriate to the scene content according to the environmental conditions, such as the weather and the existing effect target objects 20.

[0172] In this embodiment, the object information definition table 60 includes a controlled device 2 that is compatible with the control signal C2 transmitted by the control unit 10 as an effect target object 20. The control unit 10 then performs control processing to execute the operation of the controlled device 2 in accordance with the relevant scene of the visual content, as an example. For example, the controlled device 2, which can operate in accordance with the control signal C2, such as an IoT device, is designated as the effect target object 20. The control unit 10 controls the operation of the controlled device 2 so that effects appropriate to the scene are executed. For example, effects that change the viewing space itself using lighting equipment 21, a system that opens and closes curtains 22, a fan 24, etc., can be made possible, thereby enhancing the user's sense of immersion in the world of visual content.

[0173] In this embodiment, the object information definition table 60 includes information on the effect generation conditions for the effect target object 20 and the effect content corresponding to the effect generation conditions. The control unit 10 then performs control processing to execute the operation of the controlled device 2 shown in the effect content corresponding to the effect generation conditions, in accordance with the relevant scene of the image content, when the controlled device 2 is the target of processing. The control unit 10 ensures that the actions indicated in the effect content, according to the effect generation conditions, are executed as effects suitable for the content scene of the controlled device 2. This allows the controlled device 2 to execute appropriate actions as effects, taking the effect generation conditions into consideration. Furthermore, by registering various effect generation conditions and effect content for a single controlled device 2, the diverse operations of the controlled device 2 can be used as effects.

[0174] In this embodiment, the operation of the controlled device 2, which is executed and controlled by the control unit 10, is an operation that causes a change in the viewing space. For example, environmental changes such as dimming of lighting equipment 21, opening and closing of curtains 22 to introduce or block sunlight, and environmental changes caused by electric fans 24 and air conditioners 23 were given as examples of effects. In this way, it is possible to change the viewing space using controllable devices 2 that are commonly found in a room, and by using these as effects, it is possible to enhance the user's sense of immersion in the world of visual content.

[0175] In this embodiment, the object information definition table 60 includes target object 3, which is an object whose operation cannot be controlled by the control unit 10, as an effect target object 20. When the control unit 10 processes target object 3, it performs control processing to execute an image effect on the captured image of target object 3 in accordance with the relevant scene of the visual content. The example vase 31, window 32, and other objects present in the room—that is, objects that cannot communicate with the control unit 10, or objects that do not have communication capabilities in the first place—can also be designated as target objects 3 for the effect target objects 20. In this case, by employing an image effect technique applied to the captured image of the target object 3, the user can perceive visual effects such as the target object 3 moving or exploding. This can enhance the user's immersion in the content.

[0176] In this embodiment, the target object 3 is included as the effect target object 20, and the object information definition table 60 includes information on the effect generation conditions for the effect target object 20 and the effect content corresponding to the effect generation conditions. The control unit 10 then performs control processing to execute the image effect processing shown in the effect content corresponding to the effect generation conditions on the captured image of the target object 3, in accordance with the relevant scene of the visual content, as shown in the effect content corresponding to the effect generation conditions, when the target object 3 is the target of processing. The control unit 10 ensures that an image effect, as an effect appropriate to the content scene created by the target object 3, is executed, as indicated by the effect content according to the effect generation conditions. This allows for the application of appropriate image effects to the captured image of the target object 3, taking the effect generation conditions into consideration. Furthermore, by registering various effect generation conditions and effect content for a single target object 3, a variety of images using the target object 3 can be used as effects.

[0177] In this embodiment, when the control unit 10 selects multiple effect target objects 20 from the object information definition table 60 for a scene shown in the content information definition table 50, it determines which effect target object 20 to process based on a priority order derived from the distance from the user. For example, when determining the matching of a scene and an effect target object 20 based on the commonality of scene attributes in the content information definition table 50 and the object information definition table 60, there may be cases where multiple effect target objects 20 are applicable. In such cases, the control unit 10 can prioritize selecting the effect target object 20 that is closest to the user and make it the target of the effect processing (see steps S134 and S135 in Figure 21). This allows the user to experience the effect at as close a position as possible. This means that the effect is executed using an object in the viewing space image 42 that is as close as possible to the content image 41 within the displayed image 40, that is, an object that is easily visible in a video see-through state, thereby increasing the visibility of the effect to the user.

[0178] In this embodiment, an example was given in which the control unit 10 determines to execute an effect targeting multiple effect targets 20 when multiple effect targets 20 are selected from the object information definition table 60 for a scene shown in the content information definition table 50. For example, as in the processing of the first embodiment shown in Figures 15 and 17, when determining the matching of a scene and an effect target object 20 based on the commonality of scene attributes in the content information definition table 50 and the object information definition table 60, there may be cases where multiple effect target objects 20 are applicable. In such cases, the control unit 10 can make each effect target object 20 a target of effect processing. This allows the user to experience effects from multiple effect target objects 20 simultaneously.

[0179] In this embodiment, an example was given in which the control unit 10, when viewing of visual content begins, controls the display of an introductory image selected according to the attributes of the visual content to be viewed, according to the position of objects present in the viewing space (see Figures 24 to 27). For example, when viewing visual content using the HMD1's video see-through feature, an introductory animation into the content's world is displayed along the user's viewing space before the content is viewed. This makes it possible to naturally guide the user from the real world into the world of the content.

[0180] In this embodiment, the control unit 10 performs a search process for objects that can become effect targets 20 in the viewing space, and registers the discovered objects that can become effect targets 20 in the object information definition table 60 (see Figures 22 and 23). This allows for the appropriate setting and use of the effect target 20 when there is no effect target object 20 suitable for the scene, or when viewing in a new environment. Furthermore, the controlled devices 2, such as IoT devices, can be recognized by the control unit 10 through mutual communication. Therefore, for example, if a new controlled device 2 is introduced into a room, the control unit 10 can recognize it through communication and register it as an effect target object 20.

[0181] In this embodiment, the control unit 10 performs object recognition from captured images of the surroundings as part of the object search process, and determines whether the recognized object can be an effect target object 20. This makes it possible to designate items present in the user's viewing space as effect target objects 20.

[0182] In this embodiment, an example was given in which the control unit 10 performs execution control of presentations requesting user actions for the search process. For example, the system can notify the user wearing the HMD1 to walk around the room or look around. When the user responds, the imaging unit 12 can capture images of the room, making it easier to search for objects in the room from the captured images.

[0183] In this embodiment, an example was given in which the control unit 10 performs execution control of a presentation requesting the placement of an object that may be the target of the effect 20 within the viewing space. For example, when viewing in a new room, or when a suitable object that can serve as the effect target 20 cannot be found, it is recommended to purchase an object that can serve as the effect target 20, bring it into the room, and place it there. This encourages users to create a viewing space that enhances immersion through the effect.

[0184] In this embodiment, an information processing device configured as HMD1 was given as an example. The HMD1 is configured to have a control unit 10 and other components located within it, so that the HMD1 functions as an information processing device that performs the processing described in the embodiment. This allows the user to build a system that realizes effects using effect target objects 20 in the viewing space using only the HMD1.

[0185] The viewing system of this embodiment comprises an effect target object 20, which is a real object in the viewing space; a control unit 10; and a display unit 16 that allows the user to see the viewing space along with the playback image of the visual content. For example, it is configured as shown in Figures 1 and 4. Such a viewing system can provide the user with a highly immersive viewing space.

[0186] In this embodiment, the viewing space is an environment for playing both images and sound, but it may also be an environment for playing only images.

[0187] The program of this embodiment is a program that causes various processors such as CPUs, GPUs, GPGPUs, AI processors, or information processing devices including these to perform the processes shown in Figures 10, 15, 16, 17, 21, 22, 23, 26, and 27. In other words, the program of the embodiment is a program that causes an information processing device to execute a determination process that determines an effect that corresponds to a scene in the visual content and targets the effect object 20, based on content information that includes scene attribute information for some or all of the scenes in the visual content and object information that includes scene attribute information for an object that is an effect target 20 among the objects that actually exist in the viewing space, and a control process that executes the determined effect in accordance with the corresponding scene in the visual content.

[0188] Such a program can realize the HMD1 of the embodiment, and the effect control device 4 of the embodiment can be realized in, for example, a computer device, a mobile terminal device, or other device capable of performing information processing.

[0189] Such programs can be pre-recorded in storage media such as HDDs (hard disk drives) built into computer devices, or in ROM (remote-controlled memory) within microcomputers with CPUs. Alternatively, programs can be temporarily or permanently stored (recorded) on removable recording media such as flexible disks, CD-ROMs (Compact Disc Read Only Memory), MO (Magneto Optical) disks, DVDs (Digital Versatile Discs), Blu-ray Discs (registered trademark), magnetic disks, semiconductor memory, and memory cards. Such removable recording media can be provided as so-called packaged software. In addition to being installed from the removable recording media to a personal computer, such programs can also be downloaded from download sites via networks such as LANs (Local Area Networks) and the Internet.

[0190] Furthermore, such a program is suitable for providing a wide range of information processing devices according to the embodiment. For example, by downloading the program to mobile terminal devices such as smartphones and tablets, imaging devices, mobile phones, personal computers, game consoles, video equipment, PDAs (Personal Digital Assistants), etc., these devices can be made to function as information processing devices according to the disclosure.

[0191] Furthermore, the effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur.

[0192] Furthermore, this technology can also be configured as follows. (1) A determination process that determines an effect that corresponds to a scene in the visual content and targets the target object, based on content information including scene attribute information for some or all scenes in the visual content and object information including scene attribute information for an object that is an object in the viewing space that is designated as an effect target. The control unit includes a control process that executes an effect in accordance with the corresponding scene of the visual content based on the determination of the determination process. Information processing device. (2) It features a display unit that allows the user to visualize the viewing space along with the playback image of the visual content. The information processing device described in (1) above. (3) The system includes an image processing unit that takes as input images of playback visual content and images of the viewing space captured by an imaging device, and is capable of performing effect processing on the captured images and a synthesis process that combines the captured images and the playback images. The information processing device described in (1) or (2) above. (4) The control unit selects an effect target object from the object information whose scene attribute information matches the scene shown in the content information, and determines that it will execute an effect targeting the selected effect target object. An information processing device as described in any of (1) to (3) above. (5) The aforementioned object information includes information on the effect generation conditions for the object to be affected and the effect content corresponding to the effect generation conditions. The control unit determines whether the effect generation conditions are met for the effect target object included in the object information and determines the effect content. An information processing device as described in any of (1) to (4) above. (6) The object information includes controlled devices that are capable of corresponding to the control signals transmitted by the control unit as effect targets. When the control unit is processing the controlled device, it performs a control process that causes the controlled device to operate in accordance with the corresponding scene of the visual content. An information processing device as described in any of (1) to (5) above. (7) The object information includes controlled devices that can respond to the control signals transmitted by the control unit as effect targets, and also includes information on effect generation conditions for the effect targets and the effect content corresponding to the effect generation conditions. When the control unit is processing the controlled device, the control unit performs a process that causes the controlled device to perform the operation indicated in the effect content according to the effect generation conditions, in accordance with the relevant scene of the visual content. An information processing device as described in any of (1) to (6) above. (8) The operation of the controlled device controlled by the control unit is an operation that causes a change in the viewing space. The information processing device described in (6) or (7) above. (9) The object information includes objects that cannot be controlled by the control unit as objects to be affected. When the control unit is to process the object, it performs a control process that causes the control unit to execute an image effect on the captured image of the object in accordance with the corresponding scene of the visual content. An information processing device as described in any of (1) to (8) above. (10) The object information includes objects whose operation cannot be controlled by the control unit as effect targets, and also includes information on effect generation conditions for effect targets and the content of the effect according to the effect generation conditions. When the control unit is to process the object, the control unit performs the following control process: In accordance with the scene of the visual content, it performs the image effect processing indicated by the effect content according to the effect generation conditions on the captured image of the object. An information processing device as described in any of (1) to (9) above. (11) When the control unit, in relation to a scene shown in the content information, selects multiple effect targets from the object information, the determination process determines which effect targets to process based on priority derived from the distance from the user. An information processing device as described in any of (1) to (10) above. (12) The control unit determines, in the determination process, that if multiple effect targets are selected from the object information for a scene shown in the content information, it will execute an effect that targets multiple effect targets. An information processing device as described in any of (1) to (10) above. (13) The control unit, upon the start of viewing visual content, controls the display of an introductory image selected according to the attributes of the visual content to be viewed, according to the position of objects present in the viewing space. An information processing device as described in any of (1) to (12) above. (14) The control unit, The process involves searching for objects in the viewing space that could potentially be targeted by effects, and registering the discovered objects as target objects in the object information. An information processing device as described in any of (1) through (13) above. (15) The control unit, As part of the aforementioned search process, object recognition is performed from the captured images of the surroundings, and it is determined whether or not the recognized objects can be targets for the effect. The information processing device described in (14) above. (16) The control unit, The system controls the execution of a presentation requesting user action for the aforementioned search process. The information processing device described in (14) or (15) above. (17) The control unit, This controls the system to execute a presentation requesting the placement of objects that could be the target of an effect within the viewing space. An information processing device as described in any of (14) to (16) above. (18) Configured as a head-mounted display An information processing device as described in any of (1) through (17) above. (19) Information processing device, A determination process that determines an effect that corresponds to a scene in the visual content and targets the target object, based on content information including scene attribute information for some or all scenes in the visual content and object information including scene attribute information for an object that is an object in the viewing space that is designated as an effect target. A control process is performed to execute an effect in accordance with the scene of the corresponding visual content based on the determination of the aforementioned determination process. Information processing methods. (20) The effect target is an object that actually exists in the viewing space, A control unit that performs a determination process that determines an effect that corresponds to a scene in the visual content and targets the target object, based on content information including scene attribute information for some or all scenes in the visual content and object information including scene attribute information for the object targeted by the effect, and a control process that executes the effect in accordance with the corresponding scene in the visual content based on the determination of the determination process. A display unit that allows the user to see the viewing space along with the playback image of the visual content, A viewing system equipped with [features / equipment]. [Explanation of Symbols]

[0193] 1 HMD 2 Controlled equipment 3. Target object 4. Effect Control Device 10 Control Unit 11 Image Processing Unit 11a Content Image Processing Unit 11b Image processing unit 11c Effects Processing Unit 11d Synthesis Processing Unit 12 Imaging Unit 13 Sensor section 14 Communications Department 15 Storage section 16 Display section 17 Audio output section 20. Effect Targets 40 Displayed Images 41 Content Images 42 Viewing Space Image 50 Content Information Definition Table 60 Object information definition table 100 Information Input Section 200 Determination Processing Unit 300 Effect Control Unit 400 Information Management Department 401 Object information definition section 402 Content Information Definition Section

Claims

1. A determination process that determines an effect that corresponds to a scene in the visual content and targets the target object, based on content information including scene attribute information for some or all scenes in the visual content and object information including scene attribute information for an object that is an object in the viewing space that is designated as an effect target. The control unit includes a control process that executes an effect in accordance with the corresponding scene of the visual content based on the determination of the determination process. Information processing device.

2. It features a display unit that allows the user to visualize the viewing space along with the playback image of the visual content. The information processing apparatus according to claim 1.

3. The system includes an image processing unit that takes as input images of playback visual content and images of the viewing space captured by an imaging device, and is capable of performing effect processing on the captured images and a synthesis process that combines the captured images and the playback images. The information processing apparatus according to claim 1.

4. The control unit selects an effect target object from the object information whose scene attribute information matches the scene shown in the content information, and determines that it will execute an effect targeting the selected effect target object. The information processing apparatus according to claim 1.

5. The aforementioned object information includes information on the effect generation conditions for the object to be affected and the effect content corresponding to the effect generation conditions. The control unit determines whether the effect generation conditions are met for the effect target object included in the object information and determines the effect content. The information processing apparatus according to claim 1.

6. The object information includes controlled devices that are capable of corresponding to the control signals transmitted by the control unit as effect targets. When the control unit is processing the controlled device, it performs a control process that causes the controlled device to operate in accordance with the corresponding scene of the visual content. The information processing apparatus according to claim 1.

7. The object information includes controlled devices that can respond to the control signals transmitted by the control unit as effect targets, and also includes information on effect generation conditions for the effect targets and the effect content corresponding to the effect generation conditions. When the control unit is processing the controlled device, the control unit performs a process that causes the controlled device to perform the operation indicated in the effect content according to the effect generation conditions, in accordance with the relevant scene of the visual content. The information processing apparatus according to claim 1.

8. The operation of the controlled device controlled by the control unit is an operation that causes a change in the viewing space. The information processing apparatus according to claim 6.

9. The object information includes objects that cannot be controlled by the control unit as objects to be affected. When the control unit is to process the object, it performs a control process that causes the control unit to execute an image effect on the captured image of the object in accordance with the corresponding scene of the visual content. The information processing apparatus according to claim 1.

10. The object information includes objects whose operation cannot be controlled by the control unit as effect targets, and also includes information on effect generation conditions for effect targets and the content of the effect according to the effect generation conditions. When the control unit is to process the object, the control unit performs the following control process: In accordance with the scene of the visual content, it performs the image effect processing indicated by the effect content according to the effect generation conditions on the captured image of the object. The information processing apparatus according to claim 1.

11. When the control unit, in relation to a scene shown in the content information, selects multiple effect targets from the object information, the determination process determines which effect targets to process based on priority derived from the distance from the user. The information processing apparatus according to claim 1.

12. The control unit determines, in the determination process, that if multiple effect targets are selected from the object information for a scene shown in the content information, it will execute an effect that targets multiple effect targets. The information processing apparatus according to claim 1.

13. The control unit, upon the start of viewing visual content, controls the display of an introductory image selected according to the attributes of the visual content to be viewed, according to the position of objects present in the viewing space. The information processing apparatus according to claim 1.

14. The control unit, The process involves searching for objects in the viewing space that could potentially be targeted by effects, and registering the discovered objects as target objects in the object information. The information processing apparatus according to claim 1.

15. The control unit, As part of the aforementioned search process, object recognition is performed from the captured images of the surroundings, and it is determined whether or not the recognized objects can be targets for the effect. The information processing apparatus according to claim 14.

16. The control unit, The system controls the execution of a presentation requesting user action for the aforementioned search process. The information processing apparatus according to claim 14.

17. The control unit, This controls the system to execute a presentation requesting the placement of objects that could be the target of an effect within the viewing space. The information processing apparatus according to claim 14.

18. Configured as a head-mounted display The information processing apparatus according to claim 1.

19. Information processing device, A determination process that determines an effect that corresponds to a scene in the visual content and targets the target object, based on content information including scene attribute information for some or all scenes in the visual content and object information including scene attribute information for an object that is an object in the viewing space that is designated as an effect target. A control process is performed to execute an effect in accordance with the scene of the corresponding visual content based on the determination of the aforementioned determination process. Information processing methods.

20. The effect target is an object that actually exists in the viewing space, A control unit that performs a determination process that determines an effect that corresponds to a scene in the visual content and targets the target object, based on content information including scene attribute information for some or all scenes in the visual content and object information including scene attribute information for the object targeted by the effect, and a control process that executes the effect in accordance with the corresponding scene in the visual content based on the determination of the determination process. A display unit that allows the user to see the viewing space along with the playback image of the visual content, A viewing system equipped with [features / equipment].

Images