Entertainment system and program

Abstract

To provide an entertainment system or the like that can notify a user of content information related to a first display device in a system using the first display device and a second display device.SOLUTION: An entertainment system using a first display device and a second display device in which a display image of the first display device can be visually recognized includes: a gaze area detection unit that detects a user's gaze area on a display screen of the first display device based on information on a user's line-of-sight direction; and a content processing unit that performs processing to cause the user to visually recognize a content image to which a display image of the second display device is added to the display image of the first display device. The content processing unit performs notification processing for notifying the user of content information corresponding to the detected gaze area by using the second display device.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an entertainment system, a program, and the like. [Background technology]

[0002] Conventionally, systems have been known that allow users to view content images that are a combination of two display images. The system disclosed in Patent Document 1 includes a transmissive monitor that displays a transparent image that transmits a specific color, and a projector that projects a basic screen that forms a game screen by combining the transparent image onto the rear of the transmissive monitor. An information-providing image is then formed by combining the transparent image and the basic screen. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-146913 Summary of the Invention [Problem to be solved by the invention]

[0004] However, until now, no system has been proposed that controls the first display device and the second display device in a linked manner to notify the user of content information related to the display objects, etc., of the first display device.

[0005] According to the present disclosure, it is possible to provide an entertainment system, a program, and the like that can notify a user of content information related to a first display device in a system using a first display device and a second display device. [Means for solving the problem]

[0006] One aspect of the present disclosure relates to an entertainment system using a first display device and a second display device on which a display image of the first display device is visible, the entertainment system including: a gaze area detection unit that detects a gaze area of the user on a display screen of the first display device based on information on the user's line of sight; and a content processing unit that processes a content image in which a display image of the second display device is added to the display image of the first display device, the content processing unit performing a notification process to notify the user of content information corresponding to the detected gaze area using the second display device. Another aspect of the present disclosure relates to a program that causes a computer to function as each of the above units, or a computer-readable information storage medium that stores the program.

[0007] According to one aspect of the present disclosure, a user's gaze area on the display screen of a first display device is detected. A process is performed in which the user is allowed to view a content image in which the display image of the second display device is added to the display image of the first display device. At this time, a notification process is performed in which content information corresponding to the gaze area is notified to the user using the second display device. This allows the user to use the second display device to learn content information corresponding to the gaze area on which the user gazed on the display screen of the first display device. Therefore, it is possible to provide an entertainment system or the like that can notify the user of content information related to the first display device in a system using the first and second display devices.

[0008] In one aspect of the present disclosure, the notification process may be a process of displaying the content information using the second display device.

[0009] In this way, the user can visually recognize the content information corresponding to the gaze area that the user gazes at on the display screen of the first display device, from the display image of the second display device.

[0010] In one aspect of the present disclosure, the content processing unit may change the notification mode of the content information depending on information of an object displayed on the first display device or the display mode of the first display device.

[0011] In this way, in an entertainment system that allows a user to view a content image in which a display image of a second display device is added to a display image of a first display device, it becomes possible to notify the user of appropriate content information that corresponds to the display object or display mode of the first display device.

[0012] In another aspect of the present disclosure, the content processing unit may perform display control of an object displayed on the second display device according to information about the display object of the first display device or the display mode of the first display device.

[0013] In this way, an object whose display is controlled according to the display state or the display object of the first display device can be displayed on the second display device, and the user can view a content image in which an image of the object is added to the display image of the first display device.

[0014] In one aspect of the present disclosure, the information about the display object is information about a gaze display object that the user is gazed at on the display screen of the first display device, and the content processing unit may set the display position of the object displayed on the second display device according to the information about the gaze display object or the display mode of the first display device.

[0015] In this way, the object can be displayed at a display position that corresponds to the information of the gaze display object of the first display device or the display mode of the first display device, allowing the user to view a content image in which the image of the object has been added to the display image of the first display device.

[0016] In one aspect of the present disclosure, the information about the display object is information about a gaze display object that the user is gazed at on the display screen of the first display device, and the content processing unit may set the display mode of the object to be displayed on the second display device according to the information about the gaze display object or the display mode of the first display device.

[0017] In this way, the object is displayed in a display manner that corresponds to the information or display manner of the gaze display object of the first display device, and the user can view a content image in which the image of the object is added to the display image of the first display device.

[0018] Another aspect of the present disclosure relates to an entertainment system using a first display device and a second display device on which a display image of the first display device is visible, the entertainment system including an information acquisition unit that acquires information about an object displayed on the first display device or information about a display mode of the first display device, and a content processing unit that processes a content image in which a display image of the second display device is added to the display image of the first display device and the user is allowed to view the content image, the content processing unit performs a notification process to notify the user of content information using the second display device, and changes the notification mode of the content information depending on the information about the object of the first display device or the display mode of the first display device. Another aspect of the present disclosure relates to a program that causes a computer to function as each of the above units, or a computer-readable information storage medium that stores the program.

[0019] According to one aspect of the present disclosure, information on an object or a display mode displayed on a first display device is acquired. A process is performed to allow a user to view a content image in which a display image of a second display device is added to a display image of the first display device. At this time, a process is performed to change the notification mode of the content information according to the information on the object or the display mode of the first display device. In this way, in an entertainment system in which a user views a content image in which a display image of a second display device is added to a display image of the first display device, it is possible to notify the user of appropriate content information according to the object or the display mode of the first display device. Therefore, it is possible to provide an entertainment system, etc., that can notify a user of content information related to the first display device in a system using a first display device and a second display device.

[0020] In another aspect of the present disclosure, the content processing unit may perform display control of an object displayed on the second display device according to information about the display object of the first display device or the display mode of the first display device.

[0021] In this way, an object whose display is controlled according to the display state or the display object of the first display device can be displayed on the second display device, and the user can view a content image in which an image of the object is added to the display image of the first display device.

[0022] In another aspect of the present disclosure, the content processing unit may set the display position of the object displayed on the second display device according to information about the display object of the first display device or the display mode of the first display device.

[0023] In this way, the object can be displayed at a display position corresponding to the information or display mode of the display object of the first display device, allowing the user to view a content image in which the image of the object has been added to the display image of the first display device.

[0024] In another aspect of the present disclosure, the content processing unit may set the display mode of the object to be displayed on the second display device according to information about the display object of the first display device or the display mode of the first display device.

[0025] In this way, the object is displayed in a display manner that corresponds to the information or display manner of the display object of the first display device, and the user can view a content image in which the image of the object is added to the display image of the first display device.

[0026] In one aspect of the present disclosure, the object displayed on the second display device may be an object related to the display object displayed on the first display device.

[0027] In this way, an object related to an object displayed on the first display device can be displayed on the second display device as an object superimposed on the display image of the first display device.

[0028] In one aspect of the present disclosure, the information about the object of the first display device may be parameter information about the object, display position information about the object, or display mode information about the object.

[0029] In this way, it becomes possible to change the notification mode of the content information notified using the second display device in accordance with the parameter information, display position information, or display mode information of the display object.

[0030] Another aspect of the present disclosure relates to an entertainment system using a first display device and a second display device on which a display image of the first display device can be viewed, the entertainment system including: an image analysis unit that performs an analysis process on a captured image of the first display device taken by a camera of the second display device; and a content processing unit that performs a process of allowing the user to view a content image in which the display image of the second display device is added to the display image of the first display device, the content processing unit performing a notification process of notifying the user of content information based on the result of the analysis process of the captured image using the second display device. Another aspect of the present disclosure relates to a program that causes a computer to function as each of the above units, or a computer-readable information storage medium that stores the program.

[0031] According to one aspect of the present disclosure, an analysis process is performed on a captured image of the first display device using a camera included in the second display device. A process is also performed in which a user is allowed to view a content image in which the display image of the second display device is added to the display image of the first display device. At this time, a notification process is performed in which the second display device is used to notify the user of content information based on the results of the analysis process on the captured image. This allows the user to use the second display device to learn content information based on the results of the analysis process on the captured image obtained by capturing an image of the first display device using the camera of the second display device. Therefore, it is possible to provide an entertainment system or the like that can notify the user of content information related to the first display device in a system using the first and second display devices.

[0032] In another aspect of the present disclosure, the content processing unit may change the display content of the first display device or the second display device, or the notification mode of the content information, depending on the positional relationship between the first display device and the second display device.

[0033] In this way, an image can be displayed on the first display device or the second display device with appropriate display content according to the relative positions of the first and second display devices, or content information can be notified to the user by the second display device in an appropriate notification manner according to the relative positions of the first and second display devices.

[0034] In another aspect of the present disclosure, the content processing unit may change the display content of the first display device or the second display device, or the notification mode of the content information, depending on the positional relationship between the first display device and the second display device and information about the display object displayed on the first display device.

[0035] In this way, an image can be displayed on the first display device or the second display device with appropriate display content according to the positional relationship between the first and second display devices and the information of the displayed object on the first display device, or content information can be notified to the user by the second display device in an appropriate notification manner according to the positional relationship and the information of the displayed object. [Brief explanation of the drawings]

[0036] [Figure 1] 1 shows an example of a system configuration of an entertainment system according to an embodiment of the present invention. [Figure 2] FIG. 10 is an explanatory diagram of a content image in which a display image of a second display device is added to a display image of a first display device. [Figure 3] FIG. 10 is an explanatory diagram of a notification process for notifying a user of content information corresponding to the user's gaze area using a second display device. [Figure 4] 10A and 10B are explanatory diagrams of a notification process of content information using vibration and sound. [Figure 5] FIG. 2 is a diagram showing a see-through HMD as an example of a second display device. [Figure 6] 1 is a block diagram showing an example of the configuration of an entertainment system according to an embodiment of the present invention. [Figure 7] 7A and 7B are diagrams showing an example of the hardware configuration of the entertainment system of this embodiment. [Figure 8]4 is a flowchart illustrating a processing example of the present embodiment. [Figure 9] 4 is a flowchart illustrating a processing example of the present embodiment. [Figure 10] 10 is a flowchart illustrating a detailed processing example of the present embodiment. [Figure 11] 11(A) and 11(B) are explanatory diagrams of a method for setting the display position of an object on the second display device in accordance with the information or display mode of a display object on the first display device. [Figure 12] FIG. 10 is an explanatory diagram of a method for setting the display mode of an object on a second display device in accordance with the information or display mode of a display object on a first display device. [Figure 13] FIG. 4 is an explanatory diagram of information displayed on the first display device. [Figure 14] 10 is an explanatory diagram of a method for changing the display content of the first display device or the second display device, or the notification mode of content information, depending on the positional relationship between the first display device and the second display device. FIG. [Figure 15] 15(A) and 15(B) are explanatory diagrams of the process of acquiring real space information through the process of recognizing the real space. [Figure 16] FIG. 10 is an explanatory diagram of a method for detecting the positional relationship between the first display device and the second display device using recognition processing of the real space. [Figure 17] FIG. 10 is an explanatory diagram of a technique for making a user perceive a displayed object as if it has moved from the first display device to the second display device. DETAILED DESCRIPTION OF THE INVENTION

[0037] The present embodiment will be described below. Note that the present embodiment described below does not unduly limit the content of the claims. Furthermore, not all of the configurations described in the present embodiment are necessarily essential components.

[0038] 1. Entertainment System FIG. 1 shows an example of the system configuration of an entertainment system according to this embodiment. The entertainment system according to this embodiment is realized using a first display device 10 and a second display device 200. For example, the entertainment system according to this embodiment is a system that provides various entertainment contents to a user using the first display device 10 and the second display device 200, on which the display image of the first display device 10 can be viewed. The first display device 10 is, for example, a main display device. Specifically, the first display device 10 may be a monitor (display) of an information processing device such as a personal computer (PC), a television monitor, a game device monitor, or a large monitor installed at a venue such as a live concert or club. In FIG. 1, the first display device 10 displays an image of game content in which characters CH1, CH2, and CH3 compete against characters CE1, CE2, and CE3. The second display device 200 is, for example, a sub-display device. For example, the second display device 200 is a display device on which the display image of the first display device 10 can be viewed. For example, a user can view the display image of the first display device 10 through the display unit of the second display device 200. 1, a see-through HMD (HMD in a broad sense, see-through display device) is used as the second display device 200. The see-through HMD is a see-through head mounted display worn by a user. The see-through HMD is, for example, an optically see-through HMD.

[0039] Note that, although the following description will be given mainly taking as an example a case where the second display device 200 is a transmissive HMD, the second display device 200 may be a transmissive display device other than a transmissive HMD. For example, a transmissive HMD is a display device that a user wears on their head, but the second display device 200 may be a transmissive display device that the user does not wear on their head. For example, the second display device 200 may be a transmissive display device that the user holds in their hand or places in a desired location for use. Alternatively, the second display device 200 may be a display device other than a transmissive display device, and may be, for example, a display device that displays an image of the first display device 10 captured by a camera on a display unit, thereby enabling the display image of the first display device 10 to be viewed.

[0040] FIG. 2 is an explanatory diagram of a content image in which a display image IMH of the second display device 200 is added to a display image IMD of the first display device 10. In this embodiment, a process is performed to allow a user to view a content image in which a display image IMH of the second display device 200 is added (superimposed) to a display image IMD of the first display device 10. For example, if the second display device 200 is a see-through HMD, the display image IMD of the first display device 10 is viewed by the user after passing through an optical element of the second display device 200. The display image IMH of the second display device 200 is displayed superimposed on the display image IMD of the first display device 10 that passes through the second display device 200. The display image IMH of the second display device 200 is, for example, a virtual image displayed by the second display device 200. Taking FIG. 5, which will be described later, as an example, the display image IMH is a display image of the display device 243 of the see-through HMD 202, which is the second display device 200. This allows the user to visually recognize a content image in which the display image IMH of the second display device 200 is added to the display image IMD of the first display device 10.

[0041] 2 shows an example of an image of game content as the content image, however, the content image of this embodiment is not limited to such an image of game content, and may be an image of various content (visual content) such as streaming video, movie, environmental video, sightseeing video, or simulation video.

[0042] Furthermore, if the second display device 200 is a transparent display device other than a transparent HMD, such as a transparent tablet terminal, the display image IMD of the first display device 10 is displayed to the user through the transparent tablet terminal. The display image IMD transmitted through the transparent tablet terminal and the display image IMH generated by the transparent tablet terminal are displayed to the user, allowing the user to view a content image in which the display image IMH of the second display device 200 is added to the display image IMD (transparent image) of the first display device 10. If the second display device 200 is a terminal device other than a transparent display device, such as a smartphone or a portable game device, the display image IMD of the first display device 10 is captured by a camera, for example. The display image IMH generated by the second display device 200 is superimposed on the image captured by the camera, allowing the user to view a content image in which the display image IMH of the second display device 200 is added to the display image IMD (captured image) of the first display device 10.

[0043] 2, the second display device 200 displays content information related to the first display device 10 as a display image IMH. The content information related to the first display device 10 is, for example, content information related to an object displayed on the first display device 10 or the display content of the first display device 10. For example, the first display device 10 displays a character CH1 or the like as an object. The content information related to this character CH1 is displayed as a display image IMH on the second display device 200, and the display image IMH of this content information is displayed superimposed on the display image IMD of the first display device 10. In FIG. 2, the content information notified by the display image IMH is game information, that is, game information related to the character CH1, which is the object displayed on the first display device 10. Specifically, the game information, which is content information, is information such as the hit points (HP), magic points (MP), experience points (EXP), level (LV), and equipment (EQP) of the character CH1. In the display image IMH, information on hit points (HP) is displayed by a hit point gauge object OB1, information on magic points (MP) is displayed by a magic point gauge object OB2, information on experience points (EXP) and level (LV) is displayed by numerical objects OB3 and OB4 representing experience points and level, respectively, and information on equipment (EQP) is displayed by an icon object OB5 representing equipment.

[0044] As described above, in this embodiment, a notification process is performed in which content information related to the display object or display content of the first display device 10 is notified to the user using the second display device 200. Taking FIG. 2 as an example, content information (HP, MP, EXP, LV, EQP) related to the character CH1, which is the display object of the first display device 10, is notified to the user using the second display device 200. Specifically, the display image IMH of the content information related to the character CH1 is displayed using the second display device 200 (display device 243) and is superimposed on the display image IMD of the first display device 10 so that the user can view it, thereby notifying the user of the content information. That is, the content information notification process in this case is a process in which the content information (display image IMH) is displayed using the second display device 200.

[0045] As described above, in this embodiment, a process is performed in which the user visually recognizes a content image (FIG. 2) in which the display image IMH of the second display device 200 is added to the display image IMD of the first display device 10. Furthermore, in this embodiment, the user's gaze area on the display screen of the first display device 10 is detected based on information about the user's line of sight. Then, a notification process is performed in which the second display device 200 is used to notify the user of content information corresponding to the detected gaze area.

[0046] For example, in FIG. 3, the gaze direction LS of the user US is detected. The gaze direction LS of the user US can be detected by detecting, for example, the direction (facing direction) of the second display device 200. Specifically, information on the gaze direction LS can be acquired by tracking the direction of the second display device 200, etc. Alternatively, the gaze direction LS of the user US may be detected by gaze tracking processing (eye tracking processing) that directly detects the gaze of the user US. Alternatively, when the first display device 10 is photographed by a camera, the gaze direction LS of the user US may be detected by considering the shooting direction of the camera as the gaze direction LS of the user US. In this case, the camera can be considered as the user US, and the gaze area of the user US is the gaze area of the camera. Alternatively, the facing direction of a transparent display device such as a transparent tablet terminal may be considered as the gaze direction of the user US. In this case, the gaze area of the user US is the gaze area in the facing direction of the transparent display device (the area where a line along the facing direction intersects with the display screen of the first display device 10).

[0047] As shown in FIG. 3, the gaze direction LS of the user US is detected, thereby detecting the gaze area ARG of the user US on the display screen of the first display device 10. For example, the gaze area ARG (the gaze area of the camera and the transmissive display device) can be detected by determining the position where an extension of the gaze direction LS (the shooting direction of the camera, the direction in which the transmissive display device faces) intersects with the display screen of the first display device 10. In this case, it is desirable to determine the positional relationship between the first display device 10 and the second display device 200 (user, camera). For example, the relative positional relationship of the first display device 10 as seen from the second display device 200 is determined, and the gaze area ARG (attention area), which is the area gazed (attention) by the user US, is detected based on information on this positional relationship and information on the gaze direction LS. Then, a notification process is performed in which content information corresponding to the detected gaze area ARG is notified to the user US using the second display device 200. In FIG. 3, the notification process is performed in which a display image IMH, which is content information corresponding to the gaze area ARG, is superimposed on a display image IMD of the first display device 10. 3, for example, the user US is gazing at the character CH1 in the display image IMD of the first display device 10, and the display area of this character CH1 is the gaze area ARG. Therefore, as content information corresponding to the gaze area ARG in which the character CH1 is displayed, character status information such as the hit points, magic points, experience points, level, and equipment of the character CH1 is displayed as a display image IMH (virtual image) of the second display device 200, as shown in FIG. 2. That is, images of objects OB1, OB2, OB3, OB4, and OB5 representing the hit points, magic points, experience points, level, and equipment of the character CH1 are displayed as a display image IMH of the display device 243 of the second display device 200, superimposed on the display image IMD of the first display device 10.

[0048] In this way, when the user US gazes at the character CH1 of interest in the display image IMD of the first display device 10, the user US is notified of content information related to the character CH1, such as hit points, magic points, experience points, level, and equipment. This allows the user US to learn various content information related to the character CH1 of interest simply by gazing at the character CH1. Furthermore, if the display image IMH of content information (status information) such as hit points and magic points for all characters CH1 to CH3 and CE1 to CE3 displayed on the first display device 10 were superimposed on the display image IMD, the display of this content information would be distracting and difficult for the user US to view. In this regard, displaying only the content information corresponding to the user US's gaze area ARG can prevent the display of content information from being distracting and difficult for the user US to view.

[0049] As described above, the notification process of FIG. 3 is a process of displaying content information using the second display device 200. By performing a content information display process as the content information notification process, the user US can easily visually recognize the content of the content information corresponding to the gaze area ARG. However, the notification process of this embodiment is not limited to this content information display process. For example, in FIG. 4, the notification process is performed using vibration or sound. Specifically, a rare character RCH appears on the display screen of the first display device 10. For example, the rare character RCH appears in the gaze area in the line of sight LS of the user US. When a specific display object such as the rare character RCH appears in the first display device 10 (gaze area) in this manner, the user US is notified of the appearance of the rare character RCH by vibrating a vibration device (vibration mechanism) provided on the second display device 200. Alternatively, the user US is notified of the appearance of the rare character RCH by sound from a speaker built into the second display device 200 or a speaker of headphones provided separately from the second display device 200. In this way, the notification process of content information related to the displayed object or display content of the first display device 10 (content information corresponding to the gaze area) may be a process of vibrating a vibration device of the second display device 200, or a process of outputting sound such as voice by a sound output unit 192 (FIG. 6) provided corresponding to the second display device 200. Alternatively, a light-emitting element visible to the user US may be provided in the second display device 200, and the notification process of the content information may be realized by light emission from this light-emitting element. In this way, various notification modes for the notification process of the content information can be assumed.

[0050] Furthermore, when multiple users, such as a first user and a second user, enjoy entertainment content, first content information corresponding to the first user may be notified to the first user using a second display device, such as a first see-through HMD, worn by the first user, and second content information corresponding to the second user may be notified to the second user using a third display device, such as a second see-through HMD, worn by the second user. Specifically, a first gaze area is detected based on information about the first user's line of sight, and first content information corresponding to the first gaze area is notified to the first user using the second display device, such as the first see-through HMD, worn by the first user. Also, a second gaze area is detected based on information about the second user's line of sight, and second content information corresponding to the second gaze area is notified to the second user using a third display device, such as a second see-through HMD, worn by the second user. In this way, for example, a first content image in which only the first content information is superimposed on the display image IMD of the first display device 10 is displayed to the first user, and a second content image in which only the second content information is superimposed on the display image IMD of the first display device 10 is displayed to the second user. Therefore, the first user can view only the first content information that the first user focuses on, and the second user can view only the second content information that the second user focuses on, making it possible to display content images in a display mode that is easy for each user to view.

[0051] Furthermore, although the above description has been given of the case where the content is game content, the content of this embodiment is not limited to this and may be content of streaming videos, movies, environmental videos, sightseeing videos, or simulation videos. For example, in the case of streaming video content in which a virtual character dances or sings, information about the character may be notified as content information. In the case of movie content, various explanatory screens, subtitle information, etc. may be notified as content information. In the case of sightseeing video content, for example, various tourist information in the user's gaze area may be notified as content information. In the case of simulation video content, various information about the subject of the simulation may be notified as content information.

[0052] Fig. 5 shows an example of a transmissive HMD 202 (HMD in a broad sense, transmissive display device) which is an example of the second display device 200. The transmissive HMD in Fig. 5 is an example of an optical transmissive HMD.

[0053] The see-through HMD 202 in FIG. 5 has a temple portion 240 and a goggle portion 242. The temple portion 240 has a built-in speaker. The goggle portion 242 is provided with a display device 243 and a holographic optical element 244. For example, the display device 243, the holographic optical element 244, etc. can realize the display portion 30 of the see-through HMD 202 (broadly speaking, the second display device). The display device 243 is provided with a display unit configured with a microdisplay, a mirror, a prism, etc. The display device 243 is provided with a display unit for the left eye and a display unit for the right eye, thereby realizing stereoscopic vision. Furthermore, by providing a pair of microdisplays in each of the display units for the left eye and the right eye, the position of the virtual image can be changed.

[0054] Display light from the display device 243 is guided in front of the user's eyes while being refracted inside the light guide plate of the holographic optical element 244. The holographic optical element 244 then refracts the display light toward the eyeball and sends the display light to the eyeball. This makes it appear as if a virtual image (a reconstructed image of a hologram) is in front of the user's eyes. The light guide plate of the holographic optical element 244 is also called a combiner, and the combiner, which is a half mirror, makes it possible to see a real image of the outside world overlap with a virtual image, making it possible to realize technologies such as AR (Augmented Reality) and MR (Mixed Reality).

[0055] The goggle unit 242 is also provided with an RGB camera 246, a depth camera 247, and environment recognition cameras 248 and 249. The RGB camera 246 can be used to capture an image of the user in the front direction. The depth camera 247 can be used to acquire depth information in the front direction. The goggle unit 242 is also provided with an emitter (not shown) for the depth camera. The environment recognition cameras 248 and 249 can be used to recognize the environment around the user. The goggle unit 242 is also equipped with an inertial measurement unit (IMU), which is a motion sensor composed of an acceleration sensor, a gyro sensor, and the like. Position information and direction information of the see-through HMD 202 are detected based on images captured by the camera provided in the goggle unit 242 and measurement information from the inertial measurement unit, thereby realizing head tracking. Detecting the position information and direction information of the see-through HMD 202 also makes it possible to acquire position information and direction information (gaze direction information) of the user. Furthermore, by processing using the RGB camera 246, depth camera 247, etc., it is possible to acquire position information (relative position information), direction information (relative direction information), or posture information of other users around the user. For example, this information is acquired by processing called Kinect (registered trademark). For example, by acquiring posture information, it becomes possible to detect what actions other users have performed.

[0056] 5, the display unit 30, which is realized by, for example, a display device 243, a holographic optical element 244, and the like, allows the user to view the display image IMD and the display image IMH. For example, the display image IMD of the first display device 10 can be viewed by the user as a transmission image (a real image of the outside world) by a combiner using the holographic optical element 244. Furthermore, the display image IMH can be viewed by the user as a virtual image by displaying it on the display device 243 of the transmission type HMD 202. This makes it possible to realize a process of allowing the user to view a content image in which the display image IMH (virtual image) of the transmission type HMD 202 is added to the display image IMD (transmission image) of the first display device 10.

[0057] The see-through HMD 202 is not limited to the configuration shown in Fig. 5, and various modifications are possible, for example, with respect to the configuration of the display unit 30, the method of displaying a virtual image, etc. When the see-through HMD 202 is used, the user's operation input may be realized by the user operating a controller (not shown), or by detecting by eye tracking or the like that the user's line of sight is directed toward an input display object. Alternatively, the user's operation input may be detected by detecting the user's head movement or the like using a motion sensor provided in the see-through HMD 202.

[0058] As mentioned above, the second display device 200 is not limited to the transparent HMD 202, but may be a transparent display device having a transparent display, such as a transparent tablet terminal that the user holds in their hand, or may be a terminal device with a camera function, such as a smartphone, a portable game device, or a tablet terminal.

[0059] Fig. 6 is a block diagram showing an example of the configuration of the entertainment system (game system, content providing system, processing system) of this embodiment. Note that the amusement system of this embodiment is not limited to the configuration shown in Fig. 6, and various modifications are possible, such as omitting some of the components or adding other components.

[0060] The operation unit 160 allows the user (player) to perform various inputs (operations). The operation unit 160 can be realized by, for example, operation buttons, directional keys, a keyboard, a joystick, a lever, or the like. Alternatively, as described above, the functions of the operation unit 160 may be realized using eye tracking, motion, or the like.

[0061] The imaging unit 162 (camera) captures an image of a subject (an object to be imaged). The imaging unit 162 can be realized by an image sensor such as a CCD or CMOS sensor, an optical system including a focus lens, etc. The imaging unit 162, which is a camera, is provided in the second display device 200, for example.

[0062] The storage unit 170 stores various types of information. The storage unit 170 functions as a work area for the processing unit 100, the communication unit 196, etc. Programs and data required for executing the programs are held in this storage unit 170. The functions of the storage unit 170 can be realized by a semiconductor memory (DRAM, VRAM), an HDD (hard disk drive), an SSD (Solid State Drive), an optical disk device, or the like. The storage unit 170 stores, for example, content information. The content information is information about images and sounds that make up the content. The image information of the content is, for example, information about objects (display objects) that make up the content.

[0063] The information storage medium 180 is a computer-readable medium that stores programs, data, and the like. The information storage medium 180 can be realized by an optical disc (DVD, BD, CD), a HDD, a semiconductor memory (ROM), or the like. The processing unit 100 performs various processes of this embodiment based on the programs (data) stored in the information storage medium 180. That is, the information storage medium 180 stores programs (programs that cause the computer to execute the processes of each unit) that cause a computer (a device including an input device, a processing unit, a storage unit, and an output unit) to function as each unit of this embodiment.

[0064] The first display device 10 is, for example, a main monitor, and is realized by a television monitor, a PC monitor, a game device monitor, or a monitor installed in a facility such as a live music venue or an amusement facility. The display unit 30 is a display provided in the second display device 200 (such as a see-through HMD 202). The first display device 10 and the display unit 30 can be realized by, for example, an LCD or an organic EL display. The sound output unit 192 outputs the sound generated by this embodiment, and its function can be realized by a speaker, headphones, or the like.

[0065] The communication unit 196 communicates with the outside (other devices) via a wired or wireless network, and its function can be realized by hardware such as a communication ASIC or a communication processor, or communication firmware.

[0066] Note that the programs (data) for causing a computer to function as each unit of this embodiment may be distributed from an information storage medium possessed by a server (host device) to information storage medium 180 (or storage unit 170) via a network and communication unit 196. Such use of an information storage medium by a server can also be included within the scope of this embodiment.

[0067] The processing unit 100 (processor) performs information acquisition processing, gaze area detection processing, image analysis processing, positional relationship detection processing, content processing, etc. based on programs and data stored in the storage unit 170 and operation information from the operation unit 160.

[0068] Each process of this embodiment performed by each unit of the processing unit 100 can be realized by a processor (a processor including hardware). For example, each process of this embodiment can be realized by a processor that operates based on information such as a program and a memory that stores information such as the program. The functions of each unit of the processor may be realized by separate hardware, or the functions of each unit may be realized by integrated hardware. For example, the processor may include hardware, which may include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the processor may be configured with one or more circuit devices (e.g., ICs, etc.) mounted on a circuit board, or one or more circuit elements (e.g., resistors, capacitors, etc.). The processor may be, for example, a central processing unit (CPU). However, the processor is not limited to a CPU, and various processors such as a graphics processing unit (GPU) or a digital signal processor (DSP) can be used. The processor may also be a hardware circuit using an ASIC. The processor may also include an amplifier circuit, a filter circuit, etc. that process analog signals. The memory (storage unit 170) may be a semiconductor memory such as an SRAM or a DRAM, or a register. Alternatively, the memory may be a magnetic storage device such as a hard disk drive (HDD), or an optical storage device such as an optical disk drive. For example, the memory stores computer-readable instructions, and the instructions are executed by the processor to realize the processing of each unit of the processing unit 100. The instructions here may be an instruction set constituting a program, or may be instructions that instruct the hardware circuitry of the processor to operate.

[0069] The processing unit 100 includes an information acquisition unit 102, a gaze area detection unit 104, an image analysis unit 105, a positional relationship detection unit 106, and a content processing unit 110. The content processing unit 110 includes a notification processing unit 112. As described above, each process of this embodiment executed by each of these units can be realized by a processor (or a processor and a memory). Note that various modifications are possible, such as omitting some of these components (each unit) or adding other components.

[0070] The information acquisition unit 102 performs processing to acquire various pieces of information required for processing by the processing unit 100. For example, the information acquisition unit 102 acquires information about the user's gaze direction. For example, the information about the user's gaze direction is acquired by a tracking process using the second display device 200. For example, by the tracking process, the direction (facing direction) of the second display device 200 is detected as the user's gaze direction. Alternatively, the user's gaze direction may be detected by eye tracking processing. The position and direction of the second display device 200 can be detected based on an image captured by a camera or measurement information from an inertial measurement unit. Alternatively, the second display device 200 may be provided with a light-emitting unit, and light from the light-emitting unit may be captured by an external camera or the like to detect the position and direction of the second display device 200. Alternatively, the position and direction of the second display device 200 may be detected by light from an external light-emitting unit being captured by a light-receiving unit (camera) provided in the second display device 200. The information acquisition unit 102 also acquires information about objects displayed on the first display device 10 or information about the display mode of the first display device 10. The information on the displayed object is, for example, position information on the displayed object, display mode information on the displayed object (color, shape, brightness, transparency, etc.), or, if the displayed object is a character, various information on the character (parameter information such as status parameters, etc.), etc. The information on the display mode of the first display device 10 is information on the brightness, color, contrast, etc. of the display screen of the first display device 10.

[0071] The gaze area detection unit 104 performs a process of detecting the user's gaze area. For example, the gaze area detection unit 104 detects the user's gaze area on the display screen of the first display device 10. For example, based on information about the gaze direction acquired by the information acquisition unit 102, the gaze area detection unit 104 detects the gaze area, which is the area the user is gazing at on the display screen of the first display device 10. For example, the gaze area detection unit 104 obtains an intersection between a line along the gaze direction and the display screen of the first display device 10, and detects the area including the obtained intersection as the user's gaze area.

[0072] The gaze area is not limited to the area that the user actually gazes at by directing his or her gaze direction. For example, in an example in which the display image of the first display device 10 is made visible by photographing the first display device 10 with the camera of the second display device 200, the user's gaze area can be the area gazed at by the camera. For example, the direction in which the camera is facing is considered to be the user's gaze direction, and the gaze area detection unit 104 determines the intersection between a line along the direction in which the camera is facing and the display screen of the first display device 10, and detects the area including the determined intersection as the user's gaze area. In this case, it is desirable that the content processing unit 110 displays an indicator, such as a marker, at the position of the intersection so that the user can visually identify the position of the intersection on the display screen of the first display device 10. Alternatively, in an example in which the display image of the first display device 10 is made visible by directing the second display device 200, which is a transparent display device such as a transparent tablet terminal, toward the first display device 10, the user's gaze area can be determined as the area gazed at by directing the second display device 200. That is, for example, the direction in which the second display device 200 faces is regarded as the user's line of sight. The direction in which the second display device 200 faces is, for example, a direction perpendicular to the main surface of the second display device 200, for example, a direction perpendicular to the display unit 30 (display screen). The gaze area detection unit 104 then obtains an intersection between a line along the direction in which the second display device 200 faces and the display screen of the first display device 10, and detects the area including the obtained intersection as the user's gaze area. In this case as well, it is desirable that the content processing unit 110 display an indicator for identifying the intersection position, such as a marker, at the position of the intersection so that the user can visually recognize the position of the intersection on the display screen of the first display device 10.

[0073] The image analysis unit 105 performs an analysis process on the captured image. For example, the image analysis unit 105 analyzes the captured image of the first display device 10 captured by the camera of the second display device 200. The camera is, for example, the imaging unit 162, and in the case of the see-through HMD 202 of FIG. 5, the camera is, for example, the RGB camera 246. If the second display device 200 is a terminal device such as a smartphone, a portable game device, or a tablet terminal, the camera that captures the captured image is, for example, a camera included in the terminal device. The image analysis unit 105 performs an analysis process on the display image of the first display device 10 that appears in the captured image of the camera. For example, the image analysis unit 105 analyzes the display content of the display image of the first display device 10 or the display mode of the first display device 10. Alternatively, the image analysis unit 105 may perform an image analysis process to detect the user's gaze area on the display screen of the first display device 10.

[0074] The positional relationship detection unit 106 detects the positional relationship between the first display device 10 and the second display device 200. For example, the positional relationship detection unit 106 detects the relative positional relationship of the second display device 200 with respect to the first display device 10. Here, the positional relationship may be the relationship between the positions of the first display device 10 and the second display device 200, or the relationship between the direction (facing direction) of the first display device 10 and the direction (facing direction) of the second display device 200. For example, the positional relationship may be the relative directional relationship of the second display device 200 with respect to the first display device 10. As an example, the positional relationship detection unit 106 performs a recognition process of the real space using a camera (such as an RGB camera, a depth camera, or an environment recognition camera) provided on the second display device 200, and detects the relative positional relationship of the first display device 10 with respect to the second display device 200 based on real space information obtained by this recognition process. Alternatively, markers may be installed on the first display device 10, and the positional relationship between the second display device 200 and the first display device 10 may be detected using these markers. For example, multiple markers (AR markers, etc.) are placed on the first display device 10, and the camera of the second display device 200 captures images of these multiple markers. The captured images are then analyzed to detect the relative positional relationship of the first display device 10 with respect to the second display device 200.

[0075] The content processing unit 110 performs various content processes. The content processes include, for example, various processes related to the content, such as displaying an image of the content and outputting sound of the content. The displaying process of the image of the content is not limited to generating and displaying an image of the content, but may be a process of instructing the display of the image of the content or instructing the content of the image of the content to be displayed. The content processing unit 110 includes a notification processing unit 112. The notification processing unit 112 performs a process of notifying the user of content information using the second display device 200. For example, the notification processing unit 112 performs a process of notifying the user of content information corresponding to the user's gaze area using the second display device 200.

[0076] The entertainment system of this embodiment, which uses a first display device 10 and a second display device 200 on which the display image of the first display device 10 is visible, includes a gaze area detection unit 104 and a content processing unit 110, as shown in FIG. 6.

[0077] The gaze area detection unit 104 detects the user's gaze area on the display screen of the first display device 10 based on information about the user's gaze direction. For example, the gaze area detection unit 104 detects the user's gaze area based on information about the user's gaze direction acquired by the information acquisition unit 102. For example, the gaze area is detected based on information about the user's gaze direction acquired by a tracking process of the second display device 200. Alternatively, the gaze area is detected from the direction in which the camera faces or the direction in which the second display device 200, which is a transmissive display device, faces.

[0078] The content processing unit 110 performs processing to allow the user to view a content image in which the display image of the second display device 200 is added to the display image of the first display device 10. For example, as shown in Fig. 2, the content processing unit 110 performs control processing on the second display device 200 etc. to allow the user to view a content image in which the display image IMH of the second display device 200 is added to the display image IMD of the first display device 10. For example, the content processing unit 110 controls the second display device 200 so that the display image IMD is displayed to the user as a transparent image, and the display image IMH is displayed to the user as a superimposed image of a virtual image or a real image.

[0079] The content processing unit 110 then performs a notification process of notifying the user of content information corresponding to the detected gaze area using the second display device 200. For example, when a gaze area is detected, the content processing unit 110 performs a process of notifying the user of the content information corresponding to the gaze area using, for example, hardware (such as a display unit, a sound output unit, a vibration device, or a light-emitting element) of the second display device 200. For example, the content processing unit 110 performs a process of notifying the user of information about displayed objects in the gaze area, their display content, display mode, status information, or event occurrence information. The content information corresponding to the gaze area is, for example, content information related to the gaze area. Specifically, the content information corresponding to the gaze area is information related to the displayed objects displayed in the gaze area, the displayed content in the gaze area, the display mode of the gaze area, or events occurring in the gaze area. The content information is information about entertainment content realized by the entertainment system. If the entertainment content is game content, the content information is game information. The game information is, for example, information about displayed objects appearing in the game, game progress information, game status information, or information about the user playing the game. The information about displayed objects is, for example, parameter information, display position information, or display mode information about the displayed objects. Alternatively, the content information may be various information about entertainment content such as streaming videos, movies, environmental videos, sightseeing videos, or simulation videos. For example, the content information may be information about the content of such content, status information about the content, progress information about the content, or explanatory information about the content.

[0080] The notification process of the content information is, for example, a process of displaying the content information using the second display device 200. For example, the notification process is a process of displaying an image of the content information on the display device 243 of the second display device 200. For example, the notification process is a process of displaying the image of the content information as an image superimposed on the display image of the first display device 10. However, the notification process is not limited to this, and may be a process of outputting sound, a vibration process, or a light-emitting element light-emitting process for the content information.

[0081] Furthermore, the content processing unit 110 changes the notification mode of the content information according to the information of the display object displayed on the first display device 10 or the display mode of the first display device 10. The information of the display object is parameter information, display position information, display mode information, etc. of the display object. The display mode of the first display device 10 is the brightness, color, contrast, display content, etc. of the display screen of the first display device 10. For example, the content processing unit 110 changes the notification content, notification object, notification position, notification event, notification timing, etc. of the content information according to the information of the display object or the display mode of the first display device 10.

[0082] For example, the content processing unit 110 controls the display of an object displayed on the second display device 200 in accordance with the information about the object displayed on the first display device 10 or the display mode of the first display device 10. For example, the content processing unit 110 controls the display of an object displayed on the second display device 200 as a process for changing the notification mode of content information in accordance with the information about the object displayed on the first display device 10 or the display mode of the object. For example, the content processing unit 110 determines whether to display or hide an object or changes the display mode of the object in accordance with the information about the object displayed on the first display device 10 or the display mode of the object. For example, the content processing unit 110 sets the display mode, such as the color, brightness, size, shape, transparency, or outline of the object, in accordance with the information about the object displayed on the first display device 10 or the display mode of the object.

[0083] For example, the information on the display object is information on the gaze object gazed upon by the user on the display screen of the first display device 10. The content processing unit 110 then sets the display position of the object to be displayed on the second display device 200 according to the information on the gaze object on the first display device 10 or the display mode of the first display device 10. That is, as display control of the object according to the information on the gaze object or the display mode of the first display device 10, the content processing unit 110 sets (determines) the display position of the object to be displayed on the second display device 200. For example, when the content processing unit 110 determines the user's gaze object, it controls the object to be displayed on the second display device 200 at a position corresponding to the position of the gaze object. Alternatively, it controls the object to be displayed on the second display device 200 at a position according to the size and shape of the gaze object. Alternatively, it controls the object to be displayed on the second display device 200 at an appropriate position according to the display mode and display content of the display screen of the first display device 10. In this embodiment, for the sake of clarity, the display objects (OB1 to OB5 in FIG. 2, etc.) displayed on the display device 243 of the second display device 200 will be referred to as "objects" to distinguish them from the "display objects" of the first display device 10. These objects are displayed by the second display device 200 as superimposed images on the display image of the first display device 10.

[0084] Furthermore, the content processing unit 110 sets the display mode of the object to be displayed on the second display device 200 according to the information of the gaze object or the display mode of the first display device 10. That is, the content processing unit 110 sets (determines) the display mode of the object to be displayed on the second display device 200 as display control of the object according to the information or display mode of the gaze object of the first display device 10. For example, when the content processing unit 110 determines the user's gaze object, it controls to display the object on the second display device 200 in a display mode corresponding to the gaze object. Alternatively, it controls to display the object on the second display device 200 in a display mode according to the brightness, color, shape, or size of the gaze object. Alternatively, it controls to display the object on the second display device 200 in an appropriate display mode according to the display mode or display content of the display screen of the first display device 10.

[0085] The entertainment system of this embodiment also includes an information acquisition unit 102 and a content processing unit 110. The information acquisition unit 102 acquires information about an object to be displayed on the first display device 10 or information about the display mode of the first display device 10. The content processing unit 110 performs processing to allow the user to view a content image in which the display image of the second display device 200 is added to the display image of the first display device 10. The content processing unit 110 performs notification processing to notify the user of content information using the second display device 200, and changes the notification mode of the content information depending on the information or display mode of the object of the first display device 10. For example, the content processing unit 110 changes the notification content, notification object, notification position, notification event, notification timing, etc. of the content information depending on the information or display mode of the object of the first display device 10.

[0086] In this case, the content processing unit 110 performs display control of the object displayed on the second display device 200 in accordance with the information or display mode of the display object of the first display device 10. For example, the content processing unit 110 performs display control of the object displayed on the second display device 200 as a process for changing the notification mode of content information in accordance with the information or display mode of the display object of the first display device 10. For example, the content processing unit 110 determines whether to display or hide the object, or changes the display mode of the object, in accordance with the information or display mode of the display object of the first display device 10.

[0087] Specifically, the content processing unit 110 sets the display position of an object displayed on the second display device 200 in accordance with the information or display mode of the object on the first display device 10. That is, as display control of the object in accordance with the information of the object or the display mode of the first display device 10, the content processing unit 110 sets (determines) the display position of the object displayed on the second display device 200. For example, the content processing unit 110 controls to display the object on the second display device 200 at a position corresponding to the position of the object on the first display device 10. Alternatively, the content processing unit 110 controls to display the object on the second display device 200 at a position in accordance with the size and shape of the object. Alternatively, the content processing unit 110 controls to display the object on the second display device 200 at an appropriate position in accordance with the display mode and display content of the display screen of the first display device 10.

[0088] Furthermore, the content processing unit 110 sets the display mode of the object to be displayed on the second display device 200 in accordance with the information or display mode of the display object of the first display device 10. That is, the content processing unit 110 sets (determines) the display mode of the object to be displayed on the second display device 200 as display control of the object in accordance with the information or display mode of the display object of the first display device 10. For example, the content processing unit 110 controls the display of the object on the second display device 200 in a display mode corresponding to the display object of the first display device 10. Alternatively, the content processing unit 110 controls the display of the object on the second display device 200 in a display mode corresponding to the brightness, color, shape, or size of the display object. Alternatively, the content processing unit 110 controls the display of the object on the second display device 200 in an appropriate display mode corresponding to the display mode or display content of the display screen of the first display device 10.

[0089] Furthermore, the objects displayed on the second display device 200 are objects related to the objects displayed on the first display device 10. For example, the objects on the second display device 200 are objects that are associated in advance with the objects on the first display device 10. For example, the objects on the second display device 200 are stored in the storage unit 170 in association with the objects on the first display device 10. For example, a first object (first object group) is associated with the first object, and a second object (second object group) is associated with the second object. Then, for example, when an object is displayed on the first display device 10, the objects associated with this object are displayed on the second display device 200.

[0090] The information about the object of the first display device 10 is, for example, parameter information about the object, display position information about the object, or display mode information about the object. The parameter information about the object is information about parameters that represent the state or situation of the object, such as information about game parameters of the object that appears in a game. For example, if the object is a game character, the parameter information is game parameter information about the character that is stored in the storage unit 170 in association with the character. For example, it is information about parameters that represent the character's status, ability, level, attributes, etc. The display position information about the object is, for example, information about position coordinates, display area, display location, etc. on the display screen of the first display device 10. The display position information about the object may be information about the object's position in a virtual three-dimensional space. The display mode information about the object is, for example, information about the color, brightness, size, shape, transparency, texture, etc. of the object.

[0091] The entertainment system of this embodiment also includes an image analysis unit 105 and a content processing unit 110. The image analysis unit 105 performs analysis processing of a captured image of the first display device 10 taken by the camera of the second display device 200. For example, the information acquisition unit 102 acquires information about the captured image of the first display device 10 taken by the camera of the second display device 200, and the image analysis unit 105 performs analysis processing of the acquired captured image. For example, the image analysis unit 105 analyzes a display image of the first display device 10 or a display mode of the first display device 10. Alternatively, the image analysis unit 105 performs image analysis processing to detect a user's gaze area on the display screen of the first display device 10.

[0092] The content processing unit 110 then performs a notification process to notify the user of content information based on the results of the analysis process of the captured image using the second display device 200. For example, based on the results of the analysis process of the image captured by the camera that captured the first display device 10, information about the objects to be displayed on the first display device 10 and the display mode of the first display device 10 are identified. The content processing unit 110 then performs a process to change the notification mode of the content information in accordance with the identified information about the objects on the first display device 10 or the display mode of the first display device 10. That is, the content processing unit 110 performs a process to change the notification mode of the content information in accordance with the information about the objects or the display mode of the first display device 10 identified based on the results of the analysis process of the captured image. Specifically, the content processing unit 110 performs display control of objects displayed on the second display device 200 in accordance with the information about the objects or the display mode of the first display device 10. Alternatively, based on the results of the analysis process of the image captured by the camera that captured the first display device 10, a user's gaze area on the display screen of the first display device 10 is detected. The content processing unit 110 then performs a notification process to notify the user of content information corresponding to the gaze area detected based on the results of the analysis process of the image captured by the camera using the second display device 200. For example, suppose that a gaze object gazed upon by a user is identified based on the results of an analysis process of an image captured by a camera. In this case, the content processing unit 110 performs a process of setting the display position of the object displayed on the second display device 200 or a process of setting the display mode of the object displayed on the second display device 200, depending on the information of the gaze object or the display mode of the first display device 10.

[0093] Furthermore, the content processing unit 110 changes the display content of the first display device 10 or the second display device 200, or the notification mode of the content information, depending on the positional relationship between the first display device 10 and the second display device 200. For example, when the positional relationship detection unit 106 detects the positional relationship between the first display device 10 and the second display device 200, the content processing unit 110 changes the display content of the first display device 10 or the second display device 200, or changes the notification mode of the content information, depending on the detected positional relationship. For example, the content processing unit 110 changes the type or display mode of an object displayed on the first display device 10, or changes the type or display mode of an object displayed on the second display device 200, depending on the positional relationship between the first display device 10 and the second display device 200. Alternatively, the content processing unit 110 changes the notification content, notification object, notification position, notification event, notification timing, etc. of the content information, depending on the positional relationship between the first display device 10 and the second display device 200.

[0094] Furthermore, the content processing unit 110 changes the display content of the first display device 10 or the second display device 200, or the notification mode of the content information, depending on the positional relationship between the first display device 10 and the second display device 200 and the information on the display object displayed on the first display device 10. For example, the content processing unit 110 changes the type or display mode of the display object displayed on the first display device 10, changes the type or display mode of the object displayed on the second display device 200, or changes the notification content, notification object, notification position, notification event, notification timing, etc. of the content information, depending on the positional relationship between the first display device 10 and the second display device 200 and the information on the display object on the first display device 10.

[0095] 7A and 7B show examples of hardware configurations of the entertainment system of this embodiment. In FIG. 7A, an information processing device 40 is communicatively connected to the first display device 10 and the second display device 200 via wired or wireless communication. The information processing device 40 controls the first display device 10 and the second display device 200. The information processing device 40 is, for example, a personal computer (PC), a game device, or a communication terminal device. The personal computer is a desktop PC, a notebook PC, a tablet PC, or the like. The game device may be a home game device or an arcade game device installed in an amusement facility. The communication terminal device is, for example, a portable communication terminal device such as a smartphone. For example, in FIG. 7A, the information processing device 40 performs display processing of an image displayed on the first display device 10 and also performs display processing of an image displayed on the second display device 200. The display processing on the first display device 10 can be realized by distributed processing between the information processing device 40 and the first display device 10, and the display processing on the second display device 200 can be realized by distributed processing between the information processing device 40 and the second display device 200. The information processing device 40 then performs processing to allow the user to view a content image in which the display image of the second display device 200 is added to the display image of the first display device 10. For example, the information processing device 40 transmits information about the display image of the first display device 10 to the first display device 10, and transmits information about the display image of the second display device 200 to the second display device 200. Then, the second display device 200 performs processing to superimpose the display image of the second display device 200 on the transparent image of the display image of the first display device 10.

[0096] In FIG. 7B, the second display device 200, the server 50, and the information processing device 40 are communicatively connected via a network 60. The network 60 (distribution network, communication line) is a communication path using, for example, the Internet or a wireless LAN, and can include a dedicated line (dedicated cable) for direct connection, a LAN using Ethernet (registered trademark), a telephone communication network, a cable network, a wireless LAN, or other communication networks. The communication method may be wired or wireless. The server 50 is operated by an entertainment system operator and performs, for example, various control processes and management processes for the entertainment system. In FIG. 7B, each process in the entertainment system of this embodiment can be realized by, for example, distributed processing of the information processing device 40, the server 50, and the second display device 200. In FIG. 7B, for example, the information processing device 40 performs display processing of an image displayed on the first display device 10, and the second display device 200 performs display processing of an image displayed on the second display device 200. The server 50 performs various control processes and management processes for the information processing device 40 and the second display device 200. For example, the server 50 can perform control processing and management processing to allow the user to view a content image in which the display image of the second display device 200 is added to the display image of the first display device 10.

[0097] 2. Method of this embodiment Next, the method of this embodiment will be described in detail.

[0098] 2.1 Notification processing using the second display device As explained in Fig. 1, the entertainment system of this embodiment is an entertainment system that uses a first display device 10 and a second display device 200 on which the display image of the first display device 10 is visible. In this embodiment, various notification processes are realized using the second display device. Fig. 8 is a flowchart illustrating an example of the process of this embodiment.

[0099] First, the entertainment system of this embodiment acquires information on the gaze direction LS of the user US (step S1), as described in Fig. 3. For example, the gaze direction LS of the user US is detected by performing a tracking process on the direction of the second display device 200 worn by the user US. Alternatively, the gaze direction LS of the user US may be detected by performing gaze tracking (eye tracking) of the user US.

[0100] Next, as shown in Fig. 3, the entertainment system detects a gaze area ARG of the user US on the display screen of the first display device 10 based on information about the line of sight LS of the user US (step S2). For example, in Fig. 3, the user US is gazing at a character CH1, which is the character that the user US operates, and an area corresponding to the position of this character CH1 is detected as the gaze area ARG. As an example, an area including the intersection of a line along the line of sight LS of the user US and the display screen of the first display device 10 is detected as the gaze area ARG. Note that, as described above, the gaze area ARG may also be detected based on the facing direction of the camera of the second display device 200 or the facing direction of the transmissive display device that is the second display device 200.

[0101] When the entertainment system performs a process of making the user US view a content image in which the display image of the second display device 200 is added to the display image of the first display device 10, the entertainment system performs a notification process of notifying the user of content information corresponding to the gaze area ARG using the second display device 200 (step S3). For example, in FIG. 3, a character CH1 is displayed in the gaze area ARG where the user US is gazing on the display screen of the first display device 10. Then, information about the character CH1, which is a display object of the first display device 10, is notified as content information corresponding to the gaze area ARG. Specifically, as shown in FIG. 2, status information of the character CH1 located in the gaze area ARG is displayed to the user US as a display image IMH of the second display device 200 superimposed on the display image IMD of the first display device 10. In this way, the user US can use the second display device 200 to know content information corresponding to the gaze area ARG where the user US is gazing on the display screen of the first display device 10. That is, the user US can obtain content information corresponding to the gaze area ARG simply by gazing at a desired area on the display screen of the first display device 10. Therefore, it is possible to provide an entertainment system or the like that can notify the user of content information related to the first display device 10 in a system using the first display device 10 and the second display device 200. For example, it is possible to realize highly entertaining performance control in an entertainment system using the first display device 10 and the second display device 200.

[0102] In this embodiment, the notification process in step S3 of FIG. 8 involves displaying content information using the second display device 200. That is, as shown in FIGS. 2 and 3 , the second display device 200 displays a display image IMH corresponding to the content information superimposed on the display image IMD of the first display device 10, thereby notifying the user of the content information corresponding to the gaze area ARG. In this manner, the user US can visually recognize the content information corresponding to the gaze area ARG that the user gazes at on the display screen of the first display device 10 from the display image IMH of the second display device 200. That is, the user US can visually recognize the content information corresponding to the gaze area ARG by simply gazing at a desired area on the display screen of the first display device 10 from the display image IMH of the second display device 200, thereby realizing a suitable user interface in an entertainment system using the first display device 10 and the second display device 200. The notification process for notifying the user US of content information is not limited to the image display process shown in FIGS. 2 and 3 , but may also be notification using vibration, sound, light emission, or the like, as described with reference to FIG. 4 .

[0103] In this embodiment, the notification mode of the content information is changed according to the information or display mode of the display object of the first display device 10. Fig. 9 is a flowchart illustrating an example of processing in this embodiment.

[0104] First, the entertainment system acquires information about the display objects displayed on the first display device 10 or information about the display mode of the first display device 10 (step S11). Taking FIG. 1 as an example, the entertainment system acquires information about the display objects, such as the characters CH1-CH3, CE1-CE3, and background display objects displayed on the first display device 10, and information about the display mode, such as the brightness, color, contrast, and display content of the display screen of the first display device 10. This information about the display objects and the display mode may be acquired based on the results of an analysis process of an image captured by a camera that captured the first display device 10. Then, the entertainment system changes the notification mode of the content information according to the information about the display objects or the display mode of the first display device 10 (step S12). For example, the notification content, notification object, notification position, notification event, notification timing, etc. of the content information are changed according to the information about the display objects or the display mode of the first display device 10. For example, if the display object on the first display device 10 is a first display object, or if the display mode of the first display device 10 is a first display mode, the content information is notified using the first notification content, first notification object, first notification position, first notification event, or first notification timing. When the display object of the first display device 10 is the second display object, or when the display mode of the first display device 10 is the second display mode, content information is notified using the second notification content, the second notification display object, the second notification position, the second notification event, or the second notification timing.

[0105] Specifically, the entertainment system notifies the user of notification content or content information of the notification object corresponding to the display object of the first display device 10. For example, if the display object of the first display device 10 is a character, notification processing is performed to notify the user of content information of notification content corresponding to the character or to display a notification object corresponding to the character. For example, content information of notification content that provides information about the character is notified or a notification object related to the character is displayed. Furthermore, content information of a notification event corresponding to the display object of the first display device 10 is notified or the content information is notified at a notification timing corresponding to the display object of the first display device 10. For example, if the display object of the first display device 10 is a character, content information that a notification event related to the character occurs is notified or the content information is notified at a notification timing corresponding to the timing when the character appears, etc. Alternatively, the notification content of the content information, notification object, notification position, notification event, notification timing, etc. are changed depending on the display mode, such as brightness, color, contrast, or display content, of the display screen of the first display device 10. For example, the entertainment system notifies the user of content information of notification content or content information of the notification object suitable for the display mode of the first display device 10. Alternatively, the content information may be notified at a notification position suitable for the display mode of the first display device 10, or content information that will cause a notification event suitable for the display mode of the first display device 10 to occur, or content information may be notified at a notification timing suitable for the display mode of the first display device 10. As described above, according to the present embodiment, the notification mode of the content information is changed depending on the information or display mode of the display object of the first display device 10, so that the content information can be notified to the user in a notification mode suitable for the display object or display mode of the first display device 10. Therefore, in an entertainment system that allows the user to view a content image in which the display image of the second display device 200 is added to the display image of the first display device 10, it is possible to notify the user of appropriate content information suitable for the display object or display mode of the first display device 10.

[0106] Furthermore, in this embodiment, display control of an object displayed on the second display device 200 is performed according to the information or display mode of the display object of the first display device 10. In this manner, an object whose display is controlled according to the display object or display mode of the first display device 10 is displayed on the second display device 200, and it becomes possible for the user to visually recognize a content image in which an image of the object is added to the display image of the first display device 10. Therefore, display control of an object that reflects the display object or display mode of the first display device 10 can be performed, thereby realizing notification processing of content information using the object. FIG. 10 is a flowchart illustrating an example of processing in this embodiment.

[0107] First, the entertainment system acquires information about the object (the gazed-at object) displayed on the first display device 10 or information about the display mode of the first display device 10 (step S21). For example, information about the object, such as parameter information, display position information, or display mode information about the object, or information about the display mode, such as the brightness, color, contrast, or display content of the display screen of the first display device 10, is acquired. This information about the object or the display mode may be acquired based on the results of an analysis process of an image captured by a camera that captured the first display device 10. Then, the entertainment system sets the display position or display mode of the object displayed on the second display device 200 according to the information or display mode of the object (the gazed-at object) on the first display device 10 (step S22). That is, the entertainment system performs control to set the display position or display mode of the object displayed on the second display device 200 as display control of the object on the second display device 200 according to the information or display mode of the object on the first display device 10.

[0108] As described above, in this embodiment, the display position of an object displayed on the second display device 200 is set according to the information or display mode of the display object of the first display device 10. In this way, the object is displayed at a display position according to the information or display mode of the display object of the first display device 10, and it becomes possible for the user to visually recognize a content image in which the image of the object is added to the display image of the first display device 10. Therefore, it becomes possible to realize a notification process of content information using the object by displaying the object at a display position according to the information or display mode of the display object of the first display device 10.

[0109] For example, in FIG. 11(A), the character CH1, which is a display object of the first display device 10, is displayed at the left position on the display screen of the first display device 10. In this case, the display positions of the objects OB1 and OB2 displayed as the display image IMH on the second display device 200 are also set to the left position. Also, in FIG. 11(B), the character CH1, which is a display object of the first display device 10, is displayed at the right position on the display screen of the first display device 10. In this case, the display positions of the objects OB1 and OB2 displayed on the second display device 200 are also set to the right position. That is, the display positions of the objects displayed on the second display device 200 are set according to the position information of the objects displayed on the first display device 10. Alternatively, the display positions of the objects displayed on the second display device 200 may be set according to the shape or size information of the objects. For example, information about the objects displayed on the first display device 10, such as their display position, shape, or size, is determined, and the objects on the second display device 200 are displayed at display positions that do not obstruct the visibility of the objects. In this way, it is possible to display an object on the second display device 200 at an appropriate display position according to information about the object, such as the display position, shape, or size of the object on the first display device 10. Alternatively, in this embodiment, the display position of the object displayed on the second display device 200 may be set according to the display mode of the first display device 10. For example, the display mode of the first display device 10, such as brightness, color, contrast, or display content, is determined, and the object on the second display device 200 is displayed at a display position suitable for notifying content information using the object. Alternatively, the display mode of the first display device 10 is determined, and the object on the second display device 200 is displayed at a display position that does not interfere with the display on the first display device 10. In this way, it is possible to display an object on the second display device 200 at an appropriate display position according to the display mode of the first display device 10.

[0110] Furthermore, in this embodiment, the display mode of an object displayed on the second display device 200 is set according to the information or display mode of the display object on the first display device 10. For example, the display mode, such as the color, brightness, size, shape, transparency, or outline of the object on the second display device 200, is set. In this way, the object is displayed in a display mode according to the information or display mode of the display object on the first display device 10, and it becomes possible for the user to visually recognize a content image in which the image of the object is added to the display image on the first display device 10. Therefore, it becomes possible to realize a notification process of content information using the object by displaying the object in a display mode according to the information or display mode of the display object on the first display device 10.

[0111] For example, in FIG. 12, the display aspects, such as the color, brightness, size, shape, transparency, or contour, of objects OB1 and OB2 in the display image IMH of the second display device 200 are set according to information about the character CH1, which is a display object of the first display device 10. For example, the color and brightness of the objects OB1 and OB2 on the second display device 200 are set to a color and brightness corresponding to the character CH1, which is a display object. For example, the color and brightness of the objects OB1 and OB2 are set to the same color and brightness as the character CH1. Alternatively, the size and shape of the objects OB1 and OB2 on the second display device 200 are set to a size and shape corresponding to the character CH1, which is a display object. For example, if the character CH1 is large, the objects OB1 and OB2 are also made large, or the objects OB1 and OB2 are displayed in a shape corresponding to the shape of the character CH1. Note that the type of display object displayed on the first display device 10 can be identified based on, for example, the results of an analysis of an image captured by a camera that captured the first display device 10.

[0112] 12, the display modes of the objects OB1 and OB2 in the display image IMH of the second display device 200, such as the hue, brightness, size, shape, transparency, and contour, are set according to the display modes of the display screen (IMD) of the first display device 10, such as the brightness, hue, contrast, and display content. For example, when the display screen (IMD) of the first display device 10 is bright, the display modes of the objects OB1 and OB2 are set so that the visibility of the objects OB1 and OB2 displayed by the second display device 200 is increased even on the bright display screen. For example, when the display screen (IMD) of the first display device 10 is bright, the display modes of the objects OB1 and OB2 on the second display device 200 are set so that the hue of the objects OB1 and OB2 on the second display device 200 is darkened, the transparency is reduced (opacity is increased), or the contours of the objects OB1 and OB2 are made more noticeable. Furthermore, the hue of the objects OB1 and OB2 on the second display device 200 is set to a hue corresponding to the hue of the display screen of the first display device 10. For example, the color of the objects OB1 and OB2 on the second display device 200 may be set to the same color as that of the display screen of the first display device 10, or to a color that enhances the visibility of the objects OB1 and OB2 on the display screen. Alternatively, the size, shape, transparency, etc. of the objects OB1 and OB2 on the second display device 200 may be set to a size, shape, transparency, etc. that corresponds to the display content of the first display device 10. For example, the size, shape, transparency, etc. of the objects OB1 and OB2 on the second display device 200 may be set to a size, shape, or transparency that does not interfere with the visibility of the display content of the first display device 10. Note that an ambient light sensor that detects the brightness of ambient light may be provided, and the display of the objects OB1 and OB2 displayed by the second display device 200 may be controlled based on the detection result of the ambient light sensor. Furthermore, the display mode, such as the brightness, color, contrast, or display content of the display screen of the first display device 10, may be determined based on the results of analysis of an image captured by a camera that captured the first display device 10, for example.

[0113] Furthermore, when detecting a gaze object gazed upon by the user on the display screen of the first display device 10, the display position of the object displayed on the second display device 200 is set according to the information of the gaze object. For example, in FIG. 3, the user's gaze area ARG is detected based on information about the user's line of sight LS, and thereby the character CH1, which is the user's gaze object, is detected. In this case, the display positions of the objects (OB1, OB2) displayed on the second display device 200 are set according to the information of the detected gaze object (character CH1). For example, the display positions of the objects displayed on the second display device 200 are set according to the parameter information, display position, display mode, etc. of the gaze object. For example, the object on the second display device 200 is displayed near the display position of the gaze object, or the object is displayed at an appropriate display position according to the size or shape of the gaze object. In this way, the object is displayed at a display position according to the information of the gaze object on the first display device 10, and it becomes possible for the user to view a content image in which the image of the object is added to the display image on the first display device 10. Therefore, it becomes possible to realize a content information notification process using an object by displaying the object at a display position corresponding to the gaze object of the first display device 10. Note that the gaze object gazed upon by the user on the display screen of the first display device 10 can be identified based on the results of an analysis process of an image captured by a camera that captured the first display device 10, for example.

[0114] Furthermore, when a gaze object gazed upon by the user is detected on the display screen of the first display device 10, the display mode of the object displayed on the second display device 200 is set according to the information of the gaze object. For example, as shown in FIG. 3, when a character CH1 that is the user's gaze object is detected, the display mode of the objects (OB1, OB2) displayed on the second display device 200 is set according to the information of the detected gaze object (character CH1). For example, the display mode of the object displayed on the second display device 200 is set according to the parameter information, display position, display mode, etc. of the gaze object. For example, the display mode of the object displayed on the second display device 200, such as the color, brightness, size, shape, or transparency of the object, is changed according to the parameter information of the gaze object. Alternatively, the second display device 200 displays an object with a color, brightness, size, or shape corresponding to the color, brightness, size, or shape of the gaze object. In this way, the object is displayed in a display mode according to the information of the gaze object of the first display device 10, and the user can view a content image in which the image of the object is added to the display image of the first display device 10. Therefore, it is possible to display an object in a display mode corresponding to the gaze display object of the first display device 10, and realize a notification process of content information using the object.

[0115] Furthermore, the objects displayed on the second display device 200 are objects related to the objects displayed on the first display device 10. For example, in FIG. 2, the objects OB1 to OB5 displayed on the second display device 200 are objects that notify the user of status information, etc., of the character CH1, which is the object of the first display device 10, and are objects related to the character CH1, which is the object. In this way, the objects related to the objects of the first display device 10 can be displayed by the second display device 200 as objects superimposed on the display image of the first display device 10. As a result, the second display device 200 can superimpose objects related to the objects in the display image of the first display device 10 on the display image of the first display device 10, and can notify the user of content information using the objects.

[0116] 13, the information about the displayed object of the first display device 10 is parameter information, display position information, or display mode information about the displayed object. The parameter information about the displayed object is information about parameters that represent the status, ability, level, or attributes of a displayed object such as a character. The display position information about the displayed object is, for example, the position coordinates, display area, or display location of the displayed object. The display mode of the displayed object is, for example, the color, brightness, size, shape, or transparency of the displayed object. In this way, it becomes possible to change the notification mode of content information notified using the second display device 200 according to the parameter information, display position information, or display mode information about the displayed object.

[0117] In this embodiment, for example, an analysis process is performed on a captured image of the first display device 10 captured by the camera of the second display device 200. Then, a notification process is performed to notify the user of content information based on the results of the analysis process on the captured image using the second display device 200. For example, a notification process is performed to notify the user of content information corresponding to information or a display mode of an object identified based on the results of the analysis process on the image captured by the camera using the second display device 200. For example, a process is performed to change the notification mode of the content information depending on information or a display mode of an object identified based on the results of the analysis process. Alternatively, a notification process is performed to notify the user of content information corresponding to a gaze area detected based on the results of the analysis process on the image captured by the camera using the second display device 200.

[0118] 2.2 Processing according to positional relationships In this embodiment, as shown in Fig. 14, the positional relationship between the first display device 10 and the second display device 200 is detected. This positional relationship may be the relationship between the position of the first display device 10 and the position of the second display device 200, or the relationship between the distance between the first display device 10 and the second display device 200, or the relationship between the direction of the first display device 10 and the direction of the second display device 200. Furthermore, Fig. 14 shows an example in which the second display device 200 is a see-through HMD, but the second display device 200, whose positional relationship with the first display device 10 is detected, may be a see-through display device other than a see-through HMD, a terminal device having a camera, or the like.

[0119] In this embodiment, the display content of the first display device 10 or the second display device 200, or the notification mode of the content information notified using the second display device 200, is changed depending on the positional relationship between the first display device 10 and the second display device 200. For example, when the first display device 10 and the second display device 200 are in a first positional relationship, the display content of the first display device 10 or the second display device 200 is set to first display content, or the content information is notified using the second display device 200 in a first notification mode. When the first display device 10 and the second display device 200 are in a second positional relationship, the display content of the first display device 10 or the second display device 200 is set to second display content, or the content information is notified using the second display device 200 in a second notification mode. In this way, an image can be displayed on the first display device 10 or the second display device 200 with appropriate display content according to the relative positions of the first display device 10 and the second display device 200, or content information can be notified to the user by the second display device 200 in an appropriate notification manner according to the relative positions of the first display device 10 and the second display device 200.

[0120] For example, when the first display device 10 and the second display device 200 are positioned far apart, the display content of the first display device 10 or the second display device 200 is set to display content that is visible to the user even at a long distance. On the other hand, when the first display device 10 and the second display device 200 are positioned close together, the display content of the first display device 10 or the second display device 200 is set to display content with a higher level of detail than when the distance is long. The display content of the first display device 10 or the second display device 200 may also be changed depending on the orientation of the first display device 10 and the second display device 200. For example, when the second display device 200 is facing directly toward the display screen of the first display device 10, the display content of the first display device 10 or the second display device 200 is set to the first display content. For example, the display content with a higher level of detail is set. On the other hand, when the second display device 200 is facing in an oblique direction relative to the first display device 10, the display content of the first display device 10 or the second display device 200 is set to the second display content. For example, the display content may be low in detail, conveying simplified information, or may be hidden.

[0121] Furthermore, the notification mode, such as the notification content, notification display, or notification position of the content information notified using the second display device 200, is changed depending on the relative positions of the first display device 10 and the second display device 200. For example, when the first display device 10 and the second display device 200 are far apart, the notification content or notification display of the content information is changed to notification content or notification display for a long distance. For example, the content information is notified using notification content with less detail than when the distance is short, or a simplified notification display. On the other hand, when the first display device 10 and the second display device 200 are close together, the notification content or notification display of the content information is changed to notification content or notification display for a short distance. For example, the content information is notified using notification content with more detail than when the distance is long, or a notification display with a more detailed shape. Alternatively, the content information is displayed at an appropriate notification position depending on the relative positions of the first display device 10 and the second display device 200. For example, on the display screen of the first display device 10, the content information is displayed in an area that is easily visible from the position of the second display device 200.

[0122] Furthermore, when content information is notified by sound, vibration, or light emission as shown in FIG. 4 , the notification mode of the content information by sound, vibration, or light emission may be changed depending on the positional relationship between the first display device 10 and the second display device 200. For example, when the first display device 10 and the second display device 200 are far from each other, the sound or vibration may be reduced or the light emission may be weakened. When the first display device 10 and the second display device 200 are close to each other, the sound or vibration may be increased or the light emission may be strengthened. Furthermore, the notification of the content information may be controlled to be turned on or off depending on the positional relationship between the first display device 10 and the second display device 200. For example, when the first display device 10 and the second display device 200 are close to each other, the notification of the content information may be turned on, and when the second display device 200 is far from each other, the notification of the content information may be turned off. Alternatively, when the orientation of the first display device 10 and the orientation of the second display device 200 are such that the display screen of the first display device 10 is easily visible from the second display device 200, the notification of the content information may be turned on, and when the orientation of the second display device 200 is such that the display screen of the first display device 10 is easily visible from the second display device 200, the notification of the content information may be turned off.

[0123] Furthermore, in this embodiment, the display content of the first display device 10 or the second display device 200 or the notification mode of the content information may be changed depending on the positional relationship between the first display device 10 and the second display device 200 and the information of the display object displayed on the first display device 10. That is, the display content of the first display device 10 or the second display device 200 or the notification mode of the content information is changed taking into consideration not only the positional relationship between the first display device 10 and the second display device 200 but also the information of the display object displayed on the first display device 10. In this way, it becomes possible to display an image on the first display device 10 or the second display device 200 with appropriate display content depending on the positional relationship between the first display device 10 and the second display device 200 and the information of the display object on the first display device 10, or to notify the user of content information via the second display device 200 in an appropriate notification mode depending on the positional relationship between the first display device 10 and the second display device 200 and the information of the display object.

[0124] For example, on the condition that the first display device 10 and the second display device 200 are in a predetermined positional relationship, such as being close to each other or being easily visible to the second display device 200, the display content of the first display device 10 or the second display device 200 or the notification mode of the content information is changed according to the information of the display object on the first display device 10. For example, on the condition that the first display device 10 and the second display device 200 are in a predetermined positional relationship (close to each other or being easily visible to each other), the display content of the first display device 10 or the second display device 200 is set to display content according to the information of the display object on the first display device 10. Alternatively, on the condition that the first display device 10 and the second display device 200 are in a predetermined positional relationship (close to each other or being easily visible to each other), the notification mode of the content information is changed according to the information of the display object on the first display device 10, as described above with reference to FIG. For example, on the condition that the first display device 10 and the second display device 200 are in a predetermined positional relationship, the notification mode, such as the notification content, notification display object, or notification position, of the content information notified using the second display device 200 is changed according to the information of the display object of the first display device 10. Alternatively, when the first display device 10 and the second display device 200 are in a predetermined positional relationship (such as a close relationship or an easily visible relationship), a change process is performed to change the display content of the first display device 10 or the second display device 200 or the notification mode of the content information according to the information of the display object of the first display device 10, while when the first display device 10 and the second display device 200 are not in the predetermined positional relationship, the change process is not performed.

[0125] 3, when a user's gaze object (CH1) is detected, the display content of the first display device 10 or the second display device 200 or the notification mode of the content information is changed according to the information of the gaze object, provided that the first display device 10 and the second display device 200 are in a predetermined positional relationship. For example, when the first display device 10 and the second display device 200 are close to each other and a user's gaze object is detected on the display screen of the first display device 10, the display content of the first display device 10 or the second display device 200 or the notification mode of the content information is changed according to the information of the gaze object. For example, the display content of the first display device 10 or the second display device 200 is set so that the content is suitable for displaying the gaze object. Alternatively, as described above, the notification mode of the content information notified using the second display device 200, such as the notification object or notification position, is changed according to the parameter information, position information, display mode, etc. of the gaze object. In this way, various notification processes of content information according to the information of the gaze display object of the first display device 10 can be realized while reflecting the positional relationship between the first display device 10 and the second display device 200.

[0126] Next, an example of a process for detecting the positional relationship between the first display device 10 and the second display device 200 will be described. The second display device 200 described in FIG. 5 acquires real-space information by performing a recognition process on the real space around the user, and can generate a virtual space corresponding to the real space based on the acquired real-space information. For example, in FIG. 15(A), objects such as a desk DK and a wall WL exist in the real space. The real-space information is acquired by performing a recognition process in which the real space is scanned by a camera provided on the second display device 200 of FIG. 5. Specifically, as shown in FIG. 15(B), mesh polygon data is acquired as the real-space information. This real-space recognition process is performed, for example, in real time, and the real space around the user is sequentially scanned to acquire the real-space information. For example, space data associated with IDs is sequentially acquired and accumulated and stored as real-space information in the storage unit 170 of FIG. 6. This real-space information also includes depth information (depth values), and using this depth information enables, for example, hidden surface removal between objects in the virtual space and objects in the real space. Furthermore, generating a virtual space based on real-space information enables interactions between virtual objects (characters, placed objects) and real-space objects (desks, walls). For example, it also enables hit processing (collision processing) between virtual objects and real-space objects, enabling unprecedented types of AR (Augmented Reality) and MR (Mixed Reality). Real-space information obtained through such real-space recognition processing can also be used to acquire user position and direction information. For example, the positional and distance relationships between the user and objects around the user can be determined based on real-space information. Therefore, by using the detection results of the inertial measurement unit and real-space information, it becomes possible to acquire more accurate user position and direction information.

[0127] As shown in FIG. 16 , real space information about the first display device 10 as seen from the second display device 200 is acquired through a real space recognition process using a camera or the like of the second display device 200. Specifically, mesh polygon data of the first display device 10 is acquired. Then, based on the mesh polygon data, which is the real space information of the first display device 10, a plurality of virtual markers MK1, MK2, and MK3 are set, for example, at corners of the first display device 10. In FIG. 16 , virtual markers MK1, MK2, and MK3 are set, for example, at least three corners of the first display device 10. These markers MK1 to MK3 are, for example, virtual markers in markerless AR. Then, the positional relationship between the first display device 10 and the second display device 200 is detected using these virtual markers MK1 to MK3. For example, the positional relationship between the first display device 10 and the second display device 200 can be detected by calculating the distance between the position of the second display device 200 and the positions of the markers MK1 to MK3. In this way, it is possible to detect the positional relationship between the first display device 10 and the second display device 200 by effectively utilizing the recognition processing of the real space using the camera or the like of the second display device 200. As described above, it is possible to realize the processing of changing the display content of the first display device 10 or the second display device 200 or the notification mode of the content information according to the detected positional relationship or the like.

[0128] In this embodiment, as shown in Fig. 17, an AR or MR image may be displayed in which an object OB corresponding to an object on the first display device 10 appears to pop out from the screen of the first display device 10. For example, as described in Fig. 3, an object that the user is gazing at on the display screen of the first display device 10 is detected, and a display process is performed using the second display device 200 to allow the user to view an image in which the object OB corresponding to the detected gaze object moves from the screen of the first display device 10 toward the second display device 200. For example, such a display process can be realized by controlling the parallax in the stereoscopic display. In this way, a notification process can be realized in which the second display device 200 is used to notify the user of the object OB, which is content information corresponding to the detected gaze area (gaze object).

[0129] Although the present embodiment has been described in detail above, those skilled in the art will readily understand that many modifications are possible without substantially departing from the novel features and advantages of the present disclosure. Therefore, all such modifications are intended to be within the scope of the present disclosure. For example, a term (such as a see-through HMD) described at least once in the specification or drawings together with a broader or synonymous term (such as a second display device) may be replaced with that term anywhere in the specification or drawings. Furthermore, information acquisition processing, gaze area detection processing, image analysis processing, content processing, notification processing, positional relationship detection processing, and the like are not limited to those described in the present embodiment; equivalent methods, processes, and configurations are also within the scope of the present disclosure. [Explanation of symbols]

[0130] 10...first display device, 30...display unit, 40...information processing device, 50...server, 60... network, 100... processing unit, 102... information acquisition unit, 104... gaze area detection unit, 105... image analysis unit, 106... positional relationship detection unit, 110... content processing unit, 112... notification processing unit, 160... operation unit, 162... imaging unit, 170... storage unit, 180...information storage medium, 192...sound output unit, 196...communication unit, 200...second display device, 202...Transmissive HMD, 240...Temple part, 242...Goggle part, 243...Display device, 244...Holographic optical element, 246...RGB camera, 247...Depth camera, 248, 249...Environmental recognition camera, ARG... gaze area, CH1~CH3, CE1~CE3... character, IMD, IMH...display image, LS...line of sight, MK1~MK3...marker, OB, OB1~OB5...Objects, RCH...Rare characters, US...Users

Claims

1. An entertainment system using a first display device and a second display device on which a display image of the first display device can be viewed, a gaze area detection unit that detects a gaze area of the user on a display screen of the first display device based on information about the user's gaze direction; a content processing unit that performs processing to cause the user to visually recognize a content image in which the display image of the second display device is added to the display image of the first display device; Including, The content processing unit An entertainment system characterized by performing a notification process to notify the user of content information corresponding to the detected gaze area using the second display device.

2. In claim 1, An entertainment system, wherein the notification process is a process of displaying the content information using the second display device.

3. In claim 1 or 2, The content processing unit An entertainment system characterized in that a notification mode of the content information is changed according to information on an object displayed on the first display device or a display mode of the first display device.

4. In claim 3, The content processing unit An entertainment system characterized in that display control of an object displayed on the second display device is performed in accordance with information about the displayed object on the first display device or the display mode of the first display device.

5. In claim 4, the information about the display object is information about a gaze display object gazed at by the user on the display screen of the first display device, The content processing unit An entertainment system, characterized in that a display position of the object displayed on the second display device is set in accordance with information on the gaze display object or a display mode of the first display device.

6. In claim 4, the information about the display object is information about a gaze display object gazed at by the user on the display screen of the first display device, The content processing unit An entertainment system, characterized in that a display mode of the object displayed on the second display device is set in accordance with information on the gaze display object or a display mode of the first display device.

7. An entertainment system using a first display device and a second display device on which a display image of the first display device can be viewed, an information acquisition unit that acquires information about an object to be displayed on the first display device or information about a display mode of the first display device; a content processing unit that performs processing to cause the user to visually recognize a content image in which the display image of the second display device is added to the display image of the first display device; The content processing unit An entertainment system characterized by performing a notification process to notify the user of content information using the second display device, and changing the notification mode of the content information depending on the information of the displayed object of the first display device or the display mode of the first display device.

8. In claim 7, The content processing unit An entertainment system characterized in that display control of an object displayed on the second display device is performed in accordance with information about the displayed object on the first display device or the display mode of the first display device.

9. In claim 8, The content processing unit An entertainment system characterized in that the display position of the object displayed on the second display device is set according to information about the displayed object on the first display device or the display mode of the first display device.

10. In claim 8, The content processing unit An entertainment system characterized in that the display mode of the object displayed on the second display device is set according to information about the displayed object on the first display device or the display mode of the first display device.

11. In claim 4, 5, 6, 8, 9 or 10, An entertainment system, wherein the object displayed on the second display device is an object related to the display object displayed on the first display device.

12. In any one of claims 3 to 11, An entertainment system, wherein the information about the displayed object of the first display device is parameter information about the displayed object, display position information about the displayed object, or display mode information about the displayed object.

13. An entertainment system using a first display device and a second display device on which a display image of the first display device can be viewed, an image analysis unit that performs analysis processing of a captured image of the first display device captured by a camera of the second display device; a content processing unit that performs processing to cause the user to visually recognize a content image in which the display image of the second display device is added to the display image of the first display device; Including, The content processing unit an entertainment system that performs a notification process to notify the user of content information based on the results of the analysis process of the captured image using the second display device;

14. In any one of claims 1 to 13, The content processing unit An entertainment system characterized by changing the display content of the first display device or the second display device, or the notification mode of the content information, depending on the positional relationship between the first display device and the second display device.

15. In any one of claims 1 to 13, The content processing unit An entertainment system characterized by changing the display content of the first display device or the second display device, or the notification mode of the content information, depending on the positional relationship between the first display device and the second display device and the information of the displayed object displayed on the first display device.

16. A program used in an entertainment system using a first display device and a second display device on which a display image of the first display device can be viewed, a gaze area detection unit that detects a gaze area of the user on a display screen of the first display device based on information about the user's gaze direction; a content processing unit that performs processing to cause the user to visually recognize a content image in which the display image of the second display device is added to the display image of the first display device, Make your computer work, The content processing unit A program that performs a notification process to notify the user of content information corresponding to the detected gaze area using the second display device.

17. A program used in an entertainment system using a first display device and a second display device on which a display image of the first display device can be viewed, an information acquisition unit that acquires information about an object to be displayed on the first display device or information about a display mode of the first display device; a content processing unit that performs processing to cause the user to visually recognize a content image in which the display image of the second display device is added to the display image of the first display device, Make your computer work, The content processing unit A program characterized by performing a notification process to notify the user of content information using the second display device, and changing the notification mode of the content information depending on the information of the display object of the first display device or the display mode of the first display device.

18. A program used in an entertainment system using a first display device and a second display device on which a display image of the first display device can be viewed, an image analysis unit that performs analysis processing of a captured image of the first display device taken by a camera included in the second display device; a content processing unit that performs processing to cause the user to visually recognize a content image in which the display image of the second display device is added to the display image of the first display device, Make your computer work, The content processing unit a program for notifying the user of content information based on the result of the analysis processing of the captured image using the second display device;

Images