Entertainment System and Program

Abstract

To provide an entertainment system or the like that can notify a user of content information related to a first display device in a system using the first display device and a second display device.SOLUTION: An entertainment system using a first display device and a second display device in which a display image of the first display device can be visually recognized includes: a gaze area detection unit that detects a user's gaze area on a display screen of the first display device based on information on a user's line-of-sight direction; and a content processing unit that performs processing to cause the user to visually recognize a content image to which a display image of the second display device is added to the display image of the first display device. The content processing unit performs notification processing for notifying the user of content information corresponding to the detected gaze area by using the second display device.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to an entertainment system, a program, and the like.

Background Art

[0002] Conventionally, a system for allowing a user to visually recognize a content image formed by combining two display images has been known. The system of Patent Document 1 includes a transmissive monitor that displays a transmissive image through which a specific color is transmitted, and a projector that projects a base screen that forms a game screen in combination with the transmissive image behind the transmissive monitor. Then, an information-providing image is formed by combining the transmissive image and the base screen.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, heretofore, no system has been proposed that performs interlocking control between a first display device and a second display device to notify a user of content information related to a display object or the like of the first display device.

[0005] According to the present disclosure, it is possible to provide an entertainment system, a program, and the like that can notify a user of content information related to a first display device in a system using a first display device and a second display device.

Means for Solving the Problems

[0006] One aspect of the present disclosure relates to an entertainment system using a first display device and a second display device where the display image of the first display device is visible, the entertainment system including a fixation area detection unit that detects a fixation area of a user on a display screen of the first display device based on information on the user's line-of-sight direction, and a content processing unit that performs a process of allowing the user to view a content image in which the display image of the second display device is added to the display image of the first display device. The content processing unit performs a notification process of notifying the user of content information corresponding to the detected fixation area using the second display device, and relates to the entertainment system. Another aspect of the present disclosure relates to a program that causes a computer to function as each of the above units, or a computer-readable information storage medium storing the program.

[0007] According to one aspect of the present disclosure, a fixation area of a user on a display screen of a first display device is detected. Also, a process of allowing the user to view a content image in which the display image of the second display device is added to the display image of the first display device is performed. At this time, a notification process of notifying the user of content information corresponding to the fixation area using the second display device is performed. In this way, the user can use the second display device to know the content information corresponding to the fixation area that the user has gazed at on the display screen of the first display device. Therefore, it becomes possible to provide an entertainment system or the like that can notify the user of content information related to the first display device in a system using the first display device and the second display device.

[0008] Also, in one aspect of the present disclosure, the notification process may be a process of displaying the content information using the second display device.

[0009] In this way, the user can visually recognize the content information corresponding to the fixation area that the user has gazed at on the display screen of the first display device by the display image of the second display device.

[0010] In another aspect of the present disclosure, the content processing unit may change the notification mode of the content information according to the information of the display content on the first display device or the display mode of the first display device.

[0011] In this way, in an entertainment system that allows a user to visually recognize a content image in which the display image of the second display device is added to the display image of the first display device, it becomes possible to notify the user of appropriate content information according to the display content or display mode of the first display device.

[0012] In another aspect of the present disclosure, the content processing unit may perform display control of the object displayed on the second display device according to the information of the display content on the first display device or the display mode of the first display device.

[0013] In this way, it becomes possible to display an object whose display is controlled according to the display content or display mode of the first display device on the second display device, and allow the user to visually recognize a content image in which the image of the object is added to the display image of the first display device.

[0014] In another aspect of the present disclosure, the information of the display content is the information of the fixation display content fixated by the user on the display screen of the first display device, and the content processing unit may set the display position of the object displayed on the second display device according to the information of the fixation display content or the display mode of the first display device.

[0015] In this way, it becomes possible to display an object at a display position according to the information of the fixation display content of the first display device or the display mode of the first display device, and allow the user to visually recognize a content image in which the image of the object is added to the display image of the first display device.

[0016] Also, in one aspect of the present disclosure, the information of the display object is the information of the fixation display object gazed at by the user on the display screen of the first display device, and the content processing unit may set the display mode of the object displayed on the second display device according to the information of the fixation display object or the display mode of the first display device.

[0017] In this way, it becomes possible to display an object in a display mode according to the information or display mode of the fixation display object of the first display device, and to let the user visually recognize a content image in which the image of the object is added to the display image of the first display device.

[0018] Also, one aspect of the present disclosure is an entertainment system using a first display device and a second display device capable of visually recognizing the display image of the first display device, including an information acquisition unit that acquires information of a display object displayed on the first display device or information of the display mode of the first display device, and a content processing unit that performs a process of letting the user visually recognize a content image in which the display image of the second display device is added to the display image of the first display device. The content processing unit performs a notification process of notifying content information to the user using the second display device, and relates to an entertainment system that changes the notification mode of the content information according to the information of the display object of the first display device or the display mode of the first display device. Also, one aspect of the present disclosure relates to a program that causes a computer to function as each of the above units, or a computer-readable information storage medium storing the program.

[0019] According to one aspect of the present disclosure, information on a display object or information on a display mode displayed on a first display device is acquired. Further, a process is performed to allow a user to visually recognize a content image in which a display image of a second display device is added to the display image of the first display device. And at this time, a process is performed to change a notification mode of content information according to the information on the display object or the display mode of the first display device. In this way, in an entertainment system that allows a user to visually recognize a content image in which a display image of a second display device is added to the display image of the first display device, it becomes possible to notify the user of appropriate content information according to the display object or the display mode of the first display device. Therefore, it becomes possible to provide an entertainment system or the like that can notify a user of content information related to a first display device in a system using the first display device and the second display device.

[0020] Also, in one aspect of the present disclosure, the content processing unit may perform display control of an object displayed on the second display device according to the information on the display object of the first display device or the display mode of the first display device.

[0021] In this way, it becomes possible to display an object whose display is controlled according to the display object or the display mode of the first display device on the second display device, and allow the user to visually recognize a content image in which an image of the object is added to the display image of the first display device.

[0022] Also, in one aspect of the present disclosure, the content processing unit may set a display position of the object displayed on the second display device according to the information on the display object of the first display device or the display mode of the first display device.

[0023] In this way, it becomes possible to display an object at a display position according to the information on the display object or the display mode of the first display device, and allow the user to visually recognize a content image in which an image of the object is added to the display image of the first display device.

[0024] Also, in one aspect of the present disclosure, the content processing unit may set a display mode of the object displayed on the second display device according to information on the display content of the first display device or the display mode of the first display device.

[0025] In this way, it becomes possible to display the object in a display mode corresponding to the information or display mode of the display content of the first display device, and to let the user visually recognize a content image in which the image of the object is added to the display image of the first display device.

[0026] Also, in one aspect of the present disclosure, the object displayed on the second display device may be an object related to the display content displayed on the first display device.

[0027] In this way, an object related to the display content of the first display device can be displayed by the second display device as an object superimposed on the display image of the first display device.

[0028] Also, in one aspect of the present disclosure, the information on the display content of the first display device may be parameter information of the display content, display position information of the display content, or display mode information of the display content.

[0029] In this way, it becomes possible to change the notification mode of the content information notified using the second display device according to the parameter information, display position information, or display mode information of the display content.

[0030] Another aspect of the present disclosure relates to an entertainment system using a first display device and a second display device through which a display image of the first display device is visible, the entertainment system including an image analysis unit that performs analysis processing on a captured image obtained by photographing the first display device with a camera of the second display device, and a content processing unit that performs processing to allow a user to view a content image in which a display image of the second display device is added to a display image of the first display device, the content processing unit performing notification processing to notify the user of content information based on a result of the analysis processing of the captured image using the second display device. Another aspect of the present disclosure relates to a program that causes a computer to function as each of the above units, or a computer-readable information storage medium storing the program.

[0031] According to one aspect of the present disclosure, analysis processing is performed on a captured image obtained by photographing a first display device with a camera included in a second display device. Also, processing is performed to allow a user to view a content image in which a display image of the second display device is added to a display image of the first display device. At this time, notification processing is performed to notify the user of content information based on a result of the analysis processing of the captured image using the second display device. In this way, the user can use the second display device to know content information based on the result of the analysis processing of the captured image obtained by photographing the first display device with the camera of the second display device. Therefore, it is possible to provide an entertainment system or the like that can notify a user of content information related to the first display device in a system using the first display device and the second display device.

[0032] Also, in one aspect of the present disclosure, the content processing unit may change the display content of the first display device or the second display device, or the notification mode of the content information, according to the positional relationship between the first display device and the second display device.

[0033] By doing so, an appropriate display content according to the positional relationship between the first display device and the second display device can be used to display an image on the first display device or the second display device, or an appropriate notification mode according to the positional relationship between the first display device and the second display device can be used to notify the user of the content information by the second display device.

[0034] In addition, in one aspect of the present disclosure, the content processing unit may change the display content of the first display device or the second display device, or the notification mode of the content information according to the positional relationship between the first display device and the second display device and the information of the display object displayed on the first display device.

[0035] By doing so, an appropriate display content according to the positional relationship between the first display device and the second display device and the information of the display object on the first display device can be used to display an image on the first display device or the second display device, or an appropriate notification mode according to the positional relationship and the information of the display object can be used to notify the user of the content information by the second display device.

Brief Description of the Drawings

[0036]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Embodiments for Carrying Out the Invention

[0037] Hereinafter, this embodiment will be described. Note that the embodiment described below does not unduly limit the content described in the claims. Also, not all of the configurations described in this embodiment are essential constituent elements.

[0038] 1. Entertainment System Fig. 1 shows an example of the system configuration of the entertainment system according to the present embodiment. The entertainment system according to the present embodiment is realized using a first display device 10 and a second display device 200. For example, the entertainment system according to the present embodiment is a system that provides various entertainment contents to a user using a first display device 10 and a second display device 200 capable of visually recognizing the display image of the first display device 10. The first display device 10 is, for example, the main display device. Specifically, the first display device 10 is a monitor (display) of an information processing device such as a PC (personal computer), a monitor of a television, a monitor of a game device, or a large monitor installed in a live or club venue. In Fig. 1, an image of game content in which characters CH1, CH2, CH3 and characters CE1, CE2, CE3 are in battle is displayed on the first display device 10. The second display device 200 is, for example, a sub-display device. For example, the second display device 200 is a display device capable of visually recognizing the display image of the first display device 10. For example, the user can visually recognize the display image of the first display device 10 through the display unit of the second display device 200. In Fig. 1, a transmissive HMD (broadly, HMD, transmissive display device) is used as the second display device 200. The transmissive HMD is a transmissive head-mounted display device worn by the user. The transmissive HMD is, for example, an optical transmissive HMD.

[0039] Note that hereinafter, the case where the second display device 200 is a transmissive HMD will be mainly taken as an example for explanation, but the second display device 200 may be a transmissive display device other than the transmissive HMD. For example, although the transmissive HMD is a display device worn on the user's head, the second display device 200 may be a type of transmissive display device not worn on the user's head. For example, the second display device 200 may be a transmissive display device held by the user in the hand or placed at a desired location by the user for use. Alternatively, the second display device 200 may be a display device other than the transmissive display device, for example, a display device that enables the display image of the first display device 10 to be visually recognized by displaying the captured image of the first display device 10 by a camera on the display unit.

[0040] FIG. 2 is an explanatory diagram of a content image in which the display image IMH of the second display device 200 is added to the display image IMD of the first display device 10. In the present embodiment, a process is performed to allow the user to visually recognize a content image in which the display image IMH of the second display device 200 is added (superimposed) to the display image IMD of the first display device 10. For example, when the second display device 200 is a transmissive HMD, the display image IMD of the first display device 10 is transmitted through the optical element of the second display device 200 and visually recognized by the user. Then, the display image IMH of the second display device 200 is displayed superimposed on the display image IMD of the first display device 10 that passes through the second display device 200. The display image IMH of the second display device 200 is, for example, a virtual image displayed by the second display device 200. Taking FIG. 5 described later as an example, the display image IMH is the display image of the display device 243 of the transmissive HMD 202 that is the second display device 200. By doing so, it becomes possible to allow the user to visually recognize a content image in which the display image IMH of the second display device 200 is added to the display image IMD of the first display device 10.

[0041] Note that in FIG. 2, an example of an image of game content is shown as the content image. However, the content image of the present embodiment is not limited to such an image of game content, and images of various contents (visual contents) such as distribution videos, movies, environmental videos, tourism videos, or simulation videos can be assumed.

[0042] Also, when the second display device 200 is a transmissive display device other than a transmissive HMD, for example, a transmissive tablet terminal, the display image IMD of the first display device 10 is transmitted through the transmissive tablet terminal and displayed to the user. Then, by displaying the display image IMD transmitted through the transmissive tablet terminal and the display image IMH generated by the transmissive tablet terminal to the user, the user can visually recognize the content image in which the display image IMH of the second display device 200 is added to the display image IMD (transmissive image) of the first display device 10. Also, when the second display device 200 is a display device other than a transmissive display device, for example, a terminal device such as a smartphone or a portable game device, for example, the display image IMD of the first display device 10 is photographed by a camera. Then, by superimposing and displaying the display image IMH generated by the second display device 200 on the photographed image of the camera, the user can visually recognize the content image in which the display image IMH of the second display device 200 is added to the display image IMD (photographed image) of the first display device 10.

[0043] In FIG. 2, using the second display device 200, content information related to the first display device 10 is displayed as a display image IMH. Specifically, the content information related to the first display device 10 is, for example, content information related to the display object displayed on the first display device 10 or the display content of the first display device 10. For example, on the first display device 10, a character CH1 or the like is displayed as a display object. Then, the content information related to this character CH1 is displayed on the second display device 200 as the display image IMH, and the display image IMH of this content information is superimposed and displayed on the display image IMD of the first display device 10. In FIG. 2, the content information notified by the display image IMH is game information, which is game information related to the character CH1 that is the display object of the first display device 10. Specifically, the game information that is the content information is information such as the hit points (HP), magic points (MP), experience points (EXP), level (LV), and equipment (EQP) of the character CH1. In the display image IMH, the information of the hit points (HP) is displayed by the object OB1 of the hit point gauge, and the information of the magic points (MP) is displayed by the object OB2 of the magic point gauge. The information of the experience points (EXP) and the level (LV) are respectively displayed by the objects OB3 and OB4 of the numbers representing the experience points and the level, and the information of the equipment (EQP) is displayed by the object OB5 of the icon representing the equipment.

[0044] Thus, in this embodiment, a notification process is performed to notify the user of content information related to the display object or display content of the first display device 10 using the second display device 200. Taking FIG. 2 as an example, content information (HP, MP, EXP, LV, EQP) related to the character CH1, which is the display object of the first display device 10, is notified to the user using the second display device 200. Specifically, the display image IMH of the content information related to the character CH1 is displayed using the second display device 200 (display device 243) and superimposed on the display image IMD of the first display device 10 so that the user can visually recognize it, thereby notifying the user of the content information. That is, the notification process of the content information in this case is a process of displaying the content information (display image IMH) using the second display device 200.

[0045] Thus, in this embodiment, a process is performed to allow the user to visually recognize a content image (FIG. 2) in which the display image IMH of the second display device 200 is added to the display image IMD of the first display device 10. Also, in this embodiment, based on the information on the user's line-of-sight direction, the user's fixation area on the display screen of the first display device 10 is detected. Then, a notification process is performed to notify the user of the content information corresponding to the detected fixation area using the second display device 200.

[0046] For example, in FIG. 3, the line-of-sight direction LS of the user US is detected. The line-of-sight direction LS of the user US can be detected, for example, by detecting the direction (facing direction) of the second display device 200. Specifically, the information on the line-of-sight direction LS can be obtained by tracking processing such as the direction of the second display device 200. Alternatively, the line-of-sight direction LS of the user US may be detected by a line-of-sight tracking process (eye tracking process) that directly detects the line of sight of the user US. Or when the first display device 10 is photographed by a camera, the photographing direction of the camera may be regarded as the line-of-sight direction LS of the user US and detected. In this case, the camera can be regarded as the user US, and the fixation area of the user US becomes the fixation area of the camera. Or the direction in which a transmissive display device such as a transmissive tablet terminal faces may be regarded as the line-of-sight direction of the user US. In this case, the fixation area of the user US is the area that the direction in which the transmissive display device faces gazes at (the area where the line along the facing direction intersects the display screen of the first display device 10).

[0047] Then, as shown in FIG. 3, by detecting the line-of-sight direction LS of the user US, the attention area ARG of the user US on the display screen of the first display device 10 can be detected. For example, the attention area ARG (the attention area of the camera and the transmissive display device) can be detected by obtaining the position where the line extending the line-of-sight direction LS (the shooting direction of the camera, the direction in which the transmissive display device faces) intersects the display screen of the first display device 10. In this case, it is desirable to obtain the positional relationship between the first display device 10 and the second display device 200 (the user, the camera). For example, the relative positional relationship of the first display device 10 as seen from the second display device 200 is obtained, and based on the information of this positional relationship and the information of the line-of-sight direction LS, the attention area ARG (the attention area) that the user US is gazing (paying attention) at is detected. Then, a notification process is performed to notify the user US of the content information corresponding to the detected attention area ARG using the second display device 200. In FIG. 3, a notification process is being performed in which the display image IMH, which is the content information corresponding to the attention area ARG, is overlaid and displayed on the display image IMD of the first display device 10. For example, in FIG. 3, the user US is gazing at the character CH1 in the display image IMD of the first display device 10, and the display area of this character CH1 is the attention area ARG. Therefore, as shown in FIG. 2, the character status information such as the hit points, magic points, experience points, level, and equipment of the character CH1 is displayed as the display image IMH (virtual image) of the second display device 200 as the content information corresponding to the attention area ARG where the character CH1 is displayed. That is, the images of the objects OB1, OB2, OB3, OB4, OB5 representing the hit points, magic points, experience points, level, and equipment of the character CH1 are overlaid and displayed on the display image IMD of the first display device 10 as the display image IMH of the display device 243 of the second display device 200.

[0048] In this way, by the user US gazing at the character CH1 that the user himself / herself pays attention to in the display image IMD of the first display device 10, information such as hit points, magic points, experience points, level, equipment, etc., which are content information related to the character CH1, will be notified to the user US. As a result, the user US can know various content information related to the character CH1 that the user himself / herself pays attention to just by gazing at the character CH1. Also, for example, if display images IMH of content information (status information) such as hit points and magic points are overlaid and displayed on the display image IMD for all characters CH1 to CH3 and CE1 to CE3 displayed on the first display device 10, the display of this content information will be obstructive and will result in a display that is difficult for the user US to view. In this regard, if only the content information corresponding to the gaze area ARG of the user US is displayed, it is possible to prevent a situation where the display of the content information becomes obstructive and difficult for the user US to view.

[0049] As described above, the notification process in FIG. 3 is a process of displaying content information using the second display device 200. By performing the display process of content information as the notification process of content information in this way, the user US can visually recognize the content of the content information corresponding to the attention area ARG more easily. However, the notification process of the present embodiment is not limited to such a display process of content information. For example, in FIG. 4, the notification process is performed by vibration or sound. Specifically, a rare character RCH appears on the display screen of the first display device 10. For example, the rare character RCH appears in the attention area in the line-of-sight direction LS of the user US. When a specific display object such as the rare character RCH appears in the first display device 10 (attention area) in this way, the vibration device (vibration mechanism) provided in the second display device 200 is vibrated to notify the user US that the rare character RCH has appeared. Alternatively, the user US is notified that the rare character RCH has appeared by the sound from the speaker built in the second display device 200 or the speaker of the headphones provided separately from the second display device 200. In this way, the notification process of the content information related to the display object or display content of the first display device 10 (content information corresponding to the attention area) may be a process of vibrating the vibration device of the second display device 200 or an output process of sound such as voice by the sound output unit 192 (FIG. 6) provided corresponding to the second display device 200. Alternatively, a light emitting element visible to the user US may be provided in the second display device 200, and the notification process of the content information may be realized by the light emission of this light emitting element. Thus, various modes can be assumed as the notification mode of the notification process of the content information.

[0050] When multiple users, such as a first user and a second user, enjoy entertainment content, the first content information corresponding to the first user is notified to the first user using a second display device such as a first transmissive HMD worn by the first user, and the second content information corresponding to the second user is notified to the second user using a third display device such as a second transmissive HMD worn by the second user. Specifically, a first fixation area is detected based on the information of the line-of-sight direction of the first user, and the first content information corresponding to the first fixation area is notified to the first user using a second display device such as a first transmissive HMD worn by the first user. Also, a second fixation area is detected based on the information of the line-of-sight direction of the second user, and the second content information corresponding to the second fixation area is notified to the second user using a third display device such as a second transmissive HMD worn by the second user. In this way, for example, for the first user, a first content image in which only the first content information is superimposed on the display image IMD of the first display device 10 is displayed, and for the second user, a second content image in which only the second content information is superimposed on the display image IMD of the first display device 10 is displayed. Therefore, the first user can visually recognize only the first content information that the first user pays attention to, and the second user can visually recognize only the second content information that the second user pays attention to, so that it becomes possible to display a content image in a display mode that is easy for each user to view.

[0051] In the above description, the case where the content is game content has been explained. However, the content of the present embodiment is not limited to this, and it may be content such as a distribution video, a movie, an environmental video, a tourism video, or a simulation video. For example, in the case of content of a distribution video in which a virtual character dances or sings, information about the character may be notified as content information. In the case of movie content, various explanation screens, subtitle information, etc. may be notified as content information. In the case of tourism video content, for example, various tourism information in the user's gaze area may be notified as content information. In the case of simulation video content, various information about the object of the simulation may be notified as content information.

[0052] FIG. 5 shows an example of a transmissive HMD 202 (broadly, an HMD, a transmissive display device), which is an example of the second display device 200. The transmissive HMD in FIG. 5 is an example of an optical transmissive HMD.

[0053] The transmissive HMD 202 in FIG. 5 has a temple part 240 and a goggle part 242. A speaker is built into the temple part 240. A display device 243 and a holographic optical element 244 are provided in the goggle part 242. For example, the display unit 30 of the transmissive HMD 202 (broadly, the second display device) can be realized by these display device 243, holographic optical element 244, etc. The display device 243 is provided with a display unit composed of a microdisplay, a mirror, a prism, etc. The display device 243 is provided with a display unit for the left eye and a display unit for the right eye, whereby stereoscopic vision is realized. Also, by providing a pair of microdisplays in each of the display units for the left eye and the right eye, the position of the virtual image can be made variable.

[0054] The display light from the display device 243 is guided in front of the user's eyes while refracting inside the light guide plate of the holographic optical element 244. Then, the holographic optical element 244 refracts the display light in the direction of the eyeball to send the display light to the eyeball. As a result, a virtual image (reproduced image of the hologram) appears to be in front of the eyes. The light guide plate of the holographic optical element 244 is also called a combiner, and the combiner, which is a half mirror, enables the real image of the external world and the virtual image to overlap and be seen, making it possible to realize AR (Augmented Reality), MR (Mixed Reality), etc.

[0055] In addition, the goggle unit 242 is provided with an RGB camera 246, a depth camera 247, and environmental recognition cameras 248 and 249. By using the RGB camera 246, it becomes possible to photograph the front direction of the user. By using the depth camera 247, depth information in the front direction can be obtained. Note that the goggle unit 242 is provided with an emitter (not shown) for the depth camera. By using the environmental recognition cameras 248 and 249, it becomes possible to recognize the environment around the user. Also, the goggle unit 242 incorporates an inertial measurement unit (IMU), which is a motion sensor composed of an acceleration sensor, a gyro sensor, etc. The position information and direction information of the transmissive HMD 202 are detected based on the captured images by the cameras provided in the goggle unit 242 and the measurement information from the inertial measurement unit, thereby realizing head tracking. By detecting the position information and direction information of the transmissive HMD 202, it also becomes possible to acquire the position information and direction information (line-of-sight direction information) of the user. Also, through processing using the RGB camera 246, the depth camera 247, etc., it is possible to acquire the position information (relative position information), direction information (relative direction information), or posture information of other users around the user. For example, these information are acquired by a process called Kinect (registered trademark). For example, by acquiring the posture information, it becomes possible to detect what actions other users have taken.

[0056] According to the transmissive HMD 202 of FIG. 5, for example, the display unit 30 realized by the display device 243, the holographic optical element 244, etc. enables the user to visually recognize the display image IMD and the display image IMH. For example, regarding the display image IMD of the first display device 10, it can be visually recognized by the user as a transmissive image (real image of the external world) through the combiner using the holographic optical element 244. Also, regarding the display image IMH, it can be visually recognized by the user as a virtual image by displaying it on the display device 243 of the transmissive HMD 202. As a result, it becomes possible to realize the process of allowing the user to visually recognize the content image in which the display image IMH (virtual image) of the transmissive HMD 202 is added to the display image IMD (transmissive image) of the first display device 10.

[0057] Note that the transmissive HMD 202 is not limited to the configuration of FIG. 5, and various modifications can be made, for example, to the configuration of the display unit 30 and the method of virtual image display. Also, when using the transmissive HMD 202, the user's operation input may be realized by the user operating a controller (not shown), or by detecting that the user's line of sight is directed towards the input display object through eye tracking or the like. Alternatively, the user's operation input may be detected by detecting the movement of the user's head or the like by a motion sensor provided in the transmissive HMD 202.

[0058] Also, as described above, the second display device 200 is not limited to the transmissive HMD 202, and may be a transmissive display device such as a transmissive tablet terminal that the user holds and uses, or a terminal device having a camera function such as a smartphone, a portable game device, or a tablet terminal.

[0059] FIG. 6 is a block diagram showing a configuration example of the entertainment system (game system, content providing system, processing system) of the present embodiment. Note that the amusement system of the present embodiment is not limited to the configuration of FIG. 6, and various modifications such as omitting some of its components or adding other components are possible.

[0060] The operation unit 160 is for the user (player) to perform various inputs (operations). The operation unit 160 can be realized by, for example, operation buttons, direction keys, keyboards, joysticks, or levers. Alternatively, the function of the operation unit 160 may be realized using eye tracking, motion, etc. as described above.

[0061] The imaging unit 162 (camera) is for photographing a subject (imaging object). The imaging unit 162 can be realized by, for example, an image sensor such as a CCD or CMOS sensor, or an optical system composed of a focus lens, etc. The imaging unit 162 which is a camera is provided, for example, on the second display device 200.

[0062] The storage unit 170 stores various information. The storage unit 170 functions as a work area for the processing unit 100, the communication unit 196, etc. Programs and data necessary for the execution of the programs are held in this storage unit 170. The function of the storage unit 170 can be realized by a semiconductor memory (DRAM, VRAM), HDD (hard disk drive), SSD (Solid State Drive), or an optical disk device, etc. The storage unit 170 stores, for example, content information. The content information is information such as images and sounds that make up the content. The image information of the content is, for example, information of objects (display items) that make up the content.

[0063] The information storage medium 180 is a computer-readable medium and stores programs, data, etc. The information storage medium 180 can be realized by an optical disk (DVD, BD, CD), HDD, or a semiconductor memory (ROM), etc. The processing unit 100 performs various processes of this embodiment based on the programs (data) stored in the information storage medium 180. That is, the information storage medium 180 stores programs (programs for causing a computer to execute the processes of each part) for causing a computer (a device equipped with an input device, a processing unit, a storage unit, and an output unit) to function as each part of this embodiment.

[0064] The first display device 10 is, for example, a main monitor, which can be realized by a TV monitor, a monitor for a PC, a monitor for a game device, or a monitor installed in facilities such as live facilities and amusement facilities. The display unit 30 is a display provided in the second display device 200 (such as the transmissive HMD 202). The first display device 10 and the display unit 30 can be realized by, for example, an LCD or an organic EL display. The sound output unit 192 outputs the sound generated according to the present embodiment, and its function can be realized by a speaker, headphones, or the like.

[0065] The communication unit 196 communicates with the outside (other devices) via a wired or wireless network, and its function can be realized by hardware such as a communication ASIC or a communication processor, or communication firmware.

[0066] Note that the program (data) for causing a computer to function as each part of the present embodiment may be distributed from an information storage medium of a server (host device) to the information storage medium 180 (or the storage unit 170) via the network and the communication unit 196. The use of such an information storage medium by a server can also be included within the scope of the present embodiment.

[0067] The processing unit 100 (processor) performs information acquisition processing, fixation area detection processing, image analysis processing, positional relationship detection processing, or content processing, etc., based on the programs, data stored in the storage unit 170, and the operation information of the operation unit 160.

[0068] Each process of the present embodiment performed by each part of the processing unit 100 can be realized by a processor (a processor including hardware). For example, each process of the present embodiment can be realized by a processor that operates based on information such as a program and a memory that stores information such as a program. The processor may be, for example, a separate hardware for each function of each part, or the functions of each part may be realized by integrated hardware. For example, the processor includes hardware, and the hardware can include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the processor can also be composed of one or more circuit devices (such as ICs, etc.) mounted on a circuit board and one or more circuit elements (such as resistors, capacitors, etc.). The processor may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to the CPU, and various processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) can be used. Also, the processor may be a hardware circuit by an ASIC. Also, the processor may include an amplifier circuit or a filter circuit for processing analog signals. The memory (storage unit 170) may be a semiconductor memory such as SRAM or DRAM, or a register. Alternatively, it may be a magnetic storage device such as a hard disk drive (HDD), or an optical storage device such as an optical disk drive. For example, the memory stores instructions readable by a computer, and when the instructions are executed by the processor, the processes of each part of the processing unit 100 are realized. The instructions here may be an instruction set constituting a program or instructions for instructing operations to the hardware circuit of the processor.

[0069] The processing unit 100 includes an information acquisition unit 102, a gaze area detection unit 104, an image analysis unit 105, a positional relationship detection unit 106, and a content processing unit 110. The content processing unit 110 includes a notification processing unit 112. As described above, each process of the present embodiment executed by these units can be realized by a processor (or a processor and a memory). Note that various modifications such as omitting some of these components (each unit) or adding other components are possible.

[0070] The information acquisition unit 102 performs a process of acquiring various information necessary for the process of the processing unit 100. For example, the information acquisition unit 102 acquires information on the gaze direction of the user. For example, information on the gaze direction of the user is acquired by tracking processing using the second display device 200. For example, by tracking processing, the direction (facing direction) of the second display device 200 is detected as the gaze direction of the user. Alternatively, eye tracking processing may be performed to detect the gaze direction of the user. The position and direction of the second display device 200 can be detected based on an imaging image by a camera, measurement information from an inertial measurement unit, or the like. Alternatively, a light emitting unit may be provided on the second display device 200, and the position and direction of the second display device 200 may be detected by imaging the light from this light emitting unit with a camera or the like provided outside. Alternatively, the position and direction of the second display device 200 may be detected by imaging the light from a light emitting unit provided outside with a light receiving unit (camera) provided on the second display device 200. Further, the information acquisition unit 102 acquires information on the display object displayed on the first display device 10 or information on the display mode of the first display device 10. The information on the display object is, for example, position information of the display object, display mode information of the display object (color, shape, brightness, transparency, etc.), or, when the display object is a character, various information about the character (parameter information such as status parameters, etc.). The information on the display mode of the first display device 10 is information such as the brightness, color tone, or contrast of the display screen of the first display device 10.

[0071] The gaze area detection unit 104 performs a process of detecting the gaze area of the user. For example, the gaze area detection unit 104 detects the gaze area of the user on the display screen of the first display device 10. For example, based on the information on the line-of-sight direction acquired by the information acquisition unit 102, the gaze area, which is the area on the display screen of the first display device 10 that the user is gazing at, is detected. For example, the gaze area detection unit 104 obtains the intersection of the line along the line-of-sight direction and the display screen of the first display device 10, and detects the area including the obtained intersection as the gaze area of the user.

[0072] Note that the gaze area is not limited to the area where the user actually directs the line-of-sight and gazes. For example, in an example where the first display device 10 can be visually recognized by the camera of the second display device 200, the gaze area of the user can be set as the area that the camera is gazing at. For example, regarding the direction in which the camera is facing as the line-of-sight direction of the user, the gaze area detection unit 104 obtains the intersection of the line along the direction in which the camera is facing and the display screen of the first display device 10, and detects the area including the obtained intersection as the gaze area of the user. In this case, it is desirable for the content processing unit 110 to display a display object for identifying the intersection position, such as a marker, at the position of the intersection on the display screen of the first display device 10 so that the user can visually recognize the position of the intersection. Alternatively, in an example where the first display device 10 can be visually recognized by facing the second display device 200, which is a transmissive display device such as a transmissive tablet terminal, toward the first display device 10, the gaze area of the user can be set as the area that the user gazes at by facing the second display device 200. That is, for example, the direction in which the second display device 200 is facing is regarded as the line-of-sight direction of the user. The direction in which the second display device 200 is facing is, for example, the direction orthogonal to the main surface of the second display device 200, and for example, the direction orthogonal to the display unit 30 (display screen). Then, the gaze area detection unit 104 obtains the intersection of the line along the direction in which the second display device 200 is facing and the display screen of the first display device 10, and detects the area including the obtained intersection as the gaze area of the user. Also in this case, it is desirable for the content processing unit 110 to display a display object for identifying the intersection position, such as a marker, at the position of the intersection on the display screen of the first display device 10 so that the user can visually recognize the position of the intersection.

[0073] The image analysis unit 105 performs analysis processing on the captured image. For example, it performs analysis processing on a captured image of the first display device 10 taken by the camera of the second display device 200. The camera is, for example, the imaging unit 162, and in the case of the transmissive HMD 202 in FIG. 5, it is, for example, an RGB camera 246 or the like. Also, when the second display device 200 is a terminal device such as a smartphone, a portable game device, or a tablet terminal, the camera that captures the captured image is, for example, the camera possessed by the terminal device. The image analysis unit 105 performs analysis processing on, for example, the display image of the first display device 10 reflected in the captured image of the camera. For example, the image analysis unit 105 performs analysis processing on the display content of the display image of the first display device 10 and the display mode of the first display device 10. Alternatively, the image analysis unit 105 may perform image analysis processing for detecting the user's gaze area on the display screen of the first display device 10.

[0074] The positional relationship detection unit 106 detects the positional relationship between the first display device 10 and the second display device 200. For example, the positional relationship detection unit 106 detects the relative positional relationship of the second display device 200 with respect to the first display device 10. Here, the positional relationship may be the relationship between the position of the first display device 10 and the position of the second display device 200, or may be the relationship between the direction (facing direction) of the first display device 10 and the direction (facing direction) of the second display device 200. For example, the positional relationship may be the relative directional relationship of the second display device 200 with respect to the first display device 10. As an example, the positional relationship detection unit 106 performs real-space recognition processing using a camera (RGB camera, depth camera, environment recognition camera, etc.) provided in the second display device 200, and based on the real-space information obtained by this recognition processing, detects the relative positional relationship of the first display device 10 with respect to the second display device 200. Alternatively, a marker may be installed on the first display device 10, and the positional relationship between the second display device 200 and the first display device 10 may be detected using this marker. For example, a plurality of markers (AR markers, etc.) are installed on the first display device 10, and the camera of the second display device 200 captures these plurality of markers, and by performing analysis processing on the obtained captured image, the relative positional relationship of the first display device 10 with respect to the second display device 200 is detected.

[0075] The content processing unit 110 performs various content processes. The content processes are various processes related to content, such as display processing of the content image and output processing of the content sound. The display processing of the content image is not limited to the process of generating and displaying the content image, and may be a process of instructing the display of the content image or a process of instructing the content of the displayed content image. The content processing unit 110 includes a notification processing unit 112. The notification processing unit 112 performs a process of notifying the user of content information using the second display device 200. For example, it performs a process of notifying the user of content information corresponding to the user's gaze area using the second display device 200.

[0076] Then, the entertainment system according to this embodiment using the first display device 10 and the second display device 200 where the display image of the first display device 10 is visible includes a gaze area detection unit 104 and a content processing unit 110 as shown in FIG. 6.

[0077] The gaze area detection unit 104 detects the user's gaze area on the display screen of the first display device 10 based on the information on the user's line-of-sight direction. For example, the gaze area detection unit 104 detects the user's gaze area based on the information on the user's line-of-sight direction acquired by the information acquisition unit 102. For example, it detects the gaze area based on the information on the user's line-of-sight direction acquired by the tracking process of the second display device 200. Alternatively, it detects the gaze area from the direction in which the camera is facing or the direction in which the second display device 200, which is a transmissive display device, is facing.

[0078] The content processing unit 110 performs a process of allowing a user to visually recognize a content image in which the display image of the second display device 200 is added to the display image of the first display device 10. For example, as shown in FIG. 2, control processing for allowing the user to visually recognize a content image in which the display image IMH of the second display device 200 is added to the display image IMD of the first display device 10 is performed on the second display device 200 or the like. For example, the content processing unit 110 controls the second display device 200 so that the display image IMD is displayed to the user as a transparent image and the display image IMH is displayed to the user as a superimposed image of a virtual image or a real image.

[0079] Then, the content processing unit 110 performs a notification process of notifying the user of the content information corresponding to the detected gaze area using the second display device 200. For example, when a gaze area is detected, a process of notifying the user of the content information corresponding to the gaze area is performed using, for example, the hardware of the second display device 200 (such as a display unit, a sound output unit, a vibration device, or a light emitting element). For example, a process of notifying information such as information on the display object in the gaze area, display content, display mode, status information, or event occurrence information in the gaze area is performed. The content information corresponding to the gaze area is, for example, content information related to the gaze area. Specifically, the content information corresponding to the gaze area is information related to the display object displayed in the gaze area, the display content in the gaze area, the display mode of the gaze area, or an event occurring in the gaze area. The content information is information related to the entertainment content realized by the entertainment system. When the entertainment content is game content, the content information is game information. The game information is, for example, information on the display objects appearing in the game, game progress information, game status information, or information on the user playing the game. The information on the display object is parameter information of the display object, display position information, display mode information, or the like. Or the content information is various information about entertainment content such as a distribution video, a movie, an environmental video, a tourism video, or a simulation video. For example, the content information is information related to the content of these contents, status information of the content, progress information of the content, or explanatory information of the content.

[0080] Also, the notification process of the content information is, for example, a process of displaying the content information using the second display device 200. For example, the notification process is a process of displaying an image of the content information on the display device 243 of the second display device 200. For example, the notification process is a process of displaying an image of the content information as an image superimposed on the display image of the first display device 10. However, the notification process is not limited to this, and may be a sound output process, a vibration process, or a light emission process of the light emitting element for the content information.

[0081] Also, the content processing unit 110 changes the notification mode of the content information according to the information of the display object displayed on the first display device 10 or the display mode of the first display device 10. The information of the display object is, for example, the parameter information, display position information, or display mode information of the display object. The display mode of the first display device 10 is, for example, the brightness, color tone, contrast, or display content of the display screen of the first display device 10. For example, the content processing unit 110 changes the notification content, notification display object, notification position, notification event, or notification timing, etc. of the content information according to the information of the display object or the display mode of the first display device 10.

[0082] For example, the content processing unit 110 performs display control of the object displayed on the second display device 200 according to the information of the display object of the first display device 10 or the display mode of the first display device 10. For example, as a process of changing the notification mode of the content information according to the information of the display object or the display mode of the first display device 10, display control of the object displayed on the second display device 200 is performed. For example, the content processing unit 110 determines the display / non-display of the object or changes the display mode of the object according to the information of the display object or the display mode of the first display device 10. For example, according to the information of the display object or the display mode of the first display device 10, the display mode such as the color tone, brightness, size, shape, transparency, or contour of the object is set.

[0083] For example, the information of the object being viewed is the information of the object being gazed at by the user on the display screen of the first display device 10. Then, the content processing unit 110 sets the display position of the object to be displayed on the second display device 200 according to the information of the object being gazed at on the first display device 10 or the display mode of the first display device 10. That is, as the display control of the object to be displayed on the second display device 200 according to the information of the object being gazed at or the display mode of the first display device 10, the display position of the object to be displayed on the second display device 200 is set (determined). For example, when the object being gazed at by the user is determined, the content processing unit 110 controls to display the object on the second display device 200 at a position corresponding to the position of the object being gazed at. Alternatively, it controls to display the object on the second display device 200 at a position corresponding to the size and shape of the object being gazed at. Alternatively, it controls to display the object on the second display device 200 at an appropriate position according to the display mode and display content of the display screen of the first display device 10. In this embodiment, for the sake of clarity of description, the objects (such as OB1 to OB5 in FIG. 2) to be displayed on the display device 243 of the second display device 200 are distinguished from the "objects" on the first display device 10 and are described as "objects". This object is an object that is displayed as an overlapping image on the display image of the first display device 10 by the second display device 200.

[0084] In addition, the content processing unit 110 sets the display mode of the object to be displayed on the second display device 200 according to the information of the object being gazed at or the display mode of the first display device 10. That is, as the display control of the object to be displayed on the second display device 200 according to the information of the object being gazed at or the display mode of the first display device 10, the display mode of the object to be displayed on the second display device 200 is set (determined). For example, when the object being gazed at by the user is determined, the content processing unit 110 controls to display the object on the second display device 200 in a display mode corresponding to the object being gazed at. Alternatively, it controls to display the object on the second display device 200 in a display mode corresponding to the brightness, color, shape, or size of the object being gazed at. Alternatively, it controls to display the object on the second display device 200 in an appropriate display mode according to the display mode and display content of the display screen of the first display device 10.

[0085] Moreover, the entertainment system of this embodiment includes an information acquisition unit 102 and a content processing unit 110. The information acquisition unit 102 acquires information on the display content displayed on the first display device 10 or information on the display mode of the first display device 10. The content processing unit 110 performs a process of allowing the user to visually recognize a content image in which the display image of the second display device 200 is added to the display image of the first display device 10. The content processing unit 110 performs a notification process of notifying the user of content information using the second display device 200, and changes the notification mode of the content information according to the information or display mode of the display content of the first display device 10. For example, the content processing unit 110 changes the notification content, notification display content, notification position, notification event, notification timing, etc. of the content information according to the information or display mode of the display content of the first display device 10.

[0086] In this case, the content processing unit 110 performs display control of the object displayed on the second display device 200 according to the information or display mode of the display content of the first display device 10. For example, as a process of changing the notification mode of the content information according to the information or display mode of the display content of the first display device 10, display control of the object displayed on the second display device 200 is performed. For example, the content processing unit 110 determines whether to display or not display the object, or changes the display mode of the object, according to the information or display mode of the display content of the first display device 10.

[0087] Specifically, the content processing unit 110 sets the display position of the object displayed on the second display device 200 according to the information or display mode of the display content of the first display device 10. That is, as the display control of the object according to the information or display mode of the display content of the first display device 10, the display position of the object displayed on the second display device 200 is set (determined). For example, the content processing unit 110 controls to display the object on the second display device 200 at a position corresponding to the position of the display content of the first display device 10. Alternatively, it controls to display the object on the second display device 200 at a position corresponding to the size or shape of the display content. Alternatively, it controls to display the object on the second display device 200 at an appropriate position according to the display mode or display content of the display screen of the first display device 10.

[0088] In addition, the content processing unit 110 sets the display mode of the object displayed on the second display device 200 according to the information or display mode of the display content of the first display device 10. That is, as the display control of the object according to the information or display mode of the display content of the first display device 10, the display mode of the object displayed on the second display device 200 is set (determined). For example, the content processing unit 110 controls to display the object on the second display device 200 in a display mode corresponding to the display content of the first display device 10. Alternatively, it controls to display the object on the second display device 200 in a display mode corresponding to the brightness, color, shape, or size of the display content. Alternatively, it controls to display the object on the second display device 200 in an appropriate display mode according to the display mode or display content of the display screen of the first display device 10.

[0089] Also, the object displayed on the second display device 200 is an object related to the display item displayed on the first display device 10. For example, the object of the second display device 200 is an object that has been associated in advance with the display item of the first display device 10. For example, the object of the second display device 200 is stored in the storage unit 170 in association with the display item of the first display device 10. For example, a first object (first object group) is associated with the first display item, and a second object (second object group) is associated with the second display item. Then, for example, when a display item is displayed on the first display device 10, the object associated with this display item is displayed on the second display device 200.

[0090] Also, the information of the display item on the first display device 10 is, for example, the parameter information of the display item, the display position information of the display item, or the display mode information of the display item. The parameter information of the display item is the parameter information representing the state or situation of the display item, and for example, it is the game parameter information of the display item appearing in the game. For example, when the display item is a game character, the parameter information is the game parameter information of the character stored in the storage unit 170 in association with the character. For example, it is the parameter information representing the status, ability, level, or attribute of the character. The display position information of the display item is, for example, the position coordinates, display area, or display location on the display screen of the first display device 10. The display position information of the display item may be the position information of the display item in a virtual three-dimensional space. The display mode information of the display item is, for example, the information such as the color tone, brightness, size, shape, transparency, or texture of the display item.

[0091] Moreover, the entertainment system of the present embodiment includes an image analysis unit 105 and a content processing unit 110. The image analysis unit 105 performs analysis processing on a captured image obtained by photographing the first display device 10 with a camera of the second display device 200. For example, the information acquisition unit 102 acquires information on a captured image obtained by photographing the first display device 10 with a camera of the second display device 200, and the image analysis unit 105 performs analysis processing on the acquired captured image. For example, analysis processing is performed on a display image of the first display device 10 shown in the captured image of the camera. For example, the image analysis unit 105 performs analysis processing on a display object displayed on the first display device 10 or a display mode of the first display device 10. Alternatively, the image analysis unit 105 performs image analysis processing for detecting a user's attention area on the display screen of the first display device 10.

[0092] Then, the content processing unit 110 performs a notification process of notifying the user of content information based on the result of the analysis process of the captured image using the second display device 200. For example, based on the result of the analysis process of the captured image of the camera that captured the first display device 10, information on the display object displayed on the first display device 10 and the display mode of the first display device 10 are specified. Then, the content processing unit 110 performs a process of changing the notification mode of the content information according to the specified information on the display object of the first display device 10 or the display mode of the first display device 10. That is, the content processing unit 110 performs a process of changing the notification mode of the content information according to the information on the display object or the display mode specified based on the result of the analysis process of the captured image. Specifically, display control of the object displayed on the second display device 200 is performed according to the information on the display object or the display mode of the first display device 10. Alternatively, based on the result of the analysis process of the captured image of the camera that captured the first display device 10, the user's attention area on the display screen of the first display device 10 is detected. Then, the content processing unit 110 performs a notification process of notifying the user of the content information corresponding to the attention area detected based on the result of the analysis process of the captured image of the camera using the second display device 200. For example, assume that the attention display object stared at by the user is specified based on the result of the analysis process of the captured image of the camera. In this case, the content processing unit 110 performs a process of setting the display position of the object displayed on the second display device 200 or a process of setting the display mode of the object displayed on the second display device 200 according to the information on the attention display object or the display mode of the first display device 10.

[0093] Also, the content processing unit 110 changes the display content of the first display device 10 or the second display device 200, or the notification mode of the content information, according to the positional relationship between the first display device 10 and the second display device 200. For example, when the positional relationship detection unit 106 detects the positional relationship between the first display device 10 and the second display device 200, the content processing unit 110 changes the display content of the first display device 10 or the second display device 200, or changes the notification mode of the content information, according to the detected positional relationship. For example, the content processing unit 110 changes the type or display mode of the display object displayed on the first display device 10, or changes the type or display mode of the object displayed on the second display device 200, according to the positional relationship between the first display device 10 and the second display device 200. Or the content processing unit 110 changes the notification content, notification display object, notification position, notification event, or notification timing, etc. of the content information, according to the positional relationship between the first display device 10 and the second display device 200.

[0094] Also, the content processing unit 110 changes the display content of the first display device 10 or the second display device 200, or the notification mode of the content information, according to the positional relationship between the first display device 10 and the second display device 200 and the information of the display object displayed on the first display device 10. For example, the content processing unit 110 changes the type or display mode of the display object displayed on the first display device 10, or changes the type or display mode of the object displayed on the second display device 200, or changes the notification content, notification display object, notification position, notification event, or notification timing, etc. of the content information, according to the positional relationship between the first display device 10 and the second display device 200 and the information of the display object of the first display device 10.

[0095] FIG. 7(A) and FIG. 7(B) show an example of the hardware implementation configuration of the entertainment system according to the present embodiment. In FIG. 7(A), the information processing apparatus 40 is communicatively connected to the first display apparatus 10 and the second display apparatus 200 by wire or wirelessly. Then, the information processing apparatus 40 performs control processing on the first display apparatus 10 and the second display apparatus 200. The information processing apparatus 40 is, for example, a personal computer (PC), a game apparatus, or a communication terminal apparatus. The personal computer is a desktop PC, a notebook PC, a tablet PC, or the like. The game apparatus may be a home game apparatus or a business game apparatus installed in an amusement facility. The communication terminal apparatus is, for example, a portable communication terminal apparatus such as a smartphone. For example, in FIG. 7(A), the information processing apparatus 40 performs display processing of a display image on the first display apparatus 10 and also performs display processing of a display image on the second display apparatus 200. The display processing on the first display apparatus 10 can be realized by distributed processing between the information processing apparatus 40 and the first display apparatus 10, and the display processing on the second display apparatus 200 can be realized by distributed processing between the information processing apparatus 40 and the second display apparatus 200. Then, the information processing apparatus 40 performs processing to allow the user to visually recognize a content image in which the display image of the second display apparatus 200 is added to the display image of the first display apparatus 10. For example, the information processing apparatus 40 transmits information on the display image of the first display apparatus 10 to the first display apparatus 10 and transmits information on the display image of the second display apparatus 200 to the second display apparatus 200. Then, in the second display apparatus 200, processing is performed to superimpose the display image of the second display apparatus 200 on a transparent image of the display image of the first display apparatus 10.

[0096] In FIG. 7(B), the second display device 200, the server 50, and the information processing device 40 are communicatively connected via the network 60. The network 60 (distribution network, communication line) is a communication path using, for example, the Internet or a wireless LAN, and includes, in addition to a dedicated line (dedicated cable) for direct connection and a LAN such as Ethernet (registered trademark), a communication network such as a telephone communication network, a cable network, or a wireless LAN. Also, the communication method may be wired or wireless. The server 50 is a server operated by the operator of the entertainment system, and executes, for example, various control processes and management processes of the entertainment system. In FIG. 7(B), each process of the entertainment system of the present embodiment can be realized by, for example, distributed processing among the information processing device 40, the server 50, and the second display device 200. In FIG. 7(B), for example, the information processing device 40 performs display processing of the display image on the first display device 10, and the second display device 200 performs display processing of the display image on the second display device 200. Then, the server 50 performs various control processes and management processes on the information processing device 40 and the second display device 200. For example, the server 50 can perform control processes and management processes for allowing the user to visually recognize a content image in which the display image of the second display device 200 is added to the display image of the first display device 10.

[0097] 2. Method of the Present Embodiment Next, the method of the present embodiment will be described in detail.

[0098] 2.1 Notification Processing Using the Second Display Device The entertainment system of the present embodiment is an entertainment system using the first display device 10 and the second display device 200 where the display image of the first display device 10 is visible, as described in FIG. 1. And in the present embodiment, various notification processes using the second display device are realized. FIG. 8 is a flowchart for explaining a processing example of the present embodiment.

[0099] First, as described with reference to FIG. 3, the entertainment system according to the present embodiment acquires information on the line-of-sight direction LS of the user US (step S1). For example, the line-of-sight direction LS of the user US is detected by performing tracking processing such as the direction of the second display device 200 worn by the user US. Alternatively, the line-of-sight direction LS of the user US may be detected by performing line-of-sight tracking (eye tracking) of the user US.

[0100] Next, as shown in FIG. 3, the entertainment system detects the area ARG on the display screen of the first display device 10 that the user US is gazing at, based on the information on the line-of-sight direction LS of the user US (step S2). For example, in FIG. 3, the user US is gazing at the character CH1, which is their own operation character, and the area corresponding to the arrangement position of this character CH1 is detected as the gazing area ARG. As an example, an area including the intersection of the line along the line-of-sight direction LS of the user US and the display screen of the first display device 10 is detected as the gazing area ARG. As described above, the gazing area ARG may be detected based on the direction in which the camera of the second display device 200 faces, or the direction in which the second display device 200, which is a transmissive display device, faces.

[0101] When the entertainment system performs a process of allowing the user US to visually recognize a content image in which the display image of the second display device 200 is added to the display image of the first display device 10, the entertainment system performs a notification process of notifying the user of the content information corresponding to the attention area ARG by using the second display device 200 (step S3). For example, in FIG. 3, a character CH1 is displayed in the attention area ARG that the user US is looking at on the display screen of the first display device 10. Then, as the content information corresponding to the attention area ARG, information about the character CH1 that is the display object of the first display device 10 is notified. Specifically, as shown in FIG. 2, the status information of the character CH1 located in the attention area ARG is displayed to the user US as the display image IMH of the second display device 200 that is superimposed on the display image IMD of the first display device 10. In this way, the user US can use the second display device 200 to know the content information corresponding to the attention area ARG that the user has looked at on the display screen of the first display device 10. That is, the user US can obtain the content information corresponding to the attention area ARG by simply looking at the area desired by the user on the display screen of the first display device 10. Therefore, it becomes possible to provide an entertainment system or the like that can notify the user of the content information related to the first display device 10 in a system using the first display device 10 and the second display device 200. For example, in an entertainment system using the first display device 10 and the second display device 200, highly interesting production control can be realized.

[0102] Also, in the present embodiment, as the notification process in step S3 of FIG. 8, a process of displaying content information using the second display device 200 is performed. That is, as shown in FIGS. 2 and 3, by superimposing a display image IMH corresponding to the content information on the display image IMD of the first display device 10 using the second display device 200, the content information corresponding to the gaze area ARG is notified to the user. In this way, the user US can visually recognize the content information corresponding to the gaze area ARG that the user has gazed at on the display screen of the first display device 10 by the display image IMH of the second display device 200. That is, the user US can visually recognize the content information corresponding to the gaze area ARG by simply gazing at the area desired by the user on the display screen of the first display device 10 by the display image IMH of the second display device 200, and a suitable user interface in the entertainment system using the first display device 10 and the second display device 200 can be realized. Note that the notification process for notifying the user US of the content information is not limited to the notification process by image display as shown in FIGS. 2 and 3, and may be a notification process by vibration, sound, light emission, or the like as described with reference to FIG. 4.

[0103] Also, in the present embodiment, the notification mode of the content information is changed according to the information or display mode of the display object of the first display device 10. FIG. 9 is a flowchart for explaining a processing example of the present embodiment in this case.

[0104] First, the entertainment system acquires information on the display content displayed on the first display device 10 or information on the display mode of the first display device 10 (step S11). Taking FIG. 1 as an example, information on display content such as characters CH1 to CH3, CE1 to CE3 and background display content displayed on the first display device 10, and information on the display mode such as the brightness, color tone, contrast or display content of the display screen of the first display device 10 is acquired. This information on the display content and the display mode may be acquired based on the result of the analysis process of the captured image of the camera that has captured the first display device 10. Then, according to the information on the display content or the display mode of the first display device 10, the notification mode of the content information is changed (step S12). For example, according to the information on the display content or the display mode of the first display device 10, the notification content, the notification display content, the notification position, the notification event or the notification timing of the content information is changed. For example, when the display content of the first display device 10 is the first display content, or when the display mode of the first display device 10 is the first display mode, the content information is notified with the first notification content, the first notification display content, the first notification position, the first notification event or the first notification timing. When the display content of the first display device 10 is the second display content, or when the display mode of the first display device 10 is the second display mode, the content information is notified with the second notification content, the second notification display content, the second notification position, the second notification event or the second notification timing.

[0105] Specifically, the entertainment system notifies the user of the notification content corresponding to the display content of the first display device 10 or the content information of the notification display object. For example, when the display content of the first display device 10 is a character, it notifies the content information of the notification content corresponding to the character, or performs a notification process of displaying a notification display object corresponding to the character. For example, it notifies the content information of the notification content that notifies the character information, or displays a notification display object related to the character. Also, it notifies the content information of the notification event corresponding to the display content of the first display device 10, or notifies the content information at the notification timing corresponding to the display content of the first display device 10. For example, when the display content of the first display device 10 is a character, it notifies the content information such that a notification event related to the character occurs, or notifies the content information at the notification timing corresponding to the appearance timing of the character, etc. Or, according to the display mode such as the brightness, color tone, contrast, or display content of the display screen of the first display device 10, it changes the notification content, notification display object, notification position, notification event, or notification timing, etc. of the content information. For example, it notifies the user of the content information of the notification content or notification display object suitable for the display mode of the first display device 10. Or, it notifies the content information at a notification position suitable for the display mode of the first display device 10, notifies the content information such that a notification event suitable for the display mode of the first display device 10 occurs, or notifies the content information at the notification timing suitable for the display mode of the first display device 10. Thus, according to this embodiment, since the notification mode of the content information is changed according to the information or display mode of the display content of the first display device 10, the content information can be notified to the user in a notification mode suitable for the display content or display mode of the first display device 10. Therefore, in the entertainment system that allows the user to visually recognize the content image in which the display image of the second display device 200 is added to the display image of the first display device 10, it becomes possible to notify the user of appropriate content information corresponding to the display content or display mode of the first display device 10.

[0106] Also, in the present embodiment, display control of an object displayed on the second display device 200 is performed according to the information or display mode of the display content of the first display device 10. By doing so, an object that is display-controlled according to the display content or display mode of the first display device 10 is displayed on the second display device 200, and a content image in which the image of the object is added to the display image of the first display device 10 can be made visible to the user. Therefore, display control of an object that reflects the display content or display mode of the first display device 10 can be performed, and notification processing of content information using the object can be realized. FIG. 10 is a flowchart for explaining a processing example of the present embodiment in this case.

[0107] First, the entertainment system acquires information on the display content (gazed display content) displayed on the first display device 10 or information on the display mode of the first display device 10 (step S21). For example, information on the display content such as parameter information, display position information, or display mode information of the display content, or information on the display mode such as the brightness, color tone, contrast, or display content of the display screen of the first display device 10 is acquired. This information on the display content and display mode may be acquired based on the result of analyzing a captured image of the first display device 10 by a camera. Then, according to the information on the display content (gazed display content) or display mode of the first display device 10, the display position or display mode of the object displayed on the second display device 200 is set (step S22). That is, as display control of the object on the second display device 200 according to the information on the display content or display mode of the first display device 10, control for setting the display position or display mode of the object displayed on the second display device 200 is performed.

[0108] As described above, in this embodiment, the display position of the object displayed on the second display device 200 is set according to the information or display mode of the display object of the first display device 10. By doing so, it becomes possible to display the object at a display position corresponding to the information or display mode of the display object of the first display device 10 and let the user visually recognize a content image in which the image of the object is added to the display image of the first display device 10. Therefore, it becomes possible to display the object at a display position corresponding to the display object or display mode of the first display device 10 and realize the notification process of content information using the object.

[0109] For example, in FIG. 11(A), the character CH1, which is the display object of the first display device 10, is displayed at the left position of the display screen of the first display device 10. In this case, the display positions of the objects OB1 and OB2 displayed as the display image IMH on the second display device 200 are also set to the left position. Also, in FIG. 11(B), the character CH1, which is the display object of the first display device 10, is displayed at the right position of the display screen of the first display device 10. In this case, the display positions of the objects OB1 and OB2 displayed on the second display device 200 are also set to the right position. That is, the display positions of the objects displayed on the second display device 200 are set according to the position information of the display object, which is the information of the display object of the first display device 10. Alternatively, the display positions of the objects displayed on the second display device 200 may be set according to the information on the shape or size of the display object. For example, by judging the information of the display object such as the display position, shape, or size of the display object on the first display device 10, the object on the second display device 200 is displayed at a display position that does not obstruct the viewing of the display object. In this way, the object on the second display device 200 can be displayed at an appropriate display position according to the information of the display object such as the display position, shape, or size of the display object on the first display device 10. Alternatively, in this embodiment, the display positions of the objects displayed on the second display device 200 may be set according to the display mode of the first display device 10. For example, by judging the display mode such as the brightness, color tone, contrast, or display content of the first display device 10, the object on the second display device 200 is displayed at a display position suitable for notifying content information using the object. Alternatively, by judging the display mode of the first display device 10, the object on the second display device 200 is displayed at a display position that does not obstruct the display on the first display device 10. In this way, the object on the second display device 200 can be displayed at an appropriate display position according to the display mode of the first display device 10.

[0110] In this embodiment, the display mode of the object displayed on the second display device 200 is set according to the information or display mode of the display content of the first display device 10. For example, the display mode such as the color tone, brightness, size, shape, transparency, or outline of the object on the second display device 200 is set. By doing so, it becomes possible to display the object in a display mode corresponding to the information or display mode of the display content of the first display device 10, and to allow the user to visually recognize the content image in which the image of the object is added to the display image of the first display device 10. Therefore, it becomes possible to display the object in a display mode corresponding to the display content or display mode of the first display device 10, and to realize the notification process of the content information using the object.

[0111] For example, in FIG. 12, according to the information of the character CH1 which is the display content of the first display device 10, the display mode such as the color tone, brightness, size, shape, transparency, or outline of the objects OB1 and OB2 in the display image IMH of the second display device 200 is set. For example, the color tone and brightness of the objects OB1 and OB2 in the second display device 200 are set according to the color tone and brightness corresponding to the character CH1 which is the display content. For example, the color tone and brightness of the objects OB1 and OB2 are set to the same color tone and brightness as the character CH1. Alternatively, the size and shape of the objects OB1 and OB2 in the second display device 200 are set according to the size and shape corresponding to the character CH1 which is the display content. For example, when the character CH1 is large, the objects OB1 and OB2 are also enlarged, or the objects OB1 and OB2 having a shape corresponding to the shape of the character CH1 are displayed. Note that what kind of display content is displayed on the first display device 10 can be specified based on, for example, the result of the analysis process of the captured image of the camera that has captured the first display device 10.

[0112] Also, in FIG. 12, according to the display mode of the display screen (IMD) of the first display device 10, such as brightness, color tone, contrast, or display content, the display modes of the objects OB1 and OB2 of the display image IMH of the second display device 200, such as color tone, brightness, size, shape, transparency, or contour, are set. For example, when the display screen (IMD) of the first display device 10 is bright, the display modes of the objects OB1 and OB2 are set so that the visibility of the objects OB1 and OB2 displayed by the second display device 200 is high even on the bright display screen. For example, when the display screen (IMD) of the first display device 10 is bright, the color tone of the objects OB1 and OB2 in the second display device 200 is made black, the transparency is lowered (the opacity is increased), or the display modes of the objects OB1 and OB2 are set so that the contours of the objects OB1 and OB2 are prominent. Also, the color tone of the objects OB1 and OB2 in the second display device 200 is set to a color tone corresponding to the color tone of the display screen of the first display device 10. For example, the color tone of the objects OB1 and OB2 in the second display device 200 is set to the same color tone as the display screen of the first display device 10, or to a color tone that enhances the visibility of the objects OB1 and OB2 on the display screen. Alternatively, the size, shape, etc. of the objects OB1 and OB2 in the second display device 200 are set to the size, shape, transparency, etc. corresponding to the display content, which is the display content of the first display device 10. For example, the size, shape, transparency, etc. of the objects OB1 and OB2 in the second display device 200 are set to a size, shape, or transparency that does not interfere with the visibility of the display content of the first display device 10. Note that an ambient light sensor for detecting the brightness of ambient light may be provided, and based on the detection result of this ambient light sensor, display control of the objects OB1 and OB2 displayed by the second display device 200 may be performed. Also, the display mode of the display screen of the first display device 10, such as brightness, color tone, contrast, or display content, can be specified based on, for example, the result of the analysis process of the captured image of the camera that captured the first display device 10.

[0113] When detecting a fixation display object gazed at by a user on the display screen of the first display device 10, the display position of an object displayed on the second display device 200 is set according to the information of the fixation display object. For example, in FIG. 3, a user's fixation area ARG is detected based on the information of the user's line-of-sight direction LS, and thereby a character CH1, which is the user's fixation display object, is detected. In this case, the display positions of the objects (OB1, OB2) displayed on the second display device 200 are set according to the information of the detected fixation display object (character CH1). For example, the display positions of the objects displayed on the second display device 200 are set according to the parameter information, display position, or display mode, etc. of the fixation display object. For example, an object is displayed on the second display device 200 near the display position of the fixation display object, or an object is displayed at an appropriate display position according to the size or shape of the fixation display object. In this way, it becomes possible to cause the user to visually recognize a content image in which an image of the object is added to the display image of the first display device 10 by displaying the object at the display position corresponding to the information of the fixation display object on the first display device 10. Therefore, it becomes possible to realize a notification process of content information using the object by displaying the object at the display position corresponding to the fixation display object on the first display device 10. Note that the fixation display object gazed at by the user on the display screen of the first display device 10 can be specified based on, for example, the result of analyzing a captured image of a camera that has captured the first display device 10.

[0114] When detecting a fixation display object gazed at by a user on the display screen of the first display device 10, the display mode of the object displayed on the second display device 200 is set according to the information of the fixation display object. For example, as shown in FIG. 3, when the character CH1, which is the user's fixation display object, is detected, the display mode of the objects (OB1, OB2) displayed on the second display device 200 is set according to the information of the detected fixation display object (character CH1). For example, the display mode of the object displayed on the second display device 200 is set according to the parameter information, display position, or display mode, etc. of the fixation display object. For example, according to the parameter information of the fixation display object, the display mode such as the color tone, brightness, size, shape, or transparency of the object is changed. Or, an object with a color tone, brightness, size, or shape corresponding to the color tone, brightness, size, or shape, etc. of the fixation display object is displayed by the second display device 200. In this way, the object can be displayed in a display mode according to the information of the fixation display object of the first display device 10, and the user can be made to visually recognize a content image in which the image of the object is added to the display image of the first display device 10. Therefore, it becomes possible to realize a notification process of content information using the object by displaying the object in a display mode according to the fixation display object of the first display device 10.

[0115] Also, the object displayed on the second display device 200 is an object related to the display object displayed on the first display device 10. For example, in FIG. 2, the objects OB1 to OB5 displayed on the second display device 200 are objects that inform the user of the status information, etc. of the character CH1, which is the display object of the first display device 10, and are objects related to the character CH1, which is the display object. In this way, the object related to the display object of the first display device 10 can be displayed by the second display device 200 as an object superimposed on the display image of the first display device 10. Thereby, an object related to the display object of the display image of the first display device 10 can be superimposed and displayed by the second display device 200 on the display image of the first display device 10, and the content information by the object can be notified to the user.

[0116] Also, as shown in FIG. 13, the information of the display object of the first display device 10 is parameter information, display position information, or display mode information of the display object. The parameter information of the display object is information of parameters representing the status, ability, level, or attribute, etc. of the display object such as a character. Also, the display position information of the display object is, for example, the position coordinates, display area, or display location, etc. of the display object. The display mode of the display object is, for example, the color tone, brightness, size, shape, or transparency, etc. of the display object. In this way, it becomes possible to change the notification mode of the content information notified using the second display device 200 according to the parameter information, display position information, or display mode information of the display object.

[0117] Also, in this embodiment, for example, analysis processing of a captured image obtained by capturing the first display device 10 with the camera of the second display device 200 is performed. Then, notification processing of notifying the user of content information based on the result of the analysis processing of the captured image using the second display device 200 is performed. For example, notification processing of notifying the user of content information corresponding to the information or display mode of the display object specified based on the result of the analysis processing of the captured image of the camera using the second display device 200 is performed. For example, processing of changing the notification mode of the content information is performed according to the information or display mode of the display object specified based on the result of the analysis processing. Or notification processing of notifying the user of content information corresponding to the attention area detected based on the result of the analysis processing of the captured image of the camera using the second display device 200 is performed.

[0118] 2.2 Processing According to the Positional Relationship In the present embodiment, as shown in FIG. 14, the positional relationship between the first display device 10 and the second display device 200 is detected. This positional relationship may be the relationship between the position of the first display device 10 and the position of the second display device 200, or may be the relationship between the distance between the first display device 10 and the second display device 200, or the relationship between the direction of the first display device 10 and the direction of the second display device 200. Further, in FIG. 14, an example in the case where the second display device 200 is a transmissive HMD is shown, but the second display device 200 whose positional relationship with the first display device 10 is detected may be a transmissive display device other than the transmissive HMD, or a terminal device having a camera, or the like.

[0119] Then, in the present embodiment, according to the positional relationship between the first display device 10 and the second display device 200, the display content of the first display device 10 or the second display device 200, or the notification mode of the content information notified using the second display device 200 is changed. For example, when the first display device 10 and the second display device 200 are in the first positional relationship, the display content of the first display device 10 or the second display device 200 is set to the first display content, or the content information is notified in the first notification mode using the second display device 200. Further, when the first display device 10 and the second display device 200 are in the second positional relationship, the display content of the first display device 10 or the second display device 200 is set to the second display content, or the content information is notified in the second notification mode using the second display device 200. In this way, an appropriate display content corresponding to the positional relationship between the first display device 10 and the second display device 200 can be used to display an image on the first display device 10 or the second display device 200, or the content information can be notified to the user by the second display device 200 in an appropriate notification mode corresponding to the positional relationship between the first display device 10 and the second display device 200.

[0120] For example, when the positional relationship between the first display device 10 and the second display device 200 is such that the distance between them is far, the display content of the first display device 10 or the second display device 200 is made into display content that can be visually recognized by the user even at a far distance. On the other hand, when the positional relationship between the first display device 10 and the second display device 200 is such that the distance between them is close, the display content of the first display device 10 or the second display device 200 is made into display content with a higher level of information detail, for example, compared to when the distance is far. Also, according to the directional relationship between the first display device 10 and the second display device 200, the display content of the first display device 10 or the second display device 200 may be changed. For example, when the direction in which the second display device 200 faces is directly facing the first display device 10 and is directed straight toward the display screen from the front side of the first display device 10, the display content of the first display device 10 or the second display device 200 is made into the first display content. For example, display content with a high level of information detail is used. On the other hand, when the direction in which the second display device 200 faces is at an oblique angle with respect to the first display device 10, the display content of the first display device 10 or the second display device 200 is made into the second display content. For example, display content with a low level of information detail and simplified information transmission, or a non-display state, is used.

[0121] Also, according to the positional relationship between the first display device 10 and the second display device 200, the notification mode such as the notification content, the notification display object, or the notification position of the content information notified using the second display device 200 is changed. For example, when the positional relationship between the first display device 10 and the second display device 200 is a distant one, the notification content or the notification display object of the content information is set to the notification content or the notification display object for a distant location. For example, compared to when the distance is short, a notification with a lower level of information detail is used, or the content information is notified by a simplified notification display object. On the other hand, when the positional relationship between the first display device 10 and the second display device 200 is a close one, the notification content or the notification display object of the content information is set to the notification content or the notification display object for a close location. For example, compared to when the distance is long, a notification with a higher level of information detail is used, or the content information is notified by a more elaborate-shaped notification display object. Alternatively, the content information is displayed at an appropriate notification position according to the positional relationship between the first display device 10 and the second display device 200. For example, on the display screen of the first display device 10, the content information is displayed in an area that is easily visible from the position of the second display device 200.

[0122] Also, when the content information is notified by sound, vibration, or light as shown in FIG. 4, the notification mode of the content information by sound, vibration, or light may be changed according to the positional relationship between the first display device 10 and the second display device 200. For example, when the positional relationship between the first display device 10 and the second display device 200 is a distant one, the sound or vibration is made smaller, or the light emission is made weaker, and when the positional relationship is a close one, the sound or vibration is made larger, or the light emission is made stronger. Also, the on / off of the notification of the content information may be controlled according to the positional relationship between the first display device 10 and the second display device 200. For example, when the positional relationship between the first display device 10 and the second display device 200 is a close one, the notification of the content information is turned on, and when the positional relationship is a distant one, the notification of the content information is turned off. Alternatively, when the relationship between the direction of the first display device 10 and the direction of the second display device 200 is such that the display screen of the first display device 10 is easily visible from the second display device 200, the notification of the content information is turned on, and when the relationship is not easily visible, the notification of the content information is turned off.

[0123] Also, in this embodiment, the display content of the first display device 10 or the second display device 200, or the notification mode of the content information may be changed according to the positional relationship between the first display device 10 and the second display device 200 and the information of the display object displayed on the first display device 10. That is, not only the positional relationship between the first display device 10 and the second display device 200 but also the information of the display object displayed on the first display device 10 is considered, and the display content of the first display device 10 or the second display device 200 is changed, or the notification mode of the content information is changed. In this way, an appropriate display content according to the positional relationship between the first display device 10 and the second display device 200 and the information of the display object of the first display device 10 is used to display an image on the first display device 10 or the second display device 200, or an appropriate notification mode according to the positional relationship between the first display device 10 and the second display device 200 and the information of the display object is used to notify the user of the content information by the second display device 200.

[0124] On the condition that there is a predetermined positional relationship, such as the distance between the first display device 10 and the second display device 200 being close, or the display screen of the first display device 10 being easily visible from the second display device 200, the display content of the first display device 10 or the second display device 200, or the notification mode of the content information is changed according to the information of the displayed object of the first display device 10. For example, on the condition that the first display device 10 and the second display device 200 have a predetermined positional relationship (such as a close distance or an easily visible relationship), the display content of the first display device 10 or the second display device 200 is set to the display content corresponding to the information of the displayed object of the first display device 10. Alternatively, on the condition that the first display device 10 and the second display device 200 have a predetermined positional relationship (such as a close distance or an easily visible relationship), as described with reference to FIG. 9 above, the notification mode of the content information is changed according to the information of the displayed object of the first display device 10. For example, on the condition that the first display device 10 and the second display device 200 have a predetermined positional relationship, the notification mode, such as the notification content, the notified display object, or the notification position, of the content information notified using the second display device 200 is changed according to the information of the displayed object of the first display device 10. Alternatively, when the first display device 10 and the second display device 200 have a predetermined positional relationship (such as a close distance or an easily visible relationship), while performing a change process of changing the display content of the first display device 10 or the second display device 200, or the notification mode of the content information according to the information of the displayed object of the first display device 10, when the first display device 10 and the second display device 200 do not have a predetermined positional relationship, the change process is not performed.

[0125] Also, as shown in FIG. 3, when a user's fixation display object (CH1) is detected, on the condition that the first display device 10 and the second display device 200 are in a predetermined positional relationship, the display content of the first display device 10 or the second display device 200, or the notification mode of the content information is changed according to the information of the fixation display object. For example, when the distance between the first display device 10 and the second display device 200 is in a close positional relationship and a user's fixation display object on the display screen of the first display device 10 is detected, the display content of the first display device 10 or the second display device 200, or the notification mode of the content information is changed according to the information of the fixation display object. For example, the display content of the first display device 10 or the second display device 200 is set to be content suitable for the display of the fixation display object. Or, as described above, according to the parameter information, position information, display mode, etc. of the fixation display object, the notification mode such as the notification content, notification display object, or notification position of the content information notified using the second display device 200 is changed. By doing so, various notification processes of the content information according to the information of the fixation display object of the first display device 10 can be realized while reflecting the positional relationship between the first display device 10 and the second display device 200.

[0126] Next, an example of the detection process of the positional relationship between the first display device 10 and the second display device 200 will be described. In the second display device 200 described with reference to FIG. 5, real space information is acquired by performing recognition processing on the real space around the user, and based on the acquired real space information, a virtual space corresponding to the real space can be generated. For example, in FIG. 15(A), there are objects such as a desk DK and a wall WL in the real space. By performing recognition processing to scan this real space with a camera provided in the second display device 200 of FIG. 5, real space information is acquired. Specifically, as shown in FIG. 15(B), data of mesh polygons is acquired as real space information. This recognition processing of the real space is executed, for example, in real time, and the real space around the user is sequentially scanned to acquire real space information. For example, space data associated with an ID is sequentially acquired and stored in the storage unit 170 of FIG. 6 as real space information. This real space information also has depth information (depth value), and by using this depth information, for example, hidden surface removal processing between an object in the virtual space and an object in the real space becomes possible. Also, by generating a virtual space based on the real space information, interaction between virtual objects (characters, placement objects) and real space objects (desk DK, wall WL) becomes possible. For example, hit processing (collision processing) between a virtual object and a real space object also becomes possible, enabling the realization of types of AR (Augmented Reality) and MR (Mixed Reality) that have never existed before. Further, the real space information obtained by such real space recognition processing can also be used for the acquisition processing of the user's position information and direction information. For example, based on the real space information, the positional relationship and distance relationship between the user and the objects around the user can be grasped. Therefore, by using the detection result of the inertial measurement unit and the real space information, it becomes possible to acquire more accurate user position information and direction information.

[0127] As shown in FIG. 16, real-space information about the first display device 10 as seen from the second display device 200 is acquired through real-space recognition processing using a camera or the like of the second display device 200. Specifically, data of the mesh polygons of the first display device 10 is acquired. Then, based on the data of the mesh polygons which is the real-space information of the first display device 10, for example, a plurality of virtual markers MK1, MK2, and MK3 are set for the corner portions of the first display device 10. In FIG. 16, virtual markers MK1, MK2, and MK3 are set for at least three corner portions of the first display device 10, for example. These markers MK1 to MK3 are virtual markers in markerless AR, for example. Then, using these virtual markers MK1 to MK3, the positional relationship between the first display device 10 and the second display device 200 is detected. For example, by obtaining the distances between the position of the second display device 200 and the positions of the markers MK1 to MK3, the positional relationship between the first display device 10 and the second display device 200 can be detected. In this way, by effectively using the real-space recognition processing using a camera or the like of the second display device 200, the positional relationship between the first display device 10 and the second display device 200 can be detected. And as described above, according to the detected positional relationship and the like, processing for changing the display content of the first display device 10 or the second display device 200, or the notification mode of the content information can be realized.

[0128] Also, in the present embodiment, as shown in FIG. 17, AR or MR image display may be performed such that the object OB corresponding to the display object of the first display device 10 pops out from the screen of the first display device 10. For example, as described with reference to FIG. 3, the user's gaze display object on the display screen of the first display device 10 is detected, and a display process is performed in which an image in which the object OB corresponding to the detected gaze display object moves from the screen of the first display device 10 toward the second display device 200 is visually recognized by the user using the second display device 200. For example, such a display process can be realized by controlling the parallax in stereoscopic display. By doing so, a notification process for notifying the user of the object OB which is the content information corresponding to the detected gaze area (gaze display object) using the second display device 200 can be realized.

[0129] Although the present embodiment has been described in detail as above, those skilled in the art will easily understand that many modifications can be made without substantially departing from the novel matters and effects of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the present disclosure. For example, in the specification or drawings, a term (such as a transmissive HMD) described at least once together with a broader or synonymous different term (such as a second display device) can be replaced with the different term at any location in the specification or drawings. Also, the information acquisition process, the gaze area detection process, the image analysis process, the content process, the notification process, the positional relationship detection process, etc. are not limited to those described in the present embodiment, and equivalent methods, processes, and configurations are also included in the scope of the present disclosure.

Explanation of Reference Numerals

[0130] 10…First display device, 30…Display unit, 40…Information processing device, 50…Server, 60…Network, 100…Processing unit, 102…Information acquisition unit, 104…Gaze area detection unit, 105…Image analysis unit, 106…Positional relationship detection unit, 110…Content processing unit, 112…Notification processing unit, 160…Operation unit, 162…Imaging unit, 170…Storage unit, 180…Information storage medium, 192…Sound output unit, 196…Communication unit, 200…Second display device, 202…Transmissive HMD, 240…Temple part, 242…Goggle part, 243…Display device, 244…Holographic optical element, 246…RGB camera, 247…Depth camera, 248, 249…Environmental recognition camera, ARG…Gaze area, CH1~CH3, CE1~CE3…Characters, IMD, IMH…Display images, LS…Gaze direction, MK1~MK3…Markers, OB, OB1~OB5…Objects, RCH…Rare character, US…User

Claims

1. An entertainment system using a first display device and a second display device where the display image of the first display device is visible, comprising: a fixation area detection unit that detects a fixation area of the user on the display screen of the first display device based on information on the user's line-of-sight direction; a content processing unit that performs processing to allow the user to view a content image in which a second display image of the second display device is added to a first display image of the first display device; including the content processing unit performs a notification process of notifying the user of content information corresponding to the detected fixation area using the second display device; the content processing unit changes the display content of the second display device according to the positional relationship between the first display device and the second display device; an area where the second display image of the second display device is added to the first display image of the first display device is an inner area of the display area of the first display image; the content processing unit when the positional relationship between the first display device and the second display device is a close distance relationship, makes the display content of the second display device have a higher level of information detail compared to when the distance between the first display device and the second display device is far. An entertainment system characterized by this.

2. A program used in an entertainment system using a first display device and a second display device where the display image of the first display device is visible, causing a computer to function as: a fixation area detection unit that detects a fixation area of the user on the display screen of the first display device based on information on the user's line-of-sight direction; a content processing unit that performs processing to allow the user to view a content image in which a second display image of the second display device is added to a first display image of the first display device; such that the content processing unit performs a notification process of notifying the user of content information corresponding to the detected fixation area using the second display device; the content processing unit changes the display content of the second display device according to the positional relationship between the first display device and the second display device; an area where the second display image of the second display device is added to the first display image of the first display device is an inner area of the display area of the first display image; the content processing unit When the positional relationship is such that the distance between the first display device and the second display device is close, the display content of the second display device is made to be display content with a higher level of information detail compared to when the distance between the first display device and the second display device is far. A program characterized by this.

Images