Information processing apparatus, method of controlling information processing apparatus, and program

Abstract

To provide an information processing apparatus, a method of controlling an information processing apparatus, and a program which can display notification information of an external device in accordance with a timing desired by a user currently experiencing a video in response to the situation.SOLUTION: A CPU 201 of a HMD 100 acquires notification information of a smartphone 600, acquires a user viewing position with respect to display screens 104, 105 where a VR image are displayed, compares a specific position and the viewing position on the display screens 104, 105 to determine whether the notification information of the smartphone 600 should be displayed, and when it is determined that the notification information of the smartphone 600 is displayed, displays the notification information of the smartphone 600 on the display screens 104, 105.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] In recent years, head-mounted displays (hereinafter referred to as "HMDs") have been developed as devices that realize virtual reality, augmented reality, and mixed reality. Virtual reality (hereinafter referred to as "VR") is a technology that uses a computer to generate (simulate) an environment in which a user can interact with virtual objects. In the simulation, images, sounds, haptic feedback, and other pseudo-sensory information are used to provide the user with a perception similar to that of real space. In the following explanation, the three-dimensional (3D) space in such a simulation will be referred to as "virtual space." In other words, VR is a technology that provides users with images of a virtual space (hereinafter referred to as "VR images") as if they were real events.

[0003] Augmented reality (AR) is a simulation technology that displays virtual objects superimposed on real space. Mixed reality (MR) is a simulation technology that further extends augmented reality by superimposing objects that are not actually present on real space, allowing users to experience the experience from any position or angle. Among HMDs that realize VR, AR, and MR, users wearing HMDs that realize VR, especially VR, can block external visual information, allowing them to concentrate on watching VR videos or working, thereby achieving a high level of immersion. However, even when immersed in watching VR videos or working, users may still want to receive notification information from an external device other than the HMD. Patent Document 1, for example, discloses a technology that determines the importance of notification information from an external device and displays the notification information from the external device on an HMD based on the determined importance. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2014 / 156388 Summary of the Invention [Problem to be solved by the invention]

[0005] However, as described above, the technology disclosed in Patent Document 1 determines the display timing of notification information from an external device on an HMD based on the importance of the notification information. In other words, the display timing on an HMD is determined completely regardless of the timing at which the user desires the notification information from the external device to be displayed. Therefore, depending on the display timing determined, it may reduce the sense of immersion of a user watching VR video or working with an HMD. Furthermore, when a user is watching VR video through an HMD while wearing it, external visual information is blocked, making it difficult for the user to directly operate the external device. Furthermore, when playing a game using VR video, it is difficult for the user to let go of the game controller. Furthermore, if notification information from an external device is displayed on the HMD while VR video is being live-streamed, the notification information from the external device is also live-streamed along with the VR video, raising concerns about privacy and information leaks.

[0006] The present invention has been made in view of the above-mentioned problems, and aims to provide an information processing device, a control method for the information processing device, and a program that can display notification information from an external device at a timing desired by a user who is experiencing a video, depending on the situation. [Means for solving the problem]

[0007] In order to achieve the above object, the information processing device of the present invention is characterized by comprising: a notification acquisition means for acquiring notification information of an external device; a gaze position acquisition means for acquiring the user's gaze position relative to a display screen on which an image is being displayed; a first determination means for comparing a specific position on the display screen with the gaze position to determine whether or not to display notification information of the external device; and a display control means for displaying the notification information of the external device on the display screen when the first determination means determines that the notification information of the external device should be displayed. [Effects of the Invention]

[0008] According to the present invention, notification information from an external device can be displayed at a timing desired by a user who is experiencing a video, depending on the situation. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram illustrating the configuration of an HMD. [Figure 2] FIG. 1 is a cross-sectional view showing the HMD from the left eye side. [Figure 3] FIG. 1 is a diagram illustrating the principle of gaze detection. [Figure 4] FIG. 1A is a schematic diagram of an eyeball image projected onto an eyeball image sensor, and FIG. 1B is a diagram of the output intensity of the eyeball image sensor. [Figure 5] 10 is a flowchart of a method for calculating a gaze position using gaze detection. [Figure 6] (a) is a schematic diagram showing the configuration of a smartphone, and (b) is an external view of the smartphone. [Figure 7] 10 is a flowchart showing the operation of the smartphone when the timing of displaying notification information from the smartphone on the HMD is controlled. [Figure 8] 10 is a flowchart showing the operation of the HMD when the timing of displaying notification information from the smartphone on the HMD is controlled. [Figure 9]It is an explanatory diagram of the relationship between the direction and strength of pulling (or pushing) due to the difference in the position between the smartphone and the user. [Figure 10] In the HMD, it is a diagram showing the relationship between the user's gaze state and the display of notification information on the smartphone. [Figure 11] It is a diagram showing an example of notification information of the smartphone displayed in the HMD. [Figure 12] It is a diagram showing the change in display transparency according to the level of the user's degree of attention regarding the display of notification information of the smartphone in the HMD. [Figure 13] It is a diagram showing an example of notification information of the interphone displayed in the HMD.

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to FIGS. 1 to 13. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in the present embodiments. For example, each part constituting the present invention can be replaced with any configuration that can exhibit the same function. Also, any component may be added. Further, any two or more configurations (features) in the present embodiment can be combined. In the present embodiment, as an example of the information processing apparatus, the HMD to which the present invention is applied will be described. Also, the same reference numerals are given to the same configurations to omit the description.

[0011] <Configuration of HMD> First, the configuration of the HMD will be described. FIG. 1 is a schematic diagram showing the configuration of the HMD 100. FIG. 1 is a view of the HMD 100 as seen from the top of the user's head. When the user wears the housing 101 of the HMD 100 on their head, a display screen 104 for the left eye and a display screen 105 for the right eye are disposed in front of the user's left eyeball 102 and right eyeball 103. If the display screens 104 and 105 are see-through, the user can see real space through the display screens 104 and 105. In this case, the HMD 100 displays virtual objects on the see-through display screens 104 and 105, thereby enabling the virtual objects to be superimposed or merged with the real space seen by the user through the display screens 104 and 105. This allows the HMD 100 to realize AR and MR.

[0012] On the other hand, when the display screens 104, 105 are of a non-transparent type, eyepieces (not shown) are arranged between the display screens 104, 105 and the user's eyes 102, 103. In this case, when the display screens 104, 105 are in a non-transparent mode, internal images (such as captured video, CG, or game images) stored inside the HMD 100 are displayed on the display screens 104, 105. This allows the HMD 100 to realize VR. When the display screens 104, 105 are in a transparent mode, camera images captured by the left-eye camera 106 and the right-eye camera 107 are displayed on the display screens 104, 105. Furthermore, in this transparent mode, images in which virtual objects are superimposed on or integrated with the camera images are displayed on the display screens 104, 105, allowing the HMD 100 to realize AR or MR. The HMD 100 estimates where on the display screens 104 and 105 the user is gazing, using a left-eye line-of-sight detector 108 and a right-eye line-of-sight detector 109, as will be described later.

[0013] The HMD 100 has an inertial sensor 110. The inertial sensor 110 is configured with two sensors: an acceleration sensor that detects translational motion on the X, Y, and Z axes, and a gyro sensor that detects rotational motion on the YPR (yaw, pitch, roll) axes. By associating these two sensors, the inertial sensor 110 establishes inertial sensor fusion, enabling comprehensive detection of translational motion and rotational motion. Note that, although the inertial sensor 110 in this embodiment is configured using an acceleration sensor and a gyro sensor, it may also be configured using only a gyro sensor.

[0014] The HMD 100 has a controller 111a and a controller 111b. Although the controllers 111a and 111b are connected to the HMD 100 by wire in FIG. 1, they may also be connected wirelessly. The controllers 111a and 111b control game images, virtual objects, and the like displayed on the display screens 104 and 105. The user can control the controllers 111a and 111b independently with their left and right hands by holding them in their right and left hands, respectively. The controller 111a is provided with a cross key 112a, buttons 113a, a lever operation 114a, and a touch panel 115a. Similarly, the controller 111b is provided with a cross key 112b, buttons 113b, a lever operation 114b, and a touch panel 115b.

[0015] The cross keys 112a and 112b, buttons 113a and 113b, lever controls 114a and 114b, and touch panels 115a and 115b are operation members operated by the user. User operations on the controllers 111a and 111b are transmitted as operation signals to a CPU (201 in FIG. 2) described below. The HMD 100 has a position sensor 116. The position sensor 116 acquires information from a GPS or the like, and obtains position information for the HMD 100. However, if the position sensor constituting the illusionary tactile force sense unit (212 in FIG. 2) described below can be used for purposes other than the illusionary tactile force sense unit, the position sensor 116 is not necessary.

[0016] FIG. 2 is a cross-sectional view of the HMD 100 cut along the YZ plane defined by the Y axis and Z axis shown in FIG. 1, and is a schematic diagram showing a mechanism for detecting the gaze. FIG. 2 is a cross-sectional view showing the HMD 100 from the left eye side. Therefore, the mechanism for detecting the gaze of the left eye will be described below with reference to FIG. 2. The mechanism for detecting the gaze of the right eye is similar to the mechanism for detecting the gaze of the left eye. As shown in FIG. 2, the housing 101 of the HMD 100 contains units configured as described below. The HMD 100 has a CPU 201, a memory unit 202, and a communication unit 203.

[0017] The CPU 201 controls the entire HMD 100. The memory unit 202 stores video information, control programs, and the like. The communication unit 203 connects to a communicable external device and receives information from the external device. Examples of communicable external devices include mobile terminals such as smartphones, tablet PCs, digital cameras, and notebook PCs, as well as desktop PCs. The HMD 100 can use incoming call information and received information such as notifications and messages on SNS by connecting to, for example, a smartphone (600 in FIG. 6) described below via Bluetooth (registered trademark) or Wi-Fi (registered trademark) using the communication unit 203.

[0018] The HMD 100 is provided with a display screen 104, such as a liquid crystal display, for displaying images, and a drive circuit 204 for driving the display screen 104. When the display screen 104 is non-transmissive, the HMD 100 is provided with a camera image sensor 205, an aperture mechanism 206, and a focus mechanism 207 as mechanisms constituting a camera 106 that captures an image of the outside in transmissive mode. The HMD 100 also has a light splitter 208. The camera 106 can capture an image of the subject scene through the light splitter 208.

[0019] The HMD 100 has an illumination light source 209, a light-receiving lens 210, and an eyeball image sensor 211. The illumination light source 209 is a light source that illuminates the eyeball 102 to detect the gaze, and is composed of, for example, multiple infrared light-emitting diodes. The light-receiving lens 210 positions the pupil of the user's eyeball 102 and the eyeball image sensor 211 in a complementary imaging relationship. The eyeball image sensor 211 is a two-dimensional array of photoelectric elements such as CMOS. An eyeball image illuminated by the illumination light source 209 and an image resulting from the corneal reflection of the illumination light source 209 are formed on the eyeball image sensor 211 by the light-receiving lens 210. The gaze direction is detected from the positional relationship between the eyeball image formed on the eyeball image sensor 211 and the image resulting from the corneal reflection of the illumination light source 209 using a predetermined algorithm, which will be described later. The illumination light source 209, the light receiving lens 210, and the eye image sensor 211 constitute the left-eye line-of-sight detector 108. The memory unit 202 has a function of storing image signals from the camera image sensor 205 and the eye image sensor 211, as well as a function of storing line-of-sight correction data.

[0020] The HMD 100 also has an illusionary tactile force sense unit 212. The illusionary tactile force sense unit 212 realizes haptics (illusory tactile force sense technology) that realizes physical sensations. The illusionary tactile force sense unit 212 is composed of an acceleration sensor, a position sensor, and an eccentric motor, and acquires acceleration and position information of the user's body part in contact with it, and generates various vibrations using the eccentric motor in response to the information. As a result, the illusionary tactile force sense unit 212 can make the user feel the sensation of being pushed or pulled, and this haptic effect can provide the user with sensory instructions. Note that the illusionary tactile force sense unit 212 is not limited to being built into the HMD 100, but may also be built into the controllers 111a and 111b. The illusionary tactile force sense unit 212 may have any form as long as it is in contact with the user's body, such as a watch or a ring. The HMD 100 also has a RAM 213. The RAM 213 is a rewritable volatile memory that provides a working area for the CPU 201.

[0021] <Method for calculating gaze position using gaze detection> FIG. 3 is a diagram illustrating the principle of gaze detection. FIG. 3 corresponds to a summary diagram of the optical system for performing gaze detection in the HMD 100. In FIG. 3, the eyeball 102, pupil 301, and cornea 302 are organs of the user's eye. O' is the center of the eyeball 102. Reference numerals 303a and 303b are light sources such as infrared light-emitting diodes that emit infrared light that is insensitive to the user, and constitute the illumination light source 209. The light sources 303a and 303b are arranged approximately symmetrically with respect to the optical axis of the light-receiving lens 210, and illuminate the user's eyeball 102. A portion of the infrared light reflected by the eyeball 102 is focused by the light-receiving lens 210 onto the eye image sensor 211.

[0022] FIG. 4(a) is a schematic diagram of an eyeball image projected onto the eyeball image sensor 211. FIG. 4(b) is an output intensity diagram of the eyeball image sensor 211. FIG. 5 is a flowchart of a method for calculating a gaze position using gaze detection. Hereinafter, a method for calculating a gaze position for the left eye using gaze detection will be described with reference to FIGS. 3 to 5. The method for calculating a gaze position for the right eye using gaze detection is the same as that for the left eye. The flowchart shown in FIG. 5 is realized in the HMD 100 when the CPU 201 reads out a program stored in the memory unit 202, expands it into the RAM 213, and executes it.

[0023] 5 starts, in step S501, the CPU 201 causes the light sources 303a and 303b to emit infrared light toward the user's eyeball 102. An image of the user's eye illuminated by the infrared light is formed on the eyeball image sensor 211 through the light receiving lens 210. At this time, the eyeball image is photoelectrically converted by the eyeball image sensor 211, and can be processed as an electrical signal. In step S502, the CPU 201 receives the eyeball image signal obtained by the eyeball image sensor 211 as described above.

[0024] In step S503, the CPU 201 acquires the coordinates of points corresponding to the corneal reflection images Pd and Pe of the light sources 303a and 303b and the center c of the pupil 301 shown in FIG. 3 from the information of the eyeball image signal received in step S502, as follows: Infrared light emitted from the light sources 303a and 303b in step S501 illuminates the cornea 302 of the user's eyeball 102. At this time, the corneal reflection images Pd and Pe formed by part of the infrared light reflected from the surface of the cornea 302 are collected by the light receiving lens 210 and formed as reflection images Pd' and Pe' on the eyeball image sensor 211. Similarly, light beams from pupil edges a and b, which are the edges of the pupil 301, are also formed on the eyeball image sensor 211.

[0025] FIG. 4(a) shows example images of reflection images Pd' and Pe' obtained from the eye image sensor 211. FIG. 4(b) shows example luminance information obtained from the eye image sensor 211 in region α of the example image shown in FIG. 4(a). In FIG. 4(a), the horizontal direction is the X-axis and the vertical direction is the Y-axis. In FIG. 4(b), the horizontal direction is the X-axis and luminance is the Y-axis. Furthermore, for reflection images Pd' and Pe' formed by the corneal reflection images Pd and Pe of light sources 303a and 303b, respectively, their coordinates in the X-axis direction (horizontal direction) are designated Xd and Xe. Furthermore, for images a' and b' formed by light beams from pupil edges a and b of pupil 301, respectively, their coordinates in the X-axis direction are designated Xa and Xb.

[0026] In the example of luminance information in FIG. 4( b), the coordinates Xd and Xe of the reflection images Pd' and Pe' formed by the corneal reflection images Pd and Pe of light sources 303a and 303b, respectively, exhibit extremely high levels of luminance. In the region between coordinates Xa and Xb, which corresponds to the region of pupil 301, an extremely low level of luminance is exhibited except for the coordinates Xd and Xe. In contrast, in the region of iris 401 outside pupil 301, which has an X coordinate value smaller than coordinate Xa and an X coordinate value larger than coordinate Xb, an intermediate luminance between the two levels is exhibited. From the information on fluctuations in luminance level with respect to the X coordinate position, the coordinates Xd and Xe of the reflection images Pd' and Pe' formed by the corneal reflection images Pd and Pe of light sources 303a and 303b, respectively, and the coordinates Xa and Xb of the images a' and b' at the pupil edges a and b can be obtained.

[0027] Furthermore, when the rotation angle θx in the ZX plane of the optical axis of eyeball 102 relative to the optical axis of light receiving lens 210 is small, the coordinate Xc of point c' corresponding to the center c of pupil 301 imaged on eyeball image sensor 211 can be expressed as Xc≈(Xa+Xb) / 2. As described above, it is possible to obtain the coordinate Xc of point c' corresponding to the center c of pupil 301 and the coordinates Xd and Xe of reflected images Pd' and Pe' on the eyeball image sensor 211. In this way, the coordinates of the points corresponding to corneal reflection images Pd and Pe of light sources 303a and 303b and the coordinates of the point corresponding to the center c of pupil 301 are obtained from the information on the eyeball image signal.

[0028] Returning to FIG. 5, in step S504, CPU 201 acquires the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position of eyeball 102 relative to light receiving lens 210, and can essentially be calculated as a function of the distance between reflected images Pd' and Pe' (coordinate Xd - coordinate Xe). In step S505, CPU 201 acquires rotation angles θx and θy as follows. Note that rotation angle θy refers to the rotation angle in the ZY plane of the optical axis of eyeball 102 relative to the optical axis of light receiving lens 210.

[0029] 3, the X coordinate of the midpoint of reflected images Pd', Pe' and the X coordinate of the center of curvature O of cornea 302 almost coincide with each other. If the standard distance from center of curvature O of cornea 302 to center c of pupil 301 is Oc, then the rotation angle θx in the ZX plane of the optical axis of eyeball 102 relative to the optical axis of light receiving lens 210 can be calculated from the following equation (1): β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc…Equation (1)

[0030] Note that Figures 3 and 4 show an example in which the rotation angle θx is calculated when the user's eyeball 102 rotates within the ZX plane perpendicular to the Y axis, but the rotation angle θy when the user's eyeball 102 rotates within the ZY plane perpendicular to the X axis is calculated in the same way.

[0031] In step S506, CPU 201 uses the rotation angles θx and θy acquired in step S505 to calculate the gaze position, which is the position of the user's line of sight on display screen 104. If the gaze position is defined as coordinates (Hx, Hy) corresponding to center c of pupil 301 on display screen 104, it can be calculated from the following equations (2) and (3). Hx=m×(Ax×θx+Bx) …Equation (2) Hy=m×(Ay×θy+By) ...Equation (3)

[0032] The coefficient m is a conversion coefficient that converts the rotation angles θx and θy into position coordinates corresponding to the center c of the pupil 301 on the display screen 104. The coefficient m is a constant determined by the configuration of the optical system, and is determined in advance and stored in the memory unit 202. Furthermore, Ax, Bx, Ay, and By are line-of-sight correction coefficients that correct for individual differences in the user's line of sight, and are acquired by performing a calibration operation and stored in the memory unit 202 before the flowchart shown in FIG. 5 starts.

[0033] In step S507, the CPU 201 stores the gaze position calculated in step S506, that is, the coordinates (Hx, Hy) of the center c of the pupil 301 on the display screen 104, in the memory unit 202. Thereafter, the flowchart shown in Fig. 5 ends. Note that the above method is a method that uses the corneal reflection images Pd and Pe of the light sources 303a and 303b, but the method is not limited to this and any method may be used as long as it can acquire the rotation angles θx and θy of the eyeball 102 from the captured eyeball image.

[0034] <Smartphone configuration> Next, the configuration of a smartphone, which is an example of an external device connected to the HMD 100, will be described with reference to FIG. 6. FIG. 6(a) is a schematic diagram showing the configuration of the smartphone 600. FIG. 6(b) is an external view of the smartphone 600. As shown in FIG. 6(a), the smartphone 600 has a touchscreen display 601. The touchscreen display 601 has a liquid crystal display 602 and a touchscreen 603. The liquid crystal display 602 displays characters, images, icons, and the like. The touchscreen 603 can detect gesture operations.

[0035] The smartphone 600 further includes an in-camera 604, an out-camera 605, a light 606, and an illuminance sensor 607. The in-camera 604 and the out-camera 605 include lenses and imaging elements such as CCDs or CMOSs ​​that convert optical images into electrical signals. The in-camera 604 and the out-camera 605 are compact camera modules equipped with AF (autofocus), aperture, and shutter speed adjustment functions. The in-camera 604 captures an image of a subject facing the touchscreen display 601. The out-camera 605 captures an image of a subject located on the back of the smartphone 600. The light 606 is a light-emitting module that functions as a flash when capturing images with the in-camera 604 or the out-camera 605. The illuminance sensor 607 acquires illuminance information of the subject that is focused by the in-camera 604 or the out-camera 605. The illuminance information acquired by the illuminance sensor 607 is used to adjust the exposure time and ISO sensitivity during image capture.

[0036] The smartphone 600 further includes a CPU 608, a ROM 609, a RAM 610, and a switch 611. The CPU 608 is, for example, a central processing unit. The CPU 608 reads control programs for each block of the smartphone 600 from the ROM 609, loads them into the RAM 610, and executes them. In this way, the CPU 608 controls the operation of each block of the smartphone 600. The CPU 608 provides a camera function by controlling the touch screen 603, the in-camera 604, the out-camera 605, the light 606, the illuminance sensor 607, and the switch 611. The ROM 609 is an electrically erasable and recordable nonvolatile memory. The ROM 609 stores control programs for each block of the smartphone 600 as well as parameters required for the operation of each block. The RAM 610 is a rewritable volatile memory. The RAM 610 is used for loading programs executed by the CPU 608 and for temporarily storing data generated by the operation of each block of the smartphone 600.

[0037] The smartphone 600 further includes a speaker 612, a connector 613, a power supply unit 614, and a communication unit 615. When the switch 611 is set to "on" for sound output, the speaker 612 outputs a shutter sound during image capture and a warning sound. The connector 613 is an external interface used to connect the smartphone 600 to an external device. An AC adapter for charging a battery provided in the power supply unit 614 is connected to the connector 613, for example. The connector 613 is also used to input and output image data, audio data, and the like to and from an externally connected nonvolatile memory. The connector 613 may be a dedicated terminal such as a dock connector, or a general-purpose terminal such as a USB.

[0038] The power supply unit 614 has a rechargeable battery and supplies power to the entire smartphone 600. The battery provided in the power supply unit 614 may be, for example, a lithium-ion battery or a nickel-metal hydride battery. The communication unit 615 performs communication according to a predetermined wireless communication standard. Examples of the wireless communication standard include Wi-Fi under the IEEE 802.11 standard, Bluetooth, and NFC. The communication unit 615 only needs to support at least one of these wireless communication standards. Specific examples of communication include sending and receiving notifications or messages in applications installed on the smartphone 600, and downloading program modules for adding functions to the smartphone 600. The smartphone 600 also establishes an information processing system by connecting to the HMD 100 using the communication unit 615.

[0039] The smartphone 600 further includes an acceleration sensor 616, a position sensor 617, and a depth sensor 618. The acceleration sensor 616 detects the direction and magnitude of acceleration acting on the smartphone 600. The acceleration sensor 616 is capable of detection in three axes: X, Y, and Z. The position sensor 617 acquires information about the current location of the smartphone 600 using a GPS or the like. The depth sensor 618 measures the distance from the in-camera 604 or the out-camera 605 to a subject. Methods for measuring distance using the depth sensor 618 include measuring the time it takes for infrared light, light, or ultrasound to reflect off an object and bounce back, and acquiring depth information of the subject from a parallax image obtained by arranging multiple cameras or pixels in parallel. The communication unit 615 is used when transmitting information from the illuminance sensor 607, the acceleration sensor 616, the position sensor 617, the depth sensor 618, and the like.

[0040] <Overall flow> Next, a method for controlling the timing at which notification information from the smartphone 600 is displayed on the HMD 100 will be described. FIG. 7 is a flowchart showing the operation of the smartphone 600. The flowchart shown in FIG. 7 is implemented by the CPU 608 in the smartphone 600 reading out a program stored in the ROM 609, expanding the program in the RAM 610, and executing it. When the flowchart shown in FIG. 7 starts, first, in step S701, the CPU 608 determines whether or not notification information from an application has been received. Note that the application may be an SNS application installed via the communication unit 615, a short message, or the like.

[0041] If the CPU 608 determines that it has not received notification information for the application, the process returns to step S701. As a result, the CPU 608 repeats the process of step S701 until it receives notification information for the application. On the other hand, if the CPU 608 determines that it has received notification information for the application, the process proceeds to step S702. In step S702, the CPU 608 transmits the notification information for the application received in step S701 to the HMD 100 via the communication unit 615. At that time, the CPU 608 also transmits information regarding the current location of the smartphone 600 to the HMD 100 via the communication unit 615. Thereafter, the flowchart shown in FIG. 7 ends.

[0042] Fig. 8 is a flowchart showing the operation of the HMD 100. The flowchart (control method of an information processing device) shown in Fig. 8 is realized in the HMD 100 by the CPU 201 (computer) reading out a program stored in the memory unit 202, expanding it in the RAM 213, and executing it. Before the flowchart shown in Fig. 8 starts, it is assumed that the HMD 100 is powered on, the CPU 201 performs image processing, VR images and the like are displayed on the display screens 104 and 105, and the user can view the VR images and the like. Furthermore, it is assumed that notification information of an application of the smartphone 600 (hereinafter referred to as "notification information of the smartphone 600") and information regarding the current location of the smartphone 600 have been transmitted from the smartphone 600 to the HMD 100, as shown in Fig. 7.

[0043] 8 starts, in step S801, the CPU 201 (notification acquisition means) receives notification information of the smartphone 600 via the communication unit 203 (notification acquisition step). At this time, the CPU 201 (current location acquisition means) also receives information related to the current location of the smartphone 600 via the communication unit 203. As a result, the CPU 201 acquires the notification information and the current location of the smartphone 600.

[0044] In step S802, the CPU 201 (haptic control means) vibrates the HMD 100 using the illusionary tactile force sense unit 212, thereby notifying the user that notification information from the smartphone 600 has been received. To this end, the CPU 201 causes the illusionary tactile force sense unit 212 to generate a vibration pattern for indicating the current location of the smartphone 600 to the user. Furthermore, the CPU 201 notifies the user of the current location of the smartphone 600 by causing the HMD 100 to vibrate using the generated vibration pattern using the illusionary tactile force sense unit 212. Generally, the illusionary tactile force sense unit 212 stimulates the skin with the generated vibration pattern, thereby generating an illusion in the brain, and as a result, the user can be made to feel as if they are experiencing a sense of resistance or touch. In other words, the tactile illusion force sense unit 212 can simultaneously and freely express the "force sense" of being pulled or pushed, the "pressure sense" of tapping, knocking, squeezing, or soft and hard, and the "tactile sense" of the texture of the surface of an object, such as roughness.

[0045] The illusionary force sense unit 212 senses the head movement of the user wearing the HMD 100 using an acceleration sensor and a position sensor, and changes the acceleration pattern of the eccentric motor based on the obtained velocity, acceleration, and position information, thereby providing an illusionary force sense. In this manner, the illusionary force sense unit 212 changes the pattern of stimulation to the user. In this embodiment, the illusionary force sense unit 212 makes the user feel the direction and strength of a pull or push depending on the positional relationship between the user and the smartphone 600. In this way, the illusionary force sense unit 212 allows the user experiencing the VR video displayed on the HMD 100 to intuitively identify the current position of the smartphone 600, even in the non-transparent mode when the display screens 104 and 105 are opaque.

[0046] Here, the relationship between the direction and strength of the pull (or push) due to differences in the positions of the smartphone 600 and the user will be described with reference to Fig. 9. In Fig. 9, 900 is the position of the user wearing the HMD 100. 901, 902, and 903 are the positions of the smartphone 600. 911, 912, and 913 are the direction and strength of the pull for each position of the smartphone 600, and are shown as vectors. The direction and strength of the pull felt by the user changes depending on the vibration pattern according to the direction and distance from the position 900 of the user wearing the HMD 100 to each of the positions 901, 902, and 903 on the smartphone 600.

[0047] In this regard, since positions 901, 902, and 903 on the smartphone 600 are different from each other, the directions of vectors 911, 912, and 913 are also different from each other. Therefore, the illusionary tactile force sense unit 212 generates vibration patterns that make the user feel pulling in different directions for positions 901, 902, and 903 on the smartphone 600. Furthermore, the distances from the user's position 900 to positions 901, 902, and 903 on the smartphone 600 are farther from each of positions 901 and 902 on the smartphone 600 than from position 903 on the smartphone 600. Therefore, the pulling strength is greater for positions 901 and 902 on the smartphone 600 than for position 903 on the smartphone 600. The magnitude relationship between these pulling strengths is indicated by the lengths of vectors 911, 912, and 913 in Figure 9. The magnitude relationship between the lengths of vectors 911, 912, and 913 can be expressed by the following equation (4). Length of vector 911 = length of vector 912 > length of vector 913 ... Equation (4)

[0048] The illusionary tactile force sense unit 212 notifies the user that the smartphone 600 is located farther from the user by providing a stronger vibration pattern as the lengths of the vectors 911, 912, and 913 become longer. Note that, contrary to FIG. 9 , the illusionary tactile force sense unit 212 may notify the user that the smartphone 600 is located farther from the user by providing a weaker vibration pattern as the lengths of the vectors 911, 912, and 913 become longer. Furthermore, even for the same pulling strength, the sense of distance corresponding to that pulling strength varies from user to user. Therefore, the illusionary tactile force sense unit 212 can more accurately notify the user of the location of the smartphone 600 by previously calibrating the pulling direction and strength relative to the position in real space.

[0049] In order to reduce external stimuli to the user as much as possible, the illusionary tactile force sense unit 212 may generate vibrations that change only the pulling direction that indicates the direction in which the smartphone 600 is located, without adjusting the strength of the vibration pattern according to the distance to the smartphone 600. Furthermore, the CPU 201 (tactile control means) of the HMD 100 may generate vibrations in the illusionary tactile force sense unit 212 when, for example, the user's gaze position on the display screens 104 and 105 is far from the current position of the smartphone 600. This makes it easier for the user to gaze in the direction in which the smartphone 600 is located.

[0050] Returning to FIG. 8, in step S803, the CPU 201 (gaze position acquisition means) calculates the user's gaze position (gaze position acquisition step). The CPU 201 obtains the user's gaze position using the above-described formulas (2) and (3). Note that the CPU 201 may acquire the user's gaze position based on pre-prepared data, such as a correspondence table, instead of by calculating it as described above. In step S804, the CPU 201 (first determination means) determines whether the current position of the smartphone 600 and the user's gaze position on the display screens 104 and 105 match for a first predetermined time or longer (determination step). As a result, the CPU 201 determines whether the user's gaze is focused in the direction of the smartphone 600. Note that the current position of the smartphone 600 on the display screens 104 and 105 is the position where the user's gaze intersects with the display screens 104 and 105 when it is assumed that the user is looking at the current position of the smartphone 600 through the display screens 104 and 105. The CPU 201 calculates this position (specific position) using the current position of the smartphone 600 acquired in step S801, etc.

[0051] If the CPU 201 determines that the current position of the smartphone 600 and the user's gaze position on the display screens 104 and 105 have matched for at least the first predetermined time, the process proceeds to step S805. On the other hand, if the CPU 201 determines that the current position of the smartphone 600 and the user's gaze position on the display screens 104 and 105 have not matched for at least the first predetermined time, the process returns to step S803. As a result, the CPU 201 repeats the processes of steps S803 and S804 until the current position of the smartphone 600 and the user's gaze position match for at least the first predetermined time. However, if the process does not proceed to step S805 even after a predetermined time has elapsed since the process of step S802 was performed, the CPU 201 ends the flowchart shown in FIG. 8.

[0052] Note that the CPU 201 can acquire the current position of the smartphone 600 even when the smartphone 600 does not have the position sensor 617. In this case, the CPU 201 captures an image of the real space with the camera image sensor 205, and can acquire the current position of the smartphone 600 from the coordinates of an area in the image data acquired by the image capture where the subject image of the smartphone 600 is directly received.

[0053] Furthermore, the CPU 201 (direction determination means) may make the determination in step S804 by comparing the user's line of sight with a specific direction. The specific direction may be, for example, the direction in which the smartphone 600 is located. In this case, the CPU 201 calculates the direction in which the smartphone 600 is located using the current position of the smartphone 600 acquired in step S801, etc. Furthermore, the CPU 201 (detection means) detects the user's line of sight using the line of sight detectors 108 and 109. Note that the specific direction may be, in addition to the direction in which the smartphone 600 is located as described above, the direction in which a specific virtual object is located, the direction in which a specific physical object is located, a specific orientation, etc.

[0054] In step S805, the CPU 201 (display control means) displays the notification information of the smartphone 600 superimposed on the VR video or the like at the current position of the smartphone 600 on the display screens 104, 105 on which the VR video or the like is being displayed (display control step). Here, the relationship between the user's gaze state on the HMD 100 and the display of the notification information of the smartphone 600 will be described with reference to FIG. 10. FIG. 10(a) is a diagram showing the display of the VR video or the like on the HMD 100 from a top viewpoint when the user 1000 is not gazing at the current position of the smartphone 600. FIG. 10(b) is a diagram showing the display of the VR video or the like on the HMD 100 from a top viewpoint when the user 1000 is gazing at the current position of the smartphone 600.

[0055] As shown in FIGS. 10(a) and 10(b), VR video and the like are displayed on a spherical projection surface around the user 1000. Reference numeral 1001 denotes the projection surface that is the farthest in the display of the VR video and the like. In FIG. 10(b), the user 1000 is gazing at the current position of the smartphone 600, and therefore notification information 1002 of the smartphone 600 is displayed superimposed on the VR video and the like at the coordinates of the VR video and the like that correspond to the current position of the smartphone 600. In other words, the user 1000 can visually recognize the notification information 1002 of the smartphone 600 by gazing at the current position of the smartphone 600 only when he or she is concerned about a notification from the smartphone 600.

[0056] Here, an example of notification information of the smartphone 600 will be further described with reference to Fig. 11. Fig. 11(a) is a diagram showing an example of notification information of the smartphone 600 superimposed on VR video or the like on the display screens 104 and 105 of the HMD 100. Fig. 11(b) is a diagram showing an example of the notification information of the smartphone 600 of Fig. 11(a) displayed in detail on the display screens 104 and 105 of the HMD 100. Note that in the description using Fig. 11, the notification information of the smartphone 600 is, as an example, notification information of an SNS application capable of sending and receiving messages.

[0057] In FIG. 11(a), 1100 is a notification display frame. Notification information of the smartphone 600, which is composed of information 1101 to 1104, is displayed within the notification display frame 1100. Information 1101 is information indicating the name of the other party who sent a message to the user using the SNS application of the smartphone 600. Information 1102 is information indicating a portion of the message sent by the other party whose name is indicated in information 1101. Information 1103 is an icon indicating the profile set by the other party whose name is indicated in information 1101. Information 1104 is information indicating the name of the SNS application.

[0058] Note that a button 1105 is displayed within the notification display frame 1110. The button 1105 is a button for displaying details of the notification information of the smartphone 600. The user can operate the button 1105 with the controllers 111a and 111b. The user can also operate the button 1105 with their own line of sight. In this case, when the CPU 201 determines, for example, using the left-eye line-of-sight detector 108 and the right-eye line-of-sight detector 109, that the user is gazing at the button 1105, the notification information of the smartphone 600 is displayed in detail. Specifically, when the current position of the smartphone 600 and the user's gaze position match for a second predetermined time or longer on the display screens 104 and 105, the CPU 201 (display control means) displays the notification information of the smartphone 600 in detail, as shown in FIG. 11(b). Note that the second predetermined time is longer than the first predetermined time used in the determination of step S804. As a result, the display screens 104 and 105 display the notification information of the smartphone 600 in detail at the current location of the smartphone 600.

[0059] The notification information of smartphone 600 shown in FIG. 11(b) is displayed so that the user can see the entire text of the message sent by the other party whose name is indicated by information 1101. In FIG. 11(b), 1110 is a notification display frame. In addition to the above-mentioned information 1101 to 1104, notification information of an application configured by information 1111 is displayed within notification display frame 1110. Information 1111 is information indicating the entire text of the message sent by the other party whose name is indicated by information 1101, excluding the above-mentioned information 1102. Therefore, the entire text of the message sent by the other party whose name is indicated by information 1101 is displayed by information 1102 and information 1111. However, if the entire text of the message is too long to fit within notification display frame 1110, CPU 201 may display the entire text of the message by scrolling.

[0060] Furthermore, an operation screen 1112 is displayed within notification display frame 1110. Operation screen 1112 is an operation screen that allows the user to respond to notification information from smartphone 600. Buttons 1113 and 1114 that allow the user to select preset reply templates are displayed within operation screen 1112 so that the user can quickly reply to a message. Note that button 1115 is displayed within notification display frame 1110. Button 1115 is a button for closing notification display frame 1110. The user can operate buttons 1113 to 1115 with controllers 111a and 111b or with their own line of sight, in the same manner as button 1105 described above.

[0061] Returning to Fig. 8, in step S806, CPU 201 (second determination means) determines whether the user's gaze position has been away from the current location of smartphone 600 (i.e., the display position of notification information on smartphone 600) for a third predetermined time or more on display screens 104, 105. This determines whether the time during which the current location of smartphone 600 and the user's gaze position have shifted from a state in which they match to a state in which they do not match has continued for a third predetermined time or more.

[0062] If the CPU 201 determines that the user's gaze position on the display screens 104 and 105 has been away from the current location of the smartphone 600 for the third predetermined time or more, the process proceeds to step S807. On the other hand, if the CPU 201 determines that the user's gaze position on the display screens 104 and 105 has not been away from the current location of the smartphone 600 for the third predetermined time or more, the process returns to step S806 itself. As a result, the CPU 201 repeats the process of step S806 until the user's gaze position on the display screens 104 and 105 has been away from the current location of the smartphone 600 for the third predetermined time or more. In step S807, the CPU 201 determines that the user has lost interest in the notification information of the smartphone 600, and ends displaying the notification information of the smartphone 600 (including the buttons 1105 and 1115 and the operation screen 1112). The flowchart shown in FIG. 8 then ends.

[0063] In this way, the HMD 100 can display notification information from the smartphone 600 at a timing desired by the user experiencing VR video or the like depending on the situation. This also applies to the HMD 100 that realizes AR or MR. In particular, when the HMD 100 is displaying VR video such as a game, it can display notification information from the smartphone 600 so that the user experiencing the VR video can maintain their immersive experience.

[0064] <Transparent display> In this embodiment, when the current position of the smartphone 600 and the user's gaze position coincide on the display screens 104 and 105 of the HMD 100, the notification information of the smartphone 600 is displayed superimposed on VR video or the like. In this case, the CPU 201 (display control means) of the HMD 100 may control display to make the notification information of the smartphone 600 transparent depending on the level of the user's gaze on the notification information, so that the user can see the VR video or the like on which the notification information is superimposed. In this case, the CPU 201 measures, for example, the amount of displacement, which is the distance from the current position of the smartphone 600 to the user's gaze position, on the display screens 104 and 105, and determines that the smaller the measured amount of displacement, the higher the user's gaze level. Furthermore, the CPU 201 determines that the larger the measured amount of displacement, the lower the user's gaze level.

[0065] 12 is a diagram showing how the display transparency of the notification information display (hereinafter referred to as "notification display") of the smartphone 600 on the HMD 100 changes depending on the level of the user's gaze. In FIG. 12, the horizontal axis represents the amount of displacement (hereinafter abbreviated as "displacement amount"), which is the distance from the current position of the smartphone 600 to the user's gaze position on the display screens 104 and 105. The vertical axis represents the display transparency of the notification display. In the HMD 100, the CPU 201 determines that the greater the amount of displacement, the lower the user's gaze level, and performs display control in step S805 described above by increasing the display transparency of the notification display so that the notification information of the smartphone 600 is less visible to the user.

[0066] As shown in FIG. 12, in region 1201, the amount of shift is large and the display transparency of the notification display is 100%, so the notification information of the smartphone 600 is not displayed. In contrast, in region 1202, the display transparency of the notification display is set so that the user can simultaneously see both the notification information of the smartphone 600 and the display area of ​​the VR video or the like on which the notification information overlaps. In region 1202, the CPU 201 can detect the level of importance of the notification information of the smartphone 600 at the user's moment by using the user's gaze level (i.e., the amount of shift). In region 1203, the amount of shift is small and the display transparency of the notification display is 0%, so the notification information of the smartphone 600 is displayed non-transparently. In other words, in region 1203, the notification information of the smartphone 600 is visible to the user, but the display area of ​​the VR video or the like on which the notification information overlaps is invisible to the user.

[0067] The CPU 201 (display control means) may change the display transparency of the notification display on the display screens 104 and 105 according to the number of times the current position of the smartphone 600 and the user's gaze position coincide again within a certain period of time. In this case, the CPU 201 determines that the user's gaze level is lower as the number of times mentioned above decreases, and increases the display transparency of the notification display. The CPU 201 (display control means) may also change the display transparency of the notification display on the display screens 104 and 105 according to the time the current position of the smartphone 600 is separated from the user's gaze position. In this case, the CPU 201 determines that the user's gaze level is lower as the time mentioned above increases, and increases the display transparency of the notification display.

[0068] <Displaying intercom notification information> In this embodiment, the external device is a smartphone 600 installed with an SNS application capable of sending and receiving messages, but the external device is not limited to this and may be any device capable of sending notification information to the HMD 100. In short, the present invention can be applied to any external device that needs to notify the user of a notification, such as an intercom, an IoT-compatible microwave, or a refrigerator. However, if the external device is not configured to acquire information about its current location, the user may input the current location of the external device into the HMD 100, or the HMD 100 may acquire the current location of the external device from a camera image as described above.

[0069] FIG. 13 is a diagram showing an example of intercom notification information displayed on the HMD 100, showing the notification information displayed in step S805 when the intercom buzzer is pressed. FIG. 13(a) is a diagram showing an example of intercom notification information displayed superimposed on VR video or the like on the display screens 104 and 105 of the HMD 100. FIG. 13(b) is a diagram showing an example of the intercom notification information of FIG. 13(a) displayed in detail on the display screens 104 and 105 of the HMD 100. In FIG. 13(a), 1300 is a notification display frame. In the notification display frame 1300, intercom notification information made up of information 1301 to 1303 is displayed. As a result, the HMD 100 displays a notification that the intercom buzzer has been pressed, as simple notification information that is first shown to the user wearing the HMD 100.

[0070] Information 1301 is information showing a message that the intercom buzzer has been pressed. Information 1302 is an icon showing an intercom. Information 1303 is information showing the general name "intercom." Note that button 1304 is displayed within notification display frame 1300. Button 1304 is a button for displaying details of the intercom notification information. The user can operate button 1304 with controllers 111a and 111b or with their own line of sight, in the same way as buttons 1105, 1113 to 1115 described above.

[0071] In FIG. 13(b), 1310 is a notification display frame. In the notification display frame 1310, in addition to the above-mentioned information 1303, intercom notification information composed of information 1311 is displayed. The information 1311 is an image displayed on the intercom. Also displayed in the notification display frame 1310 are operation screens 1312 and 1313. The operation screens 1312 and 1313 are operation screens on which the user can respond to the intercom notification information. In the operation screen 1312, a button 1314 for canceling auto-lock is displayed. In the operation screen 1313, a button 1315 for starting recording on the intercom is displayed. In addition, a button 1316 is displayed in the notification display frame 1310. The button 1316 is a button for closing the notification display frame 1310.

[0072] The user can operate buttons 1314 to 1316 with controllers 111a and 111b or with their own line of sight, similar to the above-described buttons 1105, 1113 to 1115, and 1304. In this way, the user can select an action to take regarding the other party displayed on the intercom by operating buttons 1314 and 1315. Therefore, even while wearing HMD 100, the user can comfortably respond when the intercom buzzer is pressed. For example, if the user's gaze position is outside the notification display frame 1300 or 1310 on the display screen 104 or 105 for a predetermined period of time or more, CPU 201 may cause the intercom to output a message such as "Currently unavailable." In this way, the user can prioritize the experience of VR video and the like on HMD 100 while taking action regarding the other party displayed on the intercom.

[0073] <Other> Although a preferred embodiment of the present invention has been described above, the present invention is not limited to this embodiment, and various modifications and changes are possible within the scope of the gist thereof. For example, in this embodiment, an example has been described in which the position where the user's line of sight intersects with the display screens 104 and 105 when viewing the current position of the smartphone 600 through the display screens 104 and 105 is applied as the specific position in the present invention. In this regard, the specific position in the present invention is not limited to this, and may be a predetermined position with respect to the display screens 104 and 105 (for example, the display position of a specific virtual object, a corner of the screen, etc.).

[0074] Although the present embodiment illustrates a single external device communicating with the HMD 100, multiple external devices may be connected. In this case, for example, multiple external devices, such as a smartphone, an intercom, an IoT-compatible microwave, or a refrigerator, are connected to the HMD 100, and the current locations of the intercom, microwave, and refrigerator are set. This allows a user wearing the HMD 100 to switch the notification information of an external device displayed on the HMD 100 by gazing at the current location of the external device for which the user wants to check notification information. For example, if the external device is a microwave, the notification information displayed on the HMD 100 may include a notification indicating that defrosting is complete. If the external device is a refrigerator, the notification information may include a warning when the door is left open or a notification that ice has been made. In this way, notification display using the user's gaze position on the HMD 100 has a wide range of applications.

[0075] The information processing device of the present invention is not limited to the HMD 100 and may be, for example, smart glasses. The information processing device of the present invention may also be a computer connected to the HMD 100 or smart glasses via a wired or wireless connection. Examples of such computers include a small box computer that can be worn over a user's shoulder, a portable computer such as a notebook PC, a tablet PC, or a smartphone, and a stationary computer such as a desktop PC. However, even in this case, the display screens 104 and 105, the cameras 106 and 107, the gaze detectors 108 and 109, and the optical splitter 208 are included in the HMD 100 or smart glasses.

[0076] The present invention can also be realized by supplying a program that realizes one or more functions of this embodiment to a system or device via a network or storage medium, and having one or more general-purpose processors (e.g., CPUs) in the computer of that system or device read and execute the program.

[0077] The present invention can also be realized by a dedicated processor (e.g., a circuit such as an ASIC or FPGA) that realizes one or more functions. Furthermore, the present invention can also be realized by a combination of a general-purpose processor and a dedicated processor. Note that the term "processor" refers to a processor in a broad sense, and includes general-purpose processors and dedicated processors. Furthermore, the operations of a processor may not only be performed by a single processor, but may also be performed in cooperation with multiple processors located at physically separate locations.

[0078] The disclosure of this embodiment includes the following configurations, methods, and programs. (Configuration 1) A notification acquisition means for acquiring notification information of an external device; a gaze position acquisition means for acquiring a gaze position of a user on a display screen on which an image is being displayed; a first determination means for determining whether or not to display notification information of the external device by comparing a specific position on the display screen with the gaze position; and a display control means for displaying the notification information of the external device on the display screen when the first determination means determines that the notification information of the external device should be displayed. (Configuration 2) A current location acquisition unit for acquiring a current location of the external device is provided, 2. The information processing device according to configuration 1, wherein the specific position is a position where the user's line of sight intersects with the display screen when the user sees the current position of the external device through the display screen. (Configuration 3) The information processing device according to configuration 1 or 2, further comprising a tactile control means for notifying the user of the specific position by haptics. (Configuration 4) The information processing device according to Configuration 3, wherein the haptic control means activates haptics when the notification acquisition means acquires notification information from the external device. (Configuration 5) The information processing device according to Configuration 4, wherein the tactile control means further activates haptics when the specific position and the gaze position are separated from each other. (Configuration 6) The information processing device according to any one of configurations 1 to 5, wherein the display control means displays notification information of the external device at the specific position. (Configuration 7) The information processing device according to any one of configurations 1 to 6, wherein the display control means displays notification information of the external device superimposed on the video. (Configuration 8) An information processing device described in any one of configurations 1 to 7, characterized in that the first judgment means determines to display notification information of the external device when the time during which the specific position and the gaze position coincide is equal to or longer than a first predetermined time. (Configuration 9) The information processing device according to configuration 8, wherein the display control means changes the display transparency of the notification information of the external device according to the amount of deviation, which is the distance between the specific position and the gaze position. (Configuration 10) The information processing device according to configuration 9, wherein the display control means increases the display transparency of the notification information of the external device as the amount of deviation increases. (Configuration 11) The information processing device described in Configuration 8, characterized in that the display control means changes the display transparency of the notification information of the external device depending on the number of times the specific position and the gaze position coincide again within a certain period of time. (Configuration 12) The information processing device described in Configuration 11, characterized in that the display control means increases the display transparency of the notification information of the external device the fewer the number of times the specific position and the gaze position coincide again within a certain period of time. (Configuration 13) The information processing device according to configuration 8, wherein the display control means changes the display transparency of the notification information of the external device depending on the time period during which the specific position is separated from the gaze position. (Configuration 14) The information processing device described in Configuration 13, wherein the display control means increases the display transparency of the notification information of the external device the longer the distance between the specific position and the gaze position. (Configuration 15) An information processing device described in any one of configurations 8 to 14, characterized in that the display control means displays an operation screen on the display screen for the user to respond to notification information from the external device when the time during which the specific position and the gaze position coincide becomes equal to or longer than a second predetermined time that is longer than the first predetermined time. (Configuration 16) The information processing device according to configuration 15, wherein the display control means displays the operation screen at the specific position. (Configuration 17) The information processing device according to configuration 15 or 16, wherein the display control means displays the operation screen superimposed on the video. (Configuration 18) A second determination means is provided to compare the specific position with the gaze position and determine whether to terminate display of the notification information of the external device, The information processing device described in any one of configurations 1 to 17, characterized in that the display control means terminates the display of the notification information of the external device when the second determination means determines that the display of the notification information of the external device should be terminated. (Configuration 19) The information processing device described in Configuration 18, characterized in that the second judgment means determines to terminate display of notification information of the external device when the time the specific position and the gaze position are separated from each other becomes equal to or longer than a third predetermined time. (Configuration 20) The information processing device according to any one of configurations 1 to 19, wherein the information processing device is a head-mounted display having the display screen. (Configuration 21) The information processing device according to any one of configurations 1 to 20, wherein the video is a VR video. (Configuration 22) A notification acquisition means for acquiring notification information of an external device; a detection means for detecting a user's line of sight; a direction determining means for determining whether or not to display notification information of the external device by comparing the line of sight direction with a specific direction; and display control means for displaying the notification information of the external device on a display screen when the direction determination means determines that the notification information of the external device should be displayed. (Method 1) A notification acquisition step of acquiring notification information of an external device; a gaze position acquisition step of acquiring a gaze position of a user with respect to a display screen on which an image is being displayed; a determination step of determining whether or not to display notification information of the external device by comparing the specific position on the display screen with the gaze position; a display control step of displaying the notification information of the external device on the display screen when the determination step determines that the notification information of the external device should be displayed. (Program 1) A program for causing a computer to execute each means of the information processing device according to any one of configurations 1 to 21. (System 1) An information processing system having an information processing device and an external device, The external device is a transmitting means for transmitting notification information of the external device; The information processing device includes: a receiving means for receiving notification information from the external device; a gaze position acquisition means for acquiring a gaze position of a user on a display screen on which an image is being displayed; a determination means for determining whether or not to display notification information of the external device by comparing the specific position on the display screen with the gaze position; and a display control means for displaying the notification information of the external device on the display screen when the determination means determines that the notification information of the external device should be displayed. (System 2) The information processing system according to System 1, wherein the information processing device is a head-mounted display having the display screen. (System 3) The information processing system according to System 1 or 2, wherein the external device is a mobile terminal. (System 4) The information processing system according to System 1 or 2, wherein the external device is an IoT device. [Explanation of symbols]

[0079] 100 HMD (information processing device) 104, 105 Display screen 201 CPU (notification acquisition means) (gaze position acquisition means) (first determination means) (display control means) 600 Smartphone (external device)

Claims

1. a notification acquisition means for acquiring notification information of an external device; a gaze position acquisition means for acquiring a gaze position of a user on a display screen on which an image is being displayed; a first determination means for determining whether or not to display notification information of the external device by comparing a specific position on the display screen with the gaze position; and a display control means for displaying the notification information of the external device on the display screen when the first determination means determines that the notification information of the external device should be displayed.

2. a current location acquisition means for acquiring a current location of the external device; 2. The information processing apparatus according to claim 1, wherein the specific position is a position where a user's line of sight intersects with the display screen when the user sees the current position of the external device through the display screen.

3. 2. The information processing apparatus according to claim 1, further comprising a tactile control means for notifying the user of the specific position by haptics.

4. 4. The information processing apparatus according to claim 3, wherein the haptic control means activates a haptic when the notification acquisition means acquires the notification information of the external device.

5. 5. The information processing apparatus according to claim 4, wherein the tactile control means further activates a haptic when the specific position and the gaze position are separated from each other.

6. 2. The information processing apparatus according to claim 1, wherein the display control means displays notification information of the external device at the specific position.

7. 2. The information processing apparatus according to claim 1, wherein the display control means displays notification information of the external device superimposed on the video.

8. 2. The information processing device according to claim 1, wherein the first determination means determines that notification information of the external device is to be displayed when the time during which the specific position and the gaze position coincide is equal to or longer than a first predetermined time.

9. 9. The information processing apparatus according to claim 8, wherein the display control means changes a display transparency of the notification information of the external device in accordance with a displacement amount that is a distance between the specific position and the gaze position.

10. 10. The information processing apparatus according to claim 9, wherein the display control means increases the display transparency of the notification information of the external device as the amount of deviation increases.

11. 9. The information processing device according to claim 8, wherein the display control means changes a display transparency of the notification information of the external device depending on the number of times the specific position and the gaze position coincide again within a certain period of time.

12. 12. The information processing device according to claim 11, wherein the display control means increases the display transparency of the notification information of the external device as the number of times the specific position and the gaze position coincide again within a certain period of time decreases.

13. 9. The information processing apparatus according to claim 8, wherein the display control means changes a display transparency of the notification information of the external device depending on a time period during which the specific position is separated from the gaze position.

14. 14. The information processing apparatus according to claim 13, wherein the display control means increases the display transparency of the notification information of the external device as the time period during which the specific position is separated from the gaze position increases.

15. The information processing device according to claim 8, characterized in that the display control means displays an operation screen on the display screen for the user to respond to notification information from the external device when the time during which the specific position and the gaze position coincide becomes equal to or longer than a second predetermined time that is longer than the first predetermined time.

16. 16. The information processing apparatus according to claim 15, wherein the display control means displays the operation screen at the specific position.

17. 16. The information processing apparatus according to claim 15, wherein the display control means displays the operation screen superimposed on the video.

18. a second determination means for determining whether to end display of the notification information of the external device by comparing the specific position with the gaze position; 2 . The information processing apparatus according to claim 1 , wherein the display control means terminates the display of the notification information of the external device when the second determination means determines that the display of the notification information of the external device should be terminated.

19. 19. The information processing device according to claim 18, wherein the second determination means determines to end display of the notification information of the external device when the time during which the specific position and the gaze position are separated from each other is equal to or longer than a third predetermined time.

20. The information processing device according to claim 1 , wherein the information processing device is a head-mounted display having the display screen.

21. The information processing apparatus according to claim 1 , wherein the video is a VR video.

22. a notification acquisition means for acquiring notification information of an external device; a detection means for detecting a user's line of sight; a direction determining means for determining whether or not to display notification information of the external device by comparing the line of sight direction with a specific direction; and display control means for displaying the notification information of the external device on a display screen when the direction determination means determines that the notification information of the external device should be displayed.

23. a notification acquisition step of acquiring notification information of an external device; a gaze position acquisition step of acquiring a gaze position of a user with respect to a display screen on which an image is being displayed; a determination step of determining whether or not to display notification information of the external device by comparing the specific position on the display screen with the gaze position; a display control step of displaying the notification information of the external device on the display screen when the determination step determines that the notification information of the external device should be displayed.

24. 2. A program for causing a computer to execute each means of the information processing apparatus according to claim 1.

Images