Eye-tracking device, display device, control method, and program

Abstract

To provide a visual line detection device with which, in selecting a GUI using a visual line of a display device including an xR device, it is possible to adjust an influence degree of a visual line shift that corresponds to a distance of the GUI so as to reduce an erroneous selection or a misdetection due to the visual line and improve operation ability.SOLUTION: A visual line detection device 100a is capable of selecting a GUI on a display element on the basis of a visual line position of a user. The visual line detection device includes: GUI control means 113 capable of synthesizing a GUI to a discretionary point in a real space coordinate system on the display element (a display 106 for the left eye and a display 107 for the right eye of a HMD 100); visual line detection means 112 for detecting the visual line position of the user on the display element; and visual line processing means 115 for adjusting an influence degree of a change in the visual line position of the user headed to the GUI, in accordance with a distance of the GUI to the user.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a gaze detection device, and more particularly to a gaze detection device for selecting a GUI of an xR device using a gaze.

Background Art

[0002] There is a technology for estimating / detecting a gaze direction and controlling a fixation target.

[0003] One of the applications of gaze in an xR device (a general term for virtual reality (VR), augmented reality (AR), mixed reality (MR), substitute reality (SR), etc.) is the selection of a graphical user interface (GUI) such as a button or an icon using a gaze.

[0004] For example, Patent Document 1 describes a technology in which a GUI is selected when a user continuously views the GUI for a certain period of time. Further, Patent Document 2 describes a technology in which when a GUI is gazed at, the size of the determination area is changed to be enlarged.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in an xR device, when a user selects a GUI having a certain distance from the user using a gaze, there are the following problems.

[0007] In Patent Document 1, when selecting an item by continuously viewing a GUI for a certain period of time, the further away the GUI is and the smaller its size, the greater the relative impact of changes in the line of sight (eye movement). If the line of sight shifts and the user's gaze unintentionally moves away from the GUI, the selection is reset, and the time elapsed by the gaze must be restarted from the beginning.

[0008] Furthermore, as described in Patent Document 2, if the size of the detection area used to determine whether the GUI is being viewed is larger than the size of the GUI, and the GUI is close and displayed at a large size, it can lead to unintended selection of the GUI, making false detections likely.

[0009] To solve the above problem, adjustments are needed for gaze detection based on the distance of the GUI.

[0010] The present invention provides a gaze detection device that can reduce selection errors and false detections due to gaze input and improve operability in GUI selection using gaze input in a display device including an xR device, by adjusting the degree of influence of gaze shift according to the distance of the GUI. [Means for solving the problem]

[0011] One aspect of the present invention is a gaze detection device capable of selecting a GUI on a display element based on the user's gaze position, comprising: a synthesis means capable of compositing the GUI at any location in a real-space coordinate system on the display element; and a gaze detection means for detecting the user's gaze position on the display element. Based on the user's position coordinates and the GUI's position coordinates, which are set in the aforementioned real-space coordinate system, The system includes gaze processing means that adjusts the degree of influence of changes in the user's gaze position directed towards the GUI, according to the distance of the GUI to the user.

[0012] Another aspect of the present invention is a display device comprising a display element that displays an image toward the user and the gaze detection device described above.

[0013] Another aspect of this embodiment is a control method for a gaze detection device that can select a GUI on a display element based on the user's gaze position, comprising the steps of: compositing the GUI on the display element at an arbitrary location in a real-space coordinate system; and detecting the user's gaze position on the display element. Based on the user's position coordinates and the GUI's position coordinates, which are set in the aforementioned real-space coordinate system, The process includes adjusting the degree of influence of changes in the user's line of sight directed towards the GUI, according to the distance between the user and the GUI.

[0014] Another aspect of the present invention is a program that causes a computer to execute the above control method.

[0015] Other objects and features of the present invention are described in the following embodiments. [Effects of the Invention]

[0016] According to the present invention, in the selection of a GUI using gaze in a display device including an xR device, by adjusting the degree of influence of gaze misalignment according to the distance to the GUI, it is possible to reduce selection errors and false detections due to gaze and improve operability. [Brief explanation of the drawing]

[0017] [Figure 1] This is a schematic diagram showing the configuration of the display device according to the first embodiment. [Figure 2] This is a schematic diagram showing the configuration of the eye-tracking detection mechanism according to the first embodiment. [Figure 3] This figure illustrates the principle of the gaze detection method according to the first embodiment. [Figure 4] This is a schematic diagram of the eyeball image projected onto the ophthalmic image sensor and a diagram of the output intensity at the ophthalmic image sensor. [Figure 5] This is a flowchart illustrating the general process of gaze detection. [Figure 6] This flowchart outlines the process of selecting GUI elements using eye movements. [Figure 7] This diagram illustrates the GUI panel according to the first embodiment. [Figure 8] It is a diagram for explaining GUI selection by line of sight according to the first embodiment. [Figure 9] It is a diagram for explaining GUI selection by line of sight according to the second embodiment. [Figure 10] It is a diagram for explaining GUI selection by line of sight according to the third embodiment. **Modes for Carrying Out the Invention**

[0018] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and redundant descriptions are omitted. (First Embodiment) Hereinafter, a display device having a line-of-sight detection device according to the first embodiment will be described with reference to FIGS. 1 to 8. In this embodiment, a head-mounted display (HMD) will be used as the display device for explanation. **Explanation of Configuration** FIG. 1 shows the configuration of the HMD 100 according to this embodiment. The left configuration diagram in FIG. 1 shows the HMD 100 as viewed from the top of the user's head. The right block diagram in FIG. 1 shows the functional configuration of the line-of-sight detection device 100a of this embodiment included in the HMD 100, that is, the functional configuration for GUI selection using the line of sight. The line-of-sight detection device 100a is a device capable of selecting a GUI on the display element based on the line-of-sight position of the user. When the housing 103 of the HMD 100 is worn on the user's head, the left eyeball 101 and the right eyeball 102 observe the images of the left-eye display 106 and the right-eye display 107 through the left-eye eyepiece lens 104 and the right-eye eyepiece lens 105, respectively. The left-eye display 106 and the right-eye display 107 are display elements that display images toward the user, respectively.

[0019] When the HMD100 is worn in a non-transparent manner, the displays 106 and 107 may display images stored internally (such as recorded videos or game footage). When the HMD100 is worn in a transparent manner, real-time images of the real space captured by the left-eye camera 108 and the right-eye camera 109 may be displayed on the displays 106 and 107. Generally, the former is a so-called VR display method, and the latter is a display method called a video see-through type. In each display method, it is possible to expand spatial information by appropriately superimposing digital information such as CG on the displays 106 and 107.

[0020] In addition to the video see-through type, a system that views the real world through a lens that also serves as the display surface, and then superimposes digital information onto it using optical systems such as prisms or half-mirrors, is called the optical see-through type. Similarly, a system that uses a non-light-blocking transmissive display as the lens, views the real world through the display, and then superimposes digital information onto it, is also considered an optical see-through type. The gaze detection configuration and flow described in this embodiment are also applied to the optical see-through type.

[0021] Generally, AR and MR are considered technologies that span from optical see-through to video see-through. This embodiment is applicable to both non-transparent and transmissive (video see-through and optical see-through) technologies.

[0022] The location on displays 106 and 107 where the user is fixating is detected (estimated) using the left-eye gaze detector 110 and the right-eye gaze detector 111, as described later. The information from the gaze detectors 110 and 111 is sent to the gaze detection means 112 shown in the block diagram, and the point of fixation is detected (estimated) using the information from both eyes or one eye.

[0023] GUI elements such as buttons and icons are appropriately combined and superimposed onto displays 106 and 107 by a GUI control means (combination means) 113 capable of combining GUIs. By selecting a GUI, the user can proceed with processing according to the selected command or function.

[0024] In GUI selection by gaze, the distance to the GUI from the user is first obtained by the GUI distance acquisition means (distance acquisition means) 114. Depending on the distance, the gaze processing means 115, which consists of gaze adjustment means 115a and gaze determination means 115b, changes the processing pattern for gaze detection as predetermined.

[0025] The GUI selection means 116 compares the position information of the GUI positioned by the GUI control means 113 with the gaze information adjusted by the gaze adjustment means 115a or the gaze determination means 115b, and determines whether the GUI was selected by gaze.

[0026] Further details of the processing in each block diagram will be described later using the processing flow diagram in Figure 6.

[0027] Figure 2 is a cross-sectional view of the HMD100 cut along the YZ plane formed by the Y and Z axes shown in Figure 1, illustrating the mechanism for eye-line detection. Note that Figure 2 is a cross-sectional view of the HMD100 as seen from the left eye side, and is a schematic diagram of the mechanism on the left eye side, but the mechanism on the right eye side can be assumed to be similar.

[0028] In Figure 2, 103 represents the housing of the HMD100, and the configuration of the units contained within it is as follows.

[0029] The gaze detection device 100a included in the HMD 100 has a CPU 128 that controls the entire HMD 100 and a memory unit 129 that records video information. The CPU 128 executes a program stored in the memory unit 129 to realize the various functional configurations of the gaze detection device 100a described in Figure 1 and to realize the various processes in the flowchart described later. This program is a program that makes the computer operate or function as the gaze detection device 100a. The memory unit 129 includes a non-volatile storage medium for storing various information and a volatile storage medium such as RAM. The above program is stored in the non-volatile storage medium. The non-volatile storage medium stores information that is temporarily necessary when the CPU 128 is operating.

[0030] The HMD 100 also includes a display 106 made of liquid crystal or the like for displaying images, a display drive circuit 124 for driving the display 106, and an eyepiece 104 for observing the image displayed on the display 106. Furthermore, as a mechanism for a camera 108 that captures images of the outside when the HMD 100 is worn in a see-through video type, the HMD 100 includes a camera image sensor 125, an aperture mechanism 126, and a focus mechanism 127. The imaging information from the camera image sensor 125 may be replaced with the internal image as needed, or the imaging information and the internal image may be combined and displayed on the display 106. In addition, a GUI may be superimposed on the display 106 by a GUI control means 113 (not shown).

[0031] The illumination light source 120 is a light source projected onto the eyeball 101 for gaze detection, and consists of, for example, multiple infrared light-emitting diodes arranged around the eyepiece lens 104. The illuminated eyeball image and the image from the corneal reflection of the illumination light source 120 pass through the eyepiece lens 104, are reflected by the light splitter 121, and are imaged onto the eyeball image sensor 123, which has a two-dimensional arrangement of photoelectric elements such as CMOS, by the light-receiving lens 122. The light-receiving lens 122 positions the pupil of the user's eyeball 101 and the eyeball image sensor 123 in a complementary imaging relationship. The gaze direction is detected by a predetermined algorithm, described later, from the positional relationship between the eyeball image formed on the eyeball image sensor 123 and the image from the corneal reflection of the illumination light source 120. The illumination light source 120, light splitter 121, light-receiving lens 122, and eyeball image sensor 123 constitute the gaze detector 110.

[0032] The memory unit 129 also has a function for storing imaging signals from the camera image sensor 125 and the eyeball image sensor 123, as well as a function for storing gaze correction data.

[0033] Figure 3 is a diagram illustrating the principle of the gaze detection method and corresponds to a summary diagram of the optical system for gaze detection shown in Figure 2. In Figure 3, 120a and 120b are illumination light sources such as light-emitting diodes that emit infrared light insensitive to the user. Each light source 120a and 120b is arranged approximately symmetrically with respect to the optical axis of the light-receiving lens 122 and illuminates the user's eyeball 101. A portion of the illumination light reflected by the eyeball 101 is focused by the light-receiving lens 122 onto the image sensor 123 for the eyeball.

[0034] Figure 4(a) is a schematic diagram of the eyeball image projected onto the eyeball image sensor 123. Figure 4(b) is an output intensity diagram of the eyeball image sensor 123. Figure 5 shows a flowchart illustrating the general flow of gaze detection.

[0035] The following explanation of gaze detection operation will be given using Figures 3 to 5. Note that while the explanation focuses on gaze detection for the left eye, the process is similar for the right eye. <Explanation of eye-tracking operation> In Figure 5, when the gaze detection routine starts, in step S501, the light sources 120a and 120b emit infrared light toward the user's eyeball 101. The image of the user's eyeball, illuminated by the infrared light, is formed on the eyeball image sensor 123 through the light-receiving lens 122 and converted into an electrical signal by the eyeball image sensor 123. As a result, the eyeball image becomes processable as an electrical signal.

[0036] In step S502, the eyeball image signal obtained from the eyeball image sensor 123 is transmitted to the CPU 128.

[0037] In step S503, the CPU 128 obtains the coordinates of the corneal reflection images Pd and Pe from the light sources 120a and 120b shown in Figure 3, and the coordinates of the point corresponding to the pupil center c (pupil center position) from the eyeball image signal information obtained in S502. The infrared light emitted from the light sources 120a and 120b illuminates the cornea 142 of the user's eyeball 101. At this time, the corneal reflection images Pd and Pe, formed by a portion of the infrared light reflected from the surface of the cornea 142, are focused by the light-receiving lens 122 and imaged onto the eyeball image sensor 123 (points Pd' and Pe' in Figure 3). Similarly, the light beams from the pupillary ends a and b, which are the ends of the pupil 141, are also imaged onto the eyeball image sensor 123.

[0038] Figure 4(a) shows an example of a reflected image obtained from the ophthalmic image sensor 123. Figure 4(b) shows an example of luminance information obtained from the ophthalmic image sensor 123 in region α of the above image example. As shown in Figure 4(a), the horizontal direction is the X-axis and the vertical direction is the Y-axis. In this case, the coordinates in the X-axis direction (horizontal direction) of the images Pd' and Pe' formed by the corneal reflected images Pd and Pe from the light sources 120a and 120b are denoted as Xd and Xe. Also, the coordinates in the X-axis direction of the images a' and b' formed by the light beams from the pupillary apex a and b, which are the ends of the pupil 141, are denoted as Xa and Xb. In the example of luminance information in Figure 4(b), an extremely high level of luminance is obtained at positions Xd and Xe, which correspond to the images Pd' and Pe' formed by the corneal reflected images Pd and Pe from the light sources 120a and 120b. In the region corresponding to the pupil 141 between coordinates Xa and Xb, an extremely low level of brightness is obtained, except for the positions Xd and Xe. In contrast, in the region corresponding to the iris 143 outside the pupil 141, where the X coordinate value is smaller than Xa and the X coordinate value is larger than Xb, an intermediate value between the two types of brightness levels is obtained. From the brightness level variation information with respect to the above X coordinate positions, the X coordinates Xd and Xe of the images Pd' and Pe' formed by the corneal reflection images Pd and Pe of the light sources 120a and 120b, and the X coordinates Xa and Xb of the images a' and b' of the pupil ends a and b can be obtained. Furthermore, when the rotation angle θx of the eyeball 101 with respect to the optical axis of the light-receiving lens 122 is small, the coordinate Xc of the location corresponding to the pupil center c (let's call it c') that is imaged on the eyeball image sensor 123 can be expressed as Xc ≈ (Xa + Xb) / 2. Based on the above, the X-coordinate Xc of the point c' corresponding to the pupil center c that is imaged on the ocular image sensor 123, and the coordinates Xd and Xe of the corneal reflection images Pd' and Pe' of the light sources 120a and 120b can be estimated.

[0039] Furthermore, in step S504, the CPU 128 obtains (calculates) the imaging magnification β of the eyeball image. The imaging magnification β is a magnification determined by the position of the eyeball 101 relative to the light-receiving lens 122, and can be essentially obtained as a function of the interval (Xd-Xe) between the corneal reflection images Pd' and Pe'.

[0040] Furthermore, in step S505, the CPU 128 obtains (calculates) the rotation angle θx of the eyeball 101 in the ZX plane relative to the optical axis and the rotation angle θy of the eyeball 101 in the ZY plane relative to the optical axis. Since the X coordinate of the midpoint of the corneal reflection images Pd' and Pe' and the X coordinate of the curvature center O of the cornea 142 are approximately the same, if we let Oc be the standard distance between the curvature center O of the cornea 142 and the center c of the pupil 141, then the rotation angle θx of the optical axis of the eyeball 101 in the ZX plane is, β×Oc×sinθx≒{(Xd+Xe) / 2}-Xc This can be determined from the given relationship. Furthermore, while Figures 3 and 4 show examples of calculating the rotation angle θx when the user's eyeball 101 rotates in a plane perpendicular to the Y-axis, the method for calculating the rotation angle θy when the user's eyeball rotates in a plane perpendicular to the X-axis is similar.

[0041] In step 505, when the rotation angles (eyeball rotation angles) θx and θy of the user's eyeball 101 with respect to the optical axis are calculated, in step S506, the CPU 128 uses the rotation angles θx and θy to obtain (calculate) the user's gaze position on the display 106. Assuming that the gaze position is the gaze point coordinate (Hx, Hy) corresponding to the center c of the pupil 141 on the display 106, Hx and Hy can be calculated using the following formulas.

[0042] Hx = m × (Ax × θx + Bx) Hy = m × (Ay × θy + By) In the above equation, the coefficient m is a constant determined by the optical system configuration and is a conversion coefficient that converts the rotation angles θx and θy into gaze point coordinates (Hx, Hy) corresponding to the center c of the pupil 141 on the display 106. The coefficient m is predetermined and stored in the memory unit 129. The other coefficients Ax, Bx, Ay, and By are gaze correction coefficients that compensate for individual differences in the user's gaze and are obtained by performing a calibration operation. These coefficients Ax, Bx, Ay, and By are also stored in the memory unit 129 in advance before the gaze detection routine starts. The CPU 128 reads these coefficients stored in the memory unit 129 and obtains the user's gaze point coordinates (Hx, Hy) using the above equation.

[0043] After calculating the gaze point coordinates (Hx, Hy) corresponding to the center c of the pupil 141 on the display 106 as described above, in step S507, the CPU 128 stores the gaze point coordinates (Hx, Hy) in the memory unit 129 and completes the gaze detection routine.

[0044] In this embodiment, a method for acquiring the gaze point coordinates (Hx, Hy) on the display 106 using the corneal reflection images Pd, Pe of light sources 120a, 120b was described. However, the method for acquiring the gaze point coordinates (Hx, Hy) is not limited to this; any method that acquires the eye rotation angles θx, θy from the captured eyeball image can acquire the gaze point coordinates (Hx, Hy). Furthermore, it is possible to acquire the gaze point coordinates (Hx, Hy) even with techniques that do not use eyeball images, such as methods that detect (estimate) the line of sight from the detection of electrooculography. <Explanation of GUI selection based on eye gaze> Next, the gaze-based GUI selection flow in this embodiment will be explained using Figure 6. Note that the order of each step in Figure 6 may be partially changed, some steps may be repeated, some steps may be executed simultaneously, or other modifications may be made as appropriate according to the specifications.

[0045] In Figure 6, when the control routine starts, in step S601, the GUI control means 113 overlays and displays the GUI on the displays 106 and 107 at any location in the real-space coordinate system. The GUI consists of buttons, icons, or panels containing them. When the GUI is selected, various predetermined processes assigned to the GUI are executed.

[0046] These GUIs may be displayed in conjunction with an object in a virtual or real space. For example, as shown in Figure 7, detailed information about an object may be displayed as a panel nearby, with GUIs positioned for scrolling, sliding, and selecting information within the panel. Additionally, a separate GUI resembling a keyboard may be placed for writing information into the panel, and another GUI for closing the panel. These GUIs can be freely arranged according to various objects.

[0047] Furthermore, even if they are not tied to an object, execution buttons for media players and other applications placed by the user at any spatial location are also considered GUI elements. Alternatively, icons placed in a specific area on displays 106 and 107, such as the computer's taskbar, menu bar, or application launcher, are also considered GUI elements.

[0048] The GUI may be superimposed on both displays 106 and 107 for the right and left eyes, or on only one of the displays. When the GUI is superimposed on both displays 106 and 107, a sense of depth may be given by showing the images to the left and right eyes shifted by the amount of parallax, or the GUI may be superimposed without parallax.

[0049] A GUI placed in a real-space coordinate system is positioned at least a certain distance from the user. The user can change the distance between the user and the GUI by moving and changing their relative spatial position to the GUI. If the distance is close, the GUI will appear larger relative to the user, and conversely, if the distance is far, the GUI will appear smaller.

[0050] Next, in step S602, the gaze detection means 112 acquires gaze information and calculates (detects) the gaze position. In this step, gaze detection of the left eye and right eye is observed. Gaze detection is performed using the gaze detectors 110 and 111 in the manner described above, and the observed measurement data is analyzed by the gaze detection means 112.

[0051] The gaze detection means 112 observes the gaze positions of the left and right eyes as time-series data. The gaze position may be calculated using gaze information from only one eye, or using gaze information from both eyes.

[0052] Most people have a dominant eye, and the distance between the gaze position of the dominant eye and the object tends to be shorter than the distance between the gaze position of the non-dominant eye and the object. Also, the amount of time-series deviation of the gaze tends to be smaller for the dominant eye than for the non-dominant eye. For this reason, it is preferable to use the gaze information of the dominant eye to calculate the gaze position. Alternatively, the location on the display that the dominant eye is looking at can be detected and this information can be used to control the display on the non-dominant eye side. Alternatively, the gaze position can be detected (estimated) using the gaze information of both eyes, with greater weight given to the dominant eye. Furthermore, if there are means to correct the deviation of the gazes of the dominant and non-dominant eyes, the location on displays 106 and 107 that each eye is looking at can be detected (estimated) using the respective gaze information.

[0053] When the HMD100 is worn in a non-transparent manner, the gaze detection means 112 detects where the user is looking within the internal images and superimposed CG displayed on the displays 106 and 107. When the HMD100 is worn in a transparent manner, the gaze detection means 112 detects where the user is looking within the camera images and superimposed CG displayed on the displays 106 and 107.

[0054] In step S603, the GUI distance acquisition means 114 acquires the distance from the user to the GUI that is superimposed on an arbitrary location in the real-space coordinate system on the displays 106 and 107.

[0055] In typical computer graphics (CG), real-space coordinate systems such as Cartesian, polar, and cylindrical coordinate systems are used to understand the position and orientation of objects in virtual or real space. In addition to objects, real-space coordinates are also assigned to the user relative to the object, making it possible to calculate the distance between them in virtual or real space from their coordinates. This distance information changes as the user or object moves, altering their relative spatial positions. If the distance is close, the object will appear larger relative to the user; conversely, if the distance is far, the object will appear smaller.

[0056] To obtain the distance of a GUI to a user, if the GUI has a coordinate system set in real space, whether in virtual or real space, then the distance information can be obtained. In other words, the distance of a GUI to a user can be obtained based on the user's position coordinates and the GUI's position coordinates, which are set in real space coordinate systems. Furthermore, since distance and the size of the GUI are correlated, the distance of the GUI to the user can also be obtained using the GUI's size.

[0057] In step S604, the gaze adjustment means 115a adjusts the gaze detection process for the GUI according to the distance of the GUI to the user.

[0058] Furthermore, in step S605, the gaze determination means 115b adjusts the gaze determination process for the determination area to determine whether the gaze is fixed on the GUI, according to the distance of the GUI to the user.

[0059] In other words, in steps 604 and 605, the gaze processing means 115 (gaze adjustment means 115a, gaze determination means 115b) adjusts the degree of influence of changes in the user's gaze position directed towards the GUI, according to the distance of the GUI to the user. This degree of influence is the degree of influence when the gaze detection means 112 detects the gaze position.

[0060] Furthermore, in step S606, the GUI selection means 116 determines whether the GUI has been selected based on the gaze. This determination is made, for example, whether the gaze has been continuously looking at the GUI for a certain period of time or longer.

[0061] In this embodiment, the processing of the gaze adjustment means 115a and the GUI selection means 116 will be explained with reference to Figures 7 and 8. Details of the processing of the gaze determination means 115b will be described later in the third embodiment, but the gaze adjustment means 115a and the gaze determination means 115b may be combined in any way. Therefore, in the following description, the processing of the gaze adjustment means 115a and the processing of the gaze determination means 115b will not be distinguished and will be described as the processing of the gaze processing means 115.

[0062] In Figure 7, the user is attempting to manually manipulate an object. On displays 106 and 107, a manual for manipulating the object is displayed on a GUI-equipped panel superimposed near the object. The panel contains a GUI used for advancing (forward and backward) through the operation, and the GUI is selected when the user continuously views it for a certain period of time.

[0063] Furthermore, in this embodiment, the determination area for determining whether the gaze is fixed on the GUI is applied using the size or frame of the GUI itself.

[0064] Figure 7(a) shows the display when the GUI is far from the user. Figure 7(b) shows the display when the GUI is close to the user.

[0065] Figure 8(a) illustrates the effect of gaze misalignment on the GUI. Gazes 1-5 represent time-series data of gaze. For example, the GUI selection means 116 normally determines the selection of the GUI based on gaze if the time-series data of gaze has been continuously observed for five consecutive times. If the gaze moves away from the GUI while it is being observed, the GUI selection means 116 determines that gaze misalignment has occurred, and the continuity of GUI selection based on gaze is reset.

[0066] The size of the GUI changes depending on the distance to the user, but the greater the distance, the greater the relative impact of changes in the user's line of sight (eye-tracking shift).

[0067] Eye-gaze misalignment in GUI selection can be attributed to various factors, including those related to eye characteristics such as fixation tremors, those related to user concentration levels such as unintentional glances, those related to user physical behavior such as unintentional head movements, and those related to eye-gaze detection accuracy such as calibration errors.

[0068] For example, it is known that the human eyeball can momentarily change its gaze position due to so-called fixation tremors. As shown in Figure 8(a), even if a user is intentionally focusing on the GUI, their gaze may shift away from the GUI due to the influence of eye characteristics such as fixation tremors. Also, if there is another moving object on displays 106 and 107, and the user's attention is diverted by that object, it may lead to a flickering behavior where the gaze shifts significantly away and then returns, as shown in Figure 8(b). Furthermore, even while focusing on the GUI, head movements can inevitably lead to gaze misalignment as shown in Figures 8(a) and 8(b). Even with a device configuration that has very high gaze detection accuracy, the further the distance of the GUI from the user, the greater the impact of gaze misalignment caused by these human characteristics.

[0069] Furthermore, low eye-tracking accuracy itself can also lead to eye-tracking misalignment. As shown in Figure 8(c), if the eye-tracking position is shifted by an offset due to calibration errors, the greater the distance from the GUI to the user, the greater the impact of these errors on eye-tracking misalignment.

[0070] Conversely, when a user is looking at something other than the GUI, but their gaze unintentionally drifts to the GUI, this is also a type of eye-tracking misalignment for the user. In this case, it is necessary to prevent the GUI from being accidentally selected. Unintentional GUI selection is more likely to occur the closer the GUI is to the user, that is, the larger the GUI is.

[0071] The gaze processing means 115 reduces the effects of gaze misalignment when the GUI is far from the user, enabling stable selection of the GUI. Furthermore, when the GUI is close to the user, the gaze processing means 115 adjusts gaze detection to prevent unintended GUI selection.

[0072] In this embodiment, the gaze processing means 115 adjusts the tolerance for gaze misalignment detection according to the distance of the GUI to the user. That is, when the distance of the GUI to the user is far, the gaze processing means 115 sets the tolerance for gaze misalignment to be larger, allowing the continuity of GUI selection by gaze within a certain range even if gaze misalignment occurs. The tolerance may be adjusted as appropriate by considering the expected gaze misalignment situation, such as the magnitude of the gaze misalignment relative to the size of the GUI, the time the gaze is away from the GUI, and the number of times the gaze is away from the GUI.

[0073] For example, in the case of small gaze shifts caused by fixation tremors as seen in Figure 8(a), if the deviation of the gaze time-series data from the GUI is singular and not continuous, the GUI selection based on gaze may be continued. The number of consecutive deviations and the number of times the gaze time-series data deviates from the GUI due to gaze shifts can be adjusted as appropriate.

[0074] Furthermore, even in cases of relatively large gaze shifts, such as those seen in Figure 8(b), the tolerance can be adjusted by referring to the time-series data of the gaze. For example, if the gaze shifts significantly (gaze 3) and then returns to the GUI (gaze 4, gaze 5), it is acceptable to continue GUI selection by gaze.

[0075] Furthermore, in the case of gaze misalignment caused by gaze detection errors as seen in Figure 8(c), the error may be readjusted to adjust the tolerance for gaze misalignment. For example, if it is determined from the time-series gaze data that the user is fixated on something, and a GUI exists nearby, it may be determined that the user is trying to select the GUI, and the gaze misalignment error may be readjusted. Moreover, this readjustment is only applied to the GUI that the user is trying to select; if the user looks away from the GUI or releases their gaze, the calibration may be returned to its state before readjustment.

[0076] On the other hand, the gaze processing means 115 is set to reduce the tolerance for gaze misalignment when the GUI is close to the user. When the GUI is close to the user, the effect of gaze misalignment is relatively small, allowing the user to focus on the GUI more accurately. In such a situation where the user can focus accurately, it is more natural to assume that the user intentionally looks away rather than that the gaze is unintentionally misaligned. Setting a tolerance for gaze misalignment as in the case when the GUI is far from the user would lead to false detection of GUI selections based on gaze, so adjustments are necessary for when the GUI is close to the user.

[0077] Furthermore, the tolerance for distance between the GUI and the user and the detection of gaze misalignment can be arbitrarily determined, including linear and nonlinear relationships. Multiple thresholds, including upper and lower limits, can be used, and the adjustment can be changed within or outside the range of these thresholds. The adjustment can also be changed based on user-specific gaze misalignment information acquired in advance through calibration. These adjustments may only be activated when the gaze and the GUI are closer than an arbitrarily set distance range, that is, when it is determined that the user will select the GUI.

[0078] The GUI selection means 116 may have a mechanism to notify the user when it determines that a GUI has been selected. For example, the display of GUIs that are not selected and the display of GUIs that are selected may be changed. The display changes may be appropriately selected, for example, by changing the color, brightness, size, shape, depth perception, or position. The GUI selection means 116 may also have a mechanism to notify the user of this by adding labels or objects to the vicinity of the GUI or to another area.

[0079] Furthermore, the GUI selection means 116 may have a mechanism for displaying objects such as frames that represent the determination area. The GUI selection means 116 may also have a mechanism for displaying the real-time gaze position on the display using a pointer or the like. In displaying the gaze position, the smoothed position may be displayed, as described later in the second embodiment.

[0080] By displaying the user's gaze position and the display during GUI selection, users can verify how their gaze affects GUI selection.

[0081] Furthermore, the processing performed by the gaze processing means 115 and the GUI selection means 116 may be carried out individually or in appropriate combinations, as described in the second and third embodiments described later, in addition to the implementation in this embodiment.

[0082] As described in this embodiment, by adjusting the tolerance for eye-tracking error detection according to the distance of the GUI to the user, it is possible to reduce GUI selection errors and false detections due to eye movements and improve the usability of GUI selection.

[0083] In this embodiment, the HMD 100 was used as an example, but the gaze detection device 100a of the present invention is applicable to all so-called xR devices. It can also be applied to devices with external cameras, such as in-vehicle cameras, PCs, tablets, and smartphones. (Second example) Hereinafter, with reference to Figure 9, an HMD100 having a gaze detection device 100a according to a second embodiment will be described.

[0084] In this embodiment, the basic configuration and processing flow of the HMD 100 having the gaze detection device 100a are the same as in the first embodiment, but the processing performed by the gaze processing means 115 (gaze adjustment means 115a) and the GUI selection means 116 is different.

[0085] In the first embodiment, no processing was performed on the gaze position itself, and a mechanism was described that allows the gaze position to move outside the GUI. In this embodiment, as shown in Figure 9, there is a mechanism for smoothing the time-series data of the gaze.

[0086] The gaze processing means 115 (gaze adjustment means 115a) functions as a so-called time-series filter, averaging or removing data that deviates significantly from the time-series gaze data as noise. By using the smoothed gaze position, the effects of gaze misalignment are reduced, enabling stable GUI selection based on gaze.

[0087] If the GUI is far from the user, the gaze processing means 115 is set to increase the degree of smoothing by the smoothing process. Even with gaze deviation (gaze 3), the smoothed gaze position is judged to be fixed on the GUI, thus allowing for the continuity of GUI selection by gaze.

[0088] On the other hand, if the GUI is close to the user, the gaze processing means 115 is set to reduce the degree of smoothing by the smoothing process for the reasons described in Example 1.

[0089] For smoothing, known methods such as moving average filters and low-pass filters may be used. The number of data points used for smoothing and the methods used may be appropriately changed depending on the distance to the GUI user.

[0090] The distance between the GUI and the user, and the degree of smoothing, can be arbitrarily determined, including linear and nonlinear relationships. Multiple thresholds, including upper and lower limits, can be used, and the adjustment can be changed within or outside the range of these thresholds. The adjustment can also be changed based on user-specific gaze deviation information acquired in advance through calibration. These adjustments may only be activated when the gaze and the GUI are closer than an arbitrarily set distance range, i.e., when it is determined that the user will select the GUI.

[0091] As described in this embodiment, by adjusting the degree of smoothing of the gaze time-series data according to the distance of the GUI to the user, it is possible to reduce errors in GUI selection and false detections due to gaze, and improve the usability of GUI selection. (Third example) Hereinafter, with reference to Figure 10, an HMD 100 having a gaze detection device 100a according to a third embodiment will be described.

[0092] In this embodiment, the basic configuration and processing flow of the HMD 100 having the gaze detection device 100a are the same as in the first embodiment, but the processing performed by the gaze processing means 115 (gaze determination means 115b) and the GUI selection means 116 is different.

[0093] In this embodiment, as shown in Figures 10(a) and 10(b), the gaze processing means 115 changes the size of the determination area for determining whether the gaze is fixed on the GUI according to the distance of the GUI to the user.

[0094] Normally, the judgment area of ​​a GUI is defined by the size or frame of the GUI. In this embodiment, if the distance of the GUI to the user is far, the gaze processing means 115 sets the size of the judgment area to be larger than the size of the GUI, as shown in Figure 10(a). This allows for the continuity of GUI selection by gaze, even if there is a gaze shift (gaze 3), as the gaze remains within the judgment area.

[0095] On the other hand, if the GUI is close to the user, the gaze processing means 115 sets the size of the judgment area to be the same as or smaller than the size of the GUI, as shown in Figure 10(b). When the GUI is close to the user, the gaze can accurately focus on the GUI, but there is a possibility of unintentionally selecting the GUI when the gaze moves across the GUI. To avoid such false detections, the judgment area may be made even smaller than the size of the GUI.

[0096] Furthermore, as shown in Figures 10(c) to 10(e), the size of the GUI itself may be changed according to the distance of the GUI to the user, in conjunction with the judgment area described above. For example, in Figure 10(c), where the distance of the GUI to the user is greater than in Figure 10(d), the size of the GUI relative to the panel may be increased. Conversely, in Figure 10(e), where the distance of the GUI to the user is closer than in Figure 10(d), the size of the GUI relative to the panel may be decreased.

[0097] The distance between the GUI and the user, as well as the size of the selection area, can be arbitrarily determined, including linear and nonlinear relationships. Multiple thresholds, including upper and lower limits, can be used, and the adjustments can be changed within or outside the range of these thresholds. The adjustments can also be changed based on user-specific gaze deviation information acquired in advance through calibration. These adjustments may only be activated when the gaze and the GUI are closer than an arbitrarily set distance range, i.e., when it is determined that the user will select the GUI.

[0098] As described in this embodiment, by adjusting the size of the detection area used to determine whether the user's gaze is fixed on the GUI according to the distance of the GUI to the user, it is possible to reduce errors and false detections in GUI selection based on gaze, and improve the usability of GUI selection. (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0099] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence. [Explanation of Symbols]

[0100] 100a Eye-line detection device 113 Synthesis means (GUI control means) 112 Eye-line detection means 115 Eye-line processing means

Claims

1. A gaze detection device capable of selecting a GUI on a display element based on the user's gaze position, The display element includes a synthesis means capable of compositing the GUI at any point in the real-space coordinate system, A gaze detection means for detecting the user's gaze position on the display element, A gaze detection device characterized by having gaze processing means that adjusts the degree of influence of changes in the user's gaze position directed towards the GUI, according to the distance of the GUI to the user, which is obtained based on the user's position coordinates and the GUI's position coordinates set in the real space coordinate system.

2. The gaze detection device according to claim 1, characterized in that the gaze processing means adjusts the tolerance for gaze misalignment determination in the selection of the GUI according to the distance of the GUI to the user.

3. The gaze detection device according to claim 1 or 2, characterized in that the gaze processing means increases the tolerance for gaze misalignment determination in the selection of the GUI as the distance of the GUI to the user increases.

4. The gaze detection device according to any one of claims 1 to 3, characterized in that the gaze processing means adjusts the degree of smoothing of the user's gaze time-series data according to the distance of the GUI to the user.

5. The gaze detection device according to any one of claims 1 to 4, characterized in that the gaze processing means increases the degree of smoothing of the user's gaze time-series data as the distance of the GUI to the user increases.

6. The gaze detection device according to any one of claims 1 to 5, characterized in that the gaze processing means increases the size of the GUI as the distance of the GUI to the user increases.

7. The gaze detection device according to any one of claims 1 to 6, characterized in that the gaze processing means makes the size of the determination area for determining whether the GUI has been selected larger than the size of the GUI as the distance of the GUI to the user increases.

8. The gaze detection device according to any one of claims 1 to 7, characterized in that the gaze processing means reduces the size of the GUI as the distance of the GUI to the user decreases.

9. The gaze detection device according to any one of claims 1 to 8, characterized in that the gaze processing means reduces the size of the determination area for determining whether the GUI has been selected to the size of the GUI, as the distance of the GUI to the user decreases.

10. The gaze detection device according to claim 2 or 3, characterized in that the gaze processing means adjusts the tolerance based on the gaze deviation information for each user that has been acquired in advance.

11. The gaze detection device according to claim 4 or 5, characterized in that the gaze processing means adjusts the degree of smoothing based on the gaze misalignment information for each user that has been acquired in advance.

12. The gaze detection device according to claim 6 or 8, characterized in that the gaze processing means adjusts the size of the GUI based on the gaze misalignment information for each user that has been acquired in advance.

13. The gaze detection device according to claim 7 or 9, characterized in that the gaze processing means adjusts the size of the determination area based on the gaze deviation information for each user that has been acquired in advance.

14. The gaze detection device according to any one of claims 1 to 13, further comprising distance acquisition means for acquiring the distance of the GUI to the user.

15. A display element that displays an image to the user, A display device comprising: an eye-tracking device according to any one of claims 1 to 14.

16. The display device according to claim 15, characterized in that the display device is a head-mounted display.

17. A control method for a gaze detection device that can select a GUI on a display element based on the user's gaze position, The steps include: compositing the GUI onto the display element at any point in the real-space coordinate system; A step of detecting the user's line of sight position on the display element, A control method characterized by having a step of adjusting the degree of influence of changes in the user's line of sight directed towards the GUI, according to the distance between the user and the GUI, which is obtained based on the user's position coordinates and the GUI's position coordinates set in the real space coordinate system.

18. A program characterized by causing a computer to execute the control method described in claim 17.

Images