electronic machines

Abstract

To provide a technique that enables high-precision eye-tracking regardless of the user's eye condition.SOLUTION: An electronic device has control means for controlling the amount of light incident on a user's eye. The control means, during calibration of eye tracking function for obtaining eye gaze information on the user's eye gaze, controls the amount of light incident on the user's eye split to each of a plurality of amounts of light.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to an electronic device that performs control for calibration of a gaze detection function.

Background Art

[0002] In recent years, automation and intelligentization of head-mounted displays (HMDs) having a function of detecting a user's gaze, such as glasses-type devices using MR (Mixed Reality) or AR (Augmented Reality), have been progressing. In such an HMD, a deviation may occur between the detected fixation point and the actual fixation point (the fixation point intended by the user), and it may be impossible to display a pointer at the actual fixation point. Patent Documents 1 and 2 disclose performing calibration of a gaze detection function.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, since the state of the user's eyeballs changes moment by moment, even if conventional calibration is performed, it may not be possible to perform gaze detection with high accuracy.

[0005] An object of the present invention is to provide a technique that enables highly accurate gaze detection regardless of the state of the user's eyeballs.

Means for Solving the Problems

[0006] A first aspect of the present invention is control means for controlling the amount of light incident on the user's eyes The system includes an acquisition means for acquiring light quantity information indicating the amount of light incident on the user's eye, andThe control means has, during the calibration of the gaze detection function which obtains gaze information relating to the user's gaze, Based on the aforementioned light intensity information, This electronic device is characterized by controlling the amount of light incident on the user's eye to multiple light levels. A second aspect of the present invention is an electronic device comprising control means for controlling the amount of light incident on a user's eye and acquisition means for acquiring pupil information indicating the size of the user's pupil, wherein the control means controls the amount of light incident on the user's eye to a plurality of light increments based on the pupil information during calibration of a gaze detection function that acquires gaze information relating to the user's gaze.

[0007] This invention 3 This aspect controls the amount of light entering the user's eyes. control Step The acquisition step involves obtaining light quantity information indicating the amount of light incident on the user's eye. It has, In the control step described above, Calibration of the gaze detection function that obtains gaze information related to the user's gaze. During the session, Based on the aforementioned light intensity information, This is a control method for electronic equipment characterized by controlling the amount of light incident on the user's eye to multiple light levels. A fourth aspect of the present invention is a control method for electronic equipment comprising a control step of controlling the amount of light incident on the user's eye and an acquisition step of acquiring pupil information indicating the size of the user's pupil, wherein in the control step, during the calibration of a gaze detection function that obtains gaze information relating to the user's gaze, the amount of light incident on the user's eye is controlled to a plurality of light increments based on the pupil information.

[0008] This invention 5 This aspect controls the amount of light entering the user's eyes. control Step The acquisition step involves obtaining light quantity information indicating the amount of light incident on the user's eye. A program for causing a computer to execute the above control In step, during the calibration of the gaze detection function that obtains gaze information related to the user's gaze, Based on the aforementioned light intensity information, This program is characterized by controlling the amount of light incident on the user's eye to multiple light levels. A sixth aspect of the present invention is a program for causing a computer to perform a control step of controlling the amount of light incident on a user's eye and an acquisition step of acquiring pupil information indicating the size of the user's pupil, wherein the control step is characterized in that, during calibration of a gaze detection function that obtains gaze information relating to the user's gaze, the amount of light incident on the user's eye is controlled to a plurality of light increments based on the pupil information.

[0009] This invention 7 This aspect controls the amount of light entering the user's eyes. control Step The acquisition step involves obtaining light quantity information indicating the amount of light incident on the user's eye. A computer-readable storage medium that stores a program for causing a computer to execute the program, control In step, during the calibration of the gaze detection function that obtains gaze information related to the user's gaze, Based on the aforementioned light intensity information,A storage medium characterized by controlling the amount of light incident on the user's eyes to each of a plurality of light amounts. An eighth aspect of the present invention is a computer-readable storage medium that stores a program causing a computer to perform a control step of controlling the amount of light incident on a user's eye and an acquisition step of acquiring pupil information indicating the size of the user's pupil, wherein the control step, during calibration of a gaze detection function that obtains gaze information relating to the user's gaze, controls the amount of light incident on the user's eye to a plurality of light increments based on the pupil information.

Advantages of the Invention

[0010] According to the present invention, highly accurate gaze detection is possible regardless of the state of the user's eyeballs.

Brief Description of the Drawings

[0011] [Figure 1] It is an external view of a display device. [Figure 2] It is a block diagram of a display device. [Figure 3] It is a diagram for explaining the principle of a gaze detection method. [Figure 4] It is a diagram showing an eye image. [Figure 5] It is a flowchart of a gaze detection operation. [Figure 6] It is a diagram showing a visual field. [Figure 7] It is a diagram showing the correspondence between pupil size and appropriate gaze correction parameters. [Figure 8] It is a diagram showing the installation state of a dimming device. [Figure 9] It is a diagram showing the change in the transmittance of a dimming device. [Figure 10] It is a flowchart of a calibration operation. <​​​​​​​​​​​This is possible. Furthermore, the user can see the virtual object displayed on an optical see-through display device.

[0013] Furthermore, the present invention is also applicable to other display devices. For example, the present invention is also applicable to video see-through type display devices (video see-through type HMDs). Video see-through type display devices display a virtual space captured from the outside world in near real-time. A user wearing a video see-through type display device cannot directly see the outside world, but can indirectly see the outside world by viewing the virtual space displayed on the video see-through type display device. The present invention is also applicable to HMDs that display a virtual space unrelated to the outside world. Mr. / Ms. It is applicable to various control devices. For example, the present invention is not limited to display devices, but can also be applied to various electronic devices that perform control for the calibration of gaze detection functions (e.g., controllers for display devices, personal computers (PCs), etc.).

[0014] <Explanation of the structure> Figures 1(A) and 1(B) show the external appearance of the display device 100 according to Embodiment 1. Figure 1(A) is a front perspective view, and Figure 1(B) is a rear perspective view. The display device 100 is an optical see-through type display device, a type of head-mounted display (HMD) that can be attached to and removed from the head, and is a glasses-type device that utilizes MR (Mixed Reality) and AR (Augmented Reality). The display device 100 can individually detect the gaze of the right eye and the gaze of the left eye of a user wearing the display device 100 on their head. Hereafter, a user wearing the display device 100 on their head will be simply referred to as the user.

[0015] Lens 10 is an optical component that faces the user's eye. The user can see the outside world through lens 10. The display device 11 displays virtual objects (virtual images of virtual objects) to both of the user's eyes (both right and left eyes) under control (display control) from CPU 2, which will be described later. The user can see the displayed virtual objects as if they exist in the outside world. Light source driving circuit 12 drives light sources 13a and 13b. Light sources 13a and 13b are light sources that illuminate the user's eye, and are, for example, infrared light-emitting diodes that emit infrared light that is insensitive to the user. Light emitted from light sources 13a and 13b and reflected by the user's eye A portion of the light is focused onto the eye image sensor 17 by the light-receiving lens 16. These components are provided for both the left and right eyes. For example, the eye image sensor 17 includes a right image sensor for imaging the right eye and a left image sensor for imaging the left eye. The dimming device 19 (dimming panel) adjusts the light from the outside. The dimming control circuit 18 is a circuit that changes the transmittance of the dimming device 19, and is, for example, a circuit that controls the voltage applied to the dimming device 19. The dimming device 19 transmits light from the outside at a transmittance controlled by the dimming control circuit 18.

[0016] Figure 2 is a block diagram showing the electrical configuration of the display device 100. The CPU 2 is the central processing unit of the microcomputer built into the display device 100 and controls the entire display device 100. The display device 11, light source drive circuit 12, gaze detection circuit 15, dimming control circuit 18, memory unit 3, etc. are connected to the CPU 2.

[0017] The memory unit 3 has a function to store video signals from the eye image sensor 17 and a function to store gaze correction parameters that correct individual differences in gaze, which will be described later.

[0018] The gaze detection circuit 15 performs A / D conversion on the output of the eye image sensor 17 (eye image captured from the eye) when an optical image of the eye is formed on the eye image sensor 17, and transmits the result to the CPU 2. The CPU 2 extracts feature points necessary for gaze detection from the eye image according to a predetermined algorithm described later, and detects the user's gaze from the position of the feature points. For example, the CPU 2 acquires right gaze information related to the gaze of the right eye based on the right eye image obtained by the right image sensor, and acquires left gaze information related to the gaze of the left eye based on the left eye image obtained by the left image sensor.

[0019] <Explanation of eye-tracking operation> The gaze detection method will be explained using Figures 3, 4(A), 4(B), and 5. Both the gaze of the right eye and the gaze of the left eye are detected by the following gaze detection method. Figure 3 is a diagram illustrating the principle of the gaze detection method and is a schematic diagram of the optical system for performing gaze detection. Figure 4(A) is a schematic diagram of the eye image (optical image projected onto the eye image sensor 17) captured by the eye image sensor 17, and Figure 4(B) is a diagram showing the output intensity of the CCD in the eye image sensor 17. Figure 5 shows a schematic flowchart of the gaze detection operation.

[0020] When the gaze detection operation starts, in step S101 of Figure 5, the CPU 2 controls the light sources 13a and 13b using the light source drive circuit 12 to emit infrared light toward the user's eyeball 14. The optical image of the user's eye, illuminated by the infrared light, is formed on the eye image sensor 17 through the light-receiving lens 16 and converted into an electrical signal by the eye image sensor 17. This provides an electrical signal of the eye image that can be processed.

[0021] In step S102, the CPU2 acquires an eye image (eye image signal, electrical signal of the eye image) from the eye image sensor 17 via the gaze detection circuit 15.

[0022] In step S103, the CPU2 detects the coordinates of the points corresponding to the corneal reflection images Pd and Pe of light sources 13a and 13b and the pupil center c from the eye image obtained in step S102.

[0023] Infrared light emitted from light sources 13a and 13b illuminates the cornea 142 of the user's eyeball 14. At this time, the corneal reflection images Pd and Pe, formed by a portion of the infrared light reflected from the surface of the cornea 142, are focused by the light-receiving lens 16 and imaged onto the eye image sensor 17 to become the corneal reflection images Pd' and Pe' in the eye image. Similarly, the light beams from the ends a and b of the pupil 141 are also imaged onto the eye image sensor 17 to become the pupil end images a' and b' in the eye image.

[0024] Figure 4(B) shows the luminance information (luminance distribution) of region α in the eye image of Figure 4(A). In Figure 4(B), the horizontal direction of the eye image is defined as the X-axis direction, and the vertical direction as the Y-axis direction, with the brightness distribution in the X-axis direction shown. In Example 1, the X-axis coordinates (horizontal direction) of the corneal reflection images Pd',Pe' are set to Xd,Xe, and the X-axis coordinates of the pupil end images a',b' are set to Xa,Xb. As shown in Figure 4(B), extremely high levels of brightness are obtained at coordinates Xd,Xe of the corneal reflection images Pd',Pe'. In the region from coordinate Xa to coordinate Xb, which corresponds to the pupil 141 region (the region of the pupil image obtained when the light beam from the pupil 141 is imaged on the eye image sensor 17), extremely low levels of brightness are obtained except at coordinates Xd,Xe. In the iris 143 region outside the pupil 141 (the region of the iris image outside the pupil image obtained when the light beam from the iris 143 is imaged), a brightness intermediate between the two types of brightness is obtained. Specifically, an intermediate brightness level is obtained between the two brightness levels mentioned above in the region where the X-coordinate (coordinate in the X-axis direction) is greater than coordinate Xa and in the region where the X-coordinate is less than coordinate Xb.

[0025] From the luminance distribution shown in Figure 4(B), the X coordinates Xd and Xe of the corneal reflection images Pd' and Pe', and the X coordinates Xa and Xb of the pupil end images a' and b' can be obtained. Specifically, coordinates with extremely high luminance can be obtained as the coordinates of the corneal reflection images Pd' and Pe', and coordinates with extremely low luminance can be obtained as the coordinates of the pupil end images a' and b'. Furthermore, when the rotation angle θx of the optical axis of the eyeball 14 with respect to the optical axis of the light-receiving lens 16 is small, the coordinate Xc of the pupil center image c' (center of the pupil image) obtained by imaging the light beam from the pupil center c onto the eye image sensor 17 can be expressed as Xc ≈ (Xa + Xb) / 2. In other words, the coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa and Xb of the pupil end images a' and b'. In this way, the coordinates of the corneal reflection images Pd',Pe' and the coordinates of the pupillary central image c' can be estimated.

[0026] In step S104, the CPU2 calculates the imaging magnification β of the eye image. The imaging magnification β is determined by the position of the eyeball 14 relative to the light-receiving lens 16, and can be calculated using a function of the interval (Xd-Xe) between the corneal reflection images Pd',Pe'.

[0027] In step S105, the CPU2 calculates the rotation angle of the optical axis of the eyeball 14 relative to the optical axis of the light-receiving lens 16. The X-coordinate of the midpoint of the corneal reflection image Pd and the corneal reflection image Pe is approximately the same as the X-coordinate of the curvature center O of the cornea 142. Therefore, if we take Oc as the standard distance from the curvature center O of the cornea 142 to the center c of the pupil 141, the rotation angle θx of the eyeball 14 in the ZX plane (plane perpendicular to the Y axis) can be calculated using the following equation 1. The rotation angle θy of the eyeball 14 in the ZY plane (plane perpendicular to the X axis) can also be calculated using the same method as for calculating the rotation angle θx. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc (Formula 1)

[0028] In step S106, CPU2 estimates the user's point of fixation on lens 10 using the rotation angles θx and θy calculated in step S105. Assuming that the coordinates of the point of fixation (Hx, Hy) correspond to the pupil center c, the coordinates of the point of fixation (Hx, Hy) can be calculated using the following equations 2 and 3. The point of fixation can also be understood as the position where the gaze is directed, the position the user is looking at, the gaze position, or the point of focus. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0029] The parameter m in equations 2 and 3 is a transformation coefficient that converts the rotation angles θx and θy to coordinates corresponding to the pupil center c on the lens 10. The parameter m is assumed to be predetermined and stored in the memory unit 3. The parameters Ax, Bx, Ay, and By are gaze correction parameters that compensate for individual differences in gaze, and are determined (acquired) by performing the calibration process described later. The line correction parameters Ax, Bx, Ay, and By are stored in the memory unit 3 before the gaze detection operation starts.

[0030] In step S107, the CPU2 stores the coordinates (Hx, Hy) of the point of focus in the memory unit 3 and completes the gaze detection operation.

[0031] The gaze detection method is not limited to the method described above; any method that obtains gaze information from an eye image is acceptable. The final gaze information may not indicate the point of fixation, but rather the direction of the gaze. For example, processing may be performed to obtain the rotation angle (Ax×θx+Bx or Ay×θy+By) without obtaining the coordinates (Hx,Hy) of the point of fixation.

[0032] <Explanation of the necessity of calibration work> As described above, in the gaze detection operation, the rotation angles θx and θy of the eyeball 14 are obtained from the eye image, and the point of fixation can be estimated by performing a coordinate transformation on the position of the pupil center c to the position on the lens 10. Figure 6(A) is a diagram showing the user's field of view (the range that the user can see through the lens 10, the imaging range of the external imaging unit 20), and shows the state in which the display device 11 is operating. As shown in Figure 6(A), the display device 11 displays a frame or the like at the current point of fixation A (estimated position). The display device 100 may control the display device 11 to display information related to the real object that the user is fixated on as a UI (User Interface) according to the results of gaze detection (gaze information, point of fixation information). A real object is an object that actually exists in the external world (real space).

[0033] However, due to factors such as individual differences in the shape of the human eyeball, it may not be possible to estimate the point of fixation with high accuracy. Specifically, if the gaze correction parameters Ax, Ay, Bx, and By are not adjusted to values ​​suitable for the user, a discrepancy will occur between the actual point of fixation B and the estimated point of fixation C, as shown in Figure 6(B). In Figure 6(B), the user is fixating on the person, but the display device 100 has incorrectly estimated that the user is fixating on the background.

[0034] Therefore, before normal use of the display device 100, it is necessary to perform a calibration of the gaze detection function to determine the gaze correction parameters Ax, Ay, Bx, and By that are suitable for the user, and store them in the display device 100.

[0035] Conventionally, calibration has been performed by displaying multiple indicators at different positions, as shown in Figure 6(C), before normal use of the display device 100, and having the user look at these indicators. Then, a gaze detection operation is performed when the user gazes at each indicator, and a technique is known to determine gaze correction parameters Ax, Ay, Bx, and By suitable for the user from the calculated multiple gaze points (estimated positions) and the coordinates of each indicator.

[0036] However, the state of a user's eyes changes constantly. Even with conventional calibration procedures, changes in the state of the eyes alter the appropriate gaze correction parameters Ax, Ay, Bx, and By, causing a discrepancy (increasing) between the actual point of fixation and the estimated point of fixation. In other words, changes in the state of the eyes reduce the accuracy of gaze detection.

[0037] For example, when the user moves from indoors to outdoors, changing the environment in which the display device 100 is used, and the brightness of the outside world changes, the size of the pupil (pupil size, pupil diameter, pupil diameter and radius) changes. As a result, the appropriate gaze correction parameters Ax, Ay, Bx, and By change, and the accuracy of gaze detection decreases. Figure 7 shows the correspondence between pupil size and the appropriate gaze correction parameter Ax.

[0038] While repeating the calibration process can improve the accuracy of gaze detection, performing the calibration every time the eye condition changes is cumbersome. Therefore, in Example 1, high-precision gaze detection is enabled regardless of the user's eye condition without repeating the calibration process. Note that in Figure 7, the appropriate gaze correction parameter Ax increases linearly with increasing pupil size, but this trend varies from person to person. Therefore, it is difficult to estimate the appropriate gaze correction parameters Ax, Bx, Ay, and By (changes) from pupil size (changes), and it is necessary to determine the gaze correction parameters Ax, Bx, Ay, and By through the calibration process.

[0039] <Calibration Operation Explanation> In Example 1, during the calibration process, the amount of light incident on the user's eye (incident light amount) is controlled to multiple light amounts. For example, during the calibration process, the transmittance of the display device 100 (transmittance of the dimming device 19) is controlled to multiple transmittances. As shown in Figure 8, the dimming device 19 is provided so as to cover the front surface of the lens 10. Therefore, by controlling the transmittance of the dimming device 19, the amount of incident light on the eye is controlled. The dimming device 19 may also be provided so as to cover the back surface of the lens 10. Furthermore, the method of controlling the amount of incident light on the eye is not limited to the above method. For example, in the case of a video see-through type display device, the display brightness may be controlled.

[0040] Then, for each amount of light entering the eye, the gaze correction parameters Ax, Bx, Ay, and By (parameters used for gaze detection (gaze detection operation, gaze detection function)) are determined. This process can also be seen as determining the gaze correction parameters Ax, Bx, Ay, and By for each pupil size.

[0041] In this way, a user-friendly correspondence can be determined between pupil size (amount of light incident on the eye) and gaze correction parameters Ax, Bx, Ay, and By. Consequently, when the operating environment of the display device 100 changes and the brightness of the outside world changes, gaze detection can be performed with high accuracy using the gaze correction parameters Ax, Bx, Ay, and By corresponding to pupil size (amount of light incident on the eye) without having to perform calibration work again.

[0042] Figures 9(A) to 9(C) illustrate a specific example of controlling the transmittance of the dimming device 19 during calibration.

[0043] First, the display device 100 controls the transmittance of the dimming device 19 to its maximum, as shown in Figure 9(A), to determine the gaze correction parameters Ax, Ay, Bx, and By. Since the transmittance of the dimming device 19 is at its maximum, the gaze correction parameters Ax, Ay, Bx, and By can also be considered as the gaze correction parameters when the amount of incident light on the eye is at its maximum (when the pupil size is at its minimum) during the calibration process.

[0044] Next, the display device 100 controls the transmittance of the dimming device 19 to the minimum, as shown in Figure 9(B), to determine the gaze correction parameters Ax, Ay, Bx, and By. Since the transmittance of the dimming device 19 is at its minimum, the gaze correction parameters Ax, Ay, Bx, and By can also be considered as the gaze correction parameters when the amount of incident light on the eye is minimum (when the pupil size is maximum) during the calibration process.

[0045] Furthermore, as shown in Figure 9(C), the display device 100 controls the transmittance of the dimming device 19 to 40% to determine the gaze correction parameters Ax, Ay, Bx, and By. The gaze correction parameters Ax, Ay, Bx, and By obtained in this way can also be considered as gaze correction parameters when the amount of incident light on the eye is between the maximum and minimum during the calibration process (when the pupil size is between the maximum and minimum).

[0046] The above transmittance of 40% is the transmittance such that the amount of incident light to the eye is intermediate between the amount of incident light in the state shown in Figure 9(A) and the amount of incident light in the state shown in Figure 9(B) (the amount of incident light that bisects the range from the amount of incident light in the state shown in Figure 9(A) to the amount of incident light in the state shown in Figure 9(B)). The light intensity information indicating the amount of incident light to the eye can be obtained, for example, based on the brightness of the eye image. However, it is not limited to this, and the detection result of a sensor that detects the brightness inside (user side) of the display device 100 may be obtained as light intensity information, or other information based on the said detection result may be obtained as light intensity information. Light intensity information may also be obtained based on the detection result of a sensor that detects the brightness outside the display device 100 and the transmittance of the dimming device 19. From the light intensity information obtained in the state shown in Figure 9(A) (the state with maximum transmittance) and the light intensity information obtained in the state shown in Figure 9(B) (the state with minimum transmittance), the transmittance required to obtain an incident light amount intermediate between the two incident light amounts indicated by the two light intensity information can be easily determined (calculated).

[0047] Note that the maximum, minimum, and 40% transmittances are examples, and the transmittance of the dimming device 19 is not particularly limited. The transmittance of the dimming device 19 may be controlled in more than three steps. The number of steps for controlling the transmittance of the dimming device 19 may be determined by comparing the amount of incident light to the eye when the transmittance of the dimming device 19 is at its maximum with one or more thresholds. For example, the number of steps may be increased when the amount of incident light when the transmittance of the dimming device 19 is at its maximum is sufficiently large. By doing so, it is possible to obtain information that shows in detail the correspondence between pupil size (amount of incident light to the eye) and gaze correction parameters over a wide range from the amount of incident light when the transmittance of the dimming device 19 is at its minimum to the amount of incident light when the transmittance of the dimming device 19 is at its maximum. However, increasing the number of steps will increase the calibration time. Therefore, the number of steps may be reduced when the amount of incident light when the transmittance of the dimming device 19 is at its maximum is not very large. By doing so, the calibration process can be shortened without obtaining unnecessarily detailed information regarding the correspondence between pupil size (amount of light entering the eye) and gaze correction parameters.

[0048] Furthermore, during the calibration process, the display device 100 may acquire pupil information indicating the pupil size. For example, the display device 100 may calculate the pupil size as the distance from the X coordinate Xa of pupil end image a' to the X coordinate Xb of pupil end image b'.

[0049] Figure 10 is a flowchart of the calibration operation. For example, when a user attaches the display device 100 and instructs the start of calibration from a menu screen displayed on the display device 100, the calibration operation shown in Figure 10 begins. 。

[0050] In step S201, the CPU2 controls the transmittance of the dimming device 19 using the dimming control circuit 18. For example, the CPU2 controls the transmittance of the dimming device 19 to its maximum.

[0051] In step S202, the CPU2 controls the display device 11 to display a calibration screen, including calibration indicators, to both of the user's eyes. For example, multiple indicators are displayed as shown in Figure 6(C).

[0052] In step S203, CPU2 notifies the user of indicators that should be paid attention to. For example, CPU2 may highlight an indicator that should be paid attention to by changing its shape, color, brightness, etc., among several indicators. However, the method of notifying the user of indicators to pay attention to is not limited to this. For example, only the indicators to pay attention to may be displayed.

[0053] In step S204, the CPU2 determines whether the user has performed a decision operation. The decision operation is performed by the user to inform the display device 100 that they are focusing on the indicator. Therefore, The decision operation is performed while the user is looking at the indicator that they should be focusing on. The decision operation is, for example, pressing a button on a controller connected to the display device 100 by wire or wirelessly. The decision operation is not particularly limited and may be an operation in which the user continues to focus on the indicator for a longer period of time than predetermined, or it may be a voice operation. The CPU 2 waits for the decision operation, and when the decision operation is performed, it proceeds to step S205.

[0054] In step S205, CPU2 performs the gaze detection operation shown in Figure 5. The gaze detection operation is performed for both the right and left eyes. As a result, right gaze information (the result of the gaze detection operation for the right eye) and left gaze information (the result of the gaze detection operation for the left eye) are acquired (gaze acquisition).

[0055] In step S206, CPU2 determines whether or not it has performed gaze detection for all indicators. If there are any indicators for which gaze detection has not been performed, CPU2 proceeds to step S207. If it has performed gaze detection for all indicators, it proceeds to step S208.

[0056] In step S207, CPU2 notifies of a change in the indicator to be monitored. For example, CPU2 switches the indicator to be highlighted among several indicators and proceeds to step S204, waiting for a decision operation.

[0057] In step S208, the CPU2 determines (calculates) the gaze correction parameters Ax, Bx, Ay, and By mentioned above from the results of the gaze detection operation for all indicators.

[0058] In step S209, the CPU2 determines whether it has determined the line-of-sight correction parameters Ax, Bx, Ay, and By for all transmittances (transmittances of the dimming device 19). If there are any transmittances for which the line-of-sight correction parameters Ax, Bx, Ay, and By have not yet been determined, the CPU2 proceeds to step S210. If the line-of-sight correction parameters Ax, Bx, Ay, and By have been determined for all transmittances, the CPU2 terminates the calibration operation.

[0059] In step S210, the CPU2 controls (changes) the transmittance of the dimming device 19 using the dimming control circuit 18. Then, the CPU2 proceeds to step S202.

[0060] This section explains how to calculate the gaze correction parameter Bx. The gaze correction parameter By can be calculated using the same method as the gaze correction parameter Bx.

[0061] Figure 11 shows the internal structure of the eyeball 14 when the rotation angle θx shown in Figure 3 is 0 degrees. In Figure 11, the photoreceptor cells, which are responsible for sensing light incident on the eyeball 14 and sending signals to the brain, are offset from the optical axis of the eyeball 14 (the optical axis of the cornea 142 and pupil 141), as indicated by the dashed line. Therefore, when the user looks at the center of the display surface of the display device 11, the rotation angle θx is offset from 0 degrees by an offset angle corresponding to the amount of displacement of the photoreceptor cells from the optical axis of the eyeball 14. This offset angle (amount of displacement of the photoreceptor cells) corresponds to the gaze correction parameter Bx (Bx ∝ offset angle).

[0062] For example, as shown in Figure 6(C), CPU2 displays multiple indicators on the display surface of display device 11 and makes the indicator in the center of the display surface blink, thereby causing the user to focus on the center of the display surface. Using the eye image obtained while the user is focusing on the center of the display surface, CPU2 performs the gaze detection operation shown in Figure 5 to calculate the rotation angle θx as the offset angle. Then, CPU2 calculates the gaze correction parameter Bx according to the offset angle.

[0063] This section explains how to calculate the gaze correction parameter Ax. The gaze correction parameter Ay is calculated based on the gaze It can be calculated using the same method as the calculation method for the line correction parameter Ax.

[0064] Equation 1 uses the standard distance Oc (constant), which is the distance from the corneal curvature center O to the pupil center c, to calculate the rotation angle θx. However, the actual distance Oc' (variable), which is the distance from the corneal curvature center O to the pupil center c, is not necessarily the same as the distance Oc. The difference between distance Oc' and distance Oc represents the error in the rotation angle θx calculated using Equation 1. The gaze correction parameter Ax is a parameter that reduces such errors and is inversely proportional to the actual distance Oc' (Ax ∝ 1 / Oc'). The value obtained by dividing the standard distance Oc by the gaze correction parameter Ax is the actual distance Oc'.

[0065] For example, CPU2 calculates the gaze correction parameter Ax based on multiple eye images taken multiple times while the user sequentially gazes at multiple positions on the display surface of the display device 11. Specifically, CPU2 sequentially gazes at two or more indicators from among the multiple indicators shown in Figure 6(C) that have different horizontal positions. CPU2 calculates the rotation angle θx for each of the two or more eye images corresponding to the two or more indicators. The eye images corresponding to the indicators are eye images obtained while the user is gazing at the indicator in question. Then, CPU2 calculates the gaze correction parameter Ax based on the two or more calculated rotation angles θx. For example, CPU2 calculates the gaze correction parameter Ax so as to minimize the sum of errors (such as the sum of squared residuals using the least squares method) of the two or more calculated rotation angles θx. Alternatively, the gaze correction parameter Ax may be calculated so that the errors of each of the two or more calculated rotation angles θx (the difference between the target rotation angle corresponding to the position of the indicator and the calculated rotation angle θx) are approximately the same.

[0066] <Explanation of how to determine the gaze correction parameters to use> According to the calibration operation shown in Figure 10, information indicating a user-appropriate correspondence between pupil size (amount of incident light into the eye) and gaze correction parameters Ax, Bx, Ay, and By can be obtained. Consequently, gaze detection can be performed with high accuracy using gaze correction parameters Ax, Bx, Ay, and By corresponding to pupil size (amount of incident light into the eye) without having to perform the calibration work again.

[0067] According to the calibration operation shown in Figure 10, multiple combinations of pupil size (amount of incident light into the eye) and gaze correction parameters Ax, Bx, Ay, and By can be obtained by changing the pupil size (amount of incident light into the eye). The information from these multiple combinations can also be considered as a table showing the correspondence between pupil size (amount of incident light into the eye) and gaze correction parameters Ax, Bx, Ay, and By. After the calibration operation, if the pupil size (amount of incident light into the eye) is the same as during the calibration operation, the gaze correction parameters Ax, Bx, Ay, and By corresponding to the pupil size (amount of incident light into the eye) can be obtained from the table above.

[0068] However, the pupil size (amount of light entering the eye) may not be the same after the calibration operation. If the pupil size (amount of light entering the eye) differs after the calibration operation, the gaze correction parameters Ax, Bx, Ay, and By corresponding to the pupil size (amount of light entering the eye) cannot be obtained from the table above. To consider such cases, a function showing the correspondence between the pupil size (amount of light entering the eye) and the gaze correction parameters Ax, Bx, Ay, and By may be estimated based on the above multiple combinations. By doing so, gaze detection can be performed with high accuracy by obtaining and using the corresponding gaze correction parameters Ax, Bx, Ay, and By from the function for any pupil size (amount of light entering the eye).

[0069] The function showing the correspondence between pupil size (amount of incident light into the eye) and line-of-sight correction parameters Ax, Bx, Ay, and By can be determined, for example, by the least squares method using Equation 4 below. In Equation 4, s is the error, and Ri is the size of the i-th pupil. Ax(Ri) is the pupil size Ax is the gaze correction parameter Ax corresponding to Ri, and is the gaze correction parameter Ax determined by the calibration operation. Ax'(Ri) is the gaze correction parameter Ax corresponding to pupil size Ri, and is the gaze correction parameter Ax determined by a function (candidate) that shows the correspondence between pupil size and gaze correction parameter Ax. The function that shows the correspondence between pupil size and gaze correction parameter Ax is determined so as to minimize the error s (sum of squared residuals). Similarly, functions that show the correspondence between pupil size and gaze correction parameters Bx, Ay, and By are also determined. s=Σ i=1~n {Ax(Ri)-Ax'(Ri)} 2 ...(Formula 4)

[0070] Furthermore, the method for determining the function that shows the correspondence between pupil size (amount of incident light into the eye) and the gaze correction parameters Ax, Bx, Ay, and By is not limited to the least squares method. For example, linear interpolation may be performed by connecting the coordinates of multiple combinations (multiple combinations of pupil size (amount of incident light into the eye) and gaze correction parameters Ax, Bx, Ay, and By) obtained by the calibration operation with straight lines.

[0071] <<Example 2>> Embodiment 2 of the present invention will now be described. Embodiment 1 described an example in which light intensity information indicating the amount of light incident on the user's eye is acquired, and the amount of light incident on the user's eye is controlled to each of several light intensities based on the light intensity information. Embodiment 2 describes an example in which pupil information indicating the size of the user's pupil is acquired, and the amount of light incident on the user's eye is controlled to each of several light intensities based on the pupil information.

[0072] In Example 2, during the calibration process, the display device 100 acquires pupil information indicating the pupil size. For example, the display device 100 calculates the pupil size as the distance from the X coordinate Xa of pupil end image a' to the X coordinate Xb of pupil end image b'.

[0073] First, the display device 100 controls the transmittance of the dimming device 19 to its maximum to determine the gaze correction parameters Ax, Ay, Bx, and By. Since the transmittance of the dimming device 19 is at its maximum, the gaze correction parameters Ax, Ay, Bx, and By can also be considered as the gaze correction parameters when the pupil size is at its minimum (when the amount of incident light on the eye is at its maximum) during the calibration process.

[0074] Next, the display device 100 controls the transmittance of the dimming device 19 to the minimum and determines the gaze correction parameters Ax, Ay, Bx, and By. Since the transmittance of the dimming device 19 is at its minimum, the gaze correction parameters Ax, Ay, Bx, and By can also be considered as the gaze correction parameters when the pupil size is at its maximum (when the amount of light entering the eye is at its minimum) during the calibration process.

[0075] Furthermore, the display device 100 determines (calculates) a transmittance such that the user's pupil size is midway between the two pupil sizes indicated by the two sets of pupil information, based on the pupil information when the transmittance of the dimming device 19 is controlled to its maximum and when it is controlled to its minimum. Then, the display device 100 controls the transmittance of the dimming device 19 to the determined transmittance and determines the gaze correction parameters Ax, Ay, Bx, and By.

[0076] The above three types (three stages) of transmittance are merely examples, and the transmittance of the dimming device 19 is not particularly limited. The transmittance of the dimming device 19 may be controlled in more than three stages. The number of stages for controlling the transmittance of the dimming device 19 may be determined by comparing the pupil size when the transmittance of the dimming device 19 is at its maximum with one or more thresholds. For example, the dimming device If the pupil size is sufficiently small when the transmittance of the chair 19 is at its maximum, the number of steps may be increased. By doing so, detailed information showing the correspondence between pupil size (amount of light incident on the eye) and gaze correction parameters can be obtained over a wide range from the pupil size when the transmittance of the photochromic device 19 is at its maximum to the pupil size when the transmittance of the photochromic device 19 is at its minimum. However, increasing the number of steps will lengthen the calibration time. Therefore, if the pupil size is not so small when the transmittance of the photochromic device 19 is at its maximum, the number of steps may be decreased. By doing so, the calibration time can be shortened without obtaining unnecessarily detailed information showing the correspondence between pupil size (amount of light incident on the eye) and gaze correction parameters.

[0077] <<Summary>> As described above, according to Examples 1 and 2, during calibration, the amount of light incident on the user's eye is controlled to match multiple light levels. This enables highly accurate gaze detection regardless of the user's eye condition. Specifically, a correspondence between pupil size (amount of light incident on the eye) and gaze correction parameters Ax, Bx, Ay, and By can be determined to suit the user. Consequently, even if the pupil size (amount of light incident on the eye) changes, gaze detection can be performed with high accuracy using the gaze correction parameters Ax, Bx, Ay, and By corresponding to the pupil size (amount of light incident on the eye) without having to perform the calibration process again.

[0078] It should be noted that Examples 1 and 2 are merely examples, and configurations obtained by appropriately modifying or changing the configurations of Examples 1 and 2 within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the configurations of Examples 1 and 2 are also included in the present invention.

[0079] <<Other Examples>> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0080] This embodiment includes the following configurations, methods, programs, and media. (Composition 1) It has control means for controlling the amount of light that enters the user's eye, The control means controls the amount of light incident on the user's eye to a plurality of light increments during calibration of the gaze detection function which obtains gaze information related to the user's gaze. An electronic device characterized by the following features. (Configuration 2) The system further comprises calibration means for performing the aforementioned calibration. The electronic device according to configuration 1, characterized by the features described above. (Composition 3) The aforementioned gaze information is information relating to the gaze of a user wearing an optical see-through display device on their head. The control means controls the transmittance of the display device to each of the plurality of transmittances during the calibration. The electronic device according to configuration 1 or 2, characterized by the above. (Composition 4) The electronic device is the display device. The electronic device according to configuration 3, characterized by the features described above. (Composition 5) The system further includes an acquisition means for acquiring light quantity information indicating the amount of light incident on the user's eye, The control means controls the amount of light incident on the user's eye based on the light intensity information, and controls a plurality of Control the light intensity for each element. An electronic device according to any one of configurations 1 to 4, characterized by the features described herein. (Composition 6) The aforementioned gaze information is information relating to the gaze of a user wearing an optical see-through display device on their head. The control means, during the calibration, (1) Control the transmittance of the display device to the maximum, (2) Control the transmittance of the display device to the minimum, (3) Based on the first light intensity information acquired by the acquisition means in a first state in which the transmittance of the display device is controlled to the maximum, and the second light intensity information acquired by the acquisition means in a second state in which the transmittance of the display device is controlled to the minimum, the transmittance of the display device is controlled to a transmittance such that the amount of light incident on the user's eye is intermediate between the light intensity indicated by the first light intensity information and the light intensity indicated by the second light intensity information. The electronic device according to configuration 5, characterized by the features described herein. (Composition 7) The system further includes means for acquiring pupil information indicating the size of the user's pupil, The control means controls the amount of light incident on the user's eye to a plurality of light intensities based on the pupil information. An electronic device according to any one of configurations 1 to 4, characterized by the features described herein. (Composition 8) The aforementioned gaze information is information relating to the gaze of a user wearing an optical see-through display device on their head. The control means, during the calibration, (1) Control the transmittance of the display device to the maximum, (2) Control the transmittance of the display device to the minimum, (3) Based on the first pupil information acquired by the acquisition means in a first state in which the transmittance of the display device is controlled to the maximum, and the second pupil information acquired by the acquisition means in a second state in which the transmittance of the display device is controlled to the minimum, the transmittance of the display device is controlled to a transmittance such that the size of the pupil is intermediate between the size indicated by the first pupil information and the size indicated by the second pupil information. The electronic device according to configuration 7, characterized by the features described above. (Composition 9) In the calibration described above, the parameters used for the gaze detection function are determined for each amount of light incident on the user's eye. An electronic device according to any one of configurations 1 to 8, characterized by the above. (Composition 10) The process includes a step of controlling the amount of light entering the user's eye. In the above step, during the calibration of the gaze detection function that obtains gaze information relating to the user's gaze, the amount of light incident on the user's eye is controlled to each of a plurality of light intensities. A method for controlling electronic equipment characterized by the following features. (program A program for causing a computer to function as one of the electronic devices described in any one of items 1 to 9 of the configuration. (medium) A computer-readable storage medium containing a program for causing the computer to function as one of the electronic devices described in any one of items 1 to 9 of the configuration. [Explanation of Symbols]

[0081] 100:Display device 2:CPU

Claims

1. A control means for controlling the amount of light entering the user's eye, The system includes an acquisition means for acquiring light quantity information indicating the amount of light incident on the user's eye, The control means, during calibration of the gaze detection function that obtains gaze information relating to the user's gaze, controls the amount of light incident on the user's eye to each of a plurality of light intensities based on the light intensity information. An electronic device characterized by the following features.

2. The system further comprises calibration means for performing the aforementioned calibration. The electronic device according to feature 1.

3. The aforementioned gaze information is information relating to the gaze of a user wearing an optical see-through display device on their head. The control means controls the transmittance of the display device to each of the plurality of transmittances during the calibration. The electronic device according to claim 1 or 2.

4. The electronic device is the display device. The electronic device according to feature 3.

5. The aforementioned gaze information is information relating to the gaze of a user wearing an optical see-through display device on their head. The control means, during the calibration, (1) Control the transmittance of the display device to the maximum, (2) Control the transmittance of the display device to the minimum, (3) Based on the first light intensity information acquired by the acquisition means in a first state in which the transmittance of the display device is controlled to the maximum, and the second light intensity information acquired by the acquisition means in a second state in which the transmittance of the display device is controlled to the minimum, the amount of light incident on the user's eye is set to be intermediate between the light intensity indicated by the first light intensity information and the light intensity indicated by the second light intensity information. The transmittance of the display device is controlled to the transmittance of the display device. The electronic device according to feature 1.

6. In the calibration described above, the parameters used for the gaze detection function are determined for each amount of light incident on the user's eye. The electronic device according to claim 1 or 2.

7. A control means for controlling the amount of light incident on the user's eye, The system includes means for acquiring pupil information indicating the size of the user's pupil, The control means, during calibration of the gaze detection function that obtains gaze information relating to the user's gaze, controls the amount of light incident on the user's eye to a plurality of light intensities based on the pupil information. An electronic device characterized by the following features.

8. The aforementioned gaze information is information relating to the gaze of a user wearing an optical see-through display device on their head. The control means, during the calibration, (1) Control the transmittance of the display device to the maximum, (2) Control the transmittance of the display device to the minimum, (3) Based on the first pupil information acquired by the acquisition means in the first state in which the transmittance of the display device is controlled to the maximum, and the second pupil information acquired by the acquisition means in the second state in which the transmittance of the display device is controlled to the minimum, the transmittance of the display device is controlled to a transmittance such that the size of the pupil is intermediate between the size indicated by the first pupil information and the size indicated by the second pupil information. The electronic device according to feature 7.

9. A control step that controls the amount of light entering the user's eye, The system includes an acquisition step of acquiring light quantity information indicating the amount of light incident on the user's eye, In the control step, during the calibration of the gaze detection function that obtains gaze information relating to the user's gaze, the amount of light incident on the user's eye is controlled to each of a plurality of light intensities based on the light intensity information. A method for controlling electronic equipment characterized by the following features.

10. A control step for controlling the amount of light incident on the user's eye, The system includes an acquisition step of acquiring pupil information indicating the size of the user's pupil, In the control step, during the calibration of the gaze detection function that obtains gaze information relating to the user's gaze, the amount of light incident on the user's eye is controlled to each of a plurality of light intensities based on the pupil information. A method for controlling electronic equipment characterized by the following features.

11. A program for causing a computer to perform a control step of controlling the amount of light incident on the user's eye, and an acquisition step of acquiring light quantity information indicating the amount of light incident on the user's eye, In the control step, during the calibration of the gaze detection function that obtains gaze information relating to the user's gaze, the amount of light incident on the user's eye is controlled to each of a plurality of light intensities based on the light intensity information. A program characterized by the following features.

12. A program for causing a computer to perform a control step of controlling the amount of light incident on the user's eye and an acquisition step of acquiring pupil information indicating the size of the user's pupil. There is, In the control step, during the calibration of the gaze detection function that obtains gaze information relating to the user's gaze, the amount of light incident on the user's eye is controlled to each of a plurality of light intensities based on the pupil information. A program characterized by the following features.

13. A computer-readable storage medium that stores a program causing a computer to execute a control step of controlling the amount of light incident on the user's eye and an acquisition step of acquiring light quantity information indicating the amount of light incident on the user's eye, In the control step, during the calibration of the gaze detection function that obtains gaze information relating to the user's gaze, the amount of light incident on the user's eye is controlled to each of a plurality of light intensities based on the light intensity information. A storage medium characterized by the following features.

14. A computer-readable storage medium storing a program for causing a computer to perform a control step of controlling the amount of light incident on a user's eye and an acquisition step of acquiring pupil information indicating the size of the user's pupil, In the control step, during the calibration of the gaze detection function that obtains gaze information relating to the user's gaze, the amount of light incident on the user's eye is controlled to each of a plurality of light intensities based on the pupil information. A storage medium characterized by the following features.

Images