electronic machines

Abstract

To provide a technology for acquiring highly accurate information (precise information) as information regarding the gaze of the right and left eyes of a user.SOLUTION: An electronic apparatus is equipped with: first acquisition means for acquiring right gaze information about the gaze of the right eye of a user; second acquisition means for acquiring left gaze information about the gaze of the left eye of the user; and control means for performing right eye calibration to improve the accuracy of the right gaze information to be acquired based on the right gaze information acquired by the first acquisition means and for performing left eye calibration to improve the accuracy of the left gaze information to be acquired based on the left gaze information acquired by the second acquisition means.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to an electronic device capable of performing calibration of gaze detection.

Background Art

[0002] With the spread of HMD (Head-Mounted Display, head-mounted display device), technologies for integrating the real world and the virtual world have been advancing. Such technologies are called XR (Cross Reality), etc., and XR includes VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), etc. In VR, a virtual world is displayed, and in AR and MR, a virtual world or virtual object is superimposed on the real world and displayed.

[0003] In XR, etc., by detecting the gaze position of the user, identifying the object the user is looking at, the user's intention, etc., and performing processing according to the identification result, the user's sense of immersion can be enhanced or the user's convenience can be improved.

[0004] Patent Document 1 discloses a technique for detecting the gaze position of a user when looking at a real object and when looking at a virtual object, and adjusting the display position of the virtual object based on those gaze positions so as to reduce the burden on the user due to the difference in those gaze positions.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] ​However, in technologies such as those described in Patent Document 1, if the user's gaze position and fixation position cannot be detected with high accuracy, it is not possible to suitably perform the processing necessary to obtain the desired effect.

[0007] The present invention aims to provide a technology that enables the acquisition of highly accurate information (accurate information) as information regarding the gaze of the user's right and left eyes. [Means for solving the problem]

[0008] A first aspect of the present invention includes a first acquisition means for acquiring right gaze information relating to the gaze of the user's right eye, a second acquisition means for acquiring left gaze information relating to the gaze of the user's left eye, and a control means for controlling the system to perform right eye calibration based on the right gaze information acquired by the first acquisition means and left eye calibration based on the left gaze information acquired by the second acquisition means. Furthermore, the control means controls the system to perform binocular calibration based on the right gaze information acquired by the first acquisition means and the left gaze information acquired by the second acquisition means, and in the binocular calibration, it determines weight information for at least one of the right gaze information and the left gaze information used when combining the right gaze information acquired by the first acquisition means and the left gaze information acquired by the second acquisition means to obtain the user's gaze position. This is an electronic device characterized by the following features.

[0011] This invention 2 The embodiment includes a first acquisition step of acquiring right gaze information relating to the gaze of the user's right eye, a second acquisition step of acquiring left gaze information relating to the gaze of the user's left eye, and a control step of controlling the system to perform right eye calibration based on the right gaze information acquired in the first acquisition step, and left eye calibration based on the left gaze information acquired in the second acquisition step. Furthermore, in the control step, control is performed to perform binocular calibration based on the right gaze information acquired in the first acquisition step and the left gaze information acquired in the second acquisition step, and in the binocular calibration, weight information of at least one of the right gaze information and the left gaze information is determined to be used when combining the right gaze information acquired in the first acquisition step and the left gaze information acquired in the second acquisition step to obtain the user's gaze position. This is a control method for electronic equipment characterized by the following features.

[0013] This invention 3 One aspect of the present invention is a program for causing a computer to function as one of the means of the electronic device described above. A seventh aspect of the present invention is a computer-readable storage medium that stores a program for causing a computer to function as one of the means of the electronic device described above. [Effects of the Invention]

[0014] According to the present invention, high-precision information (accurate information) can be obtained as information regarding the line-of-sight of the user's right eye and left eye.

Brief Description of the Drawings

[0015] [Figure 1] It is an external view of an HMD (Head-Mounted Display). [Figure 2] It is a block diagram showing the internal configuration of the HMD. [Figure 3] It is a flowchart of a calibration operation and the like according to Example 1. [Figure 4] It is a diagram showing the display during calibration. [Figure 5] It is a diagram for explaining the principle of a method for detecting the line-of-sight of one eye. [Figure 6] It is a diagram showing an eye image. [Figure 7] It is a flowchart of a line-of-sight detection operation for one eye. [Figure 8] It is a diagram showing the visual field. [Figure 9] It is a diagram for explaining the factors causing an error in the line-of-sight position of one eye. [Figure 10] It is a diagram for explaining the factors requiring binocular calibration. [Figure 11] It is a diagram showing the index direction and index distance. [Figure 12] It is a diagram for explaining a part of the fixation position estimation operation. [Figure 13] It is a diagram for explaining the problems to be solved in Example 2. [Figure 14] It is a flowchart of a calibration operation according to Example 2. [Figure 15] It is a diagram for explaining the problems to be solved in Example 3. [Figure 16] It is a flowchart of a calibration operation and the like according to Example 3.

Modes for Carrying Out the Invention

[0016] <Example 1> Embodiment 1 of the present invention will be described. Embodiment 1 describes an example in which the present invention is applied to a video see-through type HMD (head-mounted display, a head-worn display device), but the present invention is also applicable to an optical see-through type HMD. A video see-through type HMD displays an image (virtual space) of the real space (outside world) and, if necessary, synthesizes images of virtual objects. A user wearing a video see-through type HMD (on their head) cannot directly see the real space, but can indirectly see the real space by looking at the displayed image. An optical see-through type HMD has, for example, a lens similar to that of ordinary eyeglasses, and projects an image of a virtual object onto the lens. A user wearing an optical see-through type HMD can directly see the real space through the lens. Furthermore, the user can also see the virtual object projected onto the lens.

[0017] The present invention is applicable to electronic devices other than HMDs. For example, the present invention is applicable to various electronic devices that can be controlled to perform calibration related to gaze detection in an HMD. The present invention is also applicable to PCs (personal computers) and game consoles connected to an HMD. Furthermore, while Example 1 describes an example of applying the present invention to AR (Augmented Reality) and MR (Mixed Reality), the present invention is also applicable to other XR (Cross Reality), such as VR (Virtual Reality). While Example 1 describes an example of detecting the gaze position of a user wearing an HMD, the present invention is not limited to this, and is also applicable, for example, to detecting the gaze position of a user looking at a general display device.

[0018] Figures 1(a) and 1(b) are external views of the HMD 100 according to Embodiment 1. As shown in Figure 1(a), the HMD 100 is equipped with a headband 200. The user places the HMD 100 in contact with their eyes and secures it to their head with the headband 200. The left imaging device 108a is a camera that captures the real world (outside world) for display to the left eye, and the right imaging device 108b is a camera that captures the real world (outside world) for display to the right eye. The image captured by the left imaging device 108a is displayed on the left display panel (not shown), which is a display panel visible through the left eyepiece 102a in Figure 1(b). The image captured by the right imaging device 108b is displayed on the right display panel (not shown), which is a display panel visible through the right eyepiece 102b in Figure 1(b). The user aligns their left eye with the left eyepiece 102a and looks through the left eyepiece 102a at the left display panel with their left eye, then aligns their right eye with the right eyepiece 102b and looks through the right eyepiece 102b at the right display panel with their right eye.

[0019] Figure 2 is a block diagram showing the internal configuration of the HMD100. The left eye 101a is the user's left eye, and the right eye 101b is the user's right eye.

[0020] The left eyepiece 103a is a lens for magnifying the image displayed on the left display panel 107a, and the right eyepiece 103b is a lens for magnifying the image displayed on the right display panel 107b. The user will see these magnified images.

[0021] The left light splitter 104a transmits light from the left display panel 107a (the image displayed on the left display panel 107a) and guides it to the left eyepiece lens 103a. The left light splitter 104a also transmits light from the left eyepiece lens 103a and the left eyepiece section 102a (the light reflected by the left eye 101a, and the light from the left eyepiece section 102a). The right light splitter 104b reflects the subject light (showing 101a) and directs it to the left light receiving lens 105a. The right light splitter 104b transmits the light from the right display panel 107b (the image displayed on the right display panel 107b) and directs it to the right eyepiece lens 103b. The right light splitter 104b also reflects the light from the right eyepiece lens 103b and the right eyepiece section 102b (the subject light showing the right eye 101b, reflected by the right eye 101b) and directs it to the right light receiving lens 105b.

[0022] The left light-receiving lens 105a guides light from the left light splitter 104a to the left image sensor 106a, and the right light-receiving lens 105b guides light from the right light splitter 104b to the right image sensor 106b.

[0023] The left image sensor 106a converts the light incident from the left light-receiving lens 105a into photoelectric power. This captures an image of the left eye 101a. The right image sensor 106b converts the light incident from the right light-receiving lens 105b into photoelectric power. This captures an image of the right eye 101b. The image data from the left eye 101a and the image data from the right eye 101b are sent to the necessary blocks via the bus 109.

[0024] The left display panel 107a is a display unit that displays images captured by the left imaging device 108a. The left display panel 107a may also display an image in which an image of a virtual object is superimposed on the image captured by the left imaging device 108a. When the user is wearing the HMD 100, the left display panel 107a faces the user's left eye 101a. The right display panel 107b is a display unit that displays images captured by the right imaging device 108b. The right display panel 107b may also display an image in which an image of a virtual object is superimposed on the image captured by the right imaging device 108b. When the user is wearing the HMD 100, the right display panel 107b faces the user's right eye 101b. The HMD 100 can perform a three-dimensional display by placing virtual objects in a three-dimensional space (a three-dimensional space centered on the HMD 100 (the user wearing the HMD 100)) using the display for the right eye and the display for the left eye.

[0025] The left imaging device 108a and the right imaging device 108b are positioned with a distance corresponding to the distance between the left and right eyes of a human being. The left imaging device 108a and the right imaging device 108b capture two images with parallax between the left and right eyes.

[0026] Bus 109 is a bus that enables the transmission and reception of data between blocks, allowing data to be sent and received between blocks connected to bus 109.

[0027] The CPU (Central Processing Unit) 110 controls each block of the HMD 100 and performs various processes on the HMD 100. For example, the CPU 110 can detect the user's gaze direction and gaze position using images (images of the user's eyes) captured by the left image sensor 106a and the right image sensor 106b (gaze detection). The gaze direction can be considered as the direction parallel to the user's line of sight, and the gaze position can be considered as the position the user is looking at (the position the user's gaze is directed at). The CPU 110 can also control the object detection unit 113 to detect objects from images (captured images) captured by the left image sensor 108a and the right image sensor 108b (object detection). The CPU 110 can also control the distance detection unit 114 to detect the distance from the HMD 100 (the user wearing the HMD 100) to an object (distance detection).

[0028] The ROM (Read Only Memory) 111 pre-stores programs for processes executed by the CPU 110, as well as information necessary for those processes. The RAM (Random Access Memory) 112 stores video data captured by the left image sensor 106a, the right image sensor 106b, the left imaging device 108a, and the right imaging device 108b. The RAM 112 is also used as work memory to temporarily store data necessary for processing by the CPU 110.

[0029] The object detection unit 113 detects objects present in the image (the user's field of view in three-dimensional space) captured by the left imaging device 108a and the right imaging device 108b (object detection). Through object detection, the object detection unit 113 can obtain information such as the position, type, and size of the detected object.

[0030] The distance detection unit 114 uses the difference between the image captured by the left imaging device 108a and the image captured by the right imaging device 108b to detect (calculate) the distance from the HMD 100 to the detected object (the object detected by the object detection unit 113).

[0031] Figure 3(a) is a flowchart showing the calibration operation of the HMD100 (a series of operations including calibration). The calibration operation in Figure 3(a) may be performed, for example, when the HMD100 is first started up, periodically, or in response to instructions from the user (such as an instruction to start calibration). Here, we assume that the user is wearing the HMD100.

[0032] In step S301 (first timing), the CPU 110 controls the display to show a screen for right-eye calibration (described later). For example, as shown in Figure 4(a), the left display panel 107a is turned off (hidden), and the indicator 401 (an indicator for right-eye calibration, an indicator that the user should focus on) is displayed on the right display panel 107b. This prompts the user to focus on indicator 401 with their right eye. Position 402 is the gaze position (estimated position) of the right eye detected by the gaze detection operation (described later), and does not need to be displayed on the right display panel 107b.

[0033] In step S302, the CPU 110 performs right-eye calibration. Right-eye calibration is a calibration to improve the accuracy of right-eye gaze detection (gaze detection operation). Right-eye calibration can also be considered a calibration to improve the accuracy of the acquired right-eye gaze information (information related to the gaze of the right eye).

[0034] In step S303 (second timing), the CPU 110 controls the display to show a screen for left-eye calibration (described later). For example, as shown in Figure 4(b), the right display panel 107b is turned off (hidden), and the indicator 403 (indicator for left-eye calibration, the indicator the user should focus on) is displayed on the left display panel 107a. This prompts the user to focus on indicator 403 with their left eye. Position 404 is the gaze position (estimated position) of the left eye detected by the gaze detection operation (described later), and does not need to be displayed on the left display panel 107a.

[0035] In step S304, the CPU 110 performs left-eye calibration. Left-eye calibration is a calibration to improve the accuracy of left-eye gaze detection (gaze detection operation). Left-eye calibration can also be considered a calibration to improve the accuracy of the acquired left-eye gaze information (information related to the left eye's gaze).

[0036] Figure 5 is a diagram illustrating the principle of the monocular gaze detection method and is a schematic diagram of the optical system for monocular gaze detection. Figure 6(a) is a schematic diagram of the eye image (optical image projected onto the eye image sensor 117) captured by the eye image sensor 117, and Figure 6(b) is a diagram showing the output intensity of the CCD in the eye image sensor 117. Figure 7 is a flowchart of the monocular gaze detection operation. The eye image sensor 117 is either the left image sensor 106a or the right image sensor 106b.

[0037] When the gaze detection operation starts, in step S701 in Figure 7, the CPU 110 detects the light source. By controlling 118a and 118b, infrared light is emitted toward the user's eyeball 314. The optical image of the eyeball 314 illuminated by the infrared light is formed on the eye image sensor 117 through the light-receiving lens 116 and converted into an electrical signal by the eye image sensor 117. This provides an electrical signal of a processable eye image.

[0038] In step S702, the CPU 110 acquires an eye image (eye image signal, electrical signal of the eye image) from the eye image sensor 117.

[0039] In step S703, the CPU 110 detects the coordinates of the points corresponding to the corneal reflection images Pd and Pe of the light sources 118a and 118b and the pupil center c from the eye image obtained in step S702.

[0040] Infrared light emitted from light sources 118a and 118b illuminates the cornea 342 of the user's eyeball 314. At this time, the corneal reflection images Pd and Pe, formed by a portion of the infrared light reflected from the surface of the cornea 342, are focused by the light-receiving lens 116 and imaged onto the eye image sensor 117 to become the corneal reflection images Pd' and Pe' in the eye image. Similarly, the light beams from the ends a and b of the pupil 341 are also imaged onto the eye image sensor 117 to become the pupil end images a' and b' in the eye image.

[0041] Figure 6(b) shows the luminance information (luminance distribution) of region α in the eye image of Figure 6(a). In Figure 6(b), the horizontal direction of the eye image is the X-axis direction and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In Example 1, the coordinates in the X-axis direction (horizontal direction) of the corneal reflection images Pd',Pe' are Xd,Xe, and the coordinates in the X-axis direction of the pupil end images a',b' are Xa,Xb. As shown in Figure 6(b), an extremely high level of luminance is obtained at coordinates Xd,Xe of the corneal reflection images Pd',Pe'. In the region from coordinate Xa to coordinate Xb, which corresponds to the pupil 341 region (the region of the pupil image obtained when the light beam from the pupil 341 is imaged on the eye image sensor 117), an extremely low level of luminance is obtained except for coordinates Xd,Xe. Furthermore, in the region of the iris 343 outside the pupil 341 (the region of the iris image outside the pupil image obtained by imaging the light beam from the iris 343), an intermediate brightness between the two brightness levels described above is obtained. For example, an intermediate brightness between the two brightness levels described above is obtained in the region where the X-coordinate (coordinate in the X-axis direction) is greater than coordinate Xa and in the region where the X-coordinate is less than coordinate Xb.

[0042] From the luminance distribution shown in Figure 6(b), the X coordinates Xd and Xe of the corneal reflection images Pd' and Pe', and the X coordinates Xa and Xb of the pupil end images a' and b' can be obtained. For example, coordinates with extremely high luminance can be obtained as the coordinates of the corneal reflection images Pd' and Pe', and coordinates with extremely low luminance can be obtained as the coordinates of the pupil end images a' and b'. Also, when the rotation angle θx of the optical axis of the eyeball 314 with respect to the optical axis of the light-receiving lens 116 is small, the coordinate Xc of the pupil center image c' (center of the pupil image) obtained by imaging the light beam from the pupil center c onto the eye image sensor 117 can be expressed as Xc ≈ (Xa + Xb) / 2. In other words, the coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa and Xb of the pupil end images a' and b'. In this way, the coordinates of the corneal reflection images Pd',Pe' and the coordinates of the pupillary central image c' can be estimated.

[0043] In step S704, the CPU 110 calculates the imaging magnification β of the eye image. The imaging magnification β is determined by the position of the eyeball 314 relative to the light-receiving lens 116, and can be calculated using a function of the interval (Xd-Xe) between the corneal reflection images Pd',Pe'.

[0044] In step S705, the CPU 110 calculates the rotation angle of the optical axis of the eyeball 314 relative to the optical axis of the light-receiving lens 116. The X-coordinate of the midpoint of the corneal reflection image Pd and the corneal reflection image Pe is approximately the same as the X-coordinate of the center of curvature O of the cornea 342. Therefore, if we take Oc as the standard distance from the center of curvature O of the cornea 342 to the center c of the pupil 341, the rotation angle θx of the eyeball 314 in the ZX plane (a plane perpendicular to the Y axis) can be calculated using the following equation 1. The rotation angle θy of the eyeball 314 in the plane can also be calculated using the same method as the method for calculating the rotation angle θx. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc (Formula 1)

[0045] In step S706, the CPU 110 estimates the user's gaze position using the rotation angles θx and θy calculated in step S705. In the left eye gaze detection operation, the gaze position of the left eye on the display surface of the left display panel 107a is estimated, and in the right eye gaze detection operation, the gaze position of the right eye on the display surface of the right display panel 107b is estimated. Assuming that the coordinates of the gaze position (Hx, Hy) correspond to the coordinates of the pupil center c, the coordinates of the gaze position (Hx, Hy) can be calculated using the following equations 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0046] The parameter m in equations 2 and 3 is a conversion coefficient that transforms the rotation angles θx and θy into coordinates corresponding to the pupil center c on the display surface. Parameter m is assumed to be predetermined and stored in ROM 111. Parameters Ax, Bx, Ay, and By are gaze correction parameters that compensate for individual differences in gaze, and are determined (acquired) by performing gaze detection calibration. The gaze correction parameters Ax, Bx, Ay, and By are assumed to be stored in RAM 112 before the gaze detection operation begins.

[0047] In step S707, the CPU 110 stores the coordinates (Hx, Hy) of the gaze position in the RAM 112 and ends the gaze detection operation.

[0048] The gaze detection method is not limited to the method described above; any method that obtains gaze information from an eye image may be used. The final gaze information may not indicate the gaze position, but rather the direction of the gaze. For example, processing may be performed to obtain the rotation angle (Ax×θx+Bx or Ay×θy+By) without obtaining the coordinates (Hx,Hy) of the gaze position.

[0049] As mentioned above, in the gaze detection operation, the rotation angles θx and θy of the eyeball 314 are obtained from the eye image, and the gaze position can be estimated by transforming the position of the pupil center c to a position on the display surface. Figure 8(a) shows the user's field of view (the range that the user can see). In Figure 8(a), position A, which is the same as the actual gaze position, is estimated as the gaze position.

[0050] However, due to factors such as individual differences in the shape of human eyeballs, it may not be possible to estimate the gaze position with high accuracy (i.e., to obtain high-accuracy gaze information). For example, if the gaze correction parameters Ax, Ay, Bx, and By are not adjusted to values ​​suitable for the user, a discrepancy will occur between the actual gaze position B and the estimated gaze position C, as shown in Figure 8(b). In Figure 8(b), the user is fixated on the person, but it is incorrectly estimated that they are fixated on the background.

[0051] Therefore, before normal use of the HMD100, it is necessary to perform gaze detection calibration to determine the appropriate gaze correction parameters Ax, Ay, Bx, and By for the user, and store them in the HMD100.

[0052] Traditionally, calibration has been performed by displaying multiple indicators at different positions, as shown in Figure 8(c), and having the user look at these indicators. A known technique involves performing gaze detection and determining appropriate gaze correction parameters Ax, Ay, Bx, and By for the user based on the calculated multiple gaze positions (estimated positions) and the coordinates of each indicator. Such calibration is performed in steps S302 and S304. For example, the CPU 110 acquires right gaze information in step S301 and performs right eye calibration based on the right gaze information in step S302. Similarly, the CPU 110 acquires left gaze information in step S301 and performs left eye calibration based on the left gaze information in step S302.

[0053] This section explains the factors that cause errors in the line of sight of one eye (the factors that necessitate the calibration mentioned above). Factors that cause errors in the line of sight of one eye include, for example, the size of the eyeball and the position of the photoreceptor cells.

[0054] This section explains the factors affecting eyeball size. Eyeball size varies from person to person. Furthermore, even in the same person, the size of the right and left eyes may differ. As shown in Figure 9(a), the eyeball 901 can rotate around the eyeball center 903. When detecting the line of sight, for example, the position of the pupil 902 is determined by detecting the light emitted from the LED (light-emitting diode) 904 and reflected by the eyeball 901 with an image sensor. Image 905 in Figure 9(b) is an eye image of a person with small eyeballs, and image 909 in Figure 9(c) is an eye image of a person with large eyeballs. Figures 9(b) and 9(c) show the movement of the pupil when the eyeball rotates by the same angle. Position 907 in Figure 9(b) and position 911 in Figure 9(c) are the positions of the pupil before the eyeball is rotated and correspond to the same line of sight direction. Because the eyeball size is different in Figure 9(b) and Figure 9(c), the amount of pupil movement on the eye image is different even though the rotation angle of the eyeball is the same. The distance 912 from position 911 to pupil 910 in Figure 9(c) is longer than the distance 908 from position 907 to pupil 906 in Figure 9(b). To suppress the shift in gaze position (estimated position) due to such differences, the calibration described above is necessary.

[0055] This section explains the factors affecting the position of photoreceptor cells. The position of photoreceptor cells varies from person to person. Furthermore, even in the same person, the position of photoreceptor cells differs between the right and left eyes. As shown in Figure 9(d), photoreceptor cells that detect light entering from the iris are located on the opposite side of the pupil 902 of the eyeball 901. Image 916 in Figure 9(e) is an eye image of a person whose photoreceptor cells are at position 913, and image 918 in Figure 9(f) is an eye image of a person whose photoreceptor cells are at position 914, a distance 915 from position 913, and both correspond to the same line of sight direction. Because the position of the photoreceptor cells differs between Figure 9(e) and Figure 9(f), the position of the pupils in the eye images differs despite the same line of sight direction (pupils 917, 919). The calibration described above is necessary to suppress the shift in line of sight position (estimated position) due to such differences.

[0056] Returning to the explanation of Figure 3(a), in step S305 (third timing), the CPU 110 controls the display to show a screen for binocular calibration (described later). For example, as shown in Figure 4(c), indicator 406 is displayed on the left display panel 107a and indicator 409 is displayed on the right display panel 107b. This prompts the user to fixate on indicators 406 and 409 with both eyes. Indicators 406 and 409 are indicators corresponding to a point in three-dimensional space (indicators for binocular calibration, indicators that the user should fixate on) and are displayed with parallax. In Figure 4(c), indicator 406 is displayed at a distance of 405 to the right of the center of the left display panel 107a, and indicator 409 is displayed at a distance of 408 to the left of the center of the right display panel 107b. Position 407 is the left eye's gaze position (estimated position) detected by the left eye's gaze detection operation and does not need to be displayed on the left display panel 107a. Similarly, position 410 is the gaze position (estimated position) of the right eye detected by the gaze detection operation of the right eye, and does not necessarily have to be displayed on the right display panel 107b.

[0057] This section explains the factors that necessitate binocular calibration. Binocular calibration is a calibration procedure designed to improve the accuracy of estimating the user's gaze position (with both eyes).

[0058] Figures 10(a) and 10(b) show an ideal state in which the lines of sight of the right eye (right eye line of sight) and the lines of sight of the left eye (left eye line of sight) accurately traverse an object (a single point) in three-dimensional space.

[0059] Figure 10(a) is a schematic diagram showing the relationship between indicator 1001 and user 1003. Indicator 1001 is located at a distance of 1002 from user 1003, and the left eye line of sight 1004 and the right eye line of sight 1005 are pointing at indicator 1001.

[0060] Figure 10(b) shows the display screen and gaze position (estimated position, gaze detection result) in the state shown in Figure 10(a). The left display panel 107a displays display screen 1006, which includes index 1007 corresponding to index 1001, and the right display panel 107b displays display screen 1010, which includes index 1011 corresponding to index 1001. Indicators 1007 and 1011 are displayed with parallax corresponding to the distance 1002. Indicator 1007 is displayed at a distance of 1008 to the right of the center of the left display panel 107a (display screen 1006), and index 1011 is displayed at a distance of 1012 to the left of the center of the right display panel 107b (display screen 1010). Since left eye calibration has been completed, the gaze position 1009 of the left eye coincides with the position of index 1007. Similarly, since the right eye calibration has been completed, the line of sight position 1013 of the right eye coincides with the position of index 1011.

[0061] Display screens 1006 and 1010 are combined in the user's brain to form the user's visual field image 1014. In the user's visual field image 1014, index 1015, which corresponds to index 1001, is located in the center (towards the center). The user can also perceive the distance to index 1015 (distance 1002) from the parallax between index 1007 and index 1011. By combining the gaze position of the left eye 1009 and the gaze position of the right eye 1013, the user's (bilateral) fixation position 1016 is estimated. The fixation position 1016 is, for example, the average position of gaze position 1009 (the relative position of gaze position 1009 to display screen 1006) and gaze position 1013 (the relative position of gaze position 1013 to display screen 1010). The fixation position 1016 coincides with the position of index 1015.

[0062] However, the lines of sight of the right and left eyes may not be able to accurately target an object (a single point) in three-dimensional space. For example, due to the influence of the characteristics of the eye muscles or the dominant eye, users may not be able to set the convergence angle between the lines of sight of the right and left eyes so that the lines of sight of the right and left eyes accurately target an object. Figures 10(c) and 10(d) illustrate such a situation.

[0063] Figure 10(c) is a schematic diagram showing the relationship between indicator 1001 and user 1003. Indicator 1001 is located at a distance of 1002 from user 1003. Neither the left eye line of sight 1017 nor the right eye line of sight 1018 can hit indicator 1001. Here, we assume that user 1003's dominant eye is the left eye. Therefore, the left eye line of sight 1017 is pointed closer to indicator 1001 than the right eye line of sight 1018.

[0064] Figure 10(d) shows the display screen and gaze position (estimated position, gaze detection result) in the state shown in Figure 10(c). The left display panel 107a displays display screen 1019 including index 1020 corresponding to index 1001, and the right display panel 107b displays display screen 1023 including index 1024 corresponding to index 1001. Indicators 1020 and 1024 are displayed with parallax corresponding to the distance 1002. Indicator 1020 is displayed on the left display panel The indicator 1024 is displayed at a distance of 1021 units to the right of the center of the Nell 107a (display screen 1019), and the index 1024 is displayed at a distance of 1025 units to the left of the center of the right display panel 107b (display screen 1023). Since left-eye calibration has been completed, the left eye's line of sight position 1022 can be estimated with high accuracy as a monocular line of sight position. However, as shown in Figure 10(c), the left eye's line of sight 1017 does not hit index 1001, so the line of sight position 1022 is shifted from the position of index 1020. Similarly, since right-eye calibration has been completed, the right eye's line of sight position 1026 can be estimated with high accuracy as a monocular line of sight position. However, as shown in Figure 10(c), the right eye's line of sight 1018 does not hit index 1001, so the line of sight position 1026 is shifted from the position of index 1024. Since the dominant eye is the left eye, the difference (distance) between the left eye's line of sight position 1022 and the position of index 1020 is smaller than the difference between the right eye's line of sight position 1026 and the position of index 1024.

[0065] Display screens 1019 and 1023 are combined in the user's brain to form the user's visual field image 1027. In the user's visual field image 1027, index 1028, which corresponds to index 1001, is located in the center (towards the center). The user can also perceive the distance to index 1028 (distance 1002) from the parallax between index 1020 and index 1024. By combining the gaze position of the left eye 1022 and the gaze position of the right eye 1026, the user's (binocular) gaze position 1029 is estimated. The gaze position 1029 does not coincide with the position of index 1028. Binocular calibration is necessary to suppress such discrepancies (misestion of gaze position).

[0066] Returning to the explanation of Figure 3(a), in step S306, the CPU 110 performs a right eye gaze detection operation and a left eye gaze detection operation, and acquires right eye gaze information (right eye gaze position) and left eye gaze information (left eye gaze position).

[0067] In step S307, the CPU 110 performs binocular calibration based on the right and left eye gaze information acquired in step S306.

[0068] An example of binocular calibration is explained using Figures 10(c) and 10(d). If the average position of the left eye's gaze position 1022 and the right eye's gaze position 1026 is determined (estimated) as the fixation position, a position that deviates from the index 1028, such as fixation position 1029, will be determined as the fixation position. By combining gaze positions 1022 and 1026 with appropriate weights, a position closer to the index 1028 can be determined as the fixation position. In binocular calibration, for example, when combining right gaze information and left gaze information to obtain the fixation position, the weight (weight information, weight coefficient) of at least one of the right gaze information and left gaze information is determined.

[0069] Assume that in step S306, the gaze position of the left eye 1022 and the gaze position of the right eye 1026 are obtained. Then, assume that the position of index 1028 is the user's actual gaze position (a predetermined gaze position). In this case, using the X coordinate (horizontal position) of index 1028 as XI, the X coordinate (horizontal position) of gaze position 1022 as XL, and the X coordinate of gaze position 1026 as XR, a suitable weight Z can be calculated from the following equation 4. Z×XL+(1-Z)×XR=XI (Formula 4)

[0070] According to Equation 4, weight information is determined based on the gaze position corresponding to the right gaze information and the gaze position corresponding to the left gaze information, and the right gaze information and the left gaze information are combined to obtain a predetermined gaze position. The CPU 110 stores the weight information determined based on Equation 4 in the RAM 112, associating it with the index direction and the index distance. In the cases of Figures 10(c) and 10(d), the index position is the direction of index 1028, and the index distance is distance 1002.

[0071] Let's return to the explanation of Figure 3(a). In Example 1, multiple indicator distances (multiple depth positions as seen from the user) are predetermined, and for each of the multiple indicator distances, multiple indicator directions (multiple directions as seen from the user) are predetermined. In step S305 described above, an indicator corresponding to one of the multiple indicator distances and one of the multiple indicator directions is displayed.

[0072] In step S308, the CPU 110 determines whether or not binocular calibration has been performed for all indicator directions corresponding to the current indicator distance. This determination can also be interpreted as determining whether or not all of the multiple indicators (multiple indicators with different indicator directions) corresponding to the current indicator distance have been displayed. If the CPU 110 determines that binocular calibration has been performed for all indicator directions, it proceeds to step S310; if it determines that there are still indicator directions for which binocular calibration has not been performed, it proceeds to step S309.

[0073] In step S309, the CPU 110 changes (updates) the indicator direction setting. Then, the process returns to step S305, where the indicator corresponding to the previous indicator direction is hidden, and the indicator corresponding to the new indicator direction is displayed. This can also be seen as a change in the display position of the indicator.

[0074] In step S310, the CPU 110 determines whether or not binocular calibration has been performed for all indicator distances. This determination can also be interpreted as determining whether or not all indicators (multiple indicators with different indicator distances and multiple indicators with different indicator directions) have been displayed. If the CPU 110 determines that binocular calibration has been performed for all indicator distances, it terminates the calibration operation shown in Figure 3(a). If it determines that there are still indicator distances for which binocular calibration has not been performed, it proceeds to step S311.

[0075] In step S311, the CPU 110 changes (updates) the indicator distance setting. Then, the process returns to step S305, where the indicator corresponding to the indicator distance before the change is hidden, and the indicator corresponding to the indicator distance after the change is displayed. This can also be seen as a change in the display position of the indicator.

[0076] An example of the processing in steps S308 to S311 will be explained using Figure 11. In the example in Figure 11, three indicator distances are predetermined based on the user 1110: 50 cm, 2 m, and infinity. For the 50 cm indicator distance, three indicator directions are predetermined: 0 degrees, 20 degrees, and 50 degrees, and three indicators C, F, and G corresponding to the three indicator directions are displayed. For the 2 m indicator distance, two indicator directions are predetermined: 0 degrees and 20 degrees, and two indicators B and E corresponding to the two indicator directions are displayed. For the infinity indicator distance, two indicator directions are predetermined: 0 degrees and 20 degrees, and two indicators A and D corresponding to the two indicator directions are displayed.

[0077] In step S309, the change from 0 degrees to 20 degrees (→ 50 degrees) in Figure 11 is performed. In step S311, the change from 50 cm to 2 m to infinity in Figure 11 is performed. For example, if the indicator distance is 50 cm, the change from 0 degrees to 20 degrees to 50 degrees is performed in step S309, and the displayed indicator changes from C to F to G. If the indicator distance is changed from 50 cm to 2 m in step S311, the change from 0 degrees to 20 degrees is performed in step S309, and the displayed indicator changes from B to E. If the indicator distance is changed from 2 m to infinity in step S311, the change from 0 degrees to 20 degrees is performed in step S309, and the displayed indicator changes from A to D. In the case of infinity, the indicator on the left display panel 107a and the indicator on the right display panel 107b are displayed without parallax. The indicator distances and indicator directions shown in Figure 11 are examples. The number of indicator distances (resolution) and the number of indicator directions (resolution) are not particularly limited. For example, the number of indicator distances and indicator directions may be increased from the numbers shown in Figure 11.

[0078] The calibration process has been described above, but either steps S308 and S309 or steps S310 and S311 may be omitted. Right eye calibration and left eye calibration may be performed, while binocular calibration may be omitted. Right eye calibration and left eye calibration may be omitted, and binocular calibration may be performed.

[0079] Figure 3(b) is a flowchart showing the gaze position estimation operation of the HMD100. The gaze position estimation operation in Figure 3(b) is performed after the calibration operation in Figure 3(a). Here, it is assumed that the user is wearing the HMD100.

[0080] In step S321, the CPU 110 performs a right eye gaze detection operation and a left eye gaze detection operation, and acquires right eye gaze information (right eye gaze position) and left eye gaze information (left eye gaze position).

[0081] In steps S322 to S324, the CPU 110 obtains the results of binocular calibration corresponding to the right and left gaze information acquired in step S321.

[0082] In step S322, the CPU 110 identifies the approximate range (field of view) that the user is looking at, based on the right eye gaze information (right eye gaze position) and left eye gaze information (left eye gaze position) acquired in step S321. For example, multiple ranges based on the HMD 100 (user) are predetermined, and the CPU 110 identifies (determines) the area that includes the midpoint between the detection position of the right eye and the detection position of the left eye as the approximate range that the user is looking at.

[0083] In step S323, the CPU 110 controls the object detection unit 113 to detect objects within the range identified in step S322, and controls the distance detection unit 114 to detect the distance from the HMD 100 (user) to the object (acquisition of distance information).

[0084] An example of the processing in steps S322 and S323 will be explained using Figure 12. In the example in Figure 12, the left display panel 107a displays a display screen 1201 including object 1202, and the right display panel 107b displays a display screen 1205 including object 1206. Objects 1202 and 1206 correspond to the same object 1210 and are displayed with parallax. Object 1202 is displayed at a distance of 1203 to the right of the center of the left display panel 107a (display screen 1201), and object 1206 is displayed at a distance of 1207 to the left of the center of the right display panel 107b (display screen 1205). In the user's field of view image 1209, object 1210 is located in the center (towards the center). The left eye's line of sight position 1204 and the right eye's line of sight position 1208 are detected. Neither the left eye's line of sight nor the right eye's line of sight accurately targets object 1210. Therefore, the line of sight position 1204 is shifted from the position of object 1202 (a distance of 1203 to the right of the center of the display screen 1201). Similarly, the line of sight position 1208 is shifted from the position of object 1206 (a distance of 1207 to the left of the center of the display screen 1205). In step S322, a region 1212 is identified that includes the midpoint 1211 of the line of sight positions 1204 and 1208. In step S323, the distance to object 1210 located in region 1212 is detected.

[0085] Returning to the explanation of Figure 3(b), in step S324, the CPU 110 acquires weight information (weights) corresponding to the range (position within the range, indicator direction) identified in step S322 and the distance (indicator distance) detected in step S323 as a result of binocular calibration. For example, the CPU 110 obtains weight information in which the combination of index direction and index distance is closest to the combination of the direction within the range identified in step S322 and the distance detected in step S323.

[0086] In step S325, the CPU 110 estimates the user's gaze position based on the right gaze information (right eye position) acquired in step S321, the left gaze information (left eye position) acquired in step S321, and the weight information acquired in step S324. For example, the CPU 110 uses the following equation 5 to estimate the X-coordinate XR of the right eye position and the X-coordinate X of the left eye position. L From the weight Z, the X-coordinate XT of the gaze position is calculated. The CPU 110 may estimate (determine) the Y-coordinate of the gaze position as the average of the Y-coordinate (vertical coordinate) of the right eye's line of sight position and the Y-coordinate of the left eye's line of sight position, or it may estimate the Y-coordinate in the same way as the method for estimating the X-coordinate XT. Alternatively, the Y-coordinate of the right eye's line of sight position or the Y-coordinate of the left eye's line of sight position may be estimated as the Y-coordinate of the gaze position. XT=Z×XL+(1-Z)×XR (Formula 5)

[0087] As described above, according to Example 1, by performing calibration related to gaze detection, highly accurate information (accurate information) regarding the gaze of the user's right and left eyes can be obtained. For example, by performing right-eye calibration, left-eye calibration, or binocular calibration, the gaze position of the right eye, the gaze position of the left eye, and the gaze position of both eyes can be estimated with high accuracy.

[0088] Although an example was described in which the left display panel 107a is turned off during right-eye calibration and the right display panel 107b is turned off during left-eye calibration, this is not the only example. For example, the CPU 110 may be controlled to instruct the user to close their left eye during right-eye calibration and to instruct the user to close their right eye during left-eye calibration. Instructions to the user may be given, for example, by voice or on the screen. However, since closing the left eye changes the degree to which the right eye is open, and closing the right eye changes the degree to which the left eye is open, it is preferable to perform both right-eye and left-eye calibration in the same state as when the HMD 100 is normally used (for example, with both eyes open).

[0089] <Example 2> Embodiment 2 of the present invention will now be described. Note that explanations of the same aspects (configuration and processing) as in Embodiment 1 will be omitted as appropriate.

[0090] The gaze position fluctuates due to unintended eye movements by the user (e.g., micro-movements during fixation). The range of fluctuation in gaze position corresponding to unintended eye movements may differ between the dominant and non-dominant eyes, and such differences reduce the accuracy of gaze position estimation. Example 2 suppresses this decrease in estimation accuracy.

[0091] Figure 13(a) is a schematic diagram showing the relationship between index 1301 and user 1303. Index 1301 is located at a distance of 1302 from user 1303. In Figure 13(a), the left eye line of sight 1304 and the right eye line of sight 1305 are shown to be pointing towards index 1301, but the left eye line of sight 1304 fluctuates within a range 1306, and the right eye line of sight 1305 fluctuates within a wider range 1307 than the range 1306. The centers of the respective ranges 1306 and 1307 coincide with index 1301.

[0092] Figure 13(b) shows the display screen and gaze position (estimated position, gaze detection result) in the state shown in Figure 13(a). The left display panel 107a shows index 1309 corresponding to index 1301. Display screen 1308 is shown, and display screen 1313, which includes index 1314 corresponding to index 1301, is displayed on the right display panel 107b. Indicators 1309 and 1314 are displayed with parallax corresponding to the distance 1302. Indicator 1309 is displayed at a distance of 1310 to the right of the center of the left display panel 107a (display screen 1308), and index 1314 is displayed at a distance of 1315 to the left of the center of the right display panel 107b (display screen 1313). Since left eye calibration has been completed, the gaze position 1311 of the left eye can be estimated with high accuracy. The gaze position 1311 is shown to coincide with the position of index 1309, but it fluctuates within the variation range 1312. The center of the variation range 1312 coincides with index 1309. Similarly, since right eye calibration has been completed, the gaze position 1316 of the right eye can be estimated with high accuracy. The line of sight position 1316 is shown to coincide with the position of index 1314, but it fluctuates within a wider range of variation 1317 than the range of variation 1312. The center of the range of variation 1317 coincides with index 1314.

[0093] Display screens 1308 and 1313 are combined in the user's brain to form the user's field of view image 1318. In the user's field of view image 1318, index 1319, which corresponds to index 1301, is located in the center (towards the center). The user can also perceive the distance to index 1319 (distance 1302) from the parallax between index 1309 and index 1314. In Example 2, the decrease in the accuracy of gaze position estimation due to the difference between the fluctuation range 1312 and the fluctuation range 1317 is suppressed, and the gaze position 1320 that coincides with the position of index 1319 is estimated.

[0094] Figure 14 is a flowchart of the calibration operation according to Example 2. Steps S1401 to S1405 are the same as steps S301 to S305 in Figure 3(a), and steps S1408 to S1411 are the same as steps S308 to S311 in Figure 3(a).

[0095] In step S1406, the CPU 110 performs a right eye gaze detection operation and a left eye gaze detection operation for a predetermined time, and acquires time-series information (time-series data) that shows the time change of the right eye gaze information (right eye gaze position) and the left eye gaze information (left eye gaze position).

[0096] In step S1407, the CPU 110 performs binocular calibration based on the time-series information acquired in step S1406. For example, the CPU 110 identifies the right fluctuation range (fluctuation range of the right eye's gaze position) based on the time change of the right gaze information, and identifies the left fluctuation range (fluctuation range of the left eye's gaze position) based on the time change of the left gaze information. The CPU 110 then considers the reciprocal of the size of the fluctuation range as the confidence level of the gaze position, and determines weights from the size of the right fluctuation range and the size of the left fluctuation range. For example, the CPU 110 can calculate the weight Z of the left eye's gaze position from the reciprocal α of the size of the left fluctuation range and the reciprocal β of the size of the right fluctuation range using the following equation 6. The size of the fluctuation range is, for example, the number of pixels in the fluctuation range. Z = α / (α+β) ... (Equation 6)

[0097] As described above, according to Example 2, the weight (weight information) is determined based on the size of the right fluctuation range and the size of the left fluctuation range. This makes it possible to suppress the decrease in the accuracy of estimating the gaze position due to the difference between the right fluctuation range and the left fluctuation range.

[0098] <Example 3> Embodiment 3 of the present invention will now be described. Embodiment 3 combines Embodiments 1 and 2. Note that explanations of aspects (configuration and processing) that are the same as those in Embodiments 1 and 2 will be omitted as appropriate.

[0099] Figure 15(a) is a schematic diagram showing the relationship between index 1501 and user 1503. Index 1501 is located at a distance of 1502 from user 1503. Neither the left eye line of sight 1504 (line of sight passing through the center of the fluctuation range 1506) nor the right eye line of sight 1505 (line of sight passing through the center of the fluctuation range 1507) can hit index 1501. Here, we assume that user 1503's dominant eye is the left eye. Therefore, the left eye line of sight 1504 is pointed closer to index 1501 than the right eye line of sight 1505. The left eye line of sight 1504 fluctuates within the fluctuation range 1506, while the right eye line of sight 1505 fluctuates within a wider fluctuation range 1507 than the fluctuation range 1506.

[0100] Figure 15(b) shows the display screen and gaze position (estimated position, gaze detection result) in the state shown in Figure 15(a). The left display panel 107a displays display screen 1508, which includes index 1509 corresponding to index 1501, and the right display panel 107b displays display screen 1513, which includes index 1514 corresponding to index 1501. Indicators 1509 and 1514 are displayed with parallax corresponding to the distance 1502. Indicator 1509 is displayed at a distance of 1510 to the right of the center of the left display panel 107a (display screen 1508), and index 1514 is displayed at a distance of 1515 to the left of the center of the right display panel 107b (display screen 1513). Since left-eye calibration has been completed, the gaze position 1511 of the left eye (center of the variation range 1512) can be estimated with high accuracy as a gaze position of one eye. However, as shown in Figure 15(a), the left eye's line of sight 1504 does not hit the target 1501, so the line of sight position 1511 is shifted from the position of the target 1509. Similarly, since the right eye calibration has been completed, the right eye's line of sight position 1516 (center of the variation range 1517) can be estimated with high accuracy for a single eye's line of sight position. However, as shown in Figure 15(a), the right eye's line of sight 1505 does not hit the target 1501, so the line of sight position 1516 is shifted from the position of the target 1514. Since the dominant eye is the left eye, the amount of shift (distance) between the left eye's line of sight position 1511 and the position of the target 1509 is smaller than the amount of shift between the right eye's line of sight position 1516 and the position of the target 1514. The left eye's line of sight position 1511 fluctuates within the variation range 1512, and the right eye's line of sight position 1516 fluctuates within a wider variation range 1517 than the variation range 1512.

[0101] Display screens 1508 and 1513 are combined in the user's brain to form the user's field of view image 1518. In the user's field of view image 1518, index 1519, which corresponds to index 1501, is located in the center (towards the center). The user can also perceive the distance to index 1519 (distance 1502) from the parallax between index 1509 and index 1514. Similar to Example 1 (Figure 10(d)), the gaze position 1520, which is a combination of the left eye's line of sight position 1511 and the right eye's line of sight position 1516, does not coincide with the position of index 1519. Example 3 suppresses such discrepancies (misestion of gaze position). Also, similar to Example 2, it suppresses the decrease in the accuracy of gaze position estimation due to the difference between the variation range 1512 and the variation range 1517.

[0102] Figure 16(a) is a flowchart of the calibration operation according to Example 3. Steps S1601 to S1605 are the same as steps S301 to S305 in Figure 3(a), step S1606 is the same as step S1406 in Figure 14, and steps S1608 to S1611 are the same as steps S308 to S311 in Figure 3(a).

[0103] In step S1607, the CPU 110 performs binocular calibration based on the time-series information acquired in step S1606. For example, the CPU 110 identifies the right variation range (variation range of the right eye's line of sight position) based on the time change of the right line of sight information, and identifies the left variation range (variation range of the left eye's line of sight position) based on the time change of the left line of sight information. Then, the CPU 110 calculates the weight Z of the left eye's line of sight position from the X coordinate XI of the index, the X coordinate XL' of the position in the left variation range, and the X coordinate XR' of the position in the right variation range using the following equation 7. The position of the variation range is, for example, the centroid position of the variation range. Using a predetermined cutoff frequency... Filter processing or similar methods can be used to identify the location of the fluctuation range from the time change in the line of sight. Z×XL'+(1-Z)×XR'=XI (Formula 7)

[0104] Figure 16(b) is a flowchart illustrating the gaze position estimation operation according to Embodiment 3. In step S1621, the CPU 110 performs right eye gaze detection and left eye gaze detection operations for a predetermined time and acquires time-series information (time-series data) showing the time changes in right gaze information (right eye gaze position) and left gaze information (left eye gaze position). Then, the CPU 110 identifies the position of the right fluctuation range (fluctuation range of the right eye gaze position) based on the time change of the right gaze information, and identifies the position of the left fluctuation range (fluctuation range of the left eye gaze position) based on the time change of the left gaze information. Steps S1622 to S1625 are the same as steps S322 to S325 in Figure 3(b). However, gaze position is read as the position of the fluctuation range.

[0105] As described above, according to Example 3, by combining Example 1 and Example 2, the gaze position can be estimated with higher accuracy than in Examples 1 and 2.

[0106] The above-described embodiments (including modified examples) are merely examples, and configurations obtained by appropriately modifying or changing the above-described configurations within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the above-described configurations are also included in the present invention.

[0107] <Other examples> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0108] This embodiment includes the following configurations, methods, programs, and media. (Composition 1) A first acquisition means for acquiring right-eye gaze information relating to the user's right eye gaze, A second acquisition means for acquiring left-eye gaze information relating to the user's left eye gaze, A control means controls the following: performing right eye calibration to improve the accuracy of the acquired right eye gaze information based on the right eye gaze information acquired by the first acquisition means, and performing left eye calibration to improve the accuracy of the acquired left eye gaze information based on the left eye gaze information acquired by the second acquisition means. An electronic device characterized by having the following features. (Configuration 2) While the user is wearing a head-mounted display device, The first display unit of the head-mounted display device is positioned opposite the user's right eye, The second display unit of the head-mounted display device is positioned opposite the user's left eye, The first acquisition means acquires the right gaze information of the user who is looking at the first display unit while the user is wearing the head-mounted display device, The second acquisition means acquires the left gaze information of the user who is looking at the second display unit while the user is wearing the head-mounted display device, The control means is During the right-eye calibration, the second display unit is turned off and the indicator is displayed on the first display unit. During the left-eye calibration, the first display unit is turned off and the indicator is displayed on the second display unit. The electronic device according to configuration 1, characterized by the features described above. (Composition 3) The control means is During the right eye calibration, the system controls the user to be instructed to close their left eye. During the left eye calibration, the system controls the user to be instructed to close their right eye. The electronic device according to configuration 1, characterized by the features described above. (Composition 4) The control means further controls the system to perform binocular calibration to improve the accuracy of estimating the user's gaze position, based on the right gaze information acquired by the first acquisition means and the left gaze information acquired by the second acquisition means. An electronic device according to any one of configurations 1 to 3, characterized by the features described herein. (Composition 5) While the user is wearing a head-mounted display device, The first display unit of the head-mounted display device is positioned opposite the user's right eye, The second display unit of the head-mounted display device is positioned opposite the user's left eye, The first acquisition means acquires the right gaze information of the user who is looking at the first display unit while the user is wearing the head-mounted display device, The second acquisition means acquires the left gaze information of the user who is looking at the second display unit while the user is wearing the head-mounted display device, The control means controls the display of indicators on the first display unit and the second display unit, respectively, during the binocular calibration. The electronic device according to configuration 4, characterized by the features described above. (Composition 6) The control means controls the first display unit and the second display unit to display multiple indicators, each with a different depth position as seen from the user's perspective, during the binocular calibration. The electronic device according to configuration 5, characterized by the features described herein. (Composition 7) The electronic device according to configuration 5 or 6, characterized in that the control means controls the first display unit and the second display unit to display a plurality of indicators that are viewed from the user's perspective during binocular calibration. (Composition 8) In the binocular calibration described above, weight information is determined for at least one of the right gaze information and the left gaze information acquired by the second acquisition means, which is used to obtain the user's gaze position by combining the right gaze information acquired by the first acquisition means and the left gaze information acquired by the second acquisition means. An electronic device according to any one of the configurations 4 to 7, characterized by the features described herein. (Composition 9) In the binocular calibration described above, weight information is determined based on the gaze position corresponding to the right gaze information acquired by the first acquisition means and the gaze position corresponding to the left gaze information acquired by the second acquisition means, by combining the right gaze information and the left gaze information to obtain a predetermined gaze position. The electronic device according to configuration 8, characterized by the above. (Composition 10) In the binocular calibration described above, the weight information is determined based on the size of the variation range of the gaze position corresponding to the right gaze information acquired by the first acquisition means and the size of the variation range of the gaze position corresponding to the left gaze information acquired by the second acquisition means. The electronic device according to configuration 8, characterized by the above. (Composition 11) In the binocular calibration described above, the right gaze information acquired by the first acquisition means is used in relation to the right gaze information acquired by the first acquisition means. The weight information is determined based on the position of the range of variation of the corresponding gaze position and the position of the range of variation of the gaze position corresponding to the left gaze information acquired by the second acquisition means. The electronic device according to configuration 8, characterized by the above. (Composition 12) A first acquisition means for acquiring right-eye gaze information relating to the user's right eye gaze, A second acquisition means for acquiring left-eye gaze information relating to the user's left eye gaze, A third acquisition means for acquiring the results of binocular calibration to improve the estimation accuracy of the user's gaze position, corresponding to the right gaze information acquired by the first acquisition means and the left gaze information acquired by the second acquisition means, An estimation means for estimating the user's gaze position based on the right gaze information acquired by the first acquisition means, the left gaze information acquired by the second acquisition means, and the results of the binocular calibration acquired by the third acquisition means. An electronic device characterized by having the following features. (Composition 13) The system further includes a fourth acquisition means for acquiring distance information to objects located within the range based on the right-line-of-sight information acquired by the first acquisition means and the left-line-of-sight information acquired by the second acquisition means from the user. The third acquisition means acquires information corresponding to the range and distance information as a result of the binocular calibration. The electronic device according to configuration 12, characterized by the features described above. (Composition 14) The estimation means estimates the user's gaze position based on the position of the range of variation of the gaze position corresponding to the right gaze information acquired by the first acquisition means, the position of the range of variation of the gaze position corresponding to the left gaze information acquired by the second acquisition means, and the result of the binocular calibration acquired by the third acquisition means. The electronic device according to configuration 12 or 13, characterized by the features described herein. (Composition 15) It has control means for controlling the calibration of gaze detection in a head-mounted display device, While the user is wearing the head-mounted display device, The first display unit of the head-mounted display device is positioned opposite the user's right eye, The second display unit of the head-mounted display device is positioned opposite the user's left eye, The control means is At a first timing of the series of operations including the calibration, the second display unit is controlled to turn off and the indicator is displayed on the first display unit. At a second timing of the series of operations including the calibration, the first display unit is controlled to turn off and the indicator is displayed on the second display unit. An electronic device characterized by the following features. (Method 1) The first acquisition step involves obtaining right-eye gaze information regarding the user's right eye's gaze, A second acquisition step involves acquiring left-eye gaze information relating to the user's left eye gaze, A control step that controls the following: performing right eye calibration to improve the accuracy of the acquired right eye gaze information based on the right eye gaze information acquired in the first acquisition step, and performing left eye calibration to improve the accuracy of the acquired left eye gaze information based on the left eye gaze information acquired in the second acquisition step. A method for controlling electronic equipment, characterized by having the following features. (Method 2) The first acquisition step involves obtaining right-eye gaze information regarding the user's right eye's gaze, A second acquisition step involves acquiring left-eye gaze information relating to the user's left eye gaze, The right gaze information acquired in the first acquisition step and the information acquired in the second acquisition step A third acquisition step involves obtaining the results of binocular calibration to improve the estimation accuracy of the user's gaze position, corresponding to the obtained left gaze information, An estimation step is performed to estimate the user's gaze position based on the right gaze information obtained in the first acquisition step, the left gaze information obtained in the second acquisition step, and the results of the binocular calibration obtained in the third acquisition step. A method for controlling electronic equipment, characterized by having the following features. (program) A program for causing a computer to function as one of the electronic devices described in any one of configurations 1 to 15. (medium) A computer-readable storage medium containing a program for causing the computer to function as one of the electronic devices described in any one of items 1 to 15. [Explanation of symbols]

[0109] 100: HMD (Head-Mounted Display) 110: CPU

Claims

1. A first acquisition means for acquiring right-eye gaze information relating to the user's right eye gaze, A second acquisition means for acquiring left-eye gaze information relating to the user's left eye gaze, A control means that controls the system to perform right eye calibration based on right eye gaze information acquired by the first acquisition means, and left eye calibration based on left eye gaze information acquired by the second acquisition means. It has, The control means further controls the system to perform binocular calibration based on the right gaze information acquired by the first acquisition means and the left gaze information acquired by the second acquisition means. In the binocular calibration described above, weight information is determined for at least one of the right gaze information and the left gaze information acquired by the second acquisition means, which is used to obtain the user's gaze position by combining the right gaze information acquired by the first acquisition means and the left gaze information acquired by the second acquisition means. An electronic device characterized by the following features.

2. While the user is wearing a head-mounted display device, The first display unit of the head-mounted display device is positioned opposite the user's right eye, The second display unit of the head-mounted display device is positioned opposite the user's left eye, The first acquisition means acquires the right gaze information of the user who is looking at the first display unit while the user is wearing the head-mounted display device, The second acquisition means acquires the left gaze information of the user who is looking at the second display unit while the user is wearing the head-mounted display device, The control means is During the right-eye calibration, the second display unit is turned off and the indicator is displayed on the first display unit. During the left-eye calibration, the first display unit is turned off and the indicator is displayed on the second display unit. The electronic device according to feature 1.

3. The control means is During the right eye calibration, the system controls the user to be instructed to close their left eye. During the left eye calibration, the system controls the user to be instructed to close their right eye. The electronic device according to feature 1.

4. While the user is wearing a head-mounted display device, The first display unit of the head-mounted display device is positioned opposite the user's right eye, The second display unit of the head-mounted display device is positioned opposite the user's left eye, The first acquisition means acquires the right gaze information of the user who is looking at the first display unit while the user is wearing the head-mounted display device, The second acquisition means acquires the left gaze information of the user who is looking at the second display unit while the user is wearing the head-mounted display device, The control means controls the first display unit and the second display unit to display indicators during the binocular calibration. The electronic device according to feature 1.

5. The control means controls the first display unit and the second display unit to display multiple indicators, each with a different depth position as seen from the user's perspective, during the binocular calibration. The electronic device according to feature 4.

6. The electronic device according to claim 4, characterized in that the control means controls the first display unit and the second display unit to display a plurality of indicators that are in different directions from the user's perspective during binocular calibration.

7. In the binocular calibration described above, weight information is determined based on the gaze position corresponding to the right gaze information acquired by the first acquisition means and the gaze position corresponding to the left gaze information acquired by the second acquisition means, by combining the right gaze information and the left gaze information to obtain a predetermined gaze position. The electronic device according to feature 1.

8. In the binocular calibration described above, the weight information is determined based on the size of the variation range of the gaze position corresponding to the right gaze information acquired by the first acquisition means and the size of the variation range of the gaze position corresponding to the left gaze information acquired by the second acquisition means. The electronic device according to feature 1.

9. In the binocular calibration described above, the weight information is determined based on the position of the range of variation in the gaze position corresponding to the right gaze information acquired by the first acquisition means and the position of the range of variation in the gaze position corresponding to the left gaze information acquired by the second acquisition means. The electronic device according to feature 1.

10. The system further comprises estimation means for estimating the user's gaze position based on the right gaze information acquired by the first acquisition means, the left gaze information acquired by the second acquisition means, and the results of the binocular calibration. The electronic device according to feature 1.

11. The system further includes a third acquisition means for acquiring distance information to objects located within the range based on the right-line-of-sight information acquired by the first acquisition means and the left-line-of-sight information acquired by the second acquisition means from the user. The estimation means uses information corresponding to the range and distance information as a result of the binocular calibration. The electronic device according to feature 10.

12. The estimation means estimates the user's gaze position based on the position of the range of variation of the gaze position corresponding to the right gaze information acquired by the first acquisition means, the position of the range of variation of the gaze position corresponding to the left gaze information acquired by the second acquisition means, and the results of the binocular calibration. The electronic device according to feature 10.

13. The first acquisition step involves obtaining right-eye gaze information regarding the user's right eye's gaze, A second acquisition step involves acquiring left-eye gaze information relating to the user's left eye gaze, A control step that controls the system to perform right eye calibration based on the right gaze information acquired in the first acquisition step, and left eye calibration based on the left gaze information acquired in the second acquisition step. It has, In the control step, further control is performed to perform binocular calibration based on the right gaze information acquired in the first acquisition step and the left gaze information acquired in the second acquisition step. In the binocular calibration described above, weight information is determined for at least one of the right gaze information and the left gaze information obtained in the second acquisition step, which is used to obtain the user's gaze position by combining the right gaze information obtained in the first acquisition step and the left gaze information obtained in the second acquisition step. A method for controlling electronic equipment characterized by the following features.

14. A program for causing a computer to function as one of the means of an electronic device according to any one of claims 1 to 12.

15. A computer-readable storage medium storing a program for causing a computer to function as one of the means of an electronic device according to any one of claims 1 to 12.

Images