Electronic device, electronic device control method, program, and recording medium

Abstract

To enable an easy GUI operation with high precision.SOLUTION: An electronic device according to the present invention has: control means for performing control to display a GUI on a display surface of a display device that can be detachably mounted on a head of a user; visual line detection means for detecting a visual line position of the user on the basis of a captured image of a user's eye; pupil detection means for detecting a pupil position of the user from the eye image; motion detection means for detecting a motion of the head or a neck of the user; determination means for determining whether or not the user intends to operate the GUI near the visual line position on the basis of the visual line position detected by the visual line detection means, the motion detected by the motion detection means, and a change of the pupil position during the motion; and execution means for executing processing corresponding to the GUI in a case where the determination means determines that the user intends to operate the GUI near the visual line position.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an electronic device, a control method for an electronic device, a program, and a recording medium. [Background technology]

[0002] In recent years, head-mounted displays (HMDs) with a function for detecting a user's gaze, such as eyeglass-type devices using mixed reality (MR) and augmented reality (AR), have become increasingly automated and intelligent. For such HMDs, methods have been proposed that allow users to easily operate operation icons arranged on the display surface in a hands-free manner. For example, in the technology disclosed in Patent Document 1, when a gaze point is detected within a predetermined threshold distance from an operation icon displayed superimposed on the background on the display surface, processing corresponding to the operation icon is performed. In the technology disclosed in Patent Document 2, an operation corresponding to the operation icon (element) at which the user is gazing is determined based on the gaze and neck movement. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2018-530798 [Patent Document 2] International Publication No. 2017 / 038248 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology disclosed in Patent Document 1 malfunctions if the user happens to gaze at an operation icon even if they have no intention of operating it. Also, the technology disclosed in Patent Document 2 makes it easier to perform a desired operation based on the user's line of sight and neck movement, but when multiple operation icons are placed close together, the user cannot determine which operation icon they want to operate, resulting in a malfunction.

[0005] Therefore, an object of the present invention is to enable accurate and simple GUI operation. [Means for solving the problem]

[0006] A first aspect of the present invention is an electronic device characterized by having: a control means for controlling a display device detachable from a user's head to display a GUI; a gaze detection means for detecting the user's gaze position based on an image of the user's eyes; a pupil detection means for detecting the user's pupil position from the image of the eyes; a movement detection means for detecting movement of the user's head or neck; a determination means for determining whether the user intends to operate a GUI near the gaze position based on the gaze position detected by the gaze detection means, the movement detected by the movement detection means, and a change in the pupil position during the movement; and an execution means for executing processing corresponding to the GUI when the determination means determines that the user intends to operate the GUI near the gaze position.

[0009] The present invention 2 This aspect is a control method for an electronic device, characterized by comprising: a control step of controlling a display device detachable from a user's head to display a GUI on a display surface thereof; a gaze detection step of detecting the user's gaze position based on an image of the user's eyes; a pupil detection step of detecting the user's pupil position from the image of the eyes; a movement detection step of detecting a movement of the user's head or neck; a determination step of determining whether the user intends to operate a GUI near the gaze position based on the gaze position detected by the gaze detection step, the movement detected by the movement detection step, and a change in the pupil position during the movement; and an execution step of executing processing corresponding to the GUI when it is determined by the determination step that the user intends to operate the GUI near the gaze position.

[0010] The present invention 3 This aspect is a control method for an electronic device, characterized by comprising: a control step of controlling a display device detachable from a user's head to display a GUI on a display surface thereof; a gaze detection step of detecting the user's gaze position based on an image of the user's eyes; a movement detection step of detecting a movement of the user's head or neck; a determination step of determining whether the user intends to operate a GUI near the gaze position based on the gaze position detected by the gaze detection step, the movement detected by the movement detection step, and a change in the gaze position during the movement; and an execution step of executing processing corresponding to the GUI when it is determined by the determination step that the user intends to operate the GUI near the gaze position.

[0011] The present invention 4 The third aspect is a program for causing the electronic device to function as each of the means described above.

[0012] The present invention 5 The third aspect is a computer-readable recording medium storing a program for causing the electronic device to function as each of the means described above. [Effects of the Invention]

[0013] According to the present invention, accurate and simple GUI operations can be made possible. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram illustrating the configuration of an HMD. [Figure 2] FIG. 1 is a cross-sectional view of an HMD. [Figure 3] FIG. 1 is a diagram for explaining the principle of a visual field detection method. [Figure 4] FIG. 10 shows an eye image. [Figure 5] 10 is a flowchart of a gaze detection operation. [Figure 6]FIG. 1 shows the background seen by a user through an HMD. [Figure 7] 10 is a flowchart of the operation of the HMD. [Figure 8] 10 is a flowchart of a process for acquiring operational intention information based on a gaze position. [Figure 9] 10 is a flowchart of a process for acquiring operational intention information based on head movement. [Figure 10] FIG. 3 is a diagram for explaining an output value of an inertial sensor. [Figure 11] 10 is a flowchart of a process for acquiring operation intention information based on the pupil center position. [Figure 12] FIG. 10 is a diagram showing changes in pupil center position during head movement. [Figure 13] FIG. 10 is a diagram illustrating an example of processing corresponding to an operation icon. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings, but the present invention is not limited to the following embodiments.

[0016] <Configuration> Fig. 1 shows a schematic diagram of the configuration of an HMD 100 (head-mounted display) according to this embodiment. In Fig. 1, the configuration diagram on the left shows the configuration of the HMD 100 as seen from the top of the user's head, and the block diagram on the right shows the functional configuration of the HMD 100. The HMD 100 is a display device that can be attached to and detached from the user's head.

[0017] When the housing 103 of the HMD 100 is worn on the head, the left eye 101 and right eye 102 can observe the real space through a transmissive left-eye display 104 and a transmissive right-eye display 105, respectively. By displaying images such as operation icons on the left-eye display 104 and right-eye display 105, the HMD 100 can superimpose the displayed images onto the real world seen by the user through the left-eye display 104 and right-eye display 105. In other words, the HMD 100 is an optical see-through display (optical see-through HMD).

[0018] The present invention can also be applied to other configurations. For example, a non-transmissive display is used, with eyepieces placed between the eyes and the non-transmissive display. In non-transmissive mode, the non-transmissive display displays an internally stored image (such as a captured video or game image). In transparent mode, the non-transmissive display displays an image captured by the left-eye camera 106 or the right-eye camera 107 so that the real space appears transparent. The non-transmissive display may display an image that combines the internal image with the captured image. The present invention can also be applied to such configurations. In other words, the present invention can also be applied to an HMD that displays an image of a virtual space unrelated to the real space, a video see-through display (a video see-through HMD), an HMD that can selectively perform these operations, and the like.

[0019] The left-eye gaze detector 108 is used to estimate the area on the left-eye display 104 that the user is gazing at. Similarly, the right-eye gaze detector 109 is used to estimate the area on the right-eye display 105 that the user is gazing at.

[0020] The inertial sensor 110 is composed of two sensors: an acceleration sensor that detects translational motion on the X, Y, and Z axes, and a gyro sensor that detects rotational motion around the YPR (yaw, pitch, and roll) axes, as shown in FIG. 10(a). The inertial sensor 110 can comprehensively detect translational motion and rotational motion by associating these two sensors (inertial sensor fusion). Specifically, as shown in FIG. 10(b), angular velocity and acceleration are input to the inertial sensor 110, and angle information (angle and angular velocity) in each of the yaw, pitch, and roll directions is output. In this embodiment, an inertial sensor is applied to the HMD 100, but a configuration in which only a gyro sensor is applied may also be used.

[0021] The movement detection unit 111 detects the movement of the head or neck of the user wearing the HMD 100 on their head (hereinafter referred to as "head movement"). The movement detection unit 111 detects the head movement based on the output value (output result) of the inertial sensor 110 or the output value of the image determination unit 112. The movement detection unit 111 may also detect the head movement based on the output value of a gyro sensor. stomach.

[0022] Image determination unit 112 determines changes in the image captured by left-eye camera 106 or right-eye camera 107. Image determination unit 112 calculates a difference image in the time direction for the image captured by left-eye camera 106 or right-eye camera 107, for example, to calculate a motion vector within the image.

[0023] Display control unit 113 controls so that a GUI (Graphical User Interface) including operation icons is displayed on left-eye display 104 and right-eye display 105. Display control unit 113 can also control so that an image captured by left-eye camera 106 is displayed on left-eye display 104, and can also control so that an image captured by right-eye camera 107 is displayed on right-eye display 105.

[0024] The user can view a range of three-dimensional space (real space or virtual space) that corresponds to the position of the head, along with the GUI displayed on the display surface (display surfaces of the left-eye display 104 and right-eye display 105) of the HMD 100. The GUI is a first GUI whose position on the display surface is fixed, or a second GUI that is arranged in three-dimensional space and whose position on the display surface changes in response to changes in the position of the head.

[0025] The gaze detection unit 114 detects the gaze position of the user on the display surface. For example, the gaze detection unit 114 estimates the area on the left-eye display 104 that the user is gazing at, or estimates the area on the right-eye display 105 that the user is gazing at. The gaze detection method will be described in detail later.

[0026] The HMD 100 also has a function of communicating with an external device and acquiring information about the subject. The HMD 100 may be provided with a GPS (Global Positioning System) module that acquires its own position information using radio waves from artificial satellites.

[0027] Figure 2 is a cross-sectional view of the HMD 100 cut along the YZ plane formed by the Y and Z axes shown in Figure 1, and shows a schematic diagram of the mechanism that performs gaze detection. Note that Figure 2 is a cross-sectional view seen from the user's left eye side, but the right eye side can be considered similar. In Figure 2, housing 103 is the housing of the HMD 100, and the units contained therein are configured as follows:

[0028] The HMD 100 includes a CPU 128 that controls the entire HMD 100, and a memory unit 129 that records video information. The HMD 100 also includes a transmissive left-eye display 104 that is configured with a liquid crystal or the like for displaying video, and a display driver circuit 124 that drives the left-eye display 104.

[0029] In addition, an image sensor 125, an aperture mechanism 126, and a focus mechanism 127 are arranged as mechanisms constituting the left-eye camera 106 that captures images of the outside world, and the left-eye camera 106 can capture images of the subject scene through a light splitter 121.

[0030] The illumination light source 120 is a light source that illuminates the left eye 101 for gaze detection, and is composed of, for example, a plurality of infrared light emitting diodes. An image of the illuminated eyeball of the left eye 101 and an image due to the corneal reflection of the illumination light source 120 (corneal reflection image) are formed by a light receiving lens 122 on an ocular imaging element 123, which is made up of a two-dimensional array of photoelectric elements such as CMOS.

[0031] The light receiving lens 122 positions the pupil of the user's left eye 101 and the ocular image sensor 123 in a complementary imaging relationship. The left-eye gaze detector 108 is composed of the illumination light source 120, the light-receiving lens 122, and the ocular image sensor 123.

[0032] The memory unit 129 is a storage medium (recording medium) having a function of storing image signals from the image sensor 125 and the eye image sensor 123, and a function of storing line-of-sight correction parameters.

[0033] <Gaze detection operation> The gaze detection method will be described using Figures 3, 4(a), 4(b), and 5. Figure 3 is a diagram for explaining the principle of the gaze detection method and is a schematic diagram of an optical system for performing gaze detection. As shown in Figure 3, light sources 13a and 13b are arranged approximately symmetrically with respect to the optical axis of light-receiving lens 16 and illuminate user's eyeball 14. A portion of the light emitted from light sources 13a and 13b and reflected by eyeball 14 is collected by light-receiving lens 16 onto ocular imaging element 17. Figure 4(a) is a schematic diagram of an eye image captured by ocular imaging element 17 (eyeball image projected onto ocular imaging element 17), and Figure 4(b) is a diagram showing the output intensity of the CCD in ocular imaging element 17. Figure 5 is a schematic flowchart of the gaze detection operation.

[0034] 5, light sources 13a and 13b emit infrared light toward user's eyeball 14. An image of the user's eyeball illuminated by the infrared light is formed on ocular imaging element 17 through light receiving lens 16 and is photoelectrically converted by ocular imaging element 17. As a result, an electrical signal of the eye image that can be processed is obtained.

[0035] In step S502, the line-of-sight detection circuit 201 sends the eye image (eye image signal; electric signal of the eye image) obtained from the eye imaging device 17 to the CPU 3.

[0036] In step S503, the CPU 3 obtains the coordinates of the points corresponding to the corneal reflection images Pd and Pe of the light sources 13a and 13b and the pupil center c from the eye image obtained in step S502.

[0037] Infrared light emitted from light sources 13a and 13b illuminates cornea 142 of user's eyeball 14. At this time, corneal reflection images Pd and Pe formed by part of the infrared light reflected from the surface of cornea 142 are collected by light receiving lens 16 and formed on ocular imaging element 17 as corneal reflection images Pd' and Pe' in the eye image. Similarly, light beams from edges a and b of pupil 141 are also formed on ocular imaging element 17 as pupil edge images a' and b' in the eye image.

[0038] FIG. 4(b) shows luminance information (luminance distribution) of region α in the eye image of FIG. 4(a). In FIG. 4(b), the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In this embodiment, the X-axis (horizontal) coordinates of the corneal reflection images Pd', Pe' are set to Xd, Xe, and the X-axis coordinates of the pupil edge images a', b' are set to Xa, Xb. As shown in FIG. 4(b), an extremely high level of luminance is obtained at the coordinates Xd, Xe of the corneal reflection images Pd', Pe'. In the region from coordinate Xa to coordinate Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained when the light beam from the pupil 141 is focused on the ocular imaging element 17), an extremely low level of luminance is obtained except for coordinates Xd, Xe. A luminance intermediate between the two types of luminance is obtained in the region of iris 143 outside pupil 141 (the region of the iris image outside the pupil image obtained by focusing the light beam from iris 143). Specifically, a luminance intermediate between the two types of luminance is obtained in the region where the X coordinate (coordinate in the X-axis direction) is smaller than coordinate Xa and the region where the X coordinate is larger than coordinate Xb.

[0039] From the brightness distribution shown in Figure 4(b), the X coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X coordinates Xa and Xb of the pupil edge images a' and b' can be obtained. Specifically, the coordinates where the brightness is extremely high can be obtained as the coordinates of the corneal reflection images Pd' and Pe', and the coordinates where the brightness is extremely low can be obtained as the coordinates of the corneal reflection images Pd' and Pe'. The coordinates Xc of the pupil-centered image c' (center of the pupil image) obtained when the light beam from the pupil center c is focused on the ocular imaging element 17 can be expressed as Xc ≒ (Xa + Xb) / 2 when the rotation angle θx of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16 is small. In other words, the coordinate Xc of the pupil-centered image c' can be calculated from the X-coordinates Xa and Xb of the pupil-edge images a' and b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinate of the pupil-centered image c' can be estimated.

[0040] In step S504, CPU 3 calculates the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position of eyeball 14 relative to light receiving lens 16, and can be calculated using a function of the distance (Xd-Xe) between corneal reflection images Pd' and Pe'.

[0041] In step S505, CPU 3 calculates the rotation angle of the optical axis of eyeball 14 relative to the optical axis of light receiving lens 16. The X coordinate of the midpoint between corneal reflection images Pd and Pe and the X coordinate of the center of curvature O of cornea 142 approximately coincide. Therefore, if the standard distance from the center of curvature O of cornea 142 to the center c of pupil 141 is Oc, then the rotation angle θx of eyeball 14 in the ZX plane (plane perpendicular to the Y axis) can be calculated using the following equation 1. The rotation angle θy of eyeball 14 in the ZY plane (plane perpendicular to the X axis) can also be calculated using a method similar to that for calculating rotation angle θx. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc (Formula 1)

[0042] In step S506, the CPU 3 uses the rotation angles θx and θy calculated in step S505 to determine (estimate) the user's viewpoint (the position where the gaze is fixed; the position where the user is looking) on ​​the display surface of the HMD 100. If the gaze position (coordinates of the viewpoint) (Hx, Hy) is the coordinate corresponding to the pupil center c, the gaze position (Hx, Hy) can be calculated using the following equations 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0043] The parameter m in Expressions 2 and 3 is a constant determined by the configuration of the finder optical system (such as the light receiving lens 16) of camera 1, and is a conversion coefficient for converting the rotation angles θx and θy into coordinates corresponding to the pupil center c on the display surface (screen) of HMD100. It is assumed that the parameter m is determined in advance and stored in the memory unit 129. The parameters Ax, Bx, Ay, and By are line-of-sight correction parameters for correcting individual differences in the line of sight, and are obtained by performing a calibration operation. It is assumed that the parameters Ax, Bx, Ay, and By are stored in the memory unit 129 before the line-of-sight detection operation starts.

[0044] In step S507, the CPU 3 stores the line-of-sight position (Hx, Hy) in the memory unit 129 and ends the line-of-sight detection operation.

[0045] <GUI operation method of HMD> Hereinafter, the GUI operation method of HMD100 in the present embodiment will be described using FIGS. 6 to 10.

[0046] FIG. 6(a) is a diagram showing the background seen by the user through HMD100. HMD100 superimposes the operation icon display area 602 on the background 601 in a semi-transparent manner, and further superimposes various operation icons 603 to 606 in the operation icon display area 602. The operation icon display area 602 and the various operation icons 603 to 606 are fixed on the display surface and are fixed. The operation icons 603 to 606 are an example of a first GUI whose positions on the display surface are fixed. Further, HMD100 superimposes an operation icon 608 indicating that there is information about the subject (building) 607 in the background 601 near the subject 607. This operation icon 608 is associated with the subject 607 and is fixed to the position of the subject in the real space so that it remains located near the subject 607 even when the range of the background seen by the user changes. The operation icon 608 is arranged in a three-dimensional space and is an example of a second GUI whose position on the display surface changes according to the change in the head posture.

[0047] As shown in FIG. 6(b), depending on the position of the subject in the background, the operation icon 613 fixed to the subject position in the real space may be displayed near the operation icon 605 fixed on the display surface. In that case, the user's line-of-sight position 615 may be detected in an area where the area 612 determined to be gazing at the operation icon 605 and the area 614 determined to be gazing at the operation icon 613 overlap. In such a situation, it is impossible to determine whether the user intends to operate either the operation icon 605 or the operation icon 613 based only on the user's line-of-sight position and head movement. Therefore, in the present embodiment, the HMD 100 determines whether there is an intention to operate the GUI in the vicinity of the line-of-sight position based on the user's line-of-sight position, head movement, and change in the pupil center position during the head movement.

[0048] <HMD Operation> FIG. 7 is a flowchart for explaining the operation of the HMD 100 according to the present embodiment.

[0049] First, in step S701, the CPU 128 performs startup processing of the HMD 100. In step S702, the CPU 128 uses the images captured by the left-eye camera 106 and the right-eye camera 107 to detect a subject in the background (within the viewing angle) that the user is viewing through the HMD 100.

[0050] In step S703, the CPU 128 acquires information about the subject based on the image of the subject detected in step S702 and the position information by GPS. The CPU 128 displays an operation icon indicating that there is information about the subject detected in step S702 near the subject, like the operation icon 608 in FIG. 6(a). The information about the subject is acquired, for example, by the CPU 128 connecting to a network such as the Internet and communicating with an external device such as a server.

[0051] In step S704, the CPU 128 performs the above-described line-of-sight detection operation to calculate the user's line-of-sight position.

[0052] In step S705, the CPU 128 performs processing to acquire operation intention information based on the gaze position, based on the gaze position calculated in the gaze detection operation in step S704. In step S706, the CPU 128 performs processing to acquire operation intention information based on head movement. In step S707, the CPU 128 performs processing to acquire operation intention information based on pupil center position, using the information on the pupil center position calculated in the gaze detection operation in step S704. Details of the processing to acquire operation intention information based on gaze position, head movement, and pupil center position will be described later. Note that the operation intention information is information that indicates the user's intention to operate an operation icon.

[0053] In steps S708 to S711, the CPU 128 performs a series of processes for all operation icons displayed on the display surface, focusing on each icon one by one in order. Hereinafter, the order in which the processes are performed is indicated by i.

[0054] In step S708, the CPU 128 determines whether the gaze count value d_cnt(i) of the i-th operation icon currently being focused on is equal to or greater than a predetermined threshold Cnt_Th, based on the result of the process of acquiring operation intention information based on the gaze position (step S705). If the gaze count value d_cnt(i) is equal to or greater than the threshold Cnt_Th, the CPU 128 proceeds to step S709; otherwise, the CPU 128 proceeds to step S711. The gaze count value d_cnt(i) corresponds to the gaze time. In order to determine whether gaze is continuing during a period in which a predetermined head movement is being performed, it is advisable to set the threshold Cnt_Th to be approximately the same time as the time required for the predetermined head movement. A method for calculating the gaze count value d_cnt(i) will be described later.

[0055] In step S709, CPU 128 determines whether the head movement indicates a predetermined movement based on the result of the process of acquiring operational intention information based on head movement (step S706). If the head movement indicates a predetermined movement, CPU 128 proceeds to step S710; if not, CPU 128 proceeds to step S711.

[0056] In step S710, CPU 128 determines whether the user intends to operate the currently focused operation icon i (i-th operation icon) based on the result of the process of acquiring operation intention information based on the pupil center position (step S707). If it is determined that the user intends to operate operation icon i, CPU 128 executes a process corresponding to the operation icon.

[0057] In step S711, CPU 128 determines whether or not the processes of steps S708 to S710 have been performed for all operation icons displayed on the display surface. If the processes of all operation icons displayed on the display surface have been performed, CPU 128 proceeds to step S712; if not, CPU 128 returns to step S708 and performs the processes of steps S708 to S710 for operation icons that have not been processed.

[0058] In step S712, the CPU 128 determines whether or not an instruction has been issued to stop the HMD 100. If an instruction has been issued to stop the HMD 100, the CPU 128 proceeds to step S713, and if not, the CPU 128 returns to step S702.

[0059] In step S713, the CPU 128 performs a process of stopping the HMD 100. When the CPU 128 has finished the process of stopping the HMD 100, it ends this flow.

[0060] <Acquisition of Operation Intention Information from Gaze Position> FIG. 8 is a flowchart for explaining the process of acquiring operational intention information based on gaze position in step S705 of the HMD operation flow in FIG.

[0061] In steps S801 to S805, the CPU 128 performs a series of processes for all operation icons displayed on the display surface, focusing on each icon one by one in order. Hereinafter, the order of the icons to be processed is indicated by i.

[0062] First, in step S801, the CPU 128 calculates the distance d(i) on the display surface from the gaze position to the currently focused operation icon i. The distance d(i) is, for example, the shortest distance from the gaze position to the operation icon i. Note that the distance d(i) may also be, for example, the distance from the gaze position to the center position of the operation icon i.

[0063] In step S802, the CPU 128 determines whether the distance d(i) is smaller than a predetermined threshold value ds. If the distance d(i) is smaller than the threshold value ds, the CPU 128 proceeds to step S803, and if not, the CPU 128 proceeds to step S804. If the distance d(i) is equal to the threshold value ds, the process may proceed to step S803. Here, the threshold value ds may be set to define a relatively small range (e.g., a minimum range) that can include the variation in gaze position detected when a person continues to gaze at a certain point. This variation in gaze position occurs due to the accuracy of gaze detection, head movement, small eye movements, etc.

[0064] In step S803, the CPU 128 increments the gaze counter d_cnt(i) of the operation icon i by one.

[0065] In step S804, the CPU 128 clears (resets) the gaze counter d_cnt(i) of the operation icon i to zero.

[0066] In step S805, CPU 128 determines whether or not the processes of steps S801 to S804 have been performed for all operation icons on the display surface. If the processes of all operation icons on the display surface have been performed, CPU 128 ends this flow, and if not, returns to step S801 and performs the processes of steps S801 to S804 for operation icons that have not been processed.

[0067] <Acquisition of operational intention information from head movements> FIG. 9 is a flowchart for explaining the process of acquiring operational intention information based on head movement in step S706 of the HMD operation flow in FIG.

[0068] First, in step S901, the CPU 128 acquires the output value of the inertial sensor 110.

[0069] In step S902, the CPU 128 determines whether or not the output value of the inertial sensor 110 satisfies a predetermined condition using the motion detection unit 111. The predetermined condition may be, for example, whether or not the output value of the inertial sensor 110 is equal to or greater than a threshold value Th_p in the positive direction during a predetermined time Int_t. If the output value of the inertial sensor 110 satisfies the predetermined condition, the CPU 128 proceeds to step S903, and if not, the CPU 128 proceeds to step S904.

[0070] In step S903, the CPU 128 determines that the head movement has become a predetermined movement. The predetermined movement is, for example, a "nodding movement," a "shaking of the head from side to side," or a "tilting of the head."

[0071] Here, an example of the processing of steps S901 to S903 will be described. As shown in Fig. 10(b), the inertial sensor 110 outputs angle information (angle, angular velocity) for each of the three directions, i.e., the yaw direction, pitch direction, and roll direction. In this embodiment, for example, the CPU 128 uses the output angular velocity information for the pitch direction to detect a "nodding gesture" (a short-term gesture of looking down and then returning to the original position) by the user.

[0072] FIG. 10(c) is a diagram showing the change over time in the output value in the pitch direction output by the inertial sensor 110 when the user performs a "nodding motion." The output value of the inertial sensor 110 increases in the positive direction when the user quickly looks down, and then increases in the negative direction when the user quickly returns to the original posture (faces forward). The CPU 128 determines whether the output value of the inertial sensor 110 becomes equal to or greater than the threshold value Th_p in the positive direction within a predetermined time Int_t. Then, by determining whether or not the value has become equal to or less than the threshold value Th_m in the negative direction, it is possible to determine whether or not a "nodding motion" has been performed. Note that the CPU 128 determines a "motion of shaking the head from side to side" from the output value in the yaw direction and a "motion of shaking the head from side to side" from the output value in the roll direction in the same way as for the "nodding motion." It is possible to detect the "head tilting action."

[0073] Returning to the description of Figure 9, in step S904, CPU 128 determines whether a change in the image captured by at least one of left-eye camera 106 and right-eye camera 107, using image determination unit 112 and motion detection unit 111, satisfies a predetermined condition. The predetermined condition is satisfied, for example, when the sum of motion vectors in the image suddenly changes from an upward direction to a downward direction within a predetermined period of time. If the change in the image satisfies the predetermined condition, CPU 128 proceeds to step S905; if not, CPU 128 ends this flow.

[0074] In step S905, the CPU 128 determines that the head movement has become a predetermined movement, and ends this flow.

[0075] An example of the processing of steps S904 to S905 will now be described. Left-eye camera 106 or right-eye camera 107 periodically captures images. Image determination unit 112 can calculate a motion vector within the captured image by calculating a difference image in the time direction for the captured image. CPU 128 can estimate the user's head movement based on the motion vector calculated by image determination unit 112. For example, CPU 128 can determine that a "nodding movement" has been made when the sum of the motion vectors within the image suddenly changes from pointing upward to pointing downward within a predetermined period of time. CPU 128 can also determine a "head shaking movement" and a "head tilting movement" in the same manner as for a "nodding movement."

[0076] <Acquisition of operation intention information based on pupil center position> FIG. 11 is a flowchart illustrating the process of acquiring operational intention information based on the pupil center position in step S707 of the HMD operation flow in FIG.

[0077] First, in step S1101, CPU 128 acquires the pupil center position from the results of pupil detection performed during the gaze detection operation. For example, as shown in Fig. 12(a), horizontal and vertical coordinates (Px1, Py1) are acquired as the pupil center position. Note that the gaze position indicates the coordinates of the position on the display surface where the user is looking, and the pupil center position indicates the coordinates of the center position of the pupil in an image of the user's eyes.

[0078] In step S1102, the CPU 128 performs band-pass filtering in the horizontal and vertical directions along the time axis using the current pupil center position acquired in step S1101 and the previously acquired pupil center position.

[0079] Here, band-pass filtering in the vertical direction will be described as an example with reference to FIG. 6(a) and FIGS. 12(a) to 12(c). FIGS. 12(a) to 12(c) are diagrams illustrating changes in the pupil center position during head movement. For example, when a user performs a "nodding motion" while gazing at an operation icon fixed near the subject in real space, such as operation icon 608 in FIG. 6(a), the pupil center position in the eye image quickly moves upward and then quickly returns to its original position. Specifically, the pupil center position quickly moves upward from (Px1, Py1) shown in FIG. 12(a) to (Px2, Py2) shown in FIG. 12(b), and then quickly returns to its original position (Px1, Py1). FIG. 12(c) illustrates the change over time in the vertical coordinate of the pupil center position during this "nodding motion."

[0080] Furthermore, when band-pass filtering is performed on the values ​​in Fig. 12(c) in the time direction, the output values ​​(BPF output values) shown in Fig. 12(d) are obtained. The CPU 128 determines whether the BPF output values ​​have become equal to or greater than a threshold value Py_th_p in the positive direction in conjunction with head movement, and then become equal to or less than a threshold value Py_th_m in the negative direction within a predetermined time Int_t. When a user gazes at an operation icon fixed near a subject in real space and performs a "nodding motion," the vertical coordinate of the pupil center position changes in conjunction with the head movement. On the other hand, when a user gazes at an operation icon fixed on the display surface and performs a "nodding motion," the vertical coordinate of the pupil center position does not change in conjunction with the head movement, or changes less than in the case of an operation icon fixed near a subject in real space. Therefore, CPU 128 can determine whether the operation icon being gazed at is an operation icon fixed near a subject in real space by determining whether the pupil center position does not change or has changed in conjunction with the head movement in a way that offsets the head movement. The same method is also applied to "shaking the head from side to side" and "tilting the head," and CPU 128 determines the operation icon being gazed at by the user.

[0081] Returning to the description of Figure 11, in step S1103, it is determined whether the output value after the band-pass filter processing in step S1102 satisfies a predetermined condition. The predetermined condition is satisfied, for example, when the output value becomes equal to or greater than a threshold value Py_th_p and then becomes equal to or less than a threshold value Py_th_m within a predetermined time Int_t. If the output value satisfies the predetermined condition, the CPU 128 proceeds to step S1104, and if not, proceeds to step S1105.

[0082] In step S1104, CPU 128 determines that the pupil center position has undergone a predetermined change (for example, a change linked to a "nodding action").

[0083] In step S1105, the CPU 128 stores the pupil center position in the memory unit 129. After the CPU 128 records the pupil center position in the memory unit 129, the CPU 128 ends this flow.

[0084] The CPU 128 determines whether the user is gazing at a specific operation icon while performing a head movement, and therefore also determines whether the user is gazing at the icon during the head movement. Head movements, for example, do not necessarily involve rotating the head in one direction and then stopping. It is also possible for the head to rotate in the opposite direction to the direction of rotation. If such a head movement is performed while an operation icon fixed to a subject in real space is displayed near an operation icon fixed on the display surface, both operation icons may remain near the gaze position during the head movement. Therefore, the operation icon the user wishes to operate cannot be accurately determined based solely on the gaze position and head movement. In this embodiment, the CPU 128 determines changes in the pupil center position in addition to the gaze position and head movement. This allows for more accurate determination of the operation icon the user wishes to operate.

[0085] It should be noted that CPU 128 may not determine whether or not the gaze is being observed during head movement, but may determine whether or not the gaze was observed before the head movement, and may determine whether or not there is a change in the pupil center position linked to the head movement during the head movement.

[0086] When estimating the gaze position from multiple detected gaze positions, the gaze position used for estimating the gaze position may be selected based on the correlation between gaze movement and head movement. For example, if the gaze moves in a way that offsets the head movement, a relatively new gaze position is selected to estimate the gaze position. This can improve the responsiveness of the gaze position to actual eye movement. Also, if the gaze does not move in a way that offsets the head movement (the gaze movement is relatively small), multiple gaze positions are selected to estimate the gaze position. This can reduce the influence of involuntary eye movement and the like on the gaze position, thereby stabilizing the gaze position.

[0087] <Example of processing corresponding to the operation icon> In the processing of steps S708 to S710 in FIG. 7, the CPU 128 determines whether or not the user has an intention to operate the operation icon, and 00. For example, when the CPU 128 detects a predetermined head movement indicating an intention to operate a specific operation icon while gazing at the operation icon, the CPU 128 executes a process corresponding to the operation icon.

[0088] A specific example will be described. First, as shown in Fig. 13(a), the user performs a "nodding motion" while the gaze position 1301 calculated in the gaze detection operation (step S704 in Fig. 7) remains within a predetermined range 1302 relative to the operation icon 1303. The user looks at the vicinity of the operation icon 1303 and then performs a "nodding motion," thereby indicating the intention to execute a process corresponding to the operation icon 1303 near the gaze position.

[0089] 13(b), CPU 128 changes the color or brightness of operation icon 1303 to indicate to the user that operation icon 1303 has been selected for execution of an operation, and then executes the process corresponding to operation icon 1303. In this way, CPU 128 controls the operation icons for which a process is being executed to have different colors or brightness from those for which a process is not being executed. Note that CPU 128 may also control the same operation icon to have different colors or brightness for those for which a process is being executed and those for which a process is not being executed.

[0090] Next, as shown in Fig. 13(c), CPU 128 displays icon 1304 indicating that processing corresponding to operation icon 1303 is being executed. In the example of Fig. 13(c), icon 1304 displays "Recording," indicating that recording processing is being executed as processing corresponding to operation icon 1303. Here, CPU 128 may prepare for processing corresponding to the operation icon in response to a first determination that the user intends to operate an operation icon near the gaze position, and may execute the processing corresponding to the operation icon in response to a second determination. This makes it possible to prevent malfunctions in processing.

[0091] Another example of processing corresponding to an operation icon will be described. As shown in Fig. 13(d), the user performs a "nodding motion" while keeping their gaze position 1305 within a predetermined range 1308 of an operation icon 1306 associated with a subject 1307. As shown in Fig. 13(e), the CPU 128 displays information display 1309 about the subject 1307. In the example of Fig. 13(e), the CPU 128 displays store information such as the name of the store of the subject 1307, business hours, whether it is open or not, and a phone number.

[0092] For example, in a similar manner, when the user performs a "motion of shaking his head from side to side," CPU 128 may perform processing to return the selected operation icon to a non-selected state or to suspend the processing in progress. When the user performs a "motion of tilting his head," CPU 128 may perform processing to switch to another operation icon or display an explanation of the operation icon.

[0093] In this embodiment, the change in pupil center position is used to determine the operation icon indicating the user's intention to operate. However, the operation icon indicating the user's intention to operate may be determined using information corresponding to the pupil position, not limited to the pupil center position. The operation icon indicating the user's intention to operate may be determined using the user's gaze position instead of the pupil center position. Furthermore, the CPU 128 may display the operation icon whose position on the display surface is fixed and the operation icon arranged in three-dimensional space at different distances. By displaying them at different distances, the CPU 128 may estimate the distance the user is gazing at using convergence angle information and determine the operation icon the user is gazing at.

[0094] In this embodiment, the CPU 128 acquires both the output value of the inertial sensor 110 and the output value of the image determination unit 112, and determines that the head movement has become a predetermined movement when either one of them satisfies a predetermined condition. Alternatively, it may be determined that the head movement has become the predetermined movement when both the output values ​​of the CPU 128, the inertial sensor 110, and the image determination unit 112 satisfy a predetermined condition. Alternatively, it may be determined whether the head movement has become the predetermined movement by acquiring the output value of either the CPU 128, the inertial sensor 110, or the image determination unit 112.

[0095] As described above, according to this embodiment, the user can easily operate the HMD in a hands-free manner.

[0096] It should be noted that the above-described embodiment (including modifications) is merely an example, and configurations obtained by appropriately modifying or changing the above-described configuration within the scope of the gist of the present invention are also included in the present invention. For example, although an example in which the present invention is applied to a display device has been described, the present invention can be applied to various electronic devices that control a display device. For example, the present invention can also be applied to a controller or personal computer (PC) separate from the display device.

[0097] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0098] 100: HMD 111: Motion detection unit 113: Display control unit 114: Gaze detection unit 128: CPU

Claims

1. a control means for controlling a display device detachably attached to the user's head so as to display a GUI on the display surface thereof; a gaze detection means for detecting a gaze position of the user based on an image of the user's eyes; a pupil detection means for detecting a pupil position of the user from the image of the eye; a movement detection means for detecting a movement of the user's head or neck; a determination means for determining whether the user has an intention to operate a GUI near the gaze position, based on the gaze position detected by the gaze detection means, the action detected by the action detection means, and a change in the pupil position during the action; and an execution means for executing a process corresponding to a GUI when the determination means determines that the user has an intention to operate the GUI in the vicinity of the gaze position; An electronic device comprising:

2. Further comprising a gyro sensor, The motion detection means detects the motion based on the output result of the gyro sensor.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

3. the user can visually recognize a range in three-dimensional space corresponding to the posture of the head together with the GUI displayed on the display surface; the electronic device further includes an acquisition unit that acquires an image of a range in the three-dimensional space according to the posture of the head, The motion detection means detects the motion based on a change in the image acquired by the acquisition means.

3. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

4. the user can visually recognize a range in three-dimensional space corresponding to the posture of the head together with the GUI displayed on the display surface; The GUI comprises: a first GUI whose position on the display surface is fixed; or a second GUI that is arranged in the three-dimensional space and whose position on the display surface changes in response to a change in the posture of the head; 4. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

5. the gaze detection means detects the gaze position within a predetermined range from the first GUI and the second GUI, When the action detection means detects a predetermined action indicating an intention to operate the GUI, The determination means If the change in the pupil position during the predetermined operation is smaller than a predetermined threshold, it is determined that the user has an intention to operate the first GUI located near the gaze position; The electronic device according to claim 4, characterized in that if the change in the pupil position during the specified operation is greater than a specified threshold, it is determined that the user intends to operate the second GUI located near the gaze position.

6. The control means controls the GUI in which a process is being executed and the GUI in which a process is not being executed to have different colors or brightness.

6. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

7. The execution means prepares a process corresponding to the GUI in response to a first determination that the user has an intention to operate a GUI near the gaze position, and executes the process corresponding to the GUI in response to a second determination.

7. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

8. The display device is an optical see-through display.

8. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

9. The display device is a video see-through display.

9. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

10. The pupil position is the pupil center position.

10. The electronic device according to claim 1.

11. a control step of controlling the display device detachably attached to the user's head so that the GUI is displayed on a display surface of the display device; a gaze detection step of detecting a gaze position of the user based on an image of the user's eyes; a pupil detection step of detecting a pupil position of the user from the eye image; a movement detection step of detecting a movement of the user's head or neck; a determination step of determining whether the user has an intention to operate a GUI near the gaze position, based on the gaze position detected by the gaze detection step, the action detected by the action detection step, and a change in the pupil position during the action; an execution step of executing a process corresponding to the GUI when it is determined in the determination step that the user has an intention to operate the GUI in the vicinity of the gaze position; 1. A method for controlling an electronic device, comprising:

12. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 10.

13. A computer functions as each of the means of the electronic device according to any one of claims 1 to 10. A computer-readable recording medium storing a program for causing a computer to

Images