Gaze detection device, gaze detection program, and head-mounted display

Abstract

[Problem] To provide a visual line detection device for detecting a visual line direction even when a plurality of bright spots cannot be observed on a cornea. [Solution] In this visual line detection device, when the state of bright spots observed on the cornea of an eye on the basis of light applied from a plurality of light sources is a first state, a processor (1-1) identifies a center position of a cornea on the basis of a plurality of bright spots, (1-2) identifies the position of the pupil of an eye on the basis of an image captured by an imaging device, and (1-3) identifies a visual line direction on the basis of the center position of a cornea and the position of a pupil. Meanwhile, when the state of bright spots observed on the cornea of an eye on the basis of the plurality of light sources is a second state, the processor (2-1) identifies the shape of an eyeball in accordance with data about a center position of the eyeball and the radius of the eyeball stored in a storage unit, and (2-2) identifies the visual line direction on the basis of the image captured by the imaging device and the center position of the eyeball.

Description

[Technical Field]

[0001] The present invention relates to a gaze detection device for detecting a gaze direction, a gaze detection program, and a head-mounted display. [Background technology]

[0002] Currently, head-mounted displays capable of providing three-dimensional images are becoming increasingly popular in order to provide virtual reality (VR) in games and videos. A head-mounted display is an image providing device that provides a three-dimensional image to a user inside a housing that covers the user's head. Some head-mounted displays are equipped with a gaze detector that detects the user's gaze direction, and are configured to change the three-dimensional image by, for example, increasing the resolution of only the image in the gaze direction (see, for example, Patent Document 1).

[0003] The user's gaze direction is detected by irradiating the eye with infrared light and detecting the position of the bright spot of light in the image of the eye illuminated with infrared light. In this case, to accurately grasp the gaze direction, it is necessary to irradiate the eye with multiple beams of light from multiple light sources and observe multiple bright spots on the cornea.

[0004] However, even with multiple light sources, it is difficult to constantly observe multiple bright spots on the cornea. For example, blinking or shaking of the head-mounted display housing can cause some of the bright spots to be projected outside the cornea, making them unusable as information for gaze detection. In such cases, the gaze direction cannot be accurately detected, and the head-mounted display cannot perform the desired operation. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-134371 Summary of the Invention [Problem to be solved by the invention]

[0006] SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and has as its object to provide a gaze direction detection device that detects the gaze direction even when a plurality of bright spots cannot be observed on the cornea. [Means for solving the problem]

[0007] In order to achieve the above object, a gaze detection device according to the present invention includes a plurality of light sources that irradiate an eye with light, an imaging device that captures an image of the eye, and a processor, wherein when a state of a bright spot observed on the cornea of ​​the eye based on the plurality of light sources is a first state, the processor: (1-1) identifying the center position of the cornea based on the plurality of bright points; (1-2) identifying the position of the pupil of the eye based on the image of the imaging device; (1-3) Identifying the gaze direction based on the center position of the cornea and the position of the pupil. On the other hand, when the state of the bright spot observed on the cornea of ​​the eye based on the plurality of light sources is a second state, (2-1) Identifying the shape of the eyeball according to the data of the radius of the eyeball and the center position of the eyeball stored in a storage unit; (2-2) The line of sight direction is identified from the image of the imaging device and the center position of the eyeball. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a gaze direction detection device for detecting the gaze direction even when a plurality of bright spots cannot be observed on the cornea. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram schematically illustrating an overview of a video system 1 according to a first embodiment.

[0010] [Figure 2]2 is a perspective view schematically showing the configuration of a video display unit 140 according to the first embodiment. FIG.

[0011] [Figure 3] 2 is a schematic cross-sectional view showing a configuration of an image display unit 140 according to the first embodiment. FIG.

[0012] [Figure 4] 1 is a block diagram showing the configuration of a main part of a head-mounted display 100 and a video playback device 200 according to a first embodiment.

[0013] [Figure 5] FIG. 2 is a schematic diagram illustrating a method for detecting a gaze direction using a detection unit 203 according to the first embodiment.

[0014] [Figure 6] FIG. 2 is a schematic diagram illustrating a method for detecting a gaze direction using a detection unit 203 according to the first embodiment.

[0015] [Figure 7] 10 is a flowchart illustrating a method for detecting a gaze direction using a detection unit 203 according to the first embodiment.

[0016] [Figure 8] FIG. 2 is a schematic diagram illustrating a method for detecting a gaze direction using a detection unit 203 according to the first embodiment.

[0017] [Figure 9] 10 is a flowchart illustrating a method for detecting a gaze direction using a detection unit 203 according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, the present embodiment will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be designated by the same numerals. Note that the accompanying drawings show embodiments and implementation examples according to the principles of the present disclosure, but these are for understanding the present disclosure and are not to be used to interpret the present disclosure in a limiting manner. The descriptions in this specification are merely typical examples and are not intended to limit the scope or application of the present disclosure in any way.

[0019] Although the present embodiment has been described in sufficient detail to enable those skilled in the art to implement the present disclosure, it should be understood that other implementations and forms are possible, and that changes in configuration and structure and substitutions of various elements are possible without departing from the scope and spirit of the technical ideas of the present disclosure. Therefore, the following description should not be interpreted as being limited thereto.

[0020] [First embodiment] A video system 1 according to a first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram schematically illustrating an overview of the video system 1. The video system 1 according to the first embodiment includes a head-mounted display 100 and a video playback device 200. As shown in Fig. 1, the head-mounted display 100 may be a shielded type head-mounted display that is worn on the head of a user.

[0021] The video playback device 200 generates image data to be displayed on the head mounted display 100 and transmits the image data to the head mounted display 100 via wired or wireless communication. As an example, the video playback device 200 is a device that can play images from a personal computer, a stationary game console, a portable game console, a tablet terminal, a smartphone, a phablet, a video player, a television, etc.

[0022] The wireless connection between the video playback device 200 and the head mounted display 100 can be realized using known wireless communication technologies such as Wi-Fi (registered trademark), Bluetooth (registered trademark), and infrared communication. Furthermore, image transmission between the head mounted display 100 and the video playback device 200 is performed in accordance with standards such as Miracast (trademark), WiGig (trademark), WHDI (trademark), or Ethernet (trademark). The video playback device 200 may be configured integrally with the head mounted display 100 or may be built into the head mounted display 100.

[0023] The head mounted display 100 includes a housing 110, a head fixing unit 120, headphones 130, and a video display unit 140. As will be described later, the housing 110 houses the video display unit 140, as well as a transmission module and various sensors. The head fixing unit 120 is a member that causes the head mounted display 100 to be worn on the user's head. The headphones 130 output audio of images played back by the video playback device 200. Although not shown, the head mounted display 100 may also include a gyro sensor or the like for detecting the tilt direction of the head mounted display 100. The video playback device 200 can change the display state of the image according to the detection result of this gyro sensor. The video display unit 140 is housed in the housing 110 and displays the video transmitted from the video playback device 200.

[0024] Fig. 2 is a perspective view showing a schematic configuration of the video display unit 140, and Fig. 3 is a schematic cross-sectional view thereof. Fig. 4 is a block diagram showing the configuration of the main parts of the head-mounted display 100 and the video playback device 200.

[0025] As shown in Fig. 2, the image display unit 140 presents an image to each of the user's left and right eyes E (EL, ER), thereby making it possible to present a stereoscopic image to the user. As shown in Fig. 3, the image display unit 140 includes an infrared light source 141 (141a-c), an objective lens 142, a half mirror 143, a camera 144, a hot mirror 145, and a display 146. Note that the configuration of the image display unit 140 is the same for both the left and right eyes E, and therefore only the configuration of one side will be described below.

[0026] The infrared light source 141 is a light source used to detect the positions of the center of the pupil of the eye, the center of the cornea, and further the center of the eyeball, and is used to detect the line of sight of the eye E, and is a light source capable of emitting light in the near-infrared wavelength band (approximately 780 nm to 2500 nm). At least three infrared light sources 141 are provided. Here, as an example, it is assumed that three infrared light sources 141a to 141c are provided (infrared light source 141c is not shown in FIG. 3). Note that, hereinafter, the infrared light sources 141a to 141c may be collectively referred to as "infrared light source 141."

[0027] The objective lens 142 is disposed so as to be positioned in front of the user's eye E when the head mounted display 100 is in use. The infrared light sources 141a to 141c can be disposed around the objective lens 142. Here, the three infrared light sources 141a to 141c are disposed at positions that are not at least line-symmetrical with respect to the optical axis of the objective lens 142, and are preferably disposed at approximately equal angles around the optical axis.

[0028] A half mirror 143 and a hot mirror 145 are provided in the optical path between the objective lens 142 and the display 146. The half mirror 143 transmits a portion of the light from the eye E and reflects the remainder. A camera 144 is provided in the optical path on the reflection side of the half mirror 143, and the hot mirror 145 is provided in the optical path on the transmission side.

[0029] Camera 144 is an imaging device that captures an image of eye E on which light from infrared light sources 141a to 141c is projected and a bright spot is formed based on the light. The image of the eye including this bright spot is used to detect the positions of the center of the pupil and the center of the cornea of ​​the eye, and ultimately the direction of the gaze. Display 146 is a video display device for displaying images transmitted from video playback device 200, and can be realized using, for example, a known liquid crystal display or organic EL display. Hot mirror 145 has the property of transmitting visible light emitted by display 146 while reflecting near-infrared light.

[0030] Infrared light emitted from infrared light sources 141a to 141c and reaching user's eye E is reflected by eye E and travels toward objective lens 142. A portion of the light that passes through objective lens 142 is reflected by half mirror 143 and enters camera 144. Camera 144 may be equipped with a filter (not shown) that blocks visible light.

[0031] 4, the head mounted display 100 includes a control unit 101 and a communication unit 102 in addition to an infrared light source 141, a camera 144, and a display 146. The control unit 101 controls the entire head mounted display 100 including the infrared light source 141, the camera 144, and the display 146. The communication unit 102 also controls data communication with the video playback device 200.

[0032] The video playback device 200 also includes a control unit 201 (processor) for controlling the detection unit 203 and the video generation unit 204, and a communication unit 202 for managing data communication with the head mounted display 100. The communication unit 202 communicates with the communication unit 102 in the head mounted display 100, receives various data obtained by the head mounted display 100 and supplies it to the control unit 201, and conversely, transmits data obtained by the video playback device 200 to the head mounted display 100. The video playback device 200 also includes the detection unit 203 and the video generation unit 204.

[0033] The detection unit 203 detects the positions of the pupil center, the cornea center, and the eyeball center of the user's eye based on an image of the eye E captured by the camera 144 with infrared light from the infrared light sources 141a to 141c projected onto the eye E. The detection unit 203 then detects the gaze direction of the eye E based on the data on the positions of the pupil center, the cornea center, and / or the eyeball center. The detection unit 203 employs different gaze direction detection methods depending on the number of bright points of the infrared light sources 141a to 141c that can be observed on the cornea of ​​the eye E.

[0034] When the positions of three bright points on eye E based on the three infrared light sources 141 can be observed (first state), detection unit 203 identifies the center position of the pupil and the center position of the cornea, thereby detecting a gaze vector. Furthermore, gaze vectors are detected at different times, and the position of the center of the eyeball is detected based on the position of the intersection of these gaze vectors. When the center position of the eyeball is detected, data on the radius of the eyeball is also calculated based on data on the position of the detected pupil center, etc.

[0035] On the other hand, when fewer than three bright points are observable on the cornea of ​​eye E (second state), the detection unit 203 generates eyeball model data using the data on the center position of the eyeball and the radius of the eyeball stored in the storage unit, and further identifies the position of the pupil. Then, the vector from the center position of the eyeball toward the center of the pupil is detected as the gaze vector, i.e., the gaze direction. This point will be described in detail later.

[0036] As described above, the video playback device 200 may be a personal computer, a stationary game console, a portable game console, a tablet terminal, a smartphone, a phablet, a video player, or the like. The detection unit 203 may be realized by, for example, an image signal processor (ISP) included in a personal computer or the like, and an image processing program stored in a storage device. The image processing program may be stored in a built-in storage device such as a ROM or RAM, or may be stored in a portable storage device such as a memory card, DVD-RAM, or CD-ROM.

[0037] Note that, if the head mounted display 100 includes a memory, and the control unit 101 and the memory can function as computational resources, the control unit 101 of the head mounted display 100 may execute a program that realizes the gaze detection unit. Similarly, instead of the detection unit 203, the control unit (processor) 201 of the video playback device 200 may execute a program that realizes the gaze detection unit.

[0038] The image generating unit 204 has a function of generating an image to be displayed on the display 146 in accordance with a detection signal from a gyro sensor (not shown) or the like. Note that the image generating unit 204 may change the image to be generated in accordance with the detection result of the above-mentioned detection unit 203 in addition to the output of the gyro sensor or the like.

[0039] Next, a method for detecting the gaze direction using the detection unit 203 according to the first embodiment will be described with reference to FIGS.

[0040] As shown in FIG. 5, a human eye E has an eyeball B with an eyeball center Bc and a cornea C formed in a part of the eyeball B. The cornea C has a different radius (corneal radius) from the eyeball B, and its curvature is determined by the corneal center Cc. The cornea C also contains an iris, and the opening formed in the center of the iris is the pupil P. The pupil center Pc, which is the center of the pupil P, the corneal center Cc, and the eyeball center Bc are aligned on a single straight line Ls. The straight line Ls is the gaze direction of the eye E. Therefore, when detecting the gaze direction of the eye E, the gaze direction of the eye E (straight line Ls) can be determined by identifying the eyeball center Bc, the pupil center Pc, and the corneal center Cc.

[0041] In the case of the video system 1 shown in Fig. 1, the user's head is fixed to a head-mounted display 100. As shown in Fig. 6, it is considered that the gaze direction of eye E changes as eyeball B rotates around eyeball center Bc. In this embodiment, when the gaze direction changes over time, the positions of the pupil center Pc (Pc0-2) and corneal center Cc (Cc0-2) are identified at a plurality of different timings, and further, a straight line Ls (Ls0-2) connecting the pupil center Pc and corneal center Cc is identified.

[0042] The intersection of the multiple straight lines Ls thus identified is identified as the eyeball center Bc. The difference (distance) between the eyeball center Bc and the pupil center Pc is identified as the radius Re of the eyeball B. Note that the multiple straight lines Ls may not intersect at a single point. In this case, the point closest to the multiple straight lines Ls (nearest point) may be calculated using the least squares method or the like and used as the eyeball center Bc. Note that the position of the eyeball center Bc is estimated to be constant unless the user's head is displaced relative to the head-mounted display 100. For this reason, in this embodiment, position information of the identified eyeball center Bc is stored in a storage unit and used to identify the gaze direction thereafter.

[0043] Next, the procedure for detecting the gaze direction in the video system 1 according to this embodiment will be described with reference to the flowchart in Fig. 7. In this embodiment, as described above, the detection unit 203 employs different gaze direction detection methods depending on the number of bright points of the infrared light sources 141a-c observable on the cornea of ​​the eye E.

[0044] When an image of the eye E is captured by the camera 144 with light projected from the infrared light sources 141a-c (step S11), the detection unit 203 counts the number of bright spots formed on the cornea C in the image (step S12). If three bright spots are confirmed on the cornea C, the process proceeds to step S21. On the other hand, if the number of bright spots observed on the cornea C is two or less, the process proceeds to step S31.

[0045] In step S21, the detection unit 203 analyzes the positions of the three confirmed bright spots and calculates the position of the corneal center Cc of the cornea C. In the following step S22, the detection unit 203 identifies the pupil P in the image captured by the camera 144 and calculates the position of the pupil center Pc. Then, in step S23, the detection unit 203 calculates a straight line Ls connecting the corneal center Cc and the pupil center Pc calculated in steps S21 and S22 as the gaze direction of the eye E. The calculated gaze direction is output to the control unit 201.

[0046] The control unit 201 executes various controls on the head mounted display 100 and the video playback device 200 according to the obtained data on the gaze direction. Data on the line Ls indicating the gaze direction is calculated at predetermined time intervals and stored in a memory (not shown) of the detection unit 203. For example, at times t0, t1, t2, data on different lines Ls0, Ls1, Ls2, etc. is calculated and stored.

[0047] In the following step S24, it is determined whether the radius Re of the eyeball B is known and has already been stored in a memory (not shown) of the detection unit 203. If the radius of the eyeball B is not known and has not been stored in the memory (No), the process proceeds to step S25, where the radius Re of the eyeball B is calculated according to the data of the multiple straight lines Ls described above. Specifically, as shown in FIG. 6, data of three or more straight lines Ls (Ls0 to Ls2) is obtained, and the position of the eyeball center Bc of the eyeball B is determined by finding the intersection or nearest point of the three or more straight lines Ls. Once the position of the eyeball center Bc is identified, the radius Re of the eyeball B is determined from its relationship with the position of the pupil center Pc. In step S26, the position data of the eyeball center Bc of the eyeball B thus determined and the value of the radius Re are stored in the memory of the detection unit 203.

[0048] On the other hand, if it is determined in step S24 that the radius Re of the eyeball B and the position of the eyeball center Bc of the eyeball B are already stored in the memory (Yes), step S25 is skipped. However, even if the radius Re of the eyeball B and the position of the eyeball center Bc are already stored in the memory, step S25 may be periodically executed to newly calculate the data of the eyeball radius Re and the eyeball center Bc and update the data stored in the memory.

[0049] The above describes the operation when three bright spots are observed on the cornea C in step S12. The following describes the operation (steps S31 to S33) when two or fewer bright spots are observed. As described above, even if multiple (e.g., three) infrared light sources 141a to 141c are used, it is not always possible to observe multiple bright spots on the cornea C. In this case, the center position of the cornea C cannot be identified, making it difficult to detect the gaze direction based on this. However, in this embodiment, the eyeball center Bc is identified in advance and its position data is stored in memory. The gaze direction can be detected based on the position data of this eyeball center Bc and the pupil center Pc. The procedure (steps S31 to S33) is described below.

[0050] In step S31, the position of the eyeball center Bc and the radius Re of the eyeball B, which are shape data of the eyeball B stored in the memory of the detection unit 203, are read out. As shown in Fig. 8(a), the detection unit 203 generates shape data Me of the eyeball B based on the read position data of the eyeball center Bc and the radius Re.

[0051] Next, in step S32, as shown in Fig. 8(b), position data of the pupil center Pc of the pupil P is calculated according to the image captured by the camera 144. Then, in step S33, data of a straight line Ls connecting the eyeball center Bc and the pupil center Pc is calculated (see Fig. 8(c)), and this straight line Ls is identified as the line of sight.

[0052] In the above description, three infrared light sources 141 are provided for each of the left and right eyes. However, this is merely an example, and the number of infrared light sources provided for each eye E may be four or more. Also, in the above example, in step S12, the gaze direction detection method is switched depending on whether three or more bright points are observed on the cornea C, but this is also just an example, and it goes without saying that multiple detection methods may be switched based on other criteria.

[0053] As described above, the system of the first embodiment projects light from multiple infrared light sources, determines the position of the corneal center Cc based on multiple bright spots formed on the cornea C, and also determines the position of the pupil center Pc, thereby making it possible to detect the gaze direction. Even if multiple bright spots cannot be observed on the cornea C, the gaze direction can be detected based on pre-stored data on the eyeball center Bc and data on the pupil center Pc. Therefore, according to this embodiment, it is possible to accurately detect the gaze direction regardless of changes in the situation.

[0054] [Second embodiment] Next, a video system 1 according to a second embodiment will be described with reference to FIG. 9. FIG. 9 is a flowchart showing the procedure for executing a gaze detection operation in the video system 1. The overall configuration of the video system of the second embodiment is substantially the same as that of the first embodiment (FIGS. 1 to 4), so duplicated explanations will be omitted. In this second embodiment, as shown in FIG. 9, step S24 of FIG. 7 is omitted, and calculation of the eyeball radius Re is performed every time, regardless of whether the eyeball radius Re is known or not, and data on the positions of the eyeball radius Re and the eyeball center Bc is updated. This second embodiment can also achieve the same effects as the first embodiment.

[0055] [Third embodiment] Next, a video system 1 according to a third embodiment will be described with reference to FIG. 10. FIG. 10 is a flowchart showing the procedure for executing a gaze detection operation in the video system 1. The overall configuration of the video system of the third embodiment is substantially the same as that of the first embodiment (FIGS. 1 to 4), so duplicated explanations will be omitted. The gaze detection operation of this third embodiment (FIG. 10) differs from that of the first embodiment (FIG. 7) in the procedure for detecting the pupil center Pc. In FIG. 10, the operations of the blocks assigned the same step numbers as those in FIG. 7 are the same, so explanations will be omitted below.

[0056] In Fig. 10, the position of the pupil center Pc is detected in step S22, but the operation differs depending on whether the detection is successful or unsuccessful. If the detection is successful (Yes in step S41), the gaze detection procedure proceeds in the same manner as in the first embodiment (steps S23 to S26). On the other hand, if the detection is unsuccessful (No in step S41), it is determined whether the eyeball center Bc is known (step S42). If the eyeball center Bc is known (Yes), gaze detection is performed based on the eyeball center Bc and the corneal center Cc instead of gaze detection based on the pupil center Pc and the corneal center Cc (step S23). If the eyeball center Bc is not known, there is insufficient information for gaze detection, so gaze detection is not performed and the process returns to step S11.

[0057] According to the third embodiment, even if detection of the position of the pupil center Pc fails, gaze detection can be performed as long as the position of the corneal center Cc is detected and the position of the eyeball center Bc is obtained. Therefore, compared to the previous embodiments, the number of opportunities to perform gaze detection increases, and it becomes possible to more accurately perform a given operation of the head mounted display.

[0058] [Fourth embodiment] Next, a video system 1 according to a fourth embodiment will be described with reference to Fig. 11 and Fig. 12. Fig. 11 is a flowchart showing the procedure for executing a gaze detection operation in the video system 1, and Fig. 12 is a schematic diagram showing the operation of the fourth embodiment. The overall configuration of the video system of the fourth embodiment is substantially the same as that of the first embodiment (Figs. 1 to 4), so duplicated explanations will be omitted.

[0059] The gaze detection operation in this fourth embodiment (FIG. 11) differs from that in the first embodiment (FIG. 7) in the operation that occurs after it is determined that the number of bright spots is less than three (after the determination of No in step S12). Specifically, in this third embodiment, when it is determined that the number of bright spots is less than three, the gaze direction is detected based on the eyeball shape data BS and the direction Dr of the line toward the eyeball. In FIG. 11, the operations of the blocks assigned the same step numbers as in FIG. 7 are the same, so their explanation will be omitted below.

[0060] In this fourth embodiment, after generating eyeball shape data BS in step S31, the position of the pupil center CP in camera 144 is detected as shown in Fig. 12, and the direction Dr of a line from camera 144 toward pupil center CP is analyzed based on this detection information via an optical system such as lens 142 (step S32A). The direction Dr from camera 144 toward pupil center CP can be calculated from the 2D pupil center in the 2D image captured by camera 144. The direction Gd from the eyeball center Bc toward the intersection of this line direction Dr and eyeball shape data BS is identified as the gaze direction.

[0061] As explained above, according to the fourth embodiment, the direction Dr of the line pointing toward the pupil center CP is analyzed, and the gaze direction is analyzed based on this direction Dr and the shape data of the eyeball. This method can also achieve the same effects as the previous embodiments.

[0062] [others] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, remove, or replace part of the configuration of each embodiment with other configurations. For example, in the above-described embodiment, an example in which the gaze detection device is installed in a video system having a head-mounted display 100 has been mainly described, but the scope of application of the gaze detection device of the present invention is not limited thereto and can be applied to various devices and systems that can use gaze detection functions.

[0063] In the above-described embodiment, the processor in the gaze detection device executes a gaze detection program or the like to identify the user's gaze direction, but the gaze detection device may be realized by hardware such as a logic circuit or dedicated circuit formed on an integrated circuit. These circuits may be realized by one or more integrated circuits, and the functions of the multiple functional units described in the above-described embodiment may be realized by a single integrated circuit.

[0064] In addition, the gaze detection program may be recorded on a processor-readable recording medium, and the recording medium may be a "non-transitory tangible medium," such as a tape, disk, card, semiconductor memory, or programmable logic circuit. [Explanation of symbols]

[0065] 1...Video system, 100...Head-mounted display, 101...Control unit, 102...Communication unit, 110...Housing, 120...Head fixing unit, 130...Headphones, 140...Video display unit, 141, 141a-c...Infrared light source, 142...Objective lens, 143...Half mirror, 144...Camera, 145...Hot mirror, 146...Display, 200...Video playback device, 201...Control unit (processor), 202...Communication unit, 203...Detection unit, 204...Video generation unit, B...Eyeball, Bc...Eyeball center, C...Cornea, Cc...Corneal center, E...Eye, Ls...Straight line, Me...Shape data, P...Pupil, Pc...Pupil center, Re...Eyeball radius.

Claims

1. a plurality of light sources for irradiating light onto the eye; an imaging device that captures an image of the eye; Processor and Equipped with The processor: When the number of bright points observed on the cornea of ​​the eye based on the plurality of light sources is equal to or greater than the number that allows the center position of the cornea to be identified, (1-1) Identifying the center position of the cornea based on the plurality of bright points; (1-2) identifying the position of the pupil of the eye based on the image of the imaging device; (1-3) Identifying a gaze direction based on the center position of the cornea and the position of the pupil; If the number of bright points observed on the cornea of ​​the eye based on the plurality of light sources is less than the number that allows the center position of the cornea to be identified, (2-1) Identifying the shape of the eyeball according to the data of the radius of the eyeball and the center position of the eyeball stored in a storage unit; (2-2) Identifying the gaze direction from the image of the imaging device and the center position of the eyeball A gaze detection device characterized by:

2. The gaze detection device described in Claim 1, wherein, if the number of bright spots observed on the cornea of ​​the eye is equal to or greater than the number that allows the central position of the cornea to be identified, the processor further identifies the central position of the eyeball and the radius of the eyeball based on the identified gaze direction, and stores the identified central position of the eyeball and the radius of the eyeball in the memory unit.

3. 2. The gaze detection device according to claim 1, wherein the processor is configured to identify a radius of the eyeball and a center position of the eyeball according to the gaze direction data acquired at a plurality of times, and store the radius and center position of the eyeball in the memory unit.

4. 2. The gaze detection device according to claim 1, wherein the processor identifies the center position of the eyeball by determining the intersection of the identified multiple lines of sight, and identifies the radius of the eyeball based on the difference between the center position of the eyeball and the position of the pupil.

5. When the number of the bright points observed on the cornea of ​​the eye based on the plurality of light sources is less than a number that can identify a center position of the cornea, the processor: The gaze detection device according to claim 1 , wherein the position of the pupil is identified from the image of the imaging device, and the gaze direction is identified based on the center position of the eyeball and the position of the pupil.

6. When the number of the bright points observed on the cornea of ​​the eye based on the plurality of light sources is equal to or greater than a number that can identify a center position of the cornea, The gaze detection device according to claim 1 , wherein, if the position of the pupil of the eye fails to be identified, the gaze direction is identified according to the shape of the eyeball and the center position of the cornea.

7. When the number of the bright points observed on the cornea of ​​the eye based on the plurality of light sources is less than a number that can identify a center position of the cornea, the processor: The gaze detection device according to claim 1 , further comprising: determining a direction of a light ray emitted from the eye from the image captured by the imaging device; and determining the gaze direction based on the center position of the eyeball and the direction of the light ray.

8. A gaze detection program for detecting a gaze direction of an eye, illuminating the eye with light from a plurality of light sources and determining the number of bright spots observed on the cornea of ​​the eye; If the number of bright points is equal to or greater than the number that allows the center position of the cornea to be identified, (1-1) Identifying the center position of the cornea based on the plurality of bright points; (1-2) determining the position of the pupil of the eye based on the image of the eye; (1-3) Identifying the gaze direction based on the center position of the cornea and the position of the pupil, and if the number of the bright points is less than the number that allows the center position of the cornea to be identified, (2-1) Identifying the shape of the eyeball according to the data of the radius of the eyeball and the center position of the eyeball stored in the storage unit; (2-2) determining the gaze direction from the image of the eye and the center position of the eyeball; A gaze detection program that causes a computer to execute the above.

9. A gaze detection program as described in Claim 8, wherein, if the number of bright spots is equal to or greater than the number that allows the central position of the cornea to be identified, the computer is further caused to identify the central position of the eyeball and the radius of the eyeball based on the identified gaze direction, and to store the identified central position of the eyeball and the radius of the eyeball in the memory unit.

10. 9. The gaze detection program according to claim 8, further comprising: specifying a radius of the eyeball and a center position of the eyeball according to the gaze direction data acquired at a plurality of times; and storing the specified radius and center position of the eyeball in the storage unit.

11. 9. The gaze detection program according to claim 8, wherein the center position of the eyeball is identified by finding an intersection of the identified plurality of gazes, and the radius of the eyeball is identified based on a difference between the center position of the eyeball and the position of the pupil.

12. When the number of the bright points is less than the number that can identify the center position of the cornea, The gaze detection program according to claim 8 , further comprising: identifying a position of the pupil from the image of the eye; and identifying the gaze direction based on the center position of the eyeball and the position of the pupil.

13. When the number of the bright points is equal to or greater than the number that allows the center position of the cornea to be identified, The gaze detection program according to claim 8 , wherein, if the position of the pupil of the eye fails to be identified, the gaze direction is identified according to the shape of the eyeball and the center position of the cornea.

14. When the number of the bright points is less than the number that can identify the center position of the cornea, The gaze detection program according to claim 7 , further comprising: identifying a direction of a light ray emitted from the eye from the image of the eye; and identifying the gaze direction based on the center position of the eyeball and the direction of the light ray.

15. A head-mounted display including a housing to be worn on a user's head and a gaze detection device installed on the housing to detect the gaze of the user, The gaze detection device a plurality of light sources that irradiate the user's eyes with light; an imaging device that captures an image of the eye; Processor and Equipped with The processor: When the number of bright points observed on the cornea of ​​the eye based on the plurality of light sources is equal to or greater than the number that allows the center position of the cornea to be identified, (1-1) Identifying the center position of the cornea based on the plurality of bright points; (1-2) identifying the position of the pupil of the eye based on the image of the imaging device; (1-3) Identifying a gaze direction based on the center position of the cornea and the position of the pupil; If the number of bright points observed on the cornea of ​​the eye based on the plurality of light sources is less than the number that allows the center position of the cornea to be identified, (2-1) Identifying the shape of the eyeball according to the data of the radius of the eyeball and the center position of the eyeball stored in a storage unit; (2-2) Identifying the gaze direction from the image of the imaging device and the center position of the eyeball A head-mounted display characterized by:

Images