Electronic device

Abstract

To provide a technique capable of performing both line-of-sight detection and personal identification of a user with high frequency and high accuracy.SOLUTION: An electronic device according to an aspect of the present disclosure includes a control unit configured to perform control to turn on at least one of a plurality of light sources capable of irradiating an eye of a user with light, an acquisition unit configured to acquire an eye image that is an image of the eye captured in a state where at least one of the plurality of light sources is turned on, a line-of-sight detection unit configured to acquire line-of-sight information that is information on a line of sight of the eye, and an identification unit configured to identify the user. The control unit controls a plurality of first light sources among the plurality of light sources to be turned on and then controls a plurality of second light sources including a light source not included in the plurality of first light sources among the plurality of light sources to be turned on, and the identification unit identifies the user based on a first eye image which is an eye image captured in a state where the plurality of first light sources are turned on and a second eye image which is an eye image captured in a state where the plurality of second light sources are turned on.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an electronic device, and more particularly to a technology for detecting a user's gaze and identifying the user based on an image of the user's eyes. [Background technology]

[0002] A technique for detecting a gaze based on an image of a user's eyes (for example, the technique disclosed in Patent Document 1) and a technique for identifying a person based on an image of a user's eyes (for example, the technique disclosed in Patent Document 2) have been proposed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-93624 [Patent Document 2] Japanese Patent Application Laid-Open No. 2004-23734 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, when gaze detection and personal identification are performed simultaneously using the same eye image, it is not possible to perform both gaze detection and personal identification with high accuracy. If an eye image for personal identification is acquired separately from the eye image for gaze detection, both gaze detection and personal identification can be performed with high accuracy, but the frequency of performing gaze detection and personal identification will decrease.

[0005] An object of the present invention is to provide a technology that can perform both user gaze detection and personal identification with high frequency and high accuracy. [Means for solving the problem]

[0006] The electronic device of the present invention comprises a control means for controlling the turning on of at least one of a plurality of light sources capable of irradiating light onto a user's eye; an acquisition means for acquiring an eye image which is an image of the eye captured with at least one of the plurality of light sources turned on; a gaze detection means for acquiring gaze information which is information regarding the gaze of the eye based on the eye image; and an identification means for identifying the user based on the eye image, wherein the control means controls the turning on of a plurality of first light sources among the plurality of light sources, and then controls the turning on of a plurality of second light sources among the plurality of light sources, including light sources not included in the plurality of first light sources, and the identification means identifies the user based on the first eye image which is an eye image captured with the plurality of first light sources turned on and the second eye image which is an eye image captured with the plurality of second light sources turned on. [Effects of the Invention]

[0007] According to the present invention, both the gaze detection and personal identification of a user can be performed with high frequency and high accuracy. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an external view of a digital camera. [Figure 2] FIG. 1 is a block diagram of a digital camera. [Figure 3] FIG. 2 is a schematic diagram of an eyepiece and an infrared light-emitting element. [Figure 4] FIG. 2 is a schematic diagram of an eye image. [Figure 5] FIG. 10 is a schematic diagram showing operation timing. [Figure 6] 10 is a flowchart showing the operation of the digital camera. [Figure 7] FIG. 2 is a schematic diagram of a display screen. [Figure 8] FIG. 1 is an external view of an HMD. [Figure 9] FIG. 1 is a block diagram of an HMD. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] (First embodiment) As a first embodiment of the present invention, an embodiment in which the present invention is applied to a digital camera, which is an imaging device capable of taking still images and moving images, will be described in detail with reference to the accompanying drawings.

[0011] <Device configuration> The configuration and functions of a digital camera 100 according to the first embodiment will be described with reference to FIGS. 1(A) to 3. FIG.

[0012] FIG. 1A is a front perspective view of the digital camera 100, and FIG. 1B is a rear perspective view of the digital camera 100. As shown in FIG.

[0013] The rear display unit 101 is a display device such as a liquid crystal panel or an organic EL panel provided on the rear surface of the digital camera 100 body outside the viewfinder, and displays images and various information so that the user can view them. For example, the rear display unit 101 displays captured (recorded) still images and videos, and displays live view images that show a subject in almost real time. The rear display unit 101 is provided with a touch panel 271. The touch panel 271 can detect contact (touch operation) with the display surface of the rear display unit 101 (the operation surface of the touch panel 271). The outside-finder display unit 243 is a display device such as a liquid crystal panel or an organic EL panel provided on the top surface of the digital camera 100 body, and displays various setting values ​​of the digital camera 100, such as shutter speed and aperture.

[0014] The shutter button 102 is a push-button type operating member used to issue a shooting instruction. The mode switch 103 is a dial type operating member used to switch between various modes. The mode switch 103 switches the operating mode of the system control unit 201 (described later) between a still image shooting mode and a video recording mode. The still image shooting modes include, for example, an auto shooting mode, an auto scene determination mode, a manual mode (M mode), an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). The still image shooting modes may also include various scene modes, program AE mode, and custom mode, which provide shooting settings for specific shooting scenes. The user directly switches to one of the multiple modes included in the still image shooting mode using the mode switch 103. Alternatively, the user may first switch to the still image shooting mode using the mode switch 103, and then use another operating member to switch to one of the multiple modes included in the still image shooting mode. Similarly, the video recording mode may also include multiple modes. The terminal cover 104 is a cover member that protects a connector (not shown) for connecting an external device to the digital camera 100 via a cable such as a USB cable.

[0015] The main electronic dial 105 is a rotary operation member, and by turning this main electronic dial 105, settings such as shutter speed and aperture can be changed. The power switch 106 is an operation member that switches the power of the digital camera 100 on and off. The control wheel 107 is a rotary operation member for moving the selection frame, scrolling through images, etc. The cross key 108 is a movement instruction member that can perform processing corresponding to the part of the cross key 108 that is pressed by pressing one of the four directional buttons consisting of the up button, down button, left button, and right button. The SET button 109 is a push button type operation member that is mainly used to confirm a selection item, etc.

[0016] The record button 110 is a push-button type operating member used to start or stop video shooting (recording). The enlargement button 111 is a push-button type operating member used to turn enlarged display on / off during live view and to change the magnification ratio during enlarged display. The enlargement button 111 is also used to enlarge the playback image and increase the magnification ratio in playback mode. After turning on the enlarged display, the live view can be enlarged or reduced by operating the main electronic dial 105. The AE lock button 112 is a push-button type operating member that can fix the exposure state by pressing it in a shooting standby state. The playback button 113 is a push-button type operating member used to switch between shooting mode and playback mode. Pressing the playback button 113 during shooting mode switches to playback mode, and the most recent image recorded on the recording medium 250 can be displayed on the rear display unit 101.

[0017] The menu button 114 is a push-button type operating member that, when pressed, displays a menu screen on the rear display unit 101 that allows various settings to be made. The user can intuitively make various settings using the menu screen displayed on the rear display unit 101, the cross key 108, and the SET button 109. The grip unit 115 has a shape that makes it easy for the user to hold the digital camera 100 in their right hand. When the digital camera 100 is held by gripping the grip unit 115 with the little finger, ring finger, and middle finger of the right hand, the shutter button 102 and main electronic dial 105 are located in positions that can be operated with the index finger of the right hand. In the same state, the sub electronic dial 107 is located in a position that can be operated with the thumb of the right hand. The lid 116 is a member that opens and closes a slot for inserting and removing a recording medium 250 into and from the digital camera 100.

[0018] The system control unit 201 (described later) performs display control, and the rear display unit 101 and the viewfinder display unit 229 (described later) function as an electronic viewfinder (hereinafter referred to as EVF) to display images (screens) according to the various operating modes described above. The user can view the image displayed on the viewfinder display unit 229 through the eyepiece unit 216, and can check the focus and composition of the subject image captured through the lens unit 200. The eyepiece detection unit 217 is disposed near the eyepiece unit 216 and can detect the approach of any object to the eyepiece unit 216. An infrared proximity sensor, for example, is used as the eyepiece detection unit 217. The communication terminal 210 is an electrical contact that enables the digital camera 100 to communicate with the lens unit 200 (described later).

[0019] 2 is a block diagram showing the internal configuration of the digital camera 100 and the lens unit 200. In FIG. 2, the same components as those in FIG. 1 are denoted by the same reference numerals.

[0020] The lens unit 200 has a photographing lens 207 and is detachable from the digital camera 100. The photographing lens 207 is usually made up of multiple lenses, but for simplicity's sake, only one lens is shown here. The communication terminal 206 is an electrical contact that allows the lens unit 200 to communicate with the digital camera 100. The communication terminal 210 is an electrical contact that allows the digital camera 100 to communicate with the lens unit 200. The lens control unit 204 built into the lens unit 200 communicates with the system control unit 201 via the communication terminal 206, and controls the aperture drive circuit 202 to drive the aperture 205. The lens control unit 204 also controls the AF drive circuit 203 to adjust the position of the photographing lens 207, thereby adjusting the focus.

[0021] The focal plane shutter 221 can freely control the exposure time of the imaging unit 222 in response to instructions from the system control unit 201. The imaging unit 222 is an image sensor composed of imaging elements such as CCD or CMOS that convert a subject image into an electrical signal. The A / D converter 223 converts the analog signal of each pixel output from the imaging unit 222 into, for example, a 10-bit digital signal. The image processing unit 224 performs predetermined pixel interpolation, resizing processing such as reduction, and color conversion processing on data from the A / D converter 223 or data from the memory control unit 215. The image processing unit 224 also performs predetermined arithmetic processing using the captured image data, and the system control unit 201 controls exposure and distance measurement based on the arithmetic results. This allows TTL (through-the-lens) AF (autofocus) processing, AE (autoexposure) processing, and EF (pre-flash) processing to be performed. The image processing unit 224 also performs predetermined arithmetic processing using the captured image data and performs TTL AWB (auto white balance) processing based on the arithmetic results.

[0022] The memory control unit 215 controls the exchange of data between the A / D converter 223, the image processing unit 224, and the memory 232. Digital data output from the A / D converter 223 is written to the memory 232 via the image processing unit 224 and the memory control unit 215, or via the memory control unit 215 without going through the image processing unit 224. The memory 232 stores image data obtained from the imaging unit 222 and the A / D converter 223, and image data to be displayed on the rear display unit 101 or the in-finder display unit 229. The memory 232 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio.

[0023] The memory 232 also serves as a memory (video memory) for image display. The D / A converter 219 converts image data for display stored in the memory 232 into an analog signal and supplies it to the rear display unit 101 or the in-finder display unit 229. The image data for display written to the memory 232 is displayed on the rear display unit 101 or the in-finder display unit 229 via the D / A converter 219. The rear display unit 101 or the in-finder display unit 229 performs display according to the analog signal from the D / A converter 219. In this way, by converting the digital signal stored in the memory 232 into an analog signal and sequentially transferring and displaying it on the rear display unit 101 or the in-finder display unit 229, an EVF function for live view (LV) display (through-image display) is realized. Various camera setting values ​​such as shutter speed and aperture are displayed on the outside-finder display unit 243 via an outside-finder display unit drive circuit 244.

[0024] The nonvolatile memory 256 is electrically erasable and recordable, such as an EEPROM. The nonvolatile memory 256 stores constants and programs for the operation of the system control unit 201. The programs referred to here refer to programs for executing the flowcharts described below. The system control unit 201 includes a CPU and an MPU that control the entire digital camera 100, and executes programs stored in the nonvolatile memory 256 to realize the processes in the flowcharts described below. The system memory 252 is a RAM or the like, and is also used as a work memory for expanding constants and variables for the operation of the system control unit 201, programs read from the nonvolatile memory 256, and the like. The system control unit 201 also controls display by controlling the memory 232, the D / A converter 219, the rear display unit 101, the viewfinder display unit 229, and the like.

[0025] The system timer 253 is a timing unit that measures the time used for various controls and the time of a built-in clock. The first shutter switch 211 and the second shutter switch 212 input the following operation instructions to the system control unit 201. The first shutter switch 211 is used when the shutter button 102 provided on the digital camera 100 is pressed halfway (i.e., when the shutter button is pressed halfway). The second shutter switch 212 is turned on when the shutter button 102 is pressed fully (a shooting instruction) and generates a first shutter switch signal SW1. The system control unit 201 receives the first shutter switch signal SW1 and starts AF processing, AE processing, AWB processing, EF processing, etc. The second shutter switch 212 is turned on when the shutter button 102 is pressed fully (a shooting instruction) and generates a second shutter switch signal SW2. The system control unit 201 starts a series of shooting processes, from reading out a signal from the imaging unit 222 to writing image data to the recording medium 250, based on the second shutter switch signal SW2.

[0026] The operation unit 270 is made up of operation members such as various switches and buttons that accept various operations from the user and notify the system control unit 201, and includes at least the following operation members: a mode change switch 103, a main electronic dial 105, a sub electronic dial 107, a cross key 108, a SET button 109, a record button 110, a magnification button 111, an AE lock button 112, a playback button 113, and a menu button 114.

[0027] The power supply control unit 280 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, etc., and detects whether a battery is installed, the battery type, and the remaining battery power. The power supply control unit 280 also controls the DC-DC converter based on the detection results and instructions from the system control unit 201, and supplies the required voltage for the required period to each unit, including the recording medium 250. The power supply unit 230 is composed of primary batteries such as alkaline batteries or lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, or Li-ion batteries, an AC adapter, etc.

[0028] The recording medium I / F 218 is an interface with a recording medium 250 such as a memory card or a hard disk. The recording medium 250 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like. The communication unit 254 is communicatively connected to an external device via a wireless antenna or a wired cable, and transmits and receives video and audio. The communication unit 254 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 254 can transmit image data (including live view images) captured by the imaging unit 222 and image files recorded on the recording medium 250 to an external device, and can also receive image data and various other information from an external device. Note that the communication unit 254 is not limited to a wireless LAN wireless communication interface (wireless communication module). For example, the communication unit 254 may be a wireless communication interface such as infrared communication, Bluetooth (registered trademark), Bluetooth (registered trademark) Low Energy, or Wireless USB. The communication unit 254 may also use a wired connection interface such as a USB cable, HDMI (registered trademark), or IEEE 1394.

[0029] The orientation detection unit 255 detects the orientation of the digital camera 100 with respect to the direction of gravity. Based on the orientation detected by the orientation detection unit 255, it is possible to determine whether an image captured by the imaging unit 222 was captured with the digital camera 100 held horizontally or vertically. The system control unit 201 can add orientation information corresponding to the orientation detected by the orientation detection unit 255 to the image file of the image captured by the imaging unit 222, or rotate and record the image. An acceleration sensor, a gyro sensor, or the like can be used as the orientation detection unit 255. By using an acceleration sensor or a gyro sensor, the orientation detection unit 255 can also detect movements of the digital camera 100 (panning, tilting, lifting, standing still, etc.).

[0030] The touch panel 271 is a touch sensor that can detect a touch operation on the rear display unit 101. The touch panel 271 and the rear display unit 101 can be configured as an integrated unit. For example, the touch panel 271 is configured so that the light transmittance does not interfere with the display of the rear display unit 101, and is attached to the upper layer of the display surface of the rear display unit 101. The input coordinates in the input screen are associated with the display coordinates on the rear display unit 101. This makes it possible to configure a GUI that makes it appear as if the user can directly operate the screen displayed on the rear display unit 101.

[0031] The system control unit 201 can detect the following operations or states on the touch panel 271. A finger or pen that has not been touching the touch panel 271 touches the touch panel 271 again. That is, the start of touching (hereinafter referred to as touch-down). A state in which the touch panel 271 is touched with a finger or a pen (hereinafter referred to as Touch-On). A finger or a pen is moved while touching the touch panel 271. That is, a movement of the touch (hereinafter referred to as Touch-Move). The finger or pen that has been touching the touch panel 271 is removed from the touch panel 271. That is, the touch ends (hereinafter referred to as "touch-up"). A state in which nothing is touching the touch panel 271 (hereinafter referred to as Touch-Off).

[0032] When a touch down is detected, a touch on is also detected at the same time. After a touch down, a touch on is usually continued to be detected unless a touch up is detected. A touch move is also detected when a touch on is detected. Even if a touch on is detected, a touch move will not be detected unless the touch position moves. After a touch up is detected for all fingers or pens that were touching, a touch off occurs.

[0033] These operation states and the position coordinates of the finger or pen touching the touch panel 271 are notified to the system control unit 201 via the internal bus. The system control unit 201 determines what kind of operation (touch operation) has been performed on the touch panel 271 based on the notified information. For touch moves, the direction of movement of the finger or pen on the touch panel 271 can also be determined. For touch moves, the vertical and horizontal components on the touch panel 271 can be determined based on changes in the position coordinates. If a touch move of a predetermined distance or more is detected, it is determined that a slide operation (drag) has been performed. An operation in which a finger is touched on the touch panel, moved quickly for a certain distance, and then released is called a flick. In other words, a flick is an operation in which a finger is quickly traced across the touch panel 271 as if flicking. If a touch move of a predetermined distance or more at a predetermined speed or faster is detected and a touch up is then detected, it can be determined that a flick has been performed (it can be determined that a drag was followed by a flick). Furthermore, a touch operation in which multiple points (for example, two points) are touched simultaneously and the touch positions are brought closer together is called a pinch in, and a touch operation in which the touch positions are moved farther apart is called a pinch out. Pinch out and pinch in are collectively called a pinch operation (or simply pinch). Touch panel 271 may use any of a variety of touch panel types, including a resistive film type, a capacitance type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type. Depending on the type, there are types that detect a touch by contact with the touch panel, and types that detect a touch by the approach of a finger or pen to the touch panel, but either type is acceptable.

[0034] The eyepiece detection unit 217 detects the approach (eyepiece) and departure (eye separation) of the eye (object) from the eyepiece unit 216 (eyepiece / eye separation detection). The system control unit 201 switches the display (display state) / non-display (non-display state) of the rear display unit 101 and the in-viewfinder display unit 229 according to the state detected by the eyepiece detection unit 217. At least when the operation mode is shooting mode and the display destination switching is automatic, the system control unit 201 switches the display destination to rear display when the eye is not in contact with the subject. The display unit 101 is used, and the in-viewfinder display unit 229 is not displayed.

[0035] Specifically, when an object approaches, infrared light emitted from a light-emitting unit (not shown) of the eyepiece detection unit 217 is reflected and incident on a light-receiving unit (not shown) of the infrared proximity sensor. The amount of incident infrared light received by the infrared proximity sensor can be used to detect the approach of an object to the eyepiece 216 and to determine the distance from the eyepiece 216 to the object (eyepiece distance). When the eyepiece detection unit 217 detects the approach of an object to the eyepiece 216, it transmits information about the object's approach to the eyepiece 216 to the system control unit 201. The system control unit 201 then starts displaying information on the viewfinder display unit 229. This allows the viewfinder display unit 229 to start displaying information with as little delay as possible when the user looks through the eyepiece 216.

[0036] Furthermore, when the object that the eyepiece detection unit 217 has detected as being close to the eye moves away by a predetermined distance or more from the state where the eyepiece detection unit 217 is detecting the eyepiece state (close state), the eyepiece detection unit 217 transmits information indicating that the object has left the eyepiece unit 216 to the system control unit 201. Then, the system control unit 201 stops display on the in-finder display unit 229 and starts display on the rear display unit 101.

[0037] The threshold for detecting eye contact and the threshold for detecting eye separation may be different, for example, by providing hysteresis. After detecting eye contact, the eye contact state is maintained until eye separation is detected. After detecting eye separation, the eye contact state is maintained until eye contact is detected. As a result, the system control unit 201 controls the display on the rear display unit 101 and the in-viewfinder display unit 229 according to the eye contact state or eye separation state detected by the eye contact detection unit 217. The eye contact detection unit 217 is not limited to an infrared proximity sensor, and may be any other sensor that can detect the approach of an eye or an object that can be considered as eye contact.

[0038] Digital camera 100 further has a gaze detection unit 260 that can detect information about the user's gaze when the user (photographer) is looking into eyepiece 216. Gaze detection unit 260 has a dichroic mirror 262, an imaging lens 263, a gaze detection sensor 264, a gaze detection circuit 265, and an infrared light emitting element 266, and detects not only the presence or absence of the user's gaze but also the position and movement of the gaze.

[0039] The infrared light-emitting element 266 is a diode that emits infrared light for detecting the user's gaze position within the viewfinder screen. The infrared light is emitted toward the center of the eyepiece 216 so that the infrared light is irradiated onto the user's eyeball (eye) 261. The infrared light emitted from the infrared light-emitting element 266 is reflected by the eyeball (eye) 261, and the reflected infrared light reaches the dichroic mirror 262. The dichroic mirror 262 has the function of reflecting only infrared light and transmitting visible light. The reflected infrared light, whose optical path has been changed, forms an image on the imaging surface of the gaze detection sensor 264 via the imaging lens 263. The imaging lens 263 is an optical element that constitutes the gaze detection optical system. The gaze detection sensor 264 includes an image sensor such as a CCD or CMOS. The gaze detection sensor 264 photoelectrically converts the incident reflected infrared light into an electrical signal and outputs it to the gaze detection circuit 265. Based on the output signal of the gaze detection sensor 264, the gaze detection circuit 265 detects the gaze position of the user from the movement of the user's eyeball (eye) 261 and the position of the pupil, and outputs the detected information to the system control unit 201. The gaze detection sensor 264 can detect the pupil of a person's eye, so it does not detect that a person's gaze is being input even if another object is approaching or in contact with the eyepiece unit 216. The eyepiece unit 216 has a function as a gaze operation unit. Note that the configuration of the gaze detection unit 260 is not limited to this.

[0040] The line of sight detection sensor 264 can detect the following information when the eyepiece 216 is placed close to the eye. Based on this information, the line of sight detection sensor 264 determines whether the person is looking into the eyepiece 216. The viewfinder 222 detects gaze information, which is information about the gaze of the eye. The gaze information is, for example, coordinate information indicating the gaze position, which is the position of the gaze on the screen displayed on the viewfinder display unit 229. The gaze information may also be angle information indicating the direction of the gaze. The detection state of at least one of the pupil edge 401 (shown in FIG. 4) and the corneal reflection image (Purkinje image = P image) 402 (shown in FIG. 4) of the eye looking into the eyepiece 216 The distance between the P images 402 (shown in FIG. 4), the amount of change in the position of the P image 402, and the direction of change in the position of the P image 402

[0041] As described above, since the gaze information is detected using infrared light emitted from the infrared light-emitting element 266, the detection accuracy of the gaze information decreases when light from another light source is irradiated onto the eye. Therefore, when using a light source of the eye proximity detection unit 217 in addition to the infrared light-emitting element 266, it is preferable to turn off the light source of the eye proximity detection unit 217 when performing gaze detection. Note that the user can enable / disable gaze detection by the gaze detection unit 260 (enable / disable gaze input that performs operations with the gaze) via a menu screen, for example.

[0042] Fig. 3 is a schematic diagram showing an example of the eyepiece 216 and infrared light emitting elements 266. There are no particular limitations on the arrangement or number of the infrared light emitting elements 266, but in Fig. 3, eight infrared light emitting elements 266a to 266h are arranged in a circle around the outer periphery of the eyepiece 216. The system control unit 201 can individually control the turning on / off of the infrared light emitting elements 266a to 266h.

[0043] <Control processing> Control of user gaze detection and personal identification in the first embodiment will be described with reference to Figures 4(A) to 7. Below, a control method that enables gaze detection and personal identification to be performed frequently and with high accuracy will be described.

[0044] 4(A) and 4(B) show images of the user's eye (eye image) captured by the gaze detection sensor 264 with at least one of the infrared light-emitting elements 266a to 266h turned on. When the eyeball (eye) is illuminated with infrared light, a bright spot called a P image 402 appears as a reflected image. The pupil is a hole in the center of the iris, and the pupil edge 401 can be detected from the difference in brightness between regions. FIG. 4(A) shows an eye image captured with the four upper infrared light-emitting elements 266a to 266d turned on. FIG. 4(B) shows an eye image captured with the four lower infrared light-emitting elements 266e to 266h turned on. By taking into account the positional relationship of the multiple P images 402, gaze detection accuracy can be improved. Therefore, the eye images in FIGS. 4(A) and 4(B) are suitable for gaze detection.

[0045] Fig. 4(C) shows an eye image suitable for personal identification. Fig. 4(C) is an eye image obtained with only one infrared light-emitting element 266 turned on. By reducing the number of infrared light-emitting elements 266 that are turned on, the number of P images 402 can be reduced, and the loss of eye images due to the P images 402 can be reduced. As a result, the accuracy of personal identification can be improved.

[0046] To summarize the above, it is preferable that a plurality of P images 402 exist in an eye image for gaze detection, and it is preferable that no P image 402 exists in an eye image for personal identification.

[0047] FIG. 4(D) shows an eye image (composite image) obtained by comparative dark combination of the eye image of FIG. 4(A) and the eye image of FIG. 4(B). In comparative dark combination, of two corresponding pixels (areas) in the two images, the darker pixel is adopted as the pixel of the composite image. The P image 402 generated by the infrared light-emitting element 266 is bright. Therefore, the P image 402 that exists only in one of the two eye images can be removed by comparative dark combination of the two eye images. The infrared light-emitting elements 266a to 266d that are turned on in FIG. 4(A) do not overlap with the infrared light-emitting elements 266e to 266h that are turned on in FIG. 4(B). In other words, the infrared light-emitting elements 266a to 266d that are turned on in FIG. 4(B) do not overlap with the infrared light-emitting elements 266e to 266h that are turned on in FIG. 4(B). None of the infrared light-emitting elements 266e-266h are included in the infrared light-emitting elements 266a-266d that are turned on in Figure 4(A). Therefore, by performing comparative dark synthesis of the eye image of Figure 4(A) and the eye image of Figure 4(B), the eye image of Figure 4(D) without the P image 402 can be obtained. The eye image of Figure 4(D) is suitable for personal identification.

[0048] 5(A) and 5(B) are diagrams showing the operation timing of the gaze detection unit 260 and the system control unit 201. Here, it is assumed that the gaze detection sensor 264 operates 30 times per second (the gaze detection sensor 264 acquires eye images at 30 fps). The system control unit 201 causes the gaze detection circuit 265 to perform gaze detection, but the following description will be given assuming that the system control unit 201 performs gaze detection.

[0049] FIG. 5(A) shows conventional operation timing. For gaze detection, the infrared light emitting element 266 is turned on (501gb), and the gaze detection sensor 264 performs image capture (light accumulation) 501ga. During the illumination 501gb, for example, multiple infrared light emitting elements 266 are illuminated. Then, the gaze detection sensor 264 outputs an eye image (reads out the accumulated results) 501gc, and the system control unit 201 performs gaze detection 501gd. Similarly, for gaze detection, the infrared light emitting element 266 is turned on (502gb), and the gaze detection sensor 264 performs image capture 502ga. During the illumination 502gb, for example, multiple infrared light emitting elements 266 are illuminated. Then, the gaze detection sensor 264 outputs an eye image 502gc, and the system control unit 201 performs gaze detection 502gd. Thereafter, for personal identification, the infrared light emitting element 266 is turned on (503gb), and the gaze detection sensor 264 performs image capture 503ga. In lighting 503gb, for example, one infrared light-emitting element 266 is turned on. Then, eye image output 503gc is performed by gaze detection sensor 264, and personal identification 503ge is performed by system control unit 201. Because gaze detection and personal identification are performed separately using different eye images, gaze detection and personal identification can be performed with high accuracy, but cannot be performed frequently. In FIG. 5(A), gaze detection is performed 20 times per second, and personal identification is performed only 10 times per second.

[0050] Fig. 5(B) shows the operation timing according to the first embodiment. As in Fig. 5(A), for gaze detection, lighting 501gb is performed by the infrared light emitting element 266, and image capture 501ga is performed by the gaze detection sensor 264. Then, eye image output 501gc is performed by the gaze detection sensor 264, and gaze detection 501gd is performed by the system control unit 201. Similarly, for gaze detection, lighting 502gb is performed by the infrared light emitting element 266, and image capture 502ga is performed by the gaze detection sensor 264. Then, eye image output 502gc is performed by the gaze detection sensor 264, and gaze detection 502gd is performed by the system control unit 201.

[0051] A plurality of infrared light emitting elements 266 are lit in each of lighting 501gb and lighting 502gb, but the plurality of infrared light emitting elements 266 that are lit in lighting 502gb include infrared light emitting elements 266 that are not included in the plurality of infrared light emitting elements 266 that are lit in lighting 501gb. For example, in lighting 501gb, infrared light emitting elements 266a to 266d are lit as shown in FIG. 4(A), and in lighting 502gb, infrared light emitting elements 266e to 266h are lit as shown in FIG. 4(B).

[0052] Furthermore, in Fig. 5(B), without capturing images solely for personal identification, the system control unit 201 performs personal detection 502ge based on two eye images obtained from lighting 501gb to output 502gc. For example, the eye image (composite image) of Fig. 4(D) is generated by comparative dark composition of the eye image of Fig. 4(A) and the eye image of Fig. 4(B). Then, personal identification is performed based on the composite image.

[0053] In Figure 5(B), imaging is not performed solely for personal identification, so gaze detection and personal identification are performed. In FIG. 5(B), gaze detection can be performed 30 times per second, and personal identification can be performed 15 times per second. Also, in FIG. 5(B), gaze detection and personal identification can be performed at high frequencies. Using the same eye image for gaze detection and personal identification allows only one of gaze detection and personal identification to be performed with high accuracy. Using eye images obtained by illuminating a large number of infrared light-emitting elements 266 allows gaze detection to be performed with high frequency, but personal identification cannot be performed with high accuracy. Using eye images obtained by illuminating a small number of infrared light-emitting elements 266 allows personal identification to be performed with high frequency, but gaze detection cannot be performed with high accuracy. In FIG. 5(B), gaze detection can be performed with high accuracy by using eye images obtained by illuminating multiple infrared light-emitting elements 266. Furthermore, in FIG. 5(B), using two eye images with different illuminating infrared light-emitting elements 266 allows personal identification to be performed with high accuracy (for example, by obtaining a composite image with fewer P images 402). In FIG. 5B, the gaze detection 502gd and the individual detection 502ge are performed in parallel, but the individual detection 502ge may be performed after the gaze detection 502gd, or may be performed before the gaze detection 502gd.

[0054] Fig. 6(A) is a flowchart showing the operation of digital camera 100, including gaze detection and personal identification. The operation of Fig. 6 is realized by system control unit 201 expanding a program stored in nonvolatile memory 256 into system memory 252 and executing the program. For example, when digital camera 100 is powered on, the operation of Fig. 6(A) starts, and is repeated until digital camera 100 is powered off.

[0055] In S600, the system control unit 201 determines whether or not the user's eyeball (eye) 261 is placed in contact with the eyepiece unit 216. The system control unit 201 waits until the eyeball (eye) 261 is placed in contact with the eyepiece unit 216, and when the eyeball (eye) is placed in contact with the eyepiece, the system control unit 201 proceeds to S601.

[0056] In S601, the system control unit 201 performs a process of determining which infrared light emitting elements 266 to turn on. FIG. 6B is a flowchart of the process of determining which infrared light emitting elements 266 to turn on. In S610, the system control unit 201 determines whether the eye image to be acquired is an odd-numbered eye image since the start of the operation in FIG. 6A. If it is an odd-numbered eye image, the process proceeds to S611; otherwise (if it is an even-numbered eye image), the process proceeds to S612. In S611, the system control unit 201 determines the plurality of first infrared light emitting elements 266 as the infrared light emitting elements 266 to turn on. In S612, the system control unit 201 determines the plurality of second infrared light emitting elements 266 as the infrared light emitting elements 266 to turn on. What is important here is that, in order to improve the accuracy of personal identification, the plurality of second infrared light emitting elements 266 include infrared light emitting elements 266 that are not included in the plurality of first infrared light emitting elements 266. It is preferable that the plurality of first infrared light emitting elements 266 and the plurality of second infrared light emitting elements 266 do not overlap. For example, in S611, infrared light emitting elements 266a to 266d are determined (selected) as shown in Fig. 4(A), and in S612, infrared light emitting elements 266e to 266h are determined as shown in Fig. 4(B). Note that the infrared light emitting elements 266 to be turned on may be determined based on the attitude of the digital camera 100 detected by the attitude detection unit 255, the previous line of sight position, etc.

[0057] Returning to the explanation of Fig. 6(A), in S602, the system control unit 201 turns on the infrared light emitting element 266 determined in S601.

[0058] In S603, the system control unit 201 causes the line-of-sight detection sensor 264 to capture an image (accumulate light). The accumulation is carried out for a predetermined time.

[0059] In S604, the system control unit 201 turns off the infrared light emitting element 266 that was turned on in S601.

[0060] In S605, the system control unit 201 reads out the eye image from the gaze detection sensor 264. The system control unit 201 stores the read out eye image in the memory 232. can store multiple eye images.

[0061] In S606, the system control unit 201 performs gaze detection (calculation of gaze position) using the eye image acquired in S605.

[0062] In S607, the system control unit 201 determines whether the eye image acquired in S605 is an odd-numbered eye image since the start of the operation of Fig. 6(A). If it is an odd-numbered eye image, the process proceeds to S609, and if not (if it is an even-numbered eye image), the process proceeds to S608.

[0063] In S608, the system control unit 201 performs personal identification. FIG. 6(C) is a flowchart of personal identification. In S620, the system control unit 201 generates a composite image from which the P image 402 has been removed by comparative dark combination of odd-numbered eye images (eye images in the previous frame) and even-numbered eye images (eye images in the current frame). In S621, the system control unit 201 performs personal identification processing using the composite image generated in S620. The system control unit 201 may erase the eye images stored in the memory 232 when personal identification is completed. The method of personal identification processing is not particularly limited, but for example, the personal identification processing is iris authentication processing in which a registered iris is compared with the iris in the composite image.

[0064] Returning to the explanation of Fig. 6(A), in S609, the system control unit 201 determines whether or not the user's eyeball (eye) 261 is placed in contact with the eyepiece unit 216. If the eyeball (eye) 261 is placed in contact with the eyepiece unit 216, the process proceeds to S601, and if the eyepiece is released, the operation of Fig. 6(A) ends.

[0065] Note that if the eyes move significantly (if the gaze changes significantly), it may be difficult to properly generate an eye image (synthetic image) for personal identification. Therefore, the system control unit 201 may determine whether the amount of change in the gaze is greater than a threshold based on the eye image, and may not perform personal identification if the amount of change in the gaze is greater than the threshold. The method for determining whether the amount of change in the gaze is greater than the threshold is not particularly limited. For example, the determination of whether the amount of change in the gaze is greater than the threshold may be based on whether the amount of change in the cornea in the eye image is greater than a threshold, or may be based on whether the amount of change in gaze information (gaze position) obtained by gaze detection is greater than a threshold. The system control unit 201 may control the system to capture only one frame of the eye image as shown in FIG. 4(C) if the amount of change in the gaze is greater than the threshold.

[0066] In FIG. 6, personal identification is performed every time two frames of eye images are acquired. However, personal identification may also be performed every time N frames of eye images (N is an integer equal to or greater than 3, for example, 10) are acquired. This reduces the frequency of personal identification and the processing load. Here, the N frames of eye images may include one or more eye images not used for personal identification. When the N frames of eye images include multiple eye images not used for personal identification, the multiple eye images may be multiple eye images captured with the same light source turned on. The eye images not used for personal identification may be eye images in which a larger number of infrared light-emitting elements 266 are turned on than the eye images used for personal identification. For example, the eye images not used for personal identification may be eye images captured with all infrared light-emitting elements 266 turned on. Using such eye images enables more accurate gaze detection. The system control unit 201 may determine the value of N according to the operation mode of the digital camera 100, or may determine a value (number) designated by the user as the value of N.

[0067] The system control unit 201 may perform control so that a predetermined notification is given to the user when personal identification of the user is successful. The predetermined notification may be, for example, a notification that personal identification has been successful, such as displaying an icon, outputting a sound, or turning on a lamp.

[0068] Fig. 7 shows an example of a screen displayed on the viewfinder display unit 229. Fig. 7 shows the screen when personal identification is successful. The AF frame 700 is an item that indicates the position to focus on. Upon receiving the first shutter switch signal SW1, the system control unit 201 starts AF processing to focus on the subject at the position of the AF frame 700.

[0069] The shooting condition display section 701 is an item that indicates the shooting conditions, and in Fig. 7, it indicates still image shooting conditions. For example, the shooting condition display section 701 indicates the still image shooting mode, shutter speed, aperture, exposure level, ISO sensitivity, etc. In Fig. 7, the still image shooting mode is M mode, the shutter speed is 1 / 125 seconds, the aperture is F5.6, the exposure level is proper exposure, and the ISO sensitivity is 100. Note that information such as the AF method, continuous shooting setting, and the number of shots that can be taken may also be displayed.

[0070] The personal identification completion icon 702 is an item that is displayed when personal identification is successful. In Fig. 7, the personal identification completion icon 702 indicates that the current user has been identified as a person named User1 who has been registered in advance. Note that the system control unit 201 may hide the personal identification completion icon 702 when a predetermined time has elapsed since the display of the personal identification completion icon 702 began.

[0071] Although an example of generating an eye image for personal identification by comparative dark compositing of two consecutive frames of eye images has been described, this is not limiting. More than two frames of eye images may be composited, and the multiple eye images to be composited do not have to be captured consecutively. As long as the number of P images 402 in the composite image is smaller than the number of P images 402 in each eye image used for the composite, a compositing method other than comparative dark compositing may be used. A composite image may be generated by stitching together areas where no P image 402 exists based on the infrared light-emitting element 266 (light source) that is turned on. For example, a composite image may be generated by stitching together the lower half of the eye image in FIG. 4(A) and the upper half of the eye image in FIG. 4(B). It is sufficient to use eye information from the portion where no P image 402 exists; it is not necessary to generate a composite image.

[0072] Although an example has been described in which the multiple infrared light emitting elements 266 that are turned on do not completely overlap among the multiple eye images to be synthesized, it is also possible for the multiple infrared light emitting elements 266 to be turned on to partially overlap. If the multiple infrared light emitting elements 266 that are turned on do not partially overlap among the multiple eye images to be synthesized, the P image 402 can be removed, thereby improving the accuracy of personal identification.

[0073] As described above, according to the first embodiment, in a digital camera, both the detection of a user's line of sight and personal identification can be performed with high frequency and high accuracy.

[0074] (Second embodiment) As a second embodiment of the present invention, an embodiment in which the present invention is applied to a head mounted display (HMD), which is one type of head mounted device, will be described in detail with reference to the accompanying drawings.

[0075] In recent years, XR (Extended Reality) technology has become well known, which combines the real world and the virtual world to create new experiences. XR technology is a general term for VR (Virtual Reality), MR (Mixed Reality), AR (Augmented Reality), and SR (Substitutional Reality). VR technology allows users to experience virtual reality. MR technology allows users to experience mixed reality, which is a seamless blend of reality and virtual reality. AR technology allows users to project a virtual world onto real space and show it to them. SR technology allows users to project past footage onto the real world and show it to them. This can be done.

[0076] One XR system that uses XR technology is a system that uses an HMD. In such an XR system, the actual surroundings and virtual objects are displayed on the HMD and presented to the user (the HMD wearer), allowing the user to experience the XR space. The HMD also has the function of detecting the user's gaze position, and may use this gaze position information to control the display.

[0077] <Device configuration> The configuration and functions of the HMD 800 according to the second embodiment will be described below with reference to FIGS.

[0078] 8 is an external view of the HMD 800. The HMD 800 is a display device worn on the user's head, and includes a display unit, operation members, an image processing unit, and the like. For example, the HMD 800 includes multiple operation members for operating the HMD 800, such as a power switch for turning the power of the HMD 800 on and off, and setting buttons for setting parameters for the HMD 800. The image processing unit generates virtual objects, synthesizes images, and the like.

[0079] The head mounting member 801 is a member for stably fixing the HMD 800 to the user's head. The head mounting member 801 fixes the HMD 800 to the head so that the HMD 800 does not shift relative to the head even if the user's head moves. The head mounting member 801 may be a member that is wrapped around the user's head or a member that is hung on the user's ears.

[0080] The display unit 802 presents virtual objects, composite images of real images and virtual objects, etc., in front of the user's eyes. The display unit 802 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display, and includes a display panel such as a liquid crystal panel or an organic EL panel, a driver circuit that controls the display panel, and a memory that stores images to be displayed. The display unit 802 may be a non-transmissive display unit that does not allow the outside scene to be directly viewed, or an optically transmissive display unit that allows the outside scene to be directly viewed.

[0081] The imaging unit 803 is a device (camera unit) that captures an image of the environment around the user. The imaging unit 803 may capture an image of the environment in front of the user as the environment around the user. The area in front of the user may refer to the area directly in front of the user's head.

[0082] 9 is a block diagram showing the internal configuration of an HMD 800. The HMD shown in FIG. 9 includes an imaging unit 900, a line-of-sight detection unit 901, a movement detection unit 902, a rotation detection unit 903, an information processing unit 904, an image processing unit 905, and a display unit 906. The imaging unit 900 corresponds to the imaging unit 803 in FIG. 8, and the display unit 906 corresponds to the display unit 802 in FIG. 8. The information processing unit 904 includes a captured image acquisition unit 907, a virtual object holding unit 908, a virtual object generation unit 909, a line-of-sight information acquisition unit 910, an HMD movement amount calculation unit 911, an HMD rotation amount calculation unit 912, and an HMD operation determination unit 913. The image processing unit 905 includes a virtual object selection unit 914, a virtual object control unit 915, and a display image generation unit 916.

[0083] The imaging unit 900 has an optical system, an image sensor, a driver circuit for controlling the image sensor, an A / D conversion circuit for converting a signal acquired by the image sensor into a digital signal, and a development circuit for developing the acquired digital signal as an image. Image data captured by the imaging unit 900 is acquired by a captured image acquisition unit 907 of the information processing unit 904 and transmitted to a display image generation unit 916 of the image processing unit 905. The display image generation unit 916 generates a display image by combining the image captured by the imaging unit 900 (captured image; real image) with a virtual object (to be described later). Furthermore, the captured image acquisition unit 907 transmits the captured image to the virtual object generation unit 909, causing the virtual object generation unit 909 to generate a virtual object.

[0084] The gaze detection unit 901 is a device for detecting the gaze direction of a user who is wearing an HMD. For example, the gaze detection unit 901 is a device that irradiates the user's eye with infrared light from an infrared light-emitting element, as used in a single-lens reflex camera, and detects the gaze direction from the relationship between a P image (a reflected image of the light source due to corneal reflection) and the pupil. The gaze information detected by the gaze detection unit 901 is acquired by a gaze information acquisition unit 910 of the information processing unit 904 and transmitted to a virtual object control unit 915 of the image processing unit 905. The virtual object control unit 915 controls virtual objects, which will be described later, and transmits virtual object data to a display image generation unit 916 of the image processing unit 905. The display image generation unit 916 generates a display image by combining a captured image and a virtual object.

[0085] The movement detection unit 902 is a device for detecting movement of the HMD 800, and detects movement of the HMD 800 using, for example, position information from a Global Positioning System (GPS), an acceleration sensor, etc. The movement information detected by the movement detection unit 902 is transmitted to an HMD movement amount calculation unit 911 of the information processing unit 904. The HMD movement amount calculation unit 911 calculates the amount of movement of the HMD 800 based on the movement information, and transmits the calculated amount to an HMD operation determination unit 913.

[0086] The rotation detection unit 903 is a device for detecting the rotation of the HMD 800. Rotation information detected by the rotation detection unit 903 is transmitted to an HMD rotation amount calculation unit 912 of the information processing unit 904. The HMD rotation amount calculation unit 912 calculates the amount of rotation in the HMD 800 based on the rotation information, and transmits it to an HMD operation determination unit 913.

[0087] The HMD operation determination unit 913 determines the operation of the HMD 800 based on the amount of movement (direction and magnitude of movement) of the HMD 800 transmitted from the HMD movement amount calculation unit 911 and the amount of rotation (direction and magnitude of rotation) of the HMD 800 transmitted from the HMD rotation amount calculation unit 912. The operation information determined by the HMD operation determination unit 913 is transferred to a virtual object control unit 915 of the image processing unit 905 and used to control a virtual object, which will be described later.

[0088] The virtual object holding unit 908 holds virtual space data (virtual object data), such as data (shape information and position and orientation information) related to virtual objects that make up the virtual space and data related to a virtual light source that illuminates the virtual space. The virtual object selection unit 914 of the image processing unit 905 selects the virtual object data held in the virtual object holding unit 908 and transmits it to the virtual object control unit 915. The virtual object holding unit 908 also transmits the virtual object data to the virtual object generation unit 909, causing it to generate another virtual object.

[0089] The virtual object generation unit 909 generates a virtual object based on the image data transmitted from the captured image acquisition unit 907 and the virtual object data transmitted from the virtual object storage unit 908. The virtual object generation unit 909 transmits the generated virtual object data to a virtual object selection unit 914 of the image processing unit 905.

[0090] The virtual object selection unit 914 selects at least one of the virtual object data held in the virtual object holding unit 908 and the virtual object data generated by the virtual object generation unit 909 as virtual object data to be displayed. Then, the virtual object selection unit 914 transmits the virtual object data to be displayed to the virtual object control unit 915. Note that the virtual object data to be displayed includes image data of the virtual object, position data, data on the order of overlap with other virtual objects, etc. .

[0091] The virtual object control unit 915 controls the virtual object data transmitted from the virtual object selection unit 914 based on the line-of-sight information transmitted from the line-of-sight information acquisition unit 910 and the operation information of the HMD 800 transmitted from the HMD operation determination unit 913. Then, the virtual object control unit 915 transmits the controlled virtual object data to the display image generation unit 916.

[0092] The display image generation unit 916 generates a display image by combining the captured image data sent from the captured image acquisition unit 907 and the virtual object data sent from the virtual object control unit 915, and sends the display image to the display unit 906.

[0093] <Control processing> The HMD 800 described above can perform gaze detection and personal identification, similar to the digital camera 100 of the first embodiment. Also, similar to the first embodiment, both gaze detection and personal identification of a user can be performed with high frequency and high accuracy. Due to these effects, for example, a user can wear an HMD and conveniently perform web shopping. In web shopping, for example, a product is selected based on the user's gaze information (gaze position), and payment is made based on personal identification. Note that gaze detection and personal identification may be performed by an external device, such as a personal computer (PC), connected to the HMD.

[0094] When gaze detection fails, it is highly likely that the user has changed to another user. Therefore, personal identification must be performed. When gaze detection fails frequently, personal identification must be performed more frequently. As described in the first embodiment, in conventional technology, gaze detection and personal identification are performed separately using different eye images, so although gaze detection and personal identification can be performed with high accuracy, they cannot be performed frequently. According to the first and second embodiments, both user gaze detection and personal identification can be performed with high accuracy and frequency.

[0095] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0096] The above processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0097] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0098] Furthermore, in the above-described embodiments, the present invention has been described with reference to examples in which it is applied to a digital camera or an HMD. However, this is not limited to this example, and the present invention can be applied to any electronic device capable of performing gaze detection and personal identification based on eye images. For example, the present invention can be applied to personal computers, PDAs, mobile phone terminals, portable image viewers, printers, digital photo frames, music players, game consoles, e-book readers, etc. The present invention can also be applied to video players, display devices (including projectors), tablet terminals, smartphones, AI speakers, home appliances, in-vehicle devices, etc.

[0099] (Other embodiments) The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more functions.

[0100] The disclosure of this embodiment includes the following configuration, method, program, and medium. (Configuration 1) a control means for controlling at least one of a plurality of light sources capable of irradiating light onto the user's eyes to be turned on; an acquisition means for acquiring an eye image, which is an image of the eye captured in a state where at least one of the plurality of light sources is turned on; a gaze detection means for acquiring gaze information, which is information about the gaze of the eye, based on the eye image; an identification means for identifying the user based on the eye image; and the control means controls a plurality of first light sources among the plurality of light sources to be turned on, and then controls a plurality of second light sources among the plurality of light sources to be turned on, the second light sources including a light source not included in the plurality of first light sources; The identification means identifies the user based on a first eye image that is an eye image captured with the plurality of first light sources turned on and a second eye image that is an eye image captured with the plurality of second light sources turned on. An electronic device characterized by: (Configuration 2) The first eye image and the second eye image are two consecutive frames of eye images. 2. The electronic device according to configuration 1. (Configuration 3) a generating means for generating a composite image in which the Purkinje image is smaller than the Purkinje image in the first eye image and the Purkinje image in the second eye image by combining the first eye image and the second eye image; and The identification means identifies the user based on the composite image. 3. The electronic device according to configuration 1 or 2. (Configuration 4) A generating means for generating a composite image by comparatively dark-combining the first-eye image and the second-eye image. and The identification means identifies the user based on the composite image. 3. The electronic device according to configuration 1 or 2. (Configuration 5) None of the plurality of second light sources is included in the plurality of first light sources. 5. The electronic device according to any one of configurations 1 to 4. (Configuration 6) The identification means identifies the user each time N frames (N is an integer equal to or greater than 3) of eye images including the first eye image and the second eye image are acquired. 6. The electronic device according to any one of configurations 1 to 5. (Configuration 7) The N is a number according to the operation mode of the electronic device or a number designated by the user. 7. The electronic device according to configuration 6. (Configuration 8) The N frames of eye images include a third eye image that is not used to identify the user. 8. The electronic device according to configuration 6 or 7. (Configuration 9) the N frames of eye images include a plurality of third-eye images; The plurality of third-eye images are a plurality of eye images captured with the same light source turned on. 9. The electronic device according to configuration 8. (Configuration 10) The third eye image is an eye image captured in a state where a plurality of third light sources out of the plurality of light sources are turned on, the number of third light sources being greater than each of the plurality of first light sources and the plurality of second light sources. 10. The electronic device according to configuration 8 or 9. (Configuration 11) A determination means for determining whether the amount of change in the line of sight is greater than a threshold value based on the eye image. and When it is determined that the amount of change in the line of sight is greater than a threshold, the identification means does not identify the user. 11. The electronic device according to any one of configurations 1 to 10. (Configuration 12) The determining means determines whether the amount of change in the line of sight is greater than a threshold by determining whether the amount of change in the cornea in the eye image is greater than a threshold. 12. The electronic device according to configuration 11. (Configuration 13) The determining means determines whether the amount of change in the line of sight information is greater than a threshold value, thereby determining whether the amount of change in the line of sight is greater than a threshold value. 12. The electronic device according to configuration 11. (Configuration 14) a notification control means for controlling to give a predetermined notification to the user when the user is successfully identified; Further having 14. The electronic device according to any one of configurations 1 to 13. (method) a control step of controlling at least one of a plurality of light sources capable of irradiating light onto the user's eyes to be turned on; an acquisition step of acquiring an eye image, which is an image of the eye captured in a state where at least one of the plurality of light sources is turned on; a gaze detection step of acquiring gaze information, which is information about a gaze of the eye, based on the eye image; an identification step of identifying the user based on the eye image; and In the control step, after controlling to turn on a plurality of first light sources among the plurality of light sources, control to turn on a plurality of second light sources among the plurality of light sources, the second light sources including a light source not included in the plurality of first light sources; In the identifying step, the user is identified based on a first eye image that is an eye image captured with the plurality of first light sources turned on and a second eye image that is an eye image captured with the plurality of second light sources turned on. A method for controlling an electronic device. (program) A program for causing a computer to function as each means of the electronic device according to any one of configurations 1 to 14. (medium) A computer-readable storage medium storing a program for causing a computer to function as each means of the electronic device according to any one of configurations 1 to 14. [Explanation of symbols]

[0101] 100: Digital camera 201: System control unit 800: HMD 904: Information processing unit

Claims

1. a control means for controlling at least one of a plurality of light sources capable of irradiating light onto the user's eyes to be turned on; an acquisition means for acquiring an eye image, which is an image of the eye captured in a state where at least one of the plurality of light sources is turned on; a gaze detection means for acquiring gaze information, which is information about the gaze of the eye, based on the eye image; an identification means for identifying the user based on the eye image; and the control means controls a plurality of first light sources among the plurality of light sources to be turned on, and then controls a plurality of second light sources among the plurality of light sources to be turned on, the second light sources including a light source not included in the plurality of first light sources; The identification means identifies the user based on a first eye image that is an eye image captured with the plurality of first light sources turned on and a second eye image that is an eye image captured with the plurality of second light sources turned on. An electronic device characterized by:

2. The first eye image and the second eye image are two consecutive frames of eye images.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

3. a generating means for generating a composite image in which the Purkinje image is smaller than each of the Purkinje image in the first eye image and the Purkinje image in the second eye image by combining the first eye image and the second eye image; and The identification means identifies the user based on the composite image.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

4. a generating means for generating a composite image by comparatively dark-combining the first-eye image and the second-eye image; and The identification means identifies the user based on the composite image.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

5. None of the plurality of second light sources is included in the plurality of first light sources.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

6. The identification means identifies the user each time N frames (N is an integer equal to or greater than 3) of eye images including the first eye image and the second eye image are acquired.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

7. The N is a number according to the operation mode of the electronic device or a number designated by the user.

7. The electronic device according to claim 6, wherein the electronic device is a semiconductor device.

8. The N frames of eye images include a third eye image that is not used to identify the user.

7. The electronic device according to claim 6, wherein the electronic device is a semiconductor device.

9. the N frames of eye images include a plurality of third-eye images; The plurality of third eye images are a plurality of eye images captured with the same light source turned on.

9. The electronic device according to claim 8.

10. The third eye image is an eye image captured in a state in which a plurality of third light sources, which are greater than the plurality of first light sources and the plurality of second light sources, are turned on among the plurality of light sources.

9. The electronic device according to claim 8.

11. A determination means for determining whether the amount of change in the line of sight is greater than a threshold value based on the eye image. and When it is determined that the amount of change in the line of sight is greater than a threshold, the identification means does not identify the user.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

12. The determining means determines whether the amount of change in the line of sight is greater than a threshold by determining whether the amount of change in the cornea in the eye image is greater than a threshold.

12. The electronic device according to claim 11.

13. The determining means determines whether the amount of change in the line of sight information is greater than a threshold value, thereby determining whether the amount of change in the line of sight is greater than a threshold value.

12. The electronic device according to claim 11.

14. a notification control means for controlling to give a predetermined notification to the user when the user is successfully identified; Further having 2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

15. a control step of controlling at least one of a plurality of light sources capable of irradiating light onto the user's eyes to be turned on; an acquisition step of acquiring an eye image, which is an image of the eye captured in a state where at least one of the plurality of light sources is turned on; a gaze detection step of acquiring gaze information, which is information about a gaze of the eye, based on the eye image; an identification step of identifying the user based on the eye image; and In the control step, after controlling a plurality of first light sources among the plurality of light sources to be turned on, a plurality of second light sources including a light source not included in the plurality of first light sources among the plurality of light sources are controlled to be turned on; In the identifying step, the user is identified based on a first eye image that is an eye image captured with the plurality of first light sources turned on and a second eye image that is an eye image captured with the plurality of second light sources turned on. A method for controlling an electronic device.

16. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 14.

17. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 14.

Images