Identification device

Abstract

To provide a technique that enables highly accurate identification (authentication) of users (persons) using a simple configuration.SOLUTION: An identification device of the present invention comprises image acquisition means configured to acquire a captured image of eyeballs of a user, information acquisition means configured to acquire three-dimensional information of the eyeballs based on the image, and identification means configured to identify the user on the basis of the three-dimensional information.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an identification device for identifying a person. [Background technology]

[0002] As a method for identifying (authenticating) a person based on a facial image of the person captured by a camera, a method for identifying a person based on the iris part has been proposed. For example, Patent Document 1 proposes a method for identifying a person by extracting the iris part from an image of the person's eye (eyeball), encoding the iris part, and comparing the iris part code with a reference code. Patent Document 2 proposes a method for improving the success rate of identification based on the iris part by further using additional information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 8-504979 [Patent Document 2] Japanese Patent Application Publication No. 11-007535 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technologies disclosed in Patent Documents 1 and 2 identify people based on two-dimensional iris patterns, making it possible for people to be impersonated using contact lenses with printed iris patterns or video of the eye. In other words, people cannot be identified with high accuracy. Using an image sensor capable of obtaining high-resolution images (images with a large number of pixels) of the eye, or a high-performance CPU with the processing speed to handle high-resolution images, would enable high-accuracy identification of people based on two-dimensional iris patterns, but this would increase costs.

[0005] An object of the present invention is to provide a technology that can identify (authenticate) a user (person) with a simple configuration and high accuracy. [Means for solving the problem]

[0006] A first aspect of the present invention is a method for acquiring an image of a user's eyeball, and an information acquisition means for acquiring three-dimensional information of the eyeball based on the image; a feature acquisition means for inputting the image acquired by the image acquisition means and the three-dimensional information acquired by the information acquisition means into a convolutional neural network to acquire a feature of the user; The aforementioned Features and an identification means for identifying the user based on the above.

[0007] A second aspect of the present invention provides a method for detecting an eyeball of a user, the method comprising: acquiring an image of the eyeball of a user; and acquiring three-dimensional information of the eyeball based on the image. inputting the image and the three-dimensional information into a convolutional neural network to acquire features of the user; The aforementioned Features and identifying the user based on the

[0008] A third aspect of the present invention is a program for causing a computer to function as each means of the above-mentioned identification device.A fourth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each means of the above-mentioned identification device. [Effects of the Invention]

[0009] According to the present invention, a user (person) can be identified (authenticated) with a simple configuration and with high accuracy. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is an external view of a camera according to an embodiment of the present invention. [Figure 2] FIG. 2 is a cross-sectional view of the camera according to the embodiment. [Figure 3] FIG. 1 is a block diagram of a camera according to an embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing a field of view within a finder according to the present embodiment. [Figure 5] FIG. 2 is a block diagram of a CPU according to the present embodiment. [Figure 6]FIG. 1 is a diagram illustrating an optical system according to an embodiment of the present invention. [Figure 7] FIG. 2 is a diagram showing an eye image according to the present embodiment. [Figure 8] 10 is a flowchart of a personal identification operation according to the present embodiment. [Figure 9] 10 is a correspondence table between feature amounts and people according to the present embodiment. [Figure 10] 10 is a flowchart of a three-dimensional information acquisition operation according to the present embodiment. [Figure 11] 10A and 10B are diagrams showing a pair of corneal reflection images according to the present embodiment. [Figure 12] FIG. 2 is a diagram showing the arrangement of light sources according to the present embodiment. [Figure 13] 10 is a graph showing the relationship between ΔP, Z, and R according to the present embodiment. [Figure 14] 10 is a graph showing the relationship between ΔP, Z, and R according to the present embodiment. [Figure 15] FIG. 2 is a diagram showing the state inside an eyeball according to the present embodiment. [Figure 16] FIG. 1 is a diagram illustrating a configuration of a CNN according to an embodiment of the present invention. [Figure 17] 1A and 1B are diagrams illustrating a feature detection process and a feature integration process according to the present embodiment. [Figure 18] FIG. 10 is an external view of another electronic device to which the present invention can be applied. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0012] <Configuration explanation> 1(a) and 1(b) show the exterior of a camera 1 (digital still camera; interchangeable lens camera) according to this embodiment. As will be described in detail later, the camera 1 has the function of acquiring information about the user's (person's) eyeballs and the function of identifying (authenticating) the user. FIG. 1(a) is a front perspective view, and FIG. 1(b) is a rear perspective view. As shown in FIG. 1(a), the camera 1 has a photographing lens unit 1A and a camera housing 1B. A release button 5, which is an operation member that accepts an image capture operation from the user (photographer), is disposed on the camera housing 1B. As shown in FIG. 1(b), an eyepiece window frame 121 and an eyepiece 12 are disposed on the rear of the camera housing 1B, through which the user can view a display device 10 (display panel) (described later) contained within the camera housing 1B. Operation members 41 to 43 that accept various operations from the user are also disposed on the rear of the camera housing 1B. For example, operation member 41 is a touch panel that accepts touch operations, operation member 42 is an operation lever that can be pushed in any direction, and operation member 43 is a four-way key that can be pressed in each of four directions. Operation member 41 (touch panel) is equipped with a display panel such as a liquid crystal panel and has the function of displaying images on the display panel. In addition, four light sources 13a to 13d that illuminate the user's eyes are provided around eyepiece lens 12.

[0013] FIG. 2 is a cross-sectional view of the camera 1 taken along the YZ plane defined by the Y axis and Z axis shown in FIG. 1(a), and shows the general internal configuration of the camera 1. As shown in FIG.

[0014] The photographing lens unit 1A contains two lenses 101 and 102, an aperture 111, an aperture driver 112, a lens drive motor 113, a lens drive member 114, a photocoupler 115, a pulse plate 116, a mount contact 117, and a focus adjustment circuit 118. The lens drive member 114 is made up of a drive gear and the like, and the photocoupler 115 detects the rotation of the pulse plate 116 that is linked to the lens drive member 114 and transmits this to the focus adjustment circuit 118. The focus adjustment circuit 118 drives the lens drive motor 113 based on information from the photocoupler 115 and information from the camera housing 1B (information on the lens drive amount), moves the lens 101, and changes the focus position. The mount contact 117 connects the photographing lens unit 1A and the camera housing It is an interface with 1B. For simplicity, two lenses 101 and 102 are shown, but in reality, more than two lenses are included in taking lens unit 1A.

[0015] The camera housing 1B contains an image sensor 2, a CPU 3, a memory unit 4, a display device 10, a display device drive circuit 11, and the like. The image sensor 2 is located at the intended imaging plane of the photographing lens unit 1A. The CPU 3 is the central processing unit of the microcomputer, and controls the entire camera 1. The memory unit 4 stores images captured by the image sensor 2, etc. The display device 10 is composed of a liquid crystal or the like, and displays the captured image (subject image), etc. on the screen (display surface) of the display device 10. The display device drive circuit 11 drives the display device 10. The user can view the screen of the display device 10 through the eyepiece window frame 121 and the eyepiece 12.

[0016] The camera housing 1B also contains light sources 13a-13d, a beam splitter 15, a light-receiving lens 16, and an eye image sensor 17. Light sources 13a-13d are conventionally used in single-lens reflex cameras and the like to detect the line of sight from the relationship between the pupil and the image of light reflected by the cornea (corneal reflection image), and are used to illuminate the user's eyeball 14. Specifically, light sources 13a-13d are infrared light-emitting diodes or the like that emit infrared light insensitive to the user, and are arranged around the eyepiece 12. An optical image of the illuminated eyeball 14 (eye image; an image formed by light emitted from light sources 13a-13d and reflected by the eyeball 14) passes through the eyepiece 12 and is reflected by the beam splitter 15. The eye image is then focused by the light-receiving lens 16 onto the eye image sensor 17, which is a two-dimensional array of photoelectric elements such as a CCD or CMOS. The light receiving lens 16 positions the pupil of the eyeball 14 and the ocular imaging element 17 in a conjugate imaging relationship. Using a predetermined algorithm described later, the line of sight of the eyeball 14 is detected from the position of the corneal reflection image in the eyeball image formed on the ocular imaging element 17. Specifically, information about the line of sight, such as the line of sight direction (direction of the line of sight) and the viewpoint (position where the line of sight is fixed) on the screen of the display device 10, can be obtained. The viewpoint can also be considered as the position where the user is looking or the position of the line of sight.

[0017] 3 is a block diagram showing the electrical configuration within camera 1. Connected to CPU 3 are gaze detection circuit 201, photometry circuit 202, autofocus detection circuit 203, signal input circuit 204, display device drive circuit 11, light source drive circuit 205, and the like. CPU 3 also transmits signals via mount contacts 117 to focus adjustment circuit 118 disposed within photographing lens unit 1A and aperture control circuit 206 included in aperture drive section 112 within photographing lens unit 1A. Memory section 4 associated with CPU 3 has the function of storing image capture signals from image sensor 2 and eye image sensor 17, and the function of storing gaze correction parameters that correct for individual differences in gaze, as described below.

[0018] The gaze detection circuit 201 A / D converts the output of the eye image pickup element 17 (CCD-EYE) (eye image of the eye (eyeball 14)) when an eyeball image is formed on the eye image pickup element 17, and sends the result to the CPU 3. The CPU 3 extracts feature points required for gaze detection from the eye image according to a predetermined algorithm described later, and detects the user's gaze from the positions of the feature points.

[0019] The photometry circuit 202 amplifies, logarithmically compresses, and A / D converts the signal obtained from the image sensor 2, which also functions as a photometry sensor, specifically the luminance signal corresponding to the brightness of the field, and sends the result to the CPU 3 as field luminance information.

[0020] The auto focus detection circuit 203 A / D converts the signal voltage from the multiple detection elements (multiple pixels) used for phase difference detection, which are included in the CCD in the image sensor 2, and sends it to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point from the signals from the multiple detection elements. This is a well-known technology known as image plane phase difference AF. In the embodiment, as an example, it is assumed that there are focus detection points at each of 180 locations on the imaging surface corresponding to the 180 locations shown in the viewfinder field image (screen of the display device 10) in FIG. 4(a).

[0021] Switches SW1 and SW2 are connected to signal input circuit 204. Switch SW1 is turned on with the first stroke of release button 5 to start photometry, distance measurement, line of sight detection, and other operations of camera 1, and switch SW2 is turned on with the second stroke of release button 5 to start a photographing operation. ON signals from switches SW1 and SW2 are input to signal input circuit 204 and sent to CPU 3.

[0022] The light source drive circuit 205 drives the light sources 13a to 13d.

[0023] FIG. 4(a) is a diagram showing the field of view within the viewfinder, and shows the state in which the display device 10 is operating (the state in which an image is displayed). As shown in FIG. 4(a), the field of view within the viewfinder includes a focus detection area 400, 180 ranging point indices 401, a field of view mask 402, and the like. Each of the 180 ranging point indices 401 is displayed superimposed on a through image (live view image) displayed on the display device 10 so as to be displayed at a position corresponding to the focus detection point on the imaging surface. Furthermore, of the 180 ranging point indices 401, the ranging point indices 401 that corresponds to the current viewpoint A (estimated position) is displayed highlighted with a frame or the like.

[0024] <Explanation of personal identification operation> The personal identification operation for identifying a user will be described with reference to FIGS. 5 to 9. Details will be described later, but in this embodiment, the personal identification operation includes a gaze detection operation. FIG. 5 is a block diagram showing functional units required for the personal identification operation. The eyeball information acquisition unit 501, feature acquisition unit 502, and feature matching unit 503 shown in FIG. 5 are implemented by the CPU 3. FIG. 6 is a schematic diagram of an optical system for performing the personal identification operation. As shown in FIG. 6, light sources 13a and 13b are disposed approximately symmetrically with respect to the optical axis of the light receiving lens 16 and illuminate the user's eyeball 14. A portion of the light emitted from light sources 13a and 13b and reflected by the eyeball 14 is focused by the light receiving lens 16 onto the eye image sensor 17. Similarly, light sources 13c and 13d are disposed approximately symmetrically with respect to the optical axis of the light receiving lens 16 and illuminate the user's eyeball 14. A portion of the light emitted from light sources 13c and 13d and reflected by the eyeball 14 is focused by the light receiving lens 16 onto the eye image sensor 17. Fig. 7(a) is a schematic diagram of an eye image captured by the eye image sensor 17 (eyeball image projected onto the eye image sensor 17), and Fig. 7(b) is a diagram showing the output intensity of the CCD in the eye image sensor 17. Fig. 8 is a schematic flowchart of the personal identification operation. Fig. 9 is a table (correspondence table) showing the correspondence between feature amounts and people.

[0025] 8, light sources 13a to 13d emit infrared light toward user's eyeball 14. An image of the user's eyeball illuminated by the infrared light is formed on eye image sensor 17 through light receiving lens 16 and is photoelectrically converted by eye image sensor 17. As a result, a processable electrical signal of the eye image is obtained (eye image acquisition).

[0026] In step S802, the line-of-sight detection circuit 201 sends the eye image (eye image signal; electric signal of the eye image) obtained from the eye imaging device 17 to the CPU 3.

[0027] In steps S803 and S804, the eyeball information acquisition unit 501 realized by the CPU 3 acquires three-dimensional information of the user's eyeball 14 based on the eye image obtained in step S802.

[0028] In step S803, the eyeball information acquisition unit 501 acquires the corneal reflection images Pd, Pe, Pf, and Pg of the light sources 13a to 13d and the pupil center c (pupil 14) from the eye image acquired in step S802. Find the coordinates of the point corresponding to the center of 1.

[0029] Infrared light emitted from light sources 13a to 13d illuminates cornea 142 of user's eyeball 14. At this time, corneal reflection images Pd, Pe, Pf, and Pg formed by part of the infrared light reflected from the surface of cornea 142 are collected by light receiving lens 16 and formed on ocular imaging element 17 as corneal reflection images Pd', Pe', Pf', and Pg' in the eye image. Similarly, light beams from edges a and b of pupil 141 are also formed on ocular imaging element 17 as pupil edge images a' and b' in the eye image.

[0030] FIG. 7(b) shows luminance information (luminance distribution) of region α in the eye image of FIG. 7(a). In FIG. 7(b), the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In this embodiment, the X-axis (horizontal) coordinates of the corneal reflection images Pd', Pe' are set to Xd, Xe, and the X-axis coordinates of the pupil edge images a', b' are set to Xa, Xb. As shown in FIG. 7(b), an extremely high level of luminance is obtained at the coordinates Xd, Xe of the corneal reflection images Pd', Pe'. In the region from coordinate Xa to coordinate Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained when the light beam from the pupil 141 is focused on the ocular imaging element 17), an extremely low level of luminance is obtained except for coordinates Xd, Xe. A luminance intermediate between the two types of luminance is obtained in the region of iris 143 outside pupil 141 (the region of the iris image outside the pupil image obtained by focusing the light beam from iris 143). Specifically, a luminance intermediate between the two types of luminance is obtained in the region where the X coordinate (coordinate in the X-axis direction) is smaller than coordinate Xa and the region where the X coordinate is larger than coordinate Xb.

[0031] From the luminance distribution shown in FIG. 7(b), the X-coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X-coordinates Xa and Xb of the pupil edge images a' and b' can be obtained. Specifically, the coordinates of the corneal reflection images Pd' and Pe' can be obtained as the coordinates of extremely high luminance, and the coordinates of the pupil edge images a' and b' can be obtained as the coordinates of extremely low luminance. Furthermore, when the rotation angle θx of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16 is small, the X-coordinate Xc of the pupil center image c' (center of the pupil image) obtained when the light beam from the pupil center c is focused on the ocular imaging element 17 can be expressed as Xc ≒ (Xa + Xb) / 2. In other words, the X-coordinate Xc of the pupil center image c' can be calculated from the X-coordinates Xa and Xb of the pupil edge images a' and b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupil center image c' can be estimated. The coordinates of the corneal reflection images Pf' and Pg' can also be estimated in a similar manner.

[0032] Returning to the description of FIG. 8 , in step S804, the eyeball information acquisition unit 501 acquires three-dimensional information of the user's eyeball 14 based on the coordinates of the corneal reflection images Pd′, Pe′, Pf′, and Pg′ (three-dimensional information acquisition operation). In this embodiment, the three-dimensional information includes information about the surface shape of the eyeball 14 and information about the internal structure of the eyeball 14. The information about the surface shape of the eyeball 14 is, for example, information about the corneal radius of curvature R (the radius of curvature of the cornea 142) shown in FIG. 6. The information about the internal structure of the eyeball 14 is, for example, information about the amount of deviation of the photoreceptor cells of the eyeball 14 from the optical axis passing through the pupil center c and the corneal center of curvature O (the center of curvature of the cornea 142), and information about the distance between the pupil center c and the corneal center of curvature O. The three-dimensional information may include all of these pieces of information, or may not include at least one of them.

[0033] In steps S805 and S806, the CPU 3 identifies the user based on the three-dimensional information obtained in step S805.

[0034] In step S805, the feature amount acquiring unit 502 realized by the CPU 3 acquires features for identifying the user based on the eye image acquired in step S802 and the three-dimensional information acquired in step S804.

[0035] In step S806, the feature matching unit 503, which is realized by the CPU 3, compares the feature obtained in step S805 with feature values ​​pre-recorded in the memory unit 4 to identify the user. The feature matching unit 503 then outputs the identification result. For example, the correspondence table shown in FIG. 9 is pre-recorded in the memory unit 4. The feature matching unit 503 selects, from the multiple feature values ​​shown in the correspondence table of FIG. 9, the feature value that has the highest degree of match with the feature value obtained in step S805, and determines that the person associated with the selected feature value in the correspondence table of FIG. 9 is the user. If none of the multiple feature values ​​shown in the correspondence table of FIG. 9 has a degree of match with the feature value obtained in step S805 that is equal to or greater than a predetermined threshold, it may be determined that the person corresponding to the user is not registered (cannot be identified). Alternatively, three-dimensional information of each person may be pre-recorded in the memory unit 4, and the user may be identified by comparing the three-dimensional information obtained in step S805 with the three-dimensional information recorded in the memory unit 4.

[0036] Even if the resolution of the eye image obtained by the eye imaging element 17 is relatively low, highly accurate information can be obtained as the above-mentioned three-dimensional information. Therefore, according to this embodiment, it is possible to identify (authenticate) a user (person) with high accuracy using a simple configuration (low-cost configuration).

[0037] The above-mentioned three-dimensional information can be used to prevent impersonation using contact lenses or video of the eye. Suppose a user is attempting to impersonate a specific person by using contact lenses with the iris pattern of that person printed on them. If the user were identified based on a two-dimensional iris pattern, the user would be determined to be the specific person, making impersonation impossible. On the other hand, if the user were identified based on three-dimensional information, the large radius of curvature of the contact lenses would be estimated as the corneal curvature radius R, making it possible to determine that the user is not the specific person, preventing impersonation. Furthermore, because the amount of photoreceptor misalignment and the distance between the pupil center c and the corneal curvature center O (both estimated values) depend on whether or not contact lenses are worn, using this three-dimensional information can also prevent impersonation. Contact lens wearers typically register their information while wearing contact lenses. Therefore, if the user is a legitimate individual, they can be correctly identified even if they wear contact lenses. Furthermore, because three-dimensional information about the eyeball depends on the shape of the eyeball and the type of contact lens, it can also prevent impersonation of contact lens wearers. Even if a video of the eyes is used for spoofing, the corneal reflection images Pd', Pe', Pf', and Pg' are unlikely to appear in the video, so spoofing can be suppressed.

[0038] Whether or not three-dimensional information about the eyeball 14 is used in the personal identification operation can be verified by, for example, the following methods. The first method is to compare the identification result when a first pseudo eyeball is used with the identification result when a second pseudo eyeball, whose corneal curvature radius is different from that of the first pseudo eyeball, is used. If the identification results are different, it can be determined that information about the corneal curvature radius is used in the personal identification operation. The second method is to compare the identification result when the pseudo eyeball is pointed in a predetermined direction with the identification result when the pseudo eyeball is pointed in a direction different from the predetermined direction. Here, pointing the pseudo eyeball in a predetermined direction or a different direction corresponds to having the user gaze at the center of the screen of the display device 10, as described below. If the identification results are different, it can be determined that information about the amount of misalignment of photoreceptors is used in the personal identification operation. The third method is to compare the identification result obtained when the pseudo eyeball is rotated by a first rotation amount from a state in which it is facing a predetermined direction with the identification result obtained when the pseudo eyeball is rotated by a second rotation amount different from the first rotation amount from a state in which it is facing a predetermined direction. Here, rotating the pseudo eyeball from a state in which it is facing a predetermined direction corresponds to having the user sequentially gaze at multiple positions on the screen of the display device 10, as described below. If the identification results differ, it can be determined that information about the distance between the pupil center c and the corneal curvature center O is used in the personal identification operation.

[0039] <Explanation of 3D information acquisition operation> The three-dimensional information acquisition operation (operation of step S804) will now be described. Fig. 10 is a flowchart of the three-dimensional information acquisition operation. In steps S1001 and S1002, information about the surface shape of the eyeball 14 (surface shape information), specifically information about the corneal radius of curvature R, is acquired. In steps S1003 to S1006, information about the internal structure of the eyeball 14 (internal structure information), specifically information about the amount of displacement of photoreceptor cells and information about the distance between the pupil center c and the corneal center of curvature O, is acquired.

[0040] <<Explanation of how to obtain surface shape information>> In step S1001 of FIG. 10, the eyeball information acquisition unit 501 calculates the interval (image interval) ΔP between the corneal reflection images. Specifically, the eyeball information acquisition unit 501 selects a combination of two corneal reflection images (a corneal reflection pair) from the corneal reflection images Pd', Pe', Pf', and Pg', and calculates the interval ΔP between the two corneal reflection images. The eyeball information acquisition unit 501 selects two corneal reflection pairs and calculates two image intervals ΔP corresponding to the two corneal reflection pairs. For example, as shown in FIG. 11(a), the corneal reflection images Pd' and Pe' are selected as the first pair, and the difference (Xe-Xd) between the X coordinates Xd and Xe of the corneal reflection images Pd' and Pe' is calculated as the interval ΔPde between the corneal reflection image Pd' and the corneal reflection image Pe'. Then, the corneal reflection images Pf' and Pg' are selected as the second pair, and the interval ΔPfg between the corneal reflection images Pf' and Pg' is calculated in the same manner as the interval ΔPde.

[0041] The corneal reflection image pair is not limited to the one described above. It is sufficient that the position of at least one of the two light sources corresponding to the second pair is different from the position of the two light sources corresponding to the first pair in a direction parallel to the optical axis for capturing an image of the eyeball 14 (a direction along the optical axis of the ocular imaging element 17 and the light receiving lens 16; the Z-axis direction in FIG. 6). In the example of FIG. 12(a), the Z coordinate (coordinate in the Z-axis direction) of the light sources 13a and 13b that form the corneal reflection images Pd' and Pe' (first pair) is Z1, and the Z coordinate of the light sources 13c and 13d that form the corneal reflection images Pf' and Pg' (second pair) is Z2 (≠Z1).

[0042] 12(b), eye imaging element 17 and light receiving lens 16 may be disposed below eyepiece window frame 121 (finder window). In this case, the direction parallel to the optical axis for capturing an image of eyeball 14 (Z-axis direction) is not parallel to the optical axis of eyepiece lens 12, but is obliquely angled relative to the optical axis of eyepiece lens 12. In the example of FIG. 12(b), light sources 13a, 13b, 13c, and 13d are positioned at the same position in the direction parallel to the optical axis of eyepiece lens 12, but in the Z-axis direction, the positions of light sources 13a and 13b (Z coordinate Z1) are different from the positions of light sources 13c and 13d (Z coordinate Z2).

[0043] In the example of Fig. 11(a), both of the two corneal reflection images in the first pair are different from both of the two corneal reflection images in the second pair, but this is not limited to this. As shown in Fig. 11(b), either of the two corneal reflection images in the first pair may be the same (common) as either of the two corneal reflection images in the second pair. In the example of Fig. 11(b), corneal reflection images Pd' and Pe' are selected as the first pair, and corneal reflection images Pf' and Pd' are selected as the second pair. In this case, the distance ΔP between the corneal reflection images Pf' and Pd' in the Y-axis direction is calculated.

[0044] Returning to the description of Fig. 10, in step S1002, the eyeball information acquisition unit 501 calculates the corneal radius of curvature R and the eyeball distance Z based on the image distance ΔP calculated in step S1001. The eyeball distance Z is, for example, the distance from the light receiving lens 16 to the eyeball 14 in the direction along the Z axis. In this embodiment, it is assumed that the corneal radius of curvature R and the eyeball distance Z are calculated from the image distances ΔPde and ΔPfg shown in Fig. 11(a). The corneal radius of curvature R and the eyeball distance Z may also be calculated from the image distances ΔPde and ΔPdf shown in Fig. 11(b).

[0045] Figures 13(a) and 13(b) are graphs showing the relationship between image distance ΔP, eyeball distance Z, and corneal radius of curvature R. The graph in Figure 13(a) shows the relationship between image distance ΔP and eyeball distance Z, and as eyeball distance Z increases, image distance ΔP monotonically decreases nonlinearly. The relationship (relationship curve) between image distance ΔP and eyeball distance Z varies depending on the corneal radius of curvature R, and as the corneal radius of curvature R increases, the relationship curve shifts upward on the paper so that the image distance ΔP increases. The graph in Figure 13(b) shows the relationship between image distance ΔP and corneal radius of curvature R, and as the corneal radius of curvature R increases, the image distance ΔP monotonically increases, while showing some nonlinearity. The relationship (relationship curve) between image distance ΔP and corneal radius of curvature R varies depending on eyeball distance Z, and as the eyeball distance Z increases, the relationship curve shifts downward on the paper so that the image distance ΔP decreases.

[0046] Furthermore, as described above, the Z coordinate Z1 of the light sources 13a and 13b that form the first pair (corneal reflection images Pd', Pe') is different from the Z coordinate Z2 of the light sources 13c and 13d that form the second pair (corneal reflection images Pf', Pg'). Therefore, the image spacing ΔP1 (= ΔPde) of the first pair and the image spacing ΔP2 (= ΔPfg) of the second pair behave differently with respect to the eyeball distance Z. This is shown in Figures 14(a) and 14(b). Figure 14(a) is a graph showing the relationship between the image spacing ΔP1 of the first pair and the eyeball distance Z, and Figure 14(b) is a graph showing the relationship between the image spacing ΔP2 of the second pair and the eyeball distance Z. Both the image spacing ΔP1 and the image spacing ΔP2 decrease monotonically and nonlinearly with increasing eyeball distance Z, but for the same eyeball distance Z, the image spacing ΔP2 in Figure 14(b) is larger than the image spacing ΔP1 in Figure 14(a).

[0047] The eyeball information acquisition unit 501 calculates the user's corneal radius of curvature R and eyeball distance Z, taking into account the behavior of the image distance ΔP1 of the first pair and the behavior of the image distance ΔP2 of the second pair, which differ from each other. Here, it is assumed that an image of a user's eye with a corneal radius of curvature R=Rc is captured at an eyeball distance Z=Zc, and an image distance ΔP1=Dp1 of the first pair and an image distance ΔP2=Dp2 of the second pair are obtained. Then, the eyeball information acquisition unit 501 calculates (estimates) the user's corneal radius of curvature Rc and eyeball distance Zc based on the image distance ΔP1=Dp1 and the image distance ΔP2=Dp2.

[0048] 14(a), the combinations of corneal curvature radius R and eyeball distance Z that result in image distance ΔP1=Dp1 include the combination of R=7.0 mm and Z=Z1a, the combination of R=7.5 mm and Z=Z1b, and the combination of R=8.0 mm and Z=Z1c. Therefore, the corneal curvature radius R and eyeball distance Z cannot be uniquely determined from the image distance ΔP1=Dp1 alone.

[0049] Therefore, the image distance ΔP2 = Dp2 is further used. The combinations of the corneal curvature radius R and the eyeball distance Z that result in the image distance ΔP2 = Dp2 include the combination of R = 7.0 mm and Z = Z2a, the combination of R = 7.5 mm and Z = Z2b, and the combination of R = 8.0 mm and Z = Z2c.

[0050] FIG. 14(c) is a graph showing the relationship (first relationship curve) between the corneal radius of curvature R and the eyeball distance Z when the image distance ΔP1=Dp1, and the relationship (second relationship curve) between the corneal radius of curvature R and the eyeball distance Z when the image distance ΔP2=Dp2. Both the image distance ΔP1=Dp1 and the image distance ΔP2=Dp2 are values ​​calculated under the same corneal radius of curvature R=Rc and the same eyeball distance Z=Zc. Therefore, the combination of the corneal radius of curvature R and the eyeball distance Z that is common to the first relationship curve and the second relationship curve is the combination of the user's corneal radius of curvature Rc and the eyeball distance Zc. Therefore, the eyeball information acquisition unit 501 can uniquely determine (estimate) the corneal radius of curvature R at the intersection of the first relationship curve and the second relationship curve as the user's corneal radius of curvature Rc, and the eyeball distance Z at the intersection as the user's eyeball distance Zc. In the example of FIG. 14(c), the user's corneal curvature radius Rc=7.5 mm and the user's eyeball distance Zc=Z1b=Z2b are determined.

[0051] The angle at which a specific image distance ΔP is obtained, such as the first and second relationship curves shown in FIG. 14(c), The relationship curve between the corneal radius of curvature R and the eyeball distance Z is determined in advance by actual measurement or simulation and is stored as data in the memory unit 4. A plurality of relationship curves corresponding to a plurality of image distances ΔP are stored in advance in the memory unit 4. The relationship curve data may be, for example, table data indicating discrete combinations of the corneal radius of curvature R and the eyeball distance Z, or parameters such as coefficients of a function (theoretical formula or approximate formula) indicating the correspondence relationship between the corneal radius of curvature R and the eyeball distance Z. The eyeball information acquisition unit 501 reads out from the memory unit 4 a relationship curve (first relationship curve) corresponding to the image distance ΔP1 according to the obtained image distance ΔP1, and reads out from the memory unit 4 a relationship curve (second relationship curve) corresponding to the image distance ΔP2 according to the obtained image distance ΔP2. The eyeball information acquisition unit 501 then calculates the intersection of the read first relationship curve and second relationship curve to calculate the user's corneal radius of curvature Rc and the eyeball distance Zc.

[0052] Since the surface shape (corneal curvature radius R) of the eyeball 14 estimated by the above-mentioned method or the like varies from person to person, information about the surface shape can be used as information for identifying the user.

[0053] <<Explanation of how to obtain internal structure information>> 10, the eyeball information acquisition unit 501 calculates the imaging magnification β of the eyeball image. The imaging magnification β is a magnification determined by the position of the eyeball 14 with respect to the light receiving lens 16, and can be found using a function of the eyeball distance Z calculated in step S1002. For example, the relationship between the imaging magnification β and the eyeball distance Z is determined in advance by performing actual measurements or simulations, and is recorded in the memory unit 4 as table data, function parameters, or the like. The eyeball information acquisition unit 501 then reads data relating to the relationship between the imaging magnification β and the eyeball distance Z from the memory unit 4, and acquires the imaging magnification β corresponding to the eyeball distance Z calculated in step S1002 based on this relationship.

[0054] In step S1004, the eyeball information acquisition unit 501 calculates the rotation angle of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16. The X coordinate of the midpoint between the corneal reflection images Pd and Pe and the X coordinate of the corneal curvature center O approximately coincide. Therefore, if the standard distance from the corneal curvature center O to the pupil center c is Oc, the rotation angle θx of the eyeball 14 in the ZX plane (plane perpendicular to the Y axis) can be calculated using the following equation 1. The rotation angle θy of the eyeball 14 in the ZY plane (plane perpendicular to the X axis) is also calculated using a method similar to that for calculating the rotation angle θx. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc (Formula 1)

[0055] In step S1005, the eyeball information acquisition unit 501 uses the rotation angles θx and θy calculated in step S1004 to estimate the user's viewpoint on the screen of the display device 10. If the viewpoint coordinates (Hx, Hy) correspond to the pupil center c, the viewpoint coordinates (Hx, Hy) can be calculated by the following equations 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0056] The parameter m in equations 2 and 3 is a constant determined by the configuration of the finder optical system (light receiving lens 16, etc.) of the camera 1, and is a conversion coefficient that converts the rotation angles θx and θy into coordinates corresponding to the pupil center c on the screen of the display device 10. The parameter m is determined in advance and recorded in the memory unit 4. The parameters Ax, Bx, Ay, and By are gaze correction parameters that correct individual differences in gaze, and are used for calibration (calibration of gaze detection). The parameters Ax, Bx, Ay, and By are stored in the memory unit 4 before the personal identification operation starts. Calibration is performed for each person, and the parameters Ax, Bx, Ay, and By are determined for each person and stored in the memory unit 4.

[0057] Due to factors such as individual differences in the shape and structure of the human eyeball, a discrepancy between the actual viewpoint B and the estimated viewpoint C can occur, as shown in Figure 4(b). In Figure 4(b), the user is gazing at a person, but Camera 1 mistakenly estimates that the user is gazing at the background, resulting in a state in which viewpoint-based focus detection and focus adjustment cannot be performed appropriately. By using values ​​appropriate for the user for the parameters Ax, Ay, Bx, and By, it is possible to reduce the viewpoint discrepancy shown in Figure 4(b).

[0058] It is also possible to perform only the gaze detection operation included in the individual identification operation by performing only the processes of steps S801 to S803 in FIG. 8 and steps S1003 to S1005 in FIG.

[0059] Returning to the description of FIG. 10, in step S1006, the eyeball information acquisition unit 501 calculates the parameters Ax, Bx, Ay, and By. The shape and structure of a human eyeball can change with aging. Therefore, the CPU 3 may update the parameters Ax, Bx, Ay, and By (parameters of the person determined to be the user in step S806 of FIG. 8) stored in the memory unit 4 with the parameters Ax, Bx, Ay, and By calculated in step S1006. In this case, the personal identification operation can also be considered to include calibration. The eye image used in step S1006 can also be considered to be an image of the eyeball 14 captured during calibration.

[0060] A method for calculating the parameter Bx will be described. FIG. 15 shows the state inside the eyeball 14 when the rotation angle θx shown in FIG. 6 is 0 degrees. In FIG. 15, the photoreceptors, which sense light incident on the eyeball 14 and send signals to the brain, are offset from the optical axis of the eyeball 14 (the optical axis of the cornea 142 and the pupil 141) shown by the dashed-dotted line. Therefore, when the user looks at the center of the screen of the display device 10, the rotation angle θx is offset from 0 degrees by an offset angle corresponding to the amount of displacement of the photoreceptors from the optical axis of the eyeball 14. This offset angle (amount of displacement of the photoreceptors) corresponds to the parameter Bx (Bx ∝ offset angle). Because the offset angle (amount of displacement of the photoreceptors), and therefore the appropriate parameter Bx, vary from person to person, the parameter Bx can be used as information for identifying the user.

[0061] For example, CPU 3 displays multiple indices 403 in FIG. 4(c) on the screen of display device 10 and causes index 403 in the center of the screen to blink, thereby guiding the user to gaze at the center of the screen. Note that the method of guiding gaze is not particularly limited, and for example, only index 403 in the center of the screen may be displayed. CPU 3 calculates rotation angle θx as an offset angle by performing the processes of steps S801 to S803 in FIG. 8 and steps S1003 and S1004 in FIG. 10 using an eye image obtained while the user is gazing at the center of the screen. Then, CPU 3 calculates parameter Bx according to the offset amount.

[0062] The parameter By is calculated in the same manner as the parameter Bx. As information relating to the displacement amount of the photoreceptor cells, both the parameter Bx and the parameter By may be acquired, or only one of the parameter Bx and the parameter By may be acquired. As information relating to the displacement amount of the photoreceptor cells, information other than the parameter Bx and the parameter By may be acquired.

[0063] The calculation method of the parameter Ax will be explained. In Equation 1, the standard distance Oc (constant) from the corneal curvature center O to the pupil center c is used to calculate the rotation angle θx. However, However, the actual distance Oc' (variable) from the corneal curvature center O to the pupil center c is not necessarily the same as the distance Oc. The difference between the distance Oc' and the distance Oc results in an error in the rotation angle θx calculated by Equation 1. The parameter Ax is a parameter for reducing such errors and is inversely proportional to the actual distance Oc' (Ax ∝ 1 / Oc'). The actual distance Oc' is the value obtained by dividing the standard distance Oc by the parameter Ax. Because the actual distance Oc' (a distance related to the size of the eyeball 14) from the corneal curvature center O to the pupil center c, and therefore the appropriate parameter Ax, vary from person to person, the parameter Ax can be used as information for identifying the user.

[0064] For example, the CPU 3 calculates the parameter Ax based on a plurality of eye images captured multiple times of the eyeball 14 while the user gazes sequentially at a plurality of positions on the screen of the display device 10. Specifically, the CPU 3 sequentially gazes at two or more indices 403 of the plurality of indices 403 in FIG. 4(c) that are positioned differently in the horizontal direction. The CPU 3 calculates the rotation angle θx for each of two or more eye images corresponding to the two or more indices 403. The eye image corresponding to the indices 403 is an eye image obtained while the user gazes at the indices 403. The CPU 3 then calculates the parameter Ax based on the calculated two or more rotation angles θx. For example, the CPU 3 calculates the parameter Ax so that the sum of errors (such as the sum of squared residuals of the least squares method) between the calculated two or more rotation angles θx is minimized. The parameter Ax may be calculated so that the errors (the difference between the target rotation angle according to the position of the indices 403 and the calculated rotation angle θx) of the calculated two or more rotation angles θx are approximately the same.

[0065] The parameter Ay is calculated in the same manner as the parameter Ax. Both the parameter Ax and the parameter Ay may be acquired, or only one of the parameter Ax and the parameter Ay may be acquired, as information regarding the distance Oc' between the pupil center c and the corneal curvature center O. Information other than the parameter Ax and the parameter Ay may be acquired as information regarding the distance Oc'.

[0066] <Explanation of feature acquisition operation> The feature acquisition operation (operation of step S805) will be described below. The configuration of the feature acquisition unit 502 is not particularly limited, but in this embodiment, it is assumed to be a CNN (convolutional neural network).

[0067] FIG. 16 shows the configuration of the CNN serving as the feature acquisition unit 502. Hereinafter, the feature acquisition unit 502 will be referred to as the CNN 302. The CNN 302 receives an eye image and 3D information about the eyeball 14 as inputs and outputs features. The CNN 302 has multiple hierarchical sets, each consisting of two layers called a feature detection layer (S layer) and a feature integration layer (C layer). In the S layer, the next feature is detected based on the feature detected in the previous layer. In the first S layer, features are detected based on the eye image and 3D information. The features detected in the S layer are integrated in the C layer of the same layer and sent to the next layer as the detection result for that layer. The S layer consists of one or more feature detection cell planes, and each feature detection cell plane detects a different feature. The C layer also consists of one or more feature integration cell planes, and pools the detection results from the feature detection cell planes of the same layer. Hereinafter, unless there is a need to distinguish between them, the feature detection cell planes and feature integration cell planes will be collectively referred to as feature planes. In this embodiment, the output layer, which is the final layer, does not have a C layer, but has only an S layer.

[0068] The details of the feature detection process on the feature detection cell plane and the feature integration process on the feature integration cell plane will be explained using Figure 17. The feature detection cell plane is composed of multiple feature detection neurons, which are connected to the C layer of the previous layer in a predetermined structure. The feature integration cell plane is composed of multiple feature integration neurons, which are connected to the S layer of the same layer in a predetermined structure. Within the Mth cell plane of the S layer of the Lth layer, The output of the feature detection neuron in (ξ,ζ) is calculated as y M LS (ξ,ζ), in the Mth cell plane of the C layer of the Lth layer, the output value of the feature integration neuron at position (ξ,ζ) is y M LC (ξ,ζ). Then, the coupling coefficient of each neuron is written as w M LS (n,u,v), w M LC Assuming that the output values ​​are (u, v), each output value can be expressed as in the following equations 4 and 5.

[0069]

number

[0070]

number

[0071] In Equation 4, f is an activation function, which may be a sigmoid function such as a logistic function or a hyperbolic tangent function, for example, a tanh function. M LS (ξ,ζ) is the internal state of the feature detection neuron at position (ξ,ζ) on the Mth cell plane of the Sth layer of the Lth hierarchy. In Equation 5, a simple linear sum is calculated without using an activation function. When an activation function is not used as in Equation 5, the internal state of the neuron u M LC (ξ,ζ) and output value y M LC (ξ,ζ) are equal. Also, y in Eq. n L-1C (ξ+u,ζ+v), y in Eq. M LS (ξ+u, ζ+v) are called the output values ​​of the feature detection neuron and the feature integration neuron, respectively.

[0072] The following explains ξ, ζ, u, v, and n in Equations 4 and 5. The position (ξ, ζ) corresponds to the position coordinates in the input image. For example, y M LS If (ξ,ζ) has a high output value, it means that there is a high possibility that the feature to be detected on the Mth cell plane in the Sth layer of the Lth hierarchical level exists at pixel position (ξ,ζ) of the input image. In Equation 4, n refers to the nth cell plane in the Cth layer of the L-1th hierarchical level, and is called the target feature number. Basically, a product-sum operation is performed on all cell planes that exist in the Cth layer of the L-1th hierarchical level. (u,v) are the relative position coordinates of the connection coefficient, and the product-sum operation is performed within a finite range (u,v) depending on the size of the feature to be detected. This finite range of (u,v) is called the receptive field. The size of the receptive field is called the receptive field size below, and is expressed as the number of horizontal pixels x the number of vertical pixels in the connected range.

[0073] Also, in Equation 4, when L=1, that is, the first S layer, y n L-1C (ξ+u,ζ+v) is the input image y in_image (ξ+u,ζ+v) or input position map y in_posi_map (ξ+u,ζ+v). Note that the distribution of neurons and pixels is discrete, and the connection destination feature numbers are also discrete, so ξ, ζ, u, v, and n are not continuous variables but take discrete values. Here, ξ and ζ are non-negative integers, n is a natural number, and u and v are integers, all of which have values ​​within a finite range.

[0074] w in Equation 4 M LS (n,u,v) is the distribution of coupling coefficients for detecting a specific feature, and by adjusting it to an appropriate value, it becomes possible to detect the specific feature. Adjusting this distribution of coupling coefficients is called learning. In the construction of CNN302, various test patterns are presented to learn y M LS The coupling coefficients are adjusted by repeatedly modifying them gradually so that (ξ,ζ) becomes an appropriate output value.

[0075] w in Equation 5 M LC (u, v) can be expressed as the following equation 6 using a two-dimensional Gaussian function.

[0076]

number

[0077] Here again, (u,v) is a finite range, so just as in the explanation of feature detection neurons, this finite range is called the receptive field, and the size of the range is called the receptive field size. Here, this receptive field size can be set to an appropriate value depending on the size of the Mth feature in the Sth layer of the Lth hierarchy. σ in Equation 6 is the feature size factor, and can be set to an appropriate constant depending on the receptive field size. Specifically, it is best to set σ so that the outermost value of the receptive field can be considered to be almost 0.

[0078] By performing the above calculations at each layer, the features used to identify users can be obtained at the final layer, layer S. Note that the process up to user identification may be performed using CNN, and the user identification results may be output from the CNN.

[0079] <Summary> As described above, according to this embodiment, by using three-dimensional information of the eyeball, it is possible to identify (authenticate) a user (person) with a simple configuration and with high accuracy.

[0080] The above embodiment is merely an example, and configurations obtained by appropriately modifying or changing the configuration of the above embodiment within the scope of the gist of the present invention are also included in the present invention. For example, although an example using four light sources has been described, the number of light sources is not particularly limited and may be more or less than four. When calculating the corneal curvature radius R using the above method, three or more light sources are required.

[0081] Although the present invention has been described as being applied to an imaging device (camera), the present invention can be applied to any device that can acquire an image of a user's eye. The eye imaging element and the light source may be provided in a device separate from the device to which the present invention is applied.

[0082] <Examples of application to other electronic devices> FIG. 18(a) is an external view of a notebook personal computer 1810 (notebook PC). In FIG. 18(a), an imaging unit 1815 that captures an image of a user looking at a display unit 1811 of the notebook PC 1810 is connected to the notebook PC 1810, and the notebook PC 1810 acquires the imaging results from the imaging unit 1815. The notebook PC 1810 then acquires three-dimensional information of the user's eyeballs based on the imaging results and identifies the user. The imaging unit 1815 may acquire the three-dimensional information to identify the user and output the identification result to the notebook PC 1810. In this way, the present invention is also applicable to the notebook PC 1810 and the imaging unit 1815.

[0083] Fig. 18(b) is an external view of a smartphone 1820. In Fig. 18(b), the smartphone 1820 acquires three-dimensional information of the eyeballs of a user looking at a display unit 1822 of the smartphone 1820 based on the imaging results of an in-camera 1821 (front camera), and identifies the user. In this way, the present invention is also applicable to the smartphone 1820. Similarly, the present invention is also applicable to various tablet terminals.

[0084] FIG. 18(c) is an external view of a game machine 1830. In FIG. 18(c), a head-mounted display 1835 (HMD) that displays a VR (Virtual Reality) image of the game on a display unit 1836 is connected to the game machine 1830. The HMD 1835 has a camera 1837 that captures an image of the eyes of the user wearing the HMD 1835, and The game machine 1830 acquires the imaging results from the HMD 1835. Then, the game machine 1830 acquires three-dimensional information of the user's eyeballs based on the imaging results and identifies the user. The HMD 1835 may acquire the three-dimensional information, identify the user, and output the identification result to the game machine 1830. In this way, the present invention is applicable to the game machine 1830 and the HMD 1835. Just as the present invention is applicable to viewing VR images displayed on an HMD, the present invention is also applicable to viewing AR (Augmented Reality) images displayed on the lens portion of a glasses-type wearable terminal. Just as the present invention is applicable to VR technology and AR technology, the present invention is also applicable to other xR technologies such as MR (Mixed Reality) technology and SR (Substitutional Reality) technology.

[0085] <Other Examples> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0086] 1: Camera 3: CPU 501: Eyeball information acquisition unit 502: Feature acquisition unit 503: Feature matching unit

Claims

1. image acquisition means for acquiring an image of the user's eyeball; an information acquisition means for acquiring three-dimensional information of the eyeball based on the image; a feature acquisition means for inputting the image acquired by the image acquisition means and the three-dimensional information acquired by the information acquisition means into a convolutional neural network to acquire a feature of the user; an identification means for identifying the user based on the feature amount; An identification device comprising:

2. The image acquisition means is an imaging means for imaging the eyeball.

2. The identification device according to claim 1.

3. the image acquisition means acquires the image by illuminating the eyeball with a plurality of light sources, The information acquisition means acquires the three-dimensional information based on corneal reflection images of the plurality of light sources.

3. The identification device according to claim 1 or 2.

4. The plurality of light sources is further included.

4. The identification device according to claim 3.

5. The three-dimensional information includes information about the surface shape of the eyeball.

5. The identification device according to claim 1, wherein the identification device is a semiconductor integrated circuit.

6. The information about the surface shape includes information about the radius of curvature of the cornea of ​​the eyeball.

6. The identification device according to claim 5.

7. the image acquisition means acquires the image captured by illuminating the eyeball with three or more light sources, the information acquiring means acquires information about the radius of curvature based on corneal reflection images of the three or more light sources; In a direction parallel to an optical axis for imaging the eyeball, a position of at least one of the three or more light sources is different from positions of the remaining three or more light sources.

7. The identification device according to claim 6.

8. The three-dimensional information includes information about the internal structure of the eyeball. The identification device according to any one of claims 1 to 7.

9. The information about the internal structure includes information about the amount of deviation of the photoreceptor cells of the eyeball from the optical axis passing through the center of the pupil of the eyeball and the center of curvature of the cornea of ​​the eyeball.

9. The identification device according to claim 8.

10. the image acquisition means acquires the image capturing the eyeball of the user looking at the display screen, The information acquiring means acquires information about the amount of deviation based on an image of the eyeball captured while the user is gazing at a center of the display screen.

10. The identification device according to claim 9.

11. The information about the internal structure includes information about the distance between the center of the pupil of the eye and the center of curvature of the cornea of ​​the eye. The identification device according to any one of claims 8 to 10.

12. the image acquisition means acquires the image capturing the eyeball of the user looking at the display screen, The information acquiring means acquires information about the distance based on a plurality of images obtained by capturing images of the eyeball a plurality of times while the user gazes at a plurality of positions on the display surface in sequence.

12. The identification device according to claim 11.

13. The image is also used for gaze detection to obtain information about the user's gaze; The information acquisition means acquires information about the internal structure of the eyeball based on an image of the eyeball captured during calibration to obtain parameters used in the gaze detection. The identification device according to any one of claims 8 to 12.

14. The information about the internal structure of the eyeball is the parameter used in the gaze detection.

14. The identification device according to claim 13.

15. acquiring an image of the user's eyeball; acquiring three-dimensional information of the eyeball based on the image; inputting the image and the three-dimensional information into a convolutional neural network to acquire features of the user; identifying the user based on the feature amount; 10. A method for identifying a target object, comprising:

16. A program for causing a computer to function as each means of the identification device according to any one of claims 1 to 14.

17. A computer-readable storage medium storing a program for causing a computer to function as each means of the identification device according to any one of claims 1 to 14.

Images