Identification device

Abstract

To provide a technique that enables highly accurate identification of a user without requiring the user to manually input information even when a low-resolution eye image is used.SOLUTION: An identification device of the present invention comprises first acquisition means configured to acquire an eye image obtained by photographing an eye of a user looking through a viewfinder of an image capturing device, second acquisition means configured to acquire eye information regarding at least either of the position and posture of the eye with respect to the viewfinder from the eye image, and identification means configured to identify the user on the basis of the eye image and the eye information.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] This invention relates to an identification device for identifying people. [Background technology]

[0002] As a technology for identifying (authenticating) a person based on a facial image captured by a camera, a technology for identifying a person based on the iris has been proposed. Patent Document 1 discloses a method for identifying a person by extracting the iris from an image of the person's eye (eyeball), encoding the iris, and comparing the iris code with a reference code. Patent Document 2 discloses a technology for improving the success rate of iris-based identification by further using additional information. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Special Publication No. 8-504979 [Patent Document 2] Japanese Patent Application Publication No. 11-007535 [Overview of the project] [Problems that the invention aims to solve]

[0004] However, the technology disclosed in Patent Document 1 requires high-resolution eye images to identify iris patterns. Using high-resolution eye images increases the processing time for identification and worsens the responsiveness of the identification. Using low-resolution eye images can shorten the processing time for identification, but it reduces the accuracy of the identification.

[0005] The technology disclosed in Patent Document 2 can improve the success rate of identification by further using additional information, but the user has to manually input the additional information, which makes the process cumbersome (poor inconvenient).

[0006] An object of the present invention is to provide a technique capable of highly accurately identifying a user without requiring manual input of information by the user even when using a low-resolution eye image.

Means for Solving the Problems

[0007] The identification device of the present invention includes: a first acquisition means for acquiring an eye image obtained by imaging the eye of a user looking into the finder of an imaging device; a second acquisition means for acquiring eye information regarding at least one of the position and posture of the eye with respect to the finder from the eye image; and an identification means for identifying the user based on the eye image and the eye information.

Effects of the Invention

[0010] <<Example 1>> The following describes Example 1 of the present invention.

[0011] <Explanation of the structure> Figures 1(a) and 1(b) show the external appearance of the camera 100 (digital still camera; interchangeable lens camera) according to Embodiment 1. Figure 1(a) is a front perspective view, and Figure 1(b) is a rear perspective view. As shown in Figure 1(a), the camera 100 has a shooting lens unit 100A and a camera housing 100B. The camera housing 100B has a release button 101, which is an operating member that receives imaging operations from the user (photographer). As shown in Figure 1(b), the rear of the camera housing 100B has an eyepiece lens 102 (finder) that the user looks through to view the display device 214 (display panel) contained within the camera housing 100B, which will be described later. Operating members 103 to 105 that receive various operations from the user are also located on the rear of the camera housing 100B. For example, operating member 103 is a touch panel that accepts touch operations, operating member 104 is an operating lever that can be pushed down in each direction, and operating member 105 is a four-way key that can be pressed in each of the four directions. Operating member 103 (touch panel) is equipped with a display panel (e.g., a liquid crystal panel) and has the function of displaying images on the display panel.

[0012] Figure 2 is a cross-sectional view of camera 100 cut along the YZ plane formed by the Y and Z axes shown in Figure 1(a), and shows the general internal structure of camera 100.

[0013] The shooting lens unit 100A includes two lenses 201 and 202, an aperture 203, an aperture drive unit 204, a lens drive motor 205, a lens drive member 206, a photocoupler 207, a pulse plate 208, a mount contact 209, and a focus adjustment circuit 210. The lens drive member 206 includes a drive gear, and the photocoupler 207 detects the rotation of the pulse plate 208, which is linked to the lens drive member 206, and transmits this information to the focus adjustment circuit 210. The focus adjustment circuit 210 drives the lens drive motor 205 based on information from the photocoupler 207 and information from the camera housing 100B (information on the amount of lens drive), moving the lens 201 to change the focus position. The mount contact 209 is the interface between the shooting lens unit 100A and the camera housing 100B. Although only two lenses 201 and 202 are shown for simplicity, the shooting lens unit 100A actually contains more than two lenses.

[0014] The camera housing 100B includes an image sensor 211, a CPU 212, a memory unit 213, a display device 214, and a display device drive circuit 215. The image sensor 211 is positioned on the intended imaging plane of the imaging lens unit 100A. The CPU 212 is the central processing unit of the microcomputer and controls the entire camera 100. The memory unit 213 stores various information. For example, the memory unit 213 stores images captured by the image sensor 211. The display device 214 displays various information on its screen (display surface). For example, the display device 214 is a liquid crystal panel and displays captured images (subject images) on its screen. The display device driving circuit 215 drives the display device 214. The user can view the screen of the display device 214 through the eyepiece 102.

[0015] The camera housing 100B also includes light sources 216a, 216b, an optical divider 217, a light-receiving lens 218, and an eye image sensor 219. The light sources 216a and 216b are light sources that have been conventionally used in single-lens reflex cameras to detect the line of sight from the relationship between the reflected image (corneal reflection image) due to corneal reflection of light and the pupil, and are light sources for illuminating the eyeball 220 of the user looking through the viewfinder (eyepiece 102). For example, the light sources 216a and 216b are infrared light-emitting diodes that emit infrared light that is insensitive to the user, and are arranged around the eyepiece 102. The optical image of the illuminated eyeball 220 (eye optical image; optical image due to reflected light emitted from the light sources 216a and 216b and reflected by the eyeball 220) passes through the eyepiece 102 and is reflected by the optical divider 217. The optical image of the eye is then formed by the light-receiving lens 218 onto an eye image sensor 219, which has multiple photoelectric conversion elements (e.g., CCD or CMOS) arranged in two dimensions. The light-receiving lens 218 positions the pupil of the eyeball 220 and the eye image sensor 219 in a conjugate imaging relationship. The line of sight of the eyeball 220 is detected from the position of the corneal reflection image in the optical image formed on the eye image sensor 219 using a predetermined algorithm described later. For example, as information regarding the line of sight, at least one of the following is obtained: information indicating the direction of the line of sight (direction of the gaze) and information indicating the viewpoint on the screen of the display device 214 (the position where the gaze is directed). The viewpoint can be considered as the position the user is looking at, or as the line of sight position.

[0016] Figure 3 is a block diagram showing the electrical configuration within the camera 100. The CPU 212 is connected to a gaze detection circuit 301, a photometering circuit 302, an autofocus detection circuit 303, a signal input circuit 304, a display device drive circuit 215, and a light source drive circuit 305. The CPU 212 transmits signals via mount contacts 209 to a focus adjustment circuit 210 located in the imaging lens unit 100A and an aperture control circuit 306 included in the aperture drive unit 204 within the imaging lens unit 100A. The memory unit 213 attached to the CPU 212 has a function to store imaging signals from the image sensor 211 and the eye image sensor 219, and a function to store gaze correction parameters, which will be described later.

[0017] The gaze detection circuit 301 performs A / D conversion on the output of the eye image sensor 219 (eye image captured from the eye (eyeball 220)) when an optical image of the eye is formed on the eye image sensor 219, and transmits the result to the CPU 212. The CPU 212 extracts feature points necessary for gaze detection from the eye image according to a predetermined algorithm described later, and detects the user's gaze from the position of the feature points.

[0018] The photometering circuit 302 performs predetermined processing (e.g., amplification, logarithmic compression, and A / D conversion) on the signal obtained from the image sensor 211, which also acts as a photometering sensor, for example, a luminance signal corresponding to the brightness of the field of view, and sends the result to the CPU 212 as field of view luminance information.

[0019] The autofocus detection circuit 303 performs A / D conversion on signals from multiple detection elements (multiple pixels) included in the image sensor 211 that are used for phase difference detection, and sends the results to the CPU 212. The CPU 212 calculates the distance to the subject corresponding to each focus detection point from the signals of the multiple detection elements. This is a known technique known as image plane phase difference AF. In Example 1, as an example, it is assumed that there are 180 focus detection points on the image plane corresponding to the 180 locations shown in the viewfinder field of view (screen of display device 214) in Figure 4(a).

[0020] Switches SW1 and SW2 are connected to the signal input circuit 304. Switch SW1 is turned ON on the first stroke of the release button 101, preparing the camera 100 for shooting. This is a switch for starting an operation (e.g., metering and distance measurement). Switch SW2 is turned ON by the second stroke of the release button 101 and is a switch for starting the shooting operation. ON signals from switches SW1 and SW2 are input to the signal input circuit 304 and transmitted to the CPU 212. Eye-line detection may be started when switch SW1 is turned ON.

[0021] The light source driving circuit 305 drives the light sources 216a and 216b.

[0022] Furthermore, the CPU 212 is also connected to the control members 103 to 105. When the user operates the control members 103 to 105, they output operation signals to the CPU 212 corresponding to the user's operation, and the CPU 212 performs processing (control) according to the operation signals. For example, the CPU 212 moves the frame (item, indicator) displayed in the viewpoint according to the operation signal.

[0023] Figure 4(a) shows the viewfinder field of view, indicating that the display device 214 is operational (displaying an image). As shown in Figure 4(a), the viewfinder field of view includes a focus detection area 400, 180 autofocus point indicators 401, and a field of view mask 402. Each of the 180 autofocus point indicators 401 is displayed overlaid on the through image (live view image) displayed on the display device 214, so as to be displayed at the position corresponding to the focus detection point on the image sensor. Of the 180 autofocus point indicators 401, the autofocus point indicator 401 corresponding to the current viewpoint A (estimated position) is displayed highlighted, for example, with a frame.

[0024] <Explanation of personal identification process> Camera 100 can perform personal identification operations to identify the user. The personal identification operation according to Embodiment 1 will be explained using Figures 5 to 11. The personal identification operation according to Embodiment 1 includes gaze detection. Figure 5 is a diagram illustrating the principle of the gaze detection method and is a schematic diagram of the optical system for performing gaze detection. As shown in Figure 5, the light sources 216a and 216b are arranged approximately symmetrically with respect to the optical axis of the light-receiving lens 218 and illuminate the user's eyeball 220. A portion of the light emitted from the light sources 216a and 216b and reflected by the eyeball 220 is focused onto the eye image sensor 219 by the light-receiving lens 218. Figure 6(a) is a schematic diagram of the eye image (eye optical image projected onto the eye image sensor 219) captured by the eye image sensor 219, and Figure 6(b) is a diagram showing the output intensity of the eye image sensor 219. Figure 7 is a flowchart of the personal identification operation. For example, when camera 100 is activated, the personal identification operation shown in Figure 7 begins. It is also conceivable that the user may not use the viewfinder but instead take images by looking at the image displayed on the display panel provided on the operating member 103. In such cases, the user can disable the function of displaying the image on the display device 214 by operating an unillustrated operating member or menu on the camera 100. Since it is unlikely that the user will look through the viewfinder when the display on the display device 214 is disabled, the personal identification operation shown in Figure 7 may not be performed. Alternatively, since the display panel on the operating member 103 is positioned to be viewed from the same direction as the eyepiece lens 102 of the viewfinder, it is possible that personal identification may be possible even when the display on the display device 214 is disabled, if the user's eye is close enough. Therefore, even when the display on the display device 214 is disabled, the personal identification operation shown in Figure 7 may be performed.

[0025] Figure 8 is a block diagram showing the functional components necessary for personal identification. The eye information acquisition unit 801, feature acquisition unit 802, feature matching unit 804, and parameter selection unit 805 in Figure 8 are implemented by the CPU 212, and the storage unit 803 is implemented by the memory unit 213. Figures 9(a) and 9(b) are diagrams illustrating the method for determining the eyepiece distance (the distance from the finder to the eyeball 220 in the optical axis direction of the finder (eyepiece lens 102)). Figure 10 is a flowchart of the eye information acquisition operation. Figure 11 is a diagram showing the correspondence between feature quantities and gaze correction parameters.

[0026] When the personal detection operation shown in Figure 7 begins, in step S701, the CPU 212 controls the light sources 216a and 216b via the light source drive circuit 305 to emit infrared light toward the user's eyeball 220. The optical image of the user's eye, illuminated by the infrared light, is formed on the eye image sensor 219 through the light-receiving lens 218 and is photoelectrically converted by the eye image sensor 219. This provides an electrical signal of the eye image that can be processed.

[0027] In step S702, the CPU 212 acquires an eye image (eye image signal; electrical signal of the eye image) from the eye image sensor 219 via the gaze detection circuit 301.

[0028] Through the processing in steps S703 and S704, the CPU 212 (eye information acquisition unit 801) acquires eye information regarding the position of the eyeball 220 relative to the viewfinder from the eye image obtained in step S702.

[0029] In step S703, the CPU 212 (eye information acquisition unit 801) detects the coordinates of the points corresponding to the corneal reflection images Pd and Pe of the light sources 216a and 216b and the pupil center c from the eye image obtained in step S702.

[0030] Infrared light emitted from light sources 216a and 216b illuminates the cornea 501 of the user's eyeball 220. At this time, the corneal reflection images Pd and Pe, formed by a portion of the infrared light reflected from the surface of the cornea 501, are focused by the light-receiving lens 218 and imaged onto the eye image sensor 219 to become the corneal reflection images Pd' and Pe' in the eye image. Similarly, the light beams from the ends a and b of the pupil 502 are also imaged onto the eye image sensor 219 to become the pupil end images a' and b' in the eye image.

[0031] Figure 6(b) shows the luminance information (luminance distribution) of region α in the eye image of Figure 6(a). In Figure 6(b), the horizontal direction of the eye image is the X-axis direction and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In Example 1, the X-axis coordinates (horizontal direction) of the corneal reflection images Pd',Pe' are set to Xd,Xe, and the X-axis coordinates of the pupil end images a',b' are set to Xa,Xb. As shown in Figure 6(b), an extremely high level of luminance is obtained at coordinates Xd,Xe of the corneal reflection images Pd',Pe'. In the region from coordinate Xa to coordinate Xb, which corresponds to the pupil 502 region (the region of the pupil image obtained when the light beam from the pupil 502 is imaged on the eye image sensor 219), an extremely low level of luminance is obtained except for coordinates Xd,Xe. Furthermore, in the region of the iris 503 outside the pupil 502 (the region of the iris image outside the pupil image obtained by imaging the light beam from the iris 503), an intermediate brightness between the two brightness levels described above is obtained. For example, an intermediate brightness between the two brightness levels described above is obtained in the region where the X-coordinate (coordinate in the X-axis direction) is greater than coordinate Xa and in the region where the X-coordinate is less than coordinate Xb.

[0032] From the luminance distribution shown in Figure 6(b), the coordinates Xd and Xe of the corneal reflection images Pd' and Pe', and the coordinates Xa and Xb of the pupillary end images a' and b' can be obtained. For example, coordinates with extremely high luminance can be obtained as the coordinates of the corneal reflection images Pd' and Pe', and coordinates with extremely low luminance can be obtained as the coordinates of the pupillary end images a' and b'. When the rotation angle θx of the optical axis of the eyeball 220 with respect to the optical axis of the light-receiving lens 218 is small, the coordinate Xc of the pupillary center image c' (center of the pupillary image) obtained by imaging the light beam from the pupillary center c onto the eye image sensor 219 can be expressed as Xc ≈ (Xa + Xb) / 2. In other words, the coordinate Xc of the pupillary center image c' can be calculated from the coordinates Xa and Xb of the pupillary end images a' and b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupillary center image c' can be estimated.

[0033] In step S704, the CPU 212 (eye information acquisition unit 801) acquires eyepiece distance information as eye information based on the interval ΔP=Xe-Xd of the corneal reflection images Pd',Pe' detected in step S703.

[0034] The process of step S704 (eye information acquisition operation in Figure 10) will be explained in more detail. As shown in Figure 9(a), the interval ΔP2 between the corneal reflection images Pd',Pe' when the eyepiece distance is long is smaller than the interval ΔP1 between the corneal reflection images Pd',Pe' when the eyepiece distance is short. Thus, as the eyepiece distance increases (as the eyeball 220 moves away from the viewfinder), the interval ΔP between the corneal reflection images Pd',Pe' decreases. Figure 9(b) shows the correspondence between the eyepiece distance and the interval ΔP between the corneal reflection images Pd',Pe'. In Embodiment 1, a conversion table showing the correspondence in Figure 9(b) is stored in the memory unit 213 in advance. In step S1001 of Figure 10, the CPU 212 (eye information acquisition unit 801) calculates the interval ΔP = Xe - Xd between the corneal reflection images Pd', Pe' from the coordinates Xd, Xe of the corneal reflection images Pd', Pe' detected in step S703. In step S1002, the CPU 212 (eye information acquisition unit 801) determines the eyepiece distance corresponding to the interval ΔP calculated in step S1001 from the conversion table showing the correspondence relationship in Figure 9(b). Alternatively, a function showing the correspondence relationship in Figure 9(b) may be predetermined, and the eyepiece distance may be calculated from the interval ΔP calculated in step S1001 using this function. There are individual differences in how people look through the viewfinder, and the eyepiece distance changes depending on how they look through it. Therefore, the eyepiece distance information can be used as personal identification information (authentication information). Furthermore, when the display on the display device 214 is disabled, it is assumed that the eyepiece distance is longer than when the display on the display device 214 is enabled. Therefore, the eyepiece distance information may be stored separately for both cases where the display on the display device 214 is disabled and when it is enabled, and used accordingly depending on whether the display on the display device 214 is enabled or disabled.

[0035] Returning to the explanation of Figure 7, in step S705, the CPU 212 (feature acquisition unit 802) acquires user features based on the eye image obtained in step S702 and the eye information obtained in step S704.

[0036] In step S706, the CPU 212 (feature matching unit 804) matches (compares) the feature obtained in step S705 with each of the multiple feature quantities stored in the memory unit 213 (storage unit 803). The CPU 212 (feature matching unit 804) then selects the feature quantity with the highest degree of agreement with the feature obtained in step S705 from the multiple feature quantities stored in the memory unit 213 (storage unit 803). The multiple feature quantities stored in the memory unit 213 (storage unit 803) correspond to multiple people. Therefore, selecting one of the multiple feature quantities stored in the memory unit 213 (storage unit 803) can be considered as identifying (specifying) a user. In Embodiment 1, as shown in Figure 11, multiple gaze correction parameters are associated with each of the multiple feature quantities (multiple people) stored in the memory unit 213 (storage unit 803). The multiple gaze correction parameters are also stored in the memory unit 213 (storage unit 803). The CPU 212 (parameter selection unit 805) selects a gaze correction parameter corresponding to the selected feature (user) from among multiple gaze correction parameters stored in the memory unit 213 (storage unit 803). Then, the CPU 212 (parameter selection unit 805) sets the selected gaze correction parameter to be used for gaze detection.

[0037] The gaze correction parameter is a parameter that corrects individual differences in gaze, and is obtained by performing gaze detection calibration. The gaze correction parameter is stored in the memory unit 213 (storage unit 803) before the individual identification operation starts. Calibration is performed for each person, and the gaze correction parameter is determined for each person and stored in the memory unit 213 (storage unit 803). During calibration, feature quantities are obtained using the method described above, associated with the gaze correction parameter, and stored in the memory unit 213 (storage unit 803). The gaze correction parameter may also be stored separately for both cases where display on the display device 214 is disabled and when it is enabled, and used differently depending on whether display on the display device 214 is enabled or disabled.

[0038] In step S707, the CPU 212 detects (estimates) the user's viewpoint.

[0039] The process in step S707 will be explained in more detail. The CPU 212 calculates the imaging magnification β of the eye image. The imaging magnification β is a magnification determined by the position of the eyeball 220 relative to the light-receiving lens 218, and can be calculated using the function ΔP = Xe - Xd, which is the distance between the corneal reflection images Pd', Pe'.

[0040] Next, the CPU 212 calculates the rotation angle of the optical axis of the eyeball 220 relative to the optical axis of the light-receiving lens 218. The X-coordinate of the midpoint of the corneal reflection image Pd and the corneal reflection image Pe is approximately the same as the X-coordinate of the curvature center O of the cornea 501. Therefore, if we take Oc as the standard distance from the curvature center O of the cornea 501 to the center c of the pupil 502, the rotation angle θx of the eyeball 220 in the ZX plane (plane perpendicular to the Y axis) can be calculated using the following equation 1. The rotation angle θy of the eyeball 220 in the ZY plane (plane perpendicular to the X axis) can also be calculated using the same method as for calculating the rotation angle θx. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc (Formula 1)

[0041] The CPU 212 then uses the calculated rotation angles θx and θy to estimate the user's viewpoint on the display device 214 screen. Assuming that the viewpoint coordinates (Hx, Hy) correspond to the pupil center c, the viewpoint coordinates (Hx, Hy) can be calculated using the following equations 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0042] The parameter m in equations 2 and 3 is a constant determined by the configuration of the optical system for gaze detection, and is a transformation coefficient that converts the rotation angles θx and θy to coordinates corresponding to the pupil center c on the display device 214 screen. The parameter m is assumed to be predetermined and stored in the memory unit 213. The parameters Ax, Bx, Ay, and By are the gaze correction parameters described above.

[0043] <Explanation of Feature Acquisition Operation> The feature acquisition operation (operation in step S705) will now be explained. The configuration of the feature acquisition unit 802 is not particularly limited, but in Example 1, it is assumed to be a neural network.

[0044] Figure 12 shows the configuration of the CNN (Convolutional Neural Network) as the feature acquisition unit 802. The CNN takes eye images and eye information as input and outputs features. The CNN has multiple hierarchical sets, with each set consisting of two layers called feature detection layers (S layers) and feature integration layers (C layers). In the S layer, the next features are detected based on the features detected in the previous layer. In the first S layer, features are detected based on the eye image and eye information. The features detected in the S layer are integrated in the C layer of the same hierarchical level and sent to the next layer as detection results for that level. The S layer consists of one or more feature detection cell surfaces, and different features are detected for each feature detection cell surface. The C layer also consists of one or more feature integration cell surfaces and pools the detection results from the feature detection cell surfaces of the same hierarchical level. Hereafter, unless there is a need to distinguish between them, feature detection cell surfaces and feature integration cell surfaces will be collectively referred to as feature surfaces. In Example 1, the final output layer does not have a C layer and only has an S layer.

[0045] The details of the feature detection process on the feature detection cell plane and the feature integration process on the feature integration cell plane will be explained using Figure 13. The feature detection cell plane is composed of multiple feature detection neurons, and the feature detection neurons are connected to the previous layer C in a predetermined structure. The feature integration cell surface is composed of multiple feature integration neurons, and these feature integration neurons are connected in a predetermined structure to the S layer of the same hierarchy. Within the M-th cell surface of the L-th S layer, the output value of the feature detection neuron at position (ξ,ζ) is y M LS (ξ,ζ), within the M-th cell plane of the L-th layer C, the output value of the feature-integrating neuron at position (ξ,ζ) is y M LC Let (ξ,ζ) be denoted as such. Then, the connection coefficient of each neuron is wM LS (n, u, v), w M LC If we set (u, v), each output value can be expressed by the following equations (4) and (5).

[0046] [Number]

[0047] [Number]

[0048] f in Equation (4) is an activation function, which can be a sigmoid function such as a logistic function or a hyperbolic tangent function, for example, it can be a tanh function. u M LS (ξ, ζ) is the internal state of the feature detection neuron at the position (ξ, ζ) on the M-th cell surface of the S layer in the L-th layer. In Equation (5), a simple linear sum is calculated without using an activation function. When no activation function is used as in Equation (5), the internal state u of the neuron M LC (ξ, ζ) and the output value y M LC (ξ, ζ) are equal. Also, y in Equation (4) n L-1C (ξ + u, ζ + v), y in Equation (5) M LS (ξ + u, ζ + v) are respectively called the output value of the connection destination of the feature detection neuron and the output value of the connection destination of the feature integration neuron.

[0049] We will explain ξ, ζ, u, v, and n in Equations (4) and (5). The position (ξ, ζ) corresponds to the position coordinates in the input image. For example, y M LSA high output value for (ξ,ζ) indicates a high probability that the feature to be detected exists at the pixel position (ξ,ζ) of the input image in the M-th cell plane of the L-th layer S. In Equation 4, n represents the n-th cell plane of the L-1-th layer C, and is called the integration target feature number. Basically, a sum-of-products operation is performed on all cell planes present in the L-1-th layer C. (u,v) is the relative position coordinate of the connection coefficient, and the sum-of-products operation is performed within a finite range ((u,v) range) corresponding to the size of the feature to be detected. This finite range of (u,v) is called the receptive field. The size of the receptive field is referred to as the receptive field size below and is expressed as the number of horizontal pixels × the number of vertical pixels in the connected range.

[0050] Also, in equation 4, when L=1, i.e., the first S layer, y n L-1C (ξ+u,ζ+v) represents the input image y in_image (ξ+u,ζ+v) or input position map y in_posi_map The equation becomes (ξ+u, ζ+v). Note that the distribution of neurons and pixels is discrete, and the feature numbers of the connected nodes are also discrete; therefore, ξ, ζ, u, v, and n are not continuous variables but take discrete values. Here, ξ and ζ are non-negative integers, n is a natural number, and u and v are integers, all of which take values ​​within a finite range.

[0051] w in equation 4 M LS (n,u,v) is a coupling coefficient distribution for detecting a given feature, and by adjusting this to an appropriate value, it becomes possible to detect the given feature. This adjustment of the coupling coefficient distribution is learning (deep learning). Various test patterns (eye images and eye information as input, and correct feature quantities as training data) are presented to the learning model, which is a CNN, and y M LS The connection coefficients are repeatedly and gradually modified until (ξ,ζ) yields an appropriate output value. In this way, a trained CNN can be constructed.

[0052] w in equation 5 M LC(u,v) can be expressed using a two-dimensional Gaussian function as shown in equation 6 below.

[0053]

number

[0054] Here, the range (u,v) is finite, and similar to the explanation of feature detection neurons, this finite range is called the receptive field, and the size of this range is called the receptive field size. This receptive field size can be set to an appropriate value depending on the size of the M-th feature in the S-th layer of the L-th hierarchical layer. σ in equation 6 is the feature size factor, and it can be set to an appropriate constant depending on the receptive field size. Specifically, it is best to set σ to a value such that the outermost value of the receptive field can be considered approximately 0.

[0055] By performing the above-described operations at each layer, the features used for user identification can be obtained in the final S-layer. Alternatively, the user identification process could be constructed using a CNN, and the user identification result could be output from the CNN.

[0056] While the example described uses the CPU (Central Processing Unit) 212 of camera 100 to perform neural network calculations, this is not the only option. For example, camera 100 may have a GPU (Graphics Processing Unit), and the GPU may perform the neural network calculations. A GPU is a computing unit capable of performing processing specialized for computer graphics calculations, and can generally perform the calculations required for neural networks (e.g., matrix operations) in less time than the CPU 212. Neural network calculations may also be performed on a server in the cloud. Neural network calculations consist of two main phases: a learning phase and an inference phase. Learning and inference may be performed on different devices.

[0057] <Summary> When only low-resolution eye images are used, it is not possible to identify the user with high accuracy. According to Example 1, by further using eye information, it is possible to identify the user with high accuracy. Furthermore, since the eye information is obtained from the eye image, manual input of information by the user is unnecessary. Thus, according to Example 1, by obtaining eye information from the eye image and identifying the user based on the eye image and eye information, it is possible to identify the user with high accuracy even when using low-resolution eye images, without requiring manual input of information by the user.

[0058] Furthermore, the processing after user identification is not limited to the selection of eye-tracking correction parameters. For example, the system may automatically set the user's preferred imaging parameters (e.g., shutter speed and aperture value). In this case, a table of data linking users to imaging parameters is maintained, and the imaging parameters are obtained by referring to this table according to the identified user.

[0059] Furthermore, gaze detection calibration may be performed when performing personal identification operations. The user may then be identified based on the eye image, eye information, and the gaze correction parameters resulting from the calibration. The gaze may not be accurately estimated due to factors such as individual differences in the shape of the human eyeball. Specifically, if the gaze correction parameters mentioned above are not adjusted to values ​​suitable for the user, a discrepancy will occur between the actual gaze point B and the estimated gaze point C, as shown in Figure 4(b). In Figure 4(b), the user is fixated on the person, but camera 100 incorrectly estimates that the user is fixated on the background, resulting in a state where proper focus detection and adjustment are not possible. Gaze detection calibration is necessary to ensure the gaze point is suitable for the user. This is done to obtain gaze correction parameters. Calibration is performed, for example, by highlighting multiple indicators at different positions on the screen of the display device 214, as shown in Figure 4(c), and having the user look at these indicators. When the user fixates on each indicator, gaze detection is performed, and gaze correction parameters suitable for the user are determined from the calculated multiple viewpoints (estimated positions) and the coordinates of the multiple indicators. The method of displaying the indicators is not particularly limited as long as the position where the user should look is indicated; the indicators may be displayed as graphics, or they may be displayed by changing at least one of the brightness and color of an image (for example, an captured image). Since the gaze correction parameters obtained in this way vary from person to person, these gaze correction parameters can be used as personal identification information (authentication information).

[0060] <<Example 2>> The following describes Example 2 of the present invention. In the following, the same points as in Example 1 (e.g., configuration and processing) will be omitted, and the differences from Example 1 will be described.

[0061] Example 1 describes an example of identifying a user using eyepiece distance information (information about the position of the eye relative to the viewfinder) as eye information. Example 2 describes an example of using eye information that includes other information about the position of the eye relative to the viewfinder and information about the attitude of the eye relative to the viewfinder. For example, as other information about the position of the eye relative to the viewfinder, information about the amount of deviation (shift) of the eye from the optical axis of the viewfinder is used. As information about the attitude of the eye relative to the viewfinder, information about at least one of the roll rotation amount, pitch rotation amount, and yaw rotation amount is used. The roll rotation amount is the amount of rotation (tilt) of the eye in the roll direction from a predetermined attitude of the eye facing the optical axis of the viewfinder. The pitch rotation amount is the amount of rotation (tilt) of the eye in the pitch direction from the above predetermined attitude. The yaw rotation amount is the amount of rotation (tilt) of the eye in the yaw direction from the above predetermined attitude.

[0062] Figure 14 shows eye shift and rotation. Depending on how the viewfinder is looked through, not only the eyepiece distance but also the amount of eye shift (shift in the X-axis direction and the Y-axis direction) and the amount of eye rotation (roll rotation, pitch rotation, and yaw rotation) change. Therefore, both the information on the amount of shift and the information on the amount of rotation can be used as personal identification information (authentication information).

[0063] Figures 15(a) to 15(g) show the state of the eye (relationship between pupil, eyelid, and corneal reflection image) according to Example 2. In Example 2, four light sources are used in step S704 of Figure 7. Therefore, four corneal reflection images P1, P2, P3, and P4 are shown in Figures 15(a) to 15(g). Figure 16 is a flowchart of the eye information acquisition operation according to Example 2.

[0064] In step S1601 of Figure 16, the CPU 212 (eye information acquisition unit 801) calculates the intervals ΔPx and ΔPy between the four corneal reflection images P1, P2, P3, and P4. The interval ΔPx is the interval in the X-axis direction, and is, for example, the difference between the X coordinate of corneal reflection image P3 and the X coordinate of corneal reflection image P4, the difference between the X coordinate of corneal reflection image P1 and the X coordinate of corneal reflection image P2, or the average of these two differences. The interval ΔPy is the interval in the Y-axis direction, and is, for example, the difference between the Y coordinate of corneal reflection image P1 and the Y coordinate of corneal reflection image P3, the difference between the Y coordinate of corneal reflection image P2 and the Y coordinate of corneal reflection image P4, or the average of these two differences.

[0065] Figure 15(a) shows the state where the eye is facing directly forward (along the optical axis of the viewfinder). Assume that in the state of Figure 15(a), intervals ΔPx = ΔPx1 and ΔPy = ΔPy1 can be calculated. Figure 15(b) shows the state where the eye has rotated in the yaw direction from the state of Figure 15(a). Assume that in the state of Figure 15(b), intervals ΔPx = ΔPx2 and ΔPy = ΔPy2 can be calculated. Figure 15(c) shows the state where the eye has rotated in the pitch direction from the state of Figure 15(a). Assume that in the state of Figure 15(c), intervals ΔPx = ΔPx3 and ΔPy = ΔPy3 can be calculated.

[0066] In step S1602, the CPU 212 (eye information acquisition unit 801) calculates the aspect ratio ΔPy / ΔPx of the intervals ΔPx and ΔPy calculated in step S1601.

[0067] In step S1603, the CPU 212 (eye information acquisition unit 801) determines the amount of pitch rotation or yaw rotation based on the aspect ratio calculated in step S1602. In the state shown in Figure 15(b) (when the eye is rotated in the yaw direction), an aspect ratio ΔPy2 / ΔPx2 is calculated that is larger than the aspect ratio ΔPy1 / ΔPx1 in the state shown in Figure 15(a). Then, in the state shown in Figure 15(c) (when the eye is rotated in the pitch direction), an aspect ratio ΔPy3 / ΔPx3 is calculated that is smaller than the aspect ratio ΔPy1 / ΔPx1 in the state shown in Figure 15(a). Therefore, if the aspect ratio ΔPy1 / ΔPx1 is stored in the memory unit 213 as a reference value, the amount of pitch rotation or yaw rotation can be determined according to the difference between the reference value and the aspect ratio ΔPy / ΔPx calculated in step S1602. A function or conversion table showing the correspondence between the aspect ratio ΔPy / ΔPx calculated in step S1602 and the amount of rotation may be prepared in advance for determining the amount of rotation. Information for calculating the reference value (for example, the coordinates of the corneal reflection images P1, P2, P3, P4, or the intervals ΔPx, ΔPy) may be stored in the memory unit 213 in advance, and the reference value may be calculated based on this information.

[0068] In the state shown in Figure 15(b) (when the eye is rotated in the yaw direction), a smaller interval ΔPx2 is calculated than the interval ΔPx1 in the state shown in Figure 15(a). Therefore, the amount of yaw rotation may be determined according to the difference between the interval ΔPx1 and the interval ΔPx calculated in step S1601. In the state shown in Figure 15(c) (when the eye is rotated in the pitch direction), a smaller interval ΔPy3 is calculated than the interval ΔPy1 in the state shown in Figure 15(a). Therefore, the amount of pitch rotation may be determined according to the difference between the interval ΔPy1 and the interval ΔPy calculated in step S1601.

[0069] In step S1604, the CPU 212 (eye information acquisition unit 801) calculates the central position Pc of the four corneal reflection images P1, P2, P3, and P4.

[0070] In step S1605, the CPU 212 (eye information acquisition unit 801) determines (calculates) the difference between the position of the pupil central image c' and the center position Pc calculated in step S1604 as the amount of eye displacement. Figure 15(d) shows the state in which the eye is not displaced from the optical axis of the viewfinder. In the state of Figure 15(d), the center position Pc = Pc1 coincides with the position of the pupil central image c'. Figure 15(e) shows the state in which the eye is displaced from the state of Figure 15(d). In the state of Figure 15(e), the center position Pc = Pc2 is displaced from the position of the pupil central image c'. Therefore, the difference between the position of the pupil central image c' and the center position Pc calculated in step S1604 can be used as the amount of eye displacement from the optical axis of the viewfinder. The difference in the X coordinate can be used as the amount of displacement in the X-axis direction, and the difference in the Y coordinate can be used as the amount of displacement in the Y-axis direction.

[0071] In step S1606, as shown in Figures 15(f) and 15(g), the CPU 212 (eye information acquisition unit 801) detects the eyelid line from the eye image. For example, because the light reflectivity differs between the eyelid and the pupil, the brightness of the eyelid region and the brightness of the pupil region are different in the eye image. Therefore, based on the brightness difference, the boundary between the eyelid region and the pupil region can be detected from the eye image as the eyelid line.

[0072] In step S1607, the CPU 212 (eye information acquisition unit 801) calculates the inclination of the eyelid line detected in step S1606. For example, as shown in Figures 15(f) and 15(g), the distance from the straight line passing through the corneal reflection images P3 and P4 to the eyelid line is calculated for the position of the left end of the pupil and the position of the right end of the pupil, respectively. In the state of Figure 15(f), distances Dif1 and Dif2 are calculated, and in the state of Figure 15(g), distances Dif3 and Dif4 are calculated. Then, the inclination of the eyelid line is calculated based on the two calculated distances. For example, the difference between the two distances The tilt of the eyelid line can be calculated as the slope of the eyelid line, or it can be calculated from the difference between two distances and the pupil diameter. A predetermined value may be used as the pupil diameter, or a value detected from the eye image may be used.

[0073] In step S1608, the CPU 212 (eye information acquisition unit 801) determines the amount of roll rotation according to the inclination of the eyelid line calculated in step S1607. Figure 15(f) shows the state in which the eye is in a predetermined position. Figure 15(g) shows the state in which the eye has rotated in the roll direction from the state in Figure 15(f). As shown in Figures 15(f) and 15(g), the inclination of the eyelid line changes as the eye rotates in the roll direction. Therefore, the amount of roll rotation can be determined according to the inclination of the eyelid line. A function or conversion table showing the correspondence between the inclination of the eyelid line and the amount of roll rotation may be prepared in advance for determining the amount of roll rotation.

[0074] In step S1609, the CPU 212 (eye information acquisition unit 801) determines the eyepiece distance in the same manner as in Example 1.

[0075] According to Example 2, the eye information includes not only information on the eyepiece distance, but also information on the amount of eye displacement and information on the amount of eye rotation. As a result, the user can be identified with higher accuracy than in Example 1.

[0076] Furthermore, at least one of the following may be used as eye information: information regarding the position of the eye relative to the viewfinder, and information regarding the attitude of the eye relative to the viewfinder. Only the information regarding the position of the eye relative to the viewfinder may be used as eye information, or only the information regarding the attitude of the eye relative to the viewfinder may be used as eye information. Three pieces of information may be used as eye information: information regarding the eyepiece distance, information regarding the amount of eye displacement, and information regarding the amount of eye rotation, or one or two of these three pieces of information may be omitted. For the amount of displacement, only the amount of displacement in the X-axis direction and the amount of displacement in the Y-axis direction may be used, or a vector quantity obtained by combining the amount of displacement in the X-axis direction and the amount of displacement in the Y-axis direction may be used. For the amount of rotation, at least one of the three rotation amounts—roll rotation, pitch rotation, and yaw rotation—may be used, or one or two of these three rotation amounts may be omitted. Rotation amounts in directions different from the roll, pitch, and yaw directions may be used.

[0077] <<Other Examples>> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0078] This embodiment includes the following configurations, methods, and programs. (Composition 1) A first acquisition means for acquiring an eye image by capturing the user's eye as they look through the viewfinder of the imaging device, A second acquisition means for acquiring eye information from the eye image relating to at least one of the position and orientation of the eye relative to the viewfinder, The eye image and the eye information are used to identify the user. An identification device characterized by having the following features. (Configuration 2) The second acquisition means acquires the eye information based on at least one of the positions of the multiple corneal reflection images in the eye image, the spacing between the multiple corneal reflection images, and the tilt of the eyelids in the eye image. The identification device according to configuration 1, characterized in that it is a device. (Composition 3) The eye information includes, as information relating to the position of the eye relative to the viewfinder, information relating to the distance from the viewfinder to the eye in the optical axis direction of the viewfinder. An identification device according to configuration 1 or 2, characterized by the above. (Composition 4) The aforementioned eye information includes, as information relating to the position of the eye relative to the viewfinder, information relating to the amount of deviation of the eye from the optical axis of the viewfinder. An identification device according to any one of configurations 1 to 3, characterized by the above. (Composition 5) The eye information includes, as information relating to the orientation of the eye relative to the viewfinder, information relating to at least one of the following: the amount of rotation of the eye in the roll direction, the amount of rotation of the eye in the pitch direction, and the amount of rotation of the eye in the yaw direction, from a predetermined orientation of the eye facing the optical axis direction of the viewfinder. An identification device according to any one of configurations 1 to 4, characterized by the above. (Composition 6) The system further includes a third acquisition means for acquiring parameters used for eye gaze detection as a calibration result of the eye gaze detection, The identification means identifies the user based on the eye image, the eye information, and the parameters. An identification device according to any one of configurations 1 to 5, characterized in that it is a device. (Composition 7) The system further includes a selection means that, based on the identification result of the identification means, selects a parameter corresponding to the user from a plurality of parameters corresponding to a plurality of persons. An identification device according to any one of configurations 1 to 6, characterized in that it is an identification device. (Composition 8) The identification means identifies the user using a neural network that takes the eye image and eye information as input and outputs the user's feature quantities. An identification device according to any one of configurations 1 to 7, characterized by the above. (Method 1) The steps include acquiring an eye image of a user's eye looking through the viewfinder of an imaging device, and eye information relating to at least one of the position and orientation of the eye relative to the viewfinder, The steps include: using the acquired eye image and eye information to perform deep learning on a neural network that takes the eye image and eye information as input and outputs the user's features; and A learning method characterized by having the following features. (Method 2) The steps include acquiring an eye image by capturing the user's eye as they look through the viewfinder of the imaging device, The steps include obtaining eye information from the eye image relating to at least one of the position and orientation of the eye relative to the viewfinder, A step of identifying the user based on the eye image and the eye information. An identification method characterized by having the following features. (program) A program for causing a computer to function as one of the means of an identification device described in any one of configurations 1 to 8. [Explanation of Symbols]

[0079] 100: Camera 212: CPU 801: Eye information acquisition unit 802: Feature acquisition unit 804: Feature matching unit 805: Parameter selection unit

Claims

1. A first acquisition means for acquiring an eye image by capturing the user's eye as they look through the viewfinder of the imaging device, A second acquisition means for acquiring eye information from the aforementioned eye image, relating to at least one of the position and orientation of the eye relative to the viewfinder, The eye image and the eye information are used to identify the user. An identification device characterized by having the following features.

2. The second acquisition means acquires the eye information based on at least one of the positions of the multiple corneal reflection images in the eye image, the spacing between the multiple corneal reflection images, and the tilt of the eyelids in the eye image. The identification device according to feature 1.

3. The eye information includes, as information relating to the position of the eye relative to the viewfinder, information relating to the distance from the viewfinder to the eye in the optical axis direction of the viewfinder. The identification device according to feature 1.

4. The aforementioned eye information includes, as information relating to the position of the eye relative to the viewfinder, information relating to the amount of deviation of the eye from the optical axis of the viewfinder. The identification device according to feature 1.

5. The eye information includes, as information relating to the orientation of the eye relative to the viewfinder, information relating to at least one of the following: the amount of rotation of the eye in the roll direction, the amount of rotation of the eye in the pitch direction, and the amount of rotation of the eye in the yaw direction, from a predetermined orientation of the eye facing the optical axis direction of the viewfinder. The identification device according to feature 1.

6. The system further includes a third acquisition means for acquiring parameters used for eye gaze detection as a calibration result of the eye gaze detection, The identification means, based on the eye image, the eye information, and the parameters, determines the user Identify the user The identification device according to feature 1.

7. The identification means identifies the user using a neural network that takes the eye image and eye information as input and outputs the user's feature quantities. The identification device according to feature 1.

8. The steps include acquiring an eye image of a user's eye looking through the viewfinder of an imaging device, and eye information relating to at least one of the position and orientation of the eye relative to the viewfinder, The steps include: using the acquired eye image and eye information to perform deep learning on a neural network that takes the eye image and eye information as input and outputs the user's features; and A learning method characterized by having the following features.

9. The steps include acquiring an eye image by capturing the user's eye as they look through the viewfinder of the imaging device, The steps include obtaining eye information from the eye image relating to at least one of the position and orientation of the eye relative to the viewfinder, A step of identifying the user based on the eye image and the eye information. An identification method characterized by having the following features.

10. A program for causing a computer to function as one of the means of an identification device according to any one of claims 1 to 7.

Images