Imaging device, control method for imaging device, and program

Abstract

Conventional technologies make it difficult to selectively acquire registered images that meet multiple different criteria. [Solution] An imaging device characterized by comprising: an imaging means for capturing an image of a user's eyeball; a first determination means for determining whether a first eyeball image captured by the imaging means satisfies a first condition; a second determination means for determining whether a second eyeball image captured by the imaging means satisfies a second condition; and an information presentation means for presenting first presentation information to the user when the first condition is used and when the second condition is used, upon obtaining a user operation to register an eyeball image.

Description

Technical Field

[0001] The present invention relates to an imaging device, a control method for the imaging device, and a program.

Background Art

[0002] In an imaging device that is held and used by hand, user authentication may be performed by comparing a pre-registered image with an image obtained by an imaging means built in the imaging device for imaging the user. Such authentication is highly convenient because it is completed only by capturing an image of the user with the imaging means. Conventionally, it has been used in use cases assuming that the user captures an image suitable for authentication as much as possible, such as a login function to the imaging device.

[0003] In order to correctly perform authentication using an image, it is necessary to determine that the person shown in the pre-registered image (hereinafter, referred to as "registered image" as necessary) and the person shown in the image captured by the imaging means (hereinafter, referred to as "authentication image" as necessary) are the same person. Therefore, conventionally, user authentication has been performed using iris authentication based on the iris of the eye, and efforts have been made to make the appearance of the iris of the eye shown in the registered image and the authentication image as similar as possible. Patent Document 1 describes a technique for obtaining an authentication image that is easy to perform iris authentication by changing the displayed UI (User Interface) according to the brightness of the environment at the time of obtaining the authentication image, which is an eye image of the user, and the size of the pupil in the authentication image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

[0005] When there is a significant difference in appearance between the registered image and the authentication image, the accuracy of user authentication decreases. However, with conventional technology, it was difficult to selectively acquire the appropriate registered image according to the authentication situation in order to maintain authentication accuracy.

[0006] This invention was made to solve the above-mentioned problems and aims to enable the acquisition of registered images according to the authentication situation. [Means for solving the problem]

[0007] The imaging device of the present invention includes: an imaging means for capturing an image of a user's eyeball; a first determination means for determining whether a first eyeball image captured by the imaging means satisfies a first condition; a second determination means for determining whether a second eyeball image captured by the imaging means satisfies a second condition; an information presentation means for presenting first presentation information to the user when the first condition is used and when the second condition is used, when a user operation for registering an eyeball image is obtained; a first registration means for controlling the first determination means to register at least one of the first eyeball image or image features obtainable from the first eyeball image when the first determination means determines that the first eyeball image satisfies the first condition; and a second registration means for controlling the second determination means to register at least one of the second eyeball image or image features obtainable from the second eyeball image when the second determination means determines that the second eyeball image satisfies the second condition. [Effects of the Invention]

[0008] It is possible to retrieve registered images according to the authentication situation. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the external appearance of an imaging device according to the first embodiment. [Figure 2] This figure shows an example of the internal mechanism of the imaging device according to the first embodiment. [Figure 3]This figure shows an example of the electrical configuration of the imaging device according to the first embodiment. [Figure 4] This figure shows a first embodiment and an example of the display on the screen of a display device. [Figure 5] (a) A diagram showing an example of the functional configuration of the imaging device according to the first embodiment. (b) A diagram showing the first embodiment and an example of various tables managed by the registration data management unit shown in (a). (c) A diagram showing the first embodiment and an example of various tables managed by the registration data management unit shown in (a). (d) A diagram showing the first embodiment and an example of various tables managed by the registration data management unit shown in (a). (e) A diagram showing the first embodiment and an example of an authentication status table managed by the authentication status management unit shown in (a). [Figure 6] This flowchart shows an example of a detailed processing procedure for the registration process in the control method of the imaging device according to the first embodiment. [Figure 7] Figure 6 is a flowchart showing an example of a detailed processing procedure for acquiring eye images in step S604. [Figure 8] This flowchart shows an example of a detailed processing procedure for the registration process in the control method of the imaging device according to the first embodiment. [Figure 9] This is an example of a visual guidance display in the first embodiment. [Figure 10] This is an example of a display for guiding the user's gaze when the device is held vertically in the first embodiment. [Figure 11] This is an example of a display that guides the eyepiece position in Embodiment 1. [Figure 12] This is an example of the display during the eye image acquisition process in Modification Example 2. [Figure 13] This is a flowchart of the eye image acquisition process in Modification Example 3. [Figure 14] This is an example of eye-tracking guidance in Modification 3. [Figure 15] This is an example of an eyepiece position guidance display in Modification 4. [Figure 16] This is an example of displaying eyepiece position guidance, including the current position, in modified example 4. [Figure 17] This is an example of a display for guiding the eye relief distance and the degree of eyelid opening in Modification 5. [Figure 18] This is a flowchart showing an example of a registration process in the control method of the imaging device according to the second embodiment. [Figure 19] This is an example of a display during the acquisition of an eye image according to the second embodiment. [Figure 20] This is an example of a display when the acquisition of an eye image fails according to the second embodiment. [Figure 21] This is a flowchart of the eye image acquisition process in Modification 1 of the second embodiment. [Figure 22] This is an example of a display during the acquisition of an eye image in Modification 1 of the second embodiment.

Mode for Carrying Out the Invention

[0010] Hereinafter, the present invention will be described in detail based on its preferred embodiments with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

[0011] Hereinafter, modes (embodiments) for carrying out the present invention will be described while referring to the drawings.

[0012] (First Embodiment) First, the first embodiment will be described.

[0013] In the present embodiment, in an imaging device having a first eye authentication function and a second eye authentication function for a user, a mode for acquiring the user's eye image used for each authentication as a registration image will be described. Further, a mode for acquiring the user's eye image suitable for each authentication as an authentication image using the acquired registration image will be described. ]>

[0014] Here, the first eye authentication (hereinafter simply referred to as "first authentication") is performed when the user begins using the imaging device, before capturing an image of the subject based on the user's operation. This first authentication requires strict accuracy. In other words, it is assumed that when the first eye authentication function is executed, an authentication image is acquired under specific environmental conditions. Therefore, the registered image used for the first eye authentication function must be an image that matches the conditions under which the first eye authentication function is executed.

[0015] On the other hand, the second eye authentication (hereinafter simply referred to as "second authentication") is performed after the first authentication described above, and is performed when capturing an image of a subject based on user operation (for example, authentication performed in conjunction with the timing when the user presses the shutter button). Compared to the first authentication described above, this second authentication is required to interfere with the user's shooting action as little as possible. In other words, when performing the second eye authentication function, it is thought that authentication images will be acquired in a variety of environments. Therefore, the registered images used for the second eye authentication function need to be images acquired in a variety of environments.

[0016] [Configuration of the imaging device (camera)] Figure 1 shows an example of the external appearance of the imaging device 100 according to the first embodiment. Specifically, the imaging device 100 according to this embodiment can be, for example, a lens-interchangeable digital still camera. Figure 1(a) is a front perspective view showing an example of the external appearance of the imaging device 100 according to the first embodiment. Figure 1(b) is a rear perspective view showing an example of the external appearance of the imaging device 100 according to the first embodiment. In Figures 1(a) and 1(b), the same components are denoted by the same reference numerals. Figures 1(a) and 1(b) also illustrate an XYZ coordinate system in which the optical axis direction of the imaging device 100 is defined as the Z-axis direction, and two directions perpendicular to the Z-axis direction that are mutually orthogonal are defined as the X-axis direction and the Y-axis direction.

[0017] As shown in Figure 1(a), the imaging device 100 includes a shooting lens unit 110 and a camera housing 120. A release button 121, which is an operating member that receives imaging operations from the user (photographer), is located on the front of the camera housing 120.

[0018] As shown in Figure 1(b), an eyepiece 122 (finder) is positioned on the back of the camera housing 120, into which the user looks to view the display device (display device 214, described later in Figure 2) contained inside the camera housing 120. Furthermore, operating members 123 to 125 that receive various operations from the user are also positioned on the back of the camera housing 120. For example, operating member 123 is a touch panel that accepts touch operations, operating member 124 is an operating lever that can be pushed down in each direction, and operating member 125 is a four-way key that can be pressed in each of the four directions. Operating member 123, which is a touch panel, is equipped with a display panel (e.g., a liquid crystal panel), and this display panel constitutes the rear display of the imaging device 100. In addition, operating member 123, which is a touch panel, has the function of displaying various images on this display panel.

[0019] Figure 2 shows an example of the internal mechanism of the imaging device 100 according to the first embodiment. Specifically, Figure 2 is a cross-sectional view of the imaging device 100 when it is cut along the YZ plane formed by the Y-axis and Z-axis directions shown in Figure 1(a). In Figure 2, the same reference numerals are used for components that are the same as those shown in Figure 1, and their detailed explanations are omitted. Also, Figure 2 shows an XYZ coordinate system corresponding to the XYZ coordinate system shown in Figure 1.

[0020] The photographic lens unit 110 has, as an internal mechanism, lenses 201 and 202, an aperture 203, an aperture drive unit 204, a lens drive motor 205, a lens drive member 206, a pulse plate 207, a photocoupler 208, a focus adjustment circuit 209, and a mount contact 210. Here, the lens drive member 206 includes a drive gear. The photocoupler 208 detects the rotation of the pulse plate 207, which is linked to the lens drive member 206, and transmits this information to the focus adjustment circuit 209. The focus adjustment circuit 209 drives the lens drive motor 205 based on information from the photocoupler 208 and information from the camera housing 120 (information on the amount of lens drive), moving the lens 201 to change the focus position. The mount contact 210 is the interface between the photographic lens unit 110 and the camera housing 120. Note that while Figure 2 shows two lenses 201 and 202 for simplicity, in reality, the imaging lens unit 110 contains more than two lenses.

[0021] The camera housing 120 includes, as internal components, an image sensor 211, a CPU 212, a memory unit 213, a display device 214, a display device driving circuit 215, light sources 216a and 216b, an optical splitter 217, a light-receiving lens 218, and an eye image sensor 219. Furthermore, the camera housing 120 also includes, as internal components, a speaker 220 and an eyepiece sensor 221.

[0022] The image sensor 211 is positioned on the planned imaging plane of the imaging lens unit 110. This image sensor 211 is a component that is an example of a first imaging means that captures (photographs) a subject based on user operation and acquires a subject image. The CPU 212 is the central processing unit of the microcomputer and controls the operation of the entire imaging device 100 and performs various processes. The memory unit 213 stores various information and programs that the CPU 212 executes when performing various processes. For example, the memory unit 213 stores the subject image captured (photographed) by the image sensor 211. The display device 214 displays various information on the screen (display surface) of the display device 214. For example, the display device 214 is a liquid crystal panel and displays the captured image (subject image) on the screen. The display device driving circuit 215 drives the display device 214. The user's eye (eyeball) E can see the screen of the display device 214 through the eyepiece lens 122.

[0023] Light sources 216a and 216b are light sources that have been conventionally used in single-lens reflex cameras to detect the line of sight of the eye (eyeball) E from the relationship between the reflected image (corneal reflection image) due to corneal reflection of light and the pupil. Specifically, light sources 216a and 216b are light sources for illuminating the eye (eyeball) E of the user looking through the viewfinder (eyepiece 122). For example, light sources 216a and 216b are infrared light-emitting diodes that emit infrared light that is insensitive to the user's eye (eyeball) E, and are arranged around the eyepiece 122. The optical image of the eye (eyeball) E illuminated by light sources 216a and 216b (eye optical image; optical image due to reflected light emitted from light sources 216a and 216b and reflected by the eye (eyeball) E) passes through the eyepiece 122 and is reflected by the light divider 217. The optical image of the eye is then formed by a light-receiving lens 218 onto an eye image sensor 219, which has multiple photoelectric conversion elements (e.g., CCD or CMOS) arranged in two dimensions. The light-receiving lens 218 positions the pupil of the user's eye (eyeball) E and the eye image sensor 219 in a conjugate imaging relationship. By performing gaze detection processing, the gaze of the eye (eyeball) E is detected from the position of the corneal reflection image in the optical image of the eye formed on the eye image sensor 219. For example, as information regarding the gaze, at least one piece of information is obtained, which includes information indicating the direction of the gaze and information indicating the viewpoint (the position where the gaze is directed) on the screen of the display device 214. The viewpoint can be considered as the position the user is looking at, or as the gaze position. The eye image sensor 219 is also a component that is an example of a second imaging means for capturing (shooting) an eye image including the user's eye (eyeball) E as an authentication image.

[0024] The speaker 220 is a component that serves as an example of a sound output means for playing and outputting shutter sounds and voice guidance. The eyepiece sensor 221 is a component that acquires eyepiece information of the user's eye (eyeball) E as a state of the user's eye (eyeball) E. For example, as an example of eyepiece information, the eyepiece sensor 221 measures the user's eyepiece distance, which is the distance from the eyepiece sensor 221 to the user's eye (eyeball) E, and detects whether the user is looking through the viewfinder (eyepiece lens 122).

[0025] Figure 3 shows an example of the electrical configuration of the imaging device 100 according to the first embodiment. In Figure 3, the same reference numerals are used for components that are the same as those shown in Figures 1 and 2, and their detailed descriptions are omitted.

[0026] The photographic lens unit 110 has an electrical configuration consisting of a focus adjustment circuit 209 and an aperture control circuit 306, as shown in Figure 2.

[0027] The camera housing 120 has, as an electrical configuration, a release button 121 and operating members 123-125 as shown in Figure 1. The camera housing 120 also has, as an electrical configuration, an image sensor 211, a CPU 212, a memory unit 213, a display device 214, a display device driving circuit 215, light sources 216a, 216b, an eye image sensor 219, a speaker 220, and an eyepiece sensor 221 as shown in Figure 2. Furthermore, as an electrical configuration, the camera housing 120 has, as shown in Figure 3, a gaze detection circuit 301, a photometering circuit 302, an autofocus detection circuit 303, a signal input circuit 304, and a light source driving circuit 305.

[0028] As shown in Figure 3, the CPU 212 is connected to the gaze detection circuit 301, the photometering circuit 302, the autofocus detection circuit 303, the signal input circuit 304, and the light source drive circuit 305. Furthermore, as shown in Figure 3, the CPU 212 is connected to the image sensor 211, the memory unit 213, the display device drive circuit 215, the speaker 220, the eyepiece sensor 221, and the operating members 123-125. The CPU 212 also transmits signals via the mount contact 210 to the focus adjustment circuit 209 located in the imaging lens unit 110 and the aperture control circuit 306 included in the aperture drive unit 204 in the imaging lens unit 110. The memory unit 213 attached to the CPU 212 has, for example, a function to store imaging signals from the image sensor 211 and the eye image sensor 219.

[0029] The gaze detection circuit 301 performs A / D conversion on the output of the eye image sensor 219 (eye image captured from the user's eye (eyeball) E) when an optical image of the eye is formed on the eye image sensor 219, and transmits the result to the CPU 212. The CPU 212 extracts feature points necessary for gaze detection from the eye image according to the gaze detection process, and detects the user's gaze from the position of the feature points.

[0030] The photometering circuit 302 performs predetermined processing (e.g., amplification, logarithmic compression, and A / D conversion) on the signal obtained from the image sensor 211, which also acts as a photometering sensor, for example, a luminance signal corresponding to the brightness of the field of view, and sends the result to the CPU 212 as field of view luminance information.

[0031] The autofocus detection circuit 303 performs A / D conversion on signals from multiple detection elements (multiple pixels) in the image sensor 211 used for phase-difference detection and sends them to the CPU 212. The CPU 212 calculates the distance to the subject corresponding to each focus detection point from the signals of the multiple detection elements. This is a known technique known as image-plane phase-difference autofocus.

[0032] Figure 4 shows a first embodiment and illustrates an example of the display on the screen of the display device 214. As an example, in this embodiment, each of the 180 locations on the imaging surface corresponding to the 180 locations shown on the screen of the display device 214 (viewfinder field of view) in Figure 4(a) is assumed to have a focus detection point as described by the autofocus detection circuit 303. Figure 4(a) shows the display device 214 in operation (with an image displayed), and the viewfinder field of view includes the focus detection area 401, the field of view mask 402, and 180 autofocus point indicators 410 within the focus detection area 401. Each autofocus point indicator 410 shown in Figure 4(a) is displayed superimposed on the through image (live view image) displayed on the display device 214 so as to be displayed at the position corresponding to the focus detection point on the imaging surface. In addition, of the 180 autofocus point indicators 410 shown in Figure 4(a), the autofocus point indicator 410A corresponding to the current viewpoint A (estimated position) is displayed highlighted, for example, with a frame. Furthermore, the viewpoint may not be able to be estimated with high accuracy due to factors such as individual differences in the shape of the human eye E. Specifically, as shown in Figure 4(b), there may be a discrepancy between the actual viewpoint B (410B) and the estimated viewpoint C (410C).

[0033] Now, let's return to the explanation of Figure 3.

[0034] The signal input circuit 304 is connected to switches SW1 and SW2 of the release button 121. Switch SW1 is turned ON on the first stroke of the release button 121 and is used to start the shooting preparation operation of the imaging device 100 (e.g., photometry and distance measurement). Switch SW2 is turned ON on the second stroke of the release button 121 and is used to start the shooting operation. When the ON signals from switches SW1 and SW2 of the release button 121 are input to the signal input circuit 304, the signal input circuit 304 transmits the input ON signals to the CPU 212. At this point, user gaze detection may be started when switch SW1 of the release button 121 is turned ON.

[0035] The light source driving circuit 305 drives the light sources 216a and 216b.

[0036] Furthermore, when the user operates the operating members 123 to 125, the operating members 123 to 125 output an operation signal to the CPU 212 corresponding to the user's operation. The CPU 212 then performs processing (control) according to the operation signal. For example, the CPU 212 moves the selection frame of the displayed menu according to the operation signal.

[0037] Figure 5(a) is a diagram showing an example of the functional configuration of the imaging device 100 according to the first embodiment. The imaging device 100 is an imaging device that authenticates a user. The imaging device 100 has the following functional configurations: an eye image acquisition unit 501, a registration unit 502, a registration data management unit 503, a first authentication unit 504, a second authentication unit 505, and an authentication state management unit 506. Furthermore, the imaging device 100 has the following functional configurations: an eyeball state acquisition unit 507, a first presentation information determination unit 508, a second presentation information determination unit 509, and an information presentation unit 510. For example, the CPU 212 shown in Figures 2 and 3 executes a program stored in the memory unit 213 shown in Figures 2 and 3, thereby realizing each of the functional configurations (501 to 510) shown in Figures 5(a) to (e).

[0038] In this embodiment, the imaging device 100 authenticates whether the user is a registered person by examining the user's eye (eyeball) E as they look through the viewfinder (eyepiece 122) as a form of user authentication. Specifically, a first authentication is performed before the user uses the imaging device 100 to photograph a subject, and a second authentication is performed when the user uses the imaging device 100 to photograph a subject (for example, at the time of shooting). In this embodiment, the imaging device 100 stores these authentication results along with the captured image.

[0039] The eye image acquisition unit 501 is an eye image acquisition means that acquires an eye image including the eye (eyeball) E of the user looking through the viewfinder (eyepiece lens 122). Specifically, the eye image acquisition unit 501 acquires an eye image (eye image signal; electrical signal of the eye image) from the eye image sensor 219 shown in Figure 3 via the gaze detection circuit 301. In this embodiment, the eye image acquisition unit 501 acquires an eye image as a registration image in the case of registration processing for a registered user, and acquires an eye image as an authentication image in the case of authentication processing for a user to be authenticated.

[0040] The registration unit 502 is a registration means that creates data to be registered in the registration data management unit 503 based on the eye image of the registered user acquired by the eye image acquisition unit 501.

[0041] The registration data management unit 503 is a registration data management means that manages authentication registration information of users authorized to use the imaging device 100. Specifically, the registration data management unit 503 stores in the memory unit 213 the eye image of the registered user, which is a registered eye image acquired in advance, a feature vector calculated from the registered user's eye image, and the position of the index corresponding to the feature vector, linked to the registered user's name and person ID. In this embodiment, the registered user enters their name at the beginning of the registration process, and the registration data management unit 503 manages the entered name (and person ID) of the registered user in a table linked to the registered data.

[0042] Figures 5(b) to 5(d) show a first embodiment and illustrate examples of various tables managed by the registration data management unit 503 shown in Figure 5(a). Specifically, the registered personal information table 520 shown in Figure 5(b) is a table that associates the person ID with the name of the registered user. The first authentication registration data table 530 shown in Figure 5(c) is a table that associates the person ID with the eye image of the registered user used in the first authentication unit 504, the feature vector used in the first authentication unit 504, and the position of the index corresponding to the feature vector. The second authentication registration data table 540 shown in Figure 5(d) is a table that associates the person ID with the eye image of the registered user used in the second authentication unit 505, the feature vector used in the second authentication unit 505, and the position of the index corresponding to the feature vector. These various tables 520 to 540 shown in Figures 5(b) to 5(d) link information of the same registered user by person ID. Here, a neural network, for example, is used to extract feature vectors, and the extracted feature vectors are used for user authentication. Note that the method of user authentication using eye images is not limited to the neural network method described above; other known methods, such as iris recognition, may also be used.

[0043] Furthermore, Figure 5(e) shows the first embodiment and is a diagram illustrating an example of the authentication status table 550 managed by the authentication status management unit 506 shown in Figure 5(a). This authentication status table 550 shown in Figure 5(e), like the various tables 520 to 540 shown in Figures 5(b) to (d), links information of the same registered user by person ID.

[0044] Now, let's return to the explanation of Figure 5(a).

[0045] The first authentication unit 504 is a first authentication means that authenticates the user to be authenticated based on the eye image (authentication image) of the user to be authenticated, obtained by the eye image sensor 219 and acquired by the eye image acquisition unit 501, before the subject image is captured by the image sensor 211. In this embodiment, the first authentication unit 504 performs a process to authenticate whether the user to be authenticated is a person (registered user) registered in the registration data management unit 503 when the subject image is not being captured by the image sensor 211.

[0046] The second authentication unit 505 is a second authentication means that authenticates the user to be authenticated based on the eye image (authentication image) of the user to be authenticated, obtained by the eye image sensor 219 and acquired by the eye image acquisition unit 501 when the image sensor 211 is capturing a subject image. In this embodiment, the second authentication unit 505 performs a process to authenticate whether or not the person (user to be authenticated) authenticated by the first authentication unit 504 is capturing the subject image when the image sensor 211 is capturing the subject image. The second authentication unit 505 also controls the authentication state of the second authentication to continue only while the subject image is being captured.

[0047] In this embodiment, it is desirable for the first authentication unit 504 to use an authentication setting that lowers the false acceptance rate. On the other hand, it is desirable for the second authentication unit 505 to use an authentication setting that lowers the true rejection rate. One way to achieve such authentication settings is, for example, to change the similarity threshold when the authentication method used by the first authentication unit 504 and the second authentication unit 505 is the same. In this case, specifically, a high similarity threshold is set for the first authentication unit 504, and a low similarity threshold is set for the second authentication unit 505. This makes it possible to lower the false acceptance rate in the first authentication unit 504 and lower the true rejection rate in the second authentication unit 505. From the above, when it is difficult to lower both the false acceptance rate and the true rejection rate simultaneously during shooting, this can be achieved by performing two-stage authentication.

[0048] The authentication status management unit 506 is an authentication status management means that manages the authentication status of the first authentication by the first authentication unit 504 and the authentication status of the second authentication by the second authentication unit 505. In addition, the authentication status management unit 506 also manages whether or not the system is being used by another person. For example, the authentication status management unit 506 maintains the authentication status table 550 shown in Figure 5(e) in the memory unit 213 and performs management.

[0049] The authentication status table 550 shown in Figure 5(e) will be explained below.

[0050] The "First Authentication Status" in the Authentication Status Table 550 indicates whether or not the first authentication by the first authentication unit 504 is being performed, and can take one of the following values: "Not Authenticated," "Authentication Failed," or "Authentication Successful." The Person ID is the ID of the person identified by the first authentication. When the first authentication status is not authenticated, the Person ID will be a value indicating that it is empty, such as NULL. The "Second Authentication Status" in the Authentication Status Table 550 indicates whether or not the second authentication by the second authentication unit 505 is being performed, and can take one of the following values: "Not Authenticated," "Authentication Failed," or "Authentication Successful." The "Number of First Authentication Failures" in the Authentication Status Table 550 indicates the number of first authentication failures by the first authentication unit 504, and can take a non-negative integer value. The "Number of Second Authentication Failures" in the Authentication Status Table 550 indicates the number of second authentication failures by the second authentication unit 505, and can take a non-negative integer value. The authentication status table 550 shown in Figure 5(e) is stored and maintained in the memory unit 213. The method of maintaining the authentication status table 550 is not limited to a table structure; for example, a key-value structure may also be used.

[0051] The eyeball state acquisition unit 507 is an eyeball state acquisition means that acquires the state of the user's eye (eyeball) E from eye images acquired by the eye image acquisition unit 501 and eye images stored in the registration data management unit 503. Here, the state of the user's eyeball refers to, for example, the user's gaze information. User gaze detection is performed, for example, by using the method described in Patent Document 2 to determine the coordinates of the corneal reflection images of light sources 216a and 216b observed on the eye image and the center of the pupil, and then determining the gaze coordinates on the user's display device 214 from these coordinates. For example, if the gaze of a user looking into the viewfinder (eyepiece 122) is acquired, the UI display can show the user how to move from the current gaze position to obtain a registration image and authentication image suitable for authentication. Furthermore, the gaze information acquired as the state of the user's eye (eyeball) E by the eyeball state acquisition unit 507 includes the direction of the user's gaze, and in this case, the eyeball state acquisition unit 507 is configured to include a gaze sensor that acquires the user's gaze information. Furthermore, the state of the user's eye (eyeball) E acquired by the eyeball state acquisition unit 507 is not limited to the user's gaze information described above. For example, the state of the user's eye (eyeball) E acquired by the eyeball state acquisition unit 507 may include the user's eyepiece information. In this case, the eyeball state acquisition unit 507 includes an eyepiece sensor 221 that acquires the user's eyepiece information. Also, for example, the state of the user's eye (eyeball) E acquired by the eyeball state acquisition unit 507 may include eye image information obtained by image processing of eye images acquired by the eye image acquisition unit 501 or eye images stored in the registration data management unit 503. In this case, the eye image information may include, for example, information such as the degree of eyelid and pupil dilation of the user, the user's eyepiece position and angle, and whether or not eyewear is being worn.

[0052] The standard setting unit 508 sets the standards for eye images to be registered in the registration data management unit 503 from the eye images acquired by the eye image acquisition unit 501. Multiple standards can be set, and the registration data management unit 503 decides whether or not to register each standard (condition). Specifically, this will be explained later using an example of standards in Figure 6.

[0053] The presentation information determination unit 509 is a first determination means that determines first presentation information, which includes guidance information for guiding at least one of the position and orientation of the user's eye E when the eye image sensor 219 captures an eye image when the first authentication unit 504 authenticates the user. In this case, for example, the first presentation information determination unit 508 may determine the first presentation information based on the criteria (conditions) set by the criteria setting unit 508 and the current state of the user's eye (eyeball) E acquired by the eyeball state acquisition unit 507. Also, for example, the first presentation information determination unit 508 determines UI information as the first presentation information so that the eye image acquisition unit 501 can acquire an eye image that satisfies the first criteria (conditions) for use in first authentication and the second criteria (conditions) for use in second authentication. In this embodiment, the presentation information determination unit 508 determines the first or second presentation information by selecting an appropriate one from among the presentation information, including multiple UI displays and multiple audio guides, stored in the memory unit 213, based on the criteria set by the criteria setting unit 508. The presentation information determination unit 509 may also make the above selection from the presentation information stored in the memory unit 213 based on the authentication status of the first authentication managed by the authentication status management unit 506 and the eye image acquired by the eye image acquisition unit 501 during the first authentication. Similarly, the presentation information determination unit 509 may also make the above selection from the presentation information stored in the memory unit 213 based on the authentication status of the second authentication managed by the authentication status management unit 506 and the eye image acquired by the eye image acquisition unit 501 during the second authentication. Furthermore, the presentation information determination unit 509 selects the information to be presented based on the state of the user's eye E acquired by the eyeball state acquisition unit 507 during the first and second authentications and the registered eye image registered in the registration data management unit 503.

[0054] The information presentation unit 510 is an information presentation means that, upon receiving a user operation to register a registered image, presents to the user either the first presentation information or the second presentation information determined by the information determination unit 509 based on the reference setting unit 508. Furthermore, the information presentation unit 510 is an information presentation means that presents to the user the presentation information determined by the presentation information determination unit 509 when the eye image sensor 219 captures an eye image when the first authentication unit 504 and the second authentication unit 505 authenticate the user. For example, the information presentation unit 510 that processes the presentation of the first and second presentation information is composed of at least one means, which is a display means such as a display device 214 or an operating member 123 such as a touch panel, and a sound output means such as a speaker 220. The information presentation unit 510 then displays the first and second presentation information on the display means as a UI display or outputs it as a voice guide from the sound output means.

[0055] Figure 6 is a flowchart showing an example of a detailed processing procedure for the registration process in the control method of the imaging device 100 according to the first embodiment. The processing in the flowchart shown in Figure 6 is mainly executed by the CPU 212 via the registration unit 502. Furthermore, the processing in the flowchart shown in Figure 6 is assumed to be performed by the user operating the imaging device 100 at times other than when taking images. Therefore, the processing in the flowchart shown in Figure 6 is executed when the user operates the imaging device 100 and calls up this process from a menu or the like. For example, a menu screen (not shown) is displayed on the touch panel (operating member 123) of the imaging device 100, and when the user operates the operating members 123 to 125 on that menu screen and selects a menu that calls up this process, this process is executed.

[0056] First, in step S601 of Figure 6, the registration unit 502 accepts input of personal information of the person to be registered (registered user). In this embodiment, the "name" is accepted as input. Specifically, the information display unit 510 displays a screen for entering a name (not shown) on the touch panel (operation member 123), and the registration unit 502 accepts the name input from the user through operation of operation members 123 to 125. When the user finishes inputting, they notify that the name input is complete by pressing the completion button or the like displayed on the screen, and the process proceeds to step S602.

[0057] In step S602, the registration unit 502 receives input to determine which of two different types of authentication, first eyeball authentication and second eyeball authentication, the eye image to be registered for. Specifically, the information display unit 510 displays a GUI (Graphical User Interface) for selecting authentication (not shown) on the touch panel (operating member 123), and the registration unit 502 accepts the user's input. Once the user selects the authentication type by touching a GUI such as a button, the process transitions to step S603.

[0058] In step S603 of Figure 6, the settings related to the registration criteria (registration conditions) for eye images are acquired. That is, for two different types of authentication, first eyeball authentication and second eyeball authentication, the criteria (conditions) for the eye images to be registered as registered images for each authentication are acquired. The criteria are assumed to be pre-set by the administrator user according to the purpose of using the camera. For example, a user who does not want to impair usability during shooting can select criteria that allow for variations in the number of registered images or increase the number of registered images. Alternatively, the system may use pre-set criteria from the factory without user selection. In step S603, the registration unit 502 acquires the criteria associated with the type of authentication selected in step S602. After acquisition, the system proceeds to step S604.

[0059] Figure 7 shows examples of the first criterion (first condition) and the second criterion (second condition). In Figure 7, Table 700 is an example of the first and second criteria. Also, 701 is a superimposed display of points P0 to P4 with predetermined coordinates on the image acquired when the user looks through the camera's viewfinder (hereinafter referred to as the camera image). In this embodiment, the first criterion is "the distance between the center point P0 of the camera image and the pupil center position is within 10 pixels," and an image in which the pupil center is near the center of the camera image is registered as the first image. The second criterion is "the distance between points P1 to P4 of the camera image and the pupil center position is within 10 pixels," and an image in which the pupil center is near points P1 to P4 is registered as the second image. In other words, the registration criteria are set according to the distance between the detected pupil position and a predetermined position. The first criterion used for the first authentication is set so that the distance between the pupil position and a predetermined position is less than or equal to a predetermined distance. In the above example, the value "within 10 pixels" is set, but other values ​​may be set, and the value may be set appropriately depending on the resolution of the camera image, the detection accuracy of the pupil center, etc. A criterion regarding the number of images taken may also be established. In this embodiment, it is explained that the criterion (condition) and the display content to be displayed to the user are associated and stored for each of the first and second criterions.

[0060] Next, in step S604 of Figure 6, the information display unit 510 displays the registration method for eye images to the user based on the registration criteria acquired in step S602. Specifically, the information display unit 510 displays instructions on the touch panel (operating member 123) to the user to look through the viewfinder. The information display unit 510 also displays instructions to look at the indicators in the viewfinder. In addition, the information display unit 509 determines the information to be displayed according to the criteria acquired in step S602. Furthermore, the information display unit 510 displays the determined information. For example, it displays instructions that enable the capture of desirable eye images, such as not blinking and keeping the eyes wide open.

[0061] Next, the indicators 411 to 415 shown in Figure 4(c) are displayed sequentially on the display device 214 through the processing of steps S605 to S611 in Figure 6. Then, the user's eye image when viewing the displayed indicator, the feature vector obtained from that eye image, and the position of the indicator are stored in the registration data management unit 503. The process will be explained in order.

[0062] In step S605 of Figure 6, the information display unit 510 displays an indicator on the display device 214. Specifically, it displays only indicator 411 as shown in Figure 4(c), and does not display the other indicators. Alternatively, all indicators 411 to 415 as shown in Figure 4(c) may be displayed, and indicator 411 may be highlighted with a different color. Other display methods are also acceptable as long as they effectively communicate to the user that they should look at indicator 411.

[0063] Next, in step S605 of Figure 6, the eye image acquisition unit 501 acquires an eye image when the user looks through the viewfinder (eyepiece 122). Here, the detailed processing of step S605 in Figure 6 will be explained using Figure 8.

[0064] Figure 8 is a flowchart showing an example of a detailed processing procedure for acquiring eye images in step S605 of Figure 6. The processing shown in the flowchart of Figure 8 is mainly executed by the CPU 212 via the eye image acquisition unit 501.

[0065] First, in step S801 of Figure 8, the eye image acquisition unit 501 performs user gaze detection processing. This user gaze detection processing can be performed, for example, using the method described in Patent Document 2, to determine the coordinates of the corneal reflection images of the light sources 216a and 216b observed on the eye image and the center of the pupil, and then to determine the gaze coordinates on the user's display device 214 from these coordinates.

[0066] Next, in step S802 of Figure 7, the eye image acquisition unit 501 determines whether or not it has acquired an eye image that meets the registration criteria for use in authentication. Specifically, the eye image acquisition unit 501 determines whether or not it has acquired an eye image suitable for registration for use in authentication based on whether or not the gaze detection process in step S801 was successful. For example, in the gaze detection process in step S801, the eye image acquisition unit 501 acquires an eye image (eye image signal; electrical signal of the eye image) from the eye image sensor 219 via the gaze detection circuit 301. Next, the eye image acquisition unit 501 determines the coordinates of the corneal reflection images of the light sources 216a and 216b and the pupil center observed on the eye image. Next, the eye image acquisition unit 501 determines the gaze coordinates on the user's display device 214 from the obtained coordinates. If the eye image acquisition unit 501 cannot detect the coordinates of the pupil center, etc., it considers the gaze detection process to have failed. Therefore, if the coordinates of the pupil center and other coordinates cannot be obtained during the process of obtaining those coordinates, the eye image acquisition unit 501 may determine that it was not possible to obtain an eye image suitable for authentication.

[0067] Next, in step S803 of Figure 8, the eye image acquisition unit 501 determines whether or not it has succeeded in acquiring an eye image suitable for authentication based on the result of the determination in step S802.

[0068] In step S803 of Figure 8, if the eye image acquisition unit 501 determines that it has successfully acquired an eye image suitable for registration for use in authentication (S703 / YES), the process proceeds to step S804.

[0069] When the process proceeds to step S804 in Figure 8, the eye image acquisition unit 501 crops and acquires an eye image. Specifically, first, the eye image acquisition unit 501 obtains the eye image acquired in step S802 in Figure 8. Then, using the coordinates of the pupil center image c' obtained in step S803 in Figure 8, the eye image acquisition unit 501 crops a certain size so that the pupil center image c' becomes the center of the image. Furthermore, the eye image acquisition unit 501 generates and acquires an eye image resized to the input size of the neural network used for feature vector extraction.

[0070] Next, in step S805 of Figure 8, the eye image acquisition unit 501 records, using a flag or similar, that it has successfully acquired an eye image.

[0071] Furthermore, if the eye image acquisition unit 501 determines in step S803 of Figure 8 that it has not succeeded in acquiring an eye image suitable for authentication (S803 / NO), the process proceeds to step S806.

[0072] When the process proceeds to step S806 in Figure 7, the eye image acquisition unit 501 performs a process of waiting for a predetermined time, such as several hundred milliseconds. This process in step S806 is a process that expects the eye image to change and for gaze detection to be successful.

[0073] Next, in step S807 of Figure 8, the eye image acquisition unit 501 determines whether the failure to acquire an eye image suitable for authentication occurs a predetermined number of times consecutively. Here, failure refers to the case where it is determined in step S803 of Figure 7 that it is not possible to acquire an eye image suitable for authentication (S703 / NO).

[0074] In step S807 of Figure 87, if the eye image acquisition unit 501 determines that the number of consecutive failures to acquire an eye image suitable for authentication has not reached a predetermined number (S807 / NO), the process returns to step S801 and repeats the processing from step S801 onwards.

[0075] Furthermore, in step S807 of Figure 8, if the eye image acquisition unit 501 determines that the acquisition of an eye image suitable for authentication has failed a predetermined number of times consecutively (S807 / YES), the process proceeds to step S808.

[0076] When the process proceeds to step S808 in Figure 7, the eye image acquisition unit 501 records, using a flag or similar, that it failed to acquire the eye image. At this point, the eye image acquisition unit 501 may also record the eye image acquired in step S802 in Figure 8.

[0077] When the process in step S805 in Figure 8 is completed, or when the process in step S808 in Figure 8 is completed, the process in the flowchart in Figure 8 is completed. Then, when the process in the flowchart in Figure 8 is completed, the eye image acquisition process in step S605 in Figure 6 is completed.

[0078] Now, let's return to the explanation of Figure 6.

[0079] Once the process in step S605 in Figure 6 is completed, the process proceeds to step S606.

[0080] When the process proceeds to step S606 in Figure 6, the eye image acquisition unit 501 determines whether or not it has successfully acquired an eye image. Specifically, the eye image acquisition unit 501 determines whether or not it has successfully acquired an eye image based on the flag recorded in step S805 or step S808 in Figure 8.

[0081] In step S606 of Figure 6, if the eye image acquisition unit 501 determines that it has successfully acquired an eye image (S606 / YES), the process proceeds to step S608.

[0082] When the process proceeds to step S608 in Figure 6, the eye image acquisition unit 501 or the registration unit 502 extracts feature vectors from the eye image acquired in step S605 in Figure 6 as authentication registration information for authenticating the user. Here, a neural network, for example, is used to extract the feature vectors, and the extracted feature vectors are used for user personal authentication. Note that the method of user personal authentication using eye images is not limited to the neural network method described above, and other known methods such as iris recognition may also be used.

[0083] Next, in step S608 of Figure 6, the information display unit 510 displays on the display device 214 that it has been successful in capturing an eye image at the index currently displayed on the display device 214. For example, the information display unit 510 may display a message such as "Eye image captured successfully" on the display device 214, or it may display an icon or the like to indicate success.

[0084] Furthermore, if the eye image acquisition unit 501 determines in step S606 of Figure 6 that it has not succeeded in acquiring the eye image (failed) (S606 / NO), the process proceeds to step S607.

[0085] When the process proceeds to step S607 in Figure 6, the information display unit 510 displays on the display device 214 that the capture of the eye image at the index currently displayed on the display device 214 has failed. For example, the information display unit 510 may display a message such as "Failed to capture eye image" on the display device 214, or it may display an icon or the like to indicate that it has failed. When the processing in step S607 is completed, the process returns to step S605.

[0086] Furthermore, once the process in step S607 in Figure 6 is completed, the process proceeds to step S609.

[0087] When the process proceeds to step S609 in Figure 6, the registration unit 502 determines whether or not to perform re-registration. Specifically, if it receives a continuation operation from the user, it decides to perform re-registration and proceeds to step S604. If no operation is received from the user, or if a user operation to terminate this control is received, this control is terminated.

[0088] In step S610 of Figure 6, it is determined whether there are any indicators that have not yet been displayed. The registration unit 502 checks whether all indicators 411 to 415 shown in Figure 4(c) are displayed and determines whether there are any indicators that have not yet been displayed.

[0089] In step S609 of Figure 6, if the registration unit 502 determines that there are indicators that have not yet been displayed (S609 / YES), the process proceeds to step S610.

[0090] When the process proceeds to step S610 in Figure 6, the registration unit 502 displays the next indicator from among the indicators that have not yet been displayed on the display device 214. For example, if indicator 411 shown in Figure 4(c) was being displayed, the registration unit 502 displays indicator 412 as the next indicator on the display device 214. In this way, indicators 411 to 415 are selected and displayed on the display device 214 in numerical order. When the processing of step S610 is completed, the process returns to step S604.

[0091] Furthermore, in step S610 of Figure 6, if the registration unit 502 determines that there are no indicators that have not yet been displayed (S609 / NO), the process proceeds to step S611.

[0092] When the process proceeds to step S611 in Figure 6, the registration unit 502 saves and registers the obtained information in the registered personal information table 520, the first authentication registration data table 530, and the second authentication registration data table 540 of the registration data management unit 503. Specifically, the person IDs in the three tables 520 to 540 shown in Figures 5(b) to 5(d) are IDs used to establish relationships between the tables, so the same ID value is used in the three tables 520 to 540. Specifically, the registration unit 502 adds the name of the personal information obtained in step S601 in Figure 6 to the registered personal information table 520 shown in Figure 5(b). The registration unit 502 also registers the feature vector obtained in step S606 in Figure 6, and the corresponding eye image and index positions, separately in the first authentication registration data table 530 shown in Figure 5(c) and the second authentication registration data table 540 shown in Figure 5(d). The method of this separate registration is as follows.

[0093] In the first authentication process shown in the first authentication registration data table 530 in Figure 5(c), an indicator is displayed on the display device 214, and authentication is performed based on the eye image when the user views that indicator. Therefore, only the feature extraction vector of the indicator displayed at that time is registered as the feature vector for first authentication registration in the first authentication registration data table 530 shown in Figure 5(c). In this embodiment, the first authentication registration data table 530 shown in Figure 5(c) contains the feature vector 1 obtained when the indicator 411 shown in Figure 4 was displayed.

[0094] Furthermore, in the second authentication process shown in the second authentication registration data table 540 in Figure 5(d), the display device 214 displays the subject image captured by the image sensor 211, so the indicators are not displayed. Therefore, it is unclear where the user will focus their attention on the display device 214. Accordingly, in this embodiment, the second authentication registration data table 540 shown in Figure 5(d) registers all of the feature vectors 1 to 5 obtained when the indicators 411 to 415 shown in Figure 4 are displayed.

[0095] Next, in step S612 of Figure 6, the registration unit 502 displays a message on the touch panel (operating member 123) or display device 214 to inform the user that registration is complete.

[0096] When the process in step S612 in Figure 6 is completed, the process in the flowchart in Figure 6 is completed. However, in the flowchart in Figure 6, the process will fall into an infinite loop unless it is determined in step S605 that the acquisition of the eye image was successful. Therefore, it is desirable to configure the system to interrupt the process if, for example, a predetermined number of failures are observed. Furthermore, it is desirable to add processing for abnormal cases as appropriate, regardless of this configuration.

[0097] The information displayed by the information display unit 510 in step S604 will be changed according to the authentication type as follows. For example, as shown in Figure 9(a), a marker 902 is displayed on the display device 214 in the viewfinder, and a message 901 is displayed to the user instructing them to focus on the marker. By having the user focus on the marker, the user's gaze is fixed, and it is possible to capture an image at a time when the eye movement is stable. In addition, by moving the position of the marker, it is possible to capture multiple eye image patterns. Here, when registering a registration image for the first authentication, the marker position in Figure 9(a) indicates the position where the eyeball is directly in front when the eyeball is photographed by the eyeball camera, and the UI display is changed so that an eye image that satisfies criterion 1 is obtained. Similarly, when registering a registration image for the second authentication, the UI display is changed continuously as shown in Figures 10(b) to (e), and the display is switched at regular intervals so that an eye image that satisfies criterion 2 can be obtained. Here, the marker display positions in Figures 9(b) to (e) are positions suitable for guiding the eyeball to an eyeball position that satisfies criterion 2. The eye-tracking process shown in Figure 9 may be performed simultaneously with the calibration process for the eye-tracking function.

[0098] Furthermore, considering the possibility of shooting with the camera held vertically, it is necessary to register images in this vertical orientation. For example, the display should be optimized for vertical orientation as shown in Figures 10(a) to (j). The display device 214 displays a message 1001 and a marker 1002 that are compatible with vertical orientation. When registering the registration image for the first authentication, Figures 11(a) and (f) guide the user to capture an eye image corresponding to criterion 1. In addition, when registering the registration image for the second authentication, the UI display is changed in Figures 10(b) to (e) and (g) to (j) to guide the eye to a position where an eye image that satisfies criterion 2 can be captured.

[0099] Furthermore, instead of guiding the user's gaze as shown in Figures 9 and 10, a method of guiding the user's eyepiece position may be used, for example, as shown in Figure 11. When registering a registration image for the first authentication, illustrations 1102 and 1103 are displayed on the display device 214 with a message 1101 guiding the user so that the eyepiece position is centered in the viewfinder, as shown in Figure 11(a). In addition, when registering a registration image for the second authentication, Figures 11(b) to (e) guide the user's eyepiece position to satisfy the second criterion by shifting the positional relationship between the camera illustration 1102 and the eye illustration 1103.

[0100] As described above, when registering a registration image to be used for first or second authentication, by determining the UI to be displayed according to the first or second criterion, it becomes possible to acquire a registration image in a way that minimizes the visual difference compared to the authentication image.

[0101] <Modification 1 of the first embodiment> In the first embodiment, an example was described in which a registered image is registered using an eye image obtained when the user looks through the viewfinder (eyepiece 122). Specifically, the ability to register an eye image was determined using a criterion based on the distance to the center of the pupil in the eye image obtained when looking through the viewfinder. However, the system is not limited to this, and images may be registered using other criteria. Examples of criteria are described below.

[0102] For example, criteria for first and second authentication may be set based on the detection of a specified number of infrared reflective areas from the pupil or iris region. First, multiple infrared emitting devices are installed in the camera's viewfinder beforehand. Next, when capturing eye images, an infrared camera is used to capture images in which infrared reflective areas exist on the pupil or iris region. Subsequently, infrared reflective areas are detected from the acquired image. Finally, the image is registered only if a number of infrared reflective areas equal to or greater than the threshold is detected. Here, for example, the threshold may be set to 10 for the first criterion and to 5 for the second criterion, and these can be used as both criteria.

[0103] Alternatively, the gaze direction may be defined as a predetermined direction. For example, gaze directions could be classified into nine categories (up, down, left, right, diagonal, and forward), and one image for each of the nine gaze directions could be registered. In this case, the criteria could be determined to be met if the degree of agreement between the predetermined gaze direction and the acquired gaze direction is high. In this case, the degree of agreement set as the first criterion is higher than the degree of agreement set as the second criterion. Alternatively, the criteria could be set as a predetermined range instead of a degree of agreement. In this case, the range set as the second criterion is described as encompassing the range set as the first criterion.

[0104] Furthermore, as another example, the second criterion could be defined as "the difference in features between an image that satisfies the first criterion and an image that satisfies the first criterion is within a certain range." First, only images that satisfy the first criterion are registered. Next, for subsequent input images, the image features of the registered images and the input images are compared, and only if the difference in features is within a certain range is it determined that the second criterion is satisfied. The image features used here could be, for example, the image features and feature differences used in the authentication process.

[0105] Another example is using "variations in pupil diameter." For instance, the first criterion could be a commonly used pupil diameter size, while the second criterion could be a standard for registering multiple pupil diameter variations. The second authentication method is used in situations where the environment at the shooting location may change, and it is necessary to increase the variety of registered images to facilitate successful authentication in a short time in various environments, such as bright outdoor locations and dark indoor locations.

[0106] <Modification 2 of the first embodiment> In the first embodiment, a method was shown in which the user's gaze is guided by moving a marker as shown in Figures 9 and 10 while the user is looking through the viewfinder. However, the UI display may reflect the actual user's usage. For example, when changing the position of the marker, the UI display may instruct the user to take their eye off the viewfinder and look through it again each time. Specifically, as shown in Figure 12(b), one of the marker displays 1002 shown in Figure 10 is displayed, and then as shown in Figure 12(c), the display device 214 displays the message 1201 "Please lower the camera once and look through the viewfinder again." When the eyepiece sensor 107 detects that the user has taken their eye off the viewfinder, Figure 12(a) is displayed on the touch panel 103 to instruct the user to look through the viewfinder. Once the user has looked through the viewfinder, the next display is the marker with the changed position shown in Figure 12(b). At this time, the marker positions are displayed randomly in order from the marker positions shown in Figure 9 or, if registered in portrait orientation, in Figure 10. By using this modified method, it is possible to acquire registered images that include variations in the actual eyepiece movements of users, which is expected to improve the accuracy of authentication during use.

[0107] <Modification 3 of the first embodiment> As shown in Figure 6, the UI display for eye images in the registration process may be determined not only by a standard but also by the state of the user's eyeball while using the eyepiece. Figure 13 is a flowchart illustrating the flow of this modified process. The same reference numerals are used for the same processes as in Figure 6 of the first embodiment. The differences in the processes will be explained below.

[0108] In S1301, the current eye state is acquired. Here, we will explain the case where gaze information is acquired as the eye state.

[0109] In step S1302, the UI is displayed based on the criteria obtained in step S603 and the eye condition obtained in S1301.

[0110] When eye movement is acquired as eye state, the marker position that the user should focus on is calculated from the difference between the current gaze position and the gaze position that satisfies the criteria. As shown in Figure 14(a), the marker position 1403 moves from the gazed-on marker 1401 to the gaze position direction 1402 that satisfies the criteria. By guiding the user's gaze as the marker position moves, it is possible to get closer to the target position, and an improvement in the quality of the registered image can be expected.

[0111] <Modification 4 of the first embodiment> In Modification 3, it is explained that the eye position may be obtained as the current state of the eye, but the invention is not limited to this. For example, the eye is detected using the eye detection method described in Embodiment 1 on the acquired eye image. Furthermore, if the coordinates of its center are taken as the current eye position coordinate C1, then when using the references described in Figures 5(a) to (e), it is possible to calculate the orientation that minimizes the difference between the reference coordinates being used and C1 among the reference coordinates P0 to P4. As shown in Figure 15, by displaying a message 1501 that guides the user's eye position in the determined direction, an illustration 1503 indicating the eye position, and an arrow 1502, it is possible to get closer to the target eye position.

[0112] As shown in Figure 16, an icon 1601 representing the current eye position may be displayed in real time on the display device 214. Alternatively, a message 1603 instructing the user to align their eye position with the guide, and a guide 1602 representing the eye position where the difference between the current eye position coordinates and the reference coordinates is minimized, may be displayed on the display device 214. As a result, the user can intuitively adjust the eyepiece position to a position suitable for meeting the criteria.

[0113] <Modification 5 of the First Embodiment> In addition to line of sight, the eye distance and eye angle may also be acquired as the current state of the eyeball. The eye distance is calculated using the output value of the eye sensor 107 to determine the distance between the camera and the user's eyeball. Alternatively, the eye distance and angle may be calculated from the acquired eye image using image processing. When the size and visibility of the iris region in the eye image are used as a criterion, the size of the area occupied by the eyeball and the size of the iris region in the image will change depending on the eye distance and angle, affecting the results of region detection. For example, if the criteria are "the ability to detect a specified number or more infrared reflection areas from the pupil region or iris region" or "the degree of eyelid opening," the state of the eyeball will change. At this time, a message 1701 to guide the user's eye distance, as shown in Figure 17(a), and an arrow 1702 indicating the direction to adjust the eye distance are displayed between the illustration of the user's eyeball 1703 and the illustration of the camera 1704. Also, a message 1705 prompting adjustment of the eye angle is displayed, as shown in Figure 17(b). Furthermore, the eyepiece angle can be adjusted by displaying illustration 1706, which shows the user's current eyepiece angle, illustration 1708, which shows an eyepiece angle that meets the standard, and arrow 1707, which indicates the direction in which to adjust the eyepiece angle.

[0114] <Modification 6 of the First Embodiment> The UI display examples (Figures 9, 10, 11, 12, 13, 14, 16, and 17) shown in this embodiment, which guide the user to satisfy the criteria, may be provided via voice guidance instead of being displayed on the display device 214. The voice guidance is played back by the speaker 106. The content of the voice guidance can be the same as reading aloud the message displayed on the display device. Furthermore, if an arrow is displayed on the display device, the direction of the arrow can be expressed in words and played back as a voice guide, which is expected to improve user accessibility.

[0115] (Second embodiment) In the first embodiment, as shown in Figures 9 to 17, in the eye image acquisition process used for registration images, the user's gaze and eye position were guided by the UI display, and the eye image was acquired in step S611 when an eye image that met the criteria was obtained. In this embodiment, we will describe a case in which the user themselves determines the timing of eye image acquisition.

[0116] The imaging device in this embodiment has the hardware configuration shown in Figures 1, 2, and 3. It is also a camera with the functional block configuration shown in Figures 5(a) to 5(e). Furthermore, while the registration process flow shown in Figure 6 is the same as in the first embodiment, the eye image acquisition process differs from that of the first embodiment. Therefore, the following explanation will focus on the differences between the two embodiments, omitting explanations of similar parts. Also, parts corresponding to those in the first embodiment will be denoted by the same reference numerals.

[0117] Figure 18 shows the eye image acquisition process in this embodiment. It is basically the same as the process shown in Figure 13, but the processes from S1801 to S1805 in Figure 18 have been added.

[0118] In step S1301 of Figure 18, an eye condition score is calculated and displayed, representing the degree to which the criteria are met based on the current eye condition acquired in S1301. For example, in the case of the criteria shown in Figure 6 of Embodiment 1, if the eye condition score is S, and the Manhattan distance between the acquired coordinates of the current eye condition and the reference coordinates is D, then S=f(D) The eye state score S is calculated using the function f, where f is high when the distance D is close to 0 and decreases proportionally to the distance D. For clarity, the score may be normalized to be expressed in the range of 0 to 100. The function used is not limited to the one described, as long as it has the same qualitative properties. Also, the method of calculating the score is not limited to the method using distance D; other methods that indicate whether the image is suitable for registration as an eye recognition image may also be used.

[0119] As shown in Figure 19, the eye condition score 1901 calculated during the user's eye image acquisition process is displayed in real time on the display device 214. By displaying this information, the user can intuitively understand how to move their eyes from their current position to register an appropriate image that meets the criteria, and an improvement in the quality of registered images can be expected.

[0120] In step S1801, if it is determined that the criteria are met, a message 1904 is displayed prompting the user to press the registration button 1902 at any time, as shown in the diagram, and the process proceeds to step S1802. If it is determined that the criteria are not met, the process proceeds to step S1803.

[0121] In step S1802, it is determined whether the user pressed the registration button 1902. If the registration button is pressed, the process proceeds to step S611; otherwise, it proceeds to step S601. This flow allows eye images to be acquired at any time the user chooses.

[0122] In S1803, a process is performed to wait for a predetermined time, such as several hundred milliseconds. If the resulting eye image changes and the determination based on the criteria is successful, it is determined that the criteria are met and the process transitions to S1802, where the image is discarded.

[0123] In S1804, it is determined whether there have been consecutive failures. A failure means that in S1801, it was determined that a suitable image could not be obtained. If there have been a predetermined number of consecutive failures, the process proceeds to S1805. Otherwise, the process proceeds to S601.

[0124] In S1805, the failure to acquire the eye image is recorded using a flag or similar method.

[0125] Furthermore, the color of the score display 1901 and icon 1903 may be changed according to the score. The example above is not the only way to change the color of parts of the UI that allow the user to intuitively recognize whether the score is high or low. For example, the UI could display the current gaze position or eyepiece position as a marker, and the color of that marker could be changed according to the score.

[0126] In the first embodiment, the acquisition of eye images that satisfy the criteria is performed automatically based on whether the eye image satisfies the criteria. In this case, the burden on the user is reduced and the time required for acquisition is shortened, but since the eye image is acquired at the moment when the judgment is made and the criteria are satisfied, it does not necessarily acquire the eye image at the moment when the criteria are most satisfied. Therefore, in this embodiment, the degree of satisfaction of the eye image with the criteria is displayed as a score on the display device in real time. By performing the above display processing, the user who is registering can adjust the eye condition, such as the gaze and eyepiece position, to capture an eye image with a high score. Furthermore, by the user selecting and capturing at a moment with a high score, it becomes possible to register an eye image that is more suitable for authentication as a registered image.

[0127] <Modification 1 of the second embodiment> The failure information displayed in S607 may include the cause of the failure. For example, the eye condition is scored using multiple indicators. Furthermore, an eye image acquisition process is performed such that an eye image is acquired when all indicators meet the standard values. At this time, when registration fails, the indicator with the smallest score is presented to the user as cause information or improvement advice when the failure information is displayed, as shown in Figures 20(a) and (b). A message is displayed on the touch panel 103 or display device 214, as in 2001 or 2002.

[0128] Furthermore, if the cause of the failure is due to factors other than the condition of the eyeball, that information may also be displayed. For example, if the iris region of the eye image is obscured by unwanted reflection due to wearing glasses, the message "Please remove your glasses" may be displayed. Also, if the eye image is generally dark and unclear, the advice to "Move to a brighter location" may be displayed. In addition, if water droplets or fogging are detected on the eyepiece 102, a message such as "Please wipe the eyepiece" may be displayed to prompt the user to take action.

[0129] As shown in this modified example, by correctly informing the user of the cause of the failure to acquire eye images and prompting them to take action, the quality of the eye images that were originally intended for registration can be ensured.

[0130] <Modification 2 of the second embodiment> The processing flow of this modified example is shown in flowchart 21. The basic processing flow is the same as in Figure 13, but in Figure 21, S2101 is added, and S1302 is changed to the processing of S2102. As a result, in this modified example, the display UI can be determined taking into account the registered data, and biases or deficiencies in the registered data can be eliminated.

[0131] S2101 calculates the degree of satisfaction from registered data (satisfaction information). Satisfaction indicates the degree to which data satisfying the criteria have been collected. For example, when registering multiple images, the number of registered images that satisfy each criterion is used as the satisfaction level. Alternatively, the average, maximum, and minimum values ​​may be calculated and used from the eye condition score for each criterion.

[0132] In S2102, the UI to be displayed is determined based on the criteria, eye condition, and satisfaction level.

[0133] For example, when using eye position as a criterion, the degree of satisfaction is calculated from the registered data, and the direction of the eye position with a low degree of satisfaction is determined. Using the calculated direction and the information of the current eye position, a display is presented to the user to adjust the eye position. Specifically, the displays shown in Figures 17 and 18 are performed. Similarly, when using gaze direction as a criterion, the degree of satisfaction regarding gaze direction is determined from the registered data, and using the gaze information obtained from the current eye state, a gaze guidance UI as shown in Figure 16 can be displayed.

[0134] Alternatively, for example, eye position information with a high degree of satisfaction from the registered data may be displayed on the display device 214. As shown in Figure 22, the already registered eye position 2201 and the current eye position 2202 are displayed. By displaying the above, the user can recognize the positions with a high and low degree of satisfaction of eye position and aim to photograph the positions that are lacking. In addition, the variety of eye positions that can be registered will increase, and an improvement in authentication accuracy can be expected.

[0135] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0136] This program and a computer-readable storage medium storing said program are included in the present invention. Furthermore, the present invention can take the form of, for example, the systems, devices, methods, programs, or storage media described above. For example, it may be applied to a system consisting of multiple devices, or to a device consisting of a single device.

[0137] The embodiments of the present invention described above are merely examples of how the invention can be implemented, and the technical scope of the invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features. [Explanation of symbols]

[0138] 501 Eye image acquisition unit 502 Registration Department 503 Registration Data Management Department 508 Standard setting section 507 Eyeball state acquisition unit 509 Presentation information determination unit 510 Intelligence Probe Department

Claims

1. An imaging means for capturing images of the user's eyeballs, A first determination means for determining whether the first eyeball image captured by the imaging means satisfies the first condition, A second determination means for determining whether the second eyeball image captured by the imaging means satisfies the second condition, When user input for registering an eyeball image is obtained, the first information to be presented to the user is used when the first condition is applied. Information presentation means for presenting second presentation information to the user when the second condition described above is used, If the first determination means determines that the first eyeball image satisfies the first condition, the first registration means controls the registration of at least one of the first eyeball image or image features obtainable from the first eyeball image, If the second determination means determines that the second eyeball image satisfies the second condition, the second registration means controls the registration of at least one of the second eyeball image or image features obtainable from the second eyeball image, An imaging device characterized by having the following features.

2. A first authentication means that authenticates the user using the first eyeball image registered in the first registration means or image features obtainable from the first eyeball image, A second authentication means that authenticates the user using the second eyeball image registered in the second registration means or image features obtainable from the second eyeball image, The imaging device according to claim 1, characterized by having the following features.

3. The first determination means determines that the first condition is met when the distance between the position of the eye in the first eyeball image and a predetermined position in the first eyeball image is less than or equal to a first distance. The second determination means determines that the second condition is met if the distance between the position of the eye in the second eyeball image and the predetermined position in the second eyeball image is less than or equal to the second distance. The imaging apparatus according to claim 1, characterized in that the first distance is shorter than the second distance.

4. The first determination means determines that the first condition is met if the direction of the line of sight in the first eyeball image coincides with a predetermined direction set in advance and within a first range. The second determination means determines that the second condition is met if the direction of the line of sight in the second eyeball image coincides with the predetermined direction set in advance within the second range, The imaging apparatus according to claim 1, characterized in that the first range is included in the second range.

5. The information presentation means is at least one of the display device in the viewfinder and the rear display. The imaging device according to claim 1, characterized in that the first and second presented information are information for setting any of the user's eyelid opening, eyepiece position, eyepiece angle, eyepiece distance, or line of sight to a predetermined state.

6. The information for setting the gaze direction to a predetermined state is a marker display shown on the display device. The information presentation means is characterized by presenting the position of the marker display based on the first condition or the second condition. The imaging apparatus according to claim 5.

7. A state acquisition means for acquiring information regarding the state of the user's eyeballs, Equipped with, The information presentation means is characterized in that it determines the presentation information based on the first condition or the second condition and the state of the eyeball acquired by the state acquisition means. The imaging apparatus according to claim 5.

8. The state acquisition means is a gaze sensor, The information presentation means is characterized in that it determines the presentation information based on the direction of the user's gaze obtained by the gaze sensor. The imaging apparatus according to claim 7.

9. The state acquisition means is an eyepiece sensor, The information determination means is characterized by determining the presentation information based on the eyepiece distance between the user's eyeball and the imaging means, which is obtained by the eyepiece sensor. The imaging apparatus according to claim 7.

10. The state acquisition means is image information obtained by image processing the eyeball image, The information presentation means is characterized by determining the presentation information based on the image information. The imaging apparatus according to claim 7.

11. The device includes a calculation means for calculating a numerical value representing the degree to which the first or second condition is met, based on at least one piece of information from the eyelid opening, gaze direction, eye-eye distance, and image information obtained by image processing of the eyeball image. The information presentation means is characterized by presenting the user with a numerical value calculated by the calculation means and at least one piece of information of multiple colors. The imaging apparatus according to claim 7.

12. If the first determination means or the second determination means determines that the first condition or the second condition is not met, The information presentation means presents third information relating to the reason for determining that the first condition or the second condition is not met. The imaging apparatus according to feature 1.

13. The system includes a registration data management means for managing the data registered by the registration means, The information presentation means is characterized by determining the information to be presented based on information regarding the degree of satisfaction obtained from the registered data. The imaging apparatus according to claim 4.

14. The presented information is characterized by including information on the position of the eye with a high degree of satisfaction, obtained based on the information on the degree of satisfaction. The imaging apparatus according to claim 12.

15. The aforementioned presented information is characterized by including information regarding the position of the eyes or the direction of gaze, which is of low sufficiency. The imaging apparatus according to claim 12.

16. The information presentation means is characterized by being a sound output means that outputs the first presentation information and the second presentation information as sound. The imaging apparatus according to claim 1.

17. The imaging process involves capturing images of the user's eyeballs, A first determination step of determining whether the first eyeball image captured in the imaging step satisfies the first condition, A second determination step, which determines whether the second eyeball image captured in the imaging step satisfies the second condition, When user input for registering an eyeball image is obtained, the first information to be presented to the user is used when the first condition is applied. An information presentation step in which second information is presented to the user when the second condition described above is used, If the first determination step determines that the first eyeball image satisfies the first condition, the first registration step controls the system to register at least one of the first eyeball image or image features obtainable from the first eyeball image. If the second determination step determines that the second eyeball image satisfies the second condition, a second registration step controls the system to register at least one of the second eyeball image or image features obtainable from the second eyeball image. A control method for an imaging device, characterized by comprising:

18. A program for causing a computer to perform each step in the control method of the imaging apparatus described in claim 17.

Images