Imaging device, control method for imaging device, and program

Abstract

The goal is to ensure usability during image capture using an imaging device while acquiring authentication images in a way that minimizes the visual difference between the acquired image and the registered image. [Solution] The system includes a first authentication unit 504 that authenticates the user based on an eye image (authentication image) including the user's eyes before taking a subject image based on the user's operation, a second authentication unit 505 that authenticates the user based on the user's eye image when taking a subject image, and an information presentation unit 510 that presents the user with first presentation information including guidance information for guiding at least one of the position and orientation of the user's eyes when taking an eye image when the first authentication unit 504 authenticates the user, and presents the user with second presentation information related to the authentication process by the second authentication unit 505 when taking an eye image when the second authentication unit 505 authenticates the user.

Description

Technical Field

[0001] The present invention relates to an imaging device, a control method of the imaging device, and a program.

Background Art

[0002] In an imaging device that is held and used by hand, user authentication may be performed by comparing a pre-registered image with an image obtained by an imaging unit built in the imaging device for imaging the user. Such authentication is highly convenient because it is completed only by imaging the user's image with the imaging unit. Conventionally, it has been used in use cases assuming that the user captures an image suitable for authentication as much as possible, such as a login function to the imaging device.

[0003] In order to correctly perform authentication using an image, it is necessary to determine that the person shown in the pre-registered image (hereinafter, described as "registered image" as necessary) and the person shown in the image captured by the imaging unit (hereinafter, described as "authentication image" as necessary) are the same person. Therefore, conventionally, user authentication has been performed using iris authentication based on the iris of the eye, and efforts have been made to make the appearance of the iris of the eye shown in the registered image and the authentication image as similar as possible. Patent Document 1 describes a technique for obtaining an authentication image that is easy to perform iris authentication by changing the displayed UI (User Interface) according to the brightness of the environment at the time of obtaining the authentication image, which is the user's eye image, and the size of the pupil in the authentication image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] When there is a significant difference in appearance between the registered image and the authentication image, the accuracy of user authentication decreases. However, with conventional technology, it has been difficult to acquire authentication images that minimize the difference in appearance from the registered image while ensuring usability during image capture using an imaging device.

[0006] This invention has been made in view of these problems, and aims to enable the acquisition of authentication images that minimize the change in appearance from registered images, while ensuring usability during shooting using an imaging device. [Means for solving the problem]

[0007] The imaging device of the present invention includes: a first imaging means for capturing a subject image based on user operation; a second imaging means for capturing an eye image including the user's eyes as an authentication image; a first authentication means for authenticating the user based on the eye image obtained by the second imaging means before the subject image is captured by the first imaging means; a second authentication means for authenticating the user based on the eye image obtained by the second imaging means when the subject image is captured by the first imaging means; and an information presentation means for presenting the user with first presentation information including guidance information for guiding at least one of the position and orientation of the user's eyes when the second imaging means captures the eye image when the first authentication means authenticates the user, and for presenting the user with second presentation information relating to the authentication process by the second authentication means when the second imaging means captures the eye image when the second authentication means authenticates the user. [Effects of the Invention]

[0008] According to the present invention, it is possible to acquire an authentication image while ensuring usability during shooting using an imaging device, and minimizing the change in appearance compared to the registered image. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the external appearance of an imaging device according to the first embodiment. [Figure 2] This figure shows an example of the internal mechanism of the imaging device according to the first embodiment. [Figure 3] This figure shows an example of the electrical configuration of the imaging device according to the first embodiment. [Figure 4] This figure shows a first embodiment and an example of the display on the screen of a display device. [Figure 5A] This figure shows an example of the functional configuration of the imaging device according to the first embodiment. [Figure 5B] This figure shows an example of various tables managed by the registration data management unit shown in Figure 5A, illustrating the first embodiment. [Figure 5C] This figure shows an example of various tables managed by the registration data management unit shown in Figure 5A, illustrating the first embodiment. [Figure 5D] This figure shows an example of various tables managed by the registration data management unit shown in Figure 5A, illustrating the first embodiment. [Figure 5E] This figure shows an example of an authentication status table managed by the authentication status management unit shown in Figure 5A, illustrating the first embodiment. [Figure 6] This flowchart shows an example of a detailed processing procedure for the registration process in the control method of the imaging device according to the first embodiment. [Figure 7] Figure 6 is a flowchart showing an example of a detailed processing procedure for acquiring eye images in step S604. [Figure 8] This flowchart shows an example of a detailed processing procedure for the first authentication process in the control method of the imaging device according to the first embodiment. [Figure 9] Figure 8 is a flowchart showing an example of a detailed processing procedure for when eye image acquisition fails in step S805. [Figure 10] This flowchart shows an example of a detailed processing procedure for the matching process in step S806 of Figure 8. [Figure 11]It is a flowchart showing an example of the detailed processing procedure at the time of successful authentication in step S808 of FIG. 8. [Figure 12] It is a flowchart showing an example of the detailed processing procedure at the time of failed authentication in step S809 of FIG. 8. [Figure 13] It is a diagram showing an example of the display screen displayed in the display processing of step S901 of FIG. 9. [Figure 14] It is a diagram showing an example of the display screen displayed in the display processing of step S903 of FIG. 9. [Figure 15] It is a diagram showing an example of the display screen displayed in the display processing of step S1103 of FIG. 11. [Figure 16] It is a diagram showing an example of the display screen displayed in the display processing of step S1203 of FIG. 12. [Figure 17] It is a flowchart showing an example of the detailed processing procedure of the second authentication process in the control method of the imaging device according to the first embodiment. [Figure 18] It is a flowchart showing an example of the detailed processing procedure of the collation process in step S1705 of FIG. 17. [Figure 19] It is a flowchart showing an example of the detailed processing procedure at the time of successful authentication in step S1707 of FIG. 17. [Figure 20] It is a flowchart showing an example of the detailed processing procedure at the time of failed authentication in step S1709 of FIG. 17. [Figure 21] It is a diagram showing an example of the display screen displayed in the display processing of step S1704 of FIG. 17. [Figure 22] It is a diagram showing an example of the display screen displayed in the display processing of step S1902 of FIG. 19. [Figure 23] It is a diagram showing an example of the display screen displayed in the display processing of step S2002 of FIG. 20. [Figure 24] It is a diagram showing an example of the display screen displayed in the display processing of step S1205 of FIG. 12. [Figure 25]This figure shows an example of the display screen that appears during the display process in step S2004 of Figure 20. [Figure 26] This figure shows a modified example of the first embodiment, illustrating an example of a display screen that shows guidance information included in the first presented information. [Figure 27] This figure shows a modified example of the first embodiment, illustrating an example of a display screen that shows guidance information included in the first presented information. [Figure 28] This figure shows a modified example of the first embodiment, specifically a second example of a display screen that shows guidance information included in the first presented information. [Figure 29] This figure shows a modified example of the first embodiment, specifically a third example of a display screen 2900 that displays the first information to be presented. [Figure 30] This figure shows a modified example 5 of the first embodiment, and is an example of a display screen that displays guidance information included in the first presented information. [Figure 31] This figure shows a modified example 6 of the first embodiment, illustrating an example of a display screen that displays guidance information included in the first presented information. [Figure 32] The second embodiment is shown in the flowchart, which is an example of a detailed processing procedure for handling authentication failure in step S1709 of Figure 17. [Figure 33] This figure shows an example of the display screen that appears during the display process in step S3203 of Figure 32. [Figure 34] This figure shows a modified example of the second embodiment, specifically an example of the display screen displayed in step S3203 of Figure 32. [Figure 35] This figure shows a modified example of the second embodiment, specifically an example of the display screen displayed in the display processing step S3203 of Figure 32. [Figure 36] This figure shows a modified example of the second embodiment, specifically an example of the display screen displayed in the display processing step S3203 of Figure 32. [Figure 37] This figure shows a modified example of the second embodiment, part 3, and illustrates an example of a display screen that shows guidance information included in the second presentation information. [Figure 38]This figure shows a modified example 4 of the second embodiment, illustrating an example of a display screen that shows guidance information included in the second presented information. [Figure 39] This figure shows a modified example 5 of the second embodiment, illustrating an example of a display screen that shows the second presentation information. [Figure 40] This figure shows a modified example of the second embodiment, part of a diagram illustrating an example of a display screen that shows guidance information included in the second presented information. [Figure 41] This figure shows a modified example of the second embodiment, part 8, and illustrates an example of a display screen that shows the second set of information. [Modes for carrying out the invention]

[0010] The embodiments for carrying out the present invention will be described below with reference to the drawings.

[0011] (First Embodiment) First, let me describe the first embodiment.

[0012] In this embodiment, a configuration for acquiring user eye images suitable for each authentication method as authentication images in an imaging device having a first eyeball authentication function and a second eyeball authentication function for a user will be described. Here, the first eyeball authentication (hereinafter simply referred to as "first authentication") is performed when the user starts using the imaging device and before capturing a subject image based on the user's operation. This first authentication requires strictness. On the other hand, the second eyeball authentication (hereinafter simply referred to as "second authentication") is performed after the first authentication described above and when capturing a subject image based on the user's operation (for example, authentication performed in conjunction with the timing when the user presses the shutter). Compared to the first authentication described above, this second authentication requires that it interfere with the user's shooting operation as little as possible.

[0013] [Configuration of the imaging device (camera)] Figure 1 shows an example of the external appearance of the imaging device 100 according to the first embodiment. Specifically, the imaging device 100 according to this embodiment can be, for example, a lens-interchangeable digital still camera. Figure 1(a) is a front perspective view showing an example of the external appearance of the imaging device 100 according to the first embodiment. Figure 1(b) is a rear perspective view showing an example of the external appearance of the imaging device 100 according to the first embodiment. In Figures 1(a) and 1(b), the same components are denoted by the same reference numerals. Figures 1(a) and 1(b) also illustrate an XYZ coordinate system in which the optical axis direction of the imaging device 100 is defined as the Z-axis direction, and two directions perpendicular to the Z-axis direction that are mutually orthogonal are defined as the X-axis direction and the Y-axis direction.

[0014] As shown in Figure 1(a), the imaging device 100 includes a shooting lens unit 110 and a camera housing 120. A release button 121, which is an operating member that receives imaging operations from the user (photographer), is located on the front of the camera housing 120.

[0015] As shown in Figure 1(b), an eyepiece 122 (finder) is positioned on the back of the camera housing 120, into which the user looks to view the display device (display device 214, described later in Figure 2) contained inside the camera housing 120. Furthermore, operating members 123 to 125 that receive various operations from the user are also positioned on the back of the camera housing 120. For example, operating member 123 is a touch panel that accepts touch operations, operating member 124 is an operating lever that can be pushed down in each direction, and operating member 125 is a four-way key that can be pressed in each of the four directions. Operating member 123, which is a touch panel, is equipped with a display panel (e.g., a liquid crystal panel), and this display panel constitutes the rear display of the imaging device 100. In addition, operating member 123, which is a touch panel, has the function of displaying various images on this display panel.

[0016] Figure 2 shows an example of the internal mechanism of the imaging device 100 according to the first embodiment. Specifically, Figure 2 is a cross-sectional view of the imaging device 100 when it is cut along the YZ plane formed by the Y-axis and Z-axis directions shown in Figure 1(a). In Figure 2, the same reference numerals are used for components that are the same as those shown in Figure 1, and their detailed explanations are omitted. Also, Figure 2 shows an XYZ coordinate system corresponding to the XYZ coordinate system shown in Figure 1.

[0017] The photographic lens unit 110 has, as an internal mechanism, lenses 201 and 202, an aperture 203, an aperture drive unit 204, a lens drive motor 205, a lens drive member 206, a pulse plate 207, a photocoupler 208, a focus adjustment circuit 209, and a mount contact 210. Here, the lens drive member 206 includes a drive gear. The photocoupler 208 detects the rotation of the pulse plate 207, which is linked to the lens drive member 206, and transmits this information to the focus adjustment circuit 209. The focus adjustment circuit 209 drives the lens drive motor 205 based on information from the photocoupler 208 and information from the camera housing 120 (information on the amount of lens drive), moving the lens 201 to change the focus position. The mount contact 210 is the interface between the photographic lens unit 110 and the camera housing 120. Note that while Figure 2 shows two lenses 201 and 202 for simplicity, in reality, the imaging lens unit 110 contains more than two lenses.

[0018] The camera housing 120 includes, as internal components, an image sensor 211, a CPU 212, a memory unit 213, a display device 214, a display device driving circuit 215, light sources 216a and 216b, an optical splitter 217, a light-receiving lens 218, and an eye image sensor 219. Furthermore, the camera housing 120 also includes, as internal components, a speaker 220 and an eyepiece sensor 221.

[0019] The image sensor 211 is positioned on the planned imaging plane of the imaging lens unit 110. This image sensor 211 is a component that is an example of a first imaging means that captures (photographs) a subject based on user operation and acquires a subject image. The CPU 212 is the central processing unit of the microcomputer and controls the operation of the entire imaging device 100 and performs various processes. The memory unit 213 stores various information and programs that the CPU 212 executes when performing various processes. For example, the memory unit 213 stores the subject image captured (photographed) by the image sensor 211. The display device 214 displays various information on the screen (display surface) of the display device 214. For example, the display device 214 is a liquid crystal panel and displays the captured image (subject image) on the screen. The display device driving circuit 215 drives the display device 214. The user's eye (eyeball) E can see the screen of the display device 214 through the eyepiece lens 122.

[0020] Light sources 216a and 216b are light sources that have been conventionally used in single-lens reflex cameras to detect the line of sight of the eye (eyeball) E from the relationship between the reflected image (corneal reflection image) due to corneal reflection of light and the pupil. Specifically, light sources 216a and 216b are light sources for illuminating the eye (eyeball) E of the user looking through the viewfinder (eyepiece 122). For example, light sources 216a and 216b are infrared light-emitting diodes that emit infrared light that is insensitive to the user's eye (eyeball) E, and are arranged around the eyepiece 122. The optical image of the eye (eyeball) E illuminated by light sources 216a and 216b (eye optical image; optical image due to reflected light emitted from light sources 216a and 216b and reflected by the eye (eyeball) E) passes through the eyepiece 122 and is reflected by the light divider 217. The optical image of the eye is then formed by a light-receiving lens 218 onto an eye image sensor 219, which has multiple photoelectric conversion elements (e.g., CCD or CMOS) arranged in two dimensions. The light-receiving lens 218 positions the pupil of the user's eye (eyeball) E and the eye image sensor 219 in a conjugate imaging relationship. By performing gaze detection processing, the gaze of the eye (eyeball) E is detected from the position of the corneal reflection image in the optical image of the eye formed on the eye image sensor 219. For example, as information regarding the gaze, at least one piece of information is obtained, which includes information indicating the direction of the gaze and information indicating the viewpoint (the position where the gaze is directed) on the screen of the display device 214. The viewpoint can be considered as the position the user is looking at, or as the gaze position. The eye image sensor 219 is also a component that is an example of a second imaging means for capturing (shooting) an eye image including the user's eye (eyeball) E as an authentication image.

[0021] The speaker 220 is a component that serves as an example of a sound output means for playing and outputting shutter sounds and voice guidance. The eyepiece sensor 221 is a component that acquires eyepiece information of the user's eye (eyeball) E as a state of the user's eye (eyeball) E. For example, as an example of eyepiece information, the eyepiece sensor 221 measures the user's eyepiece distance, which is the distance from the eyepiece sensor 221 to the user's eye (eyeball) E, and detects whether the user is looking through the viewfinder (eyepiece lens 122).

[0022] Figure 3 shows an example of the electrical configuration of the imaging device 100 according to the first embodiment. In Figure 3, the same reference numerals are used for components that are the same as those shown in Figures 1 and 2, and their detailed descriptions are omitted.

[0023] The photographic lens unit 110 has an electrical configuration consisting of a focus adjustment circuit 209 and an aperture control circuit 306, as shown in Figure 2.

[0024] The camera housing 120 has, as an electrical configuration, a release button 121 and operating members 123-125 as shown in Figure 1. The camera housing 120 also has, as an electrical configuration, an image sensor 211, a CPU 212, a memory unit 213, a display device 214, a display device driving circuit 215, light sources 216a, 216b, an eye image sensor 219, a speaker 220, and an eyepiece sensor 221 as shown in Figure 2. Furthermore, as an electrical configuration, the camera housing 120 has, as shown in Figure 3, a gaze detection circuit 301, a photometering circuit 302, an autofocus detection circuit 303, a signal input circuit 304, and a light source driving circuit 305.

[0025] As shown in Figure 3, the CPU 212 is connected to the gaze detection circuit 301, the photometering circuit 302, the autofocus detection circuit 303, the signal input circuit 304, and the light source drive circuit 305. Furthermore, as shown in Figure 3, the CPU 212 is connected to the image sensor 211, the memory unit 213, the display device drive circuit 215, the speaker 220, the eyepiece sensor 221, and the operating members 123-125. The CPU 212 also transmits signals via the mount contact 210 to the focus adjustment circuit 209 located in the imaging lens unit 110 and the aperture control circuit 306 included in the aperture drive unit 204 in the imaging lens unit 110. The memory unit 213 attached to the CPU 212 has, for example, a function to store imaging signals from the image sensor 211 and the eye image sensor 219.

[0026] The gaze detection circuit 301 performs A / D conversion on the output of the eye image sensor 219 (eye image captured from the user's eye (eyeball) E) when an optical image of the eye is formed on the eye image sensor 219, and transmits the result to the CPU 212. The CPU 212 extracts feature points necessary for gaze detection from the eye image according to the gaze detection process, and detects the user's gaze from the position of the feature points.

[0027] The photometering circuit 302 performs predetermined processing (e.g., amplification, logarithmic compression, and A / D conversion) on the signal obtained from the image sensor 211, which also acts as a photometering sensor, for example, a luminance signal corresponding to the brightness of the field of view, and sends the result to the CPU 212 as field of view luminance information.

[0028] The autofocus detection circuit 303 performs A / D conversion on signals from multiple detection elements (multiple pixels) in the image sensor 211 used for phase-difference detection and sends them to the CPU 212. The CPU 212 calculates the distance to the subject corresponding to each focus detection point from the signals of the multiple detection elements. This is a known technique known as image-plane phase-difference autofocus.

[0029] Figure 4 shows a first embodiment and illustrates an example of the display on the screen of the display device 214. As an example, in this embodiment, each of the 180 locations on the imaging surface corresponding to the 180 locations shown on the screen of the display device 214 (viewfinder field of view) in Figure 4(a) is assumed to have a focus detection point as described by the autofocus detection circuit 303. Figure 4(a) shows the display device 214 in operation (with an image displayed), and the viewfinder field of view includes the focus detection area 401, the field of view mask 402, and 180 autofocus point indicators 410 within the focus detection area 401. Each autofocus point indicator 410 shown in Figure 4(a) is displayed superimposed on the through image (live view image) displayed on the display device 214 so as to be displayed at the position corresponding to the focus detection point on the imaging surface. In addition, of the 180 autofocus point indicators 410 shown in Figure 4(a), the autofocus point indicator 410A corresponding to the current viewpoint A (estimated position) is displayed highlighted, for example, with a frame. Furthermore, the viewpoint may not be able to be estimated with high accuracy due to factors such as individual differences in the shape of the human eye E. Specifically, as shown in Figure 4(b), there may be a discrepancy between the actual viewpoint B (410B) and the estimated viewpoint C (410C).

[0030] Now, let's return to the explanation of Figure 3. The signal input circuit 304 is connected to switches SW1 and SW2 of the release button 121. Switch SW1 is turned ON on the first stroke of the release button 121 and is used to start the shooting preparation operation of the imaging device 100 (e.g., photometry and distance measurement). Switch SW2 is turned ON on the second stroke of the release button 121 and is used to start the shooting operation. When the ON signals from switches SW1 and SW2 of the release button 121 are input to the signal input circuit 304, the signal input circuit 304 transmits the input ON signals to the CPU 212. At this point, user gaze detection may be started when switch SW1 of the release button 121 is turned ON.

[0031] The light source driving circuit 305 drives the light sources 216a and 216b.

[0032] Furthermore, when the user operates the operating members 123 to 125, the operating members 123 to 125 output an operation signal to the CPU 212 corresponding to the user's operation. The CPU 212 then performs processing (control) according to the operation signal. For example, the CPU 212 moves the selection frame of the displayed menu according to the operation signal.

[0033] Figure 5A shows an example of the functional configuration of the imaging device 100 according to the first embodiment. The imaging device 100 is an imaging device that authenticates a user. The imaging device 100 has the following functional configurations: an eye image acquisition unit 501, a registration unit 502, a registration data management unit 503, a first authentication unit 504, a second authentication unit 505, and an authentication state management unit 506. Furthermore, the imaging device 100 has the following functional configurations: an eyeball state acquisition unit 507, a first presentation information determination unit 508, a second presentation information determination unit 509, and an information presentation unit 510. For example, the CPU 212 shown in Figures 2 and 3 executes a program stored in the memory unit 213 shown in Figures 2 and 3, thereby realizing each of the functional configurations (501 to 510) shown in Figure 5.

[0034] In this embodiment, the imaging device 100 authenticates whether the user is a registered person by examining the user's eye (eyeball) E as they look through the viewfinder (eyepiece 122) as a form of user authentication. Specifically, a first authentication is performed before the user uses the imaging device 100 to photograph a subject, and a second authentication is performed when the user uses the imaging device 100 to photograph a subject (for example, at the time of shooting). In this embodiment, the imaging device 100 stores these authentication results along with the captured image.

[0035] The eye image acquisition unit 501 is an eye image acquisition means that acquires an eye image including the eye (eyeball) E of the user looking through the viewfinder (eyepiece lens 122). Specifically, the eye image acquisition unit 501 acquires an eye image (eye image signal; electrical signal of the eye image) from the eye image sensor 219 shown in Figure 3 via the gaze detection circuit 301. In this embodiment, the eye image acquisition unit 501 acquires an eye image as a registration image in the case of registration processing for a registered user, and acquires an eye image as an authentication image in the case of authentication processing for a user to be authenticated.

[0036] The registration unit 502 is a registration means that creates data to be registered in the registration data management unit 503 based on the eye image of the registered user acquired by the eye image acquisition unit 501.

[0037] The registration data management unit 503 is a registration data management means that manages authentication registration information of users authorized to use the imaging device 100. Specifically, the registration data management unit 503 stores in the memory unit 213 the eye image of the registered user, which is a registered eye image acquired in advance, a feature vector calculated from the registered user's eye image, and the position of the index corresponding to the feature vector, linked to the registered user's name and person ID. In this embodiment, the registered user enters their name at the beginning of the registration process, and the registration data management unit 503 manages the entered name (and person ID) of the registered user in a table linked to the registered data.

[0038] Figures 5B to 5D show a first embodiment and illustrate examples of various tables managed by the registration data management unit 503 shown in Figure 5A. Specifically, the registered personal information table 520 shown in Figure 5B is a table that associates the person ID with the name of the registered user. The first authentication registration data table 530 shown in Figure 5C is a table that associates the person ID with the eye image of the registered user used in the first authentication unit 504, the feature vector used in the first authentication unit 504, and the position of the index corresponding to the feature vector. The second authentication registration data table 540 shown in Figure 5D is a table that associates the person ID with the eye image of the registered user used in the second authentication unit 505, the feature vector used in the second authentication unit 505, and the position of the index corresponding to the feature vector. These various tables 520 to 540 shown in Figures 5B to 5D link information of the same registered user by person ID. Here, a neural network, for example, is used to extract feature vectors, and the extracted feature vectors are used for user authentication. Note that the method of user authentication using eye images is not limited to the neural network method described above; other known methods, such as iris recognition, may also be used.

[0039] Furthermore, Figure 5E shows the first embodiment and is a diagram illustrating an example of the authentication status table 550 managed by the authentication status management unit 506 shown in Figure 5A. This authentication status table 550 shown in Figure 5E, like the various tables 520 to 540 shown in Figures 5B to 5D, links information of the same registered user by person ID.

[0040] Now, let's return to the explanation of Figure 5A. The first authentication unit 504 is a first authentication means that authenticates the user to be authenticated based on the eye image (authentication image) of the user to be authenticated, obtained by the eye image sensor 219 and acquired by the eye image acquisition unit 501, before the subject image is captured by the image sensor 211. In this embodiment, the first authentication unit 504 performs a process to authenticate whether the user to be authenticated is a person (registered user) registered in the registration data management unit 503 when the subject image is not being captured by the image sensor 211.

[0041] The second authentication unit 505 is a second authentication means that authenticates the user to be authenticated based on the eye image (authentication image) of the user to be authenticated, obtained by the eye image sensor 219 and acquired by the eye image acquisition unit 501 when the image sensor 211 is capturing a subject image. In this embodiment, the second authentication unit 505 performs a process to authenticate whether or not the person (user to be authenticated) authenticated by the first authentication unit 504 is capturing the subject image when the image sensor 211 is capturing the subject image. The second authentication unit 505 also controls the authentication state of the second authentication to continue only while the subject image is being captured.

[0042] In this embodiment, it is desirable for the first authentication unit 504 to use an authentication setting that lowers the false acceptance rate. On the other hand, it is desirable for the second authentication unit 505 to use an authentication setting that lowers the true rejection rate. One way to achieve such authentication settings is, for example, to change the similarity threshold when the authentication method used by the first authentication unit 504 and the second authentication unit 505 is the same. In this case, specifically, a high similarity threshold is set for the first authentication unit 504, and a low similarity threshold is set for the second authentication unit 505. This makes it possible to lower the false acceptance rate in the first authentication unit 504 and lower the true rejection rate in the second authentication unit 505. From the above, when it is difficult to lower both the false acceptance rate and the true rejection rate simultaneously during shooting, this can be achieved by performing two-stage authentication.

[0043] The authentication status management unit 506 is an authentication status management means that manages the authentication status of the first authentication by the first authentication unit 504 and the authentication status of the second authentication by the second authentication unit 505. In addition, the authentication status management unit 506 also manages whether or not the system is being used by another person. For example, the authentication status management unit 506 maintains the authentication status table 550 shown in Figure 5E in the memory unit 213 and performs management.

[0044] The authentication status table 550 shown in Figure 5E will be explained below. The "First Authentication Status" in the Authentication Status Table 550 indicates whether or not the first authentication by the first authentication unit 504 is being performed, and can take one of the following values: "Not Authenticated," "Authentication Failed," or "Authentication Successful." The Person ID is the ID of the person identified by the first authentication. When the first authentication status is not authenticated, the Person ID will be a value indicating that it is empty, such as NULL. The "Second Authentication Status" in the Authentication Status Table 550 indicates whether or not the second authentication by the second authentication unit 505 is being performed, and can take one of the following values: "Not Authenticated," "Authentication Failed," or "Authentication Successful." The "Number of First Authentication Failures" in the Authentication Status Table 550 indicates the number of first authentication failures by the first authentication unit 504, and can take one of the following integer values: "Not Authenticated," "Authentication Failed," or "Authentication Successful." The "Number of Second Authentication Failures" in the Authentication Status Table 550 indicates the number of second authentication failures by the second authentication unit 505, and can take one of the following integer values: "Not Authenticated," "Authentication Failed," or "Authentication Successful." The Authentication Status Table 550 shown in Figure 5E is stored and maintained in the memory unit 213. The method for maintaining the authentication status table 550 is not limited to a table structure; for example, a key-value structure may also be used.

[0045] The eyeball state acquisition unit 507 is an eyeball state acquisition means that acquires the state of the user's eye (eyeball) E from eye images acquired by the eye image acquisition unit 501 and eye images stored in the registration data management unit 503. Here, the state of the user's eyeball refers to, for example, the user's gaze information. User gaze detection is performed, for example, by using the method described in Patent Document 2 to determine the coordinates of the corneal reflection images of light sources 216a and 216b observed on the eye image and the center of the pupil, and then determining the gaze coordinates on the user's display device 214 from these coordinates. For example, if the gaze of a user looking into the viewfinder (eyepiece 122) is acquired, the UI display can show the user how to move from the current gaze position to obtain a registration image and authentication image suitable for authentication. Furthermore, the gaze information acquired as the state of the user's eye (eyeball) E by the eyeball state acquisition unit 507 includes the direction of the user's gaze, and in this case, the eyeball state acquisition unit 507 is configured to include a gaze sensor that acquires the user's gaze information. Furthermore, the state of the user's eye (eyeball) E acquired by the eyeball state acquisition unit 507 is not limited to the user's gaze information described above. For example, the state of the user's eye (eyeball) E acquired by the eyeball state acquisition unit 507 may include the user's eyepiece information. In this case, the eyeball state acquisition unit 507 includes an eyepiece sensor 221 that acquires the user's eyepiece information. Also, for example, the state of the user's eye (eyeball) E acquired by the eyeball state acquisition unit 507 may include eye image information obtained by image processing of eye images acquired by the eye image acquisition unit 501 or eye images stored in the registration data management unit 503. In this case, the eye image information may include, for example, information such as the degree of eyelid and pupil dilation of the user, the user's eyepiece position and angle, and whether or not eyewear is being worn.

[0046] The first presentation information determination unit 508 is a first determination means that determines first presentation information, which includes guidance information for guiding at least one of the position and orientation of the user's eye E when the eye image sensor 219 captures an eye image when the first authentication unit 504 authenticates the user. In this case, for example, the first presentation information determination unit 508 may determine the first presentation information based on the current state of the user's eye (eyeball) E acquired by the eyeball state acquisition unit 507. Alternatively, for example, the first presentation information determination unit 508 may determine UI information as the first presentation information so that the eye image acquisition unit 501 can acquire an eye image suitable for first authentication by the first authentication unit 504. In this embodiment, the first presentation information determination unit 508 determines the first presentation information by selecting appropriate information from among presentation information, including multiple UI displays and multiple audio guides, stored in the memory unit 213. In this case, the first presentation information determination unit 508 may make the above selection from the presentation information stored in the memory unit 213 based on the authentication status of the first authentication managed by the authentication status management unit 506 and the eye image acquired by the eye image acquisition unit 501 during the first authentication. Furthermore, the first presentation information determination unit 508 may make the above selection from the presentation information stored in the memory unit 213 based on the state of the user's eye E acquired by the eyeball state acquisition unit 507 during the first authentication and the registered eye image registered in the registration data management unit 503.

[0047] The second presentation information determination unit 509 is a second determination means that determines second presentation information related to the authentication process by the second authentication unit 505 when the eye image sensor 219 captures an eye image when the second authentication unit 505 authenticates the user. In this case, for example, the second presentation information determination unit 509 may determine the second presentation information based on the current state of the user's eye (eyeball) E acquired by the eyeball state acquisition unit 507. Alternatively, for example, the second presentation information determination unit 509 may determine UI information as the second presentation information so that the eye image acquisition unit 501 can acquire an eye image suitable for the second authentication by the second authentication unit 505. In this embodiment, the second presentation information determination unit 509 determines the second presentation information by selecting appropriate information from among the presentation information, including multiple UI displays and multiple voice guides, stored in the memory unit 213. In this case, the second presentation information determination unit 509 may make the above selection from the presentation information stored in the memory unit 213 based on the authentication status of the second authentication managed by the authentication status management unit 506 and the eye image acquired by the eye image acquisition unit 501 during the second authentication. Furthermore, the second presentation information determination unit 509 may make the above selection from the presentation information stored in the memory unit 213 based on the state of the user's eye E acquired by the eyeball state acquisition unit 507 during the second authentication and the registered eye image registered in the registration data management unit 503.

[0048] The information presentation unit 510 is an information presentation means that presents the first presentation information determined by the first presentation information determination unit 508 to the user when the eye image sensor 219 captures an eye image when the first authentication unit 504 authenticates the user. The information presentation unit 510 is also an information presentation means that presents the second presentation information determined by the second presentation information determination unit 509 to the user when the eye image sensor 219 captures an eye image when the second authentication unit 505 authenticates the user. For example, the information presentation unit 510 that processes the presentation of the first and second presentation information is composed of at least one means from among a display means such as a display device 214 or an operating member 123 such as a touch panel, and a sound output means such as a speaker 220. The information presentation unit 510 then displays the first and second presentation information on the display means as a UI display, or outputs it as a voice guide from the sound output means.

[0049] [Registration Process] Figure 6 is a flowchart showing an example of a detailed processing procedure for the registration process in the control method of the imaging device 100 according to the first embodiment. The processing in the flowchart shown in Figure 6 is mainly executed by the CPU 212 via the registration unit 502. Furthermore, the processing in the flowchart shown in Figure 6 is assumed to be performed by the user operating the imaging device 100 at times other than when taking images. Therefore, the processing in the flowchart shown in Figure 6 is executed when the user operates the imaging device 100 and calls up this process from a menu or the like. For example, a menu screen (not shown) is displayed on the touch panel (operating member 123) of the imaging device 100, and when the user operates the operating members 123 to 125 on that menu screen and selects a menu that calls up this process, this process is executed.

[0050] First, in step S601 of Figure 6, the registration unit 502 accepts input of personal information of the person to be registered (registered user). In this embodiment, it accepts input of "name". Specifically, the registration unit 502 displays a screen for entering a name (not shown) on the touch panel (operating member 123) and accepts the name input from the user by operating the operating members 123 to 125. When the user finishes inputting, they notify that the name input is complete by pressing the completion button or the like displayed on the screen.

[0051] Next, in step S602 of Figure 6, the registration unit 502 displays instructions to the user regarding the registration method for eye images. Specifically, the registration unit 502 displays instructions on the touch panel (operating member 123) to the user to look through the viewfinder. The registration unit 502 also displays instructions to look at the indicators in the viewfinder. In addition, the registration unit 502 may display instructions to enable the capture of desirable eye images, such as not blinking and keeping the eyes wide open.

[0052] Next, the indicators 411 to 415 shown in Figure 4(c) are displayed sequentially on the display device 214 through the processing of steps S603 to S610 in Figure 6. Then, the user's eye image when viewing the displayed indicator, the feature vector obtained from that eye image, and the position of the indicator are stored in the registration data management unit 503. The process will be explained in order.

[0053] In step S603 of Figure 6, the registration unit 502 displays an indicator on the display device 214. Specifically, it displays only indicator 411 as shown in Figure 4(c), and does not display the other indicators. Alternatively, all indicators 411 to 415 as shown in Figure 4(c) may be displayed, and indicator 411 may be highlighted with a different color. Other display methods are also acceptable as long as they effectively communicate to the user that they should look at indicator 411.

[0054] Next, in step S604 of Figure 6, the eye image acquisition unit 501 acquires an eye image when the user looks through the viewfinder (eyepiece 122). Here, the detailed processing of step S604 of Figure 6 will be explained using Figure 7.

[0055] Figure 7 is a flowchart showing an example of a detailed processing procedure for acquiring eye images in step S604 of Figure 6. The processing shown in the flowchart in Figure 7 is mainly executed by the CPU 212 via the eye image acquisition unit 501.

[0056] First, in step S701 of Figure 7, the eye image acquisition unit 501 performs user gaze detection processing. This user gaze detection processing can be performed, for example, using the method described in Patent Document 2, to determine the coordinates of the corneal reflection images of the light sources 216a and 216b observed on the eye image and the center of the pupil, and then to determine the gaze coordinates on the user's display device 214 from these coordinates.

[0057] Next, in step S702 of Figure 7, the eye image acquisition unit 501 determines whether or not an eye image suitable for authentication has been acquired. Specifically, the eye image acquisition unit 501 determines whether or not an eye image suitable for authentication has been acquired based on whether or not the gaze detection process in step S701 was successful. For example, in the gaze detection process in step S701, the eye image acquisition unit 501 acquires an eye image (eye image signal; electrical signal of the eye image) from the eye image sensor 219 via the gaze detection circuit 301. Next, the eye image acquisition unit 501 determines the coordinates of the corneal reflection images of the light sources 216a and 216b and the pupil center observed on the eye image. Next, the eye image acquisition unit 501 determines the gaze coordinates on the user's display device 214 from the obtained coordinates. If the eye image acquisition unit 501 cannot detect the coordinates of the pupil center, etc., it considers the gaze detection process to have failed. Therefore, if the coordinates of the pupil center and other coordinates cannot be obtained during the process of obtaining those coordinates, the eye image acquisition unit 501 may determine that it was not possible to obtain an eye image suitable for authentication.

[0058] Next, in step S703 of Figure 7, the eye image acquisition unit 501 determines whether or not it has succeeded in acquiring an eye image suitable for authentication based on the result of the determination in step S702.

[0059] In step S703 of Figure 7, if the eye image acquisition unit 501 determines that it has successfully acquired an eye image suitable for authentication (S703 / YES), the process proceeds to step S704. When the process proceeds to step S704 in Figure 7, the eye image acquisition unit 501 crops and acquires an eye image. Specifically, first, the eye image acquisition unit 501 obtains the eye image acquired in step S702 in Figure 7. Then, using the coordinates of the pupil center image c' obtained in step S703 in Figure 7, the eye image acquisition unit 501 crops a certain size so that the pupil center image c' becomes the center of the image. Furthermore, the eye image acquisition unit 501 generates and acquires an eye image resized to the input size of the neural network used for feature vector extraction.

[0060] Next, in step S705 of Figure 7, the eye image acquisition unit 501 records, using a flag or similar, that it has successfully acquired an eye image.

[0061] Furthermore, if the eye image acquisition unit 501 determines in step S703 of Figure 7 that it has not succeeded in acquiring an eye image suitable for authentication (S703 / NO), the process proceeds to step S706. When the process proceeds to step S706 in Figure 7, the eye image acquisition unit 501 performs a process of waiting for a predetermined time, such as several hundred milliseconds. This process in step S706 is a process that expects the eye image to change and for gaze detection to be successful.

[0062] Next, in step S707 of Figure 7, the eye image acquisition unit 501 determines whether the failure to acquire an eye image suitable for authentication occurs a predetermined number of times consecutively. Here, failure refers to the case in step S703 of Figure 7 where it is determined that an eye image suitable for authentication cannot be acquired (S703 / NO).

[0063] In step S707 of Figure 7, if the eye image acquisition unit 501 determines that the number of consecutive failures to acquire an eye image suitable for authentication has not reached a predetermined number (S707 / NO), the process returns to step S701 and repeats the processing from step S701 onwards.

[0064] Furthermore, in step S707 of Figure 7, if the eye image acquisition unit 501 determines that the acquisition of an eye image suitable for authentication has failed a predetermined number of times consecutively (S707 / YES), the process proceeds to step S708. When the process proceeds to step S708 in Figure 7, the eye image acquisition unit 501 records, using a flag or similar, that it failed to acquire the eye image. At this point, the eye image acquisition unit 501 may also record the eye image acquired in step S702 in Figure 7.

[0065] When the process in step S705 in Figure 7 is completed, or when the process in step S708 in Figure 7 is completed, the process in the flowchart in Figure 7 is completed. Then, when the process in the flowchart in Figure 7 is completed, the eye image acquisition process in step S604 in Figure 6 is completed.

[0066] Now, let's return to the explanation of Figure 6. Once the process in step S604 in Figure 6 is completed, the process proceeds to step S605. When the process proceeds to step S605 in Figure 6, the eye image acquisition unit 501 determines whether or not it has successfully acquired an eye image. Specifically, the eye image acquisition unit 501 determines whether or not it has successfully acquired an eye image based on the flag recorded in step S705 or step S708 in Figure 7.

[0067] In step S605 of Figure 6, if the eye image acquisition unit 501 determines that it has successfully acquired an eye image (S605 / YES), the process proceeds to step S606. When the process proceeds to step S606 in Figure 6, the eye image acquisition unit 501 or the registration unit 502 extracts feature vectors from the eye image acquired in step S604 in Figure 6 as authentication registration information for authenticating the user. Here, a neural network, for example, is used to extract the feature vectors, and the extracted feature vectors are used for user personal authentication. Note that the method of user personal authentication using eye images is not limited to the neural network method described above, and other known methods such as iris recognition may also be used.

[0068] Next, in step S607 of Figure 6, the registration unit 502 displays on the display device 214 that it has been successful in capturing an eye image at the index currently displayed on the display device 214. For example, the registration unit 502 may display a message such as "Eye image captured successfully" on the display device 214, or it may display an icon or the like to indicate success.

[0069] Furthermore, if the eye image acquisition unit 501 determines in step S605 of Figure 6 that it has not succeeded in acquiring the eye image (failed) (S605 / NO), the process proceeds to step S608. When the process proceeds to step S608 in Figure 6, the registration unit 502 displays on the display device 214 that it has failed to capture an eye image using the index currently displayed on the display device 214. For example, the registration unit 502 may display a message such as "Failed to capture eye image" on the display device 214, or it may display an icon or the like to indicate that it has failed. When the processing in step S608 is completed, the process returns to step S604.

[0070] Furthermore, once the process in step S607 in Figure 6 is completed, the process proceeds to step S609. When the process proceeds to step S609 in Figure 6, the registration unit 502 determines whether there are any indicators that have not yet been displayed. The registration unit 502 checks whether all indicators 411 to 415 shown in Figure 4(c) are displayed and determines whether there are any indicators that have not yet been displayed.

[0071] In step S609 of Figure 6, if the registration unit 502 determines that there are indicators that have not yet been displayed (S609 / YES), the process proceeds to step S610. When the process proceeds to step S610 in Figure 6, the registration unit 502 displays the next indicator from among the indicators that have not yet been displayed on the display device 214. For example, if indicator 411 shown in Figure 4(c) was being displayed, the registration unit 502 displays indicator 412 as the next indicator on the display device 214. In this way, indicators 411 to 415 are selected and displayed on the display device 214 in numerical order. When the processing of step S610 is completed, the process returns to step S604.

[0072] Furthermore, in step S609 of Figure 6, if the registration unit 502 determines that there are no indicators that have not yet been displayed (S609 / NO), the process proceeds to step S611. When the process proceeds to step S611 in Figure 6, the registration unit 502 saves and registers the obtained information in the registered personal information table 520, the first authentication registration data table 530, and the second authentication registration data table 540 of the registration data management unit 503. Specifically, the person IDs in the three tables 520 to 540 shown in Figures 5B to 5D are IDs for establishing relationships between the tables, so the same ID value is used in the three tables 520 to 540. Specifically, the registration unit 502 adds the name of the personal information obtained in step S601 in Figure 6 to the registered personal information table 520 shown in Figure 5B. The registration unit 502 also registers the feature vector obtained in step S606 in Figure 6, and the eye image and index positions corresponding to the feature vector, separately in the first authentication registration data table 530 shown in Figure 5C and the second authentication registration data table 540 shown in Figure 5D. The method of this separate registration is as follows.

[0073] In the first authentication process shown in the first authentication registration data table 530 in Figure 5C, an indicator is displayed on the display device 214, and authentication is performed based on the eye image when the user views that indicator. Therefore, only the feature extraction vector of the indicator displayed at that time is registered in the first authentication registration data table 530 shown in Figure 5C as the feature vector for first authentication registration. In this embodiment, the first authentication registration data table 530 shown in Figure 5C contains the feature vector 1 obtained when the indicator 411 shown in Figure 4 was displayed.

[0074] Furthermore, in the second authentication process shown in the second authentication registration data table 540 in Figure 5D, the display device 214 displays the subject image captured by the image sensor 211, so the indicators are not displayed. Therefore, it is unclear where the user will focus their attention on the display device 214. Accordingly, in this embodiment, the second authentication registration data table 540 shown in Figure 5D registers all of the feature vectors 1 to 5 obtained when the indicators 411 to 415 shown in Figure 4 are displayed.

[0075] Next, in step S612 of Figure 6, the registration unit 502 displays a message on the touch panel (operating member 123) or display device 214 to inform the user that registration is complete.

[0076] When the process in step S612 in Figure 6 is completed, the process in the flowchart in Figure 6 is completed. However, in the flowchart in Figure 6, the process will fall into an infinite loop unless it is determined in step S605 that the acquisition of the eye image was successful. Therefore, it is desirable to configure the system to interrupt the process if, for example, a predetermined number of failures are observed. Furthermore, it is desirable to add processing for abnormal cases as appropriate, regardless of this configuration.

[0077] [First Authentication Process] Next, the first authentication process performed by the first authentication unit 504 will be described. In this embodiment, the first authentication is a so-called proactive authentication in which the user cooperates in acquiring eye images, and there is a high probability that high-quality eye images can be obtained at the time of registration with the user's cooperation. Therefore, by acquiring high-quality eye images during the first authentication, the accuracy of the first authentication can be improved.

[0078] Figure 8 is a flowchart showing an example of a detailed processing procedure for the first authentication process in the control method of the imaging device 100 according to the first embodiment. The processing in the flowchart shown in Figure 8 is mainly executed by the CPU 212 by the first authentication unit 504. The processing in the flowchart shown in Figure 8 is assumed to be performed by the user operating the imaging device 100 at times other than when taking images. Therefore, the processing in the flowchart shown in Figure 8 is executed when the user operates the imaging device 100 and calls up this process from a menu or the like. For example, a menu screen (not shown) is displayed on the touch panel (operating member 123) of the imaging device 100, and when the user operates the operating members 123 to 125 on that menu screen and selects a menu that calls up this process, this process is executed.

[0079] First, in step S801 of Figure 8, the first authentication unit 504 displays a predetermined screen on the touch panel operating member 123 via the first information determination unit 508 and the information display unit 510, and instructs the user on the authentication method. Specifically, the first authentication unit 504 displays instructions on the touch panel operating member 123 via the first information determination unit 508 and the information display unit 510, instructing the user to look through the viewfinder and view the indicator displayed on the display device 214. In addition, the first authentication unit 504 may also display instructions via the first information determination unit 508 and the information display unit 510 to the user to take desirable eye images, such as not blinking, keeping eyes wide open, and firmly holding the imaging device 100.

[0080] Next, in step S802 of Figure 8, the first authentication unit 504 displays an index on the display device 214 via the first presentation information determination unit 508 and the information presentation unit 510. Specifically, as shown in Figure 4(d), only the index 411 is displayed. This is because, during the registration process described above, the feature vector when viewing the index 411 is registered in the first authentication registration data table 530 of Figure 5C as the registration feature vector for the first authentication process. In this way, eye images with the same gaze are obtained during registration and authentication, so the change in appearance between the registration image and the authentication image can be reduced, and matching between the two becomes easier.

[0081] Next, in step S803 of Figure 8, the first authentication unit 504 acquires an eye image of the user when they look through the viewfinder (eyepiece 122) via the eye image acquisition unit 501. Specifically, the eye image is acquired by the process shown in Figure 7.

[0082] Next, in step S804 of Figure 8, the first authentication unit 504 determines whether or not it succeeded in acquiring the eye image in step S803. Specifically, it determines the success or failure of acquiring the eye image based on the flags recorded in steps S705 and S708 of Figure 7.

[0083] In step S804 of Figure 8, if the first authentication unit 504 determines that it did not succeed in acquiring the eye image in step S803 (S804 / NO), the process proceeds to step S805. When the process proceeds to step S805 in Figure 8, the first authentication unit 504 performs processing when eye image acquisition fails.

[0084] Figure 9 is a flowchart showing an example of a detailed processing procedure for when eye image acquisition fails in step S805 of Figure 8.

[0085] First, in step S901 of Figure 9, the first authentication unit 504 displays information on the display device 214, via the first presentation information determination unit 508 and the information presentation unit 510, indicating that the eye image capture failed during the first authentication. Figure 13 is a diagram showing an example of the display screen 1300 displayed in the display process of step S901 of Figure 9. As shown in the display screen 1300 of Figure 13, in the display process of step S901 of Figure 9, the message 1302 "Eye image capture failed" may be displayed, or an icon 1301 indicating eye image capture failure may be displayed.

[0086] Next, in step S902 of Figure 9, the eyeball state acquisition unit 507 acquires the state of the user's eye (eyeball) E from the eye image acquired in step S803 of Figure 8. Here, the user's gaze information is acquired as the state of the user's eye (eyeball) E.

[0087] Next, in step S903 of Figure 9, the information presentation unit 510 displays a UI via the first presentation information determination unit 508, based on the first authentication registration data table 530 shown in Figure 5C and the state of the user's eyes (eyeballs) E acquired in step S902. The UI display in step S903 displays first presentation information that includes guidance information to guide at least one of the position and orientation of the user's eyes (eyeballs) E. Specifically, when the user's gaze information is acquired as the state of the user's eyes (eyeballs) E, the first presentation information determination unit 508 determines a UI that guides the gaze from the current gaze position to the position of the indicator in the first authentication registration data table 530 shown in Figure 5C. The information presentation unit 510 then displays the UI determined by the first presentation information determination unit 508 across the entire screen of the touch panel (operation member 123) or display device 214. Figure 14 shows examples of display screens 1410 and 1420 displayed in the display process of step S903 of Figure 9. In step S903 of Figure 9, first, a marker 1401 is displayed at the current line of sight, as shown in the display screen 1410 in Figure 14(a). Next, in step S903 of Figure 9, the marker 1401 is moved to the position 1402 of the index in the first authentication registration data table 530 shown in Figure 14(a), as shown in the display screen 1420 in Figure 14(b). Furthermore, as shown in Figures 14(a) and 14(b), a message 1403 may be displayed as one of the guidance pieces of information to guide the user to focus on the index. By guiding the user's gaze in conjunction with the movement of the marker 1401 as shown in Figures 14(a) and 14(b), the user's gaze can be brought closer to the position of the index in the first authentication registration data table 530, thereby improving the quality of the authentication image.

[0088] If the process in step S903 in Figure 9 is completed, the process in the flowchart in Figure 9 is completed. Then, when the process in the flowchart in Figure 9 is completed, the process in step S805 in Figure 8 for when eye image acquisition fails is completed.

[0089] Now, let's return to the explanation of Figure 8. In step S804 of Figure 8, if the first authentication unit 504 determines that it successfully acquired the eye image in step S803 (S804 / YES), the process proceeds to step S806. When the process proceeds to step S806 in Figure 8, the first authentication unit 504 performs a comparison process between the registered eye image, which is the registered eye image, and the authentication image, which is the eye image acquired in step S803.

[0090] Figure 10 is a flowchart showing an example of a detailed processing procedure for the matching process in step S806 of Figure 8.

[0091] First, in step S1001 of Figure 10, the first authentication unit 504 extracts feature vectors from the eye image acquired in step S803. Here, a neural network, for example, is used to extract the feature vectors, and the extracted feature vectors are used for user personal authentication. Note that the method of user personal authentication using eye images is not limited to the neural network method described above, and other known methods such as iris recognition may also be used.

[0092] Next, in step S1002 of Figure 10, the first authentication unit 504 obtains the registered feature vectors to be used for the first authentication from the registered data management unit 503. Specifically, the first authentication unit 504 obtains all the feature vectors in the first authentication registered data table 530 shown in Figure 5C.

[0093] Next, in step S1003 of Figure 10, the first authentication unit 504 compares the feature vector of the authentication target extracted in step S1001 with the registered feature vector obtained in step S1002. Specifically, the first authentication unit 504 calculates the cosine similarity of the two feature vectors and determines whether or not the cosine similarity exceeds a predetermined threshold. Then, the first authentication unit 504 identifies the person ID of the registered feature vector whose cosine similarity exceeds the threshold.

[0094] When the process in step S1003 of Figure 10 is completed, the process in the flowchart of Figure 10 is completed. Then, when the process in the flowchart of Figure 10 is completed, the matching process in step S806 of Figure 8 is completed.

[0095] Now, let's return to the explanation of Figure 8. Next, in step S807 in Figure 8, the first authentication unit 504 determines whether the first authentication was successful. Specifically, the first authentication unit 504 determines that the first authentication was successful if there was a registered feature vector whose cosine similarity exceeded a threshold in step S806 in Figure 8 (specifically, S1003 in Figure 10).

[0096] In step S807 of Figure 8, if the first authentication unit 504 determines that the first authentication was successful (S807 / YES), the process proceeds to step S808. When the process proceeds to step S808 in Figure 8, the first authentication unit 504 performs the processing required when authentication is successful.

[0097] Figure 11 is a flowchart showing an example of a detailed processing procedure for the successful authentication process in step S808 of Figure 8.

[0098] In step S1101 of Figure 11, the first authentication unit 504 instructs the authentication status management unit 506 to update the first authentication status in the authentication status table 550 shown in Figure 5E to "authentication successful" and to update the second authentication status to "not authenticated". Furthermore, the first authentication unit 504 instructs the authentication status management unit 506 to update the number of first authentication failures in the authentication status table 550 shown in Figure 5E to 0.

[0099] Next, in step S1102 of Figure 11, the first authentication unit 504 instructs the authentication status management unit 506 to update the person ID in the authentication status table 550 shown in Figure 5E with the person ID identified in step S806 of Figure 8 (specifically, S1003 of Figure 10).

[0100] Next, in step S1103 of Figure 11, the first authentication unit 504 displays information indicating that the first authentication was successful on the touch panel (operating member 123) or display device 214 via the first information determination unit 508 and the information presentation unit 510. Figure 15 is a diagram showing an example of the display screen 1500 displayed in the display process of step S1103 of Figure 11. As shown in the display screen 1500 of Figure 15, in the display process of step S1103 of Figure 11, the message 1502 "Authentication successful" may be displayed, or an icon 1501 indicating successful authentication may be displayed.

[0101] When the process in step S1103 in Figure 11 is completed, the process in the flowchart in Figure 11 is completed. Then, when the process in the flowchart in Figure 11 is completed, the process for successful authentication in step S808 in Figure 8 is completed.

[0102] Now, let's return to the explanation of Figure 8. In step S807 of Figure 8, if the first authentication unit 504 determines that the first authentication was unsuccessful (failed) (S807 / NO), the process proceeds to step S809. When the process proceeds to step S809 in Figure 8, the first authentication unit 504 performs the processing required when authentication fails.

[0103] Figure 12 is a flowchart showing an example of a detailed processing procedure for handling authentication failure in step S809 of Figure 8.

[0104] In step S1201 of Figure 12, the first authentication unit 504 instructs the authentication status management unit 506 to update the first authentication status in the authentication status table 550 shown in Figure 5E to "authentication failed" and to update the second authentication status to "not authenticated". Furthermore, the first authentication unit 504 instructs the authentication status management unit 506 to update the number of first authentication failures in the authentication status table 550 shown in Figure 5E by incrementing it by 1.

[0105] Next, in step S1202 of Figure 12, the first authentication unit 504 instructs the authentication status management unit 506 to delete the person ID from the authentication status table 550 shown in Figure 5E. For example, a NULL value could be prepared to indicate that the field is empty, and the person ID could be overwritten with this NULL value.

[0106] Next, in step S1203 of Figure 12, the first authentication unit 504 displays information indicating that the first authentication failed on the touch panel (operating member 123) or display device 214 via the first presentation information determination unit 508 and the information presentation unit 510. Figure 16 is a diagram showing an example of the display screen 1600 displayed in the display process of step S1203 of Figure 12. As shown in the display screen 1600 of Figure 16, in the display process of step S1203 of Figure 12, the message 1602 "Authentication failed" may be displayed, or an icon 1601 indicating authentication failure may be displayed. Furthermore, as shown in the display screen 1600 of Figure 16, in the display process of step S1203 of Figure 12, an illustration 1603 showing the cos similarity and threshold obtained in step S806 of Figure 8 (specifically, S1003 of Figure 10) may be displayed. In the display screen 1600 of Figure 16, the cosine similarity and threshold are displayed directly on illustration 1603. However, any display method that is easy for the user to understand would be acceptable, such as converting them to integer values ​​by a constant multiplier before displaying them.

[0107] Next, in step S1204 of Figure 12, the first authentication unit 504, via the authentication status management unit 506, determines whether the number of first authentication failures (consecutive failures) in the authentication status table 550 shown in Figure 5E is equal to or greater than a predetermined first threshold. Here, the first threshold is assumed to be a positive integer and a fixed value, but the user may set an arbitrary threshold.

[0108] In step S1204 in Figure 12, if the first authentication unit 504 determines that the number of first authentication failures (consecutive failures) in the authentication status table 550 shown in Figure 5E is equal to or greater than the first threshold (S1204 / YES), the process proceeds to step S1205. When the process proceeds to step S1205 in Figure 12, the first authentication unit 504 first initializes by instructing the authentication status management unit 506 to update the first authentication status in the authentication status table 550 shown in Figure 5E to "unauthenticated" and updating the number of first authentication failures to "0". Next, in step S1205 in Figure 12, the first authentication unit 504 displays information indicating the termination of the first authentication on the touch panel (operation member 123) or display device 214 via the first presentation information determination unit 508 and the information presentation unit 510. Figure 24 shows an example of the display screen 2400 displayed in the display process of step S1205 in Figure 12. As shown in the display screen 2400 of Figure 24, the display process of step S1205 in Figure 12 may display a message 2402 saying "Authentication will be terminated because authentication has failed consecutively", or it may display an icon 2401 indicating the termination of authentication.

[0109] Next, in step S1206 of Figure 12, the first authentication unit 504 records the completion of the first authentication using a flag or the like.

[0110] Furthermore, in step S1204 of Figure 12, if the first authentication unit 504 determines that the number of first authentication failures (consecutive failures) in the authentication status table 550 shown in Figure 5E is less than the first threshold (S1204 / NO), the process proceeds to step S1207. When the process proceeds to step S1207 in Figure 12, the eyeball state acquisition unit 507 acquires the state of the user's eye (eyeball) E from the eye image acquired in step S803 in Figure 8. Here, the user's gaze information is acquired as the state of the user's eye (eyeball) E.

[0111] Next, in step S1208 of Figure 12, the information display unit 510 displays a UI based on the first authentication registration data table 530 shown in Figure 5C and the state of the user's eyes (eyeballs) E acquired in step S1207, via the first information display determination unit 508. Specifically, when the user's gaze information is acquired as the state of the user's eyes (eyeballs) E, the first information display determination unit 508 determines a UI that guides the gaze from the current gaze position to the position of the indicator in the first authentication registration data table 530 shown in Figure 5C. The information display unit 510 then displays the UI determined by the first information display determination unit 508 on the touch panel (operation member 123) or display device 214. For example, as shown in the display screen 1410 in Figure 14(a), a marker 1401 is first displayed at the current gaze position. Next, as shown in the display screen 1420 in Figure 14(b), the marker 1401 is moved to the position 1402 of the indicator in the first authentication registration data table 530 shown in Figure 14(a). Furthermore, as shown in Figures 14(a) and 14(b), a message 1403 may be displayed as one of the guidance pieces of information to guide the user to focus on the indicator. By guiding the user's gaze in conjunction with the movement of the marker 1401 shown in Figures 14(a) and 14(b), the user's gaze can be brought closer to the position of the indicator in the first authentication registration data table 530, thereby improving the quality of the authentication image.

[0112] Next, in step S1209 of Figure 12, the first authentication unit 504 records, using a flag or similar, that it will continue the first authentication.

[0113] When the process in step S1206 in Figure 12 is completed, or when the process in step S1209 in Figure 12 is completed, the process in the flowchart of Figure 12 is completed. Then, when the process in the flowchart of Figure 12 is completed, the authentication failure process in step S809 in Figure 8 is completed.

[0114] Now, let's return to the explanation of Figure 8. Next, in step S810 of Figure 8, the first authentication unit 504 determines whether or not to terminate the first authentication process. Specifically, the first authentication unit 504 determines whether or not to terminate the first authentication process based on the flag recorded in step S1206 or step S1209 of Figure 12.

[0115] In step S810 of Figure 8, if the first authentication unit 504 determines not to terminate (continue) the first authentication process (S810 / NO), it returns to step S803 and repeats the processes from step S803 onward.

[0116] Furthermore, in step S810 of Figure 8, if the first authentication unit 504 determines that the first authentication process is complete (S810 / YES), or if the process in step S808 of Figure 8 is completed, the process in the flowchart of Figure 8 is terminated. Note that the process in the flowchart of Figure 8 falls into an infinite loop where the process will not terminate unless it is determined in step S804 that the acquisition of the eye image was successful. Therefore, it is desirable to configure the system to interrupt the process if, for example, a predetermined number of failures are observed. Moreover, it is desirable to add processing for abnormal cases as appropriate, regardless of this configuration.

[0117] [Second Authentication Process] Next, the second authentication process performed by the second authentication unit 505 will be explained. In this embodiment, since the second authentication is performed simultaneously with the user's action of capturing an image of a subject, it is preferable to perform an authentication that does not interfere with the user's action of capturing an image of a subject as much as possible.

[0118] Figure 17 is a flowchart showing an example of a detailed processing procedure for the second authentication process in the control method of the imaging device 100 according to the first embodiment. The processing in the flowchart shown in Figure 17 is mainly executed by the CPU 212 by the second authentication unit 505. The processing in the flowchart shown in Figure 17 is assumed to be performed when the user looks through the viewfinder (eyepiece 122) when capturing an image of a subject. Therefore, the processing in the flowchart shown in Figure 17 is activated when the eyepiece sensor 221 detects that the user has brought their eye (eyeball) E close to the viewfinder (eyepiece 122). The eyepiece sensor 221 may be, for example, a sensor that detects when the skin around the user's eye (eyeball) E comes into contact with the periphery of the eyepiece 122, or a sensor that detects the eyepiece distance, which is the distance between the eyepiece 122 and the user's eye (eyeball) E. For example, if this eyepiece distance becomes less than or equal to a predetermined value, it can be determined that the user is looking through the viewfinder (eyepiece 122).

[0119] Furthermore, the activation condition for the flowchart shown in Figure 17 may be set to activate when the release button 121 is pressed down to the first stroke. Additionally, the activation condition for the flowchart shown in Figure 17 may be set to activate the gaze detection process in advance, and then activate the flowchart when the gaze detection process is successful.

[0120] First, in step S1701 of Figure 17, the second authentication unit 505 determines whether the first authentication is in progress and whether the user is continuing to take pictures of the subject (taking pictures). Here, whether the first authentication is in progress is determined by whether the first authentication status in the authentication status table 550 shown in Figure 5E is "authentication successful" via the authentication status management unit 506. Furthermore, whether the user is continuing to take pictures of the subject (taking pictures) is determined by the eyepiece sensor 221 described above, determining whether the user is keeping their eye close to the viewfinder (eyepiece lens 122). Note that whether or not the user is taking pictures of the subject may also be determined by other methods such as pressing the release button 121 or eye-tracking detection.

[0121] In step S1701 of Figure 17, if the second authentication unit 505 determines that the first authentication is in progress and the user is continuing to take a picture of the subject (taking a picture) (S1701 / YES), the process proceeds to step S1702. When the process proceeds to step S1702 in Figure 17, the eye image acquisition unit 501 acquires an eye image when the user looks through the viewfinder (eyepiece 122). The specific processing in step S1702 in Figure 17 is the same as the processing in step S604 in Figure 6 (the processing in the flowchart shown in Figure 7), so its explanation is omitted. If the gaze detection process has already been started, the gaze detection in step S701 in Figure 7 may be skipped, and the results of the already started gaze detection may be used.

[0122] Next, in step S1703 of Figure 17, the eye image acquisition unit 501 determines whether or not it has successfully acquired an eye image. Specifically, the eye image acquisition unit 501 determines whether or not it has successfully acquired an eye image based on the flag recorded in step S705 or step S708 of Figure 7.

[0123] In step S1703 of Figure 17, if the eye image acquisition unit 501 determines that it has not succeeded in acquiring the eye image (failed) (S1703 / NO), the process proceeds to step S1704. When the process proceeds to step S1704 in Figure 17, the second authentication unit 505 displays information indicating that the eye image capture failed during the second authentication on the edge of the touch panel (operation member 123) or display device 214 screen via the second presentation information determination unit 509 and the information presentation unit 510. Figure 21 is a diagram showing an example of the display screen 2100 displayed in the display processing of step S1704 in Figure 17. The display screen 2100 in Figure 21 displays the subject image being captured by the image sensor 211 based on the user's operation. Furthermore, as shown in the display screen 2100 in Figure 21, the display processing of step S1704 in Figure 17 may also display the message 2102 "Eye image capture failed" on the edge of the touch panel (operation member 123) or display device 214 screen. Alternatively, the display processing of step S1704 in Figure 17 may also display an icon 2101 indicating eye image capture failure on the edge of the touch panel (operation member 123) or display device 214 screen. Once the process in step S1704 in Figure 17 is completed, the process returns to step S1701.

[0124] Furthermore, in step S1703 of Figure 17, if the eye image acquisition unit 501 determines that it has successfully acquired an eye image (S1703 / YES), the process proceeds to step S1705. When the process proceeds to step S1705 in Figure 17, the second authentication unit 505 performs a comparison process between the registered eye image, which is the registered eye image, and the authentication image, which is the eye image acquired in step S1702.

[0125] Figure 18 is a flowchart showing an example of a detailed processing procedure for the matching process in step S1705 of Figure 17.

[0126] First, in step S1801 of Figure 18, the second authentication unit 505 extracts feature vectors from the eye image acquired in step S1702. Here, a neural network, for example, is used to extract the feature vectors, and the extracted feature vectors are used for user personal authentication. Note that the method of user personal authentication using eye images is not limited to the neural network method described above, and other known methods such as iris recognition may also be used.

[0127] Next, in step S1802 of Figure 18, the second authentication unit 505 obtains the registered feature vectors to be used for the second authentication from the registered data management unit 503. Specifically, the second authentication unit 505 obtains all the feature vectors in the second authentication registered data table 540 shown in Figure 5D.

[0128] Next, in step S1803 of Figure 18, the second authentication unit 505 compares the feature vector of the authentication target extracted in step S1801 with the registered feature vector obtained in step S1802. Specifically, the second authentication unit 505 calculates the cosine similarity of the two feature vectors and determines whether or not the cosine similarity exceeds a predetermined threshold. Then, the second authentication unit 505 identifies the person ID of the registered feature vector whose cosine similarity exceeds the threshold.

[0129] When the process in step S1803 in Figure 18 is completed, the process in the flowchart in Figure 18 is completed. Then, when the process in the flowchart in Figure 18 is completed, the matching process in step S1705 in Figure 17 is completed.

[0130] Now, let's return to the explanation of Figure 17. Next, in step S1706 in Figure 17, the second authentication unit 505 determines whether the second authentication was successful. Specifically, in step S1705 in Figure 17 (specifically, S1803 in Figure 18), the second authentication unit 505 determines that the second authentication was successful if there is a registered feature vector whose cosine similarity exceeds the threshold.

[0131] In step S1706 of Figure 17, if the second authentication unit 505 determines that the second authentication was successful (S1706 / YES), the process proceeds to step S1707. When the process proceeds to step S1707 in Figure 17, the second authentication unit 505 performs the processing for successful authentication.

[0132] Figure 19 is a flowchart showing an example of a detailed processing procedure for the successful authentication process in step S1707 of Figure 17.

[0133] In step S1901 of Figure 19, the second authentication unit 505 instructs the authentication status management unit 506 to update the second authentication status in the authentication status table 550 shown in Figure 5E to "authentication successful". Furthermore, the second authentication unit 505 instructs the authentication status management unit 506 to update the number of second authentication failures in the authentication status table 550 shown in Figure 5E to 0.

[0134] Next, in step S1902 of Figure 19, the second authentication unit 505 displays information indicating that the second authentication was successful on the touch panel (operating member 123) or the edge of the display device 214 screen via the second information determination unit 509 and the information presentation unit 510. Figure 22 is a diagram showing an example of the display screen 2200 displayed in the display process of step S1902 of Figure 19. The display screen 2200 in Figure 22 displays the subject image being captured by the image sensor 211 based on the user's operation. Furthermore, as shown in the display screen 2200 of Figure 22, the display process of step S1902 of Figure 19 may display the message 2202 "Authentication successful" or an icon 2201 indicating successful authentication.

[0135] Next, in step S1903 in Figure 19, the second authentication unit 505 determines whether the user is continuing to take pictures of the subject (taking pictures). The specific method of determination in step S1903 is the same as in step S1701 in Figure 17, so its explanation is omitted.

[0136] In step S1903 of Figure 19, if the second authentication unit 505 determines that the user is continuing to take pictures of the subject (taking pictures) (S1903 / YES), it waits in step S1903.

[0137] Furthermore, in step S1903 of Figure 19, if the second authentication unit 505 determines that the user is not continuing to take a picture of the subject (not taking a picture) (S1903 / NO), the process proceeds to step S1904. When the process proceeds to step S1904 in Figure 19, the second authentication unit 505 instructs the authentication status management unit 506 to update the second authentication status in the authentication status table 550 shown in Figure 5E to "unauthenticated".

[0138] When the process in step S1904 in Figure 19 is completed, the process in the flowchart in Figure 19 is completed. Then, when the process in the flowchart in Figure 19 is completed, the process for successful authentication in step S1707 in Figure 17 is completed.

[0139] Now, let's return to the explanation of Figure 17. When the process in step S1707 in Figure 17 is completed, the process proceeds to step S1708. Also, in step S1701 in Figure 17, if the second authentication unit 505 determines that the first authentication is not in progress or that the user is not continuing to take a subject image (not taking an image) (S1701 / NO), the process proceeds to step S1708. When the process proceeds to step S1708 in Figure 17, the second authentication unit 505 updates the display on the display device 214 via the second information display determination unit 509 and the information display unit 510. If shooting is no longer in progress, the authentication status is not displayed.

[0140] Furthermore, if the second authentication unit 505 determines in step S1706 of Figure 17 that the second authentication was unsuccessful (failed) (S1706 / NO), the process proceeds to step S1709. When the process proceeds to step S1709 in Figure 17, the second authentication unit 505 performs the processing required when authentication fails.

[0141] Figure 20 is a flowchart showing an example of a detailed processing procedure for handling authentication failure in step S1709 of Figure 17.

[0142] In step S2001 of Figure 20, the second authentication unit 505 instructs the authentication status management unit 506 to update the second authentication status in the authentication status table 550 shown in Figure 5E to "authentication failed". Furthermore, the second authentication unit 505 instructs the authentication status management unit 506 to update the number of second authentication failures in the authentication status table 550 shown in Figure 5E by incrementing it by 1.

[0143] Next, in step S2002 of Figure 20, the second authentication unit 505 displays information indicating that the second authentication failed on the touch panel (operating member 123) or the edge of the display device 214 screen via the second information determination unit 509 and the information presentation unit 510. Figure 23 is a diagram showing an example of the display screen 2300 displayed in the display process of step S2002 of Figure 20. The display screen 2300 in Figure 23 displays the subject image being captured by the image sensor 211 based on the user's operation. Furthermore, as shown in the display screen 2300 of Figure 23, the display process of step S2002 of Figure 20 may display the message 2302 "Authentication failed" or an icon 2301 indicating authentication failure. Furthermore, as shown in the display screen 2300 of Figure 23, the display process in step S2002 of Figure 20 may also display illustration 2303 showing the cosine similarity and threshold obtained in step S1705 of Figure 17 (specifically, S1803 of Figure 18). In the display screen 2300 of Figure 23, the cosine similarity and threshold are displayed directly in illustration 2303, but any display method that is easy for the user to understand is acceptable, such as converting them to integer values ​​by a constant multiplier before displaying them.

[0144] Next, in step S2003 of Figure 20, the second authentication unit 505, via the authentication state management unit 506, determines whether the number of failures (consecutive failures) of the second authentication in the authentication state table 550 shown in Figure 5E is greater than or equal to a predetermined second threshold. Here, the second threshold is assumed to be a positive integer and a fixed value, but the user may set an arbitrary threshold. The second threshold used in step S2003 of Figure 20 is set to a value smaller than the first threshold used in step S1204 of Figure 12 (in other words, the first threshold used in S1204 of Figure 12 is set to a value larger than the second threshold used in S2003 of Figure 20). By setting the second threshold in this way, if the second authentication fails consecutively when capturing a subject image, the authentication UI display can be terminated early, thereby suppressing interference with the user's subject image capture operation during the second authentication. This improves the usability when capturing subject images.

[0145] In step S2003 of Figure 20, if the second authentication unit 505 determines that the number of failures (consecutive failures) of the second authentication in the authentication status table 550 shown in Figure 5E is equal to or greater than the second threshold (S2003 / YES), the process proceeds to step S2004. When the process proceeds to step S2004 in Figure 20, the second authentication unit 505 first initializes by instructing the authentication status management unit 506 to update the second authentication status in the authentication status table 550 shown in Figure 5E to "unauthenticated" and updating the number of second authentication failures to "0". Next, in step S2004 in Figure 20, the second authentication unit 505 displays information indicating the termination of the second authentication on the touch panel (operation member 123) or the edge of the display device 214 screen via the second information display determination unit 509 and the information display unit 510. Figure 25 shows an example of the display screen 2500 displayed in the display processing of step S2004 in Figure 20. The display screen 2500 in Figure 25 displays the subject image being captured by the image sensor 211 based on the user's operation. Furthermore, as shown in the display screen 2500 of Figure 25, in the display process of step S2004 in Figure 20, either the message 2502 "Authentication will be terminated because authentication has failed consecutively" is displayed, or an icon 2501 indicating the end of authentication is displayed.

[0146] Next, in step S2005 of Figure 20, the second authentication unit 505 records the completion of the second authentication using a flag or similar method.

[0147] Furthermore, in step S2003 of Figure 20, if the second authentication unit 505 determines that the number of failures (consecutive failures) of the second authentication in the authentication status table 550 shown in Figure 5E is less than the second threshold (S2003 / NO), the process proceeds to step S2006. When the process proceeds to step S2006 in Figure 20, the second authentication unit 505 records, using a flag or similar, that the second authentication will continue.

[0148] When the process in step S2005 of Figure 20 is completed, or when the process in step S2006 of Figure 20 is completed, the process in the flowchart of Figure 20 is completed. Then, when the process in the flowchart of Figure 20 is completed, the authentication failure process in step S1709 of Figure 17 is completed.

[0149] Now, let's return to the explanation of Figure 17. Next, in step S1710 of Figure 17, the second authentication unit 505 determines whether or not to terminate the second authentication process. Specifically, the second authentication unit 505 determines whether or not to terminate the second authentication process based on the flag recorded in step S2005 or step S2006 of Figure 20.

[0150] In step S1710 of Figure 17, if the second authentication unit 505 determines not to terminate (continue) the second authentication process (S1710 / NO), it returns to step S1701 and repeats the processes from step S1701 onwards.

[0151] Furthermore, in step S1710 of Figure 17, if the second authentication unit 505 determines that the second authentication process is finished (S1710 / YES), or if the process in step S1708 of Figure 17 is completed, the process in the flowchart of Figure 17 is completed.

[0152] The imaging device 100 according to the first embodiment described above has the following configuration. Specifically, the imaging device 100 according to the first embodiment includes an image sensor 211 (first imaging means) that captures a subject image based on user operation, and an eye image sensor 219 (second imaging means) that captures an eye image including the user's eye (eyeball) E as an authentication image. The imaging device 100 according to the first embodiment also includes a first authentication unit 504 that authenticates the user based on the eye image obtained by the eye image sensor 219 before the subject image is captured by the image sensor 211. The imaging device 100 according to the first embodiment also includes a second authentication unit 505 that authenticates the user based on the eye image obtained by the eye image sensor 219 when the subject image is captured by the image sensor 211. The imaging device 100 according to the first embodiment also includes an information presentation unit 510 that presents information to the user. Specifically, when the first authentication unit 504 authenticates the user, the information presentation unit 510 presents first presentation information to the user, which includes guidance information for guiding at least one of the position and orientation of the user's eye E, when the eye image sensor 219 captures an eye image. Furthermore, when the second authentication unit 505 authenticates the user, the information presentation unit 510 presents second presentation information to the user regarding the authentication process by the second authentication unit 505, when the eye image sensor 219 captures an eye image. With this configuration, it is possible to acquire an authentication image that minimizes the change in appearance from the registered image while ensuring usability when shooting with the imaging device 100 (for example, when shooting a subject image).

[0153] In the first embodiment, the information presentation unit 510 presents guidance information to the user as first presentation information during first authentication, so as to bring the state of the user's eyes (eyeballs) E to a desirable state before authentication and in the event of authentication failure (S903 in Figure 9, S1208 in Figure 12, Figure 14). This allows for a rigorous check during first authentication, thereby improving the authentication accuracy of first authentication. Furthermore, in the first embodiment, the information presentation unit 510 does not present guidance information to the user as second presentation information during second authentication, thereby suppressing interference with the user's action of capturing subject images and further improving usability when capturing subject images. Thus, in the first embodiment, the second presentation information presented during second authentication is different from the first presentation information presented during first authentication.

[0154] According to the imaging device 100 of the first embodiment, it is possible to obtain authentication images that satisfy different criteria for first authentication and second authentication while ensuring usability when shooting with the imaging device 100 (for example, when shooting a subject image).

[0155] Furthermore, in the first embodiment, the threshold for the number of consecutive failures before the authentication process is terminated is set such that the second threshold for the second authentication is smaller than the first threshold for the first authentication. As a result, if the second authentication fails consecutively, the authentication UI display can be terminated earlier than if the first authentication fails consecutively, thereby suppressing interference with the user's subject image capture operation during the second authentication. This further improves the usability when capturing subject images.

[0156] Furthermore, in the first embodiment, the information display unit 510 displays second display information, including the authentication status and whether the authentication was successful or not, at the edge of the display means screen during the second authentication (Figures 21 to 23, 25). This allows the display means screen to display the subject image without being obstructed by the second display information, further improving the usability when capturing the subject image.

[0157] <Modification 1 of the first embodiment> If the position of the user's eyes (eyeballs) E differs between registration and authentication, the authentication rate will decrease. In this modified example 1, we describe a method that improves the quality of the authentication image and thus improves the authentication accuracy of the first authentication by guiding the user's eyes (eyeballs) E to the same position as during registration when eye image acquisition fails or authentication fails during the first authentication.

[0158] In the first embodiment described above, in the processing when eye image acquisition fails in the first authentication and in the processing when authentication fails, the eyeball state acquisition unit 507 acquired gaze information as the current state of the user's eye (eyeball) E, but it may also acquire the position of the eye (eyeball) E.

[0159] Specifically, in step S902 in Figure 9 and step S1207 in Figure 12, the eyeball state acquisition unit 507 detects the eyeball from the eye image acquired by the eye image acquisition unit 501 using a known technique, and sets the coordinates of its center as the current eyeball position coordinate C1. The eyeball state acquisition unit 507 also detects the eyeball from the eye image in the first authentication registration data table 530 in Figure 5C using the same method, and sets the coordinates of its center as the registered eyeball position coordinate C2. Here, the process of detecting the registered eyeball position coordinate C2 may be performed during the eye image registration process. The eyeball state acquisition unit 507 can then calculate the direction of the vector from the current eyeball position coordinate C1 to the registered eyeball position coordinate C2.

[0160] Figure 26 shows a modified example of the first embodiment, illustrating an example of a display screen 2600 that displays guidance information included in the first presentation information. The first authentication unit 504 can display the display screen 2600 shown in Figure 26 via the first presentation information determination unit 508 and the information presentation unit 510 in step S903 in Figure 9 and step S1208 in Figure 12. Specifically, as shown in the display screen 2600 of Figure 26, a message 2601 that guides the user's eye position in the direction of the vector calculated above, an illustration 2603 indicating the eye position, and an arrow 2602 are displayed on the touch panel (operating member 123) or display device 214. By displaying the information as shown in the display screen 2600 of Figure 26, the position of the user's eye (eyeball) E can be brought closer to the position of the eye (eyeball) E in the eye image of the first authentication registration data table 530, thereby improving the quality of the authentication image.

[0161] Figure 27 shows a modified example of the first embodiment, illustrating an example of a display screen 2700 that displays guidance information included in the first presentation information. The first authentication unit 504 can display the display screen 2700 shown in Figure 27 via the first presentation information determination unit 508 and the information presentation unit 510 in step S903 in Figure 9 and step S1208 in Figure 12. Specifically, as shown in the display screen 2700 of Figure 27, an icon 2701 indicating the current eyeball position coordinates C1 is displayed in real time on the touch panel (operation member 123) or display device 214. Also, as shown in the display screen 2700 of Figure 27, a guide 2702 indicating the registered eyeball position coordinates C2 and a message 2703 instructing the user to align the eye (eyeball) E with the guide 2702 are displayed on the touch panel (operation member 123) or display device 214. As shown in the display screen 2700 in Figure 27, the user can intuitively adjust the position of their own eye (eyeball) E to match the position of the eye (eyeball) E in the eye image of the first authentication registration data table 530, thereby improving the quality of the authentication image.

[0162] According to this modified example 1, for example, when the first authentication fails, the user's eye (eyeball) E position can be guided to be the same as at the time of registration, thereby improving the quality of the authentication image and improving the authentication accuracy of the first authentication.

[0163] <Modification 2 of the first embodiment> If the eye distance and eye angle of the user's eye (eyeball) E differ between registration and authentication, the authentication rate will decrease. In this modified example 2, we describe a method that improves the quality of the authentication image and thus improves the authentication accuracy of the first authentication by guiding the user's eye (eyeball) E to have the same eye distance and eye angle as during registration when eye image acquisition fails or authentication fails during the first authentication.

[0164] In the first embodiment described above, during the processing when eye image acquisition fails in the first authentication and during the processing when authentication fails, the eyeball state acquisition unit 507 acquired gaze information as the current state of the user's eye (eyeball) E. However, eyepiece distance and eyepiece angle may also be acquired.

[0165] Specifically, in step S902 in Figure 9 and step S1207 in Figure 12, the eyeball state acquisition unit 507 uses the output value of the eyepiece sensor 221 to calculate the distance between the imaging device 100 and the user's eye (eyeball) E as the current eyepiece distance. The eyeball state acquisition unit 507 also calculates the current eyepiece angle from the eye image acquired by the eye image acquisition unit 501 using, for example, the method described in Patent Document 2. In step S611 of the registration process in Figure 6, the eyepiece distance and eyepiece angle corresponding to the feature vector are calculated in a similar manner and registered in the first authentication registration data table 530. The eyeball state acquisition unit 507 can then calculate the direction that brings the current eyepiece closer to the latter, based on the current eyepiece distance and the eyepiece distance in the first authentication registration data table 530. The eyeball state acquisition unit 507 can also calculate the direction that brings the current eyepiece closer to the latter, based on the current eyepiece angle and the eyepiece angle in the first authentication registration data table 530.

[0166] Figure 28 shows a modified example of the first embodiment, specifically a second example of display screens 2810 and 2820 that display guidance information included in the first presented information. The first authentication unit 504 can display the display screen 2810 shown in Figure 28(a) and the display screen 2820 shown in Figure 28(b) via the first presented information determination unit 508 and the information presentation unit 510 in step S903 in Figure 9 and step S1208 in Figure 12.

[0167] Specifically, as shown in the display screen 2810 of Figure 28(a), a message 2801 guiding the user's eyepiece distance is displayed on the touch panel (operating member 123) or display device 214. Furthermore, as shown in the display screen 2810 of Figure 28(a), an arrow 2802 indicating the direction to adjust the eyepiece distance is displayed on the touch panel (operating member 123) or display device 214 between the illustration of the user's eyeball 2803 and the illustration of the imaging device 2804. By displaying the information as shown in the display screen 2810 of Figure 28(a), the user can adjust the eyepiece distance, thereby improving the quality of the authentication image.

[0168] Furthermore, as shown in the display screen 2820 of Figure 28(b), a message 2805 prompting the user to adjust the eyepiece angle is displayed on the touch panel (operating member 123) or display device 214. Also, as shown in the display screen 2820 of Figure 28(b), an illustration 2806 representing the user's current eyepiece angle, an illustration 2808 representing the target eyepiece angle, and an arrow 2807 indicating the direction to adjust the eyepiece angle are displayed on the touch panel (operating member 123) or display device 214. By displaying the information as shown in the display screen 2820 of Figure 28(b), the user can adjust the eyepiece angle, thereby improving the quality of the authentication image.

[0169] According to this modified version 2, for example, when the first authentication fails, the eyepiece distance and eyepiece angle of the user's eye (eyeball) E can be brought closer together during registration, thereby improving the quality of the authentication image and improving the authentication accuracy of the first authentication.

[0170] <Modification 3 of the first embodiment> If the user's eyewear wearing condition (such as glasses or sunglasses) differs between registration and authentication, the authentication rate will decrease. In this modified example 3, we describe a method that improves the quality of the authentication image and thus the authentication accuracy of the first authentication by guiding the user to wear the same eyewear as during registration when eye image acquisition fails or authentication fails during the first authentication.

[0171] In the first embodiment described above, during the processing when eye image acquisition fails in the first authentication and during the processing when authentication fails, the eyeball state acquisition unit 507 acquired gaze information as the current state of the user's eye (eyeball) E. However, it may also acquire the user's eyewear wearing status.

[0172] Specifically, in step S902 in Figure 9 and step S1207 in Figure 12, the eyeball state acquisition unit 507 determines the eyewear wearing state E1 from the eye image acquired by the eye image acquisition unit 501 using a known technology method. Similarly, the eyewear wearing state E2 is determined from the eye image in the first authentication registration data table 530 in Figure 5C using the same method. Here, the process of determining the wearing state E2 may also be performed during the registration process.

[0173] Figure 29 shows a third modified example of the first embodiment, illustrating an example of a display screen 2900 that displays the first information to be presented. In step S903 of Figure 9 and step S1208 of Figure 12, if the wearing state E1 and wearing state E2 are different, the first authentication unit 504 can display the display screen 2900 shown in Figure 29 via the first information determination unit 508 and the information presentation unit 510. Specifically, as shown in the display screen 2900 of Figure 29, messages 2902 and icons 2901 that guide the user to make the wearing state E1 and wearing state E2 the same are displayed on the touch panel (operating member 123) and the display device 214. By displaying the information as shown in the display screen 2900 of Figure 29, the user's wearing state of the eyewear can be made closer to the state at the time of registration, and the quality of the authentication image can be improved.

[0174] According to this modified example 3, for example, when the first authentication fails, the user can be guided to wear the eyewear in the same condition as when it was registered, thereby improving the quality of the authentication image and improving the authentication accuracy of the first authentication.

[0175] <Modification 4 of the first embodiment> This modified example 4 describes a configuration in which usability is improved by presenting first and second information to the user through sound output from a speaker 220, which is an example of a sound output means.

[0176] In the first embodiment described above, the information presentation unit 510 displayed the first and second presentation information on the touch panel (operating member 123) and the display means of the display device 214. In this modified example 4, the first and second presentation information are output as audio guidance from the speaker 220. In this case, the content of the audio guidance can be a reading of the message displayed on the display means, and if an arrow is displayed on the display means, the direction of the arrow is expressed in words and output as audio guidance.

[0177] According to this modified example 4, usability can be improved.

[0178] <Modification 5 of the First Embodiment> In this modified example 5, when the acquisition of the eye image during the first authentication fails and when authentication fails, the system guides the user to focus on an indicator, and by indicating the time the user should focus on the indicator, the effectiveness of guiding the user is enhanced.

[0179] In the first embodiment described above, in step S903 in Figure 9, which is the process when eye image acquisition for the first authentication fails, and in step S1208 in Figure 12, which is the process when authentication for the first authentication fails, message 1403 in Figure 14 was displayed to guide the user to focus on the indicator.

[0180] Figure 30 shows a modified example of the first embodiment, part 5, and is a diagram showing an example of display screens 3010 and 3020 that display guidance information included in the first presented information. In Figure 30, the same reference numerals are used for components that are the same as those shown in Figure 14, and their detailed descriptions are omitted. The first authentication unit 504 can display the display screen 3010 shown in Figure 30(a) and the display screen 3020 shown in Figure 30(b) via the first presented information determination unit 508 and the information presentation unit 510 in step S903 in Figure 9 and step S1208 in Figure 12.

[0181] Specifically, as shown in the display screen 3010 of Figure 30(a) and the display screen 3020 of Figure 30(b), a message 3001 indicating the remaining time for attention may be displayed, or a progress bar 3002 may be displayed. In this case, the attention time may be a fixed value, or the user may be able to set an arbitrary value.

[0182] According to this modified example 5, the effect of guiding the user can be enhanced, and therefore the quality of the authentication image can be improved, thereby improving the authentication accuracy of the first authentication.

[0183] <Modification 6 of the First Embodiment> This modified example 6 describes a configuration that enhances the user guidance effect by presenting the user with an eyeball state score, which shows the difference between the current state of the eye (eyeball) E and the state at the time of registration, when the eye image acquisition for the first authentication fails and when authentication fails.

[0184] In the first embodiment described above, the eyeball state acquisition unit 507 acquired the current state of the user's eye (eyeball) E during the processing when eye image acquisition fails in the first authentication and during the processing when authentication fails. However, it may also acquire the eyeball state score described above.

[0185] For example, if the eyeball state acquisition unit 507 has acquired gaze information as the current state of the user's eye (eyeball) E, it acquires the eyeball state score S using a function f such that S = f(D), where S is the eyeball state score described above. Here, the function f is a function that is high when the distance D is close to 0 and decreases in proportion to the distance D. Here, the function f may be, for example, f(D) = 1 / (D + ε). In this case, ε is a small value to prevent division by zero. For clarity, the eyeball state score S may be normalized to be expressed in the range of 0 to 100. Note that the function used to acquire the eyeball state score S is not limited to the one described above, as long as it has the same qualitative properties. Also, the method for calculating the eyeball state score S is not limited to using the distance D; it may be calculated by other methods that show the difference between the current state of the user's eye (eyeball) E and the state at the time of registration.

[0186] Figure 31 shows a modified example 6 of the first embodiment, illustrating an example of display screens 3110 and 3120 that display guidance information included in the first presented information. In Figure 31, the same reference numerals are used for components that are the same as those shown in Figure 14, and their detailed descriptions are omitted. In step S903 of Figure 9 and step S1208 of Figure 12, the first authentication unit 504 can display the display screen 3110 shown in Figure 31(a) and the display screen 3120 shown in Figure 31(b) via the first presented information determination unit 508 and the information presentation unit 510.

[0187] Specifically, as shown in the display screen 3110 in Figure 31(a) and the display screen 3120 in Figure 31(b), the eye condition score 3101, which corresponds to the eye condition score S described above, is displayed in real time on the touch panel (operating member 123) and the display device 214.

[0188] According to this modified example 6, the user can intuitively adjust the gaze of the eye (eyeball) E, thereby improving the quality of the authentication image and enhancing the authentication accuracy of the first authentication.

[0189] (Second embodiment) Next, a second embodiment will be described. In the description of the second embodiment below, matters common to the first embodiment described above will be omitted, and matters that differ from the first embodiment described above will be explained.

[0190] The schematic configuration of the imaging device according to the second embodiment is the same as the schematic configuration of the imaging device 100 according to the first embodiment shown in Figures 1 to 3 and Figure 5A.

[0191] In the second embodiment, the flowcharts shown in Figures 6 to 12 and Figures 17 to 19 are the same as in the first embodiment described above. In the second embodiment, the flowchart for handling authentication failure in the second authentication process shown in Figure 20 performs a different process than in the first embodiment.

[0192] In the first embodiment described above, when the first authentication process failed, first presentation information was presented that included guidance information for guiding at least one of the position and orientation of the user's eye (eyeball) E in order to improve the quality of the authentication image. In this respect, the second embodiment adopts a form in which second presentation information including the guidance information described above is presented not only when the first authentication process fails, but also when the second authentication process fails, in a manner that does not interfere with the user's subject image capture operation as much as possible. As a result, in the second embodiment as well, it is possible to improve the quality of the authentication image in the second authentication and enhance authentication accuracy while ensuring usability when capturing subject images.

[0193] Figure 32 shows a second embodiment and is a flowchart illustrating an example of a detailed processing procedure for handling authentication failure in step S1709 of Figure 17. In Figure 32, the same processing steps as those shown in Figure 20 are given the same step numbers, and their detailed explanations are omitted.

[0194] In step S2001 of Figure 32, similar to step S2001 of Figure 20, the second authentication unit 505 instructs the authentication status management unit 506 to update the second authentication status in the authentication status table 550 shown in Figure 5E to "authentication failed". Furthermore, the second authentication unit 505 instructs the authentication status management unit 506 to update the number of second authentication failures in the authentication status table 550 shown in Figure 5E by incrementing it by 1.

[0195] In step S2002 of Figure 32, similar to step S2002 in Figure 20, the second authentication unit 505 displays information indicating that the second authentication failed on the operating member 123 or the edge of the display device 214 screen via the second presentation information determination unit 509 and the information presentation unit 510.

[0196] Next, in step S3201 in Figure 32, the second authentication unit 505, via the authentication status management unit 506, determines whether the number of second authentication failures (consecutive failures) in the authentication status table 550 shown in Figure 5E is equal to or greater than a predetermined second threshold.

[0197] In step S3201 in Figure 32, if the second authentication unit 505 makes a positive determination (S3201 / YES), the process proceeds to step S2004. When the process proceeds to step S2004 in Figure 32, the same processing as in step S2004 and subsequent steps in Figure 20 is performed, and the processing in the flowchart of Figure 32 is completed.

[0198] Furthermore, if the second authentication unit 505 determines that the result is negative (S3201 / NO) in step S3201 of Figure 32, the process proceeds to step S3202. When the process proceeds to step S3202 in Figure 32, the eyeball state acquisition unit 507 acquires the state of the user's eye (eyeball) E from the eye image acquired in step S1702 in Figure 17. Here, the user's gaze information is acquired as the state of the user's eye (eyeball) E.

[0199] Next, in step S3203 of Figure 32, the information presentation unit 510 displays a UI via the second presentation information determination unit 509, based on the second authentication registration data table 540 shown in Figure 5D and the state of the user's eyes (eyeballs) E acquired in step S3202. Specifically, if the user's gaze information is acquired as the state of the user's eyes (eyeballs) E, the second presentation information determination unit 509 determines a UI that guides the user's gaze. The information presentation unit 510 then displays the UI determined by the second presentation information determination unit 509 on the touch panel (operation member 123) or the display device 214.

[0200] Figure 33 shows an example of the display screen 3320 that is displayed in step S3203 of Figure 32. First, in the screen 3310 shown in Figure 33(a), the user's current gaze position is set to p1=(x1,y1). Then, the second information display unit 509 selects the index position in the second authentication registration data table 540 that is the shortest distance from position p1, and sets this to position p2=(x2,y2) in the screen 3310 shown in Figure 33(a). Then, as shown in Figure 33(a), the second information display unit 509 calculates position p3=((x1+x2) / 2,(y1+y2) / 2) as the midpoint of position p1 and position p2. The information display unit 510 displays the index 3301 at the position corresponding to position p3 in Figure 33(a), as shown in the display screen 3320 of Figure 33(b). Furthermore, the information display unit 510 displays a message 3302 at the edge of the screen, as shown in the display screen 3320 of Figure 33(b), which guides the user to focus on the indicator 3301. That is, the information display unit 510 displays (presents) second information, which includes the indicator 3301 and message 3302 shown in Figure 33(b) as the guidance information described above. Here, position p3 is defined as the midpoint of positions p1 and p2, but it may also be defined as, for example, the weighted average of positions p1 and p2. By guiding the user's gaze to the position of indicator 3301 shown in Figure 33(b), the user's gaze can be brought closer to the position of the indicator in the second authentication registration data table 540, thereby improving the quality of the authentication image in the second authentication. In addition, by displaying indicator 3301 at a position close to the current gaze position (P1), the user's gaze movement during the subject image capture operation can be reduced, thereby suppressing interference with the user's subject image capture operation.

[0201] Once the process in step S3203 in Figure 32 is completed, the process proceeds to step S2006. When the process proceeds to step S2006 in Figure 32, the same process as in step S2006 in Figure 20 is performed, and the process in the flowchart of Figure 32 is completed.

[0202] In the second embodiment, the second presentation information determination unit 509 does not need to determine guidance information as the second presentation information before the user authentication process by the second authentication unit 505.

[0203] According to the second embodiment, when authentication fails in the second authentication process, a UI is displayed that guides the user's gaze while minimizing interference with the user's action of taking a subject image, thereby improving the quality of the authentication image in the second authentication. This makes it possible to improve the authentication accuracy of the second authentication while ensuring usability when taking a subject image.

[0204] <Modification 1 of the second embodiment> In this modified example 1, when the second authentication fails, a gaze indicator is displayed at the midpoint between the current focus frame and the eye-tracking position at the time of registration. This describes a configuration that improves the accuracy of the second authentication while ensuring usability when capturing subject images.

[0205] In step S3203 of Figure 32 in the second embodiment, the current line of sight position was set to p1, but the current position of the focus frame may also be set to p1.

[0206] Figure 34 shows a modified example of the second embodiment, specifically an example of the display screen 3420 displayed in step S3203 of Figure 32. Specifically, as shown in the display screen 3420 in Figure 34(b), the distance measurement point index corresponding to the current viewpoint is set as the focus frame 3403.

[0207] In the screen 3410 of Figure 34(a), the center position of the focus frame 3403 in Figure 34(b) is set to p1=(x1,y1), and the position of the index in the second authentication registration data table 540 that is closest to position p1 is selected as position p2=(x2,y2). Then, as shown in Figure 34(a), the second information display unit 509 calculates position p3=((x1+x2) / 2,(y1+y2) / 2) as the midpoint of positions p1 and p2. The information display unit 510 displays index 3401 at the position corresponding to position p3 in Figure 34(a), as shown in the display screen 3420 of Figure 34(b). Furthermore, as shown in the display screen 3420 of Figure 34(b), the information display unit 510 displays a message 3402 at the edge of the screen to guide the user to focus on index 3401. In other words, the information display unit 510 displays (presents) second information, which includes the indicator 3401 and message 3402 shown in Figure 34(b) as the guidance information described above. Here, position p3 is defined as the midpoint of positions p1 and p2, but it may also be defined as, for example, the weighted average of positions p1 and p2.

[0208] According to this modified example 1, by guiding the user's gaze to the position of indicator 3401, the gaze can be brought closer to the position at the time of registration, thereby improving the quality of the authentication image in the second authentication. This makes it possible to improve the authentication accuracy of the second authentication while ensuring usability when taking subject images. In addition, by displaying indicator 3401 in a position close to the object being photographed, the user's gaze movement during the subject image shooting operation can be reduced, further improving usability when taking subject images.

[0209] <Modification 2 of the second embodiment> In this modified example 2, when the second authentication fails, the user is guided to focus on the UI element for taking photos that is closest to their eye position at the time of registration. This describes a configuration that improves the accuracy of the second authentication while ensuring usability when taking photos of the subject.

[0210] In step S3203 of Figure 32 in the second embodiment, an indicator was displayed and the user was guided to focus on the indicator. If the display device 214 is displaying the distance measurement point indicator and the focus frame, the user may be guided to focus on the distance measurement point indicator and the focus frame.

[0211] Figure 35 shows a modified example of the second embodiment, specifically an example of the display screen 3520 displayed in step S3203 of Figure 32. Specifically, the second information display unit 509 determines positions p1 and p2 shown in screen 3510 of Figure 35(a) in the same manner as the second embodiment described above. The second information display unit 509 then selects the distance-measuring point indicator and focus frame that have the smallest distance between position p2 and the center position of each respective distance-measuring point indicator and focus frame. As shown in the display screen 3520 of Figure 35(b), the frame line 3501 is changed from a solid line to a dotted line to highlight the selected distance-measuring point indicator and focus frame. Here, the method of highlighting could also be changing the color of the frame line 3501 or changing the thickness of the frame line 3501; any method that allows the user to distinguish between the selected distance-measuring point indicator and focus frame and the unselected distance-measuring point indicator and focus frame is acceptable. Furthermore, as shown in the display screen 3520 of Figure 35(b), a message 3502 may be displayed at the edge of the screen to guide the user to focus on the selected autofocus point indicator and focus frame. By guiding the user's gaze to the selected autofocus point indicator and focus frame in this way, the user's gaze can be brought closer to the position of the indicator in the second authentication registration data table 540, thereby improving the quality of the authentication image in the second authentication. In addition, highlighting the UI parts for shooting can reduce screen changes during the subject image shooting operation, thereby suppressing interference with the user's subject image shooting operation.

[0212] Additionally, if the display device 214 is displaying a grid, the user may be instructed to focus on the grid.

[0213] Figure 36 shows a modified example of the second embodiment, specifically an example of the display screen 3620 displayed in step S3203 of Figure 32. Specifically, the second information display unit 509 determines positions p1 and p2 shown in the screen 3610 of Figure 36(a) in the same manner as the second embodiment described above. The second information display unit 509 then selects the grid cell with the smallest distance between position p2 and the center position of that cell. As shown in the display screen 3620 of Figure 36(b), the border line 3601 is changed from a solid line to a dotted line to highlight the selected grid cell. The method of highlighting could also be changing the color of the border line 3601 or changing the thickness of the border line 3601; any method that allows the user to distinguish between the selected grid cell and the unselected grid cell is acceptable. Furthermore, as shown in the display screen 3620 of Figure 36(b), a message 3602 may be displayed at the edge of the screen to guide the user to focus on the selected grid cell. In this way, by guiding the user's gaze to the selected grid cell, the user's gaze can be brought closer to the position of the indicator in the second authentication registration data table 540, thereby improving the quality of the authentication image in the second authentication. Furthermore, by highlighting the UI part for shooting without adding new UI parts, screen changes during the subject image shooting operation can be reduced, thereby suppressing interference with the user's subject image shooting operation.

[0214] According to this modified version 2, by guiding the user's gaze to the UI element for shooting that is closest to the user's gaze position at the time of registration, the user's gaze can be brought closer to the position at the time of registration, thereby improving the quality of the authentication image in the second authentication and increasing the accuracy of the second authentication. In addition, by guiding the user to focus on the UI element for shooting without adding any new UI elements, screen changes during the act of taking a subject image can be reduced, thereby improving the usability when taking a subject image.

[0215] <Modified example 3 of the second embodiment> If the position of the user's eyes (eyeballs) E differs between registration and authentication, the authentication rate will decrease. In this modified example 3, when the second authentication fails, the system guides the user so that the position of their eyes (eyeballs) E is the same as during registration, so as not to interfere with the user's action of taking a subject image. Thus, this modified example 3 describes a configuration that improves the quality of the authentication image in the second authentication and enhances the authentication accuracy of the second authentication while ensuring usability when taking subject images.

[0216] In the second embodiment described above, during the processing of authentication failure in the second authentication, the eyeball state acquisition unit 507 acquired gaze information as the current state of the user's eye (eyeball) E, but it may also acquire the position of the eye (eyeball) E.

[0217] Specifically, in step S3202 of Figure 32, the eyeball state acquisition unit 507 detects the eyeball from the eye image acquired by the eye image acquisition unit 501 using a known technique, and sets the coordinates of its center as the current eyeball position coordinate C3. The eyeball state acquisition unit 507 also detects the eyeball from each eye image in the second authentication registration data table 540 of Figure 5D using the same method, and sets the registered eyeball position coordinate C4 to be the one with the closest distance from the current eyeball position coordinate C3 among the center coordinates. The eyeball state acquisition unit 507 can then calculate the direction of the vector from the current eyeball position coordinate C3 to the registered eyeball position coordinate C4.

[0218] Figure 37 shows a third modified example of the second embodiment, illustrating an example of a display screen 3700 that displays guidance information included in the second presentation information. In step S3203 of Figure 32, the second authentication unit 505 can display the display screen 3700 shown in Figure 37 via the second presentation information determination unit 509 and the information presentation unit 510. Specifically, as shown in the display screen 3700 of Figure 37, a message 3701 and an arrow 3702 that guide the user's eye position in the direction of the vector calculated above are displayed on the edge of the touch panel (operation member 123) or the display device 214 screen. Furthermore, as shown in the display screen 3700 of Figure 37, an indicator 3703 indicating the position of the user's eye (eyeball) E is displayed on the edge of the touch panel (operation member 123) or the display device 214 screen.

[0219] According to this modified version 3, the position of the user's eye (eyeball) E can be brought closer to the eyeball position at the time of registration, thereby improving the quality of the authentication image in the second authentication and increasing the accuracy of the second authentication. In addition, by displaying second presentation information, which includes guidance information to guide the user's gaze, at the edge of the screen, it is possible to suppress interference with the user's action of taking a subject image, thereby improving the usability when taking a subject image.

[0220] <Modification 4 of the second embodiment> If the eyepiece distance and eyepiece angle of the user's eye (eyeball) E differ between registration and authentication, the authentication rate will decrease. In this modified example 4, a configuration is described in which, in the event of a failure in the second authentication, the user is guided to have the same eyepiece distance and eyepiece angle as during registration, so as not to interfere with the user's action of taking a subject image. As a result, in this modified example 4, usability during subject image capture is ensured while improving the quality of the authentication image used in the second authentication, thereby improving the authentication accuracy of the second authentication.

[0221] In the second embodiment described above, during the processing of authentication failure in the second authentication, the eyeball state acquisition unit 507 acquired gaze information as the current state of the user's eye (eyeball) E, but it may also acquire eyepiece distance or eyepiece angle.

[0222] Specifically, in step S3202 of Figure 32, the eyeball state acquisition unit 507 uses the output value of the eyepiece sensor 221 to calculate the distance between the imaging device 100 and the user's eye (eyeball) E as the current eyepiece distance. The eyeball state acquisition unit 507 also calculates the current eyepiece angle from the eye image acquired by the eye image acquisition unit 501, for example, using the method described in Patent Document 2. In step S611 of the registration process in Figure 6, the eyepiece distance and eyepiece angle corresponding to the feature vector are calculated in a similar manner and registered in the second authentication registration data table 540. The eyeball state acquisition unit 507 can then calculate the direction that brings the eye closer to the user based on the current eyepiece distance and the eyepiece distance in the second authentication registration data table 540. The eyeball state acquisition unit 507 can also calculate the direction that brings the eye closer to the user based on the current eyepiece angle and the eyepiece angle in the second authentication registration data table 540.

[0223] Figure 38 shows a modified example 4 of the second embodiment, illustrating an example of display screens 3810 and 3820 that display guidance information included in the second presented information. In step S3203 of Figure 32, the second authentication unit 505 can display the display screen 3810 shown in Figure 38(a) and the display screen 3820 shown in Figure 38(b) via the second presented information determination unit 509 and the information presentation unit 510.

[0224] Specifically, as shown in the display screen 3810 of Figure 38(a), a message 3801 guiding the user to adjust their eyepiece distance is displayed on the touch panel (operating member 123) or at the edge of the display device 214 screen. By displaying this message as shown in the display screen 3810 of Figure 38(a), the user can adjust their eyepiece distance.

[0225] Furthermore, as shown in the display screen 3820 of Figure 38(b), a message 3802 prompting the user to adjust the eyepiece angle is displayed on the touch panel (operating member 123) or at the edge of the display device 214 screen. By displaying this message as shown in the display screen 3820 of Figure 38(b), the user can adjust the eyepiece angle.

[0226] According to this modified version 4, when the second authentication fails, the eyepiece distance and eyepiece angle of the user's eye (eyeball) E can be brought closer to the time of registration, improving the quality of the authentication image and thus improving the accuracy of the second authentication. In addition, by displaying second presentation information, which includes guidance information to guide the user's gaze, at the edge of the screen, interference with the user's action of capturing subject images can be suppressed, thereby improving the usability when capturing subject images.

[0227] <Modification 5 of the second embodiment> If the user's eyewear, such as glasses or sunglasses, is worn differently during registration and authentication, the authentication rate will decrease. In this modified example 5, a mechanism is described in which, in the event of a failure in the second authentication, the user is guided to wear the same eyewear as during registration, so as not to interfere with the user's action of taking a subject image. As a result, in this modified example 5, usability during subject image capture is ensured while improving the quality of the authentication image used in the second authentication, thereby improving the authentication accuracy of the second authentication.

[0228] In the second embodiment described above, during the processing of authentication failure in the second authentication, the eyeball state acquisition unit 507 acquired gaze information as the current state of the user's eye (eyeball) E. However, it may also acquire the user's eyewear wearing status.

[0229] Specifically, in step S3202 of Figure 32, the eyeball state acquisition unit 507 determines the eyewear wearing state E3 from the eye image acquired by the eye image acquisition unit 501 using a known technology method. Similarly, the eyewear wearing state E4 is determined from the eye image in the second authentication registration data table 540 of Figure 5D using the same method. Here, the process of determining the wearing state E4 may also be performed during the registration process.

[0230] Figure 39 shows a modified example 5 of the second embodiment, and is a diagram showing an example of a display screen 3900 that displays second presentation information. In step S3203 of Figure 32, if wearing state E3 and wearing state E4 are different, the second authentication unit 505 can display the display screen 3900 shown in Figure 39 via the second presentation information determination unit 509 and the information presentation unit 510. Specifically, as shown in the display screen 3900 of Figure 39, a message 3901 that guides the user to make wearing state E3 and wearing state E4 the same is displayed on the touch panel (operating member 123) or the edge of the display device 214 screen. By displaying the information as shown in the display screen 3900 of Figure 39, the user's wearing state of the eyewear can be made closer to that at the time of registration, and the quality of the authentication image can be improved.

[0231] According to Modification 5, for example, when the second authentication fails, the user can be guided to wear the eyewear in the same condition as when it was registered, thereby improving the quality of the authentication image and enhancing the authentication accuracy of the second authentication. In addition, by displaying the second presentation information at the edge of the screen, interference with the user's action of taking a subject image can be suppressed, thereby improving the usability when taking a subject image.

[0232] <Modification 6 of the second embodiment> This modified example 6 describes a configuration in which, during the second authentication process, the color of the UI elements for shooting is changed according to the second authentication status, thereby indicating the second authentication status to the user and improving usability when taking images of the subject.

[0233] In the second embodiment described above, the second authentication unit 505 displayed the authentication status as an icon via the second presentation information determination unit 509 and the information presentation unit 510 during the second authentication process. In this modified example 6, the color of at least one UI part for shooting, selected from the focus frame, distance measurement point indicator, grid, characters indicating the shooting mode, and icons indicating the shooting mode, may be changed. In this case, the information presentation unit 510 may display a UI with the color of the aforementioned UI part for shooting changed as second presentation information, according to at least one piece of information from the first authentication status and the second authentication status in the authentication status table 550 managed by the authentication status management unit 506. Alternatively, the information presentation unit 510 may display a UI that includes at least one piece of information from the first authentication status and the second authentication status in the authentication status table 550 managed by the authentication status management unit 506 as first presentation information.

[0234] For example, in this modified example 6, in step S1708 in Figure 17, step S1902 in Figure 19, and step S2002 in Figure 20, the second authentication unit 505 instructs the authentication status management unit 506 to update the second authentication status in the authentication status table 550 shown in Figure 5E. At the same time, the second authentication unit 505 changes the color of the UI parts to the color corresponding to the updated second authentication status via the second presentation information determination unit 509 and the information presentation unit 510 and displays them. Here, for example, the colors corresponding to each second authentication status are fixed, such as "Unauthenticated: Black," "Authentication Failed: Red," and "Authentication Successful: Green." The colors described here are just examples, and any colors that allow the user to intuitively recognize the difference in the second authentication status are acceptable. Alternatively, the user may be allowed to set the colors corresponding to each second authentication status.

[0235] According to Modification 6, by changing the color of the UI elements used for shooting, without adding any new UI elements, screen changes during the act of taking a subject image can be reduced. This reduces the disruption to the user's act of taking a subject image and improves the usability of taking subject images.

[0236] <Modification 7 of the second embodiment> In this modified version 7, when the second authentication fails, the user is presented with an eyeball state score, which shows the difference between the current state of the eye (eyeball) E and the state at the time of registration, in a manner that does not interfere with the user's action of taking a subject image. As a result, in this modified version 7, the usability during subject image capture is ensured while enhancing the effect of guiding the user and improving the authentication accuracy of the second authentication.

[0237] In the second embodiment described above, in step S3202 of Figure 32 in the second authentication process, the eyeball state acquisition unit 507 acquired the current state of the user's eye (eyeball) E, but it may also acquire the eyeball state score described above.

[0238] For example, if the eyeball state acquisition unit 507 has acquired gaze information as the current state of the user's eye (eyeball) E, it acquires the eyeball state score S2 using a function f2 such that S2 = f2(D), where S2 is the eyeball state score described above. Here, the function f2 is a function that is high when the distance D is close to 0 and decreases in proportion to the distance D. Here, the function f2 may be, for example, f2(D) = 1 / (D + ε). In this case, ε is a small value to prevent division by zero. For clarity, the eyeball state score S2 may be normalized to be expressed in the range of 0 to 100. Note that the function used to acquire the eyeball state score S2 is not limited to the one described above, as long as it has the same qualitative properties. Also, the method for calculating the eyeball state score S2 is not limited to using the distance D; it may be calculated by other methods that show the difference between the current state of the user's eye (eyeball) E and the state at the time of registration.

[0239] FIG. 40 shows a modification example 7 of the second embodiment, and is a diagram showing an example of a display screen 4020 that displays guidance information included in the second presentation information. In this FIG. 40, the same components as those shown in FIG. 33 are denoted by the same reference numerals, and detailed descriptions thereof are omitted. The screen 4010 shown in FIG. 40(a) is the same as the screen 3310 shown in FIG. 33(a). In step S3203 of FIG. 32, the second authentication unit 505 can display the display screen 4020 shown in FIG. 40(b) via the second presentation information determination unit 509 and the information presentation unit 510.

[0240] Specifically, as shown in the display screen 4020 of FIG. 40(b), an eye state score 4001 corresponding to the above-described eye state score S2 is displayed in real time at the screen edge of the touch panel (operation member 123) or the display device 214.

[0241] According to this modification example 7, since the user can intuitively adjust the line of sight of the eye (eyeball) E, the quality of the authentication image can be improved and the authentication accuracy of the second authentication can be improved. In addition, by displaying the second presentation information including the guidance information at the screen edge, it is possible to suppress an inhibition in the photographing operation of the subject image by the user, and to enhance the usability of the user when photographing the subject image.

[0242] <Modification Example 8 of the Second Embodiment> In this modification example 8, a form will be described in which, at the time of the second authentication, information corresponding to the first authentication state is presented to the user, so that the user recognizes that the photographer has not been identified, and the usability is enhanced.

[0243] In step S1708 of FIG. 17 in the second authentication process, the second authentication unit 505 may update the display according to the first authentication state managed by the authentication state management unit 506 via the second presentation information determination unit 509 and the information presentation unit 510.

[0244] Figure 41 shows a modified example 8 of the second embodiment, illustrating an example of a display screen 4100 that displays the second presentation information. The second authentication unit 505 displays the message 4101 shown on the display screen 4100 in Figure 41 via the second presentation information determination unit 509 and the information presentation unit 510 when the first authentication status managed by the authentication status management unit 506 is "unauthenticated" or "authentication failed". Specifically, the information presentation unit 510 displays a message as message 4101 indicating the first information that the first authentication status is "unauthenticated" (or "authentication failed"), the second information that the photographer has not been identified, and the third information that the first authentication is required to identify the photographer. In this modified example 8, it is also possible to display at least one of the first, second, and third pieces of information described here. Furthermore, the information presentation unit 510 displays an icon 4102 on the display screen 4100 in Figure 41 to indicate that the first authentication status is "unauthenticated" or "authentication failed". In this case, the information display unit 510 displays the message 4101 and the icon 4102 on the edge of the touch panel (operating member 123) or the display device 214 screen, as shown in the display screen 4100 of Figure 41.

[0245] In other words, in this modified example 8, as shown in Figure 41, the second information presentation determination unit 509 determines the second information presentation based on the authentication status of the first authentication unit 504, and the information presentation unit 510 presents the second information presentation based on the authentication status of the first authentication unit 504.

[0246] According to this modified example 8, the user can be made aware that the photographer has not been identified. Furthermore, by displaying message 4101 and icon 4102 on the edge of the touch panel (operating member 123) or display device 214 screen, interference with the user's action of capturing an image of a subject can be suppressed. This improves the usability when capturing an image of a subject.

[0247] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. This program and a computer-readable storage medium storing said program are included in the present invention. Furthermore, the present invention can take the form of, for example, the systems, devices, methods, programs, or storage media described above. For example, it may be applied to a system consisting of multiple devices, or to a device consisting of a single device.

[0248] The embodiments of the present invention described above are merely examples of how the invention can be implemented, and the technical scope of the invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0249] This embodiment includes the following configurations, methods, and programs. [Configuration 1] A first imaging means that captures a subject image based on user operation, A second imaging means for capturing an eye image, including the user's eye, as an authentication image, Prior to capturing the subject image by the first imaging means, a first authentication means performs user authentication based on the eye image obtained by the second imaging means, When capturing the subject image with the first imaging means, a second authentication means performs user authentication based on the eye image obtained by the second imaging means, When the first authentication means authenticates the user and the second imaging means captures the eye image, it presents the user with first presentation information that includes guidance information for guiding at least one of the user's eye position and orientation. When the second imaging means captures the eye image while the second authentication means is authenticating the user, an information presentation means presents to the user second presentation information relating to the authentication process by the second authentication means, An imaging device characterized by having the following features. [Configuration 2] The second piece of information presented is different from the first piece of information presented. The imaging apparatus according to configuration 1, characterized in that it is a device. [Configuration 3] The second presentation information is presentation information that includes the guidance information. The imaging apparatus according to configuration 1 or 2, characterized by the above. [Structure 4] A first determination means for determining the first presented information, A second determination means for determining the second presented information, It further possesses, The aforementioned information presentation means is When the first authentication means authenticates the user, the second imaging means captures the eye image and presents the first presentation information determined by the first determination means to the user. When the second authentication means authenticates the user, the second imaging means captures the eye image and presents the second presentation information determined by the second determination means to the user. An imaging apparatus according to any one of configurations 1 to 3, characterized by the above. [Composition 5] The first authentication means terminates the user authentication process if the number of consecutive user authentication failures is equal to or greater than a first threshold. The first determination means determines the first presentation information which includes information indicating that the user authentication process will be terminated if the number of consecutive failures by the first authentication means is equal to or greater than the first threshold, The second authentication means terminates the user authentication process if the number of consecutive user authentication failures is equal to or greater than the second threshold. The second determination means determines the second presentation information including information indicating that the authentication process of the user is to be terminated when the number of consecutive failures by the second authentication means is equal to or greater than the second threshold value. The imaging device according to Configuration 4, characterized in that. [Configuration 6] The second threshold value is a value smaller than the first threshold value. The imaging device according to Configuration 5, characterized in that. [Configuration 7] The imaging device further includes acquisition means for acquiring the state of the user's eyes, The first determination means determines the first presentation information based on the current state of the user's eyes acquired by the acquisition means, The second determination means determines the second presentation information based on the current state of the user's eyes acquired by the acquisition means. The imaging device according to any one of Configurations 4 to 6, characterized in that. [Configuration 8] The acquisition means includes a gaze sensor that acquires the user's gaze information as the state of the user's eyes, The first determination means determines the first presentation information based on the gaze information acquired by the gaze sensor, The second determination means determines the second presentation information based on the gaze information acquired by the gaze sensor. The imaging device according to Configuration 7, characterized in that. [Configuration 9] The acquisition means includes an eyepiece sensor that acquires the user's eyepiece information as the state of the user's eyes, The first determination means determines the first presentation information based on the eyepiece information acquired by the eyepiece sensor, The second determination means determines the second presentation information based on the eyepiece information acquired by the eyepiece sensor. The imaging device according to Configuration 7, characterized in that. [Configuration 10] The acquisition means acquires eye image information obtained by image processing of the user's eye image as the state of the user's eyes. The first determination means determines the first presentation information based on the eye image information acquired by the acquisition means, The second determination means determines the second presentation information based on the eye image information acquired by the acquisition means. The imaging device according to configuration 7, characterized by the features described above. [Composition 11] The first determination means determines the guidance information included in the first presentation information based on the current eye condition of the user obtained by the acquisition means, before the user authentication process by the first authentication means. The second determination means does not determine the guidance information as the second presentation information before the user authentication process by the second authentication means. The imaging device according to configuration 7, characterized by the features described above. [Composition 12] The system further includes a management means for managing the registered eye image of the user acquired in advance and the user's gaze information at the time the registered eye image was acquired. The first determination means determines the guidance information included in the first presentation information based on the gaze information, The second determination means determines the guidance information to be included in the second presentation information based on the line of sight information and the current position of the focus frame. An imaging apparatus according to any one of configurations 4 to 11, characterized by the features described herein. [Composition 13] The second determination means determines the second presentation information based on the authentication status of the first authentication means. An imaging apparatus according to any one of configurations 4 to 12, characterized by the features described herein. [Composition 14] The second decision-making means is, If the authentication status of the first authentication means is unauthenticated or authentication failed, The second presentation information is determined to include at least one of the following pieces of information: the authentication status of the first authentication means, that the user has not been identified, and that authentication by the first authentication means is necessary to identify the user. The imaging apparatus according to configuration 13, characterized by the features described above. [Composition 15] The information presentation means is a display means for displaying the first presentation information and the second presentation information. An imaging apparatus according to any one of configurations 1 to 14, characterized by the features described herein. [Composition 16] The display means is at least one of the display device in the viewfinder of the imaging device and the rear display of the imaging device. The information presentation means displays the first presentation information as a UI at any position on the screen of the display means, and displays the second presentation information as a UI at the edge of the screen of the display means. The imaging apparatus according to configuration 15, characterized by the features described above. [Composition 17] The display means displays the subject image on the screen when it is displaying the second presentation information as the UI at the edge of the screen. The imaging apparatus according to configuration 16, characterized in that... [Composition 18] The aforementioned information presentation means is The UI is displayed as the first presented information, which includes at least one of the authentication status of the first authentication means and the authentication status of the second authentication means. As information presented in XL2, Depending on at least one piece of information from the authentication status of the first authentication means and the authentication status of the second authentication means, The UI displays a color change for at least one of the following: the focus frame, the autofocus point indicator, the grid, the text indicating the shooting mode, and the icon indicating the shooting mode. The imaging apparatus according to configuration 16 or 17, characterized by the above. [Composition 19] The system further includes means for acquiring the user's eye condition, The aforementioned information presentation means is As the first presentation information, if the authentication status of the first authentication means is an authentication failure, the UI including the guidance information is displayed based on the current eye state of the user obtained by the acquisition means. As information presented in XL2, If the authentication status of the second authentication means is an authentication failure, based on the current eye condition of the user obtained by the acquisition means, Display the UI which includes guidance information that guides the user to focus on at least one of the focus frame, distance point indicator, and grid. The imaging apparatus according to configuration 16 or 17, characterized by the above. [Configuration 20] The information presentation means is a sound output means that outputs the first presentation information and the second presentation information as sound. An imaging apparatus according to any one of configurations 1 to 14, characterized by the features described herein. [Method 1] A control method for an imaging device comprising: a first imaging means for capturing a subject image based on user operation; and a second imaging means for capturing an eye image including the user's eye as an authentication image, A first authentication step is performed to authenticate the user based on the eye image obtained by the second imaging means before capturing the subject image by the first imaging means, A second authentication step is performed when capturing the subject image with the first imaging means, based on the eye image obtained by the second imaging means, When the second imaging means captures the eye image in the first authentication step, it presents to the user first presentation information which includes guidance information for guiding at least one of the position and orientation of the user's eyes. When the second imaging means captures the eye image in the second authentication step, an information presentation step is performed to present the user with second presentation information relating to the authentication process in the second authentication step. A control method for an imaging device, characterized by having the following features. [Program 1] A program for causing a computer to perform each step in the method for controlling an imaging device described in Method 1. [Explanation of Symbols]

[0250] 100: Imaging device, 501: Eye image acquisition unit, 502: Registration unit, 503: Registration data management unit, 504: First authentication unit, 505: Second authentication unit, 506: Authentication status management unit, 507: Eyeball status acquisition unit, 508: First presentation information determination unit, 509: Second presentation information determination unit, 510: Information presentation unit

Claims

1. A first imaging means that captures a subject image based on user operation, A second imaging means for capturing an eye image, including the user's eye, as an authentication image, Prior to capturing the subject image by the first imaging means, a first authentication means performs user authentication based on the eye image obtained by the second imaging means, When capturing the subject image with the first imaging means, a second authentication means performs user authentication based on the eye image obtained by the second imaging means, When the second imaging means captures the eye image while the first authentication means is authenticating the user, it presents to the user first presentation information including guidance information for guiding at least one of the position and orientation of the user's eyes, and when the second imaging means captures the eye image while the second authentication means is authenticating the user, it presents to the user second presentation information relating to the authentication process by the second authentication means, An imaging device characterized by having the following features.

2. The second presented information is different from the first presented information. The imaging apparatus according to feature 1.

3. The second presentation information is presentation information that includes the guidance information. The imaging apparatus according to feature 1.

4. A first determination means for determining the first presented information, A second determination means for determining the second presented information, It further possesses, The information presentation means presents the first presentation information determined by the first determination means to the user when the second imaging means captures the eye image when the first authentication means authenticates the user, and presents the second presentation information determined by the second determination means to the user when the second imaging means captures the eye image when the second authentication means authenticates the user. The imaging apparatus according to feature 1.

5. The first authentication means terminates the user authentication process if the number of consecutive failures in user authentication is equal to or greater than a first threshold. The first determination means determines the first presentation information which includes information indicating that the user authentication process will be terminated if the number of consecutive failures by the first authentication means is equal to or greater than the first threshold, The second authentication means terminates the user authentication process if the number of consecutive user authentication failures is equal to or greater than the second threshold. The second determination means determines the second presentation information, which includes information indicating that the user authentication process will be terminated if the number of consecutive failures by the second authentication means is equal to or greater than the second threshold. The imaging apparatus according to feature 4.

6. The second threshold is a smaller value than the first threshold. The imaging apparatus according to feature 5.

7. The system further includes means for acquiring the user's eye condition, The first determination means determines the first presentation information based on the user's current eye condition acquired by the acquisition means. The second determination means determines the second presentation information based on the user's current eye condition acquired by the acquisition means. The imaging apparatus according to feature 4.

8. The acquisition means includes a gaze sensor that acquires the user's gaze information as the state of the user's eyes, The first determination means determines the first presentation information based on the gaze information acquired by the gaze sensor, The second determination means determines the second presentation information based on the gaze information acquired by the gaze sensor. The imaging apparatus according to feature 7.

9. The acquisition means includes an eyepiece sensor that acquires the user's eyepiece information as the state of the user's eyes, The first determination means determines the first presentation information based on the eyepiece information acquired by the eyepiece sensor, The second determination means determines the second presentation information based on the eyepiece information acquired by the eyepiece sensor. The imaging apparatus according to feature 7.

10. The acquisition means acquires eye image information obtained by image processing the user's eye image as the state of the user's eyes, The first determination means determines the first presentation information based on the eye image information acquired by the acquisition means, The second determination means determines the second presentation information based on the eye image information acquired by the acquisition means. The imaging apparatus according to feature 7.

11. The first determination means determines the guidance information included in the first presentation information based on the current eye condition of the user obtained by the acquisition means, before the user authentication process by the first authentication means. The second determination means does not determine the guidance information as the second presentation information before the user authentication process by the second authentication means. The imaging apparatus according to feature 7.

12. The system further includes a management means for managing the registered eye image of the user acquired in advance and the user's gaze information at the time the registered eye image was acquired. The first determination means determines the guidance information included in the first presentation information based on the gaze information, The second determination means determines the guidance information to be included in the second presentation information based on the line of sight information and the current position of the focus frame. The imaging apparatus according to feature 4.

13. The second determination means determines the second presentation information based on the authentication status of the first authentication means. The imaging apparatus according to feature 4.

14. The second determination means determines the second presentation information, which includes at least one of the following pieces of information: the authentication status of the first authentication means, that the user has not been identified, and that authentication by the first authentication means is necessary to identify the user, if the authentication status of the first authentication means is unauthenticated or authentication has failed. The imaging apparatus according to feature 13.

15. The information presentation means is a display means for displaying the first presentation information and the second presentation information. The imaging apparatus according to feature 1.

16. The display means is at least one of the display device in the viewfinder of the imaging device and the rear display of the imaging device. The information presentation means displays the first presentation information as a UI at any position on the screen of the display means, and displays the second presentation information as a UI at the edge of the screen of the display means. The imaging apparatus according to feature 15.

17. The display means displays the subject image on the screen when it is displaying the second presentation information as the UI at the edge of the screen. The imaging device according to feature 16.

18. The aforementioned information presentation means is The UI is displayed as the first presented information, which includes at least one piece of information from the authentication status of the first authentication means and the authentication status of the second authentication means. As the second presentation information, the UI is displayed with the color of at least one of the following changed: the focus frame, the distance measurement point indicator, the grid, the text indicating the shooting mode, and the icon indicating the shooting mode, depending on at least one of the information from the authentication status of the first authentication means and the authentication status of the second authentication means. The imaging device according to feature 16.

19. The system further includes means for acquiring the user's eye condition, The aforementioned information presentation means is As the first presentation information, if the authentication status of the first authentication means is an authentication failure, the UI including the guidance information is displayed based on the current eye state of the user obtained by the acquisition means. As the second presentation information, if the authentication status of the second authentication means is authentication failure, the UI is displayed which includes guidance information that guides the user to gaze at at least one of the focus frame, distance measurement point indicator, and grid, based on the user's current eye state acquired by the acquisition means. The imaging device according to feature 16.

20. The information presentation means is a sound output means that outputs the first presentation information and the second presentation information as sound. The imaging apparatus according to feature 1.

21. A control method for an imaging device comprising: a first imaging means for capturing a subject image based on user operation; and a second imaging means for capturing an eye image including the user's eye as an authentication image, A first authentication step is performed to authenticate the user based on the eye image obtained by the second imaging means before capturing the subject image by the first imaging means, A second authentication step is performed when capturing the subject image with the first imaging means, based on the eye image obtained by the second imaging means, When the second imaging means captures the eye image in the first authentication step, it presents to the user first presentation information including guidance information for guiding at least one of the position and orientation of the user's eyes, and when the second imaging means captures the eye image in the second authentication step, it presents to the user second presentation information relating to the authentication process in the second authentication step, A control method for an imaging device, characterized by having the following features.

22. A program for causing a computer to perform each step in the control method of the imaging apparatus described in claim 21.

Images