Information processing apparatus, method for controlling information processing apparatus, and storage medium

Abstract

To suppress deterioration of usability by improving authentication accuracy of a user of an information processing apparatus.SOLUTION: An information processor 100 for authenticating a user includes a registration data management part 505 for managing authentication registration information of a user who is permitted to use the information processor, and a first authentication part 507 for authenticating an authentication object user by use of a plurality of pieces of authentication object information acquired from a plurality of different parts of the authentication object user and authentication registration information managed by the registration data management part 505.SELECTED DRAWING: Figure 5A

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] Conventionally, there have been methods for personally authenticating a user of an information processing device. For example, Patent Document 1 describes a method for personally authenticating a user using an image of the user's eyes when the user looks through a finder. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2024-2562 [Patent Document 2] Patent No. 3307936 [Patent Document 3] Patent No. 7346528 Summary of the Invention [Problem to be solved by the invention]

[0004] Generally, when authenticating a user of an information processing device, an extremely low false acceptance rate is set to prevent the user from being mistakenly recognized as a different person. However, setting such a low false acceptance rate increases the false rejection rate. Therefore, when authenticating a user while using the information processing device, false rejection often occurs, resulting in a problem of reduced usability.

[0005] The present invention has been made in view of the above-mentioned problems, and has an object to improve the authentication accuracy of a user of an information processing device and to suppress a decrease in usability. [Means for solving the problem]

[0006] The information processing device of the present invention is an information processing device that authenticates users, and has a management means that manages authentication registration information of users who are permitted to use the information processing device, and an authentication means that authenticates the authentication target user using multiple pieces of authentication target information obtained from multiple different parts of the authentication target user and the authentication registration information. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the authentication accuracy of a user of an information processing device and to suppress a decrease in usability. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of the appearance of a camera corresponding to an information processing device according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of an internal mechanism of a camera corresponding to an information processing device according to the first embodiment. [Figure 3] FIG. 2 is a diagram showing an example of the electrical configuration of a camera corresponding to an information processing device according to the first embodiment. [Figure 4] FIG. 3 illustrates an example of a display on a screen of a display device according to the first embodiment. [Figure 5A] FIG. 2 is a diagram illustrating an example of the functional configuration of a camera corresponding to an information processing device according to the first embodiment. [Figure 5B] 5B illustrates an example of various tables managed by the registered data management unit illustrated in FIG. 5A according to the first embodiment. FIG. [Figure 5C] 5B illustrates an example of various tables managed by the registered data management unit illustrated in FIG. 5A according to the first embodiment. FIG. [Figure 5D] 5B illustrates an example of various tables managed by the registered data management unit illustrated in FIG. 5A according to the first embodiment. FIG. [Figure 5E] 5B illustrates an example of various tables managed by the registered data management unit illustrated in FIG. 5A according to the first embodiment. FIG. [Figure 5F]5B illustrates an example of an authentication state table managed by the authentication state management unit illustrated in FIG. 5A according to the first embodiment. FIG. [Figure 6A] 10 is a flowchart showing an example of a detailed processing procedure of a registration process in a control method for a camera corresponding to the information processing device according to the first embodiment. [Figure 6B] 6B is a flowchart showing an example of a detailed processing procedure of the registration processing in the control method for the camera corresponding to the information processing device according to the first embodiment, following FIG. 6A. [Figure 7] 6B is a flowchart showing an example of detailed processing procedures for acquiring an eye image in step S604 of FIG. 6A. [Figure 8A] 10 is a flowchart illustrating an example of a detailed processing procedure of a first authentication process in a control method for a camera corresponding to an information processing device according to the first embodiment. [Figure 8B] 8B is a flowchart showing an example of a detailed processing procedure of the first authentication processing in the control method for the camera corresponding to the information processing device according to the first embodiment, following FIG. 8A. [Figure 9] 10 is a flowchart illustrating an example of a detailed processing procedure of a second authentication process in the control method for a camera corresponding to the information processing device according to the first embodiment. [Figure 10] 10 is a flowchart showing an example of detailed processing procedures for detecting unauthorized use in step S916 of FIG. 9. [Figure 11A] 10 is a flowchart illustrating an example of a detailed processing procedure of a first authentication invalidation process in a control method for a camera corresponding to an information processing device according to the first embodiment. [Figure 11B] 10 is a flowchart illustrating an example of a detailed processing procedure of a first authentication invalidation process in a control method for a camera corresponding to an information processing device according to the first embodiment. [Figure 11C] 10 is a flowchart illustrating an example of a detailed processing procedure of a first authentication invalidation process in a control method for a camera corresponding to an information processing device according to the first embodiment. [Figure 11D]10 is a flowchart illustrating an example of a detailed processing procedure of a first authentication invalidation process in a control method for a camera corresponding to an information processing device according to the first embodiment. [Figure 12A] 10 is a flowchart showing an example of a detailed processing procedure of an authentication status storage process in the control method for a camera corresponding to the information processing device according to the first embodiment. [Figure 12B] FIG. 4 is a diagram illustrating an example of the configuration of an image file stored by an authentication status storage unit according to the first embodiment. [Figure 13] 1A to 1C are diagrams illustrating the principle of a user's gaze detection process according to the first embodiment. [Figure 14] FIG. 2 is a diagram illustrating the first embodiment and is used to explain the user's line of sight detection process. [Figure 15] 10 is a flowchart showing an example of a detailed processing procedure of a gaze detection process in a control method for a camera corresponding to the information processing device according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0010] (First embodiment) First, the first embodiment will be described.

[0011] [Camera configuration] FIG. 1 is a diagram showing an example of the appearance of a camera 100 corresponding to an information processing device according to the first embodiment. Specifically, the camera 100 corresponding to the information processing device according to this embodiment may be, for example, a digital still camera with an interchangeable lens. FIG. 1(a) is a front perspective view showing an example of the appearance of the camera 100 according to the first embodiment. FIG. 1(b) is a rear perspective view showing an example of the appearance of the camera 100 according to the first embodiment. In FIGS. 1(a) and 1(b), the same components are denoted by the same reference numerals. Also, FIGS. 1(a) and 1(b) show an XYZ coordinate system in which the optical axis direction of the camera 100 is defined as the Z-axis direction, and two mutually orthogonal directions that are orthogonal to the Z-axis direction are defined as the X-axis direction and the Y-axis direction.

[0012] 1(a), camera 100 has a photographing lens unit 110 and a camera housing 120. On the front of camera housing 120 is disposed release button 121, which is an operating member that accepts image capturing operations from the user (photographer).

[0013] As shown in FIG. 1(b), an eyepiece 122 (finder) is arranged on the back of the camera housing 120, through which the user looks to view a display device (display device 214 in FIG. 2, which will be described later) contained inside the camera housing 120. Furthermore, operation members 123 to 125 that accept various operations from the user are also arranged on the back of the camera housing 120. For example, operation member 123 is a touch panel that accepts touch operations, operation member 124 is an operation lever that can be pushed down in each direction, and operation member 125 is a four-way key that can be pressed in each of four directions. Operation member 123, which is a touch panel, is equipped with a display panel (for example, a liquid crystal panel) and has the function of displaying various images on the display panel.

[0014] Fig. 2 is a diagram showing an example of the internal mechanism of a camera 100 corresponding to an information processing device according to the first embodiment. Specifically, Fig. 2 is a cross-sectional view of the camera 100 taken along a YZ plane defined by the Y-axis direction and the Z-axis direction shown in Fig. 1(a). In Fig. 2, the same components as those shown in Fig. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted. Fig. 2 also shows an XYZ coordinate system corresponding to the XYZ coordinate system shown in Fig. 1.

[0015] The photographing lens unit 110 has, as its internal mechanisms, lenses 201 and 202, an aperture 203, an aperture driver 204, a lens drive motor 205, a lens drive member 206, a pulse plate 207, a photocoupler 208, a focus adjustment circuit 209, and a mount contact 210. The lens drive member 206 includes a drive gear. The photocoupler 208 detects the rotation of the pulse plate 207, which is linked to the lens drive member 206, and transmits this information to the focus adjustment circuit 209. The focus adjustment circuit 209 drives the lens drive motor 205 based on information from the photocoupler 208 and information from the camera housing 120 (information on the lens drive amount), thereby moving the lens 201 and changing the focus position. The mount contact 210 is an interface between the photographing lens unit 110 and the camera housing 120. For simplicity, two lenses 201 and 202 are shown in FIG. 2, but in reality, more than two lenses are included inside the photographing lens unit 110.

[0016] Camera housing 120 has, as internal mechanisms, an imaging element 211, a CPU 212, a memory unit 213, a display device 214, a display device drive circuit 215, light sources 216a and 216b, a light splitter 217, a light receiving lens 218, and an eye imaging element 219.

[0017] The image sensor 211 is disposed at a planned imaging plane of the photographing lens unit 110. The CPU 212 is a central processing unit of a microcomputer, and controls the overall operation of the camera 100 and performs various processes. The memory unit 213 stores various types of information, programs executed when the CPU 212 performs various processes, etc. For example, the memory unit 213 stores a subject image captured by the image sensor 211. The display device 214 displays various types of information on a screen (display surface) of the display device 214. For example, the display device 214 is a liquid crystal panel, and displays the captured image (subject image) on the screen. A display device drive circuit 215 drives the display device 214. The eye E of the user (photographer) can view the screen of the display device 214 through the eyepiece 122.

[0018] The light sources 216a and 216b are light sources conventionally used in single-lens reflex cameras to detect the line of sight of the eye E from the relationship between the pupil and a reflection image (corneal reflection image) of light reflected from the cornea. Specifically, the light sources 216a and 216b are light sources for illuminating the eye E of the user looking through the viewfinder (eyepiece 122). For example, the light sources 216a and 216b are infrared light-emitting diodes that emit infrared light insensitive to the user's eye E, and are arranged around the eyepiece 122. The optical image of the eye E illuminated by the light sources 216a and 216b (eye optical image; optical image formed by light emitted from the light sources 216a and 216b and reflected by the eye E) passes through the eyepiece 122 and is reflected by the light splitter 217. The eye optical image is then formed by a light receiving lens 218 on an eye imaging element 219, which has a plurality of photoelectric conversion elements (e.g., CCD or CMOS) arranged two-dimensionally. The light receiving lens 218 positions the pupil of the user's eye E and the eye imaging element 219 in a conjugate imaging relationship. By performing gaze detection processing, the gaze of the eye E is detected from the position of the corneal reflection image in the eye optical image formed on the eye imaging element 219. For example, at least one of information indicating the gaze direction and information indicating the viewpoint (position where the gaze is fixed) on the screen of the display device 214 can be obtained as information regarding the gaze. The viewpoint can be regarded as the position at which the user is looking, or as the gaze position.

[0019] Fig. 3 is a diagram showing an example of the electrical configuration of a camera 100 corresponding to the information processing device according to the first embodiment. In Fig. 3, the same components as those shown in Fig. 1 and Fig. 2 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0020] The photographing lens unit 110 has, as its electrical configuration, a focus adjustment circuit 209 and an aperture control circuit 306 shown in FIG.

[0021] Camera housing 120 has, as its electrical configuration, release button 121 and operation members 123 to 125 shown in Fig. 1. Camera housing 120 also has, as its electrical configuration, image sensor 211, CPU 212, memory section 213, display device 214, display device drive circuit 215, light sources 216a and 216b, and eye image sensor 219 shown in Fig. 2. Camera housing 120 also has, as its electrical configuration, gaze detection circuit 301, photometry circuit 302, autofocus detection circuit 303, signal input circuit 304, and light source drive circuit 305 as shown in Fig. 3.

[0022] 3, the CPU 212 is connected to the line-of-sight detection circuit 301, the photometry circuit 302, the autofocus detection circuit 303, the signal input circuit 304, the light source drive circuit 305, the image sensor 211, the display device drive circuit 215, the memory section 213, and the operation members 123 to 125. The CPU 212 also transmits signals to the focus adjustment circuit 209 disposed in the photographing lens unit 110 and the aperture control circuit 306 included in the aperture drive section 204 in the photographing lens unit 110 via the mount contacts 210. The memory section 213 attached to the CPU 212 has a function of storing image signals from the image sensor 211 and the eye image sensor 219, for example.

[0023] The gaze detection circuit 301 performs A / D conversion on the output of the eye imaging element 219 (an eye image obtained by capturing an image of the user's eye E) when an eye optical image is formed on the eye imaging element 219, and transmits the result to the CPU 212. The CPU 212 extracts feature points required for gaze detection from the eye image according to gaze detection processing, and detects the user's gaze from the positions of the feature points.

[0024] The photometry circuit 302 performs predetermined processing (e.g., amplification, logarithmic compression, and A / D conversion) on a signal obtained from the image sensor 211, which also functions as a photometry sensor, such as a luminance signal corresponding to the brightness of the field, and sends the result to the CPU 212 as field luminance information.

[0025] The autofocus detection circuit 303 A / D converts signals from multiple detection elements (multiple pixels) included in the image sensor 211 that are used for phase difference detection, and sends the converted signals to the CPU 212. The CPU 212 calculates the distance to the subject corresponding to each focus detection point from the signals from the multiple detection elements. This is a well-known technique known as image plane phase difference AF.

[0026] FIG. 4 illustrates a first embodiment and is a diagram showing an example of a display on the screen of the display device 214. As an example, in this embodiment, it is assumed that the focus detection points described in the autofocus detection circuit 303 are located at 180 points on the imaging surface corresponding to the 180 points shown on the screen (viewfinder field of view) of the display device 214 shown in FIG. 4(a). FIG. 4(a) illustrates the display device 214 in an operating state (a state in which an image is displayed), and the viewfinder field of view includes a focus detection area 401, a field of view mask 402, and 180 focus detection point indices 410 within the focus detection area 401. Each focus detection point indices 410 shown in FIG. 4(a) is displayed superimposed on a through image (live view image) displayed on the display device 214 so as to be displayed at a position corresponding to the focus detection point on the imaging surface. Of the 180 focus detection point indices 410 shown in FIG. 4(a), the focus detection point indices 410A corresponding to the current viewpoint A (estimated position) are displayed highlighted, for example, with a frame.

[0027] Here, we return to the explanation of FIG. Switches SW1 and SW2 of the release button 121 are connected to the signal input circuit 304. Switch SW1 is a switch that is turned on by the first stroke of the release button 121 and starts the shooting preparation operation (e.g., photometry and distance measurement) of the camera 100. Switch SW2 is a switch that is turned on by the second stroke of the release button 121 and starts the shooting operation. When an ON signal from switches SW1 and SW2 of the release button 121 is input to the signal input circuit 304, the signal input circuit 304 transmits the input ON signal to the CPU 212. Here, when switch SW1 of the release button 121 is turned on, detection of the user's line of sight may be started.

[0028] The light source drive circuit 305 drives the light sources 216a and 216b.

[0029] Furthermore, when the user operates the operation members 123 to 125, the operation members 123 to 125 output an operation signal corresponding to the operation from the user to the CPU 212. Then, the CPU 212 performs processing (control) corresponding to the operation signal. For example, the CPU 212 moves a selection frame of a displayed menu in response to the operation signal.

[0030] FIG. 5A is a diagram showing an example of the functional configuration of a camera 100 corresponding to an information processing device according to the first embodiment. The camera 100 is an information processing device that performs user authentication. The camera 100 has, as its functional components, an eye image acquisition unit 501, a feature vector calculation unit 502, a left / right eye determination unit 503, a user registration unit 504, a registration data management unit 505, and an authentication target person confirmation unit 506. The camera 100 also has, as its functional components, a first authentication unit 507, a second authentication unit 508, an unauthorized use detection unit 509, a first authentication status invalidation unit 510, an imaging unit 511, and an execution unit 520. For example, the CPU 212 shown in FIGS. 2 and 3 executes a program stored in the memory unit 213 shown in FIGS. 2 and 3 to realize each of the functional components (501 to 511, 520) shown in FIG. 5.

[0031] In this embodiment, camera 100 performs personal authentication of a user by checking the user's eye E looking through the viewfinder (eyepiece 122) to determine whether the user is a registered person. In particular, first authentication is performed before the user uses camera 100 to take a photograph, and second authentication is performed when the photograph is taken (for example, while the photograph is being taken). In this embodiment, camera 100 stores these authentication results together with the photographed image.

[0032] The eye image acquisition unit 501 is an eye image acquisition means that acquires an eye image, which is an image of the user's eye E looking through the finder (eyepiece 122). Specifically, the eye image acquisition unit 501 acquires the eye image (eye image signal; electric signal of the eye image) from the eye imaging element 219 shown in FIG. 3 via the line of sight detection circuit 301.

[0033] The feature vector calculation unit 502 calculates a feature vector, which is feature information used to authenticate a user (authentication target user), from the eye image acquired by the eye image acquisition unit 501. To calculate the feature vector, for example, a neural network is used as a feature extractor. In this embodiment, for example, a convolutional neural network (CNN), which is a type of neural network, is used. CNN extracts abstracted information from an input image by repeatedly performing a process consisting of a convolution process, an activation process, and a pooling process on the input image. In this case, a processing unit consisting of the convolution process, the activation process, and the pooling process is often called a layer. There are several known activation processes used in this process, and for example, a method called Rectified Linear Unit (ReLU) may be used. There are also several known pooling processes, and for example, a method called maximum value pooling may be used. For example, a ResNet or the like may be used as the CNN structure. Furthermore, a neural network known as Vision Transformer (ViT) may also be used. The configuration of the neural network is not limited to the one described above. The feature vector calculation unit 502 may store information such as the structure and weights of the neural network in the memory unit 213 or the like. The weights of the neural network used in the feature vector calculation unit 502 are acquired in advance by learning. For example, eye images of various people are acquired in advance for learning, and learning is performed using a method such as ArcFace. Although an example of using a neural network as a method for personal authentication using eye images has been shown here, known methods such as iris authentication (for example, the method described in Patent Document 2) may also be used. The method for personal authentication of a user is not limited to the one described above.

[0034] The left / right eye determination unit 503 is a determination unit that determines whether the eye image acquired by the eye image acquisition unit 501 is an eye image of the right eye or the left eye of the user (user to be authenticated). For example, in this embodiment, the left / right eye determination unit 503 determines whether the eye image is a right eye image or a left eye image based on a difference between the gaze of the user (user to be authenticated) estimated from the eye image acquired by the eye image acquisition unit 501 and the gaze of the actual user (user to be authenticated).

[0035] The user registration unit 504 creates data to be registered in the registration data management unit 505 .

[0036] The registration data management unit 505 is a management means for managing authentication registration information of users who are permitted to use the camera 100, which corresponds to an information processing device. Specifically, the registration data management unit 505 associates the feature vector of the eye image of the registered user with the name, etc. of the registered user and stores the associated information in the memory unit 213. In this embodiment, the registration data management unit 505 associates the authentication registration information used for the first authentication and the authentication registration information used for the second authentication of the same user and stores and manages the associated information in the memory unit 213.

[0037] 5B to 5E show examples of various tables managed by the registration data management unit 505 shown in FIG. 5A according to the first embodiment. Specifically, the registered user information table 530 shown in FIG. 5B is a table in which a person ID is associated with the name of a registered user. The first authentication registration right eye feature vector table 540 shown in FIG. 5C is a table in which a person ID is associated with a first authentication registration right eye feature vector used in the first authentication unit 507. The first authentication registration left eye feature vector table 550 shown in FIG. 5D is a table in which a person ID is associated with a first authentication registration left eye feature vector used in the first authentication unit 507. The second authentication registration feature vector table 560 shown in FIG. 5E is a table in which a person ID is associated with a second authentication registration feature vector used in the second authentication unit 508. The various tables 530 to 560 shown in FIGS. 5B to 5E associate information of the same registered user with the person ID.

[0038] Fig. 5F shows the first embodiment and is a diagram showing an example of authentication state table 570 managed by authentication state management unit 521 shown in Fig. 5A. Like the various tables 530 to 560 shown in Figs. 5B to 5E, authentication state table 570 shown in Fig. 5F also links information of the same registered user by person ID.

[0039] Here, we return to the description of FIG. 5A. The authentication target person confirmation unit 506 confirms whether the two eye images acquired by the eye image acquisition unit 501 are acquired from the same person (user to be authenticated).

[0040] The first authentication unit 507 authenticates the user to be authenticated using feature vectors calculated by the feature vector calculation unit 502, which are feature information based on eye images acquired from the right and left eyes of the user to be authenticated, and authentication registration information managed by the registration data management unit 505. In this embodiment, the feature vectors calculated by the feature vector calculation unit 502, which are feature information based on eye images acquired from the right and left eyes of the user to be authenticated, correspond to multiple pieces of biometric information acquired from multiple parts of the user to be authenticated, which are multiple pieces of authentication target information. This authentication of the user to be authenticated (first authentication) by the first authentication unit 507 is performed before the user to be authenticated captures an image with the camera 100 (when not capturing an image). Note that in this embodiment, the first authentication unit 507 may consider the first authentication to be successful if authentication using at least one piece of authentication target information among the multiple pieces of authentication target information is successful.

[0041] The second authentication unit 508 authenticates the user to be authenticated by using authentication target information acquired from one of the multiple parts (left and right eyes) of the user to be authenticated and the authentication registration information managed by the registration data management unit 505. This authentication of the user to be authenticated (second authentication) by the second authentication unit 508 is performed while the user to be authenticated is photographing (at the time of photographing) using the camera 100. Furthermore, for example, the second authentication by the second authentication unit 508 is performed after the first authentication by the first authentication unit 507 has been successful.

[0042] In this embodiment, the first authentication unit 507 and the second authentication unit 508 constitute an "authentication means" that authenticates the user to be authenticated. Here, it is desirable for the first authentication unit 507 to use an authentication setting that reduces the false acceptance rate. On the other hand, it is desirable for the second authentication unit 508 to use an authentication setting that reduces the false rejection rate. One method for achieving such authentication settings is, for example, to change the similarity threshold when the same authentication method is used in the first authentication unit 507 and the second authentication unit 508. Specifically, in this case, a high similarity threshold is set in the first authentication unit 507, and a low similarity threshold is set in the second authentication unit 508. This allows the first authentication unit 507 to have a low false acceptance rate, and the second authentication unit 508 to have a low false rejection rate. From the above, when it is difficult to simultaneously reduce both the false acceptance rate and the false rejection rate solely through image capture, this can be achieved by performing two-stage authentication.

[0043] The other person's use detection unit 509 detects that the photographing is being performed by a person (other person) different from the person authenticated by the first authentication unit 507. Here, the other person's use detection unit 509 determines whether or not the person is a different person from the trend of authentication failures in the second authentication unit 508. Specifically, the other person's use detection unit 509 determines that the person is a different person when authentication failures by the second authentication unit 508 occur a predetermined number of times or more in succession. Alternatively, the other person's use detection unit 509 determines that the person is a different person when the authentication score in the second authentication unit 508 is significantly low. Note that the method of determining whether or not the person is a different person is not limited to the one described here.

[0044] When the first authentication is successful by the first authentication unit 507, the first authentication status invalidation unit 510 determines whether or not the authentication status should be invalidated. There are several methods for determining whether or not to invalidate the authentication status.

[0045] The first invalidation determination method is a method based on the time elapsed since the first authentication was successful. For example, when the time elapsed since the first authentication was successful exceeds a predetermined lifetime, the first authentication state is invalidated. In this case, for example, if the second authentication is successful while the first authentication is valid, the lifetime is extended. Conversely, if the second authentication fails, the lifetime is shortened. Note that the invalidation determination method based on the elapsed time is not limited to the one described here.

[0046] The second invalidation determination method is a method based on a change in the power state of the camera 100. For example, the first authentication state is invalidated when the power of the camera 100 is turned off or when the camera 100 enters sleep mode. However, if the power of the camera 100 is turned off due to a dead battery or the like, the process of invalidating the first authentication state cannot be executed. Therefore, rather than invalidating when the power is turned off, invalidation may be performed when the power is turned on. Similarly, in the case of sleep mode, invalidation may be performed when the camera returns from sleep mode. Note that the invalidation determination method based on a change in the power state of the camera 100 is not limited to the one described here.

[0047] The third invalidation determination method is a method based on the distance and connection status of a device carried by the user. An example of the device is a smartphone. For example, the camera 100 is connected to a user's smartphone via Bluetooth, and when the connection is terminated, the first authentication status is invalidated. Alternatively, the first authentication status may be invalidated when it is determined that the user has moved away from the smartphone by a predetermined distance or more, by estimating the approximate distance from the connection status. This prevents the camera 100 from being used by others when the user leaves the camera 100 and moves away. Note that a device other than a smartphone, such as an RFID tag, may also be used. Note that the invalidation determination method based on the distance and connection status of a device carried by the user is not limited to the one described here.

[0048] The fourth invalidation determination method is a method based on an explicit invalidation operation input by the user. For example, an operation menu for "invalidate first authentication" is prepared in a menu or the like, and the user selects and executes it using the operation members 123 to 125. Alternatively, a switch button such as an invalidation button is provided on the camera 100, and the user presses it. When these operations are accepted, the first authentication state is invalidated. Note that the invalidation determination method based on an explicit invalidation operation input by the user is not limited to the one described here.

[0049] The fifth invalidation determination method is a method based on the detection result of the unauthorized use detection unit 509. Specifically, when the unauthorized use detection unit 509 detects use by another person, the first authenticated state is invalidated.

[0050] By using the first to fifth invalidation determination methods described above in combination, the possibility of erroneous acceptance of the second authentication by another person can be reduced. Specifically, the fourth invalidation determination method allows the user to consciously prevent use by another person. In addition, the first to third invalidation determination methods allow the first authenticated user to invalidate the first authentication status when the availability of the first authenticated user is low, thereby prevent use by another person. Furthermore, the fifth invalidation determination method allows the first authenticated status to be invalidated when use by another person is suspected, thereby preventing use by another person.

[0051] When the imaging unit 511 receives a signal indicating that the release button 121 has been pressed by the user, the imaging unit 511 stores the image (subject image) captured by the imaging element 211 in the memory unit 213.

[0052] The execution unit 520 executes a predetermined process based on the authentication states of the first authentication and the second authentication. As shown in FIG. 5A, the execution unit 520 includes an authentication state management unit 521, an authentication state storage unit 522, and an authentication state display unit 523.

[0053] As a predetermined process, the authentication state management unit 521 executes a management process for the authentication state of the first authentication by the first authentication unit 507 and the second authentication by the second authentication unit 508. In addition, the authentication state management unit 521 also executes a management process for determining whether or not the authentication state has been used by another person. For example, the authentication state management unit 521 stores an authentication state table 570 shown in FIG. 5E in the memory unit 213 and executes the management process.

[0054] The authentication status table 570 shown in FIG. 5E will be described. The "first authentication state" in the authentication state table 570 indicates whether or not the first authentication is being performed by the first authentication unit 507, and takes one of the values ​​"authenticated" and "unauthenticated." The person ID is the ID of the person identified by the first authentication. When the first authentication state is "unauthenticated," the person ID takes a value indicating an empty state, such as NULL. The "second authentication state" in the authentication state table 570 indicates whether or not the second authentication is being performed by the second authentication unit 508, and takes one of the values ​​"authenticated" and "unauthenticated." The "presence of use by another person" in the authentication state table 570 indicates whether or not use by another person has been detected by the other person's use detection unit 509, and takes one of the values ​​"present" and "not present." Specific processing for updating the authentication state table 570 will be described later together with the explanations of the first authentication processing (FIGS. 8A and 8B), the second authentication processing (FIG. 9), and the first authentication invalidation processing (FIGS. 11A to 11D). The method of storing the authentication state table 570 in the memory unit 213 is not limited to a table structure, but may be, for example, a key-value structure.

[0055] Here, we return to the description of FIG. 5A. As a predetermined process, the authentication status saving unit 522 executes a saving process to save the authentication status of the first authentication and the second authentication managed by the authentication status management unit 521, and whether or not the image has been used by another person, as metadata, in association with the image acquired by the imaging unit 511. Here, the process of saving image metadata can be, for example, a method known as C2PA. C2PA is a method of adding metadata indicating edits made to an image to authenticate the origin, history, and origin of the image. Therefore, the authentication status saving unit 522 may save the authentication status, etc., in accordance with C2PA. However, in this embodiment, other methods may be used for saving. Alternatively, the image file and the metadata file may be saved separately. Furthermore, the metadata may be stored in a database.

[0056] As a predetermined process, authentication status display unit 523 executes a display process of displaying the authentication status of the first authentication and the second authentication managed by authentication status management unit 521 on camera 100. For example, when the first authentication status is "authenticating", authentication status display unit 523 displays "first authentication in progress" on display device 214 or a touch panel (operation member 123). Furthermore, camera 100 may be configured to include an LED lamp (not shown) or the like, and authentication status display unit 523 may light up the LED lamp when the first authentication status is "authenticating".

[0057] In this embodiment, the execution unit 520 may take a form in which the content of predetermined processing in the authentication status management unit 521, authentication status storage unit 522, and authentication status display unit 523 is changed depending on the authentication results of the first authentication and the second authentication.

[0058] [Registration process] 6A and 6B are flowcharts showing an example of detailed processing procedures for registration processing in a control method for camera 100, which corresponds to the information processing device according to the first embodiment. The processing of the flowcharts shown in FIGS. 6A and 6B is mainly executed by CPU 212 via user registration unit 504. It is assumed that the processing of the flowcharts shown in FIGS. 6A and 6B is executed by a user operating camera 100 at a time other than when taking a photograph. Therefore, the processing of the flowcharts shown in FIGS. 6A and 6B is executed when a user operates camera 100 to call up this processing from a menu or the like. For example, a menu screen (not shown) is displayed on the touch panel (operation member 123) of camera 100, and the user operates operation members 123 to 125 on the menu screen or the like to select a menu that calls up this processing, thereby executing this processing.

[0059] First, in step S601 of FIG. 6A, the CPU 212 (user registration unit 504) accepts input of personal information of the person to be registered. In this embodiment, the CPU 212 (user registration unit 504) accepts input of "name". Specifically, the CPU 212 (user registration unit 504) displays a screen for inputting a name (not shown) on the touch panel (operation member 123), and accepts input of the name by the user operating the operation members 123 to 125. When the user has finished inputting, the user notifies the completion of name input by pressing a completion button or the like displayed on the screen.

[0060] Next, in step S602 of Fig. 6A, the CPU 212 (user registration unit 504) displays to the user a method for registering an eye image of the dominant eye. Specifically, the CPU 212 (user registration unit 504) displays instructions on the touch panel (operation member 123) to instruct the user to look into the finder with the dominant eye that will be used when taking a photograph. The CPU 212 (user registration unit 504) also displays instructions to look at an indicator in the finder. In addition, the CPU 212 (user registration unit 504) may display instructions to capture a desirable eye image, such as not blinking and keeping the eye wide open.

[0061] 6A, the indices 411 to 415 shown in Fig. 4(c) are displayed in order on the display device 214, and the feature vectors obtained from the images of the user's eyes when looking at the indices are stored in the registered data management unit 505. The processes will be explained in order.

[0062] In step S603 of FIG. 6A, the CPU 212 (user registration unit 504) displays an index on the display device 214. Specifically, only the index 411 shown in FIG. 4(c) is displayed, and the other indexes are not displayed. Alternatively, all the indexes 411 to 415 shown in FIG. 4(c) may be displayed, and only the index 411 may be displayed in an emphasizing color. Note that other display methods may be used as long as they can inform the user that they should look at the index 411.

[0063] Next, in step S604 in Fig. 6A, eye image acquisition unit 501 acquires an eye image when the user looks through the finder (eyepiece 122). Here, detailed processing of step S604 in Fig. 6A will be described with reference to Fig. 7.

[0064] Fig. 7 is a flowchart showing an example of detailed processing procedures for acquiring an eye image in step S604 of Fig. 6A. The processing of the flowchart shown in Fig. 7 is mainly executed by the eye image acquisition unit 501 in the CPU 212.

[0065] First, in step S701 of FIG. 7, the eye image acquisition unit 501 executes a process of detecting the gaze of the user.

[0066] 7, the eye image acquisition unit 501 determines whether an image suitable for authentication has been acquired. Specifically, the eye image acquisition unit 501 determines whether an image suitable for authentication has been acquired based on whether the gaze detection process of step S701 has been successful. For example, in the gaze detection process of step S701, the eye image acquisition unit 501 acquires an eye image (eye image signal; an electrical signal of the eye image) from the eye imaging element 219 via the gaze detection circuit 301. Next, the eye image acquisition unit 501 obtains the coordinates of the corneal reflection images of the light sources 216a and 216b and the pupil center observed on the eye image. Next, the eye image acquisition unit 501 obtains the gaze coordinates on the display device 214 of the user from the obtained coordinates. Therefore, if the eye image acquisition unit 501 cannot detect the coordinates of the pupil center, etc., the eye image acquisition unit 501 determines that the gaze detection process has failed. Therefore, if coordinates such as the pupil center cannot be obtained in step S603 in FIG. 6A, which is the process of obtaining those coordinates, eye image acquisition unit 501 may determine that an image suitable for authentication has not been acquired.

[0067] Subsequently, in step S703 of FIG. 7, the eye image acquisition unit 501 determines whether or not an image suitable for authentication has been successfully acquired, based on the result of the determination in step S702.

[0068] In step S703 of FIG. 7, if the eye image acquisition unit 501 determines that it has successfully acquired an eye image suitable for authentication (S703 / YES), the process proceeds to step S704. 7, the eye image acquisition unit 501 cuts out and acquires an eye image. Specifically, the eye image acquisition unit 501 first acquires the eye image acquired in step S602 of FIG. 6A. Then, using the coordinates of the pupil-centered image c' acquired in step S603 of FIG. 6A, the eye image acquisition unit 501 crops the image to a certain size so that the pupil-centered image c' is at the center of the image. Furthermore, the eye image acquisition unit 501 generates and acquires an eye image resized to the input size of the neural network of the feature vector calculation unit 502.

[0069] Subsequently, in step S705 of FIG. 7, the eye image acquisition unit 501 records, by a flag or the like, that the eye image has been successfully acquired.

[0070] Also, in step S703 of FIG. 7, if the eye image acquisition unit 501 determines that acquisition of an eye image suitable for authentication has not been successful (failed) (S703 / NO), the process proceeds to step S706. 7, the eye image acquisition unit 501 performs a process of waiting for a predetermined time. The process of step S706 is a process in which the eye image changes and the gaze detection is successful.

[0071] 7, the eye image acquisition unit 501 determines whether or not the failure to acquire an eye image suitable for authentication has occurred a predetermined number of times in succession. If the eye image acquisition unit 501 determines that the failure to acquire an eye image suitable for authentication has not occurred a predetermined number of times in succession in step S707 of FIG. 7 (S707 / NO), the process returns to step S701 and repeats the processes from step S701 onward.

[0072] Also, in step S707 of FIG. 7, if the eye image acquisition unit 501 determines that the failure to acquire an eye image suitable for authentication occurs a predetermined number of times in succession (S707 / YES), the process proceeds to step S708. When the process proceeds to step S708 in FIG. 7, the eye image acquisition unit 501 records, by using a flag or the like, that acquisition of the eye image has failed.

[0073] When the process of step S705 in Fig. 7 is completed, or when the process of step S708 in Fig. 7 is completed, the process of the flowchart in Fig. 7 is completed. Then, when the process of the flowchart in Fig. 7 is completed, the process of acquiring the eye image in step S604 in Fig. 6A is completed.

[0074] Here, we return to the description of FIG. 6A. When the process of step S604 in FIG. 6A is completed, the process proceeds to step S605. 6A, the eye image acquisition unit 501 determines whether or not the acquisition of the eye image has been successful. Specifically, the eye image acquisition unit 501 determines whether or not the acquisition of the eye image has been successful based on the flag recorded in step S705 or step S708 in FIG.

[0075] In step S605 of FIG. 6A, if the eye image acquisition unit 501 determines that acquisition of the eye image has been successful (S605 / YES), the process proceeds to step S606. 6A, the feature vector calculation unit 502 extracts a feature vector from the eye image as authentication registration information for authenticating the user. Specifically, the feature vector calculation unit 502 extracts a feature vector from the eye image acquired in step S604 in FIG. 6A.

[0076] 6A, the CPU 212 (user registration unit 504) displays on the display device 214 that the eye image has been successfully captured using the indicator currently being displayed on the display device 214. For example, the CPU 212 (user registration unit 504) may display on the display device 214 a message such as "Eye image captured successfully," or may display an icon or the like indicating success.

[0077] Also, in step S605 of FIG. 6A, if the eye image acquisition unit 501 determines that acquisition of the eye image has not been successful (failed) (S605 / NO), the process proceeds to step S608. 6A, the CPU 212 (user registration unit 504) displays on the display device 214 that capturing an eye image using the index currently being displayed on the display device 214 has failed. For example, the CPU 212 (user registration unit 504) may display on the display device 214 a message such as "Failed to capture an eye image," or may display an icon or the like indicating the failure. When the processing of step S608 ends, the processing returns to step S604.

[0078] Also, when the process of step S607 in FIG. 6A is completed, the process proceeds to step S609. 6A, the CPU 212 (user registration unit 504) determines whether there are any indices that have not yet been displayed. The CPU 212 (user registration unit 504) checks whether all of the indices 411 to 415 shown in FIG. 4C are displayed, and determines whether there are any indices that have not yet been displayed.

[0079] In step S609 of FIG. 6A, if the CPU 212 (user registration unit 504) determines that there are indices that have not yet been displayed (S609 / YES), the process proceeds to step S610. When proceeding to step S610 in FIG. 6A, the CPU 212 (user registration unit 504) displays the next index from among the indices that have not yet been displayed on the display device 214. For example, when the index 411 shown in FIG. 4(c) is displayed, the CPU 212 (user registration unit 504) displays the index 412 as the next index on the display device 214. In this way, the indices 411 to 415 are selected in numerical order and displayed on the display device 214. When the processing of step S610 ends, the processing returns to step S604.

[0080] Also, in step S609 of FIG. 6A, if the CPU 212 (user registration unit 504) determines that there are no indices that have not yet been displayed (S609 / NO), the process proceeds to step S611. When the process proceeds to step S611 in FIG. 6A, the CPU 212 (user registration unit 504) determines whether or not the eye image acquired in step S604 is an eye image acquired from the right eye.

[0081] In step S611 of FIG. 6A, if the CPU 212 (user registration unit 504) determines that the eye image acquired in step S604 is an eye image acquired from the right eye (S611 / YES), the process proceeds to step S612. When the process proceeds to step S612 in FIG. 6A, the CPU 212 (user registration unit 504) stores the obtained information in the first authentication and registration right eye feature vector table 540 shown in FIG. 5C to update it.

[0082] Also, in step S611 of FIG. 6A, if the CPU 212 (user registration unit 504) determines that the eye image acquired in step S604 is not an eye image acquired from the right eye (an eye image acquired from the left eye) (S611 / NO), the process proceeds to step S613. When the process proceeds to step S613 in FIG. 6A, the CPU 212 (user registration unit 504) stores the obtained information in the first authentication and registration left eye feature vector table 550 shown in FIG. 5D to update it.

[0083] When the process of step S612 in FIG. 6A is completed, or when the process of step S613 in FIG. 6A is completed, the process proceeds to step S614. When the process proceeds to step S614 in FIG. 6A, the CPU 212 (user registration unit 504) stores the obtained information in the second authentication registration feature vector table 560 shown in FIG. 5E to update it.

[0084] Here, in steps S612 to S614 of FIG. 6A, the obtained information is stored in the registered personal information table 530, the first authentication / registration right eye feature vector table 540, the first authentication / registration left eye feature vector table 550, and the second authentication / registration feature vector table 560. Specifically, the person IDs of the four tables 530 to 560 shown in FIGS. 5B to 5E are IDs for establishing relationships between the tables, and therefore the same ID values ​​are used in the four tables 530 to 560. The name of the personal information obtained in step S601 of FIG. 6A is added to the registered personal information table 530 shown in FIG. 5B. The feature vectors obtained in step S606 of FIG. 6A are stored separately in the first authentication / registration right eye feature vector table 540 shown in FIG. 5C, the first authentication / registration left eye feature vector table 550 shown in FIG. 5D, and the second authentication / registration feature vector table 560 shown in FIG. 5E. The separation is as follows.

[0085] In the first authentication process (described later with reference to FIGS. 8A and 8B), an index is displayed on the display device 214, and authentication is performed using an image of both eyes when the user looks at the index. Therefore, only the feature vector of the index displayed at that time is saved in a first-authentication registered feature vector table as a registered feature vector for the first authentication. In this embodiment, if the feature vector obtained when the index 411 was displayed was acquired from the right eye, it is registered in the first-authentication registered right-eye feature vector table 540 in step S612 of FIG. 6A. Also, in this embodiment, if the feature vector obtained when the index 411 was displayed was acquired from the left eye, it is registered in the first-authentication registered left-eye feature vector table 550 in step S613 of FIG. 6A.

[0086] On the other hand, in the second authentication process (described later in FIG. 9), the image captured by the image sensor 211 is displayed on the display device 214, and therefore the indices are not displayed. Therefore, it is unclear where the user will gaze on the display device 214. Therefore, in this embodiment, in step S614 of FIG. 6A, all of the feature vectors obtained when the indices 411 to 415 are displayed are registered.

[0087] When the process of step S614 in FIG. 6A is completed, the process proceeds to step S615 in FIG. 6B. When the process proceeds to step S615 in FIG. 6B, the CPU 212 (user registration unit 504) displays to the user a method for registering an eye image of the non-dominant eye. Specifically, the CPU 212 (user registration unit 504) displays instructions on the touch panel (operation member 123) to instruct the user to look into the finder with the non-dominant eye (non-dominant eye) when taking a photograph. The CPU 212 (user registration unit 504) also displays instructions to look at an indicator in the finder. In addition, the CPU 212 (user registration unit 504) may display instructions to capture a desirable eye image, such as not blinking and keeping the eye wide open.

[0088] Subsequently, in step S616 of Fig. 6B, CPU 212 (user registration unit 504) displays an indicator on display device 214. The specific processing in step S616 of Fig. 6B is similar to the processing in step S802 of Fig. 6A, and therefore description thereof will be omitted.

[0089] Next, in step S617 of Fig. 6B, eye image acquisition unit 501 acquires an eye image when the user looks through the finder (eyepiece 122). The specific processing in step S617 of Fig. 6B is the same as the processing in step S604 of Fig. 6A (the processing in the flowchart shown in Fig. 7), and therefore a description thereof will be omitted.

[0090] Next, in step S618 of Fig. 6B, the eye image acquisition unit 501 determines whether or not the acquisition of the eye image has been successful. Specifically, the eye image acquisition unit 501 determines whether or not the acquisition of the eye image has been successful based on the flag recorded in step S705 or step S708 of Fig. 7.

[0091] In step S618 of FIG. 6B, if the eye image acquisition unit 501 determines that acquisition of the eye image has not been successful (failed) (S618 / NO), the process proceeds to step S619. 6B, the CPU 212 (user registration unit 504) displays on the display device 214 that capturing an eye image using the index currently being displayed on the display device 214 has failed. For example, the CPU 212 (user registration unit 504) may display on the display device 214 a message such as "Failed to capture an eye image," or may display an icon or the like indicating the failure. When the processing of step S619 ends, the processing returns to step S617.

[0092] Also, in step S618 of FIG. 6B, if the eye image acquisition unit 501 determines that acquisition of the eye image has been successful (S618 / YES), the process proceeds to step S620. 6B, the CPU 212 (user registration unit 504) determines whether the eye in the eye image acquired in step S617 is the non-dominant eye of the eye image acquired in step S604. The specific determination method in step S620 is the same as that in step S611.

[0093] In step S620 of FIG. 6B, if the CPU 212 (user registration unit 504) determines that the eye in the eye image acquired in S617 is the eye opposite to the dominant eye in the eye image acquired in S604 (S620 / YES), the process proceeds to step S621. 6B, the feature vector calculation unit 502 extracts a feature vector from the eye image as authentication registration information for authenticating the user. Specifically, the feature vector calculation unit 502 extracts a feature vector from the eye image acquired in step S617 in FIG. 6A.

[0094] 6B, the eye image acquisition unit 501 determines whether the person who looked through the viewfinder in step S604 is the same as the person who looked through the viewfinder in step S617. A specific method of determination in step S622 is, for example, to calculate the cosine similarity between the feature vector extracted in S606 and the feature vector extracted in S621, and if the calculated cosine similarity exceeds a predetermined threshold, it is determined that the two are the same person.

[0095] 6B, if the eye image acquisition unit 501 determines that the person who looked through the finder in S604 is not the same as the person who looked through the finder in S617 (S622 / NO), the process proceeds to step S623. Also, if the CPU 212 (user registration unit 504) determines in step S620 of FIG. 6B that the eye in the eye image acquired in S617 is not the non-dominant eye of the eye image acquired in S604 (S620 / NO), the process proceeds to step S623. 6B, CPU 212 (user registration unit 504) displays on display device 214 a message that the acquired eye image is not suitable for authentication. For example, CPU 212 (user registration unit 504) may display a message such as "Eye image is not suitable for authentication" on display device 214. When the processing of step S623 ends, the processing returns to step S617.

[0096] Also, in step S622 of FIG. 6B, if the eye image acquisition unit 501 determines that the person who looked into the finder in S604 is the same as the person who looked into the finder in S617 (S622 / YES), the process proceeds to step S624. 6B, the CPU 212 (user registration unit 504) displays on the display device 214 that the eye image was successfully captured using the indicator currently displayed on the display device 214. For example, the CPU 212 (user registration unit 504) may display on the display device 214 a message such as "Eye image captured successfully," or may display an icon or the like indicating success.

[0097] Subsequently, in step S625 of FIG. 6B, the CPU 212 (user registration unit 504) determines whether or not the eye image acquired in step S617 is an eye image acquired from the right eye.

[0098] In step S625 of FIG. 6B, if the CPU 212 (user registration unit 504) determines that the eye image acquired in step S617 is an eye image acquired from the right eye (S617 / YES), the process proceeds to step S626. When the process proceeds to step S626 in FIG. 6B, the CPU 212 (user registration unit 504) stores the obtained information in the first authentication and registration right eye feature vector table 540 shown in FIG. 5C to update it.

[0099] Also, in step S625 of FIG. 6B, if the CPU 212 (user registration unit 504) determines that the eye image acquired in step S617 is not an eye image acquired from the right eye (an eye image acquired from the left eye) (S625 / NO), the process proceeds to step S627. When the process proceeds to step S627 in FIG. 6B, the CPU 212 (user registration unit 504) stores the obtained information in the first authentication registration left eye feature vector table 550 shown in FIG. 5D to update it.

[0100] When the process of step S626 in FIG. 6B is completed, or when the process of step S627 in FIG. 6B is completed, the process proceeds to step S628. When the process proceeds to step S628 in FIG. 6B, the CPU 212 (user registration unit 504) displays a message on the touch panel (operation member 123) or the display device 214 to notify the user that registration has been completed.

[0101] When the process of step S628 in FIG. 6B ends, the process of the flowcharts in FIGS. 6A and 6B ends.

[0102] In addition, in the registration process shown in Figures 6A and 6B, the process falls into an infinite loop in which the process does not end unless the acquisition of eye images is successful in steps S605 and S618. Therefore, it is desirable to configure the process to be interrupted when a predetermined number of failures are observed.

[0103] [First authentication process] 8A and 8B are flowcharts showing an example of detailed processing steps of the first authentication processing in the control method for the camera 100, which corresponds to the information processing device according to the first embodiment. The processing of the flowcharts shown in FIGS. 8A and 8B is mainly executed by the first authentication unit 507 in the CPU 212. The first authentication processing shown in FIGS. 8A and 8B is assumed to be performed by the user operating the camera 100 at times other than when taking a picture. Therefore, the processing of the flowcharts shown in FIGS. 8A and 8B is executed when the user operates the camera 100 to call up this processing from a menu or the like. For example, this processing is executed when a menu screen (not shown) is displayed on the touch panel (operation member 123) of the camera 100, and the user operates the operation members 123 to 125 on the menu screen or the like to select a menu that calls up this processing.

[0104] First, in step S801 of Fig. 8A, the CPU 212 (first authentication unit 507) instructs the user on an authentication method using an eye image of either eye. Specifically, the CPU 212 (first authentication unit 507) displays instructions on the touch panel (operation member 123) to instruct the user to look through the viewfinder with either eye and look at an index displayed on the display device 214. In addition, the CPU 212 (first authentication unit 507) may display instructions to enable the user to capture a desirable eye image, such as not blinking, keeping the eye wide open, and holding the camera 100 firmly.

[0105] Next, in step S802 of FIG. 8A, the CPU 212 (first authentication unit 507) displays an index on the display device 214. Specifically, as shown in FIG. 4(d), only the index 411 is displayed. This is because the feature vector obtained when the index 411 is viewed during the above-mentioned registration process is registered in the first authentication registration feature vector tables 540 and 550 shown in FIGS. 5C and 5D. In this way, eye images with similar gazes are obtained during registration and authentication, making it easier to match the two.

[0106] Next, in step S803 of Fig. 8A, eye image acquisition unit 501 acquires an eye image when the user looks through the finder (eyepiece 122). The specific processing in step S803 of Fig. 8A is the same as the processing in step S604 of Fig. 6A (the processing in the flowchart shown in Fig. 7), and therefore a description thereof will be omitted.

[0107] Subsequently, in step S804 of Fig. 8A, the eye image acquisition unit 501 determines whether or not the acquisition of the eye image has been successful. Specifically, the eye image acquisition unit 501 determines whether or not the acquisition of the eye image has been successful based on the flag recorded in step S705 or step S708 of Fig. 7.

[0108] In step S804 of FIG. 8A, if the eye image acquisition unit 501 determines that acquisition of the eye image has not been successful (failed) (S804 / NO), the process proceeds to step S805. 8A, the CPU 212 (first authentication unit 507) displays on the display device 214 that capturing an eye image using the index currently being displayed on the display device 214 has failed. For example, the CPU 212 (first authentication unit 507) may display on the display device 214 a message such as "Failed to capture an eye image," or may display an icon or the like indicating the failure. When the processing of step S805 ends, the processing returns to step S803.

[0109] Also, in step S804 of FIG. 8A, if the eye image acquisition unit 501 determines that acquisition of the eye image has been successful (S804 / YES), the process proceeds to step S806. 8A, the feature vector calculation unit 502 extracts a feature vector from the eye image as authentication target information for authenticating the user. Specifically, the feature vector calculation unit 502 extracts a feature vector from the eye image acquired in step S803 of FIG. 8A.

[0110] Subsequently, in step S807 of FIG. 8A, the CPU 212 (first authentication unit 507) determines whether or not the eye image acquired in step S803 is an eye image acquired from the right eye.

[0111] In step S807 of FIG. 8A, if the CPU 212 (first authentication unit 507) determines that the eye image acquired in step S803 is an eye image acquired from the right eye (S807 / YES), the process proceeds to step S808. 8A, the CPU 212 (first authentication unit 507) acquires the registered feature vector to be used for the first authentication from the first authentication registered right eye feature vector table 540 shown in Fig. 5C via the registered data management unit 505. Specifically, the CPU 212 (first authentication unit 507) acquires all feature vectors in the first authentication registered right eye feature vector table 540 shown in Fig. 5C.

[0112] Also, in step S807 of FIG. 8A, if the CPU 212 (first authentication unit 507) determines that the eye image acquired in step S803 is not an eye image acquired from the right eye (an eye image acquired from the left eye) (S807 / NO), the process proceeds to step S809. 8A, the CPU 212 (first authentication unit 507) acquires the registered feature vector to be used for the first authentication from the first authentication registered left eye feature vector table 550 shown in Fig. 5D via the registered data management unit 505. Specifically, the CPU 212 (first authentication unit 507) acquires all feature vectors in the first authentication registered left eye feature vector table 550 shown in Fig. 5D.

[0113] When the process of step S808 in FIG. 8A is completed, or when the process of step S809 in FIG. 8A is completed, the process proceeds to step S810. 8A, the CPU 212 (first authentication unit 507) performs the first authentication by comparing the feature vector, which is the authentication target information acquired in S806, with the registered feature vector, which is the authentication registration information acquired in S808 or S809. Specifically, in the processing of step S810, the cosine similarity between the two feature vectors is calculated, and the first authentication is performed depending on whether the calculated cosine similarity exceeds a predetermined threshold. More specifically, if the calculated cosine similarity exceeds the predetermined threshold, it is determined that the first authentication has been successful, and the person ID of the registered feature vector is identified.

[0114] Subsequently, in step S811 of FIG. 8A, CPU 212 (first authentication unit 507) determines whether or not the first authentication performed in step S810 has been successful.

[0115] In step S811 of FIG. 8A, if the CPU 212 (first authentication unit 507) determines that the first authentication performed in step S810 was successful (S811 / YES), the process proceeds to step S812. 8A, the CPU 212 (first authentication unit 507) instructs the user on an authentication method using the eye image of the opposite eye from that of step S801. Specifically, the CPU 212 (first authentication unit 507) displays instructions on the touch panel (operation member 123) to instruct the user to look through the viewfinder with the eye opposite to the eye in the eye image acquired in step S803 and look at an index displayed on the display device 214. In addition, the CPU 212 (first authentication unit 507) may display instructions to enable the user to capture a desirable eye image, such as not blinking, keeping the eye wide open, and holding the camera 100 firmly.

[0116] 8A, the CPU 212 (first authentication unit 507) displays an indicator on the display device 214. The specific processing in this step S813 is the same as the processing in step S802.

[0117] Next, in step S814 of Fig. 8B, eye image acquisition unit 501 acquires an eye image when the user looks through the finder (eyepiece 122). The specific processing in step S814 of Fig. 8B is the same as the processing in step S604 of Fig. 6A (the processing in the flowchart shown in Fig. 7), as well as step S803 of Fig. 8A, and therefore description thereof will be omitted.

[0118] Subsequently, in step S815 of Fig. 8B, the eye image acquisition unit 501 determines whether or not the acquisition of the eye image has been successful. Specifically, the eye image acquisition unit 501 determines whether or not the acquisition of the eye image has been successful based on the flag recorded in step S705 or step S708 of Fig. 7.

[0119] In step S815 of FIG. 8B, if the eye image acquisition unit 501 determines that acquisition of the eye image has not been successful (failed) (S815 / NO), the process proceeds to step S816. 8B, the CPU 212 (first authentication unit 507) displays on the display device 214 that capturing an eye image using the index currently being displayed on the display device 214 has failed. For example, the CPU 212 (first authentication unit 507) may display on the display device 214 a message such as "Failed to capture an eye image," or may display an icon or the like indicating the failure. When the processing of step S816 ends, the processing returns to step S814.

[0120] Also, in step S815 of FIG. 8B, if the eye image acquisition unit 501 determines that acquisition of the eye image has been successful (S815 / YES), the process proceeds to step S817. 8B, the CPU 212 (first authentication unit 507) determines whether the eye in the eye image acquired in step S814 is the opposite eye to the eye in the eye image acquired in step S803. The specific determination method in step S817 is the same as that in step S611.

[0121] In step S817 of FIG. 8B, if the CPU 212 (first authentication unit 507) determines that the eye in the eye image acquired in S814 is the opposite eye to the eye in the eye image acquired in S803 (S817 / YES), the process proceeds to step S818. 8B, the feature vector calculation unit 502 extracts a feature vector from the eye image as authentication target information for authenticating the user. Specifically, the feature vector calculation unit 502 extracts a feature vector from the eye image acquired in step S814 of FIG. 8B.

[0122] Next, in step S819 of Fig. 8B, eye image acquisition unit 501 determines whether the person who looked into the finder in step S803 is the same as the person who looked into the finder in step S814. The specific determination method in step S819 is the same as step S622 of Fig. 6B.

[0123] 8B, if the eye image acquisition unit 501 determines that the person who looked through the finder in S803 is not the same as the person who looked through the finder in S814 (S819 / NO), the process proceeds to step S820. Also, if the CPU 212 (first authentication unit 507) determines in step S817 of Fig. 6B that the eye in the eye image acquired in S814 is not the opposite eye to the eye in the eye image acquired in S803 (S817 / NO), the process proceeds to step S820. 8B, CPU 212 (first authentication unit 507) displays on display device 214 a message indicating that the acquired eye image is not suitable for authentication. For example, CPU 212 (first authentication unit 507) may display a message such as "Eye image is not suitable for authentication" on display device 214. When the processing of step S820 ends, the processing returns to step S814.

[0124] Also, in step S819 of FIG. 8B, if the eye image acquisition unit 501 determines that the person who looked into the finder in S803 is the same as the person who looked into the finder in S814 (S819 / YES), the process proceeds to step S821. When the process proceeds to step S821 in FIG. 8B, the CPU 212 (first authentication unit 507) determines whether or not the eye image acquired in step S814 is an eye image acquired from the right eye.

[0125] In step S821 of FIG. 8B, if the CPU 212 (first authentication unit 507) determines that the eye image acquired in step S814 is an eye image acquired from the right eye (S821 / YES), the process proceeds to step S822. 8B, the CPU 212 (first authentication unit 507) acquires the registered feature vector to be used for the first authentication from the first authentication registered right eye feature vector table 540 shown in Fig. 5C via the registered data management unit 505. Specifically, the CPU 212 (first authentication unit 507) acquires all feature vectors in the first authentication registered right eye feature vector table 540 shown in Fig. 5C.

[0126] Also, in step S821 of FIG. 8B, if the CPU 212 (first authentication unit 507) determines that the eye image acquired in step S814 is not an eye image acquired from the right eye (an eye image acquired from the left eye) (S821 / NO), the process proceeds to step S823. 8B, the CPU 212 (first authentication unit 507) acquires the registered feature vector to be used for the first authentication from the first authentication registered left eye feature vector table 550 shown in Fig. 5D via the registered data management unit 505. Specifically, the CPU 212 (first authentication unit 507) acquires all feature vectors in the first authentication registered left eye feature vector table 550 shown in Fig. 5D.

[0127] When the process of step S822 in FIG. 8B is completed, or when the process of step S823 in FIG. 8B is completed, the process proceeds to step S824. 8B, the CPU 212 (first authentication unit 507) performs the first authentication by comparing the feature vector, which is the authentication target information acquired in S818, with the registered feature vector, which is the authentication registration information acquired in S822 or S823. The specific processing in this step S824 is the same as that in step S810 in FIG. 8A.

[0128] Subsequently, in step S825 of FIG. 8B, CPU 212 (first authentication unit 507) determines whether or not the first authentication performed in step S824 has been successful.

[0129] In step S825 of FIG. 8B, if the CPU 212 (first authentication unit 507) determines that the first authentication performed in step S824 was successful (S824 / YES), the process proceeds to step S826. 8B, authentication state management unit 521 updates the first authentication state in authentication state table 570 shown in Fig. 5F to "authenticated." Furthermore, authentication state management unit 521 updates the second authentication state in authentication state table 570 shown in Fig. 5F to "unauthenticated."

[0130] Subsequently, in step S827 of FIG. 8B, the authentication state management unit 521 updates the person ID in the authentication state table 570 shown in FIG. 5F to the person ID identified in step S824.

[0131] Subsequently, in step S828 of FIG. 8B, the CPU 212 (first authentication unit 507) causes the authentication status display unit 523 to display on the touch panel (operation member 123) or the display device 214 to inform the user that authentication has been successful.

[0132] 8B, if the CPU 212 (first authentication unit 507) determines that the first authentication performed in step S824 was not successful (S824 / NO), the process proceeds to step S829. Also, if the CPU 212 (first authentication unit 507) determines that the first authentication performed in step S810 was not successful (S811 / NO), the process proceeds to step S829. In step S829 in FIG. 8B, authentication state management unit 521 updates the first authentication state and the second authentication state in authentication state table 570 shown in FIG. 5F to "unauthenticated."

[0133] Next, in step S830 of Fig. 8B, the authentication state management unit 521 deletes the person ID from the authentication state table 570 shown in Fig. 5F. For example, the authentication state management unit 521 may prepare a NULL value as a value indicating an empty state, and overwrite the person ID in the authentication state table 570 shown in Fig. 5F with the NULL value.

[0134] Next, in step S831 of FIG. 8B, the CPU 212 (first authentication unit 507) causes the authentication status display unit 523 to display on the touch panel (operation member 123) or the display device 214 a message informing the user that authentication has failed.

[0135] When the process of step S828 in FIG. 8B is completed, or when the process of step S831 in FIG. 8B is completed, the process of the flowcharts in FIGS. 8A and 8B is completed.

[0136] In the first authentication process shown in Figures 8A and 8B, the process falls into an infinite loop in which the process does not end unless the acquisition of eye images is successful in steps S804 and S815. Therefore, it is desirable to configure the process to be interrupted when a predetermined number of failures are observed.

[0137] [Second authentication process] FIG. 9 is a flowchart showing an example of detailed processing steps of second authentication processing in a control method for camera 100, which corresponds to the information processing device according to the first embodiment. The processing of the flowchart shown in FIG. 9 is mainly executed by second authentication unit 508 in CPU 212. The second authentication processing shown in FIG. 9 is assumed to be performed when a user looks into the viewfinder (eyepiece 122) during shooting. Therefore, the processing of the flowchart shown in FIG. 9 is initiated when an eyepiece sensor (not shown) mounted on camera 100 detects that the user has brought their eye close to the viewfinder (eyepiece 122). The eyepiece sensor is, for example, a sensor that detects contact of the skin around the user's eye with the periphery of eyepiece 122. Alternatively, the eyepiece sensor may be a sensor that detects the distance between eyepiece 122 and the user's eye. In this case, it can be determined that the user is looking into the viewfinder (eyepiece 122) when the distance is equal to or less than a predetermined distance. 9 may be started upon detecting that the release button 121 has been pressed down to the first stroke. Alternatively, the processing of the flowchart shown in Fig. 9 may be started when the gaze detection processing is started in advance and the processing is started when the gaze detection processing is successful.

[0138] First, in step S901 of FIG. 9, CPU 212 (second authentication unit 508) determines whether the first authentication is in progress and the user is continuing to capture images (capturing images). Here, whether the first authentication is in progress is determined via authentication status management unit 521 based on whether the first authentication status in authentication status table 570 shown in FIG. 5F is "authenticated." Furthermore, whether the user is continuing to capture images (capturing images) is determined by the eyepiece sensor described above based on whether the user keeps their eye close to the viewfinder (eyepiece 122). Note that whether capture is in progress may also be determined by other methods, such as pressing release button 121 or gaze detection processing.

[0139] In step S901 of FIG. 9, if the CPU 212 (second authentication unit 508) determines that the first authentication is in progress and the user is continuing to take a photograph (taking a photograph) (S901 / YES), the process proceeds to step S902. 9, the eye image acquisition unit 501 acquires an eye image when the user looks through the viewfinder (eyepiece 122). The specific processing in step S902 in FIG. 9 is the same as the processing in step S604 in FIG. 6A (the processing in the flowchart shown in FIG. 7), and therefore a description thereof will be omitted. Note that if the gaze detection processing has already started, the gaze detection in step S701 in FIG. 7 may be skipped, and the result of the gaze detection that has already started may be used.

[0140] 9, the eye image acquisition unit 501 determines whether or not the acquisition of the eye image has been successful. Specifically, the eye image acquisition unit 501 determines whether or not the acquisition of the eye image has been successful based on the flag recorded in step S705 or step S708 in FIG.

[0141] In step S903 of FIG. 9, if the eye image acquisition unit 501 determines that acquisition of the eye image has not been successful (failed) (S903 / NO), the process proceeds to step S904. 9, the CPU 212 (second authentication unit 508) displays on the display device 214 that the eye image capture failed. For example, the CPU 212 (second authentication unit 508) may display on the display device 214 a message such as "Eye image capture failed," or may display an icon or the like indicating the failure. When the processing of step S904 ends, the processing returns to step S901.

[0142] Also, in step S903 of FIG. 9, if the eye image acquisition unit 501 determines that acquisition of the eye image has been successful (S903 / YES), the process proceeds to step S905. When the process proceeds to step S905 in FIG. 9, the CPU 212 (second authentication unit 508) determines whether or not the eye image acquired in step S902 is the eye image of the eye registered as the dominant eye in the registration process.

[0143] In step S905 of FIG. 9, if the CPU 212 (second authentication unit 508) determines that the eye image acquired in S902 is not the eye image of the eye registered as the dominant eye in the registration process (S905 / NO), the process proceeds to step S906. 9, CPU 212 (second authentication unit 508) displays on display device 214 a message that the acquired eye image is not suitable for authentication. For example, CPU 212 (second authentication unit 508) may display a message such as "Eye image is not suitable for authentication" on display device 214. When the processing of step S906 ends, the processing returns to step S901.

[0144] Also, in step S905 of FIG. 9, if the CPU 212 (second authentication unit 508) determines that the eye image acquired in S902 is the eye image of the eye registered as the dominant eye in the registration process (S905 / YES), the process proceeds to step S907. 9, the feature vector calculation unit 502 extracts a feature vector from the eye image as authentication target information for authenticating the user. Specifically, the feature vector calculation unit 502 extracts a feature vector from the eye image acquired in step S902 of FIG.

[0145] Subsequently, in step S908 of FIG. 9, the CPU 212 (second authentication unit 508) acquires all feature vectors in the second authentication registration feature vector table 560 shown in FIG. 5E via the registration data management unit 505.

[0146] 9, the CPU 212 (second authentication unit 508) performs second authentication by comparing the feature vector, which is the authentication target information acquired in S907, with the registered feature vector, which is the authentication registration information acquired in S908. Specifically, the CPU 212 calculates the cosine similarity between the two feature vectors and performs second authentication depending on whether the calculated cosine similarity exceeds a predetermined threshold. More specifically, if the calculated cosine similarity exceeds the predetermined threshold, it is determined that the second authentication has been successful.

[0147] Subsequently, in step S910 of FIG. 9, the CPU 212 (second authentication unit 508) determines whether the second authentication performed in step S909 was successful.

[0148] In step S910 of FIG. 9, if the CPU 212 (second authentication unit 508) determines that the second authentication performed in step S909 was successful (S910 / YES), the process proceeds to step S911. In step S911 of FIG. 9, authentication state management unit 521 updates the second authentication state in authentication state table 570 shown in FIG. 5F to "authenticated."

[0149] Next, in step S912 of FIG. 9, the CPU 212 (second authentication unit 508) causes the authentication status display unit 523 to display on the display device 214 to inform the user that authentication has been successful.

[0150] 9, the CPU 212 (second authentication unit 508) determines whether the user is continuing to take pictures (taking pictures) or not. The method for determining whether the user is continuing to take pictures (taking pictures) or not is the same as in step S901, and therefore the description thereof will be omitted.

[0151] In step S913 of FIG. 9, if the CPU 212 (second authentication unit 508) determines that the user is continuing to take pictures (taking pictures) (S913 / YES), it waits in step S913 until it determines that the user is not continuing to take pictures (not taking pictures).

[0152] Also, in step S913 of FIG. 9, if the CPU 212 (second authentication unit 508) determines that the user is not continuing to take a photograph (not taking a photograph) (S913 / NO), the process proceeds to step S914. In step S914 in FIG. 9, authentication state management unit 521 updates the second authentication state in authentication state table 570 shown in FIG. 5F to "unauthenticated."

[0153] Also, in step S910 of FIG. 9, if the CPU 212 (second authentication unit 508) determines that the second authentication performed in step S909 was not successful (failed) (S910 / NO), the process proceeds to step S915. In step S915 of FIG. 9, authentication state management unit 521 updates the second authentication state in authentication state table 570 shown in FIG. 5F to "unauthenticated."

[0154] 9, the other-person's use detection unit 509 performs processing to detect use by another person. Specifically, the other-person's use detection unit 509 detects whether or not use of the camera 100 by a person (other person) other than the person authenticated in the first authentication is suspected.

[0155] Subsequently, in step S917 of FIG. 9, first authentication status invalidation unit 510 determines whether use by another person has been detected in step S916.

[0156] In step S917 of FIG. 9, if first authentication status invalidation unit 510 determines in step S916 that use by another person has been detected (S917 / YES), the process proceeds to step S918. 9, first authentication status invalidation unit 510 updates the first authentication status in authentication status table 570 shown in FIG. 5F to "unauthenticated" via authentication status management unit 521, and further deletes the person ID. This makes it possible to invalidate the first authentication when use by another person is suspected.

[0157] 9 is completed, the process proceeds to step S919. In addition, in step S917 of FIG. 9, if the first authentication status invalidation unit 510 determines that use by another person has not been detected in step S916 (S917 / NO), the process proceeds to step S919. 9, the CPU 212 (second authentication unit 508) causes the authentication status display unit 523 to display a message informing the user of the authentication failure on the display device 214. When the process of this step S919 ends, the process returns to step S901.

[0158] 9 is completed, the process proceeds to step S920. In addition, in step S901 of Fig. 9, if the CPU 212 (second authentication unit 508) determines that the first authentication is not in progress or that the user is not continuing to take a photograph (not taking a photograph) (S901 / YES), the process proceeds to step S920. 9, the CPU 212 (second authentication unit 508) updates the display on the display device 214 using the authentication status display unit 523. Note that the authentication status display unit 523 does not display the authentication status if, for example, image capture is no longer in progress. Also, for example, if image capture is in progress but the first authentication is "unauthenticated," the authentication status display unit 523 displays a message or an icon indicating that the first authentication has become unauthenticated.

[0159] When the process of step S920 in FIG. 9 is completed, the process of the flowchart in FIG. 9 ends.

[0160] [Detection of unauthorized use] Fig. 10 is a flowchart showing an example of detailed processing procedures for the detection of use by others in step S916 of Fig. 9. The processing of the flowchart shown in Fig. 10 is mainly executed by the CPU 212 by the detection unit 509 for use by others.

[0161] 10, the unauthorized use detection unit 509 records the failure of the second authentication, for example, the time of failure and the similarity score at that time.

[0162] Next, in step S1002 of FIG. 10, the false-use detection unit 509 analyzes the history of second authentication failures recorded in step S1001 to determine whether there is a tendency for false use. Specifically, the false-use detection unit 509 determines that the second authentication has occurred if the number of second authentication failures within a predetermined time range from the most recent time exceeds a threshold. However, the false-use detection unit 509 may count failures that occurred during the same shooting as one failure. Alternatively, the false-use detection unit 509 may count only failures with a similarity below a predetermined threshold. Furthermore, although the false-use detection unit 509 records only failures in step S1001, it may also record successes. For example, the false-use detection unit 509 may record successes of second authentication immediately after step S909 of FIG. 9. Furthermore, the false-use detection unit 509 may not count failures during the same shooting if there has been at least one success during that shooting.

[0163] Next, in step S1003 of FIG. 10, the unauthorized use detection unit 509 determines whether unauthorized use is suspected based on the analysis result in step S1002 from the history of failed second authentication attempts.

[0164] In step S1003 of FIG. 10, if the unauthorized use detection unit 509 determines that unauthorized use is not suspected based on the history of failed second authentication attempts (S1003 / NO), the process proceeds to step S1004. Proceeding to step S1004 in FIG. 10, the false-use detection unit 509 determines whether the maximum similarity obtained during the matching performed in step S909 in FIG. 9 is less than a predetermined threshold. Specifically, in step S909, similarities with multiple registered feature vectors of the same person ID are obtained. In this step, the highest similarity among the obtained similarities is obtained, and it is determined whether this similarity is less than a predetermined threshold. This is because the similarity decreases even for the person himself / herself when the shooting conditions are poor. However, this similarity tends to be higher than in the case of a false person. Therefore, in this step, a threshold is set that clearly determines that the person is a false person, and a false person is determined if the similarity falls below this threshold.

[0165] In step S1004 of FIG. 10, if the third-party use detection unit 509 determines that the maximum similarity obtained during the matching performed in step S909 of FIG. 9 is not less than a predetermined threshold (S1004 / NO), the process proceeds to step S1005. 10, the unauthorized use detection unit 509 determines that the use is not unauthorized and does not record it as unauthorized use (records it as not unauthorized use). Specifically, the unauthorized use detection unit 509 sets a flag indicating unauthorized use in the memory unit 213 and turns off the flag.

[0166] 10, if the false-use detection unit 509 determines that the maximum similarity obtained in the matching performed in step S909 of Fig. 9 is less than a predetermined threshold (S1004 / YES), the process proceeds to step S1006. Also, if the false-use detection unit 509 determines that false-use is suspected based on the history of second authentication failures in step S1003 of Fig. 10 (S1003 / YES), the process proceeds to step S1006. 10, the unauthorized use detection unit 509 determines that the use is by another person and records this as unauthorized use. Specifically, the unauthorized use detection unit 509 sets a flag indicating unauthorized use in the memory unit 213 and turns the flag ON.

[0167] When the process of step S1005 in FIG. 10 is completed, or when the process of step S1006 in FIG. 10 is completed, the process of the flowchart in FIG. 10 is completed.

[0168] [First authentication invalidation process] 11A to 11D are flowcharts showing an example of detailed processing procedures of the first authentication invalidation process in the control method for camera 100, which corresponds to the information processing device according to the first embodiment. The four types of first authentication invalidation processes shown in Fig. 11A to 11D are mainly executed by first authentication status invalidation unit 510 in CPU 212.

[0169] 11A is a flowchart illustrating an example of a detailed procedure of the first authentication invalidation process based on elapsed time. The process of the flowchart illustrated in FIG. 11A is assumed to be periodically started by a timer or the like.

[0170] 11A, first authentication status invalidation unit 510 determines whether the first authentication has been successful or not by checking whether the first authentication status in authentication status table 570 shown in FIG. 5F, which is managed by authentication status management unit 521, is "authenticated."

[0171] In step S1101 of FIG. 11A, if first authentication status invalidation section 510 determines that the first authentication has been successful (S1101 / YES), the process proceeds to step S1102. 11A, the first authentication status invalidation unit 510 calculates the elapsed time since the first authentication was successful. In this embodiment, the first authentication status invalidation unit 510 calculates the elapsed time since the first authentication status in the authentication status table 570 shown in FIG. 5F became "authenticating." Here, the first authentication status invalidation unit 510 records the time when the first authentication status in the authentication status table 570 shown in FIG. 5F was changed to "authenticating," and calculates the difference between that time and the current time, thereby calculating the elapsed time since the first authentication was successful.

[0172] 11A, the first authentication status invalidation unit 510 detects the execution of the second authentication by the second authentication unit 508. In this embodiment, the first authentication status invalidation unit 510 detects whether the second authentication has been executed since the previous execution of this process by the timer until the execution of this process.

[0173] Subsequently, in step S1104 of FIG. 11A, first authentication status invalidation unit 510 determines whether the second authentication whose execution was detected in step S1103 was successful or not.

[0174] In step S1104 of FIG. 11A, if first authentication status invalidation unit 510 determines that the second authentication detected to be executed in step S1103 was successful (S1104 / YES), the process proceeds to step S1105. 11A, first authentication status invalidation unit 510 adds a predetermined time to the lifetime. Note that the initial value of the lifetime is assumed to be initialized to the predetermined value of the lifetime when the first authentication status in authentication status table 570 shown in FIG. 5F is changed to "authenticated."

[0175] In step S1104 of FIG. 11A, if the first authentication status invalidation unit 510 determines that the second authentication detected to be executed in step S1103 was not successful (failed) (S1104 / NO), the process proceeds to step S1106. Proceeding to step S1106 in FIG. 11A, first authentication status invalidation unit 510 subtracts a predetermined time from the above-mentioned lifetime.

[0176] When the process of step S1105 in FIG. 11A is completed, or when the process of step S1106 in FIG. 11A is completed, the process proceeds to step S1107. In step S1107 of FIG. 11A, first authentication status invalidation unit 510 determines whether the elapsed time calculated in step S1102 has passed the currently acquired lifetime.

[0177] In step S1107 of FIG. 11A, if first authentication status invalidation unit 510 determines that the elapsed time calculated in step S1102 has exceeded the currently acquired lifetime (S1107 / YES), the process proceeds to step S1108. When the process proceeds to step S1108 in FIG. 11A, the first authentication status invalidation unit 510 updates the first authentication status and the second authentication status in the authentication status table 570 shown in FIG. 5F to "unauthenticated" via the authentication status management unit 521, and further deletes the person ID.

[0178] When the process of step S1108 in Fig. 11A is completed, the process of the flowchart in Fig. 11A ends. Also, when first authentication status invalidation unit 510 determines that the first authentication is not successful (failed) in step S1101 in Fig. 11A (S1101 / NO), the process of the flowchart in Fig. 11A ends.

[0179] 11B is a flowchart showing an example of detailed processing steps of the first authentication invalidation processing based on a change in the power supply state. The processing of the flowchart shown in FIG. 11B is assumed to be started when the power supply state changes. Here, a change in the power supply state refers to, for example, when the power is turned on, when the power is turned off, when the device enters sleep mode, when the device returns from sleep mode, etc.

[0180] First, in step S1111 of Fig. 11B, first authentication status invalidation unit 510 determines whether the first authentication has been successful or not by checking whether the first authentication status in authentication status table 570 shown in Fig. 5F, which is managed by authentication status management unit 521, is "authenticated."

[0181] In step S1111 of FIG. 11B, if first authentication status invalidation section 510 determines that the first authentication has been successful (S1111 / YES), the process proceeds to step S1112. 11B, first authentication status invalidation unit 510 determines whether the power state has changed. In step S1112, if there has been at least one change from the power state from ON to OFF or from OFF to ON, first authentication status invalidation unit 510 determines that the power state has changed.

[0182] In step S1112 of FIG. 11B, if first authentication status invalidation section 510 determines that the power supply status has not changed (S1112 / NO), the process proceeds to step S1113. 11B, first authentication status invalidation unit 510 determines whether the sleep state has changed. In step S1113, it determines that the sleep state has changed if at least one of the following events has occurred: a transition to sleep mode and a return from sleep mode.

[0183] 11B, if first authentication status invalidation unit 510 determines that the sleep status has changed (S1113 / YES), the process proceeds to step S1114. Also, if first authentication status invalidation unit 510 determines that the power status has changed (S1112 / YES), the process proceeds to step S1114. When the process proceeds to step S1114 in FIG. 11B, the first authentication status invalidation unit 510 updates the first authentication status and the second authentication status in the authentication status table 570 shown in FIG. 5F to "unauthenticated" via the authentication status management unit 521, and further deletes the person ID.

[0184] When the process of step S1114 in Fig. 11B is completed, the process of the flowchart in Fig. 11B ends. Furthermore, when first authentication status invalidation unit 510 determines that the sleep status has not changed in step S1113 in Fig. 11B (S1113 / NO), the process of the flowchart in Fig. 11B ends. Furthermore, when first authentication status invalidation unit 510 determines that the first authentication has not succeeded (has failed) in step S1111 in Fig. 11B (S1111 / NO), the process of the flowchart in Fig. 11B ends.

[0185] 11C is a flowchart showing an example of detailed processing steps of the first authentication deactivation processing based on the distance and connection status of the device. The processing of the flowchart shown in FIG. 11C is assumed to be periodically started by a timer or the like. The processing of the flowchart shown in FIG. 11C also assumes that a communication connection process has been executed in advance between the device carried by the user and the camera 100. For example, one example is to perform a pairing process to connect the camera 100 to a smartphone carried by the user via Bluetooth.

[0186] First, in step S1121 of Fig. 11C, first authentication status invalidation unit 510 determines whether the first authentication has been successful or not by checking whether the first authentication status in authentication status table 570 shown in Fig. 5F, which is managed by authentication status management unit 521, is "authenticated."

[0187] In step S1121 of FIG. 11C, if first authentication status invalidation section 510 determines that the first authentication has been successful (S1121 / YES), the process proceeds to step S1122. 11C, first authentication status invalidation unit 510 determines whether the connection status has deteriorated to a level equal to or greater than a predetermined condition, for example, when the radio wave strength of the communication connection falls below a certain level.

[0188] In step S1122 of FIG. 11C, if first authentication status invalidation unit 510 determines that the connection status has deteriorated to a level equal to or greater than a predetermined condition (S1122 / YES), the process proceeds to step S1123. When the process proceeds to step S1123 in FIG. 11C, the first authentication status invalidation unit 510 updates the first authentication status and the second authentication status in the authentication status table 570 shown in FIG. 5F to "unauthenticated" via the authentication status management unit 521, and further deletes the person ID.

[0189] When the process of step S1123 in Fig. 11C is completed, the process of the flowchart in Fig. 11C ends. Furthermore, when first authentication status invalidation unit 510 determines in step S1122 in Fig. 11C that the connection status has not deteriorated to a level equal to or greater than a predetermined condition (S1122 / NO), the process of the flowchart in Fig. 11C ends. Furthermore, when first authentication status invalidation unit 510 determines in step S1121 in Fig. 11C that the first authentication has not succeeded (failed) (S1121 / NO), the process of the flowchart in Fig. 11C ends.

[0190] 11D is a flowchart showing an example of detailed processing steps of the first authentication deactivation process based on an explicit deactivation operation input by a user. The processing of the flowchart shown in FIG. 11D is always executed while the power of the camera 100 is turned on.

[0191] First, in step S1131 in Fig. 11D, first authentication status invalidation unit 510 determines whether the first authentication has been successful or not by checking whether the first authentication status in authentication status table 570 shown in Fig. 5F, which is managed by authentication status management unit 521, is "authenticated."

[0192] In step S1131 of FIG. 11D, if first authentication status invalidation section 510 determines that the first authentication has been successful (S1131 / YES), the process proceeds to step S1132. 11D, first authentication status invalidation unit 510 waits for the user to perform the invalidation operation. The invalidation operation in this step is performed by a switch button (not shown) installed on camera 100 and the user pressing the switch button. Note that the invalidation operation may be selected from a menu displayed on a touch panel or the like.

[0193] Subsequently, in step S1133 in FIG. 11D, first authentication status invalidation unit 510 determines whether or not an invalidation operation by the user has been detected.

[0194] In step S1133 of FIG. 11D, if first authentication state invalidation unit 510 determines that an invalidation operation by the user has been detected (S1133 / YES), the process proceeds to step S1134. Proceeding to step S1134 in FIG. 11D, the first authentication status invalidation unit 510 updates the first authentication status and the second authentication status in the authentication status table 570 shown in FIG. 5F to "unauthenticated" via the authentication status management unit 521, and further deletes the person ID.

[0195] When the process of step S1134 in Fig. 11D is completed, the process of the flowchart in Fig. 11D ends. Furthermore, when first authentication status invalidation unit 510 determines in step S1133 in Fig. 11D that it has not detected an invalidation operation by the user (S1133 / NO), the process of the flowchart in Fig. 11D ends. Furthermore, when first authentication status invalidation unit 510 determines in step S1131 in Fig. 11D that the first authentication has not been successful (has failed) (S1131 / NO), the process of the flowchart in Fig. 11D ends.

[0196] [Authentication state saving process] Fig. 12A is a flowchart showing an example of detailed processing steps of authentication status saving processing in a control method for camera 100, which corresponds to an information processing device according to the first embodiment. The processing of the flowchart shown in Fig. 12A is mainly executed by CPU 212 via authentication status saving unit 522. Fig. 12B shows an example of the configuration of an image file 1200 saved by authentication status saving unit 522 according to the first embodiment.

[0197] The processing of the flowchart shown in Fig. 12A is processing for generating and saving an image file 1200 shown in Fig. 12B. Specifically, the processing of the flowchart shown in Fig. 12A is processing for saving image data 1220 of a subject image taken by a user, together with photographer information 1211, hash value 1212, and digital signature 1213, in association with each other as metadata 1210. The processing of the flowchart shown in Fig. 12A is also executed when release button 121 is pressed down to the second stroke.

[0198] 12A, the authentication status storage unit 522 captures an image of the subject via the imaging unit 511. Specifically, for example, the imaging unit 511 converts light received by the imaging element 211 into an electrical signal and performs imaging processing of the subject.

[0199] 12A, the authentication status saving unit 522 generates image data 1220 of the subject image via the imaging unit 511. Specifically, for example, the imaging unit 511 performs image processing such as development processing and encoding processing on the electrical signal obtained by the imaging processing in step S1201, and generates image data 1220 of the subject image.

[0200] Next, in step S1203 of FIG. 12A, the authentication status saving unit 522 generates photographer information 1211 of the user who captured the image data 1220 of the subject image. Specifically, the authentication status saving unit 522 acquires the authentication status table 570 shown in FIG. 5F. Furthermore, the authentication status saving unit 522 acquires personal information corresponding to the person ID via the registered data management unit 505. In this embodiment, "name" is acquired as the personal information. Then, the authentication status saving unit 522 generates photographer information 1211 including the user's name, first authentication status, second authentication status, and information on whether or not the image has been used by another person, from information included in the authentication status table 570, etc.

[0201] Next, in step S1204 of FIG. 12A, the authentication status storage unit 522 executes a hash function on the binary data of the image data 1220 of the subject image and the photographer information 1211, respectively, to generate a hash value 1212.

[0202] Next, in step S1205 of FIG. 12A, the authentication status storage unit 522 generates a digital signature 1213. The digital signature 1213 includes information indicating the signature value, the signer, and the date and time of signing. The signature value is generated by encrypting the hash value 1212 generated in step S1204 using a private key prepared in advance. The public key that pairs with the private key used here is also stored in the digital signature 1213. In this embodiment, information indicating the manufacturer of the camera 100 is stored as the signer. Note that the model of the camera 100 may be used instead of the manufacturer as the signer. Furthermore, the date and time when the generation of the digital signature is completed is stored as the date and time of signing.

[0203] 12A, the authentication status saving unit 522 adds the photographer information 1211, the hash value 1212, and the digital signature 1213 to the image data 1220 of the subject image as metadata 1210, and generates the image file 1200. Here, if the image data 1220 is a still image, the image file is generated in accordance with the JPEG format, and if the image data 1220 is a moving image, the image file is generated in accordance with the MPEG format.

[0204] 12A, authentication status saving unit 522 stores image file 1200 generated in step S1206 in memory unit 213. Note that memory unit 213 also includes a storage medium that is detachable from camera 100, and in this case, image file 1200 may be stored in the storage medium, for example. When the processing of step S1207 ends, the processing of the flowchart in FIG. 12A ends.

[0205] The verification method described below can be used to confirm that the image file 1200 has not been tampered with. First, the hash value 1212 is restored using the signature value with a public key. Then, the hash values ​​of the image data and the photographer information are calculated again. If the restored hash value matches the newly calculated hash value, it can be determined that the image file 1200 has not been tampered with. Conversely, if the restored hash value does not match the newly calculated hash value, it can be determined that the image file 1200 has been tampered with. This is because, even if someone were to tamper with the image data 1220, the person who tampered with the image data 1220 would not be able to change the signature value because the signature value is encrypted with a private key. Therefore, if the image data 1220 has been tampered with, the hash value calculated from the image data 1220 will not match the restored hash value. This makes it possible to detect data tampering.

[0206] 12B, the hash value 1212 is also included and stored in the image file 1200. However, since the hash value 1212 can be recalculated from the image data 1220 and photographer information 1211 included in the image file 1200, the hash value 1212 may not be included in the image file 1200.

[0207] In the case of a moving image, shooting of the moving image begins when release button 121 is pressed down to the second stroke, and shooting of the moving image is completed when it is pressed down to the second stroke again. Then, moving image data is generated by the processing of steps S1201 to S1202 in Fig. 12A. Then, a hash value or the like of the moving image data is calculated, converted into a hash value of image data and saved, and the moving image data 1220 and metadata 1210 are combined and saved as a moving image file 1200.

[0208] [Gaze detection processing] Fig. 13 shows the first embodiment and is a diagram for explaining the principle of the user's gaze detection process. In Fig. 13, the same components as those shown in Fig. 2 and Fig. 3 are given the same reference numerals, and detailed description thereof will be omitted. Fig. 13 also shows an XYZ coordinate system corresponding to the XYZ coordinate system shown in Fig. 2.

[0209] 13, light sources 216a and 216b are disposed approximately symmetrically with respect to the optical axis of light receiving lens 218, and illuminate user's eye E. A portion of the light emitted from light sources 216a and 216b and reflected by user's eye E is collected by light receiving lens 218 onto eye imaging element 219. Also, in FIG. 13, a cornea 1310, a pupil 1320, and an iris 1330 are illustrated for user's eye E.

[0210] Fig. 14 shows the first embodiment and is a diagram for explaining the gaze detection process of a user. Specifically, Fig. 14(a) is a schematic diagram of an eye image captured by the eye imaging element 219 (eye optical image projected onto the eye imaging element 219). In Fig. 14(a), the same components as those shown in Fig. 13 are assigned the same reference numerals. Fig. 14(b) is a diagram showing the output intensity of the eye imaging element 219 as luminance. Furthermore, Fig. 15 is a flowchart showing an example of a detailed processing procedure of the gaze detection process in the control method of the camera 100 corresponding to the information processing device according to the first embodiment.

[0211] 15, the CPU 212 controls the driving of the light sources 216a and 216b via the light source drive circuit 305 so as to emit infrared light toward the user's eye E. An optical image of the user's eye illuminated by the infrared light passes through the light receiving lens 218 and is formed on the eye image sensor 219, which then performs photoelectric conversion. As a result, an electrical signal of the eye image that can be processed is obtained.

[0212] 15, the CPU 212 acquires an eye image (eye image signal; electric signal of the eye image) from the eye imaging element 219 via the line of sight detection circuit 301.

[0213] Subsequently, by the processes of steps S1503 and S1504 in FIG. 15, the CPU 212 acquires eye information relating to the position of the eye E relative to the finder from the eye image obtained in step S1502.

[0214] Specifically, in step S1503 of FIG. 15, the CPU 212 detects the corneal reflection images Pd and Pe of the light sources 216a and 216b and the coordinates of the point corresponding to the pupil center c from the eye image obtained in step S1502.

[0215] In FIG. 13, infrared light emitted from light sources 216a and 216b illuminates a cornea 1310 of a user's eye E. At this time, corneal reflection images Pd and Pe formed by a portion of the infrared light reflected from the surface of the cornea 1310 are collected by a light receiving lens 218 and focused on an eye imaging element 219, becoming corneal reflection images Pd' and Pe', respectively, in the eye image. Similarly, light beams from edges a and b of a pupil 1320 are focused on the eye imaging element 219, becoming pupil edge images a' and b', respectively, in the eye image. FIG. 14(b) is a diagram showing luminance information (luminance distribution) of an area 1400 in the eye image of FIG. 14(a). In FIG. 14(b), the horizontal direction of the eye image is the X-axis direction and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In the first embodiment, the coordinates of the corneal reflection images Pd' and Pe' in the X-axis direction (horizontal direction) are Xd and Xe, respectively, and the coordinates of the pupil edge images a' and b' in the X-axis direction are Xa and Xb, respectively. As shown in FIG. 14(b), an extremely high level of luminance is obtained at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. In the region from coordinate Xa to coordinate Xb, which corresponds to the region of the pupil 1320 (the region of the pupil image obtained by focusing the light beam from the pupil 1320 on the eye imaging element 219), an extremely low level of luminance is obtained except for coordinates Xd and Xe. In the region of the iris 1330 outside the pupil 1320 (the region of the iris image outside the pupil image obtained by focusing the light beam from the iris 1330), a luminance intermediate between the two types of luminance described above is obtained. For example, a luminance intermediate between the two types of luminance described above can be obtained in an area where the X coordinate (coordinate in the X-axis direction) is greater than coordinate Xa and an area where the X coordinate is less than coordinate Xb. From the luminance distribution shown in FIG. 14(b), the coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the coordinates Xa and Xb of the pupil edge images a' and b' can be obtained. For example, coordinates of extremely high luminance can be obtained as the coordinates of the corneal reflection images Pd' and Pe', and coordinates of extremely low luminance can be obtained as the coordinates of the pupil edge images a' and b'. In FIG. 13, when the rotation angle θx of the optical axis of the eye E relative to the optical axis of the light receiving lens 218 is small, the coordinate Xc of the pupil center image c' (center of the pupil image) obtained when the light beam from the pupil center c is focused on the eye imaging element 219 can be expressed as Xc ≒ (Xa + Xb) / 2.That is, the coordinate Xc of the pupil center image c' can be calculated from the coordinates Xa and Xb of the pupil edge images a' and b'. In this way, the CPU 212 can estimate the coordinates of the corneal reflection images Pd' and Pe' and the coordinate of the pupil center image c'.

[0216] 15, CPU 212 calculates the imaging magnification β of the eye image. The imaging magnification β is determined by the position of eye E relative to light receiving lens 218, and can be calculated using a function of the distance ΔP=Xe−Xd between corneal reflection images Pd′ and Pe′.

[0217] 15, CPU 212 calculates the rotation angle of the optical axis of eye E relative to the optical axis of light receiving lens 218. The X coordinate of the midpoint between corneal reflection images Pd and Pe and the X coordinate of the center of curvature O of cornea 1310 approximately coincide. Therefore, if the standard distance from center of curvature O of cornea 1310 to center c of pupil 1320 is Oc, then rotation angle θx of eye E in the ZX plane (plane perpendicular to the Y axis) can be calculated using equation (1) below. Rotation angle θy of eye E in the ZY plane (plane perpendicular to the X axis) can also be calculated using a method similar to the method for calculating rotation angle θx described here. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc ···(1)

[0218] 15, the CPU 212 reads the line-of-sight correction parameters stored in the memory unit 213. Specifically, these are the parameters Ax, Bx, Ay, and By of equations (2) and (3) used in step S1507 of FIG.

[0219] 15, CPU 212 uses the rotation angles θx and θy calculated in step S1505 to estimate the coordinates (Hx, Hy) of the user's viewpoint on the screen of display device 214. Here, if the coordinates (Hx, Hy) of the viewpoint correspond to the pupil center c, the coordinates (Hx, Hy) of the viewpoint can be calculated by the following equations (2) and (3). Hx = m × (Ax × θx + Bx) (2) Hy=m×(Ay×θy+By) (3) The parameter m in equations (2) and (3) is a constant determined by the configuration of the optical system for performing gaze detection, and is a conversion coefficient for converting the rotation angles θx and θy into coordinates corresponding to the pupil center c on the screen of the display device 214. This parameter m is determined in advance and stored in the memory unit 213. Furthermore, the parameters Ax, Bx, Ay, and By in equations (2) and (3) are the gaze correction parameters in step S1506 described above.

[0220] Here, the line of sight correction parameters will be explained. The gaze point may not be estimated with high accuracy due to factors such as individual differences in the shape of the human eye E. Specifically, as shown in FIG. 4(b), a discrepancy occurs between the actual gaze point B (410B) and the estimated gaze point C (410C). In FIG. 4(b), the user is gazing at a person, but the camera 100 erroneously estimates that the user is gazing at the background, resulting in an inability to perform appropriate focus detection and adjustment. Therefore, the gaze correction parameter is a parameter for correcting such a discrepancy. This gaze correction parameter can be obtained by calibration of the gaze detection. In this case, calibration is performed, for example, by highlighting multiple indices 411 to 415 at different positions on the screen of the display device 214 as shown in FIG. 4(c) and having the user look at the indices. A gaze detection operation is performed when the user gazes at each indice, and gaze correction parameters appropriate for the user are obtained from the calculated multiple gaze points (estimated positions) and the coordinates of the multiple indices. As long as the position where the user should look is indicated, the method of displaying the indicator is not particularly limited; the indicator may be displayed as a graphic, or the indicator may be displayed by changing at least one of the brightness and color of the image (e.g., a captured image).

[0221] When the process of step S1507 in FIG. 15 is completed, the process of the flowchart in FIG. 15 is completed.

[0222] [Left and right eye determination processing] Next, left / right eye determination processing by the left / right eye determination unit 503 will be described. The left / right eye determination process is a process for determining which of the left and right eyes an eye image was acquired from. As described above, in the gaze detection process used in this embodiment, a deviation occurs between the actual viewpoint and the estimated viewpoint. One factor for this deviation is that the fovea of ​​eye E is not located on the visual axis. Here, the fovea is the central area of ​​the macula of the retina of eye E. The fovea is located 4 to 8 degrees offset from the visual axis toward the ear. Due to this deviation, the estimated viewpoint deviates from the actual viewpoint toward the nose. In this embodiment, the left / right eye determination unit 503 performs the left / right eye determination process based on this deviation. If the estimated viewpoint is offset to the left from the actual viewpoint as seen by the user, it can be estimated that the eye from which the eye image was acquired is the right eye. Conversely, if it is offset to the right, it can be estimated that the eye from which the eye image was acquired is the left eye. Note that the specific method for the left / right eye determination process is not limited to the form described here. For example, a neural network can be configured in which eye images acquired from the left and right eyes of a user are used as learning data, the eye images are used as input, and a determination result as to which eye, left or right, the eye image was acquired from is output. By using such a neural network, it may be possible to determine which eye, left or right, the eye image was acquired from.

[0223] [Effects of this embodiment] To ensure that an image or video was taken by a specific person, authentication must be performed at the time of capture. However, to ensure that the image was not taken by someone else (that the person in question took the photo), authentication must be performed under settings that result in a low false acceptance rate. In this case, such settings generally increase the false rejection rate. As a result, even if the person in question took the photo, authentication will fail at the time of capture. In the use case of photography, there is no second opportunity to take the same photo. For example, for a professional photographer, the decisive moment in a sports or news scoop is only a split second. Not being able to authenticate at that moment is a major problem. In other words, it becomes impossible to guarantee that the user (photographer) who captured the decisive moment is the photographer himself.

[0224] Therefore, in this embodiment, authentication is performed using multiple pieces of biometric information (both eyes) with a low false acceptance rate set by the first authentication when not photographing (before photographing). By performing authentication using multiple pieces of biometric information (both eyes), the false acceptance rate can be further reduced. In addition, authentication is performed with a low false rejection rate set by the second authentication when photographing (during photographing). This reduces the possibility of false rejection during the second authentication, while also reducing the possibility of false acceptance during the first authentication.

[0225] Note that, since a low false acceptance rate is set for the first authentication, the false rejection rate is high. Therefore, even if the user is the correct person, the first authentication may be rejected. However, since the image is not captured, authentication can be attempted again. Therefore, there is no problem with use. Furthermore, the possibility of false acceptance increases with the second authentication. In response to this issue, this embodiment prevents the possibility of false acceptance by invalidating the first authentication status under various conditions. In other words, if use by a false person is suspected during the second authentication, the first authentication is invalidated. Alternatively, the first authentication is invalidated based on the elapsed time since the first authentication was successful, a change in the power status, a change in the distance or connection status with a peripheral device, or an explicit invalidation operation by the user. This reduces the possibility of use by a false person, thereby suppressing the occurrence of false acceptance through the second authentication.

[0226] Additionally, in this embodiment, the results of the first authentication and the second authentication are recorded separately as photographer information 1211 in the metadata 1210 of the image file 1200. As a result, even if only the first authentication is successful and the second authentication fails, the success of the first authentication is recorded in the metadata 1210. In addition, if the second authentication is also successful, the success of both is recorded in the image file 1200 in a form that indicates the success of both. Therefore, the more successful the authentications, the more likely it is that the user (photographer) took the photos. In addition, whether or not the image has been used by another person is recorded as photographer information 1211 in the metadata 1210 of the image file 1200. As a result, if only the first authentication is successful and the second authentication fails, it can be determined whether or not the image has also been used by another person. Therefore, this also increases the likelihood that the image was taken by the user (photographer).

[0227] The camera 100 according to the first embodiment described above is an information processing device that authenticates users. The camera 100 according to the first embodiment includes a registration data management unit 505 that manages authentication registration information of users who are permitted to use the camera 100. The camera 100 according to the first embodiment also includes a first authentication unit 507 that authenticates the user to be authenticated using multiple pieces of authentication target information acquired from multiple different body parts of the user to be authenticated and the authentication registration information managed by the registration data management unit 505. The camera 100 according to the first embodiment also includes a second authentication unit 508 that, after the first authentication by the first authentication unit 507 is successful, performs a second authentication of the user to be authenticated using authentication target information acquired from one of the multiple body parts of the user to be authenticated. According to this configuration, it is possible to improve the accuracy of authenticating the user of the camera 100 (information processing device), and to prevent a decrease in usability.

[0228] <Modification of the first embodiment> In the first embodiment described above, only "name" was used as the personal information managed by the registration data management unit 505. However, information other than name may be used. For example, in the case of a camera 100 used within a company, an "employee number" assigned to an employee may be stored. Alternatively, account information such as an account name for another Web service may be entered. In this case, when the Web service is accessed and login is successful, this information may be used as personal information. Furthermore, a token may be issued from the Web service upon successful access, and the token may also be stored as personal information. Furthermore, this personal information may be stored in the authentication status storage unit 522 as photographer information 1211 in the metadata 1210 of the image file 1200.

[0229] In the first embodiment described above, the first authentication unit 507 and the second authentication unit 508 use the same feature vector calculation unit 502. However, the first authentication unit 507 and the second authentication unit 508 tend to obtain different eye images. Specifically, the first authentication unit 507 captures eye images with the user actively intending to be authenticated, so it targets eye images under conditions such as wide-open eyes. On the other hand, the second authentication unit 508 attempts to authenticate by capturing eye images of the user while they are being photographed, so it is considered that a wide variety of eye images, such as images at various gaze angles, are targeted. Therefore, the neural network model used in the first authentication unit 507 is a model trained on eye images intended for the first authentication. On the other hand, the neural network model used in the second authentication unit 508 is a model trained on eye images intended for the second authentication. This can further improve authentication accuracy. Note that a method other than changing the threshold or model may be used to realize the use of an authentication method with a low false acceptance rate for the first authentication and an authentication method with a low false rejection rate for the second authentication. For example, as described in Patent Document 3, there is a method for improving performance by changing the calculation method of the feature vector used at the time of registration and at the time of matching, and these authentication methods may be used.

[0230] In the first embodiment described above, only one user is registered. Therefore, when a different person registers, all data held by the registration data management unit 505 is erased and registration is restarted. That is, when the registration process is initiated, the data in the registration data management unit 505 is deleted. Alternatively, instead of deleting the data, an invalid flag is set so that the data is not used in subsequent first or second authentications. However, multiple users may be registered for the camera 100. In this case, each time the registration process is initiated, registration information (personal information and feature vector) for a different person ID is registered in the registration data management unit 505. Of course, a process for preventing registration by the same person may be added. For example, if the same name is found, a message indicating that the name has already been registered may be displayed and the process may end. Then, during the first authentication of the first authentication process, it is determined which person ID the user will use. At this time, if there are multiple person IDs exceeding a predetermined threshold, the most similar person ID may be authenticated. Alternatively, the first authentication may end as a failure. In addition, the registered feature vectors used in the second authentication and second authentication processes of the first authentication process may be limited to the feature vectors of the corresponding person ID. That is, in S822 or S823 of FIG. 8B showing the first authentication process, only the record of the person ID identified in step S810 of FIG. 8A is extracted from the first authentication registration right eye feature vector table 540 or the first authentication registration left eye feature vector table 550. Also, in S908 of FIG. 9 showing the second authentication process, only the record of the identified person ID is extracted from the second authentication registration feature vector table 560. Then, authentication is performed using only the feature vectors extracted in the first authentication process and the second authentication process. This reduces the number of feature vectors compared in the second authentication and second authentication processes of the first authentication process. This improves authentication accuracy. This is because the problem can be simplified from 1:N matching (identification) to 1:1 matching (verification). Furthermore, the number of feature extraction vectors to be matched is reduced, thereby shortening processing time.

[0231] In the first embodiment described above, the enrollment data management unit 505 stores the first authentication enrollment right eye feature vector table 540, the first authentication enrollment left eye feature vector table 550, and the second authentication enrollment feature vector table 560 as separate tables. However, the "feature vector 1r" is duplicated, resulting in inefficiency. Therefore, these may be integrated into a single table. In this case, each record may have information indicating whether it is used for the first authentication or the second authentication and information indicating which eye (left or right) it was acquired from. Then, the enrollment feature vector to be used in the first authentication process and the second authentication process may be selected. Furthermore, in the first embodiment described above, different enrollment feature vectors are used in the first authentication and the second authentication. However, the same feature vector may be used. In this case, the enrollment data management unit 505 does not need to store the first authentication enrollment right eye feature vector table 540, the first authentication enrollment left eye feature vector table 550, and the second authentication enrollment feature vector table 560; instead, the feature vectors may be managed as a single table.

[0232] In the first embodiment described above, the authentication result is displayed on the display device 214 by the authentication status display unit 523 during the second authentication process (S912, S919, and S920 in FIG. 9). However, when capturing an image, the image captured by the image sensor 211 is already displayed on the display device 214. Therefore, it is considered that the user wants to concentrate on capturing an image. Therefore, it is preferable to display the result so as not to interfere with capturing an image. Therefore, the authentication status display unit 523 may be modified as follows. For example, a display indicating success or failure may be displayed on the edge of the screen of the display device 214. Alternatively, since the user's gaze position on the display device 214 is obtained in S1507 of FIG. 15, which shows the gaze detection process, the display may be determined based on the gaze position. For example, the display may be located far from the gaze position. However, since it is difficult to understand if the display position changes frequently, the display position may be determined in advance, such as near the four corners, and the display position may be determined based on the gaze position. In other words, an icon or the like may be displayed in a predetermined position that is farthest from the gaze position. Also, the display in S904 and S906 in FIG. 9 may be omitted, and in that case, S904 and S906 may be omitted from FIG.

[0233] In the first embodiment described above, during the second authentication process, only the authentication status of the second authentication is displayed as a method of displaying the authentication status in S912 and S919 of FIG. 9 . However, the authentication status of the first authentication may also be displayed at the same time. Similarly, the authentication status of both the first authentication and the second authentication may also be displayed in step S920 of FIG. 9 . Furthermore, although these displays are not displayed when the second authentication process starts, the authentication status may be displayed on the display device 214 when the second authentication process starts, that is, when the user looks into the viewfinder, for example. Note that the display of these authentication statuses may be configured to be performed by the authentication status display unit 523.

[0234] In the first embodiment described above, in the authentication state storage process of the authentication state storage unit 522, all of the contents of the authentication state table 570 managed by the authentication state management unit 521 are stored as metadata 1210. However, it is not necessary to store all of the contents, or the contents may be processed before storage. For example, only "Name" may be stored, and "Name" may be stored only when both the first authentication and the second authentication are "Authenticating." Otherwise, a value such as "Unknown" may be stored for "Name." Alternatively, although the first authentication state and the second authentication state are separated, this may be combined into a single "Authentication State" item, and "Authenticating" may be stored only when both the first authentication and the second authentication are "Authenticating," and "Not Authenticated" may be stored otherwise.

[0235] In the first embodiment described above, both the first authentication unit 507 and the second authentication unit 508 use personal authentication using eye images. However, other authentication methods may be used. For example, other biometric authentication methods, such as fingerprint authentication, may be used for the first authentication. In the case of fingerprint authentication, a fingerprint sensor may be installed on the release button 121, and fingerprint authentication may be performed when the user places their finger on the release button. Furthermore, multi-stage biometric authentication using multiple pieces of biometric information may be performed for the first authentication. However, in this case, the biometric information used for the first authentication and the second authentication is limited to information that can be determined to have been acquired from the same person or that has been manually certified to have been acquired from the same person. Here, as an example of information that can be determined to have been acquired from the same person, for example, fingerprints acquired from different fingers are related as long as they are acquired from the same person. Therefore, a neural network trained to determine whether fingerprint data acquired from different fingers is used as training data. This makes it possible to determine whether the fingerprint data used for the first authentication and the second authentication were acquired from the same person. Once this determination is made, fingerprint authentication may be performed for both the first and second authentications. As another example of a method for determining whether the images are acquired from the same person, facial authentication may be performed for the first authentication, and personal authentication using an eye image may be performed for the second authentication. In this case, a neural network trained to be able to make a determination based on the identity between the eye region included in the facial image used for the first authentication and the eye image used for the second authentication is used. However, since the second authentication is performed at the time of shooting, it is desirable to use a method that can authenticate at the time of shooting. For example, in the case of fingerprint authentication, a fingerprint sensor may be installed on the release button 121, and fingerprint authentication may be performed at the time of shooting, which can be used for the second authentication. Also, sometimes an image captured by the image sensor 211 is displayed on the touch panel (operation member 123) and photographed without looking through the viewfinder. In such a photographing mode, facial authentication can also be used for the second authentication. Furthermore, in order to control the authentication state of the second authentication to continue only during shooting, when facial authentication is used, it may be determined that shooting is in progress while the face is being photographed.Alternatively, in the case of fingerprint authentication, it may be determined that photography is in progress while a finger is placed on the release button 121.

[0236] Furthermore, when the first authentication is successful, it is possible to retain and use the features of the eye image taken at or before and after the first authentication. During the second authentication, authentication may be performed by confirming the identity of the features with those of the successful first authentication (verification processing). This method has the advantage of reducing the effort required for registering an image for the second authentication. It is also effective when the second authentication is required in a situation where the environment is significantly different from that of the registration. Furthermore, it is possible to increase reliability by combining the above-described identity confirmation and the conventional verification. That is, it is possible to determine that the second authentication is successful if verification by either method is successful (or if verification by both methods is successful). Furthermore, another application of this method is possible, as follows: During the first authentication, a PIN is entered and fingerprint authentication is performed using the release button 121. If the first authentication is successful, a facial image of the user is simultaneously captured with the in-camera, converted into features, and stored. During the second authentication, the identity of the features of the facial image and the facial features of the photographer is confirmed. As described above, various forms of two-step authentication can be achieved by applying the first embodiment.

[0237] In the first embodiment described above, the first authentication status invalidation unit 510 invalidates the first authentication status when various conditions are met. However, when the first authentication status is invalidated, the user can re-execute the first authentication to validate it. Therefore, it is possible for the first authentication to be repeatedly successful and invalidated for the purpose of fraudulent use. Therefore, a first authentication restriction unit (not shown) may be provided. The first authentication restriction unit detects suspicion of fraudulent use and temporarily restricts the first authentication from being performed. In this case, fraud is detected when the first authentication unit 507 repeatedly performs the first authentication and the first authentication status invalidation unit 510 invalidates the first authentication within a predetermined period. Alternatively, while the first authentication status is invalidated based on the various conditions described above, the invalidation may be limited to some of these conditions. For example, the invalidation of the first authentication status by the first authentication status invalidation unit 510 may be limited to when the false use detection unit 509 suspects false use. In other words, fraud is detected when invalidation due to detection of unauthorized use and successful first authentication by the first authentication unit 507 occur repeatedly within a predetermined period of time. Another possible method for restricting the implementation of first authentication is to disable user operations via a menu or the like. That is, if a user operates the camera 100 to invoke first authentication processing from a menu or the like, the menu that invokes the first authentication processing can be invalidated. The restricted state may then be lifted after a certain period of time has passed. This makes it more difficult for a registered individual to intentionally lend the camera 100 to another person to take photos. In particular, if a method other than biometric authentication is used for first authentication, first authentication can be performed even when the registered individual is not present. For example, first authentication can be performed by sharing a password or PIN, or by lending a paired smartphone or the like. Therefore, restricting the camera in this way can prevent fraudulent use.

[0238] In the first embodiment described above, the second authentication by the second authentication unit 508 is performed at the time of image capture (during image capture). However, performing authentication at the time of image capture may be difficult due to resource constraints such as speed. In such cases, information necessary for the second authentication may be saved at the time of image capture, and the authentication process may be performed after image capture, etc. For example, an eye image may be saved at the time of image capture, and authentication may be performed after image capture. Similarly, other biometric authentication (face authentication, fingerprint authentication) may also be performed by saving biometric information (face, fingerprint), etc. Alternatively, even when determining based on the connection status between the smartphone and the camera 100, parameters of the connection status may be saved, and the connection status may be analyzed and authentication processing may be performed later. In this way, the second authentication does not necessarily have to be performed at the time of image capture. Information necessary for the second authentication may be acquired at the time of image capture, and authentication may be performed after image capture, etc. Note that if authentication is performed later, it is possible that the authentication result will not be available in time for the image file 1200 to be saved. In this case, a second authentication status different from authentication in progress or not yet authenticated, such as "processing," may be prepared and saved as the photographer information 1211 in the metadata 1210. Then, when the authentication result of the second authentication is obtained, the photographer information 1211 in the metadata 1210 may be corrected and saved again. Note that it is desirable to use biometric authentication for the second authentication. It is difficult for the photographer to enter a password or the like when taking a photo. Also, authentication based on the connection status between the smartphone and the camera 100, for example, cannot be used by others if the user lends out the smartphone along with the camera 100. On the other hand, biometric authentication does not require any authentication action, and the information is only possessed by the user. Therefore, it is desirable to use biometric authentication for the second authentication performed when taking a photo.

[0239] In the first embodiment described above, the second authentication is performed when the first authentication status is "authenticated." However, if the user suddenly needs to take a photo or if the user forgets to perform the first authentication, the first authentication may not be performed. Therefore, the second authentication may be performed when the first authentication status is "unauthenticated." Specifically, in step S901 of FIG. 9, the process proceeds to step S902 only when the first authentication is "authenticated." However, step S901 of FIG. 9 may be modified so that the process proceeds to step S902 even when the first authentication is "unauthenticated." In addition, when the first authentication is "unauthenticated," the person ID in the authentication status table 570 of FIG. 5F is a value such as NULL. Therefore, when the second authentication is successful when the first authentication is "unauthenticated," the person ID in the authentication status table 570 may be updated to the person ID identified in step S909 of FIG. 9 in step S911 of FIG. 9. In the authentication status save process, the first authentication is stored as "unauthenticated" and the second authentication is stored as "authenticated" in the photographer information 1211 of the metadata 1210. At this time, the person ID may be stored in the photographer information 1211 of the metadata 1210 so that it is clear that the person ID was identified in the second authentication. For example, it may be recorded as a different metadata item. Note that, when there is more than one person registered, the feature vectors of the person ID identified in the first authentication are limited in S908 of FIG. 9 , which shows the second authentication, as described above. However, when the first authentication has not actually been performed, all feature vectors in the second authentication registration feature vector table 560 are extracted in S908 of FIG. 9 , and these are compared in S909 of FIG. 9 . At this time, the person ID with a feature vector exceeding the threshold is authenticated. Note that when there are multiple person IDs exceeding the threshold, the most similar person ID is authenticated. Alternatively, authentication may fail. 9 may be different depending on whether the first authentication is being performed or not. Alternatively, other authentication settings, such as the model to be used, may be changed. This is because the optimal settings change depending on whether the problem is 1:1 authentication or 1:N authentication.

[0240] In the first embodiment described above, the second authentication is performed on the assumption that the person ID is the person ID identified in the first authentication. However, the person ID may be identified independently in the second authentication without using the result of the first authentication. Specifically, in the authentication status table 570 of FIG. 5F managed by the authentication status management unit 521, the person ID is stored separately as a “person ID in the first authentication” and a “person ID in the second authentication.” Then, the person IDs for the first authentication and the second authentication may be stored separately. Furthermore, in S908 of FIG. 9 showing the second authentication process, the registered feature vector of the person ID in the first authentication is limited. However, this may be omitted and the person ID may be identified from all feature vectors. Furthermore, when saving metadata as shown in FIG. 12 showing the authentication status saving process, the person IDs and authentication statuses for the first authentication and the second authentication may all be stored separately in the metadata 1210. Even in this case, if the same person is identified in the first authentication and the second authentication when using the metadata 1210, it is possible to confirm that the person in question was photographed. Alternatively, when saving metadata in Fig. 12 showing the authentication status saving process, if the person IDs in the first authentication and the second authentication are different, the second authentication may be saved as "unauthenticated." In this case, the photographer information 1211 (i.e., name) saved in the metadata 1210 is personal information (i.e., name, etc.) related to the person ID of the first authentication. The process of identifying that the same person has been authenticated in the first authentication and the second authentication may be executed when saving the metadata 1210. In this way, the first authentication and the second authentication may be performed independently, and it may be confirmed that they are the same person when the authentication results of the first authentication and the second authentication are referenced.

[0241] In the first embodiment described above, the first authentication is performed when no image is being captured, and the second authentication is performed when an image is being captured. However, it is also possible to guarantee that the photographer is a registered user based only on the authentication result of the first authentication, without performing the second authentication. Specifically, it is possible to guarantee that the photographer is a registered user without performing the second authentication until a predetermined time has passed since the first authentication was successful.

[0242] In the first embodiment described above, in the registration process and the first authentication process, the left / right eye determination process is performed based on the difference between the actual viewpoint and the estimated viewpoint in the gaze detection process. However, the left / right eye determination process may be omitted, and the left / right eye determination may be performed by the user inputting which eye image of the left / right eye is to be acquired. Specifically, the left / right eye determination may be performed by the user specifying which eye the user will use to look through the viewfinder through a user operation, such as a menu, before acquiring an eye image in S604 of FIG. 6A and S617 of FIG. 6B in the registration process. Alternatively, the left / right eye determination may be performed by the user specifying which eye the user will use to look through the viewfinder through a user operation, such as a menu, before acquiring an eye image in S803 of FIG. 8A and S814 of FIG. 8B in the first authentication process. Alternatively, the left / right eye determination may be performed by the user specifying which eye the user will use to look through the viewfinder through a user operation, such as a menu, before acquiring an eye image. Specifically, when displaying instructions for the authentication method in S801 and S812 of FIG. 8A in the first authentication process, an instruction to look through the viewfinder with either the left or right eye is also displayed to distinguish between the left and right eyes. Alternatively, the left and right eyes may be distinguished based on which eye has the most similar feature vector obtained from the feature vector to be matched, without performing the left and right eye determination process. Specifically, when performing matching in S808 of FIG. 8A in the first authentication process, feature vectors are obtained from both the first authentication registration right eye feature vector table 540 shown in FIG. 5C and the first authentication registration left eye feature vector table 550 shown in FIG. 5D, and then matched. If the feature vector with the highest similarity is obtained from the first authentication registration right eye feature vector table 540, it is determined that the eye image has been obtained from the right eye. Similarly, if the feature vector with the highest similarity is obtained from the first authentication registration left eye feature vector table 550, it is determined that the eye image has been obtained from the left eye.

[0243] (Second embodiment) Next, a second embodiment will be described. In the following description of the second embodiment, matters common to the first embodiment will be omitted, and only matters different from the first embodiment will be described.

[0244] In the above-described embodiment, if neither of the authentications using multiple pieces of biometric information (both eyes) is successful in the first authentication, the first authentication is not successful. In the second embodiment, if the required strictness of matching is not high, the first authentication is successful if authentication of either the left or right eye is successful. As an example of the second embodiment, a rental device for renting head-mounted displays (HMDs) is assumed. For example, the required strictness of matching may be low, and if authentication of either the left or right eye is successful, the user is permitted to log in to the HMD. In contrast, when making a payment on the web via the HMD, if the required strictness of matching is high, the payment cannot be made unless authentication of both the left and right eyes is successful. Note that the operations permitted by the first authentication are not limited to these.

[0245] <Modification of the second embodiment> In the second embodiment described above, when the required matching strictness is not high, if authentication of either the left or right eye is successful, the first authentication is considered successful, and if authentication of either eye fails, the first authentication is considered unsuccessful. However, even if authentication of one eye fails, the first authentication may be considered successful if authentication of the other eye is successful.

[0246] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. This program and a computer-readable storage medium storing the program are included in the present invention.

[0247] It should be noted that the above-described embodiments of the present invention are merely illustrative examples of the implementation of the present invention, and the technical scope of the present invention should not be construed as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0248] The disclosure of this embodiment includes the following configuration, method, and program. [Configuration 1] An information processing device that performs user authentication, a management means for managing authentication registration information of users who are permitted to use the information processing device; an authentication unit that authenticates the user to be authenticated by using a plurality of pieces of authentication target information acquired from a plurality of different body parts of the user to be authenticated and the authentication registration information; An information processing device comprising: [Configuration 2] The plurality of pieces of authentication target information are a plurality of pieces of biometric information acquired from the plurality of body parts of the authentication target user. 2. The information processing device according to configuration 1, [Configuration 3] The plurality of pieces of biometric information are feature information based on eye images acquired from the right eye and the left eye of the user to be authenticated. 3. The information processing device according to configuration 2. [Configuration 4] The method further includes a determination unit that determines the eye image of the right eye and the eye image of the left eye based on a difference between the line of sight of the user to be authenticated estimated from the eye image and the actual line of sight of the user to be authenticated. 4. The information processing device according to configuration 3. [Configuration 5] The authentication means performs a second authentication of the user to be authenticated using the authentication target information acquired from one of the plurality of body parts of the user to be authenticated after a first authentication, which is the authentication of the user to be authenticated using the plurality of pieces of authentication target information, is successful. 5. The information processing device according to any one of configurations 1 to 4. [Configuration 6] The authentication means performs the first authentication before the user to be authenticated takes a photograph, and performs the second authentication while the user to be authenticated takes a photograph. 6. The information processing device according to configuration 5. [Configuration 7] The authentication means determines that the first authentication is successful when authentication using at least one of the plurality of pieces of authentication object information is successful in the first authentication. 7. The information processing device according to configuration 5 or 6. [Configuration 8] The management means manages the authentication registration information used for the first authentication and the authentication registration information used for the second authentication of the same user in association with each other. 8. The information processing device according to any one of configurations 5 to 7. [Configuration 9] The authentication device further includes an execution unit that executes a predetermined process based on the authentication states of the first authentication and the second authentication. 9. The information processing device according to any one of configurations 5 to 8. [Configuration 10] The execution means executes, as the predetermined process, at least one of a management process, a storage process, and a display process in the authentication states of the first authentication and the second authentication. 10. The information processing device according to configuration 9. [Configuration 11] The execution means changes the content of the predetermined process depending on the authentication results of the first authentication and the second authentication. 11. The information processing device according to configuration 9 or 10. [Method 1] A method for controlling an information processing device that performs user authentication, comprising: a management step of managing authentication registration information of users who are permitted to use the information processing device; an authentication step of authenticating the user to be authenticated by using a plurality of pieces of authentication object information acquired from a plurality of different body parts of the user to be authenticated and the authentication registration information; 1. A method for controlling an information processing device, comprising: [Program 1] 12. A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0249] 100: camera (information processing device), 110: photographing lens unit, 120: camera housing, 121: release button, 122: eyepiece, 123-125: operation members, 501: eye image acquisition unit, 502: feature vector calculation unit, 503: left / right eye determination unit, 504: user registration unit, 505: registration data management unit, 506: authentication target person confirmation unit, 507: first authentication unit, 508: second authentication unit, 509: unauthorized use detection unit, 510: first authentication status invalidation unit, 511: imaging unit, 520: execution unit, 521: authentication status management unit, 522: authentication status storage unit, 523: authentication status display unit

Claims

1. An information processing device that performs user authentication, a management means for managing authentication registration information of users who are permitted to use the information processing device; an authentication unit that authenticates the user to be authenticated by using a plurality of pieces of authentication target information acquired from a plurality of different body parts of the user to be authenticated and the authentication registration information; An information processing device comprising:

2. The plurality of pieces of authentication target information are a plurality of pieces of biometric information acquired from the plurality of body parts of the authentication target user.

2. The information processing apparatus according to claim 1, wherein:

3. The plurality of pieces of biometric information are feature information based on eye images acquired from the right eye and the left eye of the user to be authenticated.

3. The information processing apparatus according to claim 2, wherein:

4. The method further includes a determination unit that determines the eye image of the right eye and the eye image of the left eye based on a difference between the line of sight of the user to be authenticated estimated from the eye image and the actual line of sight of the user to be authenticated.

4. The information processing apparatus according to claim 3,

5. The authentication means performs a second authentication of the user to be authenticated using the authentication target information acquired from one of the plurality of body parts of the user to be authenticated after a first authentication, which is the authentication of the user to be authenticated using the plurality of pieces of authentication target information, is successful.

2. The information processing apparatus according to claim 1, wherein:

6. The authentication means performs the first authentication before the user to be authenticated takes a photograph, and performs the second authentication while the user to be authenticated takes a photograph.

6. The information processing apparatus according to claim 5,

7. The authentication means determines that the first authentication is successful when authentication using at least one of the plurality of pieces of authentication object information is successful in the first authentication.

6. The information processing apparatus according to claim 5,

8. The management means manages the authentication registration information used for the first authentication and the authentication registration information used for the second authentication of the same user in association with each other.

6. The information processing apparatus according to claim 5,

9. The authentication device further includes an execution unit that executes a predetermined process based on the authentication states of the first authentication and the second authentication.

6. The information processing apparatus according to claim 5,

10. The execution means executes, as the predetermined process, at least one of a management process, a storage process, and a display process in the authentication states of the first authentication and the second authentication.

10. The information processing apparatus according to claim 9,

11. The execution means changes the content of the predetermined process depending on the authentication results of the first authentication and the second authentication.

10. The information processing apparatus according to claim 9,

12. A method for controlling an information processing device that performs user authentication, comprising: a management step of managing authentication registration information of users who are permitted to use the information processing device; an authentication step of authenticating the user to be authenticated by using a plurality of pieces of authentication object information acquired from a plurality of different body parts of the user to be authenticated and the authentication registration information; 1. A method for controlling an information processing device, comprising:

13. A program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 11.

Images