Two-stage authentication information processing apparatus, control method of information processing apparatus, storage medium, and computer program product

Abstract

The invention provides an information processing apparatus for two-stage authentication, a control method of the information processing apparatus, a storage medium, and a computer program product. An information processing apparatus configured to perform user authentication includes: a management unit configured to manage authentication registration information related to a user who permits use of the information processing apparatus; and an authentication unit configured to perform authentication of an authentication target user by using a plurality of pieces of authentication target information acquired from a plurality of different parts of the authentication target user and the authentication registration information.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an information processing apparatus of two-factor authentication, a control method of an information processing apparatus, a storage medium, and a computer program product. BACKGROUND

[0002] Hitherto, there has been a method for performing personal authentication of a user of an information processing apparatus. For example, Japanese Patent Application Publication No. 2024-2562 discusses a method for performing personal authentication of a user by using an eye image when the user views a viewfinder.

[0003] Generally, personal authentication of a user of an information processing apparatus is performed by setting an extremely low false acceptance rate to prevent a user from being erroneously recognized as another person. However, setting such a low false acceptance rate increases a false rejection rate. This leads to a problem of reduced usability because user authentication is frequently performed in a case where the information processing apparatus is used, and a false rejection frequently occurs. SUMMARY

[0004] The present disclosure relates to improving accuracy of user authentication with an information processing apparatus so that a decrease in usability can be prevented.

[0005] According to an aspect of the present disclosure, an information processing apparatus configured to perform user authentication includes a management unit configured to manage authentication registration information related to a user who is permitted to use the information processing apparatus, and an authentication unit configured to perform authentication of an authentication target user by using a plurality of authentication target information acquired from a plurality of different parts of the authentication target user and the authentication registration information.

[0006] The features of the present disclosure will become apparent from the following description of the embodiments with reference to the accompanying drawings. The following description of embodiments is described by way of example. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1A And Figure 1B is a diagram illustrating an example of an appearance of a camera corresponding to the information processing apparatus according to the first typical embodiment.

[0008] Figure 2 is a diagram illustrating an example of an internal mechanism of a camera corresponding to the information processing apparatus according to the first typical embodiment.

[0009] Figure 3 is a diagram illustrating an example of an electrical structure of a camera corresponding to the information processing apparatus according to the first typical embodiment.

[0010] Figures 4A to 4D is a diagram illustrating the first typical embodiment, which illustrates a display example of a screen of a display device.

[0011] Figure 5A is a diagram illustrating an example of a functional configuration of a camera corresponding to the information processing apparatus according to the first typical embodiment.

[0012] Figure 5B is a diagram illustrating the first typical embodiment, which illustrates an example of various tables managed by the registration data management unit illustrated in Figure 5A

[0013] Figure 5C is a diagram illustrating the first typical embodiment, which illustrates an example of various tables managed by the registration data management unit illustrated in Figure 5A

[0014] Figure 5D is a diagram illustrating the first typical embodiment, which illustrates an example of various tables managed by the registration data management unit illustrated in Figure 5A

[0015] Figure 5E is a diagram illustrating the first typical embodiment, which illustrates an example of various tables managed by the registration data management unit illustrated in Figure 5A

[0016] Figure 5F is a diagram illustrating the first typical embodiment, which illustrates an example of various tables managed by the registration data management unit illustrated in Figure 5A

[0017] Figure 6A is a flowchart illustrating an example of a detailed processing procedure used for the registration processing in the method for controlling a camera corresponding to the information processing apparatus according to the first typical embodiment.

[0018] Figure 6B is a flowchart following the flowchart of Figure 6A , which illustrates an example of a detailed processing procedure used for the registration processing in the method for controlling a camera corresponding to the information processing apparatus according to the first typical embodiment.

[0019] Figure 7 is a flowchart illustrating an example of a detailed processing procedure used for the eye image acquisition processing in step S604 of Figure 6A

[0020] Figure 8A is a flowchart illustrating an example of a detailed processing procedure used for the first authentication processing in the method for controlling a camera corresponding to the information processing apparatus according to the first typical embodiment.

[0021] Figure 8B is a flowchart following the flowchart of Figure 8A ​​​​​​is a flowchart illustrating an example of a detailed process procedure used for the first authentication processing in the method for controlling the camera corresponding to the information processing apparatus according to the first typical embodiment.

[0022] Figure 9 is a flowchart illustrating an example of a detailed process procedure used for the second authentication processing in the method for controlling the camera corresponding to the information processing apparatus according to the first typical embodiment.

[0023] Figure 10 is a flowchart illustrating an example of a detailed process procedure used for the other person use detection processing in step S916 of Figure 9

[0024] Figure 11A is a flowchart illustrating an example of a detailed process procedure used for the first authentication invalidation processing in the method for controlling the camera corresponding to the information processing apparatus according to the first typical embodiment.

[0025] Figure 11B is a flowchart illustrating an example of a detailed process procedure used for the first authentication invalidation processing in the method for controlling the camera corresponding to the information processing apparatus according to the first typical embodiment.

[0026] Figure 11C is a flowchart illustrating an example of a detailed process procedure used for the first authentication invalidation processing in the method for controlling the camera corresponding to the information processing apparatus according to the first typical embodiment.

[0027] Figure 11D is a flowchart illustrating an example of a detailed process procedure used for the first authentication invalidation processing in the method for controlling the camera corresponding to the information processing apparatus according to the first typical embodiment.

[0028] Figure 12A is a flowchart illustrating an example of a detailed process procedure used for the authentication state storage processing in the method for controlling the camera corresponding to the information processing apparatus according to the first typical embodiment.

[0029] Figure 12B is a diagram illustrating the first typical embodiment, which illustrates an example of a structure of an image file stored by the authentication state storage unit.

[0030] Figure 13 is a diagram illustrating the first typical embodiment, which is a diagram for explaining the principle of the line-of-sight detection processing of the user.

[0031] Figure 14A and Figure 14B is a diagram illustrating the first typical embodiment, which is a diagram for explaining the line-of-sight detection processing of the user. ​

[0032] Figure 15 is a flowchart illustrating an example of a detailed process for the line-of-sight detection process in the method for controlling the camera corresponding to the information processing apparatus according to the first typical embodiment. DETAILED DESCRIPTION

[0033] A mode (typical embodiment) for implementing the present disclosure will be described with reference to the accompanying drawings.

[0034] First, the first typical embodiment will be described.

[0035] [Camera Structure]

[0036] Figure 1A And Figure 1B is a diagram illustrating an example of the appearance of the camera 100 corresponding to the information processing apparatus according to the first typical embodiment. Specifically, for example, a replaceable lens type digital still camera can be applied as the camera 100 corresponding to the information processing apparatus according to the present typical embodiment. Figure 1A is a perspective front view illustrating an example of the appearance of the camera 100 according to the first typical embodiment. Figure 1B is a perspective back view illustrating an example of the appearance of the camera 100 according to the first typical embodiment. In Figure 1A and Figure 1B In Figure 1A and Figure 1B illustrates an XYZ coordinate system taking the optical axis direction of the camera 100 as the Z-axis direction, and taking two mutually orthogonal directions orthogonal to the Z-axis direction as the X-axis direction and the Y-axis direction.

[0037] As illustrated in Figure 1A , the camera 100 includes an imaging lens unit 110 and a camera housing 120. On the front surface of the camera housing 120, a release button 121 as an operation member for accepting an imaging operation from a user (a photographer) is arranged.

[0038] As illustrated in Figure 1B , on the back surface of the camera housing 120, a display device (to be described below) 122 included in the inside of the camera housing 120 is arranged for the user to view. Figure 2eyepiece lens 122 (viewfinder) of the display device 214). On the back surface of the camera housing 120, operation members 123 to 125 for accepting various operations from the user are also arranged. The operation member 123 is a touch screen for accepting a touch operation, for example. The operation member 124 is an operation lever that can be tilted in various directions. The operation member 125 is a four-way directional pad that can be pressed in each of four directions. The operation member 123 that is a touch screen includes a display panel such as a liquid crystal panel and has a function of displaying various images on the display panel.

[0039] Figure 2 is a diagram illustrating an example of internal mechanisms of the camera 100 corresponding to the information processing apparatus according to the present exemplary embodiment. Specifically, Figure 2 is a cross-sectional view of the camera 100 taken along a YZ plane formed by the Y-axis direction and the Z-axis direction illustrated in Figure 1A . In this Figure 2 , components similar to those illustrated in Figure 1A and Figure 1B are denoted by the same reference numerals. Detailed description thereof will be omitted. Figure 2 illustrates an XYZ coordinate system corresponding to the XYZ coordinate system illustrated in Figure 1A and Figure 1B .

[0040] The imaging lens unit 110 includes lenses 201 and 202, an aperture 203, an aperture drive unit 204, a lens drive motor 205, a lens drive member 206, a pulse board 207, a photoelectric coupler 208, a focus adjustment circuit 209, and a mounting contact 210 as its internal mechanisms.

[0041] The lens drive member 206 includes a drive gear. The photoelectric coupler 208 detects rotation of the pulse board 207 that moves together with the lens drive member 206 and transmits the detected information to the focus adjustment circuit 209. The focus adjustment circuit 209 drives the lens drive motor 205 based on the information from the photoelectric coupler 208 and information (information about the amount of lens drive) from the camera housing 120, thereby moving the lens 201 to change the focus position. The mounting contact 210 is an interface between the imaging lens unit 110 and the camera housing 120. In Figure 2 , two lenses 201 and 202 are illustrated for simplicity. Actually, the imaging lens unit 110 includes more than two lenses.

[0042] The camera housing 120 includes, as its internal mechanisms, an image sensor 211, a central processing unit (CPU) 212, a memory unit 213, a display device 214, a display device drive circuit 215, light sources 216a and 216b, a beam splitter 217, a light receiving lens 218, and an eye image sensor 219.

[0043] The image sensor 211 is located at an intended imaging plane of the photographing lens unit 110. The CPU 212 is a microcomputer CPU, and controls the operation of the entire camera 100 and performs various types of processing. The memory unit 213 stores various types of information and programs for execution by the CPU 212 in performing various types of processing. For example, the memory unit 213 stores a subject image photographed by the image sensor 211. The display device 214 displays various types of information on a screen (display surface) of the display device 214. For example, the display device 214 is a liquid crystal panel, and displays a photographed image (subject image) on its screen. The display device drive circuit 215 drives the display device 214. The user's (photographer's) eye E can view the screen of the display device 214 through the eyepiece lens 122.

[0044] The light sources 216a and 216b are light sources conventionally used in a single-lens reflex camera to detect the line of sight of the eye E according to the relationship between a reflection image (corneal reflection image) formed by the corneal reflection of light and the pupil. Specifically, the light sources 216a and 216b are light sources for illuminating the user's eye E viewing the viewfinder (eyepiece lens 122). For example, the light sources 216a and 216b are infrared light emitting diodes that emit infrared light that is not perceptible to the user's eye E, and are disposed around the eyepiece lens 122. An optical image of the eye E (eye optical image; an optical image formed by light emitted from the light sources 216a and 216b and reflected at the eye E) illuminated by the light sources 216a and 216b is transmitted through the eyepiece lens 122 and reflected at the beam splitter 217. The eye optical image is formed on the eye image sensor 219 by the light receiving lens 218, in which a plurality of photoelectric conversion elements (such as charge-coupled device [CCD] elements and complementary metal-oxide semiconductor [CMOS] elements, etc.) are arranged two-dimensionally. The light receiving lens 218 positions the pupil of the user's eye E and the eye image sensor 219 in a conjugate imaging relationship. By a line-of-sight detection process, the line of sight of the eye E is detected according to the position of the corneal reflection image on the eye optical image formed on the eye image sensor 219. For example, as information related to the line of sight, at least information indicating the direction of the line of sight, or information indicating a point of gaze (a point to which the line of sight is directed) on the screen of the display device 214 is obtained.

[0045] The point of gaze can be considered as a position at which the user is looking, or a line-of-sight position.

[0046] Figure 3 is a diagram illustrating an example of an electrical structure of the camera 100 corresponding to the information processing apparatus according to the first typical embodiment. In this Figure 3 , components similar to those illustrated in Figure 1A , 1B and Figure 2 are denoted by the same reference numerals. Detailed description thereof will be omitted.

[0047] The imaging lens unit 110 includes Figure 2 the focus adjustment circuit 209 and the aperture control circuit 306 illustrated as its electrical components.

[0048] The camera housing 120 includes Figure 1A the release button 121 and the operation members 123 to 125 illustrated as its electrical components. The camera housing 120 further includes Figure 2 the image sensor 211, the CPU 212, the memory unit 213, the display device 214, the display device drive circuit 215, the light sources 216a and 216b, and the eye image sensor 219 illustrated as its electrical components. As Figure 3 illustrated, the camera housing 120 further includes the gaze detection circuit 301, the light measurement circuit 302, the automatic focus detection circuit 303, the signal input circuit 304, and the light source drive circuit 305 as its electrical components.

[0049] As Figure 3 illustrated, the CPU 212 is connected to the gaze detection circuit 301, the light measurement circuit 302, the automatic focus detection circuit 303, the signal input circuit 304, the light source drive circuit 305, the image sensor 211, the display device drive circuit 215, the memory unit 213, and the operation members 123 to 125. The CPU 212 sends a signal to the focus adjustment circuit 209 arranged in the imaging lens unit 110 and the aperture control circuit 306 included in the aperture drive unit 204 in the imaging lens unit 110 via the mounting contact 210. The memory unit 213 accompanying the CPU 212 has, for example, a storage function of storing an imaging signal from the image sensor 211 and the eye image sensor 219.

[0050] The gaze detection circuit 301 performs analog-digital (A / D) conversion on the output of the eye image sensor 219 (an eye image of the user's eye E captured) in a state where an eye optical image is formed on the eye image sensor 219, and sends the result to the CPU 212. The CPU 212 extracts a feature point used in gaze detection from the eye image based on the gaze detection processing, and detects the user's gaze from the position of the feature point.

[0051] The photometry circuit 302 performs predetermined processing (e.g., amplification, logarithmic compression, and A / D conversion) on a signal (such as a luminance signal corresponding to the brightness of the field of view) obtained from the image sensor 211, which also functions as a photometry sensor. The photometry circuit 302 transmits the processing result as field brightness information to the CPU 212.

[0052] The autofocus detection circuit 303 A / D-converts signals from a plurality of detection elements (pixels) included in the image sensor 211 for use in phase difference detection, and transmits the A / D-converted signals to the CPU 212. The CPU 212 calculates the distance to the subject corresponding to each focus detection point from the signals of the plurality of detection elements. This is a conventional technique called image plane phase difference AF.

[0053] Figures 4A to 4D is a diagram illustrating the first typical embodiment, which illustrates a display example of the screen of the display device 214.

[0054] For example, in the present typical embodiment, the focus detection points described in connection with the autofocus detection circuit 303 are located at 180 positions on the imaging surface respectively corresponding to the 180 positions indicated on the screen (finder inner field of view) of the display device 214. Figure 4A The 180 positions indicated on the screen (finder inner field of view) of the display device 214 illustrated above. Figure 4A The display device 214 illustrates a state in operation (a state of displaying an image). The finder inner field of view includes the focus detection region 401, the field mask 402, and the 180 ranging point indicators 410 within the focus detection region 401. Figure 4A The illustrated ranging point indicators 410 are superimposed on the through image (live view image) displayed on the display device 214 so that the ranging point indicators 410 are displayed at positions respectively corresponding to the respective focus detection points on the imaging surface. In the illustrated example, the ranging point indicators 410 are displayed in the focus detection region 401. Figure 4A Among the 180 ranging point indicators 410 illustrated above, for example, the ranging point indicator 410A corresponding to the current gaze point A (estimated position) is highlighted with a frame.

[0055] Now, returning to the explanation of Figure 3 above.

[0056] The switches SW1 and SW2 of the release button 121 are connected to the signal input circuit 304. The switch SW1 is a switch that becomes on by the first stroke of the release button 121 to start the imaging preparation operation (such as photometry and ranging) of the camera 100.

[0057] The switch SW2 is a switch that becomes ON to start the image capturing operation by the second stroke of the release button 121. When the ON signals from the switches SW1 and SW2 of the release button 121 are input to the signal input circuit 304, the signal input circuit 304 transmits the input ON signals to the CPU 212. When the switch SW1 of the release button 121 becomes ON, the detection of the user's line of sight can be started.

[0058] The light source driving circuit 305 drives the light sources 216a and 216b.

[0059] When the user operates the operation members 123 to 125, the operation members 123 to 125 output operation signals based on the user's operation to the CPU 212. The CPU 212 performs processing (control) based on the operation signals. For example, the CPU 212 moves a selection frame on a displayed menu based on the operation signals.

[0060] Figure 5A is a diagram illustrating an example of a functional structure of the camera 100 corresponding to the information processing apparatus according to the first typical embodiment. The camera 100 is an information processing apparatus that performs user authentication. The camera 100 includes, as its functional components, an eye image acquisition unit 501, a feature vector calculation unit 502, a left-right eye judgment unit 503, a user registration unit 504, a registration data management unit 505, and an authentication target person confirmation unit 506. The camera 100 further includes, as its functional components, a first authentication unit 507, a second authentication unit 508, an other person use detection unit 509, a first authentication state invalidation unit 510, an image capturing unit 511, and an execution unit 520. Figures 5A-5F The illustrated functional components (501 to 511 and 520) are realized, for example, by Figure 2 and Figure 3 The illustrated CPU 212 realizes by executing a program stored in the illustrated memory unit 213. Figure 2 and Figure 3 The illustrated memory unit 213.

[0061] In the present typical embodiment, as personal authentication, the camera 100 authenticates whether the user is a registered person based on the user's eyes E that view the viewfinder (eyepiece lens 122). In particular, the camera 100 performs a first authentication before the user uses the camera 100 to capture an image, and further performs a second authentication at the time of image capturing (for example, during image capturing). In the present typical embodiment, the camera 100 stores the authentication result together with the captured image.

[0062] The eye image acquisition unit 501 is an eye image acquisition unit for acquiring an eye image that is an image of the user's eye E that views the viewfinder (eyepiece lens 122). Specifically, the eye image acquisition unit 501 acquires an eye image (eye image signal; electric signal of the eye image) from the line-of-sight detection circuit 301 via the eye image sensor 219. Figure 3 The eye image sensor 219 illustrated as an example acquires an eye image (eye image signal; electric signal of the eye image).

[0063] The feature vector calculation unit 502 calculates a feature vector that is feature information for use in authenticating the user (authentication target user) from the eye image acquired by the eye image acquisition unit 501. Here, the feature vector is calculated using, for example, a neural network as a feature extractor. For example, the present typical embodiment uses a convolutional neural network (CNN) that is a kind of neural network. The CNN extracts abstract information from an input image by repeating processing including convolution processing, activation processing, and pooling processing on the input image. Here, a processing unit composed of the convolution processing, the activation processing, and the pooling processing is often referred to as a layer. For the activation processing to be used here, various conventional techniques are known. For example, a technique called rectified linear unit (ReLU) can be used. For the pooling processing, various conventional techniques are also known. For example, a technique called max pooling can be used. As the CNN configuration, for example, a residual network (ResNet) can be used.

[0064] A neural network called Vision Transformer (ViT) can also be used. The structure of the neural network is not limited to the above. The feature vector calculation unit 502 can store information such as the configuration and the weight of the neural network in the memory unit 213. The weight of the neural network used by the feature vector calculation unit 502 is a weight acquired by pre-training. For example, eye images of various persons for training are acquired in advance, and the neural network is trained using a method such as ArcFace. Although the example using the neural network is described as a method for personal authentication based on an eye image, a conventional method called iris authentication (for example, a method discussed in Japanese Patent 3307936) can be used. The method for personal authentication of the user is not limited to the above.

[0065] The left-right eye determination unit 503 is a determination unit that determines whether the eye image acquired by the eye image acquisition unit 501 is an eye image of the right eye or an eye image of the left eye of the user (authentication target user). For example, in the present typical embodiment, the left-right eye determination unit 503 determines whether the eye image acquired by the eye image acquisition unit 501 is an eye image of the right eye or an eye image of the left eye of the user (authentication target user) based on a deviation between the line of sight of the user (authentication target user) estimated from the acquired eye image and the actual line of sight of the user (authentication target user).

[0066] The user registration unit 504 generates data to be registered in the registration data management unit 505.

[0067] The registration data management unit 505 is a management unit that manages authentication registration information on a user who is permitted to use the camera 100 corresponding to the information processing apparatus. Specifically, the registration data management unit 505 stores a feature vector of an eye image of a registered user and a name of the registered user in association with each other in the storage unit 213. In the present exemplary embodiment, the registration data management unit 505 stores authentication registration information for use in the first authentication and authentication registration information for use in the second authentication of the same user in association with each other in the storage unit 213.

[0068] Figures 5B to 5E is a diagram illustrating the first exemplary embodiment, which illustrates Figure 5A Examples of various tables managed by the registration data management unit 505 illustrated. Specifically, Figure 5B The registration person information table 530 is illustrated, which is a table that associates a personal identifier (ID) with a name of a registered user. Figure 5C The first authentication registration right eye feature vector table 540 is illustrated, which is a table that associates a personal ID with a first authentication registration right eye feature vector to be used by the first authentication unit 507.

[0069] Figure 5D The first authentication registration left eye feature vector table 550 is illustrated, which is a table that associates a personal ID with a first authentication registration left eye feature vector to be used by the first authentication unit 507. Figure 5E The second authentication registration feature vector table 560 is illustrated, which is a table that associates a personal ID with a second authentication registration feature vector to be used by the second authentication unit 508.

[0070] Figures 5B to 5E The various tables 530 to 560 illustrated associate information on the same registered user by using a personal ID.

[0071] Figure 5F is a diagram illustrating the first exemplary embodiment, which illustrates Figure 5A An example of the authentication status table 570 managed by the authentication status management unit 521 illustrated. As Figures 5B to 5E As with the Figure 5F The authentication status table 570 illustrated also associates information on the same registered user by using a personal ID.

[0072] Now, returning to Figure 5A the explanation of

[0073] The authentication target person confirming unit 506 confirms whether or not the two eye images acquired by the eye image acquiring unit 501 are eye images acquired from the same person (authentication target user).

[0074] The first authentication unit 507 authenticates the authentication target user by using feature vectors based on feature information of the eye images acquired from the right eye and the left eye of the authentication target user and calculated by the feature vector calculating unit 502 and the authentication registration information managed by the registration data managing unit 505. In the present exemplary embodiment, the feature vectors based on the feature information of the eye images acquired from the right eye and the left eye of the authentication target user and calculated by the feature vector calculating unit 502 correspond to a plurality of authentication target information of a plurality of biometric information acquired from a plurality of parts of the authentication target user. The authentication of the authentication target user by the first authentication unit 507 (first authentication) is performed before the authentication target user captures an image using the camera 100 (at a time other than the image capturing). In the present exemplary embodiment, the first authentication unit 507 can be configured to determine that the first authentication is successful in a case where the authentication using at least one of the plurality of authentication target information is successful.

[0075] The second authentication unit 508 authenticates the authentication target user by using authentication target information acquired from one of the plurality of parts (the left eye and the right eye) of the authentication target user and the authentication management information managed by the registration data managing unit 505. The authentication of the authentication target user by the second authentication unit 508 (second authentication) is performed while the authentication target user is capturing an image using the camera 100 (at the time of image capturing). For example, the second authentication by the second authentication unit 508 is performed after the first authentication by the first authentication unit 507 is successful.

[0076] In the present exemplary embodiment, the first authentication unit 507 and the second authentication unit 508 constitute an "authentication unit" that authenticates the authentication target user. Here, the first authentication unit 507 desirably uses an authentication setting having a low false acceptance rate. On the other hand, the second authentication unit 508 desirably uses an authentication setting having a low false rejection rate. For example, such an authentication setting can be realized by the first authentication unit 507 and the second authentication unit 508 using the same authentication method but using different similarity threshold values. Specifically, in such a case, a high similarity threshold value is set for the first authentication unit 507, and a low similarity threshold value is set for the second authentication unit 508. This can reduce the false acceptance rate of the first authentication unit 507 and reduce the false rejection rate of the second authentication unit 508. In this way, when it is difficult to simultaneously reduce both the false acceptance rate and the false rejection rate only at the time of image capturing, two-factor authentication can be performed to achieve such a reduction.

[0077] The other-person-use detecting unit 509 detects performance of photographing by a person different from the person authenticated by the first authentication unit 507 (an other person). The other-person-use detecting unit 509 detects whether the person is the same person or an other person based on a pattern of failure of authentication by the second authentication unit 508. Specifically, if authentication by the second authentication unit 508 fails continuously for a predetermined number of times or more, the other-person-use detecting unit 509 determines that the person is an other person. Alternatively, if the authentication score by the second authentication unit 508 is significantly low, the other-person-use detecting unit 509 determines that the person is an other person. The method for determining whether the person is an other person is not limited to the above.

[0078] If the first authentication by the first authentication unit 507 is successful, the first authentication state invalidation unit 510 determines whether to invalidate the authentication state. There are several methods of invalidation determination.

[0079] The first method of invalidation determination is based on the elapsed time since the first authentication was successful.

[0080] For example, the first authentication state is invalidated when the elapsed time since the first authentication was successful exceeds a predetermined expiration time. If the second authentication is successful during the first authentication valid period, the expiration time is extended. On the other hand, if the second authentication fails, the expiration time is shortened. The invalidation determination method based on the elapsed time is not limited to the above.

[0081] The second method of invalidation determination is based on a change in the state of the power supply of the camera 100. For example, the first authentication state is invalidated when the power supply of the camera 100 is turned off or when the camera 100 enters a sleep mode. Note that, once the power supply of the camera 100 is turned off due to battery depletion, the processing for invalidating the first authentication state cannot be performed. Therefore, instead of being invalidated when the power supply is turned off, the first authentication state can be invalidated when the power supply is turned on. The same applies to the sleep mode. The first authentication state can be invalidated when the camera 100 is recovered from the sleep mode. The invalidation determination method based on a change in the state of the power supply of the camera 100 is not limited to the above.

[0082] The third method of invalidation determination is based on the distance to or connection state with a device carried by the user. Examples of the device include a smartphone. For example, the smartphone owned by the user is connected to the camera 100 through Bluetooth The connection, and invalidates the first authentication state when the connection is disconnected. Alternatively, the approximate distance can be estimated from the connection state, and the first authentication state can be invalidated when the distance can be judged to be greater than a predetermined level. This can prevent the camera 100 from being used by other persons when the user leaves the camera 100. Other examples of the device other than the smartphone can include a radio frequency identification (RFID) tag. The invalidation judgment method based on the distance from the device carried by the user or the connection state with the device carried by the user is not limited to the above.

[0083] The fourth invalidation judgment method is based on input of a user's explicit invalidation operation. For example, an operation menu item "invalidate the first authentication" is prepared on a menu, and the user selects and runs this item using the operation members 123 to 125. Alternatively, a switch button such as an invalidation button is provided on the camera 100, and the user presses this switch button. When such an operation is accepted, the first authentication state is invalidated. The invalidation judgment method based on input of a user's explicit invalidation operation is not limited to the above.

[0084] The fifth invalidation judgment method is based on a detection result of the other person use detection unit 509. Specifically, when the other person use detection unit 509 detects use by other persons, the first authentication state is invalidated.

[0085] The first to fifth invalidation judgment methods described above can be used in combination to reduce the possibility of false acceptance of the second authentication. Specifically, the fourth invalidation judgment method enables the user to intentionally prevent use by other persons. In addition, the first to third invalidation judgment methods can invalidate the first authentication state to preventively prevent use by other persons in a situation where the possibility of use by the user authenticated by the first authentication is low. Further, the fifth invalidation judgment method can prevent use by other persons by invalidating the first authentication state when use by other persons is suspected.

[0086] The imaging unit 511 receives a press signal of the release button 121 by the user, and stores an image (object image) captured by the image sensor 211 into the storage unit 213.

[0087] The execution unit 520 executes a predetermined process based on the authentication states of the first authentication and the second authentication.

[0088] As Figure 5A illustrated, the execution unit 520 includes an authentication state management unit 521, an authentication state storage unit 522, and an authentication state display unit 523.

[0089] The authentication status management unit 521 performs management processing of the authentication status of the first authentication using the first authentication unit 507 and the second authentication using the second authentication unit 508 as predetermined processing. In addition, the authentication status management unit 521 performs management processing related to the presence or absence of use by other person. For example, the authentication status management unit 521 stores the personal ID of the person who is currently using the terminal 200 in the authentication status table 570. The authentication status management unit 521 performs the management processing of the authentication status in the authentication status table 570 in accordance with the processing of the first authentication unit 507, the processing of the second authentication unit 508, and the processing of the first authentication invalidation unit 510. Figure 5E The illustrated authentication status table 570 is held in the memory unit 213 and performs the management processing.

[0090] The processing of the first authentication unit 507 will be described. Figure 5F The illustrated authentication status table 570.

[0091] The "first authentication status" in the authentication status table 570 indicates whether the first authentication using the first authentication unit 507 is in progress, and takes one of two values "authenticated" and "not authenticated". The personal ID indicates the ID of the person identified by the first authentication. If the first authentication status is "not authenticated", the personal ID has a value for indicating null, such as null or the like. The "second authentication status" in the authentication status table 570 indicates whether the second authentication using the second authentication unit 508 is in progress, and takes one of two values "authenticated" and "not authenticated". The "used by other person" in the authentication status table 570 indicates whether the use by other person detection unit 509 has detected the use by other person, and takes one of two values "yes" and "no". The specific processing for updating the authentication status table 570 will be described below in conjunction with the description of the first authentication processing ( Figure 8A and Figure 8B ), the second authentication processing ( Figure 9 ), and the first authentication invalidation processing ( Figures 11A to 11D ). The method of the memory unit 213 holding the authentication status table 570 is not limited to the table configuration. For example, a key value configuration can be used.

[0092] Now, the description will be returned again to the Figure 5A .

[0093] The authentication status storage unit 522 performs storage processing for storing the authentication status of the first authentication and the second authentication managed by the authentication status management unit 521 and the presence or absence of use of other person as metadata in association with the image acquired by the imaging unit 511, as predetermined processing. An example of the processing for storing the image metadata includes a method called Content Authenticity and Provenance Alliance (C2PA). The C2PA is a method of adding metadata for indicating the content of editing performed on an image to the image for the purpose of authenticating the origin, scene, and provenance of the image. Thus, the authentication status storage unit 522 can store the authentication status in accordance with the C2PA. In the present exemplary embodiment, metadata can be stored by other methods. The image file and the metadata file can be stored separately. The metadata can be kept in a database.

[0094] The authentication status display unit 523 performs display processing for displaying the authentication status of the first authentication and the second authentication managed by the authentication status management unit 521 on the camera 100, as predetermined processing. For example, if the first authentication status is "authenticated", the authentication status display unit 523 displays "first authentication: authenticated" on the display device 214 or the touch panel (operation member 123). The camera 100 can include a light-emitting diode (LED) lamp (not illustrated), and the authentication status display unit 523 can cause the LED lamp to light up when the first authentication status is "authenticated".

[0095] In the present exemplary embodiment, the execution unit 520 can be configured to change the content of the predetermined processing with the authentication status management unit 521, the authentication status storage unit 522, and the authentication status display unit 523 based on the authentication result of the first authentication and the second authentication.

[0096] [Registration processing]

[0097] Figure 6A and Figure 6B is a flowchart illustrating an example of a detailed process procedure of the registration processing in the method for controlling the camera 100 corresponding to the information processing apparatus according to the first exemplary embodiment. Figure 6A and Figure 6B The processing of the illustrated flowchart is mainly performed by the user registration unit 504 on the CPU 212. Figure 6A and Figure 6B The processing of the illustrated flowchart is expected to be performed by the user operating the camera 100 at a time other than the time of imaging. Thus, Figure 6A and Figure 6BThe processing of the illustrated flowchart is executed when the user operates the camera 100 and calls the processing from a menu. For example, an unillustrated menu screen is displayed on the touch screen (operation member 123) of the camera 100, and when the user operates the menu screen using the operation members 123 to 125 and selects a menu for calling the processing, the processing is executed.

[0098] In step S601, the CPU 212 (user registration unit 504) accepts input of personal information related to a person to be registered. In the present exemplary embodiment, the CPU 212 (user registration unit 504) accepts input of "name". Specifically, the CPU 212 (user registration unit 504) displays an unillustrated screen for input of name on the touch screen (operation member 123), and accepts a name input by the user operating the operation members 123 to 125. When the input is completed, the user notifies the CPU 212 of completion of input of the name using a completion button displayed on the screen. Figure 6A In step S602, the CPU 212 (user registration unit 504) displays a method for registering an eye image of a dominant eye to the user. Specifically, the CPU 212 (user registration unit 504) displays an instruction for the user to view a viewfinder with the dominant eye (an eye with which the user views a viewfinder at the time of photographing an image) on the touch screen (operation member 123). The CPU 212 (user registration unit 504) displays an instruction to look at a marker in the viewfinder. The CPU 212 (user registration unit 504) can also display an instruction such as not to blink and keep the eyes wide open for photographing a desired eye image.

[0099] Figure 6A In step S603 to S614, the processing of the illustrated flowchart is executed.

[0100] In step S603, the CPU 212 (user registration unit 504) displays a marker on the display device 214. Specifically, the CPU 212 displays only the illustrated marker 411, and does not display other markers. Alternatively, the CPU 212 can display all the illustrated markers 411 to 415, with only the marker 411 displayed in a highlighted color. Other display methods can be used as long as the user can be notified to look at the marker 411. Figure 6A Figure 4C In step S603 to S614, the processing of the illustrated flowchart is executed.

[0101] In step S603, the CPU 212 (user registration unit 504) displays a marker on the display device 214. Specifically, the CPU 212 displays only the illustrated marker 411, and does not display other markers. Alternatively, the CPU 212 can display all the illustrated markers 411 to 415, with only the marker 411 displayed in a highlighted color. Other display methods can be used as long as the user can be notified to look at the marker 411. Figure 6A Figure 4C In step S603, the CPU 212 (user registration unit 504) displays a marker on the display device 214. Specifically, the CPU 212 displays only the illustrated marker 411, and does not display other markers. Alternatively, the CPU 212 can display all the illustrated markers 411 to 415, with only the marker 411 displayed in a highlighted color. Other display methods can be used as long as the user can be notified to look at the marker 411. Figure 4C In step S603, the CPU 212 (user registration unit 504) displays a marker on the display device 214. Specifically, the CPU 212 displays only the illustrated marker 411, and does not display other markers. Alternatively, the CPU 212 can display all the illustrated markers 411 to 415, with only the marker 411 displayed in a highlighted color. Other display methods can be used as long as the user can be notified to look at the marker 411.​​​

[0102] exist Figure 6A In step S604, the eye image acquisition unit 501 acquires an eye image when the user views the viewfinder (eyepiece lens 122). Reference will now be made to... Figure 7 To explain Figure 6A Detailed processing of step S604 in the process.

[0103] Figure 7 This is an example Figure 6A A flowchart illustrating the detailed processing steps used in step S604, specifically the eye image acquisition process. Figure 7 The processing of the illustrated flowchart is mainly performed by the eye image acquisition unit 501 on the CPU 212.

[0104] exist Figure 7 In step S701, the eye image acquisition unit 501 performs gaze detection processing on the user.

[0105] exist Figure 7 In step S702, the eye image acquisition unit 501 determines whether an image suitable for authentication has been successfully acquired. Specifically, the eye image acquisition unit 501 determines whether an image suitable for authentication has been successfully acquired based on whether the gaze detection process in step S701 was successful. For example, in the gaze detection process in step S701, the eye image acquisition unit 501 acquires an eye image (eye image signal; electrical signal of the eye image) from the eye image sensor 219 via the gaze detection circuit 301. Then, the eye image acquisition unit 501 determines the coordinates of the corneal reflection images of the light sources 216a and 216b observed on the eye image and the coordinates of the pupil center. Then, the eye image acquisition unit 501 determines the coordinates of the user's gaze on the display device 214 based on the determined coordinates. For this reason, if the coordinates of the pupil center cannot be detected, the eye image acquisition unit 501 determines that the gaze detection process has failed. If the process used to obtain the coordinates of the pupil center fails... Figure 6A If the coordinates are not obtained in step S603, the eye image acquisition unit 501 can thus determine that it has not successfully acquired an image suitable for authentication.

[0106] exist Figure 7 In step S703, the eye image acquisition unit 501 determines whether an image suitable for authentication has been successfully acquired based on the judgment result of step S702.

[0107] If in Figure 7 In step S703, if the eye image acquisition unit 501 determines that it has successfully acquired an image (eye image) suitable for authentication ("yes" in step S703), then the process proceeds to step S704.

[0108] exist Figure 7 In step S704, the eye image acquisition unit 501 acquires an eye image by cropping. Specifically, the eye image acquisition unit 501 initially acquires the eye image obtained in step S701. Using the coordinates of the pupil center image c' obtained in step S701, the eye image acquisition unit 501 crops the eye image to a certain size, making the pupil center image c' the center of the image. The eye image acquisition unit 501 further generates and acquires a resized image by adjusting the size of the cropped image to the input size of the neural network of the feature vector calculation unit 502.

[0109] exist Figure 7 In step S705, the eye image acquisition unit 501 uses a flag to record the successful acquisition of the eye image.

[0110] If in Figure 7 In step S703, if the eye image acquisition unit 501 determines that it has failed to acquire an eye image suitable for authentication (acquisition failure) (in step S703, it is "no"), then the process proceeds to step S706.

[0111] exist Figure 7 In step S706, the eye image acquisition unit 501 performs a waiting period for a predetermined time. This step S706 is performed when a change in the eye image is anticipated and gaze detection is successful.

[0112] exist Figure 7 In step S707, the eye image acquisition unit 501 determines whether the acquisition of an eye image suitable for authentication has failed consecutively a predetermined number of times. If in Figure 7 In step S707, if the eye image acquisition unit 501 determines that the acquisition of an eye image suitable for authentication has not failed for a predetermined number of consecutive times (in step S707, this is "No"), then the process returns to step S701. Then, the process from step S701 onwards is repeated.

[0113] If in Figure 7 In step S707, if the eye image acquisition unit 501 determines that the acquisition of an eye image suitable for authentication has failed for a predetermined number of consecutive times ("Yes" in step S707), then the process proceeds to step S708.

[0114] exist Figure 7 In step S708, the eye image acquisition unit 501 uses a flag to record failed acquisitions of the eye image.

[0115] exist Figure 7 The processing in step S705 is completed, or Figure 7 When step S708 is completed, Figure 7 The processing of the flowchart has ended. Figure 7 Once the flowchart processing is complete, Figure 6A The eye image acquisition and processing in step S604 is now complete.

[0116] Return to Figure 6A Explanation.

[0117] exist Figure 6A If step S604 is completed, the process proceeds to step S605.

[0118] exist Figure 6A In step S605, the eye image acquisition unit 501 determines whether the eye image has been successfully acquired. Specifically, the eye image acquisition unit 501 is based on... Figure 7 The flag recorded in step S705 or S708 is used to determine whether the eye image has been successfully acquired.

[0119] If in Figure 6A In step S605, if the eye image acquisition unit 501 determines that the eye image has been successfully acquired (in step S605, it is "yes"), then the process proceeds to step S606.

[0120] exist Figure 6A In step S606, the feature vector calculation unit 502 extracts feature vectors from the eye image as authentication registration information for user authentication. Specifically, the feature vector calculation unit 502 extracts feature vectors from the eye image. Figure 6A The feature vector is extracted from the eye image obtained in step S604.

[0121] exist Figure 6A In step S607, the CPU 212 (user registration unit 504) displays on the display device 214 an indicator that an eye image has been successfully captured using the information displayed on the display device 214. For example, the CPU 212 (user registration unit 504) may display a message "An eye image has been successfully captured" or an icon indicating success on the display device 214.

[0122] If in Figure 6A In step S605, the eye image acquisition unit 501 determines that the eye image was not successfully acquired (acquisition failure) (in step S605, it is "no"), and then the process proceeds to step S608.

[0123] exist Figure 6AIn step S608, the CPU 212 (user registration unit 504) displays on the display device 214 that the photographing of the eye image has failed with the index displayed on the display device 214. For example, the CPU 212 (user registration unit 504) can display a message "the photographing of the eye image has failed" on the display device 214, or display an icon indicating the failure. In a case where the processing of step S608 is completed, the processing returns to step S604.

[0124] In Figure 6A a case where the processing of step S607 is completed, the processing proceeds to step S609.

[0125] In Figure 6A step S609, the CPU 212 (user registration unit 504) determines whether there is any index that has not been displayed. The CPU 212 (user registration unit 504) confirms whether all of the indexes 411 to 415 exemplified above have been displayed, and determines whether there is any index that has not been displayed. Figure 4C

[0126] If the CPU 212 (user registration unit 504) determines in step S609 of the flowchart of Fig. 6 that there is an index that has not been displayed (YES in step S609), the processing proceeds to step S610. Figure 6A

[0127] In Figure 6A step S610, the CPU 212 (user registration unit 504) displays on the display device 214 the next index among the indexes that have not been displayed. For example, if the index 411 exemplified above has been displayed, the CPU 212 (user registration unit 504) displays the index 412 as the next index on the display device 214. In this way, the CPU 212 selects the indexes 411 to 415 in the order of the index numbers and displays them on the display device 214. In a case where the processing of step S610 is completed, the processing returns to step S604. Figure 4C

[0128] If the CPU 212 (user registration unit 504) determines in step S609 of the flowchart of Fig. 6 that there is no index that has not been displayed (NO in step S609), the processing proceeds to step S611. Figure 6A

[0129] In Figure 6A step S611, the CPU 212 (user registration unit 504) determines whether the eye image acquired in step S604 is the eye image acquired from the right eye.

[0130] If the CPU 212 (user registration unit 504) determines in step S611 of the flowchart of Fig. 6 that the eye image acquired in step S604 is the eye image acquired from the right eye (YES in step S611), the processing proceeds to step S612. Figure 6A ​​​​When the CPU 212 (user registration unit 504) determines in step S611 that the eye image acquired in step S604 is an eye image acquired from the right eye (YES in step S611), the processing proceeds to step S612.

[0131] In Figure 6A step S612, the CPU 212 (user registration unit 504) stores the acquired information in Figure 5C the first authentication registration right eye feature vector table 540 illustrated in

[0132] When the CPU 212 (user registration unit 504) determines in step S611 that the eye image acquired in step S604 is not an eye image acquired from the right eye (is an eye image acquired from the left eye) (NO in step S611), the processing proceeds to step S613. Figure 6A In

[0133] step S613, the CPU 212 (user registration unit 504) stores the acquired information in Figure 6A the first authentication registration left eye feature vector table 550 illustrated in Figure 5D When the processing of step S612 in

[0134] is completed, or Figure 6A the processing of step S613 in Figure 6A is completed, the processing proceeds to step S614.

[0135] In Figure 6A step S614, the CPU 212 (user registration unit 504) stores the acquired information in Figure 5E the second authentication registration feature vector table 560 illustrated in

[0136] In Figure 6A steps S612 to S614, the acquired information is stored in the registered person information table 530, the first authentication registration right eye feature vector table 540, the first authentication registration left eye feature vector table 550, and the second authentication registration feature vector table 560. Specifically, since the personal ID in the four tables 530 to 560 illustrated in Figures 5B to 5E is an ID intended to correlate the tables 530 to 560 with each other, the same ID value is used in the four tables 530 to 560. The name of the personal information acquired in step S601 is added to the registered person information table 530 illustrated in Figure 6A Figure 5B the feature vector acquired in step S606 is divided and stored in the first authentication registration right eye feature vector table 540, the first authentication registration left eye feature vector table 550, and the second authentication registration feature vector table 560 illustrated in Figure 6A Figure 5C ​​The first authentication registration right eye feature vector table 540, Figure 5D The first authentication registration left eye feature vector table 550, and Figure 5E The second authentication registration feature vector table 560. The feature vectors are divided in the following manner.

[0137] In the first authentication process (to be described below Figure 8A and Figure 8B , the index is displayed on the display device 214, and the authentication is performed using the eye images of both eyes while the user is looking at the index. For this reason, only the feature vectors having the index to be displayed in this process are stored in the first authentication registration right eye feature vector table 540 or the first authentication registration left eye feature vector table 550 as the registration feature vectors to be used in the first authentication. In the present exemplary embodiment, if the feature vector acquired using the displayed index 411 is acquired from the right eye, the feature vector is registered in the first authentication registration right eye feature vector table 540 in step S612 of Figure 6A . If the feature vector acquired using the displayed index 411 is acquired from the left eye, the feature vector is registered in the first authentication registration left eye feature vector table 550 in step S613 of Figure 6A .

[0138] On the other hand, in the second authentication process (to be described below Figure 9 , the display device 214 displays the image being captured using the image sensor 211 without any index. For this reason, the position at which the user gazes on the display device 214 is unknown. Therefore, in the present exemplary embodiment, all the feature vectors acquired using the displayed indexes 411 to 415 are registered in step S614 of Figure 6A .

[0139] When the process in step S614 of Figure 6A is completed, the process proceeds to step S615 of Figure 6B .

[0140] In step S615 of Figure 6B , the CPU 212 (user registration unit 504) displays a method for registering the eye image of the non-dominant eye to the user. Specifically, the CPU 212 (user registration unit 504) displays an instruction to the user on the touch screen (operation member 123) to look at the viewfinder using the eye on the opposite side (non-dominant eye) of the dominant eye used by the user when capturing the image. The CPU 212 (user registration unit 504) displays an instruction to look at the index in the viewfinder. In addition, the CPU 212 (user registration unit 504) can display an instruction such as not to blink and keep the eyes wide open for capturing the desired eye image.

[0141] exist Figure 6B In step S616, the CPU 212 (user registration unit 504) displays indicators on the display device 214. Figure 6B The specific processing of step S616 in the process and Figure 6A The processing of step S603 is similar. Therefore, its description will be omitted.

[0142] exist Figure 6B In step S617, the eye image acquisition unit 501 acquires an eye image when the user views the viewfinder (eyepiece lens 122). Figure 6B The specific processing of step S617 in the process and Figure 6A The processing in step S604 ( Figure 7 The processing of the illustrated flowchart is similar. Therefore, its description will be omitted.

[0143] exist Figure 6B In step S618, the eye image acquisition unit 501 determines whether the eye image has been successfully acquired. Specifically, the eye image acquisition unit 501 is based on... Figure 7 The flag recorded in step S705 or S708 is used to determine whether the eye image has been successfully acquired.

[0144] If in Figure 6B In step S618, the eye image acquisition unit 501 determines that the eye image was not successfully acquired (acquisition failure) (in step S618, it is "no"), and then the process proceeds to step S619.

[0145] exist Figure 6B In step S619, the CPU 212 (user registration unit 504) displays on the display device 214 that capturing an eye image using the indicators displayed on the display device 214 has failed. For example, the CPU 212 (user registration unit 504) may display a "Failed to capture eye image" message on the display device 214, or display an icon indicating failure. If the processing in step S619 is completed, the process returns to step S617.

[0146] If in Figure 6B If the eye image acquisition unit 501 determines that the eye image has been successfully acquired in step S618 (the value in step S618 is "yes"), then the process proceeds to step S620.

[0147] exist Figure 6B In step S620, the CPU 212 (user registration unit 504) determines whether the eye in the eye image obtained in step S617 is on the opposite side of the dominant eye in the eye image obtained in step S604. The specific determination method of step S620 is similar to that of step S611.

[0148] If in Figure 6B In step S620, if the CPU 212 (user registration unit 504) determines that the eye in the eye image obtained in step S617 is the opposite side of the dominant eye in the eye image obtained in step S604 ("Yes" in step S620), then the process proceeds to step S621.

[0149] exist Figure 6B In step S621, the feature vector calculation unit 502 extracts feature vectors from the eye image as authentication registration information for user authentication. Specifically, the feature vector calculation unit 502 extracts feature vectors from the eye image. Figure 6B The feature vector is extracted from the eye image obtained in step S617.

[0150] exist Figure 6B In step S622, the eye image acquisition unit 501 determines whether the person viewing the viewfinder in step S604 and the person viewing the viewfinder in step S617 are the same. The specific determination method for step S622 includes, for example, determining the cosine similarity between the feature vector extracted in step S606 and the feature vector extracted in step S621, and if the determined cosine similarity exceeds a predetermined threshold, then the person is determined to be the same.

[0151] If in Figure 6B In step S622, the eye image acquisition unit 501 determines that the person viewing the viewfinder in step S604 is different from the person viewing the viewfinder in step S617 ("No" in step S622), then the process proceeds to step S623. If in Figure 6B In step S620, if the CPU 212 (user registration unit 504) determines that the eye in the eye image obtained in step S617 is not on the opposite side of the dominant eye in the eye image obtained in step S604 ("No" in step S620), then the process proceeds to step S623.

[0152] exist Figure 6B In step S623, the CPU 212 (user registration unit 504) displays on the display device 214 that the acquired eye image is not suitable for authentication. For example, the CPU 212 (user registration unit 504) may display a message "Eye image is not suitable for authentication" on the display device 214. Upon completion of step S623, the process returns to step S617.

[0153] If in Figure 6BWhen the eye image acquisition unit 501 determines in step S622 that the person who looked into the viewfinder in step S604 is the same as the person who looked into the viewfinder in step S617 (YES in step S622), the process proceeds to step S624.

[0154] In step S624, the CPU 212 (user registration unit 504) displays on the display device 214 that the eye image was successfully captured using the indicator displayed on the display device 214. For example, the CPU 212 (user registration unit 504) can display a message "eye image successfully captured" on the display device 214, or display an icon indicating success. Figure 6B

[0155] In step S625, the CPU 212 (user registration unit 504) determines whether the eye image acquired in step S617 is an eye image acquired from the right eye. Figure 6B

[0156] If the CPU 212 (user registration unit 504) determines in step S625 that the eye image acquired in step S617 is an eye image acquired from the right eye (YES in step S625), the process proceeds to step S626. Figure 6B

[0157] In step S626, the CPU 212 (user registration unit 504) stores the acquired information in the first authentication registration right eye feature vector table 540 illustrated in the example to update it. Figure 6B Figure 5C If the CPU 212 (user registration unit 504) determines in step S625 that the eye image acquired in step S617 is not an eye image acquired from the right eye (is an eye image acquired from the left eye) (NO in step S625), the process proceeds to step S627.

[0158] In step S627, the CPU 212 (user registration unit 504) stores the acquired information in the first authentication registration left eye feature vector table 550 illustrated in the example to update it. Figure 6B

[0159] When the process of step S626 in the example of FIG. 6A is completed, or when the process of step S627 in the example of FIG. 6B is completed, the process proceeds to step S628. Figure 6B Figure 5D In step S628, the CPU 212 (user registration unit 504) determines whether the eye image acquisition unit 501 has acquired an eye image from the right eye in step S617.

[0160] When the eye image acquisition unit 501 has acquired an eye image from the right eye in step S617 (YES in step S628), the process proceeds to step S629. Figure 6B Figure 6B When the eye image acquisition unit 501 has not acquired an eye image from the right eye in step S617 (NO in step S628), the process proceeds to step S630.

[0161] In step S629, the CPU 212 (user registration unit 504) stores the acquired information in the first authentication registration right eye feature vector table 540 illustrated in the example to update it. Figure 6B ​​​​​​​In step S628, the CPU 212 (user registration unit 504) provides display on the touch screen (operation member 123) or the display device 214 to inform the user of the completion of registration.

[0162] In Figure 6B the processing of step S628 is completed, Figure 6A and Figure 6B the processing of the flowchart ends.

[0163] Unless the eye image is successfully acquired in steps S605 and S618, Figure 6A and Figure 6B the registration processing exemplified falls into an infinite loop. Therefore, if a predetermined number of failures are observed, the registration processing desirably is configured to be interrupted.

[0164] [First authentication processing]

[0165] Figure 8A and Figure 8B is a flowchart exemplifying a detailed process for the first authentication processing used in a method for controlling the camera 100 corresponding to the information processing device according to the first typical embodiment. Figure 8A and Figure 8B The processing of the flowchart exemplified is mainly performed by the first authentication unit 507 on the CPU 212. Figure 8A and Figure 8B The first authentication processing exemplified is expected to be performed by the user operating the camera 100 at a time other than the time of imaging. Thus, Figure 8A and Figure 8B The processing of the flowchart exemplified is performed when the user operates the camera 100 and calls the processing from a menu. For example, an unexemplified menu screen is displayed on the touch screen (operation member 123) of the camera 100, and when the user operates the menu screen using the operation members 123 to 125 and selects a menu for calling the processing, the processing is performed.

[0166] In Figure 8A step S801, the CPU 212 (first authentication unit 507) issues an instruction to the user about the authentication method using the eye image of one of the eyes. Specifically, the CPU 212 (first authentication unit 507) displays an instruction on the touch screen (operation member 123) to the user to view the viewfinder with one of the left eye and the right eye and look at an index displayed on the display device 214. In addition, the CPU 212 (user registration unit 504) can display an instruction such as not to blink, keep the eyes wide open, and firmly hold the camera 100 for capturing a desired eye image.

[0167] In Figure 8AIn step S802, the CPU 212 (first authentication unit 507) displays indicators on the display device 214. Specifically, as shown... Figure 4D As illustrated, CPU 212 only displays indicator 411. This is because, in the registration process described above, the feature vector when viewing indicator 411 is registered in... Figure 5C The first authentication registration right eye feature vector table 540 and Figure 5D The example of the left eye feature vector in the first authentication registration is shown in Table 550. This allows for the acquisition of eye images with similar gazes during registration and authentication. This facilitates the comparison of eye images.

[0168] exist Figure 8A In step S803, the eye image acquisition unit 501 acquires an eye image when the user views the viewfinder (eyepiece lens 122). Figure 8A The specific processing of step S803 in the process and Figure 6A The processing in step S604 ( Figure 7 The processing of the illustrated flowchart is similar. Therefore, its description will be omitted.

[0169] exist Figure 8A In step S804, the eye image acquisition unit 501 determines whether the eye image has been successfully acquired. Specifically, the eye image acquisition unit 501 is based on... Figure 7 The flag recorded in step S705 or S708 is used to determine whether the eye image has been successfully acquired.

[0170] If in Figure 8A In step S804, the eye image acquisition unit 501 determines that the eye image has not been successfully acquired (acquisition failure) (in step S804, it is "no"), and then the process proceeds to step S805.

[0171] exist Figure 8A In step S805, the CPU 212 (first authentication unit 507) displays a message on the display device 214 indicating that the acquisition of an eye image using the indicators displayed on the display device 214 has failed. For example, the CPU 212 (first authentication unit 507) may display a message "Failed to capture eye image" on the display device 214, or display an icon indicating failure. If the processing in step S805 is completed, the process returns to step S803.

[0172] If in Figure 8A In step S804, if the eye image acquisition unit 501 determines that the eye image has been successfully acquired (the value is "yes" in step S804), then the process proceeds to step S806.

[0173] exist Figure 8AIn step S806, the feature vector calculation unit 502 extracts feature vectors from the eye image as authentication object information for user authentication. Specifically, the feature vector calculation unit 502 extracts feature vectors from the eye image. Figure 8A In step S803, the feature vector of the eye image is extracted.

[0174] exist Figure 8A In step S807, CPU 212 (first authentication unit 507) determines whether the eye image obtained in step S803 is an eye image obtained from the right eye.

[0175] If in Figure 8A In step S807, if the CPU 212 (first authentication unit 507) determines that the eye image obtained in step S803 is an eye image obtained from the right eye ("yes" in step S807), then the process proceeds to step S808.

[0176] exist Figure 8A In step S808, CPU 212 (first authentication unit 507) obtains data from the registration data management unit 505. Figure 5C The illustrated first authentication registration right eye feature vector table 540 obtains the registration feature vector for use in the first authentication. Specifically, the CPU 212 (first authentication unit 507) obtains... Figure 5C The first authentication registration right eye feature vector table 540 shows all the feature vectors.

[0177] If in Figure 8A In step S807, CPU 212 (first authentication unit 507) determines that the eye image obtained in step S803 is not an eye image obtained from the right eye (but an eye image obtained from the left eye) (No in step S807), and then the process proceeds to step S809.

[0178] exist Figure 8A In step S809, CPU 212 (first authentication unit 507) obtains data from the registration data management unit 505. Figure 5D The illustrated first authentication registration left eye feature vector table 550 obtains the registration feature vector for use in the first authentication. Specifically, the CPU 212 (first authentication unit 507) obtains... Figure 5D The example shows all the feature vectors in the left eye feature vector table 550 of the first authentication registration.

[0179] exist Figure 8A The processing in step S808 is completed, or Figure 8A When step S809 is completed, the process proceeds to step S810.

[0180] exist Figure 8AIn step S810, the CPU 212 (the first authentication unit 507) collates the feature vector that is the authentication target information acquired in step S806 with each of the registration feature vectors that is the authentication registration information acquired in step S808 or S809 to perform the first authentication. Specifically, in the processing of this step S810, the CPU 212 (the first authentication unit 507) determines the cos similarity between the two feature vectors, and performs the first authentication based on whether the determined cos similarity exceeds a predetermined threshold. More specifically, if the determined cos similarity exceeds the predetermined threshold, the CPU 212 (the first authentication unit 507) determines that the first authentication is successful, and identifies the personal ID of the registration feature vector.

[0181] In Figure 8A In step S811, the CPU 212 (the first authentication unit 507) determines whether the first authentication performed in step S810 is successful.

[0182] If in step S811 the CPU 212 (the first authentication unit 507) determines that the first authentication performed in step S810 is not successful (NO in step S811), the processing proceeds to step S812. Figure 8A If in step S811 the CPU 212 (the first authentication unit 507) determines that the first authentication performed in step S810 is successful (YES in step S811), the processing proceeds to step S812.

[0183] In Figure 8A In step S812, the CPU 212 (the first authentication unit 507) instructs the user to use the eye image of the contralateral eye of the eye in step S801 for authentication. Specifically, the CPU 212 (the first authentication unit 507) displays on the touch panel (the operation member 123) an instruction to the user to look through the viewfinder with the contralateral eye of the eye in the eye image acquired in step S803 and look at the index displayed on the display device 214. Further, the CPU 212 (the user registration unit 504) can display an instruction such as not to blink, keep the eyes open, and firmly hold the camera 100 for capturing the desired eye image.

[0184] In Figure 8A In step S813, the CPU 212 (the first authentication unit 507) displays the index on the display device 214. The detailed processing of this step S813 is similar to that of step S802.

[0185] In Figure 8B In step S814, the eye image acquisition unit 501 acquires the eye image when the user looks through the viewfinder (the ocular lens 122). As in step S803, Figure 8A In step S814 in this embodiment, Figure 8B the detailed processing of this step S814 is similar to that of step S604 in Figure 6A Figure 7 ​The processing of the illustrated flowchart is similar. Thus, the explanation thereof will be omitted.

[0186] In Figure 8B Step S815, the eye image acquisition unit 501 determines whether the eye image is successfully acquired. Specifically, the eye image acquisition unit 501 determines whether the eye image is successfully acquired on the basis of the flag recorded in Step S705 or S708 of the processing of the illustrated flowchart. Figure 7

[0187] If the eye image acquisition unit 501 determines that the eye image is not successfully acquired (acquisition failure) in Step S815 of the processing of the illustrated flowchart (NO in Step S815), the processing proceeds to Step S816. Figure 8B

[0188] In Step S816, the CPU 212 (the first authentication unit 507) displays on the display device 214 that the eye image is not successfully captured with the index displayed on the display device 214. For example, the CPU 212 (the first authentication unit 507) can display a message of "the eye image is not successfully captured" on the display device 214, or display an icon indicating the failure. In the case where the processing of Step S816 is completed, the processing returns to Step S814.

[0189] If the eye image acquisition unit 501 determines that the eye image is successfully acquired in Step S815 of the processing of the illustrated flowchart (YES in Step S815), the processing proceeds to Step S817. Figure 8B

[0190] In Step S817 of the processing of the illustrated flowchart, the CPU 212 (the first authentication unit 507) determines whether the eye in the eye image acquired in Step S814 is the contralateral eye of the eye in the eye image acquired in Step S803. The specific determination method of this Step S817 is similar to that of Step S611. Figure 8B If the CPU 212 (the first authentication unit 507) determines that the eye in the eye image acquired in Step S814 is the contralateral eye of the eye in the eye image acquired in Step S803 in Step S817 of the processing of the illustrated flowchart (YES in Step S817), the processing proceeds to Step S818.

[0191] Figure 8B

[0192] In Step S818 of the processing of the illustrated flowchart, the feature vector calculation unit 502 extracts a feature vector from the eye image as authentication target information for authenticating the user. Specifically, the feature vector calculation unit 502 extracts a feature vector from the eye image acquired in Step S814 of the processing of the illustrated flowchart. Figure 8B Figure 8B If the eye image acquisition unit 501 determines that the eye image is not successfully acquired (acquisition failure) in Step S815 of the processing of the illustrated flowchart (NO in Step S815), the processing proceeds to Step S816.​​​​​​

[0193] In Figure 8B Step S819, the eye image acquisition unit 501 determines whether the person who viewed the viewfinder in Step S803 and the person who viewed the viewfinder in Step S814 are the same. The specific determination method of this Step S819 is similar to that of Step S622 in Figure 6B

[0194] If the eye image acquisition unit 501 determines in Step S819 in Figure 8B that the person who viewed the viewfinder in Step S803 and the person who viewed the viewfinder in Step S814 are not the same (NO in Step S819), the processing proceeds to Step S820. If the CPU 212 (the first authentication unit 507) determines in Step S817 in Figure 8B that the eye in the eye image acquired in Step S814 is not the opposite eye of the eye in the eye image acquired in Step S803 (NO in Step S817), the processing proceeds to Step S820.

[0195] In Figure 8B Step S820, the CPU 212 (the first authentication unit 507) displays on the display device 214 that the acquired eye image is not suitable for authentication. For example, the CPU 212 (the first authentication unit 507) can display a message of “eye image is not suitable for authentication” on the display device 214. In the case where the processing of Step S820 is completed, the processing returns to Step S814.

[0196] If the eye image acquisition unit 501 determines in Step S819 in Figure 8B that the person who viewed the viewfinder in Step S803 and the person who viewed the viewfinder in Step S814 are the same (YES in Step S819), the processing proceeds to Step S821.

[0197] In Figure 8B Step S821, the CPU 212 (the first authentication unit 507) determines whether the eye image acquired in Step S814 is an eye image acquired from the right eye.

[0198] If the CPU 212 (the first authentication unit 507) determines in Step S821 in Figure 8B that the eye image acquired in Step S814 is an eye image acquired from the right eye (YES in Step S821), the processing proceeds to Step S822.

[0199] In Figure 8B Step S822, the CPU 212 (the first authentication unit 507) acquires the right eye image from the Figure 5C ​The illustrated first authentication registration right eye feature vector table 540 acquires the registration feature vector to be used in the first authentication. Specifically, the CPU 212 (first authentication unit 507) acquires all the feature vectors in the first authentication registration right eye feature vector table 540. Figure 5C The illustrated first authentication registration right eye feature vector table 540 acquires the registration feature vector to be used in the first authentication. Specifically, the CPU 212 (first authentication unit 507) acquires all the feature vectors in the first authentication registration right eye feature vector table 540.

[0200] If the CPU 212 (first authentication unit 507) determines in step S821 of the first authentication that the eye image acquired in step S814 is not the eye image acquired from the right eye (is the eye image acquired from the left eye) (NO in step S821), the processing proceeds to step S823. Figure 8B

[0201] In step S823 of the first authentication, the CPU 212 (first authentication unit 507) acquires the registration feature vector to be used in the first authentication from the Figure 8B The illustrated first authentication registration right eye feature vector table 540 acquires the registration feature vector to be used in the first authentication. Specifically, the CPU 212 (first authentication unit 507) acquires all the feature vectors in the first authentication registration right eye feature vector table 540. Figure 5D The illustrated first authentication registration right eye feature vector table 540 acquires the registration feature vector to be used in the first authentication. Specifically, the CPU 212 (first authentication unit 507) acquires all the feature vectors in the first authentication registration right eye feature vector table 540. Figure 5D The illustrated first authentication registration right eye feature vector table 540 acquires the registration feature vector to be used in the first authentication. Specifically, the CPU 212 (first authentication unit 507) acquires all the feature vectors in the first authentication registration right eye feature vector table 540.

[0202] When the processing in step S822 of the first authentication is completed, or Figure 8B When the processing in step S823 of the first authentication is completed, the processing proceeds to step S824. Figure 8B In step S824 of the first authentication, the CPU 212 (first authentication unit 507) collates the feature vector to the authentication target information acquired in step S818 with each of the registration feature vectors of the authentication registration information acquired in step S822 or S823 to perform the first authentication. The detailed processing of this step S824 is similar to the detailed processing of step S810 of the first authentication.

[0203] Figure 8B In step S824 of the first authentication, the CPU 212 (first authentication unit 507) collates the feature vector to the authentication target information acquired in step S818 with each of the registration feature vectors of the authentication registration information acquired in step S822 or S823 to perform the first authentication. The detailed processing of this step S824 is similar to the detailed processing of step S810 of the first authentication. Figure 8A

[0204] In step S825 of the first authentication, the CPU 212 (first authentication unit 507) determines whether the first authentication performed in step S824 is successful. Figure 8B If the CPU 212 (first authentication unit 507) determines in step S825 of the first authentication that the first authentication performed in step S824 is successful (YES in step S825), the processing proceeds to step S826.

[0205] Figure 8B In step S826 of the first authentication, the authentication state management unit 521 sets the authentication state to the successful authentication state.

[0206] In step S826 of the first authentication, the authentication state management unit 521 sets the authentication state to the successful authentication state. Figure 8B Figure 5F ​​​​​The first authentication status in the illustrated authentication status table 570 is updated to "authenticated". The authentication status management unit 521 further... Figure 5F The second authentication status in the illustrated authentication status table 570 is updated to "unauthenticated".

[0207] exist Figure 8B In step S827, the authentication status management unit 521 updates the personal ID identified in step S824. Figure 5F The personal ID in the Authentication Status Table 570 shown.

[0208] exist Figure 8B In step S828, the CPU 212 (first authentication unit 507) uses the authentication status display unit 523 to provide a display on the touch screen (operation member 123) or display device 214 to inform the user that the authentication was successful.

[0209] If in Figure 8B In step S825, if the CPU 212 (first authentication unit 507) determines that the first authentication performed in step S824 was unsuccessful ("No" in step S825), then the process proceeds to step S829. If in Figure 8A In step S811, if the CPU 212 (first authentication unit 507) determines that the first authentication performed in step S810 was unsuccessful (in step S811 it is "no"), then the process proceeds to step S829.

[0210] In step S829, the authentication status management unit 521 will Figure 5F The first and second authentication statuses in the illustrated authentication status table 570 are updated to "unauthenticated".

[0211] exist Figure 8B In step S830, the authentication status management unit 521 from Figure 5F The personal ID is deleted from the illustrated authentication status table 570. For example, the authentication status management unit 521 can prepare a null value as a value to indicate emptiness and overwrite it with that null value. Figure 5F The personal ID in the Authentication Status Table 570 shown.

[0212] exist Figure 8B In step S831, the CPU 212 (first authentication unit 507) uses the authentication status display unit 523 to provide a display on the touch screen (operation member 123) or display device 214 to inform the user of authentication failure.

[0213] exist Figure 8B When step S828 in Figure 8 is completed, or when step S831 in Figure 8 is completed, Figure 8A and Figure 8B the flowchart ends.

[0214] Unless the eye image is successfully acquired in steps S804 and S815, Figure 8A and Figure 8B The illustrated first authentication process falls into an infinite loop. Therefore, if a predetermined number of failures are observed, the first authentication process is desirably configured to be interrupted.

[0215] [Second Authentication Process]

[0216] Figure 9 is a flowchart illustrating an example of a detailed process procedure of a second authentication process used in a method for controlling the camera 100 corresponding to the information processing apparatus according to the first typical embodiment. The Figure 9 The processing of the illustrated flowchart is mainly performed by the second authentication unit 508 on the CPU 212. Figure 9 The illustrated second authentication process is expected to be executed when the user views the viewfinder (eyepiece lens 122) at the time of imaging (during imaging). Thus, Figure 9 The processing of the illustrated flowchart is triggered by the detection of the user bringing his or her eye close to the viewfinder (eyepiece lens 122) by an eyepiece sensor (not illustrated) mounted on the camera 100. For example, the eyepiece sensor is a sensor that detects the contact of the skin in the vicinity of the user's eye with the vicinity of the eyepiece lens 122. Alternatively, the eyepiece sensor can be a sensor that detects the distance between the eyepiece lens 122 and the user's eye. In such a case, if the distance is less than or equal to a predetermined level, it can be determined that the user is viewing the viewfinder (eyepiece lens 122). Furthermore, Figure 9 The processing of the illustrated flowchart can be triggered by the detection of the release button 121 being pressed down to the first stroke. Alternatively, the line-of-sight detection process can be run in advance, and when the line-of-sight detection process is successful, the processing of the illustrated flowchart can be triggered. Figure 9 The processing of the illustrated flowchart.

[0217] In Figure 9 In step S901, the CPU 212 (second authentication unit 508) determines whether the first authentication is valid and whether the user is continuing imaging (during imaging). It is determined whether the first authentication is valid by confirming whether the first authentication state in the authentication state table 570 illustrated in Figure 5F It is determined whether the user is continuing imaging (during imaging) by confirming whether the user is keeping his or her eye close to the viewfinder (eyepiece lens 122) using the eyepiece sensor described above. It can be determined whether it is during imaging based on other methods such as the pressing of the release button 121 and the line-of-sight detection process.

[0218] If in step S901, it is determined that the first authentication is not valid and that the user is not continuing imaging (during imaging), Figure 9 In step S901, the CPU 212 (second authentication unit 508) determines that the first authentication is valid and the user is continuing imaging (during imaging) (YES in step S901), and the processing proceeds to step S902.

[0219] In Figure 9 In step S902, the eye image acquisition unit 501 acquires an eye image when the user is viewing the viewfinder (eyepiece lens 122). Figure 9 The detailed processing of this step S902 is similar to the processing of step S604 in Figure 6A the flowchart illustrated in Figure 7 Thus, the description thereof will be omitted. If the gaze detection processing is already running, the gaze detection in step S701 of Figure 7 may be skipped, and the result of the gaze detection processing that is already running can be used.

[0220] In Figure 9 In step S903, the eye image acquisition unit 501 determines whether or not the eye image has been successfully acquired. Specifically, the eye image acquisition unit 501 determines whether or not the eye image has been successfully acquired on the basis of the flag recorded in step S705 or S708 of Figure 7 .

[0221] If the eye image acquisition unit 501 determines that the eye image has not been successfully acquired (NO in step S903) (acquisition failure) in step S903 of Figure 9 , the processing proceeds to step S904.

[0222] In step S904 of Figure 9 , the CPU 212 (second authentication unit 508) displays on the display device 214 that the imaging of the eye image has failed. For example, the CPU 212 (second authentication unit 508) can display a message of "imaging of the eye image has failed" or display an icon for indicating the failure on the display device 214. In a case where the processing of step S904 is completed, the processing returns to step S901.

[0223] If the eye image acquisition unit 501 determines that the eye image has been successfully acquired (YES in step S903) in step S903 of Figure 9 , the processing proceeds to step S905.

[0224] In step S905 of Figure 9 , the CPU 212 (second authentication unit 508) determines whether or not the eye image acquired in step S902 is an eye image of an eye that is registered as a dominant eye in the registration processing.

[0225] If the eye image acquired in step S902 is not an eye image of an eye that is registered as a dominant eye in the registration processing (NO in step S905), the processing proceeds to step S906. Figure 9 In step S905, if the CPU 212 (second authentication unit 508) determines that the eye image obtained in step S902 is not the eye image of the dominant eye registered in the registration process ("No" in step S905), then the process proceeds to step S906.

[0226] exist Figure 9 In step S906, the CPU 212 (second authentication unit 508) displays on the display device 214 that the acquired eye image is not suitable for authentication. For example, the CPU 212 (second authentication unit 508) may display a message "eye image is not suitable for authentication" on the display device 214. Upon completion of step S906, the process returns to step S901.

[0227] If in Figure 9 In step S905, if the CPU 212 (second authentication unit 508) determines that the eye image obtained in step S902 is the eye image of the eye that was registered as the dominant eye in the registration process (the value is "yes" in step S905), then the process proceeds to step S907.

[0228] exist Figure 9 In step S907, the feature vector calculation unit 502 extracts feature vectors from the eye image as authentication object information for user authentication. Specifically, the feature vector calculation unit 502 extracts feature vectors from the eye image. Figure 9 The feature vector is extracted from the eye image obtained in step S902.

[0229] exist Figure 9 In step S908, the CPU 212 (second authentication unit 508) obtains the data via the registration data management unit 505. Figure 5E All feature vectors in the illustrated second authentication registration feature vector table 560.

[0230] exist Figure 9 In step S909, the CPU 212 (second authentication unit 508) compares the feature vectors of the authentication object information obtained in step S907 with the registration feature vectors of the authentication registration information obtained in step S908 to perform a second authentication. Specifically, the CPU 212 (second authentication unit 508) determines the cosine similarity between the two feature vectors and performs the second authentication based on whether the determined cosine similarity exceeds a predetermined threshold. More specifically, if the determined cosine similarity exceeds the predetermined threshold, the CPU 212 (second authentication unit 508) determines that the second authentication is successful.

[0231] exist Figure 9In step S910, the CPU 212 (the second authentication unit 508) determines whether the second authentication performed in step S909 was successful.

[0232] If in step S909 the CPU 212 (the second authentication unit 508) determines that the second authentication was not successful (NO in step S909), the process proceeds to step S911. Figure 9 In step S910, the CPU 212 (the second authentication unit 508) determines that the second authentication performed in step S909 was successful (YES in step S910), the process proceeds to step S911.

[0233] In step S911, the authentication status management unit 521 updates the second authentication status in the authentication status table 570 illustrated in FIG. 7 to "authenticated". Figure 9 Figure 5F In step S911, the authentication status management unit 521 updates the second authentication status in the authentication status table 570 illustrated in FIG. 7 to "authenticated".

[0234] In step S912, the CPU 212 (the second authentication unit 508) determines whether the user is continuing to perform image capturing (during image capturing) using the authentication status display unit 523. Figure 9 In step S912, the CPU 212 (the second authentication unit 508) determines whether the user is continuing to perform image capturing (during image capturing) using the authentication status display unit 523.

[0235] Figure 9 In step S913, the CPU 212 (the second authentication unit 508) determines whether the user is continuing to perform image capturing (during image capturing) using the authentication status display unit 523.

[0236] In step S913, the CPU 212 (the second authentication unit 508) determines whether the user is continuing to perform image capturing (during image capturing) using the authentication status display unit 523. FIG. 9 In step S913, the CPU 212 (the second authentication unit 508) determines whether the user is continuing to perform image capturing (during image capturing) using the authentication status display unit 523.

[0237] FIG. 9 In step S913, the CPU 212 (the second authentication unit 508) determines whether the user is continuing to perform image capturing (during image capturing) using the authentication status display unit 523.

[0238] In step S914, the authentication status management unit 521 updates the second authentication status in the authentication status table 570 illustrated in FIG. 7 to "not authenticated". FIG. 9 FIG. 5F In step S914, the authentication status management unit 521 updates the second authentication status in the authentication status table 570 illustrated in FIG. 7 to "not authenticated".

[0239] In step S914, the authentication status management unit 521 updates the second authentication status in the authentication status table 570 illustrated in FIG. 7 to "not authenticated". FIG. 9 ​​​​In step S910, if the CPU 212 (second authentication unit 508) determines that the second authentication performed in step S909 was unsuccessful (failed) (in step S910, it is "no"), then the process proceeds to step S915.

[0240] exist FIG. 9 In step S915, the authentication status management unit 521 will FIG. 5F The second authentication status in the illustrated authentication status table 570 is updated to "unauthenticated".

[0241] exist FIG. 9 In step S916, other individuals use the detection unit 509 to perform processing for detecting the use of the camera 100 by other individuals. Specifically, the other individuals use the detection unit 509 to detect whether they suspect that the camera 100 is being used by individuals other than those certified by the first authentication (other individuals).

[0242] exist FIG. 9 In step S917, the first authentication status invalidation unit 510 determines whether the use of other people was detected in step S916.

[0243] If in FIG. 9 In step S917, if the first authentication status invalidation unit 510 determines that the use of other people was detected in step S916 ("yes" in step S917), then the process proceeds to step S918.

[0244] exist FIG. 5F In step S918, the first authentication state invalidation unit 510 will FIG. 9 The first authentication status in the illustrated authentication status table 570 is updated to "unauthenticated," and the personal ID is further deleted via the authentication status management unit 521. This allows the first authentication to be invalidated if the use of another person is suspected.

[0245] exist FIG. 9 When step S918 is completed, the process proceeds to step S919. If in FIG. 9 In step S917, the first authentication status invalidation unit 510 determines that no other person's use was detected in step S916 (no in step S917), and then the process proceeds to step S919.

[0246] exist FIG. 9 In step S919, the CPU 212 (second authentication unit 508) uses the authentication status display unit 523 to provide a display on the display device 214 informing the user of authentication failure. Upon completion of step S919, the process returns to step S901.

[0247] exist FIG. 9When step S914 is completed, the process proceeds to step S920.

[0248] If in FIG. 9 In step S901, if the CPU 212 (second authentication unit 508) determines that the first authentication is not valid or the user is not continuing to record (not during recording) (in step S901, it is "No"), then the process proceeds to step S920.

[0249] exist FIG. 9 In step S920, the CPU 212 (second authentication unit 508) uses the authentication status display unit 523 to update the display on the display device 214. For example, if the camera is no longer recording, the authentication status display unit 523 stops displaying the authentication status. For example, if the camera is recording but the first authentication status is "unauthenticated", the authentication status display unit 523 displays that the first authentication status is "unauthenticated" by using a message or icon.

[0250] exist FIG. 10 If step S920 is completed, FIG. 9 The processing of the flowchart has ended.

[0251] [Other characters are subject to detection processing]

[0252] FIG. 10 This is an example FIG. 10 The flowchart illustrates an example of the detailed processing procedure used by other characters in step S916 for detection processing. FIG. 9 The processing of the illustrated flowchart is mainly performed by other personnel using the detection unit 509 on the CPU 212.

[0253] exist FIG. 10 In step S1001, other individuals use detection unit 509 to record the failure of the second authentication. For example, here, other individuals use detection unit 509 to record the moment of failure and the similarity score at that time.

[0254] exist FIG. 10In step S1002, the other person use detection unit 509 analyzes the history of second authentication failures recorded in step S1001 to indicate any pattern of use by other people. Specifically, if the number of second authentication failures within a predetermined recent time range exceeds a threshold, the other person use detection unit 509 determines that another person is using camera 100. Here, the other person use detection unit 509 may count failures occurring during the same video recording session as a single failure. Alternatively, the other person use detection unit 509 may only count failures with similarity below a predetermined threshold. Furthermore, although the other person use detection unit 509 only records failures in step S1001, it may also record successes. For example, the other person use detection unit 509 may record successes immediately after the failures in step S1001. FIG. 10 Before step S911, the success of the second authentication is recorded. If the second authentication is successful at least once in the same camera session, other people using the detection unit 509 can be configured not to count the failures in that camera session.

[0255] exist FIG. 9 In step S1003, other characters use detection unit 509 to determine whether to suspect the use of other characters based on the analysis in step S1002 and the history of failure of the second authentication.

[0256] If in FIG. 10 If, in step S1003, the detection unit 509 determines, based on the history of the second authentication failure, that there is no suspicion of the use of other characters (in step S1003, it is "No"), then the process proceeds to step S1004.

[0257] exist FIG. 9 In step S1004, other characters use detection unit 509 to determine... FIG. 10 The step S909 involves determining whether the maximum similarity obtained during the comparison period is less than a predetermined threshold. Specifically, in step S909, other individuals use detection unit 509 to obtain the similarity of multiple registered feature vectors with the same person ID. In this step, other individuals use detection unit 509 to obtain the highest similarity among the obtained similarities and determine whether this similarity is less than a predetermined threshold. The reason is that under poor camera conditions, the similarity to the same person may decrease. However, compared to the similarity to other individuals, such a similarity still tends to be high. In this step, the threshold is set such that it can be determined that the person is obviously someone else, and if the maximum similarity drops below this threshold, it is determined to be another person.

[0258] If in FIG. 10 In step S1004, other characters are determined by detection unit 509 to be in FIG. 9If the maximum similarity obtained during the comparison period in step S909 is not less than a predetermined threshold ("No" in step S1004), then the process proceeds to step S1005.

[0259] exist FIG. 10 In step S1005, the other characters use detection unit 509 determines that no other characters are using camera 100, and does not record the use of other characters (records that no other characters are using the camera). Specifically, the other characters use detection unit 509 stores a flag indicating the use of other characters in memory unit 213 and turns the flag off.

[0260] If in FIG. 10 In step S1004, other characters are determined by detection unit 509 to be in FIG. 10 If the maximum similarity obtained during the comparison period in step S909 is less than a predetermined threshold ("Yes" in step S1004), then the process proceeds to step S1006. If in FIG. 10 If, in step S1003, other characters use detection unit 509 to determine that, based on the history of the second authentication failure, they suspect the use of other characters (in step S1003, this is "yes"), then the process proceeds to step S1006.

[0261] exist FIG. 10 In step S1006, the other characters use detection unit 509 to determine that other characters are using camera 100 and to record their use. Specifically, the other characters use detection unit 509 turns on the flag in memory unit 213 used to indicate the use of other characters.

[0262] exist FIG. 11A to FIG. 11D The processing of step S1005 in the process is completed, or FIG. 11A to FIG. 11D When step S1006 is completed, FIG. 11A The processing of the flowchart has ended.

[0263] [First Authentication Invalidation Process]

[0264] FIG. 11A This is a flowchart illustrating an example of a detailed processing procedure used in a method for controlling a camera 100 corresponding to an information processing device according to this exemplary embodiment, specifically for a first authentication invalidation process. FIG. 11A The four types of first authentication invalidation processes illustrated are mainly performed by the first authentication status invalidation unit 510 on the CPU 212.

[0265] FIG. 5F This is a flowchart illustrating an example of the detailed processing procedure used in the first authentication invalidation process based on time. FIG. 11AThe illustrated flowchart assumes that the processing relies on the periodic activation of a timer.

[0266] exist FIG. 11A In step S1101, the first authentication status invalidation unit 510 determines whether the first authentication was successful. This is based on the authentication status managed by the authentication status management unit 521. FIG. 5F The success of the first authentication is determined by whether the first authentication status in the illustrated authentication status table 570 is "authenticated".

[0267] If in FIG. 5F If the first authentication status invalidation unit 510 determines that the first authentication is successful in step S1101 (the value is "yes" in step S1101), then the process proceeds to step S1102.

[0268] exist FIG. 11A In step S1102, the first authentication status invalidation unit 510 calculates the time elapsed since the first authentication was successful. In this typical embodiment, the first authentication status invalidation unit 510 calculates the time elapsed since the first authentication was successful. FIG. 11A The time elapsed since the first authentication status of the illustrated authentication status table 570 changed to "authenticated". The first authentication status invalidation unit 510 can record the time elapsed since the first authentication status changed to "authenticated". FIG. 11A The time elapsed since the first authentication was successful is calculated by taking the moment when the first authentication status in the illustrated authentication status table 570 changes to "authenticated" and calculating the difference between the recorded moment and the current moment.

[0269] exist FIG. 11A In step S1103, the first authentication state invalidation unit 510 detects the execution of the second authentication using the second authentication unit 508. In this typical embodiment, the first authentication state invalidation unit 510 detects whether the second authentication was performed between the last time the process was triggered using the timer and the current time the process is triggered.

[0270] If in FIG. 11A If the first authentication status invalidation unit 510 determines that the execution of the second authentication has been detected in step S1103 ("Yes" in step S1103), then the process proceeds to step S1104. If the first authentication status invalidation unit 510 determines that the execution of the second authentication has not been detected ("No" in step S1103), then the process proceeds to step S1107.

[0271] exist FIG. 5F In step S1104, the first authentication status invalidation unit 510 determines whether the second authentication detected in step S1103 was successful.

[0272] If in FIG. 11AIn step S1104, if the first authentication status invalidation unit 510 determines that the second authentication was successfully performed in step S1103 ("Yes" in step S1104), then the process proceeds to step S1105.

[0273] exist FIG. 11A In step S1105, the first authentication status invalidation unit 510 adds a predetermined time to the expiration time. FIG. 11A When the first authentication status in the illustrated authentication status table 570 changes to "authenticated", the initial value of the expiration time should be initialized to the predetermined expiration time value.

[0274] If in FIG. 11A In step S1104, if the first authentication status invalidation unit 510 determines that the second authentication was not successful (failed) in step S1103 (no in step S1104), then the process proceeds to step S1106.

[0275] exist FIG. 11A In step S1106, the first authentication status invalidation unit 510 subtracts the predetermined time from the aforementioned expiration time.

[0276] exist FIG. 11A The processing of step S1105 in the process is completed, or FIG. 11A When step S1106 is completed, the process proceeds to step S1107.

[0277] exist FIG. 5F In step S1107, the first authentication status invalidation unit 510 determines whether the elapsed time calculated in step S1102 has exceeded the currently obtained expiration time.

[0278] If in FIG. 11A In step S1107, the first authentication status invalidation unit 510 determines that the elapsed time calculated in step S1102 has exceeded the currently obtained expiration time ("yes" in step S1107), and then the process proceeds to step S1108.

[0279] exist FIG. 11A In step S1108, the first authentication status invalidation unit 510, via the authentication status management unit 521, invalidates the authentication status. FIG. 11A The first and second authentication statuses in the illustrated authentication status table 570 are updated to "unauthenticated", and the personal ID is further deleted.

[0280] exist FIG. 11B When step S1108 is completed, FIG. 11B The processing of the flowchart is complete. If in FIG. 11BIn step S1101, the first authentication status invalidation unit 510 determines that the first authentication was unsuccessful (failed) (in step S1101, it is "No"). FIG. 5F The processing of the flowchart has ended.

[0281] FIG. 11B This is a flowchart illustrating an example of the detailed processing procedure used in the first authentication invalidation process based on a change in power supply state. FIG. 11B The illustrated flowchart assumes that the processing is activated when the power state changes. Examples of power state changes include when the camera 100 is powered on, when the camera 100 is powered off, when the camera 100 enters sleep mode, and when the camera 100 resumes from sleep mode.

[0282] exist FIG. 11B In step S1111, the first authentication status invalidation unit 510 determines whether the first authentication was successful. This is based on the authentication status managed by the authentication status management unit 521. FIG. 11B The success of the first authentication is determined by whether the first authentication status in the illustrated authentication status table 570 is "authenticated".

[0283] If in FIG. 11B If the first authentication status invalidation unit 510 determines that the first authentication is successful ("Yes" in step S1111), then the process proceeds to step S1112.

[0284] exist FIG. 11B In step S1112, the first authentication state invalidation unit 510 determines whether the power state has changed. In step S1112, if there is at least one change in the power state (from on to off or from off to on), the power state is determined to have changed.

[0285] If in FIG. 5F In step S1112, the first authentication state invalidation unit 510 determines that the power state has not changed (no in step S1112), and then the process proceeds to step S1113.

[0286] exist FIG. 11B In step S1113, the first authentication state invalidation unit 510 determines whether the sleep state has changed. In step S1113, if the camera 100 has at least entered or resumed from sleep mode, the sleep state is determined to have changed.

[0287] If in FIG. 11B In step S1113, the first authentication state invalidation unit 510 determines that the dormant state has changed (in step S1113, it is "yes"), and then the process proceeds to step S1114.

[0288] If in FIG. 11B In step S1112, the first authentication state invalidation unit 510 determines that the power state has changed ("yes" in step S1112), and then the process proceeds to step S1114.

[0289] In step S1114, the first authentication status invalidation unit 510, via the authentication status management unit 521, invalidates the authentication status. FIG. 11B The first and second authentication statuses in the illustrated authentication status table 570 are updated to "unauthenticated", and the personal ID is further deleted.

[0290] exist FIG. 11B When step S1114 is completed, FIG. 11B The processing of the flowchart is complete. If in FIG. 11C In step S1113, the first authentication state invalidation unit 510 determines that the dormant state has not changed (no in step S1113). FIG. 11C The processing of the flowchart is complete. If in FIG. 11C In step S1111, the first authentication status invalidation unit 510 determines that the first authentication was unsuccessful (failed) (in step S1111, it is "No"). FIG. 11C The processing of the flowchart is also complete.

[0291] FIG. 5F This is a flowchart illustrating an example of the detailed processing procedure used in the first authentication invalidation process based on the distance from the device or the connection status with the device. FIG. 11C The illustrated flowchart relies on the periodic activation of a timer. FIG. 11C In the illustrated flowchart, the user-carried device and camera 100 will pre-process the communication connection between them. Examples include a user-carried smartphone and camera 100 pre-processing... The pairing process used for the connection.

[0292] exist FIG. 11C In step S1121, the first authentication status invalidation unit 510 determines whether the first authentication was successful. This is based on the authentication status managed by the authentication status management unit 521. FIG. 5F The success of the first authentication is determined by whether the first authentication status in the illustrated authentication status table 570 is "authenticated".

[0293] If in FIG. 11C If the first authentication status invalidation unit 510 determines that the first authentication is successful ("Yes" in step S1121), then the process proceeds to step S1122.

[0294] exist FIG. 11CIn step S1122, the first authentication state invalidation unit 510 determines whether the connection has deteriorated beyond a predetermined condition. Examples in this step include the radio wave strength of the communication connection having fallen below a predetermined level.

[0295] If in step S1122 the first authentication state invalidation unit 510 determines that the connection has deteriorated beyond a predetermined condition (YES in step S1122), the processing proceeds to step S1123. FIG. 11C

[0296] In step S1123, the first authentication state invalidation unit 510 invalidates the first authentication state via the authentication state management unit 521. FIG. 11C The first authentication state and the second authentication state in the authentication state table 570 illustrated are updated to "unauthenticated", and further, the personal ID is deleted. FIG. 11C

[0297] In step S1123 in the flowchart of FIG. 11, FIG. 11C the processing of the flowchart of FIG. 11 ends. If in step S1122 of FIG. 11 the first authentication state invalidation unit 510 determines that the connection has not deteriorated beyond a predetermined condition (NO in step S1122), FIG. 11D the processing of the flowchart of FIG. 11 ends. If in step S1121 of FIG. 11 the first authentication state invalidation unit 510 determines that the first authentication was not successful (failed) (NO in step S1121), FIG. 11D the processing of the flowchart of FIG. 11 also ends. FIG. 11D FIG. 5F FIG. 11D

[0298] FIG. 11D FIG. 12 is a flowchart illustrating an example of detailed processing procedures for the first authentication invalidation processing based on input of a user's explicit invalidation operation. This processing is performed in the flowchart of FIG. 11. FIG. 11D The processing of the flowchart illustrated is continuously performed at the time of power-on of the camera 100.

[0299] In step S1131 of FIG. 11, the first authentication state invalidation unit 510 determines whether the first authentication was successful. Whether the first authentication was successful is determined based on whether the first authentication state in the authentication state table 570 illustrated is "authenticated" which is managed by the authentication state management unit 521. FIG. 11D The first authentication state in the authentication state table 570 illustrated is updated to "unauthenticated", and further, the personal ID is deleted. FIG. 11D If in step S1131 of FIG. 11 the first authentication state invalidation unit 510 determines that the first authentication was successful (YES in step S1131), the processing proceeds to step S1132.

[0300] FIG. 5F

[0301] In step S1132, the first authentication state invalidation unit 510 invalidates the first authentication state via the authentication state management unit 521. FIG. 11D ​​​​​​​In step S1132, the first authentication state invalidation unit 510 waits for the invalidation operation by the user. An unillustrated switch button is arranged on the camera 100, and the invalidation operation in this step is deemed to be performed when the user presses the switch button. The invalidation operation can be made selectable from a menu displayed on a touch screen.

[0302] In step S1133, the first authentication state invalidation unit 510 determines whether the invalidation operation by the user is detected. FIG. 11D

[0303] If the first authentication state invalidation unit 510 determines in step S1133 that the invalidation operation by the user is not detected (NO in step S1133), the processing in the flowchart of FIG. 11 ends. FIG. 11D If the first authentication state invalidation unit 510 determines in step S1133 that the invalidation operation by the user is detected (YES in step S1133), the processing proceeds to step S1134.

[0304] In step S1134, the first authentication state invalidation unit 510 updates the first authentication state and the second authentication state in the authentication state table 570 illustrated in FIG. 5 to "unauthenticated" via the authentication state management unit 521, and further deletes the personal ID. FIG. 11D FIG. 11D In step S1134, the first authentication state invalidation unit 510 updates the first authentication state and the second authentication state in the authentication state table 570 illustrated in FIG. 5 to "unauthenticated" via the authentication state management unit 521, and further deletes the personal ID.

[0305] When the processing in step S1134 in the flowchart of FIG. 11 is completed, FIG. 11D the processing in the flowchart of FIG. 11 ends. If the first authentication state invalidation unit 510 determines in step S1133 that the invalidation operation by the user is not detected (NO in step S1133), FIG. 12A the processing in the flowchart of FIG. 11 ends. If the first authentication state invalidation unit 510 determines in step S1131 that the first authentication is not successful (failed) (NO in step S1131), FIG. 12A the processing in the flowchart of FIG. 11 also ends. FIG. 12B FIG. 12A [Authentication State Storage Processing] FIG. 12B

[0306] [Authentication State Storage Processing]

[0307] FIG. 12A is a flowchart illustrating an example of detailed processing procedure for controlling the authentication state storage processing in the method for controlling the camera 100 corresponding to the information processing apparatus according to the first typical embodiment. The FIG. 12A The processing in the flowchart illustrated in FIG. 12 is mainly performed by the authentication state storage unit 522 on the CPU 212. FIG. 12A is a diagram illustrating the first typical embodiment, which illustrates an example of the structure of the image file 1200 stored by the authentication state storage unit 522.

[0308] FIG. 12A ​​​​The processing of the illustrated flowchart is processing for generating and storing FIG. 5F The processing of the illustrated image file 1200. Specifically, FIG. 12A The processing of the illustrated flowchart is processing for storing the photographer information 1211, the hash value 1212, and the digital signature 1213 as metadata 1210 in association with the image data 1220, together with the image data 1220 related to the subject image taken by the user. When the release button 121 is pressed down to the second stroke, the processing of FIG. 12A The processing of the illustrated flowchart.

[0309] In FIG. 12A In step S1201, the authentication status storage unit 522 takes an image of a subject via the imaging unit 511. Specifically, for example, the imaging unit 511 performs an imaging process on the subject by converting light received by the image sensor 211 into an electric signal.

[0310] In FIG. 12A In step S1202, the authentication status storage unit 522 generates the image data 1220 related to the subject image via the imaging unit 511. Specifically, for example, the imaging unit 511 applies image processing such as development processing and encoding processing to the electric signal obtained by the imaging process of step S1201 to generate the image data 1220 related to the subject image.

[0311] In FIG. 12A In step S1203, the authentication status storage unit 522 generates the photographer information 1211 related to the user who took the image data 1220 related to the subject image. Specifically, the authentication status storage unit 522 acquires FIG. 12B The illustrated authentication status table 570. The authentication status storage unit 522 acquires the personal information corresponding to the personal ID via the registration data management unit 505. In the present typical embodiment, the "name" is acquired as the personal information. The authentication status storage unit 522 generates the photographer information 1211 including the name of the user, the first authentication status, the second authentication status, and information related to the presence or absence of use by other persons, from the information included in the authentication status table 570.

[0312] In FIG. 12A In step S1204, the authentication status storage unit 522 performs a hash function on the binary data of the image data 1220 related to the subject image and the binary data of the photographer information 1211 to generate the corresponding hash value 1212.

[0313] In FIG. 13In step S1205, the authentication status storage unit 522 generates a digital signature 1213. The digital signature 1213 includes information indicating a signature value, a signer, and a signature date and time. The signature value is generated by encrypting the hash value 1212 generated in step S1204 with a private key prepared in advance. A public key to be paired with the private key used here is also stored in the digital signature 1213. In the present exemplary embodiment, information indicating the manufacturer of the camera 100 is stored as the signer. Instead of the manufacturer, the model of the camera 100 can be used for the signer. The date and time at which the generation of the digital signature 1213 is completed is stored as the signature date and time.

[0314] In FIG. 13 In step S1206, the authentication status storage unit 522 generates an image file 1200 by attaching the photographer information 1211, the hash value 1212, and the digital signature 1213 as the metadata 1210 to the image data 1220 related to the subject image. If the image data 1220 is a still image, the image file 1200 is generated based on the Joint Photographic Experts Group (JPEG) format. If the image data 1220 is a moving image, the image file 1200 is generated based on the Moving Picture Experts Group (MPEG) format.

[0315] In FIG. 2 In step S1207, the authentication status storage unit 522 stores the image file 1200 generated in step S1206 to the memory unit 213. The memory unit 213 also includes a storage medium that is attachable to the camera 100 in a detachable manner, in which case the image file 1200 can be stored in the storage medium, for example. In the case where the processing of step S1207 is completed, FIG. 3 The processing of the flowchart ends.

[0316] The image file 1200 can be confirmed not to be tampered with by the following verification method. The hash value 1212 is first recovered from the signature value using the public key. Further, the hash value of the image data 1220 and the hash value of the photographer information 1211 are re-determined. If the recovered hash value 1212 and the re-determined hash value match, it can be judged that the image file 1200 is not tampered with. On the other hand, if the recovered hash value 1212 and the re-determined hash value do not match, it can be judged that the image file 1200 has been tampered with. Assume that someone tampers with the image data 1220. Since the signature value is encrypted with the private key, the person who tampers with the image data 1220 cannot modify the signature value. If the image data 1220 has been tampered with, the hash value determined from the image data 1220 and the recovered hash value thus do not match. In this way, data tampering can be detected.

[0317] InFIG. 13 In the illustrated example, the image file 1200 is stored including the hash value 1212. However, the image file 1200 can be configured not to include the hash value 1212 because the hash value 1212 can be recalculated from the image data 1220 and the photographer information 1211 included in the image file 1200.

[0318] In the case of a moving image, pressing the release button 121 downward to the second stroke causes the shooting of the moving image to start, and pressing the release button 121 downward to the second stroke again causes the shooting of the moving image to complete. The moving image data is generated by the processes of steps S1201 and S1202. FIG. 2 The hash value of the moving image data is calculated and stored instead of the hash value of the image data. The image data 1220 related to the moving image is stored together with the metadata 1210 as the image file 1200 of the moving image.

[0319] [Line-of-sight detection processing]

[0320] FIG. 13 is a diagram illustrating a first typical embodiment, a diagram for explaining the principle of line-of-sight detection processing of a user. In this FIG. 13 , components similar to those illustrated in FIG. 14A and FIG. 14B are denoted by the same reference numerals, and detailed explanation thereof will be omitted. FIG. 14A is a diagram illustrating an XYZ coordinate system corresponding to the XYZ coordinate system illustrated in FIG. 14A

[0321] As illustrated in FIG. 13 , the light sources 216a and 216b are located at positions substantially symmetrical about the optical axis of the light-receiving lens 218, and illuminate the user's eye E. A portion of light emitted from the light sources 216a and 216b and reflected at the user's eye E is converged on the eye image sensor 219 by the light-receiving lens 218. In FIG. 14B , the cornea 1310, the pupil 1320, and the iris 1330 are illustrated on the user's eye E.

[0322] FIG. 15 and FIG. 15 is a diagram illustrating a first typical embodiment, a diagram for explaining the principle of line-of-sight detection processing of a user.

[0323] Specifically, FIG. 15 is a diagram illustrating an eye image (an eye optical image projected on the eye image sensor 219) captured by the eye image sensor 219. In this FIG. 15 , components similar to those illustrated in FIG. 15 are denoted by the same reference numerals. FIG. 13 ​is a graph that illustrates the output intensity of the eye image sensor 219 from the luminance aspect. FIG. 14B is a flowchart that illustrates an example of a detailed process used for the line-of-sight detection process in the method for controlling the camera 100 corresponding to the information processing apparatus according to the first typical embodiment.

[0324] In FIG. 14A Step S1501 of FIG. 15A, the CPU 212 controls the driving of the light sources 216a and 216b via the light source driving circuit 305 so that infrared light is emitted toward the user's eye E. An optical image of the user's eye E illuminated by the infrared light is formed on the eye image sensor 219 by the light-receiving lens 218 and photoelectrically converted by the eye image sensor 219. A processable electrical signal of the eye image is thereby obtained.

[0325] In FIG. 14B Step S1502 of FIG. 15A, the CPU 212 acquires the eye image (eye image signal; electrical signal of the eye image) from the eye image sensor 219 via the line-of-sight detection circuit 301.

[0326] Through FIG. 14B the processes of Steps S1503 and S1504 of FIG. 15A, the CPU 212 acquires eye information about the position of the eye E with respect to the viewfinder from the eye image acquired in Step S1502.

[0327] Specifically, in FIG. 13 Step S1503 of FIG. 15A, the CPU 212 detects the corneal reflection images Pd and Pe of the light sources 216a and 216b and the coordinates of the points corresponding to the centers c of the pupils from the eye image acquired in Step S1502.

[0328] In FIG. 15 , the infrared light emitted from the light sources 216a and 216b illuminates the cornea 1310 of the user's eye E. Here, the corneal reflection images Pd and Pe formed by the portions of the infrared light reflected at the surface of the cornea 1310 are collected by the light-receiving lens 218 and converged on the eye image sensor 219, thereby forming the corneal reflection images Pd' and Pe' in the eye image. Similarly, the light beams from the ends a and b of the pupil 1320 are also converged on the eye image sensor 219, thereby forming the respective pupil end images a' and b' in the eye image. FIG. 15 is a graph that illustrates the luminance information (luminance distribution) about the region 1400 in the eye image of FIG. 15 FIG. 15B. FIG. 15 illustrates the luminance distribution along the X-axis direction, where the horizontal direction of the eye image is the X-axis direction and the vertical direction is the Y-axis direction.

[0329] In the first typical embodiment, the coordinates in the X-axis direction (horizontal direction) of the corneal reflection images Pd' and Pe' will be referred to as coordinates Xd and Xe, respectively. The coordinates in the X-axis direction of the pupil end images a' and b' will be referred to as coordinates Xa and Xb, respectively. As FIG. 15 As illustrated, a very high level of luminance is obtained at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. In a region from the coordinate Xa to the coordinate Xb corresponding to the region of the pupil 1320 (the region of the pupil image obtained by converging the light flux from the pupil 1320 onto the eye image sensor 219), a very low level of luminance is obtained except at the coordinates Xd and Xe. In the region of the iris 1330 outside the pupil 1320 (the region of the iris image obtained by converging the light flux from the iris 1330), a luminance between the above two types of luminance is obtained. For example, in a region where the X coordinate (the coordinate in the X-axis direction) is greater than the coordinate Xa and a region where the X coordinate is smaller than the coordinate Xb, a luminance between the above two types of luminance is obtained. According to the luminance distribution illustrated, the coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the coordinates Xa and Xb of the pupil end images a' and b' can be obtained. For example, coordinates of a very high level of luminance can be obtained as the coordinates of the corneal reflection images Pd' and Pe'. Coordinates of a very low level of luminance can be obtained as the coordinates of the pupil end images a' and b'. In FIG. 4B As illustrated, a very high level of luminance is obtained at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. In a region from the coordinate Xa to the coordinate Xb corresponding to the region of the pupil 1320 (the region of the pupil image obtained by converging the light flux from the pupil 1320 onto the eye image sensor 219), a very low level of luminance is obtained except at the coordinates Xd and Xe. In the region of the iris 1330 outside the pupil 1320 (the region of the iris image obtained by converging the light flux from the iris 1330), a luminance between the above two types of luminance is obtained. For example, in a region where the X coordinate (the coordinate in the X-axis direction) is greater than the coordinate Xa and a region where the X coordinate is smaller than the coordinate Xb, a luminance between the above two types of luminance is obtained. According to the luminance distribution illustrated, the coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the coordinates Xa and Xb of the pupil end images a' and b' can be obtained. For example, coordinates of a very high level of luminance can be obtained as the coordinates of the corneal reflection images Pd' and Pe'. Coordinates of a very low level of luminance can be obtained as the coordinates of the pupil end images a' and b'. In FIG. 4B In the first typical embodiment, the coordinates in the X-axis direction (horizontal direction) of the corneal reflection images Pd' and Pe' will be referred to as coordinates Xd and Xe, respectively. The coordinates in the X-axis direction of the pupil end images a' and b' will be referred to as coordinates Xa and Xb, respectively. As

[0330] In FIG. 4C Step S1504, the CPU 212 calculates the imaging magnification β of the eye image. The imaging magnification β is a magnification determined using the position of the eye E relative to the light-receiving lens 218, and can be calculated using a function of the distance ΔP = Xe - Xd between the corneal reflection images Pd' and Pe'.

[0331] In FIG. 15 Step S1505, the CPU 212 calculates the rotation angle of the optical axis of the eye E relative to the optical axis of the light-receiving lens 218. The X coordinate of the midpoint between the corneal reflection images Pd and Pe is approximately the same as the X coordinate of the center of curvature O of the cornea 1310. Thus, the rotation angle θx of the eye E in the ZX plane (a plane perpendicular to the Y axis) can be calculated by the following equation (1):

[0332] β×Oc×SIN(θx)≈{(Xd+Xe) / 2}-Xc...(1),

[0333] Where Oc is the standard distance from the center of curvature O of the cornea 1310 to the center c of the pupil 1320. The rotation angle θy of the eye E in the ZY plane (the plane perpendicular to the X-axis) can also be calculated using a method similar to that described above for calculating the rotation angle θx.

[0334] exist FIG. 15 In step S1506, the CPU 212 reads the gaze correction parameters stored in the memory unit 213. Specifically, the gaze correction parameters refer to... FIG. 8B The parameters Ax, Bx, Ay and By in equations (2) and (3) used in step S1507.

[0335] exist FIG. 8A In step S1507, CPU 212 uses the rotation angles θx and θy calculated in step S1505 to estimate the coordinates (Hx, Hy) of the user's gaze point on the screen of display device 214. Assuming that the coordinates (Hx, Hy) of the gaze point are the coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the gaze point can be calculated using the following equations (2) and (3):

[0336] Hx=m×(Ax×θx+Bx)...(2), and

[0337] Hy = m × (Ay × θy + By) ... (3).

[0338] The parameter m in equations (2) and (3) is a constant determined based on the structure of the optical system used for gaze detection, and is a conversion factor used to convert the rotation angles θx and θy into coordinates corresponding to the pupil center c on the screen of the display device 214. This parameter m is predetermined and stored in the memory unit 213. The parameters Ax, Bx, Ay, and By in equations (2) and (3) are gaze correction parameters read in step S1506 above.

[0339] The eye-tracking correction parameters will be explained.

[0340] Due to factors such as individual differences in the shape of the human eye (E), it may be difficult to estimate the fixation point with high accuracy. Specifically, for example... FIG. 9 As illustrated, there is a discrepancy between the actual fixation point B (410B) and the estimated fixation point C (410C). FIG. 9In this case, the user is gazing at the human figure, and the camera 100 erroneously estimates that the user is gazing at the background. In such a case, it is difficult to perform proper focus detection and adjustment. The gaze correction parameter is a parameter for correcting such a deviation. The gaze correction parameter can be obtained by calibration for gaze detection. For example, calibration is performed by displaying a plurality of indicators 411 to 415 at different positions on the screen of the display device 214 and having the user look at these indicators. The gaze detection operation is performed while gazing at each indicator, and the gaze correction parameter appropriate for the user is determined from the calculated plurality of fixation points (estimated positions) and the coordinates of the plurality of indicators. The method for displaying the indicators is not particularly limited as long as the position for the user to look at is suggested. A graphical representation of the indicators can be displayed. The indicators can be displayed by changing at least one of the brightness and the color of the image (e.g., a captured image). FIG. 15

[0341] The processing of Step S1507 of the flowchart of FIG. 15 ends. FIG. 9 The processing of the flowchart ends. FIG. 9 The processing of the flowchart ends.

[0342] [Left and right eye determination processing]

[0343] The left and right eye determination processing using the left and right eye determination unit 503 will be described.

[0344] The left and right eye determination processing is processing for determining which eye of the left eye and the right eye to acquire the eye image from. As described above, in the gaze detection processing used in the present exemplary embodiment, there is a deviation between the actual fixation point and the estimated fixation point. One of the reasons for this is that the fovea of the eye E is not located on the visual axis. The fovea refers to a central region of the macula of the retina of the eye E. The fovea is located at a position that is offset by 4 to 8 degrees from the visual axis toward the ear. This offset causes the estimated fixation point to be shifted from the actual fixation point toward the nose. In the present exemplary embodiment, the left and right eye determination unit 503 performs the left and right eye determination processing based on this shift. If the estimated fixation point is shifted to the left from the actual fixation point as viewed from the user, the eye from which the eye image is acquired can be determined to be the right eye. In contrast, if the estimated fixation point is shifted to the right, the eye from which the eye image is acquired can be determined to be the left eye. The specific method of the left and right eye determination processing is not limited thereto. For example, a neural network that outputs a determination result related to which eye of the left eye and the right eye the eye image is acquired from as input of the eye image can be constructed using the eye images acquired from the left eye and the right eye of the user as training data. Such a neural network can be used to determine which eye of the left eye and the right eye the eye image is acquired from.

[0345] [Effects of the present exemplary embodiment]

[0346] ​To ensure that an image or a moving image is taken by an intended person, authentication is required at the time of imaging. However, to ensure that imaging is not performed by another person (by the same person), authentication is required with a low false acceptance rate setting. Such a setting typically increases the false rejection rate. Thus, even when the same person takes an image, authentication at the time of imaging can fail. In the use case of photography, there is no second chance to take the same image. For example, for a professional photographer, a decisive moment in a sports event and an exclusive news is only a moment. Failure to perform authentication at the moment is a major problem. In other words, it is not ensured that a user (photographer) who takes a decisive moment is the photographer himself or herself.

[0347] In the present exemplary embodiment, by a first authentication at a time other than imaging (before imaging), authentication using a plurality of biological information (both eyes) is performed with a low false acceptance rate setting. Authentication using a plurality of biological information (both eyes) can further reduce the false acceptance rate. In addition, by a second authentication at the time of imaging (during imaging), authentication is performed with a low false rejection rate setting. This can suppress the possibility of false rejection during the second authentication while suppressing the possibility of false acceptance by the first authentication.

[0348] In the first authentication, the false rejection rate is high due to the low false acceptance rate setting.

[0349] Thus, the same person can be rejected at the time of the first authentication. However, since imaging is not in progress, authentication can be retried. This means that there is no problem in terms of use. In the second authentication, the possibility of false acceptance increases. In this regard, in the present exemplary embodiment, the first authentication state is invalidated under various conditions to prevent the possibility of acceptance of another person. More specifically, if use of another person is suspected during the second authentication, the first authentication is invalidated. The first authentication is also invalidated in accordance with acceptance of a user explicit invalidation operation, a change in the elapsed time since the first authentication succeeded, a change in the power state, a distance from a peripheral device, or a change in the connection state with the peripheral device. Thereby, the possibility of use of another person is reduced, and thus acceptance of another person at the time of the second authentication is suppressed.

[0350] Further, in the present exemplary embodiment, the results of the first authentication and the second authentication are recorded as the photographer information 1211 in the metadata 1210 of the image file 1200, respectively. If only the first authentication is successful and the second authentication fails, the success of the first authentication is thus recorded in the metadata 1210. If the second authentication is also successful, the success of both the authentications is explicitly recorded in the image file 1200. Thus, the more the authentication is successful, the higher the likelihood that the image is taken by the user (photographer) can be. Further, the presence or absence of the use of the other person is also recorded as the photographer information 1211 in the metadata 1210 of the image file 1200. Thus, in the case where only the first authentication is successful and the second authentication fails, it is thus possible to show whether or not the use of the other person is even suspected. The likelihood that the image is taken by the user (photographer) can also be increased thereby.

[0351] The camera 100 according to the first exemplary embodiment described above is an information processing apparatus that performs user authentication. The camera 100 according to the first exemplary embodiment includes a registration data management unit 505 that manages authentication registration information about a user who is permitted to use the camera 100. The camera 100 according to the first exemplary embodiment includes a first authentication unit 507 that performs authentication of an authentication target user by using a plurality of authentication target information acquired from a plurality of different parts of the authentication target user and the authentication registration information managed by the registration data management unit 505. The camera 100 according to the first exemplary embodiment further includes a second authentication unit 508 that performs second authentication of the authentication target user by using authentication target information acquired from one of the plurality of parts of the authentication target user after the first authentication by the first authentication unit 507 is successful.

[0352] Such a configuration can increase the accuracy of user authentication with the camera 100 (information processing apparatus) while preventing a decrease in usability.

[0353] A modification of the first exemplary embodiment will be described. In the first exemplary embodiment described above, only the "name" is used as the personal information to be managed by the registration data management unit 505. However, information other than the name can be used. For example, in the case of the camera 100 used in a company, an "employee number" assigned to each employee can be stored. Alternatively, account information such as an account name for a web service can be input. The account information can be used as the personal information when accessing and successfully logging in to the web service. A token can be issued when successfully accessing the web service, and the token can also be stored as the personal information. The authentication state storage unit 522 can store such personal information as the photographer information 1211 in the metadata 1210 of the image file 1200.

[0354] In the first typical embodiment described above, the first authentication unit 507 and the second authentication unit 508 use the same feature vector calculation unit 502. However, the first authentication unit 507 and the second authentication unit 508 acquire different tendencies of eye images. Specifically, in order to capture an eye image in a case where it is intended to actively authenticate a user, the first authentication unit 507 acquires an eye image under a condition such as eyes E being wide open. On the other hand, the second authentication unit 508 attempts to capture an eye image of a user during imaging and performs authentication, and thus there can be a wide variation in eye images having various gaze angles. Thus, for the neural network used in the first authentication unit 507, a model trained using eye images intended for the first authentication is used. For the neural network used in the second authentication unit 508, a model trained using eye images intended for the second authentication is used. This can further improve the authentication accuracy. The use of an authentication method with a low false acceptance rate for the first authentication and an authentication method with a low false rejection rate for the second authentication can be achieved by a method other than using different thresholds or different models. For example, as discussed in Japanese Patent 7346528, the feature vectors to be used during enrollment and during collation can be calculated by different methods to improve performance. Such an authentication method can be employed.

[0355] In the first typical embodiment described above, the number of users enrolled is one. Thus, when enrolling a different person, all the data held by the enrollment data management unit 505 is erased, and enrollment is performed again.

[0356] In other words, when the enrollment processing is activated, the data in the enrollment data management unit 505 is deleted. Alternatively, without deleting the data, an invalidation flag is set not to use the data for subsequent first authentication and second authentication. However, the camera 100 can be configured to enable enrollment of multiple users. In such a case, each time the enrollment processing is activated, enrollment information (personal information and feature vectors) having a different personal ID is enrolled in the enrollment data management unit 505. It will be understood that a process for preventing redundant enrollment of the same person can be added. For example, if the name is the same, a message can be displayed that the user has already been enrolled and the enrollment processing can be ended. Which personal ID the user using the camera 100 has is judged when performing the first round of authentication in the first authentication processing. If there are multiple personal IDs having a degree of similarity exceeding a predetermined threshold, the user can be authenticated with the personal ID having the highest degree of similarity. Alternatively, the first authentication can end in failure. Furthermore, the enrollment feature vectors for use in the second round of authentication in the first authentication processing and in the second authentication processing can be limited to the feature vectors of that personal ID. More specifically, in the example of the first authentication processing illustrated in FIG. 6, the enrollment feature vectors for use in the second round of authentication are limited to the feature vectors of the personal ID having the highest degree of similarity. In the example of the second authentication processing illustrated in FIG. 7, the enrollment feature vectors for use in the second authentication processing are limited to the feature vectors of the personal ID having the highest degree of similarity. FIG. 9In step S822 or S823, the CPU 212 (the first authentication unit 507) extracts only the record including the personal ID identified in step S810 of the first authentication processing from the first authentication registration right-eye feature vector table 540 or the first authentication registration left-eye feature vector table 550. In the second authentication processing illustrated in FIG. 9, the CPU 212 (the second authentication unit 508) extracts only the record including the personal ID identified in step S908 from the second authentication registration feature vector table 560. FIG. 5F In step S810 of the first authentication processing, the CPU 212 (the first authentication unit 507) extracts only the record including the personal ID identified in step S808 from the first authentication registration right-eye feature vector table 540 or the first authentication registration left-eye feature vector table 550. In the second authentication processing illustrated in FIG. 9, the CPU 212 (the second authentication unit 508) extracts only the record including the personal ID identified in step S908 from the second authentication registration feature vector table 560. FIG. 9 In step S908 of the second authentication processing, the CPU 212 (the second authentication unit 508) extracts only the record including the personal ID identified in step S906 from the second authentication registration feature vector table 560. In the first authentication processing and the second authentication processing described above, the feature vectors of the extracted records are used only for authentication. This reduces the number of vectors to be compared in the second round of authentication in the first authentication processing and in the second authentication processing. This can improve the accuracy of authentication. The reason is that the authentication problem can be simplified from 1:N identification to 1:1 identification. Since the feature vectors to be compared are reduced, the processing time can also be reduced.

[0357] In the first typical embodiment described above, the registration data management unit 505 holds the first authentication registration right-eye feature vector table 540, the first authentication registration left-eye feature vector table 550, and the second authentication registration feature vector table 560 as separate tables. However, such a structure is inefficient because of the redundant "feature vector 1r". Therefore, the registration data management unit 505 can hold a single integrated table. In doing so, information indicating for which authentication (the first authentication or the second authentication) each record is used, and information indicating from which of the left and right eyes each record is acquired can be stored. Then, the registration feature vectors to be used during the first authentication and the second authentication are selected. In the first typical embodiment described above, the first authentication and the second authentication use different registration feature vectors. However, the same feature vectors can be used. In such a case, the registration data management unit 505 does not need to hold the first authentication registration right-eye feature vector table 540, the first authentication registration left-eye feature vector table 550, and the second authentication registration feature vector table 560, and can manage the feature vectors using a single table.

[0358] In the first typical embodiment described above, during the second authentication processing, the authentication status display unit 523 displays the authentication result on the display device 214 (steps S912, S919, and S920). FIG. 9

[0359] ​However, at the time of imaging, the image being captured by the image sensor 211 is already displayed on the display device 214. Therefore, the user is considered to want to concentrate on imaging. Thus, the authentication result is desirably displayed in a manner that does not disturb imaging. To this end, the authentication status display unit 523 can be modified as follows. For example, the authentication status display unit 523 displays an indication of success or failure at an end portion of the screen of the display device 214. Alternatively, the authentication status display unit 523 can determine a display position based on the gaze position of the user on the display device 214 obtained in step S1507 of the line-of-sight detection processing FIG. 9 of the first typical embodiment described above, the authentication status of the second authentication is displayed in steps S912 and S919. However, the authentication status of the first authentication can also be displayed. Similarly, the authentication status of both the first authentication and the second authentication can be displayed in step S920 of the second authentication processing.

[0360] In the first typical embodiment described above, the entire contents of the authentication status table 570 managed by the authentication status management unit 521 are stored as the metadata 1210 in the authentication status storage processing using the authentication status storage unit 522. However, the entire contents need not be stored, and the contents can be processed before storage. For example, only the "name" can be stored. Only when both the first authentication status and the second authentication status are "authenticated" can the "name" be stored. In other cases, a value indicating unknown can be stored in the "name". Although the first authentication status and the second authentication status are distinguished, these statuses can be integrated into one item "authentication status". Only when both the first authentication status and the second authentication status are "authenticated" can "authenticated" be stored. In other cases, "not authenticated" can be stored. FIG. 9 FIG. 9 In the first typical embodiment described above, the entire contents of the authentication status table 570 managed by the authentication status management unit 521 are stored as the metadata 1210 in the authentication status storage processing using the authentication status storage unit 522. However, the entire contents need not be stored, and the contents can be processed before storage. For example, only the "name" can be stored. Only when both the first authentication status and the second authentication status are "authenticated" can the "name" be stored. In other cases, a value indicating unknown can be stored in the "name". Although the first authentication status and the second authentication status are distinguished, these statuses can be integrated into one item "authentication status". Only when both the first authentication status and the second authentication status are "authenticated" can "authenticated" be stored. In other cases, "not authenticated" can be stored.

[0361] In the first typical embodiment described above, the entire contents of the authentication status table 570 managed by the authentication status management unit 521 are stored as the metadata 1210 in the authentication status storage processing using the authentication status storage unit 522. However, the entire contents need not be stored, and the contents can be processed before storage. For example, only the "name" can be stored. Only when both the first authentication status and the second authentication status are "authenticated" can the "name" be stored. In other cases, a value indicating unknown can be stored in the "name". Although the first authentication status and the second authentication status are distinguished, these statuses can be integrated into one item "authentication status". Only when both the first authentication status and the second authentication status are "authenticated" can "authenticated" be stored. In other cases, "not authenticated" can be stored.

[0362] ​In the above-described first typical embodiment, both the first authentication unit 507 and the second authentication unit 508 use personal authentication based on an eye image. However, the first authentication unit 507 and the second authentication unit 508 can be configured to use other authentication methods. For example, for the first authentication, other biometric authentication such as fingerprint authentication can be used. Fingerprint authentication can be implemented by incorporating a fingerprint sensor in the release button 121 and performing authentication when the user places a finger on the release button 121. As the first authentication, multi-stage biometric authentication using a plurality of biometric information can be performed. In such a case, the biometric information used for the first authentication and the second authentication is limited to biometric information for which it can be determined whether it is acquired from the same person or biometric information for which it is guaranteed that it is manually acquired from the same person. Examples of biometric information for which it can be determined whether it is acquired from the same person include fingerprints of different fingers because if the fingerprints of different fingers are acquired from the same person, the fingerprints have a correlation. For this reason, a neural network that has been trained to be able to determine whether fingerprint data is acquired from the same person by using fingerprint data acquired from different fingers as training data is used. This makes it possible to determine whether the fingerprint data used in the first authentication and the fingerprint data used in the second authentication are acquired from the same person. With such a determination, both the first authentication and the second authentication can be performed by fingerprint authentication. As other examples of biometric information for which it can be determined whether it is acquired from the same person, the first authentication can be performed by face authentication and the second authentication can be performed by personal authentication using an eye image. In such a case, a neural network that has been trained to be able to make a determination based on the identity of an eye region included in a face image used in the first authentication and an eye image used in the second authentication is used. Since the second authentication is performed at the time of imaging, it is desirable that the second authentication unit 508 be able to perform authentication during imaging. For example, in the case where a fingerprint sensor is incorporated in the release button 121, fingerprint authentication can be performed at the time of imaging and used for the second authentication. In a shooting mode in which imaging is performed by displaying an image being captured with the image sensor 211 on a touch screen (operation member 123) without the user looking into the viewfinder, for the second authentication, face authentication can be used. In order to control the authentication state of the second authentication to continue only during imaging, in the case where face authentication is used, it can be considered that imaging is in progress during the time when a face is being captured. Alternatively, in the case of fingerprint authentication, it can be considered that imaging is in progress during the time when a finger is placed on the release button 121.

[0363] At the time of the first authentication success, the features of the eye image at that time or before or after that time can be kept for use. During the second authentication, authentication by identity confirmation (verification processing) with the features at the time of the above-mentioned successful first authentication can be performed. Such a method has the advantage of reducing the workload of registering images intended for the second authentication. This method is effective when the second authentication is expected to be performed in an environment significantly different from that during registration. Both the above-mentioned identity confirmation and the above-mentioned other matching can be performed to improve the reliability. More specifically, if the matching by either means is successful (or the matching by both means is successful), the second authentication can be judged to be successful. Other application modes of this method can include the following. During the first authentication, a personal identification number (PIN) is input and fingerprint authentication using the release button 121 is performed. If the first authentication is successful, a face image of the user is simultaneously captured using a built-in camera and converted into feature amounts, and the feature amounts are stored. For the second authentication, the identity of the features of the face image with the face features of the photographer is confirmed.

[0364] In this way, for the two-stage authentication by application of the first typical embodiment, there are various possible modes.

[0365] In the above-described first typical embodiment, the first authentication state invalidation unit 510 invalidates the first authentication state when various conditions apply. However, when the first authentication state is invalidated, the user can validate the first authentication state by re-performing the first authentication. Therefore, the successful first authentication and the invalidation of the first authentication state can be repeated for the purpose of unauthorized use. In view of this, the camera 100 can include an unillustrated first authentication restriction unit. This first authentication restriction unit detects suspicion of unauthorized use and imposes a restriction so that the first authentication cannot be performed temporarily. As for the detection method, when the successful first authentication by the first authentication unit 507 and the invalidation of the first authentication state by the first authentication state invalidation unit 510 are repeated within a predetermined period of time, the first authentication restriction unit detects suspicion of unauthorized use. Alternatively, the detection of suspicion can be limited to only a part of the above-described various conditions on which the invalidation of the first authentication state is based. For example, the detection of suspicion can be limited to the invalidation of the first authentication state by the first authentication state invalidation unit 510 when the use of the other person is suspected by the other person use detection unit 509. More specifically, when the invalidation due to the detection of the use of the other person and the successful first authentication by the first authentication unit 507 are repeated within a predetermined period of time, suspicion of unauthorized use is detected. A possible method for restricting the execution of the first authentication is to disable the user operation to the menu. In other words, if the camera 100 is configured so that the user operates the camera 100 to call the first authentication process from the menu, the menu for calling the first authentication process can be disabled. The restricted state can be released after a certain time elapses. This makes it difficult for the registered person to intentionally lend the camera 100 to the other person and have them take an image. In particular, if a technique other than the biometric authentication is used for the first authentication, the first authentication can be performed even in the absence of the registered person. For example, the first authentication can also be successfully performed by sharing a password and a PIN or by lending a paired smart phone. The above-described restriction can prevent such unauthorized use.

[0366] In the first typical embodiment, the second authentication using the second authentication unit 508 is performed at the time of imaging (during imaging). However, due to speed and other resources, it can be difficult to perform authentication at the time of imaging. In such a case, information required for the second authentication can be stored at the time of imaging, and the authentication process can be performed after imaging. For example, an eye image can be stored at the time of imaging, and authentication can be performed after imaging. Similarly, biological information (a face image, a fingerprint) can be stored to perform other biometric authentication (face authentication, fingerprint authentication). In the case of the judgment based on the connection state between the smartphone and the camera 100, a connection state parameter can be stored, and thereafter a process for analyzing the connection state to perform authentication can be performed. Thus, the second authentication does not necessarily need to be performed at the time of imaging. Information required for the second authentication can be acquired at the time of imaging, and authentication can be performed after imaging.

[0367] When authentication is performed later, it can not be possible to obtain an authentication result in time with the storage of the image file 1200. In such a case, an option different from "authenticated" and "not authenticated" can be provided for the second authentication state, such as "processing", and the option can be stored as the photographer information 1211 of the metadata 1210. When an authentication result of the second authentication is obtained, the photographer information 1211 of the metadata 1210 can be re-modified and stored. It is desirable that the second authentication uses biometric authentication. It is difficult for the photographer to perform an input action such as password input at the time of imaging. If the user lends the smartphone with the camera 100, authentication based on the connection state between the smartphone and the camera 100 has a space for use by another person. On the other hand, biometric authentication does not require an action for authentication, and information can be owned only by the same person. Thus, it is desirable to use biometric authentication for the second authentication performed at the time of imaging.

[0368] In the first typical embodiment described above, the second authentication is performed when the first authentication state is "authenticated". However, there can be cases where it is difficult to perform the first authentication, such as a case where the user suddenly needs to take an image or a case where the user has forgotten the first authentication. Then, the camera 100 can be configured so that the second authentication is performed when the first authentication state is "not authenticated". Specifically, step S901 of "when the first authentication state is "authenticated", the process proceeds from this step S901 to step S902" can be modified so that when the first authentication state is "not authenticated", the process also proceeds to step S902. When the first authentication state is "not authenticated", the personal ID in the authentication state table 570 has a null value. If the second authentication is successful in the case where the first authentication state is "not authenticated", in step S911 of FIG. 9 the process can be performed using the personal ID in the authentication state table 570. Thus, the second authentication can be performed even when the first authentication is not performed. FIG. 5F FIG. 9 FIG. 12A ​​The personal ID identified in step S909 is used to update the personal ID in the authentication status table 570. Then, in the authentication status storage process, the first authentication status "unauthenticated" and the second authentication status "authenticated" are stored in the photographer information 1211 of the metadata 1210. Here, the personal ID can be stored in the photographer information 1211 of the metadata 1210 such that it indicates the personal ID is the personal ID identified through the second authentication. For example, the personal ID can be recorded as another metadata item. In the above description, if there is not one but more registered persons, then in the example of the second authentication... FIG. 12A In step S908, the feature vector to be acquired is described as limited to a feature vector having the personal ID identified through the first authentication. However, if no first authentication is performed, then... FIG. 6A In step S908, all feature vectors in the second authentication registration feature vector table 560 are extracted and... FIG. 6B In step S909, a comparison is made. Here, personal IDs with feature vectors that have a similarity exceeding a threshold are authenticated. If multiple personal IDs have a similarity exceeding the threshold, the personal ID with the most similar feature vector is authenticated. Alternatively, authentication can end in failure. The authentication status can be determined based on whether the first authentication state is "authenticated" or "unauthenticated". FIG. 8A Different thresholds are used in step S909. Alternatively, other authentication settings, such as the model to be used, can be changed. The reason is that the optimal settings are different because the 1:1 authentication problem is now a 1:N authentication problem.

[0369] In the first typical embodiment described above, the second authentication is processed assuming that the personal ID is the same as the ID identified through the first authentication. However, instead of using the result of the first authentication, the personal ID can be independently identified in the second authentication. Specifically, the authentication status managed by the authentication status management unit 521 is modified. FIG. 8B The authentication status table 570 stores the "Personal ID for First Authentication" and the "Personal ID for Second Authentication" separately. Then, the respective personal IDs can be stored for both first and second authentication. The second authentication process is illustrated in the example. FIG. 8A In step S908, the feature vector to be acquired is limited to the feature vector of the personal ID with the first authentication. Alternatively, the second authentication unit 508 can be configured to identify the personal ID from all feature vectors. This is illustrated in the authentication state storage processing. FIG. 8A When storing metadata 1210, the personal IDs and authentication status of the first and second authentications can be stored separately in metadata 1210. When using metadata 1210, it can be confirmed that the same person was authenticated through both the first and second authentications, thus confirming that the image was taken by the same person. Alternatively, if the authentication status is stored in the example... FIG. 5CIf the personal IDs of the first and second authentications are different when storing metadata 1210, the second authentication status can be stored as "unauthenticated". In this case, the personal information (i.e., name) associated with the personal ID of the first authentication is stored in metadata 1210 as photographer information 1211 (i.e., name). When storing metadata 1210, processing can be performed to identify that the same person has been authenticated through the first and second authentications. In this way, the first and second authentications can be performed independently, and the identity of the person can be confirmed when referring to the authentication results of the first and second authentications.

[0370] In the first typical embodiment described above, the first authentication is performed when not recording video, and the second authentication is performed when recording video. However, it is possible to ensure that the photographer is a registered user based solely on the result of the first authentication without performing the second authentication. Specifically, the photographer is guaranteed to be a registered user without performing the second authentication until a predetermined time has elapsed since the first authentication was successful.

[0371] In the first typical embodiment described above, the registration process and the first authentication process include: performing left and right eye determination processing based on the deviation between the actual gaze point and the estimated gaze point in the gaze detection process. However, without performing left and right eye determination processing, the left and right eyes can be determined by the user inputting which eye image they want to obtain. Specifically, in the registration process... FIG. 5D Step S604 and ​ Before acquiring the eye image in step S617, the left and right eyes are determined by the user specifying which eye to view the viewfinder using a user operation on the menu. Alternatively, in the first authentication process... ​ Step S803 and ​ Before acquiring the eye image in step S814, the left and right eyes are determined by the user specifying which eye to view the viewfinder using a user operation on the menu. Alternatively, without performing left and right eye determination processing, the left and right eyes can be determined by instructing the user to acquire the eye image of which eye among the left and right eyes. Specifically, in the first authentication process... ​ In steps S801 and S812, when instructions related to the authentication method are displayed, instructions related to which eye (left or right) is used to view the viewfinder are also displayed to perform left-eye / right-eye determination. Alternatively, without performing left-eye / right-eye determination processing, the left-eye and right-eye can be determined based on which eye (left or right) has the highest similarity to the feature vector to be compared. Specifically, in the first authentication process... ​ When performing the comparison in step S808, from ​ The first authentication registration right eye feature vector table 540 and​ The first authentication registration left eye feature vector table 550 exemplified above acquires the feature vectors and collates the feature vectors. If the feature vector with the highest degree of similarity is acquired from the first authentication registration right eye feature vector table 540, the eye image is judged to be an eye image acquired from the right eye. Similarly, if the feature vector with the highest degree of similarity is acquired from the first authentication registration left eye feature vector table 550, the eye image is judged to be an eye image acquired from the left eye.

[0372] A second typical embodiment will be described. In the following description of the second typical embodiment, the description of items similar to those of the first typical embodiment described above will be omitted, and the differences from the first typical embodiment described above will be described.

[0373] In the first typical embodiment described above, the first authentication is not judged to be successful unless both of the two authentications using multiple biological information (both eyes) are successful. In the second typical embodiment, if the expected degree of strictness of collation is not high and the authentication using one of the left eye and the right eye is successful, the first authentication is judged to be successful. As an example of the second typical embodiment, a rental device that is a rental head-mounted display (HMD) will be assumed. For example, the expected degree of strictness of collation is not high when logging in to the HMD, and if the authentication using one of the left eye and the right eye is successful, the user is permitted to log in. In contrast, in the case of online payment via the HMD, the expected degree of strictness of collation is high, and the user cannot make a payment unless the authentication using both of the left eye and the right eye is successful. The operation to be permitted by the first authentication is not limited to this.

[0374] A modification of the second typical embodiment will be described. In the second typical embodiment described above, in the case where the expected degree of strictness of collation is not high, if the authentication using one of the left eye and the right eye is successful, the first authentication is judged to be successful, and if the authentication using the one eye fails, the first authentication is judged to be unsuccessful. However, if the authentication using the one eye fails and the authentication using the other eye is successful, the first authentication can be judged to be successful.

[0375] The typical embodiments of the present disclosure can also be realized by a process for supplying a program for realizing one or a plurality of functions of the typical embodiments described above to a system or a device, and reading and executing the program by one or a plurality of processors in a computer of the system or the device. A circuit such as an application specific integrated circuit [ASIC] or the like for realizing one or a plurality of functions can also be used for the realization.

[0376] A program and a computer-readable storage medium storing the program are included in the typical embodiments of the present disclosure.

[0377] The above-described exemplary embodiments of the present disclosure are all merely examples for implementing the present disclosure, and the technical scope of the present disclosure should not be construed as being limited thereto. In other words, the present disclosure can be implemented in various forms without departing from the technical idea or principal characteristics of the present disclosure.

[0378] According to the exemplary embodiments of the present disclosure, it is possible to improve the accuracy of user authentication using an information processing apparatus and to prevent a decrease in usability.

[0379] While the present disclosure has been described with reference to the typical embodiments, it is to be understood that the present disclosure is not limited to the disclosed typical embodiments. The scope of the claims to be attached is in accordance with the broadest interpretation so as to encompass all such modifications and equivalent constructions and functions.

Claims

1. An information processing apparatus configured to perform user authentication, the information processing apparatus comprising: one or more storages that store instructions, and one or more processors that, upon executing the stored instructions, are configured to operate as: a management unit that manages authentication registration information related to a user who is permitted to use the information processing apparatus; and an authentication unit that performs authentication of an authentication target user who attempts to use the information processing apparatus, by using a plurality of authentication target information acquired from a plurality of different parts of the authentication target user and the authentication registration information.

2. The information processing device according to claim 1, wherein The plurality of authentication target information is a plurality of biometric information acquired from the plurality of parts of the body of the authentication target user.

3. The information processing device according to claim 2, wherein The plurality of biometric information is feature information based on eye images acquired from right and left eyes of the authentication target user. 4.The information processing apparatus according to claim 3, the execution of the stored instructions further configuring the one or more processors to operate as a determination unit that determines the eye image of the right eye and the eye image of the left eye based on a deviation between a line of sight of the authentication target user estimated from the eye images and an actual line of sight of the authentication target user.

5. The information processing device according to claim 1, wherein The authentication unit performs a second authentication of the authentication target user by using authentication target information acquired from one of the plurality of parts of the authentication target user after a first authentication of the authentication target user using the plurality of authentication target information is successful.

6. The information processing device according to claim 5, wherein The authentication unit performs the first authentication before the authentication target user performs image capturing and performs the second authentication while the authentication target user is performing image capturing.

7. The information processing device according to claim 5, wherein The authentication unit determines that the first authentication is successful in a case where the first authentication is successful using authentication using at least one of the plurality of authentication target information.

8. The information processing device according to claim 5, wherein The management unit manages authentication registration information associated with a user used in each of the first authentication and the second authentication in association with each other.

9. The information processing device according to claim 5, wherein The execution of the stored instructions further configures the one or more processors to include an execution unit configured to execute a predetermined process based on authentication states of the first authentication and the second authentication.

10. The information processing device according to claim 9, wherein The execution unit is configured to execute at least one of a management process, a storage process, and a display process of the authentication states of the first authentication and the second authentication as the predetermined process.

11. The information processing device according to claim 9, wherein The execution unit is configured to change contents of the predetermined process based on the authentication states of the first authentication and the second authentication. 12.A control method of an information processing apparatus configured to perform user authentication, the control method comprising: managing authentication registration information related to a user who is permitted to use the information processing apparatus; and performing authentication of an authentication target user who attempts to use the information processing apparatus, by using a plurality of authentication target information acquired from a plurality of different parts of the authentication target user and the authentication registration information.

13. A computer-readable storage medium storing a program for causing a computer to execute the control method according to claim 12.

14. A computer program product comprising a program for causing a computer to execute the control method according to claim 12.

Images