Imaging device, operation method of imaging device, and program

Abstract

To improve usability of a device, while preventing increase in a false acceptance rate in personal authentication of a user of the device.SOLUTION: An imaging device acquires biological information of a person. The imaging device determines whether or not a user of an imaging device is a previously registered person. It is determined by biological authentication processing using the acquired biological information whether or not the user of the imaging device is the same person as the user determined previously registered. According to the result of the determination, control is performed so that a photographic image by the imaging device and the information on the user are recorded in a memory in association with each other.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a photographing device, a method for operating the photographing device, and a program, and more particularly to a technology for authenticating a device user. [Background technology]

[0002] Conventionally, methods for authenticating the user of an information processing device including an image capture device have been used. For example, Patent Document 1 discloses a method for authenticating a user using an image of the user's eye looking through the finder of the image capture device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2024-2562 Summary of the Invention [Problem to be solved by the invention]

[0004] To more reliably ensure that a registered person is using a device, authentication is generally configured to reduce the likelihood of recognizing a different person as a specific registered person (false acceptance rate). However, setting a low false acceptance rate increases the likelihood of not recognizing a registered person (false rejection rate). This increases the likelihood that the device will not operate based on personal authentication, reducing the usability of the device. Furthermore, when a highly accurate authentication method that can achieve both a low false acceptance rate and a low false rejection rate is used, the time required for authentication processing generally increases. This also reduces the usability of the device.

[0005] An object of the present invention is to improve the usability of a device while preventing an increase in the false acceptance rate in personal authentication of a device user. [Means for solving the problem]

[0006] An image capturing apparatus according to an embodiment of the present invention has the following configuration: An imaging device, an acquisition means for acquiring biometric information of a person; a first authentication means for determining whether a user of the photographing device is a person who has been registered in advance; a second authentication means for determining whether the user of the photographing device is the same person as the user determined to be registered by the first authentication means through a biometric authentication process using the biometric information acquired via the acquisition means; a control means for controlling the image captured by the photographing device to be recorded in a memory in association with information about the user in accordance with a result of the determination by the second authentication means; Equipped with. [Effects of the Invention]

[0007] The usability of the device can be improved while preventing an increase in the false acceptance rate in personal authentication of the device user. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an external view of a camera according to an embodiment. [Figure 2] FIG. 1 is a cross-sectional view of a camera according to an embodiment. [Figure 3] FIG. 1 is a block diagram showing the configuration of a camera according to an embodiment. [Figure 4] FIG. [Figure 5] FIG. 2 is a diagram for explaining the principle of a gaze detection method. [Figure 6] FIG. [Figure 7] 10 is a flowchart of a gaze detection process. [Figure 8] FIG. 2 is a block diagram showing a functional configuration related to authentication processing of a camera according to an embodiment. [Figure 9] 10 is a flowchart of a user registration process. [Figure 10] 10 is a flowchart of an eye image acquisition process. [Figure 11] 10 is a flowchart of a first authentication process. [Figure 12] 10 is a flowchart of a second authentication process. [Figure 13] 10 is a flowchart of a process for detecting use by another person. [Figure 14] 10 is a flowchart of a first authentication invalidation process. [Figure 15] 10A and 10B are a flowchart of an authentication status saving process and an example of an image file. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] An information processing device according to an embodiment has a biometric authentication function. The operation of the information processing device is controlled according to the result of the authentication process. The following describes the configuration and operation of an image capturing device, which is an example of such an information processing device.

[0011] [Configuration of imaging equipment] 1A and 1B show the exterior of a camera 100 (digital still camera; interchangeable lens camera) that is an imaging device according to this embodiment. FIG. 1A is a front perspective view. FIG. 1B is a rear perspective view. As shown in FIG. 1A, the camera 100 has a photographing lens unit 100A and a camera housing 100B. A release button 101, which is an operation member that accepts a photographing operation from a user (photographer), is located on the camera housing 100B. As shown in FIG. 1B, an eyepiece 102 (viewfinder) is located on the rear of the camera housing 100B. The user looks into the eyepiece 102 to view a display device 214 (display panel) (described below) included in the camera housing 100B. Also located on the rear of the camera housing 100B are operation members 103-105 that accept various operations from the user. For example, operation member 103 is a touch panel that accepts touch operations. Operation member 104 is an operation lever that can be tilted in any direction. The operation member 105 is a four-way key that can be pressed in each of four directions. The operation member 103 (touch panel) is equipped with a display panel (for example, a liquid crystal panel). This display panel has a function of displaying images.

[0012] 2 is a cross-sectional view of the camera 100 taken along the YZ plane defined by the Y axis and Z axis shown in FIG.

[0013] The photographing lens unit 100A includes two lenses 201 and 202, an aperture 203, an aperture driver 204, a lens drive motor 205, a lens drive member 206, a photocoupler 207, a pulse plate 208, a mount contact 209, and a focus adjustment circuit 210. The lens drive member 206 includes a drive gear. The photocoupler 207 detects the rotation of the pulse plate 208, which is interlocked with the lens drive member 206, and transmits this rotation to the focus adjustment circuit 210. The focus adjustment circuit 210 drives the lens drive motor 205 based on information from the photocoupler 207 and information from the camera housing 100B (lens drive amount information). The focus adjustment circuit 210 drives the lens drive motor 205 to move the lens 201, thereby changing the focus position. The mount contact 209 is an interface between the photographing lens unit 100A and the camera housing 100B. Although two lenses 201 and 202 are shown in FIG. 2, more than two lenses may be included in photographing lens unit 100A.

[0014] Camera housing 100B includes an image sensor 211, a CPU 212, a memory unit 213, a display device 214, and a display device drive circuit 215. Image sensor 211 is disposed at a planned imaging plane of photographing lens unit 100A. CPU 212 is a central processing unit of a microcomputer. CPU 212 controls the entire camera 100. Memory unit 213 stores various information. For example, memory unit 213 stores images captured by image sensor 211. Display device 214 displays various information on the screen (display surface) of display device 214. For example, display device 214 is a liquid crystal panel. Display device 214 can display a captured image (subject image) on the screen. Display device drive circuit 215 drives display device 214. A user can view the screen of display device 214 through eyepiece 102.

[0015] The camera housing 100B further includes light sources 216a and 216b, a light splitter 217, a light receiving lens 218, and an eye imaging element 219. The light sources 216a and 216b are light sources for illuminating the eyeball 220 of the user looking through the viewfinder (eyepiece 102). The light sources 216a and 216b have traditionally been used in single-lens reflex cameras to detect the line of sight from the relationship between the pupil and the corneal reflection image of light. The light sources 216a and 216b are arranged around the eyepiece 102. For example, the light sources 216a and 216b are infrared light-emitting diodes. The light sources 216a and 216b can emit infrared light that is invisible to the user. An optical image of the illuminated eyeball 220 (eye optical image; an optical image formed by light emitted from the light sources 216a and 216b and reflected by the eyeball 220) passes through the eyepiece 102 and is reflected by the light splitter 217. The eye optical image is then formed on the eye imaging element 219 by the light receiving lens 218. The eye imaging element 219 has a configuration in which a plurality of photoelectric conversion elements (e.g., CCD or CMOS) are arranged two-dimensionally. The light receiving lens 218 positions the pupil of the eyeball 220 and the eye imaging element 219 in a conjugate imaging relationship. A gaze detection process, which will be described later, detects the gaze of the eyeball 220 from the position of the corneal reflection image in the eye optical image formed on the eye imaging element 219. For example, the gaze detection process obtains, as information related to the gaze, at least one of information indicating the gaze direction (direction of the gaze) and information indicating the viewpoint (position where the gaze is fixed) on the screen of the display device 214. The viewpoint can be considered as the position where the user is looking. The viewpoint can also be considered as the line of sight.

[0016] 3 is a block diagram showing the electrical configuration within camera 100. Connected to CPU 212 are gaze detection circuit 301, photometry circuit 302, autofocus detection circuit 303, signal input circuit 304, display device drive circuit 215, and light source drive circuit 305. CPU 212 transmits signals via mount contacts 209 to focus adjustment circuit 210 disposed within photographing lens unit 100A and aperture control circuit 306 included in aperture drive section 204 within photographing lens unit 100A. Memory section 213 associated with CPU 212 has the function of storing image pickup signals from image pickup element 211 and eye image pickup element 219.

[0017] The gaze detection circuit 301 A / D converts the output of the eye imaging element 219 (an eye image obtained by capturing an image of the eye (eyeball 220)) when an optical image of the eye is formed on the eye imaging element 219. The gaze detection circuit 301 then transmits the conversion result to the CPU 212. The CPU 212 extracts feature points required for gaze detection from the eye image according to gaze detection processing described below. The CPU 212 then detects the user's gaze from the positions of the feature points.

[0018] The photometry circuit 302 performs predetermined processing (e.g., amplification, logarithmic compression, and A / D conversion) on a signal obtained from the image sensor 211, which also functions as a photometry sensor, such as a luminance signal corresponding to the brightness of the field. The photometry circuit 302 then sends the processing result to the CPU 212 as field luminance information.

[0019] The autofocus detection circuit 303 A / D converts signals from multiple detection elements (multiple pixels) included in the image sensor 211 that are used for phase difference detection, and sends the converted signals to the CPU 212. The CPU 212 calculates the distance to the subject corresponding to each focus detection point from the signals from the multiple detection elements. This distance calculation method is known as image plane phase difference AF. In this embodiment, there are 180 focus detection points on the image plane. The 180 focus detection points correspond to the 180 focus measurement point indicators 401 that exist in the field of view within the viewfinder (the screen of the display device 214) shown in FIG. 4(A).

[0020] Switches SW1 and SW2 are connected to the signal input circuit 304. Switch SW1 is a switch for starting a shooting preparation operation (e.g., photometry and distance measurement) of the camera 100. Switch SW1 is turned on by the first stroke of the release button 101. Switch SW2 is a switch for starting a shooting operation. Switch SW2 is turned on by the second stroke of the release button 101. ON signals from switches SW1 and SW2 are input to the signal input circuit 304 and transmitted to the CPU 212. When switch SW1 is turned on, line of sight detection may be started.

[0021] The light source drive circuit 305 drives the light sources 216a and 216b.

[0022] Furthermore, operation members 103 to 105 are also connected to CPU 212. When the user operates operation members 103 to 105, operation members 103 to 105 output operation signals corresponding to the operation from the user to CPU 212. Then, CPU 212 performs processing (control) corresponding to the operation signals. For example, CPU 212 can move a selection frame in a displayed menu in response to the operation signals.

[0023] FIG. 4(A) is a diagram showing the field of view within the viewfinder. FIG. 4 shows the display device 214 in an operating state (a state in which an image is displayed). As shown in FIG. 4(A), the field of view within the viewfinder includes a focus detection area 400, 180 ranging point indices 401, and a field of view mask 402. Each of the 180 ranging point indices 401 is displayed at a position corresponding to a focus detection point on the imaging surface. The 180 ranging point indices 401 are also displayed superimposed on a through image (live view image) displayed on the display device 214. Of the 180 ranging point indices 401, the ranging point indices 401 that corresponds to the current viewpoint 411 (estimated position) is displayed highlighted, for example, with a frame.

[0024] [Authentication configuration] 8A is a block diagram showing the functional configuration used for authentication by an information processing device according to an embodiment. Using this configuration, the camera 100 can perform personal authentication of the user.

[0025] In this embodiment, both a first authentication process and a second authentication process are used to authenticate a user. In the first authentication process, authentication is performed to determine whether the user of a device (e.g., the camera 100) is a previously registered person. In this embodiment, the first authentication process is performed before the user takes a photograph using the camera 100. In addition, the second authentication process determines whether the user of the device (e.g., the camera 100) is the same person as the user determined to be registered in the first authentication process. Such second authentication process can be performed when a predetermined operation using the device is performed. Alternatively, such second authentication process can be performed using authentication information (e.g., biometric information) acquired when performing a predetermined operation using the device. In this embodiment, the predetermined operation is a photographing operation, and the second authentication process is performed when photographing. Note that, in this specification, "when photographing" refers to immediately before, during, or immediately after photographing. Then, the operation of the device can be controlled at least according to the result of the second authentication process. For example, the operation of the camera 100 is controlled so as to record the results of the first and second authentication processes together with the photographed image. Furthermore, if the second authentication process fails, the operation of the camera 100 is controlled so as to record information indicating the authentication failure. In this embodiment, depending on the result of the second authentication process, the image captured by the camera 100 and the user information are associated with each other and recorded in memory.

[0026] In this embodiment, user authentication is performed using biometric information. For example, in the second authentication process, user authentication can be performed using biometric information. In this embodiment, the biometric information is an eye image. That is, the camera 100 authenticates whether the user is a person who has been registered in advance based on an image of the user's eyeball 220 looking through the viewfinder (eyepiece 102).

[0027] Each component will be described below. The imaging unit 813 is mainly realized by the eye imaging element 219. The other components shown in FIG. 8(A) can be realized by a processor such as the CPU 212 executing a program stored in a memory such as the memory unit 213. However, some or all of the functions of the camera 100 may be realized by dedicated hardware. The information processing device according to one embodiment of the present invention may be realized by a computer including a processor and a memory.

[0028] The camera 100 has an acquisition unit that acquires biometric information of a person. The biometric information acquired by the acquisition unit is used for biometric authentication. In this embodiment, an image acquisition unit 801 acquires the biometric information of a person. In this embodiment, the image acquisition unit 801 acquires an image of the eyeball 220 of the user looking through the viewfinder (eyepiece 102). Specifically, the image acquisition unit 801 acquires an eye image (eye image signal; electrical signal of the eye image) from the eye imaging element 219 via the gaze detection circuit 301. A specific eye image acquisition process will be described later with reference to FIG. 10.

[0029] The feature calculation unit 802 calculates feature quantities used for authentication from the biometric information. In this embodiment, the feature calculation unit 802 calculates a feature vector used for authentication from the eye image acquired by the image acquisition unit 801. A feature extractor can be used to calculate the feature vector. A neural network can be used as the feature extractor. For example, a convolutional neural network (CNN), which is a type of neural network, can be used. In processing using CNN, abstracted information is extracted from the input image by repeatedly performing processing including convolution processing, activation processing, and pooling processing on the input image. In this case, a processing unit consisting of the convolution processing, activation processing, and pooling processing is called a layer. Various activation processing methods are known. For example, a method called rectified linear unit (ReLU) may be used for the activation processing. Furthermore, various pooling processing methods are known. For example, a method called maximum pooling may be used for the pooling processing. As the CNN, ResNet, which is introduced in non-patent literature (K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In ECCV, 2016), etc., may be used. Alternatively, a neural network known as VisionTransformer (ViT), which is described in non-patent literature (Alexey Dosovitskiy, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021), may be used. The configuration of the neural network is not limited to these. Information indicating the structure and weights of the neural network used by the feature calculation unit 802 may be stored in the memory unit 213, etc.

[0030] The weights of the neural network used by the feature calculation unit 802 are obtained in advance through training. For training, various pre-acquired images of people's eyes can be used. The neural network can be trained using a method such as ArcFace, which is described in non-patent document (J. Deng, J. Guo, N. Xue, and S. Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In CVPR, 2019).

[0031] It is not essential to use an eye image or a neural network as a method for personal authentication. For example, other methods such as iris authentication (for example, the method described in Japanese Patent Laid-Open No. 8-504979) may be used. The method for personal authentication is not limited to a specific method.

[0032] The registration management unit 803 manages registration information of users of devices (camera 100 in this example). In this embodiment, the registration management unit 803 manages feature amounts of registered people and the names of the registered people in association with each other. In this embodiment, feature vectors of eye images are registered as feature amounts. The registration management unit 803 stores the information it manages in the memory unit 213. Here, the registration management unit 803 can manage, in association with each other, registration information used by the first authentication unit 805 to determine the user of the device and registration information used by the second authentication unit 806 to determine the user of the device.

[0033] The registration management unit 803 can manage registration information using, for example, the tables shown in Figures 8(B) to 8(D). Figure 8(B) shows a registered personal information table. Figure 8(C) shows a first authentication registered feature vector table. Figure 8(D) shows a second authentication registered feature vector table. A person ID is assigned to a registered person. Information about the same person is associated between these tables using this person ID. The registered personal information table records information such as "name." The first authentication registered feature vector table records feature vectors (registered feature vectors) used by the first authentication unit 805 for authentication. The second authentication registered feature vector table records feature vectors (registered feature vectors) used by the second authentication unit 806 for authentication.

[0034] The registration unit 804 creates data to be registered in the registration management unit 803. A specific example of the user registration process performed by the registration unit 804 will be described later with reference to FIG.

[0035] The first authentication unit 805 performs a first authentication process to determine whether the user of the device (the camera 100 in this example) is a person who has been registered in advance. In this embodiment, the first authentication unit 805 authenticates whether the user is a person who has been registered in the registration management unit 803 when not capturing images. In this embodiment, the first authentication unit 805 can perform authentication based on a comparison of the user's features obtained during the first authentication process with the features registered in the registration management unit 803. The first authentication unit 805 can also control the first authentication state based on the result of the first authentication process. For example, when the first authentication unit 805 determines that the user is registered, the state management unit 810 (described later) can set the first authentication state to valid. In this way, if the first authentication process is successful, the camera 100 enters the first authentication state (i.e., a state in which the first authentication process is successful). A specific example of the first authentication process will be described later with reference to FIG. 11.

[0036] The second authentication unit 806 performs second authentication processing to determine whether the device user is the same person as the user determined to be registered in the first authentication processing. In this embodiment, the second authentication unit 806 can perform authentication processing based on a comparison of the user's feature amount obtained during the second authentication processing with the feature amount registered in the registration management unit 803. In the second authentication processing, the second authentication unit 806 can perform biometric authentication processing using biometric information acquired via the image acquisition unit 801. In this embodiment, the second authentication unit 806 performs biometric authentication processing using the feature vector generated by the feature calculation unit 802 as the biometric information. The second authentication unit 806 can also perform the second authentication processing when a predetermined operation using the device is performed. The second authentication unit 806 can also perform biometric authentication processing using the biometric information acquired via the image acquisition unit 801 when the device performs a predetermined operation. In this embodiment, the second authentication unit 806 performs processing to authenticate whether the person authenticating the first authentication unit 805 is the person taking the photograph.

[0037] Furthermore, the second authentication unit 806 can control the second authentication state based on the result of the second authentication process. For example, in response to the second authentication unit 806 determining that the user is the same person as the user determined by the second authentication unit 806 to have been registered, the state management unit 810 (described later) can set the second authentication state to valid. In this way, if the second authentication process is successful, the camera 100 enters the second authentication state (i.e., a state in which the second authentication process has been successful). The second authentication unit 806 can control the second authentication state so that this second authentication state continues only while capturing images. For example, in response to the second authentication unit 806 determining that the user has finished capturing images, the state management unit 810 (described later) can set the second authentication state to invalid. A specific example of the second authentication process will be described later with reference to FIG. 12.

[0038] In this embodiment, the authentication settings used by the first authentication unit 805 for the first authentication process are different from the authentication settings used by the second authentication unit 806 for the second authentication process. For example, the first authentication unit 805 can use authentication settings that result in a low false acceptance rate. On the other hand, the second authentication unit 806 can use authentication settings that result in a low false rejection rate. In this way, the second authentication unit 806 can perform authentication using a method that results in a lower false rejection rate than the first authentication unit 805. For example, when the first authentication unit 805 and the second authentication unit 806 use the same authentication method, a method of changing the similarity threshold can be used. The first authentication unit 805 can authenticate the user by comparing information acquired from the user with information registered for the user. Here, if the similarity of the information is higher than the threshold, the first authentication unit 805 can determine that the user is registered. The second authentication unit 806 can also perform authentication using a similar method. For example, the second authentication unit 806 can perform biometric authentication processing by comparing biometric information related to a user determined to be registered with biometric information acquired via the image acquisition unit 801. If the similarity of the information is higher than a threshold, the second authentication unit 806 can determine that the user is the same person as the user determined to be registered. A high threshold can be used in the first authentication processing, and a low threshold can be used in the second authentication processing. This configuration can reduce the false acceptance rate in the first authentication processing and the false rejection rate in the second authentication processing. Even if it is difficult to reduce both the false acceptance rate and the false rejection rate in authentication performed when using a device, performing two-step authentication in this manner can reduce the false rejection rate when using the device while preventing an increase in the false acceptance rate.

[0039] In one embodiment, the second authentication unit 806 determines the first authentication status. Then, when it is determined that the first authentication status is set to valid (i.e., when it is determined that the first authentication processing is successful), the second authentication unit 806 performs the second authentication processing (e.g., biometric authentication processing). With this configuration, when the later-described invalidation unit 808 invalidates the first authentication status, it is possible to prevent other people from using the device based on the result of the second authentication processing.

[0040] The detection unit 807 detects suspicion of device use by a person other than the user determined to be registered by the first authentication unit 805. In this embodiment, the detection unit 807 detects that photography is being performed by a person (other person) other than the person authenticated by the first authentication unit 805. The detection unit 807 can detect suspicion of device use by a person other than the person when a predetermined condition indicating suspicion of device use by a person other than the person is satisfied.

[0041] For example, the detection unit 807 can detect suspicion of use of the device by another person based on the result of the second authentication process performed by the second authentication unit 806. Furthermore, the detection unit 807 can detect suspicion of use of the device by another person based on the history of the second authentication process. For example, the detection unit 807 can detect suspicion of use of the device by another person based on a trend of authentication failures in the second authentication unit 806.

[0042] Specifically, the detection unit 807 can detect suspicion of device use by another person based on the number of failures in the second authentication process. The detection unit 807 may also detect suspicion of device use by another person based on the number of failures in the biometric authentication process. In one embodiment, the detection unit 807 detects suspicion of device use by another person when authentication failures by the second authentication unit 806 occur a predetermined number of times or more consecutively. The second authentication unit 806 may also detect suspicion of device use by another person based on the authentication score. For example, as described above, the second authentication unit 806 can perform biometric authentication processing by comparing authentication information (e.g., biometric information). At this time, the similarity of the compared authentication information can be used as the authentication score. That is, the second authentication unit 806 may detect suspicion of use of the imaging device by a different person based on the similarity of the compared authentication information. For example, when the authentication score is significantly low, the second authentication unit 806 can detect suspicion of device use by another person. The detection unit 807 may use these conditions in combination. Details of the process of detecting suspicion of device use by another person will be described later with reference to FIG. 13. However, methods for detecting suspicion of device use by a third party are not limited to these methods.

[0043] The invalidation unit 808 can perform first authentication invalidation processing, which sets the first authentication status to invalid (i.e., switches to a state where the first authentication processing is not successful) according to predetermined conditions. The invalidation unit 808 determines whether or not to invalidate the first authentication status in the first authentication status where the first authentication by the first authentication unit 805 is successful. The conditions under which the invalidation unit 808 sets the first authentication status to invalid are not particularly limited. Examples of methods for invalidation determination are given below.

[0044] A first invalidation determination method is a method based on the elapsed time since the first authentication process was successful. The invalidation unit 808 can set the first authentication state to invalid based on the elapsed time since the first authentication process was set to valid. For example, when the elapsed time since the first authentication process was successful exceeds a predetermined lifetime, the invalidation unit 808 can invalidate the first authentication state. The invalidation unit 808 can also set the first authentication state to invalid based on the result of the second authentication process. For example, the invalidation unit 808 can change the lifetime based on the result of the second authentication process. Specifically, the invalidation unit 808 can extend the lifetime when the second authentication process is successful in the first authentication state. Conversely, the invalidation unit 808 can shorten the lifetime when the second authentication process fails. An example of an invalidation determination method based on the elapsed time will be described later with reference to FIG. 14(A). However, the invalidation determination method is not limited to this example.

[0045] A second invalidation determination method is a method based on a change in the power state. The invalidation unit 808 can set the first authentication state to invalid based on a change in the power state of the device. For example, the invalidation unit 808 can set the first authentication state to invalid when the camera 100 is turned off or when the camera 100 enters sleep mode. Furthermore, the invalidation unit 808 may set the first authentication state to invalid when the camera 100 is turned on or when the camera 100 returns from sleep mode. With this configuration, even if the process of setting the first authentication state to invalid cannot be executed because the power was turned off due to a dead battery or the like, the first authentication state can be set to invalid when the power is turned on again. The same applies to sleep mode. An example of an invalidation determination method based on a change in the power state will be described later with reference to FIG. 14(B). However, the invalidation determination method is not limited to this example.

[0046] A third invalidation determination method is a method based on the distance or connection status with another device. The invalidation unit 808 can invalidate the first authentication status based on the distance or connection status between a device (e.g., the camera 100) and another device. The other device may be a device carried by the user. An example of the other device is a smartphone. For example, the user's smartphone can be connected to the camera 100 via Bluetooth. When the Bluetooth connection is terminated, the invalidation unit 808 can invalidate the first authentication status. Alternatively, the invalidation unit 808 can estimate the distance between the camera 100 and the other device based on the connection status. When the estimated distance exceeds a predetermined value, the invalidation unit 808 can invalidate the first authentication status. With this configuration, when the user moves away from the camera 100, for example, by putting the camera 100 down, the first authentication status is invalidated. This makes it possible to prevent other people from using the camera 100. Note that an example of the other device is an RFID tag. The other device is not limited to these. An example of an invalidation determination method based on the distance or connection status with another device will be described later with reference to FIG. 14(C). However, the invalidation determination method is not limited to this example.

[0047] A fourth invalidation determination method is a method based on an explicit invalidation operation input by the user. The invalidation unit 808 can set the first authentication state to invalid based on an input by the user to a device (e.g., camera 100). For example, an item "Invalidate First Authentication" can be included in an operation menu. The user can select this item using operation members 103 to 105. Alternatively, the camera 100 may be provided with a switch button such as an invalidation button. The user can press this button. When this operation is received, the invalidation unit 808 can invalidate the first authentication state. An example of an explicit invalidation operation input by the user will be described later with reference to FIG. 14(D). However, the invalidation method is not limited to this example.

[0048] A fifth invalidation determination method is a method based on the detection result of the detection unit 807. The invalidation unit 808 can set the first authentication status to invalid when it is detected that a person other than the user determined to be registered is suspected of using the device. In this way, when the detection unit 807 detects that a third party is suspected of using the device, it is possible to invalidate the first authentication status.

[0049] The invalidation unit 808 can use a combination of the above-described multiple invalidation determination methods. By using these methods in combination, the possibility of recognizing a different person as a specific registered person in the second authentication process can be further reduced. For example, the fourth method can prevent use by a different person through a conscious operation by the user. In addition, the first to third methods invalidate the first authentication process when it is unlikely that the user authenticated in the first authentication process is using the device, thereby preventing use by a different person. Furthermore, the fifth method invalidates the authentication status when it is suspected that a different person is using the device, thereby preventing use by a different person.

[0050] The execution unit 809 controls the operation of the device in accordance with the determination result by the second authentication unit 806. The execution unit 809 can control the operation of the device in accordance with the second authentication state. The execution unit 809 can further control the operation of the device in accordance with the determination result by the first authentication unit 805. Specifically, the execution unit 809 can control the operation of the device in accordance with the first authentication state. In this way, the execution unit 809 can execute processing in accordance with the first authentication state and the second authentication state.

[0051] In one embodiment, the execution unit 809 can perform a specific operation when both the first authentication status and the second authentication status are valid. The specific operation is not particularly limited. For example, the specific operation may be a photographing operation using the camera 100. In another embodiment, the execution unit 809 can record the first authentication status and the second authentication status. For example, the execution unit 809 can save the records of the first authentication status and the second authentication status in association with the record of the result of the specific operation. For example, the execution unit 809 can save the captured image obtained by the photographing operation and the records of the first authentication status and the second authentication status in association with each other in the memory unit 213.

[0052] The execution unit 809 includes a state management unit 810, a state saving unit 811, and a state display unit 812. The state management unit 810 can manage the first authentication state of the device. The state management unit 810 can also manage the second authentication state of the device. Furthermore, the state management unit 810 can manage information indicating whether or not the device is suspected of being used by another person. The state management unit 810 stores the managed information in the memory unit 213. For example, the state management unit 810 can store the authentication state table shown in FIG. 8E in the memory unit 213. In the authentication state table, the "first authentication state" indicates whether or not the device is in the first authentication state. The "first authentication state" takes one of two values: "authenticated," which indicates that the first authentication state is valid, and "unauthenticated," which indicates that the first authentication state is invalid. The "person ID" is the person ID of the user identified by the first authentication process. When the "first authentication state" is "unauthenticated," the "person ID" takes a value indicating an empty value, such as NULL. The "second authentication state" indicates whether or not the device is in the second authentication state. The "second authentication status" takes one of two values: "authenticated," which indicates that the second authentication status is valid, and "unauthenticated," which indicates that the first authentication status is invalid. The "presence or absence of use by another person" indicates whether or not the detection unit 807 has detected suspicion of use by another person. The "presence or absence of use by another person" takes one of two values: "yes" and "no." Specific examples of the process for updating the authentication status table will be described later together with the explanations of the first authentication process (FIG. 11), the second authentication process (FIG. 12), and the first authentication invalidation process (FIG. 14). The method for managing the authentication status table is not limited to the method using the table structure shown in FIG. 8(E). For example, the authentication status table may be managed using a key-value structure or the like. Furthermore, the management format of this information is not limited to the above.

[0053] The imaging unit 813 records the image captured by the imaging element 211 in the memory unit 213. The imaging unit 813 may perform such a photographing operation in response to receiving a signal indicating that the release button 101 has been pressed by the user.

[0054] The state saving unit 811 performs control so that information indicating each of the first authentication status and the second authentication status is recorded in the memory. For example, the state saving unit 811 can record information indicating each of the first authentication status and the second authentication status managed by the state management unit 810 in the memory unit 213 in association with a captured image acquired by the imaging unit 813. The state saving unit 811 can record information indicating the authentication status and suspicion of use by another person as metadata of the captured image. Methods for recording image metadata are described in non-patent documents (Coalition for Content Provenance and Authenticity (C2PA), "C2PA Specifications" and<Technical Specifications Version 1.2> , [online], November 3, 2022, [Retrieved January 23, 2023], Internet<URL:https: / / c2pa.org / specifications / specifications / 1.2 / specs / C2PA_Specification.html> ) is an example of a method known as C2PA. C2PA relates to a method of adding metadata to an image indicating edits made to the image in order to authenticate the origin, history, or provenance of the image. For this reason, the authentication status and the like may be recorded according to C2PA. However, this information such as the authentication status may also be recorded by other methods. Alternatively, the image file and the metadata file may be recorded as separate files. Alternatively, the metadata may be managed in a database. However, the metadata recording method is not limited to these. A specific example of the authentication status storage process will be described later with reference to FIG. 15(A).

[0055] The status display unit 812 notifies the authentication status of the device. For example, the status display unit 812 can separately notify the first authentication status and the second authentication status. The status display unit 812 may notify the authentication status of the device via the device. For example, the status display unit 812 may display the authentication status of the device on the device. In this embodiment, the status display unit 812 notifies the first authentication status on the camera 100. For example, the status display unit 812 can display the authentication status on the camera 100 based on the authentication status managed by the status management unit 810. When the first authentication status is "authenticating," the status display unit 812 can cause the display device 214 or the touch panel (operation member 103) to display "first authentication in progress." The camera 100 may also be equipped with a lamp such as an LED lamp (not shown). The status display unit 812 can turn on the lamp when the first authentication status is "authenticating."

[0056] [User registration process] The user registration process in this embodiment will be described with reference to the flowchart in FIG. 9. This process is mainly performed by the registration unit 804. The CPU 212 can realize the operation of the registration unit 804. The user can start the registration process by operating the camera 100 at a time other than when taking a picture. For example, this process may be executed when the user operates the camera 100 to call this process from a menu or the like. For example, a menu screen (not shown) can be displayed on the touch panel (operation member 103) of the camera 100. The user can select an item to call this process by operating the operation members 103 to 105. Below, this process will be described step by step.

[0057] In S901, the registration unit 804 accepts input of personal information of the person to be registered. In this embodiment, a "name" is input. Specifically, a screen for inputting a name (not shown) is displayed on the touch panel (operation member 103). The user then operates the operation members 103 to 105 to input the name. When the user has finished inputting the name, the user notifies the user that the name input is complete by pressing a complete button or the like displayed on the screen.

[0058] In S902, the registration unit 804 instructs the user on how to register an eye image. Specifically, the registration unit 804 can cause the touch panel to display instructions that the user should look into the viewfinder. The registration unit 804 can cause the touch panel to display instructions that the user should look at an indicator in the viewfinder. In addition, the registration unit 804 can cause the touch panel to display instructions that are helpful for capturing a desirable eye image, such as not blinking and keeping the eyes wide open.

[0059] Next, in S903 to S911, the indices 421 to 425 shown in Fig. 4(C) are displayed in order on the display device 214. Then, the feature vectors obtained from the image of the user's eyes looking at these indices are stored in the registration management unit 803. The processing will be explained in order below.

[0060] In S903, the registration unit 804 displays the indices on the display device 214. Specifically, only the index 421 shown in FIG. 4C is displayed, and the other indices are not displayed. As an alternative method, all the indices may be displayed, while only the index 421 is highlighted in color. Other display methods that can inform the user that the index 421 is to be viewed may also be used. The method of displaying the indices is not limited to these.

[0061] In S904, an image of the user's eye looking through the viewfinder (eyepiece 102) is acquired. The detailed processing in S904 will be described with reference to FIG. 10. This processing is mainly performed by the image acquisition unit 801. The CPU 212 can realize the operation of the image acquisition unit 801.

[0062] In step S1001, a gaze detection process is executed. The gaze detection process will be described in detail later with reference to the flowchart in FIG.

[0063] In S1002, the image acquisition unit 801 determines whether an image suitable for authentication has been acquired. The image acquisition unit 801 can make this determination based on whether the gaze detection process has been successful. For example, in the gaze detection process, the image acquisition unit 801 acquires an eye image (eye image signal; an electrical signal of the eye image) from the eye imaging element 219 via the gaze detection circuit 301. Then, the image acquisition unit 801 obtains the corneal reflection images of the light sources 216a and 216b observed on the eye image and the coordinates of the pupil center. Then, the image acquisition unit 801 obtains the user's gaze coordinates on the display device 214 from these coordinates. Therefore, if the coordinates of the pupil center, etc. cannot be detected, the gaze detection process fails. Therefore, in S703, which will be described later, if the coordinates of the pupil center, etc. cannot be obtained, the image acquisition unit 801 may determine that an image suitable for authentication has not been acquired.

[0064] In S1003, the image acquisition unit 801 performs processing control based on the determination result in S1002. If the image acquisition unit 801 determines in S1002 that an image suitable for authentication has been acquired, the process proceeds to S1004. Otherwise, the process proceeds to S1006.

[0065] In S1004, the image acquisition unit 801 acquires an eye image. Specifically, the image acquisition unit 801 acquires the eye image acquired in S702. Then, using the coordinates of the pupil-centered image c' acquired in S703, the image acquisition unit 801 crops an image of a certain size from the eye image so that the pupil-centered image c' is located at the center of the image. Furthermore, the image acquisition unit 801 resizes the acquired image to fit the input size of the neural network used by the feature calculation unit 802.

[0066] In S1005, the image acquisition unit 801 uses a flag or the like to record that the eye image has been successfully acquired.

[0067] In S1006, the image acquisition unit 801 performs a process of waiting for a predetermined time, such as several hundred milliseconds. This is expected to change the eye image obtained, leading to successful gaze detection.

[0068] In S1007, the image acquisition unit 801 determines whether or not failures have occurred consecutively. A failure indicates that it has been determined in S1003 that an image suitable for authentication cannot be acquired. If the image acquisition unit 801 determines that failures have occurred consecutively a predetermined number of times, the process proceeds to S1008. Otherwise, the process returns to S1001.

[0069] In S1008, the image acquisition unit 801 uses a flag or the like to record the fact that acquisition of the eye image has failed.

[0070] Returning now to the description of Figure 9, in S905 the registration unit 804 determines whether or not the acquisition of the eye image in S904 was successful. The registration unit 804 can make this determination based on the flag recorded in S1005 or S1008. If the registration unit 804 determines that the acquisition of the eye image was successful, the process proceeds to S906. Otherwise, the process proceeds to S908.

[0071] In S906, a feature vector is extracted from the eye image. Specifically, the feature calculation unit 802 can extract a feature vector from the eye image acquired in S1004.

[0072] In S907, the registration unit 804 causes the display device 214 to display information indicating that the eye image corresponding to the displayed index has been successfully captured. For example, the display device 214 may display a message such as "Eye image captured successfully." Alternatively, the display device 214 may display an icon indicating success.

[0073] In S908, the registration unit 804 causes the display device 214 to display information indicating that capturing an eye image corresponding to the displayed index has failed. For example, the display device 214 may display a message such as "Failed to capture an eye image." Alternatively, the display device 214 may display an icon indicating the failure.

[0074] In S909, the registration unit 804 determines whether there are any indices that have not yet been displayed. For example, the registration unit 804 determines whether all of the indices 421 to 425 shown in FIG. 4C have already been displayed. If the registration unit 804 determines that there are any indices that have not yet been displayed, the process proceeds to S910. Otherwise, the process proceeds to S911.

[0075] In S910, the registration unit 804 performs processing to display the next index on the display device 214. For example, when the index 421 shown in Fig. 4(C) is displayed, the display device 214 can display the index 422 as the next index. In this way, the registration unit 804 selects the indexes 421 to 425 in order, and the display device 214 displays the selected index.

[0076] In S911, the registration management unit 803 registers the obtained information. The registration management unit 803 can save the obtained information in a registered personal information table, a first authentication registered feature vector table, and a second authentication registered feature vector table. The person IDs in the three tables are IDs for establishing relationships between the tables. Therefore, the same person ID is recorded in the three tables in association with information about the same person. In addition, the registration management unit 803 adds the personal information (e.g., name) obtained in S901 to the registered personal information table shown in FIG. 8(B). Furthermore, the registration management unit 803 records the feature vector obtained in S906 in the first authentication registered feature vector table shown in FIG. 8(C) and the second authentication registered feature vector table shown in FIG. 8(D). Here, the information recorded in the first authentication registered feature vector table and the second authentication registered feature vector table may differ as follows.

[0077] In this embodiment, in the first authentication process (described later with reference to FIG. 11 ), the display device 214 displays an index, and authentication is performed based on an image of the user's eye when the user is looking at the index. Therefore, only the feature vector corresponding to the index displayed in the first authentication process can be saved in the first authentication registered feature vector table as a registered feature vector for the first authentication process. In this embodiment, the feature vector extracted from the eye image obtained when the index 421 was displayed is registered.

[0078] On the other hand, in the second authentication process (described later with reference to FIG. 12), the display device 214 displays an image obtained by the image sensor 211, and authentication is performed based on the eye image when the user is looking at this image. As such, in the second authentication process, no indices are displayed. Therefore, it is unclear where on the display device 214 the user is gazing. Therefore, in the present embodiment, all feature vectors extracted in the process of FIG. 9 are stored in a second authentication registration feature vector table. That is, the second authentication registration feature vector table records each feature vector extracted from the eye image obtained when the indices 421 to 425 were displayed as a registration feature vector for the second authentication process.

[0079] In S912, the registration unit 804 notifies the user via the touch panel (operation member 103) or a display on the display device 214 that registration has been completed.

[0080] 9, if the registration unit 804 detects that it has failed to acquire eye images a predetermined number of times, the registration unit 804 can interrupt the user registration process. This process can prevent excessive repetition of the process when acquisition of eye images in S904 is unsuccessful. Such exceptional processes can be added as appropriate.

[0081] [First authentication process] The first authentication process performed in this embodiment will be described with reference to the flowchart in FIG. 11. This process is mainly performed by the first authentication unit 805. The CPU 212 can realize the operation of the first authentication unit 805. The user can start the first authentication process by operating the camera 100 at a time other than when taking a picture. For example, this process may be executed when the user operates the camera 100 to call this process from a menu or the like. For example, a menu screen (not shown) can be displayed on the touch panel (operation member 103) of the camera 100. The user can select an item to call this process by operating the operation members 103 to 105. Below, this process will be described step by step.

[0082] In S1101, the first authentication unit 805 instructs the user on the authentication method. Specifically, the first authentication unit 805 can cause the touch panel (operation member 103) to display instructions indicating that the user should look into the viewfinder and see an indicator displayed on the display device 214. In addition, the first authentication unit 805 can cause the touch panel to display instructions that are useful for capturing a desirable eye image, such as not blinking, keeping the eyes wide open, and holding the camera firmly.

[0083] In S1102, the first authentication unit 805 displays an index on the display device 214. Specifically, the display device 214 can display only the index 421, as shown in FIG. 4(D). This is because the feature vector obtained when the user was looking at the index 421 in the above-mentioned registration process is registered in the first authentication registration feature vector table. By displaying in this manner, similar eye images are obtained at the time of registration and authentication. This makes it easy to compare the eye images (or feature vectors) of both.

[0084] In S1103, an image of the eye of the user looking through the finder (eyepiece 102) is acquired. Specifically, the eye image can be acquired by the process already described with reference to FIG.

[0085] In S1104, the first authentication unit 805 determines whether or not the acquisition of the eye image in S1103 was successful. The first authentication unit 805 can make this determination based on the flag recorded in S1005 or S1008. If the first authentication unit 805 determines that the acquisition of the eye image was successful, the process proceeds to S1106. Otherwise, the process proceeds to S1105.

[0086] In S1105, the first authentication unit 805 displays information indicating that capturing an eye image has failed on the display device 214. For example, the display device 214 may display a message such as "Failed to capture an eye image." Alternatively, the display device 214 may display an icon indicating the failure.

[0087] In S1106, a feature vector is extracted from the eye image. Specifically, the feature calculation unit 802 can extract a feature vector from the eye image acquired in S1103.

[0088] In S1107, the first authentication unit 805 acquires a registered feature vector for the first authentication process from the registration management unit 803. Specifically, the first authentication unit 805 can acquire a feature vector registered in a first authentication registered feature vector table shown in Fig. 8(C).

[0089] In S1108, the first authentication unit 805 compares the feature vector obtained in S1106 with the registered feature vector obtained in S1107. The first authentication unit 805 then determines whether the registered feature vector matches the feature vector obtained in S1106. This determination can be made based on the similarity between the feature vectors. For example, the first authentication unit 805 can calculate the cosine similarity between the two feature vectors being compared. If the similarity calculated for the registered feature vector exceeds a predetermined threshold, the first authentication unit 805 can determine that this registered feature vector matches the feature vector obtained in S1106. In this case, the first authentication unit 805 identifies the person ID for the registered feature vector that matches the feature vector obtained in S1106.

[0090] In S1109, the first authentication unit 805 determines whether the first authentication process has been successful. Specifically, the first authentication unit 805 can determine that the first authentication process has been successful if the registered feature vector matches the feature vector obtained in S1106. At this time, the first authentication unit 805 has determined that the user of the camera 100 is a pre-registered person corresponding to the person ID identified in S1108. If the first authentication unit 805 determines that the first authentication process has been successful, the process proceeds to S1110. Otherwise, the process proceeds to S1113.

[0091] In S1110, the state management unit 810 enables the first authentication state. For example, the state management unit 810 can update the "first authentication state" in the authentication state table shown in FIG. 8(E) to "authenticating (authentication successful state)". At this time, the state management unit 810 can disable the second authentication state. For example, the state management unit 810 can update the "second authentication state" in the authentication state table to "unauthenticated".

[0092] In S1111, the state management unit 810 registers information about the user who has been determined to be registered by the first authentication unit 805. For example, the state management unit 810 can update the person ID in the authentication state table shown in Fig. 8(E) to the person ID identified in S1108.

[0093] In S1112, the status display unit 812 uses the touch panel (operation member 103) or the display device 214 to display to the user that the authentication has been successful.

[0094] In S1113, the state management unit 810 invalidates the first authentication state. The state management unit 810 can also invalidate the second authentication state. For example, the state management unit 810 can update the "first authentication state" and "second authentication state" in the authentication state table shown in FIG. 8(E) to "unauthenticated."

[0095] In S1114, the state management unit 810 deletes the record of the information of the user determined by the first authentication unit 805 to be registered. For example, the state management unit 810 can delete the person ID recorded in the authentication state table shown in FIG. 8(E). For example, a NULL value or the like can be prepared as a value indicating that the person ID is empty. In this case, the state management unit 810 can overwrite the person ID with a value indicating that the person ID is empty.

[0096] In S1115, the status display unit 812 uses the touch panel (operation member 103) or the display device 214 to display a message informing the user of the authentication failure.

[0097] 11, if the first authentication unit 805 detects that acquisition of an eye image has failed a predetermined number of times, the first authentication unit 805 can interrupt the first authentication process. This process can prevent excessive repetition of the process when acquisition of an eye image in S1104 is unsuccessful. Such exceptional processes can be added as appropriate.

[0098] [Second authentication process] The second authentication process performed in this embodiment will be described with reference to the flowchart in FIG. 12. This process is mainly performed by the second authentication unit 806. The CPU 212 can realize the operation of the second authentication unit 806. The second authentication process can be performed after the first authentication process. Specifically, the second authentication process can be performed when the user looks into the viewfinder during shooting. For this purpose, an eyepiece sensor (not shown) mounted on the camera 100 can detect when the user approaches the viewfinder (eyepiece 102). When such an eyepiece sensor detects that the user approaches the viewfinder, this process can be initiated. The eyepiece sensor may detect when the skin around the user's eye comes into contact with the periphery of the eyepiece 102. Alternatively, the eyepiece sensor may detect the distance between the eyepiece 102 and the user's eye. If the detected distance is equal to or less than a predetermined distance, it can be determined that the user is looking into the viewfinder. Alternatively, this process may be initiated when it is detected that the release button 101 has been pressed down to the first stroke. Alternatively, the gaze detection process may be continued. Then, when the gaze detection process is successful, this process may be started. Below, this process will be explained step by step.

[0099] In S1201, the second authentication unit 806 determines whether the first authentication status is valid. The second authentication unit 806 can determine whether the first authentication status is valid, for example, based on whether the "first authentication status" in the authentication status table shown in FIG. 8(E) is "authenticated." The second authentication unit 806 further determines whether the user is continuing to capture images. The second authentication unit 806 can determine whether the user is continuing to capture images based on whether the user is keeping their eye close to the viewfinder. Whether the user is keeping their eye close to the viewfinder can be detected by an eyepiece sensor. As described above, whether the user is continuing to capture images may also be determined according to other methods, such as pressing the release button 101 or gaze detection processing. If the second authentication unit 806 determines that the first authentication status is valid and the user is continuing to capture images, the process proceeds to S1202. Otherwise, the process proceeds to S1218.

[0100] In S1202, an image of the user's eyes looking through the viewfinder (eyepiece 102) is acquired. Specifically, the image of the eyes can be acquired by the process already described with reference to FIG. 10. Note that if the gaze detection process has already been performed, the gaze detection in S1001 can be skipped. In this case, the image of the user's eyes can be acquired using the results of the gaze detection process that have already been obtained.

[0101] In S1203, the second authentication unit 806 determines whether or not the acquisition of the eye image in S1202 was successful. The second authentication unit 806 can make this determination based on the flag recorded in S1005 or S1008. If the second authentication unit 806 determines that the acquisition of the eye image was successful, the process proceeds to S1205. Otherwise, the process proceeds to S1204.

[0102] In S1204, the second authentication unit 806 displays information indicating that capturing an eye image has failed on the display device 214. For example, the display device 214 may display an icon indicating the failure.

[0103] In S1205, a feature vector is extracted from the eye image. Specifically, the feature calculation unit 802 can extract a feature vector from the eye image acquired in S1202.

[0104] In S1206, the second authentication unit 806 acquires registered feature vectors for the second authentication process from the registration management unit 803. Specifically, the second authentication unit 806 acquires all registered feature vectors registered in the second authentication registered feature vector table shown in Fig. 8(D).

[0105] In S1207, the second authentication unit 806 compares the feature vector obtained in S1205 with each of the registered feature vectors obtained in S1206. Then, the second authentication unit 806 determines whether any of the registered feature vectors matches the feature vector obtained in S1205. This determination can be made based on the similarity, or cosine similarity, between the feature vectors, as in S1108. Then, if the similarity calculated for the registered feature vector exceeds a predetermined threshold, the second authentication unit 806 can determine that this registered feature vector matches the feature vector obtained in S1205.

[0106] In S1208, the second authentication unit 806 determines whether the second authentication process is successful. Specifically, the second authentication unit 806 can determine that the second authentication process is successful if any registered feature vector matches the feature vector obtained in S1205. If the second authentication unit 806 determines that the second authentication process is successful, the process proceeds to S1209. Otherwise, the process proceeds to S1213.

[0107] In S1209, the state management unit 810 validates the second authentication state. For example, the state management unit 810 updates the "second authentication state" in the authentication state table shown in FIG.

[0108] In S1210, the status display unit 812 displays a message informing the user of successful authentication using the display device 214. For example, the display device 214 may display an icon indicating successful authentication.

[0109] In S1211, the second authentication unit 806 determines whether the user is continuing to take pictures. The determination of whether the user is continuing to take pictures can be performed in the same manner as in S1201. If the second authentication unit 806 determines that the user is continuing to take pictures, the determination process of S1211 is repeated. Otherwise, the process proceeds to S1212.

[0110] In both S1212 and S1213, the state management unit 810 invalidates the second authentication state. For example, the state management unit 810 can update the "second authentication state" in the authentication state table shown in FIG. 8(E) to "unauthenticated."

[0111] In S1214, the detection unit 807 performs a process to detect suspicion of use by another person. That is, the detection unit 807 determines whether or not there is suspicion that the camera 100 is being used by a person other than the user determined to be registered in the first authentication process. Details of this process will be described later with reference to FIG. 13.

[0112] In S1215, the invalidation unit 808 determines whether the detection unit 807 detected suspicion of use by another person in S1214. If the invalidation unit 808 determines that suspicion of use by another person has been detected, the process proceeds to S1216. Otherwise, the process proceeds to S1217.

[0113] In S1216, the invalidation unit 808 performs processing to invalidate the first authentication state. For example, the invalidation unit 808 can control the state management unit 810 to update the "first authentication state" in the authentication state table shown in FIG. 8(E) to "unauthenticated." The invalidation unit 808 can also control the state management unit 810 to delete the person ID recorded in the authentication state table. By performing such processing, the first authentication state can be invalidated when use by another person is suspected.

[0114] In S1217, the status display unit 812 displays a message informing the user of the authentication failure using the display device 214 or the like. For example, the display device 214 may display an icon indicating the authentication failure. Note that, when the detection unit 807 detects suspicion of use by another person, the status display unit 812 may notify the user that suspicion of use of the device (camera 100 in this example) by a different person has been detected.

[0115] In S1218, the status display unit 812 updates the display on the display device 214. If image capture is no longer in progress, there is no need to display the authentication status. On the other hand, if image capture is in progress but the first authentication status has been invalidated, the display device 214 can display a message or an icon indicating that the first authentication status has been invalidated.

[0116] [Detection of unauthorized use] The process of detecting use by another person carried out in this embodiment will be described with reference to the flowchart in Fig. 13. This process is mainly carried out by the detection unit 807. The CPU 212 can realize the operation of the detection unit 807. This process is called from S1214 in Fig. 12. Each step of this process will be described below.

[0117] In S1301, the detection unit 807 records the result of the second authentication process. In this embodiment, the unauthorized use detection process is performed when the second authentication process fails. Therefore, the detection unit 807 can record a history of failed second authentication processes. For example, the detection unit 807 can record the time of failure or the similarity between feature vectors calculated in the second authentication process. The detection unit 807 may record the maximum similarity among the similarities calculated for each of the multiple registered feature vectors.

[0118] In S1302, the detection unit 807 detects the suspicion of another person using the camera 100 based on the failure history recorded in S1301. The detection unit 807 can determine whether the failure history satisfies a predetermined condition. Examples of the predetermined condition have already been described. The detection unit 807 may, for example, detect the suspicion of another person using the device when authentication failures occur a predetermined number of times or more in succession. Furthermore, the detection unit 807 may detect the suspicion of another person using the camera 100 when, for example, the number of failures in the second authentication process within a predetermined time range up to the current time exceeds a threshold. The detection unit 807 may count multiple failures occurring during the same shooting as a single failure. Furthermore, the detection unit 807 may count only the number of failures in which the similarity is below a predetermined threshold. Furthermore, the detection unit 807 may record the number of successes in the second authentication process in addition to the number of failures. For example, immediately after S1209, the detection unit 807 may record the success of the second authentication process. At this time, if the second authentication process is successful at least once during the same photographing, the detection unit 807 does not need to count failures during this photographing.

[0119] In S1303, the detection unit 807 determines whether or not the detection unit 807 detected suspicion of use of the camera 100 by another person in S1302. If the detection unit 807 detected suspicion of use by another person, the process proceeds to S1306. Otherwise, the process proceeds to S1304.

[0120] In S1304, the detection unit 807 determines whether the maximum similarity between the feature vectors obtained in S1207 is equal to or less than a predetermined threshold. In S1207, similarities are calculated for each of the registered feature vectors associated with the same person ID. The detection unit 807 obtains the highest similarity among these similarities and determines whether this similarity is equal to or less than a predetermined threshold. Under poor shooting conditions, the calculated similarity decreases even when the subject's eye image is captured. However, the similarity calculated using the subject's eye image tends to be higher than when a different person's eye image is used. Therefore, a threshold can be set in advance to determine whether the image is a different person's eye image. Then, based on the similarity being equal to or less than the set threshold, the detection unit 807 determines that a different person is using the camera 100. If the detection unit 807 determines that the similarity is equal to or less than the threshold, the process proceeds to S1306. Otherwise, the process proceeds to S1305.

[0121] In S1305, the detection unit 807 records that there is no suspicion of use of the camera 100 by another person. Specifically, the detection unit 807 can turn off a flag recorded in the memory unit 213 indicating suspicion of use by another person.

[0122] In S1306, the detection unit 807 records that there is a suspicion that the camera 100 is being used by another person. Specifically, the detection unit 807 can turn on a flag recorded in the memory unit 213 indicating that there is a suspicion that the camera 100 is being used by another person.

[0123] [First authentication invalidation process] The first authentication invalidation process performed in this embodiment will be described with reference to the flowchart in FIG. 14. This process is mainly performed by the invalidation unit 808. The CPU 212 can realize the operation of the invalidation unit 808. As described above, the invalidation unit 808 can set the first authentication status to invalid in accordance with predetermined conditions. Examples of the predetermined conditions have already been described. Below, as specific examples, processes according to four types of conditions will be described.

[0124] First, the first authentication invalidation process based on elapsed time will be described with reference to the flowchart in Fig. 14(A). This process can be started periodically using a timer or the like.

[0125] In S1401, the invalidation unit 808 determines whether the first authentication status is valid. Specifically, the invalidation unit 808 can determine whether the "first authentication status" in the authentication status table shown in FIG. 8(E) is "authenticated" via the status management unit 810. If the invalidation unit 808 determines that the first authentication status is valid, the process proceeds to S1402. Otherwise, the process ends.

[0126] In S1402, the invalidation unit 808 calculates the elapsed time since the first authentication process was successful. The invalidation unit 808 can calculate the elapsed time since the "first authentication status" in the authentication status table was updated to "authenticating". The status management unit 810 can record the time when the "first authentication status" was updated to "authenticating". The invalidation unit 808 can calculate the difference between the time when the "first authentication status" was updated to "authenticating" and the current time as the elapsed time.

[0127] In S1403, the invalidation unit 808 detects the execution of the second authentication process. The invalidation unit 808 can detect whether the second authentication process has been executed between the previous first authentication invalidation process according to a timer or the like and the current first authentication invalidation process. If the invalidation unit 808 detects the execution of such second authentication process, the process proceeds to S1404. Otherwise, the process proceeds to S1407.

[0128] In S1404, the invalidation unit 808 determines whether the second authentication process detected in S1403 was successful. If the invalidation unit 808 determines that the second authentication process was successful, the process proceeds to S1405. Otherwise, the process proceeds to S1406.

[0129] In S1405, the invalidation unit 808 adds a predetermined time to the lifetime. Note that the initial value of the lifetime is initialized to a predetermined value when the state management unit 810 updates the "first authentication state" to "authenticated."

[0130] In S1406, the invalidation unit 808 subtracts a predetermined time from the lifetime.

[0131] In S1407, the invalidation unit 808 determines whether the elapsed time calculated in S1402 has exceeded the lifetime. If the invalidation unit 808 determines that the elapsed time has exceeded the lifetime, the process proceeds to S1408. Otherwise, the process ends.

[0132] In S1408, in accordance with the instruction from the invalidation unit 808, the state management unit 810 updates the "first authentication state" in the authentication state table shown in Fig. 8(E) to "unauthenticated". At this time, the state management unit 810 can delete the person ID recorded in the authentication state table. In addition, the state management unit 810 can update the "second authentication state" in the authentication state table to "unauthenticated".

[0133] Next, the first authentication invalidation process based on a change in the power state will be described with reference to the flowchart in Fig. 14(B). This process can be started when the power state of the camera 100 changes. For example, this process can be started when the camera 100 is turned on, turned off, goes into sleep mode, or returns from sleep mode.

[0134] In S1411, the invalidation unit 808 determines whether the first authentication status is valid. Specifically, the invalidation unit 808 can determine whether the "first authentication status" in the authentication status table shown in FIG. 8(E) is "authenticated" via the status management unit 810. If the invalidation unit 808 determines that the first authentication status is valid, the process proceeds to S1412. Otherwise, the process ends.

[0135] In S1412, the invalidation unit 808 determines whether the power state of the camera 100 has changed from ON to OFF or from ON to OFF. If the invalidation unit 808 detects such a change in the power state of the camera 100, the process proceeds to S1414. Otherwise, the process proceeds to S1413. Note that the invalidation unit 808 may determine whether there has been only one of a change from ON to OFF or a change from OFF to ON. Alternatively, if the invalidation unit 808 determines that at least one of a change from ON to OFF and a change from OFF to ON has occurred, the process may proceed to S1414.

[0136] In S1413, the invalidation unit 808 determines whether the camera 100 has transitioned to sleep mode or has returned from sleep mode. If the invalidation unit 808 detects such a change in the sleep mode of the camera 100, the process proceeds to S1414. Otherwise, the process ends. Note that the invalidation unit 808 may determine whether only one of a transition to sleep mode or a return from sleep mode has occurred. Alternatively, if the invalidation unit 808 determines that at least one of a transition to sleep mode and a return from sleep mode has occurred, the process may proceed to S1414.

[0137] In S1414, in accordance with the instruction from the invalidation unit 808, the state management unit 810 updates the "first authentication state" in the authentication state table shown in Fig. 8(E) to "unauthenticated". At this time, the state management unit 810 can delete the person ID recorded in the authentication state table. In addition, the state management unit 810 can update the "second authentication state" in the authentication state table to "unauthenticated".

[0138] Next, the first authentication invalidation process based on a change in the power state due to a change in the distance or connection status with another device will be described with reference to the flowchart in FIG. 14(C). This process can be started periodically using a timer or the like. It is also assumed that the camera 100 is connected in advance to a specific device that the user carries. As an example, a pairing process may be performed in advance so that the smartphone and the camera 100 are connected via Bluetooth.

[0139] In S1421, the invalidation unit 808 determines whether the first authentication status is valid. Specifically, the invalidation unit 808 can determine whether the "first authentication status" in the authentication status table shown in FIG. 8(E) is "authenticated" via the status management unit 810. If the invalidation unit 808 determines that the first authentication status is valid, the process proceeds to S1402. Otherwise, the process ends.

[0140] In S1422, the invalidation unit 808 determines whether the connection status with the specific device has deteriorated beyond a predetermined condition. The predetermined condition may be, for example, that the signal strength of the connection falls below a certain threshold. If the invalidation unit 808 determines that the connection status has deteriorated, the process proceeds to S1423. Otherwise, the process ends.

[0141] In S1423, in accordance with the instruction from the invalidation unit 808, the state management unit 810 updates the "first authentication state" in the authentication state table shown in Fig. 8(E) to "unauthenticated". At this time, the state management unit 810 can delete the person ID recorded in the authentication state table. In addition, the state management unit 810 can update the "second authentication state" in the authentication state table to "unauthenticated".

[0142] Next, the first authentication invalidation process based on an explicit invalidation operation input by the user will be described with reference to the flowchart in Fig. 14(D). This process can be executed at any time while the camera 100 is activated.

[0143] In S1431, the invalidation unit 808 determines whether the first authentication status is valid. Specifically, the invalidation unit 808 can determine whether the "first authentication status" in the authentication status table shown in FIG. 8(E) is "authenticated" via the status management unit 810. If the invalidation unit 808 determines that the first authentication status is valid, the process proceeds to S1432. Otherwise, this process ends.

[0144] In S1432, the disabling unit 808 waits for a disabling operation by the user. The disabling operation may be, for example, the user pressing a switch button (not shown) installed on the camera 100. Alternatively, the disabling operation may be the selection of an item corresponding to the disabling operation in a menu displayed on a touch panel or the like. Examples of the disabling operation are not limited to these.

[0145] In S1433, the invalidation unit 808 determines whether or not an invalidation operation by the user has been detected. If the invalidation unit 808 detects an invalidation operation, the process proceeds to S1434. Otherwise, the process ends.

[0146] In S1434, in accordance with the instruction from the invalidation unit 808, the state management unit 810 updates the "first authentication state" in the authentication state table shown in Fig. 8(E) to "unauthenticated". At this time, the state management unit 810 can delete the person ID recorded in the authentication state table. In addition, the state management unit 810 can update the "second authentication state" in the authentication state table to "unauthenticated".

[0147] [Authentication state saving process] The authentication status save process in this embodiment will be described with reference to the flowchart in FIG. 15(A). This process is mainly performed by the state save unit 811. The CPU 212 can realize the operation of the state save unit 811. The authentication status save process can be performed after the second authentication process. Specifically, the authentication status save process can be performed after a photographing operation by the camera 100. In this embodiment, the authentication status is saved along with the image taken by the user. Specifically, the state save unit 811 can generate an image file shown in FIG. 15(B) and save it in the memory unit 213. This process can be started when the release button 101 is pressed down to the second stroke.

[0148] In S1501, the imaging unit 813 performs imaging. Specifically, the imaging unit 813 performs imaging processing to convert light received by the imaging element 211 into an electrical signal.

[0149] In S1502, the imaging unit 813 generates image data. Specifically, the imaging unit 813 generates image data 1530 by performing image processing such as development processing or encoding processing on the electrical signal obtained by the imaging processing in S1501.

[0150] In S1503, the state saving unit 811 generates photographer information 1521 about the photographer of the image data 1530. Specifically, the state saving unit 811 can acquire the authentication state table managed by the state management unit 810. The state saving unit 811 also acquires, from the registration management unit 803, personal information corresponding to the person ID indicated in the authentication state table. In this embodiment, the state saving unit 811 acquires "name" as the personal information. Note that if the person ID indicated in the authentication state table is "NULL," the state saving unit 811 can use "NULL" as the personal information (name). Then, the state saving unit 811 generates "name," "first authentication state," "second authentication state," and "whether or not another person has used the image data." The state saving unit 811 generates photographer information 1521 including this information. Through this processing, the state saving unit 811 can record the user's personal information in association with the captured image in response to a success in the second authentication process. Furthermore, in response to a failure in the second authentication process, the state saving unit 811 can control the saving operation so as not to record the user's personal information in association with the captured image. Furthermore, the state saving unit 811 can control to record information indicating that the detection unit 807 has detected that a different person has used the camera 100 in memory in association with the captured image.

[0151] In S1504, the state saving unit 811 generates a hash value 1522 by applying a hash function to each of the image data 1530 and the photographer information 1521, which are binary data.

[0152] In S1505, the state saving unit 811 generates a digital signature 1523. The digital signature 1523 may include information indicating the signature value, the signer, and the signing date and time. The state saving unit 811 can generate the signature value by encrypting the hash value 1522 generated in S1504 using a private key prepared in advance. A public key paired with the private key used here can also be stored in the digital signature 1523. In this embodiment, information indicating the manufacturer of the camera 100 is stored as the signer. Note that, instead of information indicating the manufacturer, information indicating the model of the camera 100 may be used as the signer. Furthermore, the date and time at which the generation of the digital signature is completed is stored as the signing date and time.

[0153] In S1506, the state saving unit 811 assigns the photographer information 1521, the hash value 1522, and the digital signature 1523 to the image data 1530 as metadata 1520. In this way, the state saving unit 811 generates the image file 1510. If the image data 1530 represents a still image, the state saving unit 811 can generate the image file 1510 in JPEG format. If the image data 1530 represents a moving image, the state saving unit 811 can generate the image file 1510 in MPEG format.

[0154] In S1507, the state saving unit 811 records the image file 1510 in the memory unit 213. In this embodiment, the memory unit 213 is built into the camera 100. Alternatively, the memory unit 213 may be a storage medium that is physically detachably connected to the camera 100. In this case, the image file can be recorded in the storage medium. Furthermore, the camera 100 may be connected to a server having a memory via a wired and / or wireless communication path. In this case, the state saving unit 811 can transmit the image file 1510 to the server to record it in memory. For example, the state saving unit 811 can transmit a signal to the server so that the captured image and user information are recorded in memory in association with each other.

[0155] The following verification method can be used to confirm that a file has not been tampered with. First, a hash value is restored from the signature value using the public key. Next, the hash values ​​of the image data and photographer information are recalculated. If the restored hash value matches the recalculated hash value, it can be determined that the file has not been tampered with. On the other hand, if they do not match, it can be determined that the file has been tampered with.

[0156] This is because even if someone tries to tamper with the image data, the signature value is encrypted with a private key, so the person who tampered with the data cannot change the signature value. Therefore, if the image data has been tampered with, the hash value calculated from the image data will not match the restored hash value. Therefore, the above method can detect data tampering.

[0157] In the above embodiment, the hash value is stored in the image file. However, in this verification method, the hash value stored in the file is not used. Therefore, it is not necessary to store the hash value in the image file.

[0158] When shooting a moving image, shooting of the moving image can be started when the release button 101 is pressed down to the second stroke, and can be ended when the release button 101 is pressed down to the second stroke again. After that, moving image data is generated by the processing of S1501 to S1502. Furthermore, a moving image file is generated by the processing of S1503 and subsequent steps. In this case, image data 1530 is moving image data, and image file 1510 is a moving image file.

[0159] [Gaze detection processing] FIG. 5 is a diagram for explaining the principle of the gaze detection method. FIG. 5 shows a schematic diagram of an optical system for performing gaze detection. As shown in FIG. 5, light sources 216a and 216b are arranged approximately symmetrically with respect to the optical axis of light-receiving lens 218. Light sources 216a and 216b illuminate user's eyeball 220. A portion of the light emitted from light sources 216a and 216b and reflected by eyeball 220 is collected on eye imaging element 219 via light-receiving lens 218. FIG. 6(A) is a schematic diagram of an eye image captured by eye imaging element 219 (eye optical image projected onto eye imaging element 219). FIG. 6(B) is a diagram showing the output intensity of eye imaging element 219.

[0160] The gaze detection process performed in this embodiment will be described with reference to the flowchart in Fig. 7. In S701, the CPU 212 controls the light sources 216a and 216b via the light source drive circuit 305 to emit infrared light toward the user's eyeball 220. An optical image of the user's eye illuminated by the infrared light passes through the light receiving lens 218 and is formed on the eye image sensor 219. The eye image sensor 219 then photoelectrically converts the formed optical image of the eye. In this way, an electrical signal of the eye image that can be processed is obtained.

[0161] In S702, the CPU 212 acquires an eye image (eye image signal; electric signal of the eye image) from the eye imaging element 219 via the line-of-sight detection circuit 301.

[0162] Through the processes of S703 and S704, the CPU 212 acquires eye information relating to the position of the eyeball 220 relative to the finder from the eye image obtained in S702. In S703, the CPU 212 detects the coordinates of points corresponding to the corneal reflection images Pd and Pe of the light sources 216a and 216b and the coordinates of a point corresponding to the pupil center c from the eye image obtained in S702.

[0163] Infrared light emitted from light sources 216a and 216b illuminates cornea 501 of user's eyeball 220. At this time, corneal reflection images Pd and Pe formed by part of the infrared light reflected from the surface of cornea 501 are collected by light receiving lens 218 and formed on eye imaging element 219. In this way, corneal reflection images Pd' and Pe' in the eye image are obtained. Similarly, light beams from pupil edges 510a and 510b, which are the edges of pupil 502, are also formed on eye imaging element 219. In this way, pupil edge images a' and b' in the eye image are obtained.

[0164] FIG. 6B shows luminance information (luminance distribution) in region α of the eye image shown in FIG. 6A. The horizontal direction of the eye image is the X-axis direction. The vertical direction of the eye image is the Y-axis direction. FIG. 6B shows the luminance distribution in the X-axis direction. The X-axis (horizontal) coordinates of corneal reflection images Pd' and Pe' are defined as Xd and Xe. The X-axis coordinates of pupil edge images a' and b' are defined as Xa and Xb. As shown in FIG. 6B, an extremely high level of luminance is obtained at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. In addition, in the region from coordinate Xa to coordinate Xb, which corresponds to the region of the pupil 502 (the region of the pupil image obtained when the light beam from the pupil 502 is focused on the eye imaging element 219), an extremely low level of luminance is obtained except for coordinates Xd and Xe. Then, a luminance intermediate between the above two levels is obtained in the region of iris 503 outside pupil 502 (the region of the iris image outside the pupil image obtained by focusing the light beam from iris 503). For example, a luminance intermediate between the above two levels is obtained in a region where the X coordinate (coordinate in the X-axis direction) is greater than coordinate Xa, and in a region where the X coordinate is less than coordinate Xb.

[0165] Based on the brightness distribution shown in FIG. 6B, the coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the coordinates Xa and Xb of the pupil edge images a' and b' can be obtained. For example, the coordinates of extremely high brightness can be obtained as the coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. Furthermore, the coordinates of the outer edge of the area where brightness is extremely low can be obtained as the coordinates Xa and Xb of the pupil edge images a' and b'. When the rotation angle θx of the optical axis of the eyeball 220 relative to the optical axis of the light receiving lens 218 is small, the coordinate Xc of the pupil center image c' (center of the pupil image) obtained when the light beam from the pupil center 510c is focused on the eye imaging element 219 can be expressed as Xc ≒ (Xa + Xb) / 2. In other words, the coordinate Xc of the pupil center image c' can be calculated from the coordinates Xa and Xb of the pupil edge images a' and b'. By using this method, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupil center image c' can be calculated.

[0166] In S704, the CPU 212 calculates the imaging magnification β of the eye image. The imaging magnification β is determined by the position of the eyeball 220 relative to the light receiving lens 218. The imaging magnification β can be calculated as a function of the distance ΔP=Xe−Xd between the corneal reflection images Pd′ and Pe′.

[0167] In S705, the CPU 212 calculates the rotation angle of the optical axis of the eyeball 220 relative to the optical axis of the light receiving lens 218. The X coordinate of the midpoint of the corneal reflection images Pd, Pe and the X coordinate of the center of curvature 511 of the cornea 501 approximately coincide with each other. Therefore, if the standard distance from the center of curvature 511 of the cornea 501 to the center c of the pupil 502 is Oc, the rotation angle θx of the eyeball 220 in the ZX plane (plane perpendicular to the Y axis) can be calculated using the following formula (1). The rotation angle θy of the eyeball 220 in the ZY plane (plane perpendicular to the X axis) can also be calculated in the same way as the rotation angle θx. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc...Equation (1)

[0168] In S706, the CPU 212 reads the line-of-sight correction parameters stored in the memory unit 213. Specifically, the CPU 212 can read the parameters Ax, Bx, Ay, and By used in the following equations (2) and (3).

[0169] In S707, the CPU 212 uses the calculated rotation angles θx and θy to estimate the user's viewpoint on the screen of the display device 214. The viewpoint coordinates (Hx, Hy) are set to coordinates corresponding to the pupil center c. In this case, the viewpoint coordinates (Hx, Hy) can be calculated using the following equations (2) and (3). Hx=m×(Ax×θx+Bx) Formula (2) Hy=m×(Ay×θy+By)...Equation (3) In equations (2) and (3), parameter m is a constant determined by the configuration of the optical system for detecting the line of sight. Parameter m is a conversion coefficient that converts rotation angles θx and θy into coordinates corresponding to the pupil center c on the screen of the display device 214. Parameter m is determined in advance and stored in the memory unit 213. Parameters Ax, Bx, Ay, and By are the line of sight correction parameters described above.

[0170] The gaze correction parameter will now be described. For example, due to factors such as individual differences in the shape of the human eyeball, the gaze point may not be estimated with high accuracy. In this case, as shown in FIG. 4B, a discrepancy may occur between the actual gaze point 412 and the estimated gaze point 413. In FIG. 4B, the user is gazing at a person, but the camera 100 estimates that the user is gazing at the background. In this case, appropriate focus detection or adjustment may not be possible. The gaze correction parameter is a parameter for correcting such a discrepancy. The gaze correction parameter can be obtained by calibration of the gaze detection. Calibration can be performed, for example, by highlighting multiple indices at different positions on the screen of the display device 214, as shown in FIG. 4C. The user is instructed to look at these indices. When the user gazes at each indices, a gaze detection operation is performed. Then, the gaze correction parameter suitable for the user can be calculated from the calculated positions of each gaze point (estimated positions) and the coordinates of each indices. Note that the position at which the user should gaze may be indicated using other indices display methods. For example, graphics may be displayed as the indices. Furthermore, the indicator may be displayed by changing at least one of the brightness and color of the image (for example, the captured image).

[0171] [Effects of this embodiment] Authentication at the time of capture can ensure that an image or video was taken by a specific person. However, to ensure that the image or video was not taken by someone else (that the person taking the photo) through such authentication, it is desirable to use settings that result in a low false acceptance rate. However, using such settings often increases the false rejection rate. In other words, even if the person taking the photo is the person taking the photo, there is a high possibility that authentication will fail at the time of capture. In the use case of photography, there is no chance to take the exact same photo twice. For example, the decisive moment in a sports or news scoop is fleeting. For professional photographers, not being able to authenticate at such a moment can be a problem. In other words, if authentication is not possible, there is no guarantee that the person who captured the decisive moment is the photographer himself.

[0172] Therefore, in this embodiment, a first authentication process is performed when no image is captured. In the first authentication process, authentication is performed using settings that result in a low false acceptance rate. In addition, a second authentication process is performed when an image is captured. In the second authentication process, authentication is performed using settings that result in a low false rejection rate. According to this embodiment, it is possible to reduce the possibility that a false person will be authenticated as the true person by combining the first authentication process and the second authentication process, while reducing the possibility that the true person will not be authenticated in the second authentication process.

[0173] Note that the first authentication process uses settings that result in a low false acceptance rate, so the false rejection rate may be high. As a result, there is a possibility that the person in question will not be authenticated in the first authentication process. However, because the first authentication process can be performed when no photograph is being taken, authentication can be attempted again. Therefore, problems with use are unlikely to occur.

[0174] Furthermore, the second authentication process may increase the possibility of authenticating a different person as the user. On the other hand, in this embodiment, the first authentication state is invalidated according to various conditions. Such processing can reduce the possibility of authenticating a different person as the user. That is, if use by a different person is suspected in the second authentication process, the first authentication state is invalidated. Furthermore, the first authentication state can be invalidated based on the elapsed time since the first authentication process was successful, a change in the power state, a change in the distance or connection state to a peripheral device, or an explicit invalidation operation by the user. This configuration can reduce the possibility of use by a different person. Therefore, it is possible to further suppress authentication of a different person as the user in the second authentication process.

[0175] In this embodiment, the results of the first authentication process and the second authentication process are recorded separately in the metadata of the image file. This process ensures that even if the first authentication process is successful and the second authentication process fails, the success of the first authentication process is recorded in the metadata. Additionally, if the second authentication process is also successful, the success of both authentication processes is recorded so that it is clear. Recording many successful authentications in this way indicates the high degree of certainty that the photographer recorded took the photograph.

[0176] Furthermore, in this embodiment, information indicating suspicion of use of the device (e.g., camera 100) by another person is recorded as metadata. With this configuration, if the first authentication process is successful but the second authentication process fails, it is possible to determine whether or not there is suspicion of use by another person. By recording such information, it is possible to indicate the degree of certainty that the recorded photographer took the photograph.

[0177] [Variations] In the above embodiment, the registration management unit 803 managed only "name" as personal information. However, information other than name may be used as personal information. For example, if the camera is used within a company, an "employee number" assigned to an employee may be registered as personal information. Also, account information such as an account name for any web service may be registered as personal information. In this case, if a user accesses and logs in to a web service, the account name may be registered as personal information. Also, the web service may issue a token or the like upon successful access. In this case, the issued token can be registered as personal information. Also, the state saving unit 811 may save this personal information as image metadata. The personal information used by the registration management unit 803 and the state saving unit 811 is not limited to these.

[0178] In the above embodiment, the first authentication unit 805 and the second authentication unit 806 use the same feature calculation unit 802 to perform authentication. Furthermore, the feature calculation unit 802 calculates features using the same process for the first authentication process and the second authentication process. However, the eye images used by the first authentication unit 805 for authentication and the eye images used by the second authentication unit 806 for authentication tend to differ from each other. For example, the eye images used by the first authentication unit 805 for authentication are captured while the user is aware of being authenticated. Therefore, eye images in which the photographer's eyes are wide open are often used. On the other hand, the eye images used by the second authentication unit 806 for authentication are captured while the user is taking a photo. Therefore, a wide variety of eye images, for example, eye images at various gaze angles, are likely to be used. Therefore, the neural network that calculates the features used by the first authentication unit 805 can be trained using eye images expected to be used in the first authentication process. On the other hand, the neural network that calculates the features used by the second authentication unit 806 can be trained using eye images expected to be used in the second authentication process. Such a configuration can further improve authentication accuracy.

[0179] Note that methods other than changing the threshold or model may be used to lower the false acceptance rate in the first authentication process and the false rejection rate in the second authentication process. For example, as described in Patent Document (JP 2022-182960 A), a method of improving recognition performance by changing the calculation method of the feature vector used during enrollment and verification may be used. In this way, the authentication methods used in the first authentication process and the second authentication process are not limited to the specific examples described above.

[0180] In the above embodiment, in the second authentication process, the status display unit 812 causes the display device 214 to display the authentication result (S1210, S1217, and S1218). However, during shooting, the display device 214 displays an image captured by the image sensor 211. Therefore, the status display unit 812 may cause the display device 214 to display the authentication result as follows. For example, the display device 214 may display an indication of authentication success or failure at the edge of the screen. This display method reduces the likelihood that the display of the authentication result will interfere with shooting, allowing the user to concentrate more easily on shooting. Alternatively, the display position of the authentication result may be determined based on the user's gaze position. For example, the display device 214 may display the authentication result at a position far from the gaze position. The user's gaze position on the display device 214 can be obtained by the gaze detection process (S707) already described. As another example, multiple display positions, such as near the four corners of the display device 214, may be determined in advance. Then, the display device 214 can determine, from among the plurality of display positions, the position at which to display the authentication result based on the gaze position. For example, the display device 214 can display an icon or the like at the display position farthest from the gaze position among the plurality of display positions. With such a configuration, the display position is fixed to a certain extent, making it easier for the user to understand the authentication result. However, the method of displaying the authentication result is not limited to the above example. Also, it is not necessary to display the authentication result. In this case, S1204 can be omitted.

[0181] In the above embodiment, only the second authentication status is displayed in S1210 and S1217. However, the first authentication status may be displayed at the same time. Similarly, both the first authentication status and the second authentication status may be displayed in S1218. Furthermore, in the above embodiment, these authentication statuses were not displayed when the second authentication process was started. However, when the second authentication process was started, for example, when the user looked into the viewfinder, the authentication status may already be displayed on the display device 214. The display of these authentication statuses can be performed by controlling the status display unit 812.

[0182] In the above embodiment, both the first authentication unit 805 and the second authentication unit 806 performed personal authentication using eye images. However, other authentication methods may be used. For example, other biometric authentication methods such as face authentication, fingerprint authentication, or voice authentication may be used in the first authentication process. When face authentication is performed, a face-capturing camera may be installed on the back of the camera 100 to capture a face image of the user. For example, a face authentication camera may be installed above the touch panel (operation member 103) to perform authentication when the user looks into the touch panel. Such a camera can be used to acquire a face image of the user. Then, authentication can be performed using such a face image. When fingerprint authentication is performed, a fingerprint sensor may be installed on the release button 101. Then, when the user places his / her finger on the release button, fingerprint authentication can be performed based on information acquired by the fingerprint sensor. When voice authentication is performed, a microphone may be installed on the camera 100. Then, voice authentication can be performed based on a voiceprint extracted from the acquired voice.

[0183] Furthermore, personal authentication may be performed using a method other than biometric authentication. For example, personal authentication may be performed based on a password or PIN code entered by the user. Authentication may also be performed based on the connection status with other devices. For example, the camera 100 and a smartphone may be paired via a Bluetooth connection. When the camera 100 is connected to the smartphone via Bluetooth, the first authentication status may be considered to be valid. Authentication may also be performed using a mechanism such as FIDO authentication.

[0184] Also, other authentication methods may be used for the second authentication process. Meanwhile, in the above-described embodiment, the second authentication process is performed when capturing an image. For this reason, it is convenient to use an authentication method that can be used when capturing an image for the second authentication process. For example, by using a fingerprint sensor installed on the release button 101, fingerprint authentication can be performed as the second authentication process when capturing an image. Also, the user may perform shooting while viewing an image captured by the image sensor 211 displayed on the touch panel (operation member 103) without looking through the viewfinder. In this case, by using an image of the user's face, facial authentication can be performed as the second authentication process when capturing an image. Furthermore, multiple authentication methods may be used in combination. For example, in the first authentication, multiple authentication methods are used, and if authentication is successful using all of the authentication methods, it is determined that the first authentication has been successful. On the other hand, in the second authentication, if authentication is successful using any one of the authentication methods, it is determined that the second authentication has been successful. This makes it possible to eliminate the possibility of accepting a false person in the first authentication. In addition, the second authentication can reduce the possibility of rejecting the true person.

[0185] In the above embodiment, the second authentication state is controlled to be valid only during image capture. When facial authentication is used, image capture can be considered to be in progress while a face is being captured. When fingerprint authentication is used, image capture can be considered to be in progress while a finger is placed on the release button. However, the authentication method and the method of determining whether image capture is in progress are not limited to the above examples.

[0186] In the above embodiment, the invalidation unit 808 invalidated the first authentication status in accordance with a predetermined condition. On the other hand, after the first authentication status is invalidated, another person may repeat the first authentication process. In this case, there is a possibility that the first authentication process will be successful and the first authentication status will be invalidated repeatedly.

[0187] Therefore, the camera 100 may include a first authentication restriction unit (not shown). The first authentication restriction unit detects suspicion of unauthorized use by another person. Then, the first authentication restriction unit can temporarily restrict execution of the first authentication process in response to detecting the suspicion of unauthorized use. The first authentication restriction unit can detect the suspicion of unauthorized use in response to a predetermined condition. For example, the first authentication restriction unit can restrict the first authentication process by the first authentication unit 805 based on the number of times the first authentication unit 805 has set the first authentication state to invalid. That is, the predetermined condition may be that the first authentication process by the first authentication unit 805 is successful and the first authentication state is invalidated by the invalidation unit 808, both of which occur repeatedly within a predetermined period of time. Alternatively, some of the above-described first authentication invalidation determination conditions may be used as the predetermined condition. For example, the predetermined condition may be limited to the detection unit 807 detecting suspicion of unauthorized use of the camera 100 by another person. Alternatively, the predetermined condition may be that the invalidation of the first authentication state due to suspicion of unauthorized use by another person and the success of the first authentication process are repeated within a predetermined period of time. However, the method for detecting the suspicion of unauthorized use is not limited to the above-described method.

[0188] One method for restricting the execution of the first authentication process is to disable the user's operation to start the first authentication process. For example, if the user operates camera 100 and calls the first authentication process from a menu or the like, the item for calling the first authentication process can be disabled. Furthermore, after a certain period of time has passed, this state of restriction on the execution of the first authentication process can be lifted. However, the methods for restricting the execution of the first authentication process and lifting the restriction are not limited to the above methods.

[0189] This technique makes it difficult for a registered person to intentionally hand over the camera to another person and have that person take a photo. In particular, if an authentication process other than biometric authentication is used as the first authentication process, the first authentication process may be successful even when the registered person is not present. For example, the first authentication process can be successful by sharing the password or PIN used in the first authentication process or by handing over a smartphone paired with the camera 100 to another person. On the other hand, implementing the above-mentioned restrictions can prevent fraudulent use.

[0190] In the above embodiment, the second authentication process was performed when a predetermined operation, such as a photographing operation, was performed. Alternatively, information necessary for the second authentication process may be saved when a predetermined operation is performed, and the second authentication process may be performed thereafter. For example, an image of the user's eyes may be saved when photographing, and the second authentication process may be performed using the saved eye image after photographing. Even when using other types of biometric authentication (such as face authentication or fingerprint authentication), biometric information (such as face or fingerprint) may be saved and used for the second authentication process after photographing. Furthermore, even when using the connection status with other devices, parameters of the connection status may be recorded when photographing, and the process of analyzing the connection status and performing authentication may be performed after photographing. In this way, the second authentication process does not necessarily have to be performed when photographing. Information necessary for the second authentication process may be acquired when photographing, and authentication may be performed after photographing, etc. This type of processing may be adopted when it is difficult to perform the second authentication process when photographing due to resource constraints, such as speed.

[0191] If the second authentication process is performed later, the authentication process may not be completed in time for saving the image file. In this case, a state other than "authenticated" and "unauthenticated," such as "processing," can be defined as the second authentication status. If the second authentication process is not yet complete, information indicating "processing" can be saved as metadata. Then, when the result of the second authentication process is obtained, the metadata can be modified to indicate the result of the second authentication process.

[0192] In one embodiment, biometric authentication is used for the second authentication process. It is difficult for the user to enter a password or the like when taking a photo. Furthermore, if the user hands over the smartphone along with the camera to another person, the other person can pass authentication based on the connection status with other devices. On the other hand, biometric authentication can be performed without any special action for authentication. Furthermore, the biometric information used for biometric authentication can only be held by the person himself / herself. For this reason, in the above embodiment, biometric authentication is used as the second authentication process performed when taking a photo.

[0193] In the above embodiment, the registration management unit 803 manages the first authentication registration feature vector table and the second authentication registration feature vector table as separate tables. However, these tables may be integrated into a single table. This configuration can reduce duplication of information such as "feature vector 1" and improve memory efficiency. In this case, information indicating whether a record related to each registration feature vector is to be used in the first authentication process or the second authentication process can be recorded in the table. Based on this information, the registration feature vector to be used in the first authentication process and the second authentication process can be selected.

[0194] In the above embodiment, different registered feature vector groups are used in the first authentication process and the second authentication process. However, feature vector groups may be used in these processes. In this case, the registration management unit 803 does not need to manage a first authentication registered feature vector table and a second authentication registered feature vector table. The registration management unit 803 can manage feature vectors in a single table. However, the data management method in the registration management unit 803 is not limited to the above example.

[0195] In the above embodiment, one person is registered as the user of the camera 100. Therefore, when a different person is to be registered, the data held by the registration management unit 803 is erased and the registration is restarted. For example, when the user registration process is started, the data managed by the registration management unit 803 is deleted. Alternatively, when the user registration process is started, an invalid flag may be set for the data managed by the registration management unit 803. Data with an invalid flag attached will not be used in the subsequent first and second authentication processes.

[0196] In the above embodiment, the device (e.g., camera 100) has a first authentication unit 805 that authenticates the user of the device. The device also has a second authentication unit 806 that authenticates the user of the device when performing a predetermined operation using the device. The device also has an execution unit 809 that controls the operation of the device depending on whether the same person is authenticated in both the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806. For example, if the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806 fails to authenticate the same person, the execution unit 809 can control the operation of the device to record information indicating the authentication failure. The execution unit 809 can also control the operation of the device to perform an operation to store information indicating that the same person is authenticated in both the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806. Furthermore, the execution unit 809 can notify, via the status display unit 812 or the like, that the same person has been authenticated both by the first authentication unit 805 and by the second authentication unit 806 .

[0197] In the above embodiment, when the first authentication status is valid, the second authentication unit 806 performs the second authentication process. Furthermore, the second authentication unit 806 can determine that the device user is the same person as the user determined by the first authentication unit 805 to be registered. This means that the same person was authenticated in both the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806. In response to this determination, the execution unit 809 stores the image data in the memory unit 213 together with metadata indicating that the second authentication status was valid. In response to this determination, the execution unit 809 also stores the image data in the memory unit 213 together with metadata including personal information of the authenticated user. On the other hand, a failure in the second authentication process means that the same person could not be authenticated in the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806. In such a case, the execution unit 809 can record information indicating that the authentication process failed. For example, in the above embodiment, the execution unit 809 stored the image data in the memory unit 213 together with metadata indicating that the second authentication status was invalid. In such a case, the execution unit 809 can store the image data in the memory unit 213 together with metadata including "NULL" as the identity information. In this way, the execution unit 809 can store the image data together with different metadata depending on whether the same person is authenticated in both the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806. However, the method for determining whether the user of the device is the same person as the user determined to be registered by the first authentication unit 805 is not limited to the above example.

[0198] For example, multiple users may be registered for the camera 100. In this case, each time the user registration process is performed, a different person ID and registration information corresponding to the person ID (e.g., personal information and feature vector) are registered in the registration management unit 803. At this time, additional processing may be performed to prevent duplicate registration by the same person. For example, if a name identical to an already registered name is entered during the user registration process, a notification may be sent that the user has already been registered, and the process may be terminated.

[0199] In this case, the first authentication process can determine which user corresponding to which personal ID is using the camera 100. The first authentication unit 805 may identify the user based on a comparison of feature vectors. For example, in S1107, the first authentication unit 805 may acquire all feature vectors for all personal IDs registered in the first authentication registered feature vector table. In addition, in S1108, the first authentication unit 805 may compare each of the multiple registered feature vectors with the feature vector obtained in S1106. Here, the similarity calculated for the multiple registered feature vectors may exceed a predetermined threshold. In this case, the first authentication unit 805 can identify the personal ID for the registered feature vector that is most similar to the feature vector obtained in S1106. The first authentication unit 805 can determine that the user corresponding to the identified personal ID is using the camera 100. Note that if the similarity calculated for the multiple registered feature vectors exceeds a predetermined threshold, the first authentication unit 805 may determine that the first authentication process has failed.

[0200] In this case, the registered feature vectors used in the second authentication process may be limited to those corresponding to the user identified in the first authentication process. That is, in S1206, the second authentication unit 806 can obtain only the record for the person ID identified in S1108 from the second authentication registered feature vector table. In S1207, the second authentication unit 806 can only compare the registered feature vector included in the obtained record with the feature vector obtained in S1205. If a user is identified in a single-stage authentication process, each of multiple registered feature vectors is compared with the feature vector obtained for the user. On the other hand, the above configuration can reduce the number of registered feature vectors compared in the second authentication process. This can improve authentication accuracy. This is because the problem can be simplified from so-called 1:N matching (identification) to 1:1 matching (verification). Furthermore, reducing the number of registered feature vectors to be compared can shorten processing time.

[0201] In the above embodiment, a pre-registered registered feature vector is used in the second authentication process. Alternatively, biometric information obtained when the first authentication process is successful may be used in the second authentication process. For example, feature vectors of eye images obtained when the first authentication process is successful, or before and after, can be recorded. In the second authentication process, the recorded feature vector can be used instead of the registered feature vector. That is, in the second authentication process, authentication may be performed by confirming the identity of the user's feature obtained when the first authentication process is successful and the user's feature obtained during the second authentication process (verification process). This method can omit the registration of feature vectors for the second authentication process. This method is particularly effective when the second authentication process is performed in an environment different from that when the feature vectors were registered. This identity confirmation process may also be combined with the comparison process with registered feature vectors, as described above. In this case, the second authentication process is successful if matching by either method (or if matching by both methods) is successful. This method can reduce the false rejection rate or the false acceptance rate.

[0202] As another application of this authentication method, the following method can be adopted. In the first authentication process, authentication by PIN input and fingerprint authentication via the release button are performed. If the first authentication process is successful, a facial image of the user is simultaneously captured by the in-camera. Furthermore, the facial image is converted into feature quantities, and the obtained feature quantities are recorded. In the second authentication process, the identity of the feature quantities of the recorded facial image and the facial features of a re-acquired facial image of the user is confirmed. In this way, various forms can be adopted as a two-step authentication method.

[0203] On the other hand, it is not essential for the second authentication unit 806 to determine whether the device user is the same person as the user determined to be registered by the first authentication unit 805. For example, the second authentication unit 806 may authenticate the device user independently of the first authentication unit 805.

[0204] Furthermore, the second authentication unit 806 does not necessarily perform the second authentication process when the first authentication status is valid. For example, there may be cases where the first authentication process cannot be performed, such as when a photograph is suddenly required. Also, the user may forget to perform the first authentication process. Therefore, the second authentication process may be performed while the first authentication status is invalid. Specifically, in S1201, the process can be performed so that the process proceeds to S1202 not only when the first authentication status is valid, but also when the first authentication status is invalid.

[0205] Furthermore, when the first authentication status is invalid, a value indicating an empty state, such as NULL, is recorded in the person ID in the authentication status table shown in Fig. 8(E). Therefore, when the second authentication process is successful while the first authentication status is invalid, in S1209, the person ID in the authentication status table can be updated so that the person ID identified in S1207 is recorded.

[0206] In this case, in the authentication status saving process, "unauthenticated" is recorded as the "first authentication status" and "authenticating (authentication successful state)" is recorded as the "second authentication status" in the metadata. In this case, the person ID may be recorded in the metadata so that it is clear that the person was identified in the second authentication process. For example, the person ID identified in the first authentication process and the person ID identified in the second authentication process can be described as different items in the metadata.

[0207] When multiple people are registered in the camera 100, the second authentication unit 806 can acquire in S1206 only the registered feature vector for the person ID identified in the first authentication process, as described above. On the other hand, when the first authentication status is invalid, the second authentication unit 806 can acquire in S1206 the registered feature vector for all person IDs recorded in the second authentication registered feature vector table. The second authentication unit 806 can also perform matching in S1207 for each registered feature vector. In this case, the second authentication unit 806 can identify the person ID for the registered feature vector that is most similar to the feature vector obtained in S1205. The second authentication unit 806 can then determine that the user corresponding to the identified person ID is using the camera 100. If the similarity calculated for the multiple registered feature vectors exceeds a predetermined threshold, the second authentication unit 806 may determine that the second authentication process has failed.

[0208] In S1207, the second authentication unit 806 may use different similarity thresholds depending on whether the first authentication state is valid or invalid. The second authentication unit 806 may also use different authentication settings depending on whether the first authentication state is valid or invalid. For example, the second authentication unit 806 can change the model to be used. While 1:1 authentication and 1:N authentication present different problems, such a configuration allows the use of settings appropriate for each authentication.

[0209] In the above embodiment, the second authentication process is performed according to the person ID identified in the first authentication process. That is, in the second authentication process, it is determined whether the user is the same person as the user determined to be registered in the first authentication process. However, the person ID may be identified independently in the second authentication process without using the result of the first authentication process.

[0210] In this case, the "person ID of the first authentication process" and the "person ID of the second authentication process" can be recorded separately in the authentication state table of FIG. 8(E) managed by the state management unit 810. In this case, the person ID identified in the first authentication process and the person ID identified in the second authentication process can be recorded in their respective items. Furthermore, in S1206, the second authentication unit 806 can acquire registered feature vectors for all person IDs recorded in the second authentication registered feature vector table. Furthermore, as described above, the second authentication unit 806 can identify the person ID of the person using the camera 100 by comparing each registered feature vector in S1207.

[0211] In this way, the first authentication process and the second authentication process may be performed independently. In this case, the execution unit 809 can control the operation of the device depending on whether the same person is authenticated in both the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806. For example, the execution unit 809 can store image data together with different metadata depending on whether the same person is authenticated in both the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806.

[0212] Furthermore, the state saving unit 811 can save authentication states such as the "person ID of the first authentication process," "person ID of the second authentication process," "first authentication state," and "second authentication state" separately in metadata. With this configuration, when using an image file, it is possible to confirm that the same person was authenticated in the first authentication process and the second authentication process, and to confirm that this person took the photo.

[0213] Furthermore, if the "person ID of the first authentication process" and the "person ID of the second authentication process" are different, the state saving unit 811 may record "unauthenticated" as the "second authentication state." In this case, the state saving unit 811 may save personal information related to the "person ID of the first authentication process" as personal information (for example, name) in the metadata. In this way, the process of checking whether the same person has been authenticated in the first authentication process and the second authentication process may be performed when saving the metadata.

[0214] In the authentication state saving process in the above embodiment, the state saving unit 811 saves all of the contents of the authentication state table managed by the state management unit 810 as metadata. However, the state saving unit 811 does not have to save all of this content. Furthermore, the state saving unit 811 may process the contents of the authentication state table before saving. For example, the state saving unit 811 may save only the "name." In this case, the state saving unit 811 may save the "name" corresponding to the "person ID" only when both the "first authentication state" and the "second authentication state" are "authenticating (authentication successful state)." In other cases, the state saving unit 811 may save another value (for example, a value indicating that the "name" is unknown) as the "name." In this case, the fact that the "name" is saved indicates that the authentication was successful (i.e., both the "first authentication state" and the "second authentication state" are "authenticating").

[0215] As another example, the first authentication status and the second authentication status do not need to be stored separately. For example, the state storage unit 811 may record one "authentication status" item. For example, only when both the "first authentication status" and the "second authentication status" are "authenticated," the state storage unit 811 may record "authenticated" in the "authentication status" item. On the other hand, the state storage unit 811 may record "unauthenticated" in the "authentication status" item in other cases. The information stored in the authentication status storage process is not limited to the above example.

[0216] The timing of the second authentication process is not particularly limited. In addition to when taking a photograph, the second authentication process or the acquisition of biometric information for the second authentication process can be performed when performing a predetermined operation using the device, such as viewing an image. The timing of the first authentication process is also not particularly limited. The first authentication process can be performed before a predetermined operation, such as when the power is turned on or during a login process.

[0217] Furthermore, the method of controlling the device operation by the execution unit 809 is not limited to controlling the writing of information to the memory unit 213 as described above. For example, the execution unit 809 may execute a specific process by the device depending on the result of the determination by the second authentication unit 806. Furthermore, the execution unit 809 may execute a specific process by the device depending on whether the same person is authenticated in both the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806. For example, the execution unit 809 may control whether or not the device is operable. Specifically, the execution unit 809 may permit the camera 100 to capture an image when the second authentication status is valid. Furthermore, the execution unit 809 may prohibit the camera 100 from capturing an image when the second authentication status is invalid. As another example, the execution unit 809 may execute a login process depending on whether the same person is authenticated in both the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806.

[0218] In the above embodiment, the first authentication process and the second authentication process are performed in the same device. However, the first authentication process and the second authentication process may be performed in different devices. For example, the functions of the information processing device according to the above embodiment may be realized by an information processing system. Such an information processing system may be configured, for example, by multiple information processing devices connected via a network. Such an information processing device may be realized by a computer including a processor and a memory. That is, each of the information processing devices can realize at least a part of the functions shown in FIG. 8(A) by the processor executing a program stored in the memory.

[0219] An information processing system according to one embodiment includes a head-mounted display (HMD) and a rental device that rents out the HMD. The rental device stores multiple HMDs. If a first authentication process in the rental device is successful, the HMD is rented to a user. Then, when the user puts on the HMD, a second authentication process is performed. If the second authentication process is successful, the user can log in to the HMD and use the HMD.

[0220] The authentication information for the first authentication process can be obtained via a web service or the like. Specifically, the user requests rental of an HMD from an HMD rental service online. The user also pays the usage fee online. The user also registers a facial image online. The rental device then performs a first authentication process using facial recognition. If the first authentication process identifies the user as the person who previously requested rental of the HMD, the rental device unlocks the HMD. The user can then take the HMD out of the rental device. At the same time, authentication information for the second authentication process is registered. For example, the user looks into a camera attached to the rental device that captures eye images. In this way, authentication information for the second authentication process based on the captured eye image is registered in the HMD. After that, when the user puts on the HMD, the eye image camera attached to the HMD captures the user's eye image. The HMD then compares the feature values ​​of the acquired eye image of the user with the feature values ​​of pre-registered eye images for the second authentication process. If the second authentication process is successful, the user is permitted to log in to the HMD. In this embodiment, the operation of the HMD is also controlled in response to the fact that the same person is authenticated in both the first authentication process and the second authentication process.

[0221] In such an embodiment, an information processing system having a rental device and an HMD has the components shown in FIG. 8(A). This information processing system may further include an information processing device such as a server. In this case, the functions shown in FIG. 8(A) may be distributed among the rental device, the HMD, and the information processing device. For example, the rental device and the HMD may be communicably connected to the information processing device. In this example, the information processing device may provide the above-mentioned web service. This information processing device may also manage authentication status.

[0222] In this manner, the first authentication process and the second authentication process may be implemented in different devices. Furthermore, the device used in this embodiment is not limited to a camera. Furthermore, the process executed in accordance with the first authentication status and the second authentication status is not limited to recording the authentication status along with the captured image. As in this example, a process such as logging in to a device may be performed in accordance with the first authentication status and the second authentication status.

[0223] In the above embodiment, an authentication method with a low false acceptance rate is used in the first authentication process, and an authentication method with a low false rejection rate is used in the second authentication process. However, this is not limited to this configuration. For example, an authentication method with a low false rejection rate may be used in the first authentication process, and an authentication method with a low false acceptance rate may be used in the second authentication process. Specifically, in the above example, the rental device can perform a simple and fast first authentication process using a facial image. On the other hand, the second authentication process in the HMD can perform more strict iris authentication by having the user gaze at a specific location for a certain period of time. Furthermore, the first authentication process and the second authentication process can be performed in the HMD. In this case, a fast first authentication process using an image of the eyes or face can be performed for logging in. Furthermore, the subsequent second authentication process can perform more strict iris authentication.

[0224] According to the above embodiment, user authentication is performed by combining the first authentication process and the second authentication process. Therefore, the settings of the second authentication process can be adjusted to improve the usability of the device. On the other hand, by using the first authentication process and the second authentication process in combination, an increase in the false acceptance rate due to adjustment of the settings of the authentication processes can be suppressed.

[0225] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0226] The disclosure of this specification includes the following imaging device, operation method, and program. (Item 1) An imaging device, an acquisition means for acquiring biometric information of a person; a first authentication means for determining whether a user of the photographing device is a person who has been registered in advance; a second authentication means for determining whether the user of the photographing device is the same person as the user determined to be registered by the first authentication means through a biometric authentication process using the biometric information acquired via the acquisition means; a control means for controlling the image captured by the photographing device to be recorded in a memory in association with information about the user in accordance with a result of the determination by the second authentication means; An imaging device comprising: (Item 2) 2. The photographing device according to item 1, wherein the second authentication means performs the biometric authentication process when the photographing device performs a photographing operation. (Item 3) 3. The photographing device according to any one of items 1 to 2, wherein the second authentication means performs the biometric authentication process using biometric information acquired via the acquisition means when the photographing device performs a photographing operation. (Item 4) 4. The photographing device according to any one of items 1 to 3, wherein the second authentication means performs determination using a method with a lower false rejection rate than the first authentication means. (Item 5) 5. The photographing device according to any one of items 1 to 4, further comprising a registration management means for managing, in association with each other, registration information used by the first authentication means for determining the user of the photographing device and registration information used by the second authentication means for determining the user of the photographing device. (Item 6) The photographing device described in any one of items 1 to 5, characterized in that, when the second authentication means succeeds in the biometric authentication process, the control means controls the photographed image to be associated with information about the user and recorded in the memory. (Item 7) a status management unit that manages a first authentication status of the photographing device; The photographing device described in any one of items 1 to 6, characterized in that the status management means sets the first authentication status to valid in response to the first authentication means determining that the user is registered. (Item 8) 8. The photographing device according to item 7, wherein the second authentication means performs the biometric authentication process when it is determined that the first authentication state is set to valid. (Item 9) 9. The image capturing device according to any one of items 7 to 8, further comprising a notification unit that notifies the first authentication status on the image capturing device. (Item 10) the status management means manages a second authentication status of the photographing device; The photographing device described in any one of items 7 to 9, characterized in that the state management means sets the second authentication state to valid in response to the second authentication means succeeding in the biometric authentication process. (Item 11) Item 11. The photographing device according to item 10, wherein the state management means sets the second authentication state to invalid upon completion of photographing using the photographing device. (Item 12) 12. The photographing device described in any one of items 10 to 11, characterized in that the control means controls to record information indicating each of the first authentication status and the second authentication status in the memory in association with the photographed image. (Item 13) 13. The photographing device according to any one of items 7 to 12, further comprising a disabling unit that sets the first authentication status to invalid in accordance with a predetermined condition. (Item 14) Item 14. The photographing device according to item 13, wherein the invalidation means sets the first authentication status to invalid based on the elapsed time since the first authentication status was set to valid. (Item 15) 15. The photographing device according to any one of items 13 to 14, wherein the invalidation means sets the first authentication state to invalid based on a change in the power state of the photographing device. (Item 16) The photographing device described in any one of items 13 to 15, characterized in that the invalidation means sets the first authentication status to invalid based on the distance or connection status between the photographing device and another device. (Item 17) 17. The photographing device according to any one of items 13 to 16, wherein the invalidation means sets the first authentication status to invalid based on an input by a user to the photographing device. (Item 18) The photographing device described in any one of items 13 to 17, characterized in that the invalidation means sets the first authentication status to invalid based on detection of suspicion that the photographing device is being used by a person other than the user determined to be registered. (Item 19) 19. The photographing device according to any one of items 13 to 18, further comprising a limiting means for limiting the determination by the first authentication means based on the number of times the disabling means has set the first authentication status to invalid. (Item 20) 20. The photographing device according to any one of items 1 to 19, further comprising a detection means for detecting suspicion of use of the photographing device by a person other than the user determined to be registered by the first authentication means. (Item 21) 21. The photographing device according to item 20, further comprising a notification unit that notifies that the detection unit has detected suspicion of use of the photographing device by the different person. (Item 22) 22. The photographing device according to any one of items 20 to 21, characterized in that the control means controls the recording in the memory of information indicating that the detection means has detected suspicion of use of the photographing device by the different person in association with the photographed image. (Item 23) 23. The photographing device according to any one of items 20 to 22, wherein the detection means detects suspicion of use of the photographing device by a different person based on the number of times the biometric authentication process fails. (Item 24) the second authentication means performs the biometric authentication process by comparing the biometric information related to the user determined to be registered with the biometric information acquired via the acquisition means; 24. The photographing device according to any one of items 20 to 23, wherein the detection means detects suspicion of use of the photographing device by a different person based on the similarity of the biometric information compared by the second authentication means. (Item 25) 25. The photographing device according to any one of items 1 to 24, wherein the memory is built into the photographing device or is physically detachably connected to the photographing device. (Item 26) the imaging device is connected to the server having the memory via a wired and / or wireless communication path; 26. The photographing device described in any one of items 1 to 25, characterized in that the control means transmits a signal to the server so that the photographed image and the user information are associated and recorded in the memory. (Item 27) A method of operating an imaging device, comprising: acquiring biometric information of a person; a step of performing a first authentication process to determine whether the user of the photographing device is a person who has been registered in advance; determining whether the user of the photographing device is the same person as the user determined to be registered in the first authentication process by a biometric authentication process using the acquired biometric information; a step of controlling the recording of the captured image by the image capturing device in association with information about the user in a memory in accordance with a result of the determination by the biometric authentication processing; 10. A method of operation comprising: (Item 28) 27. A program for causing a computer to function as the imaging device according to any one of items 1 to 26.

[0227] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0228] 100: camera, 801: image acquisition unit, 802: feature calculation unit, 803: registration management unit, 804: registration unit, 805: first authentication unit, 806: second authentication unit, 807: detection unit, 808: invalidation unit, 809: execution unit, 810: status management unit, 811: status storage unit, 812: status display unit, 813: imaging unit

Claims

1. An imaging device, an acquisition means for acquiring biometric information of a person; a first authentication means for determining whether a user of the photographing device is a person who has been registered in advance; a second authentication means for determining whether the user of the photographing device is the same person as the user determined to be registered by the first authentication means through a biometric authentication process using the biometric information acquired via the acquisition means; a control means for controlling the image captured by the photographing device to be recorded in a memory in association with information about the user in accordance with a result of the determination by the second authentication means; An imaging device comprising:

2. The photographing device according to claim 1 , wherein the second authentication means performs the biometric authentication process when the photographing device performs a photographing operation.

3. The photographing device according to claim 1 , wherein the second authentication means performs the biometric authentication process using biometric information acquired via the acquisition means when the photographing device performs a photographing operation.

4. 2. The photographing device according to claim 1, wherein the second authentication means performs determination using a method with a lower false rejection rate than the first authentication means.

5. 2. The photographing device according to claim 1, further comprising a registration management means for managing, in association with each other, registration information used by the first authentication means for determining a user of the photographing device and registration information used by the second authentication means for determining a user of the photographing device.

6. 2. The photographing device according to claim 1, wherein the control means controls the photographing image to be recorded in the memory in association with information about the user when the second authentication means has succeeded in the biometric authentication process.

7. a status management unit that manages a first authentication status of the photographing device; 2. The photographing device according to claim 1, wherein the state management means sets the first authentication state to valid in response to the first authentication means determining that the user has been registered.

8. The photographing device according to claim 7 , wherein the second authentication means performs the biometric authentication process when it determines that the first authentication state is set to valid.

9. The photographing device according to claim 7 , further comprising a notification unit that notifies the first authentication status on the photographing device.

10. the status management means manages a second authentication status of the photographing device; 8. The photographing device according to claim 7, wherein the state management means sets the second authentication state to valid in response to the second authentication means succeeding in the biometric authentication process.

11. The photographing device according to claim 10 , wherein the state management unit sets the second authentication state to invalid when photographing using the photographing device ends.

12. The photographing device according to claim 10 , wherein the control means performs control to record information indicating each of the first authentication status and the second authentication status in the memory in association with the photographed image.

13. The photographing device according to claim 7 , further comprising: a nullifying unit that sets the first authentication status to invalid in accordance with a predetermined condition.

14. The photographing device according to claim 13 , wherein the invalidation unit sets the first authentication status to invalid based on the amount of time that has elapsed since the first authentication status was set to valid.

15. The photographing device according to claim 13, wherein the invalidation means sets the first authentication state to invalid based on a change in a power supply state of the photographing device.

16. The photographing device according to claim 13 , wherein the invalidation unit sets the first authentication status to invalid based on a distance or a connection status between the photographing device and another device.

17. The photographing device according to claim 13, wherein the invalidation means sets the first authentication status to invalid based on an input by a user to the photographing device.

18. 14. The photographing device according to claim 13, wherein the invalidation means sets the first authentication status to invalid when a suspicion of use of the photographing device by a person other than the user determined to be registered is detected.

19. 14. The photographing device according to claim 13, further comprising a limiting unit that limits the determination by the first authenticating unit based on the number of times the invalidating unit has set the first authentication state to invalid.

20. 2. The photographing device according to claim 1, further comprising a detection unit for detecting suspicion of use of the photographing device by a person other than the user determined to be registered by the first authentication unit.

21. 21. The photographing device according to claim 20, further comprising a notification unit that notifies the user that the detection unit has detected a suspicion that the photographing device is being used by a different person.

22. 21. The photographing device according to claim 20, wherein the control means controls to record information indicating that the detection means has detected suspicion of use of the photographing device by the different person in the memory in association with the photographed image.

23. 21. The image capturing device according to claim 20, wherein the detection means detects suspicion that the image capturing device is being used by a different person based on the number of times the biometric authentication process has failed.

24. the second authentication means performs the biometric authentication process by comparing the biometric information related to the user determined to be registered with the biometric information acquired via the acquisition means; 21. The image capturing device according to claim 20, wherein the detection means detects suspicion of use of the image capturing device by the different person based on the similarity of the biometric information compared by the second authentication means.

25. 2. The photographing device according to claim 1, wherein the memory is built into the photographing device or is physically detachably connected to the photographing device.

26. the photographing device is connected to the server having the memory via a wired and / or wireless communication path; 2. The photographing device according to claim 1, wherein said control means transmits a signal to said server so that said photographed image and said user information are recorded in association with each other in said memory.

27. A method of operating an imaging device, comprising: acquiring biometric information of a person; a step of performing a first authentication process to determine whether the user of the photographing device is a person who has been registered in advance; determining whether the user of the photographing device is the same person as the user determined to be registered in the first authentication process by a biometric authentication process using the acquired biometric information; a step of controlling the recording of the captured image by the image capturing device in association with information about the user in a memory in accordance with a result of the determination by the biometric authentication processing; 10. A method of operation comprising:

28. A program for causing a computer to function as the imaging device according to any one of claims 1 to 26.

Images