Photographing apparatus, photographing apparatus operation method, and program

Abstract

The present invention improves the usability of a device while preventing an increase in the false acceptance rate during personal authentication of a user of the device.　Biological information of a person is acquired. Whether a user of a photographing apparatus is a pre-registered person is determined. Whether the user of the photographing apparatus is the same person as a user who is determined to be registered is determined by biometric authentication processing using the acquired biological information. Control is performed so that, in accordance with the result of the determination, an image captured by the photographing apparatus and information of the user are associated with each other and recorded in a memory.

Description

Photographing device, photographing device operation method, and program

[0001] The present invention relates to an image capturing device, an operation method for an image capturing device, and a program, and more particularly to a technology for authenticating a device user.

[0002] Conventionally, methods for authenticating a user of an information processing device including an image capture device have been used. For example, Patent Document 1 discloses a method for authenticating a user using an image of the user's eye looking through a viewfinder of an image capture device.

[0003] Japanese Patent Application Laid-Open No. 2024-2562

[0004] To more reliably ensure that a registered person is using a device, authentication is generally configured to reduce the likelihood of recognizing a different person as a specific registered person (false acceptance rate). However, setting a low false acceptance rate increases the likelihood of not recognizing a registered person (false rejection rate). This increases the likelihood that the device will not operate based on personal authentication, reducing the usability of the device. Furthermore, when a highly accurate authentication method that can achieve both a low false acceptance rate and a low false rejection rate is used, the time required for authentication processing generally increases. This also reduces the usability of the device.

[0005] An embodiment of the present invention can improve the usability of a device while preventing an increase in the false acceptance rate in personal authentication of a device user.

[0006] An image capturing device according to one embodiment of the present invention has the following configuration: an image capturing device comprising: an acquisition means for acquiring biometric information of a person, a first authentication means for determining whether a user of the image capturing device is a person who has been registered in advance, a second authentication means for determining whether the user of the image capturing device is the same person as a user determined to be registered by the first authentication means through biometric authentication processing using the biometric information acquired via the acquisition means, and a control means for controlling, according to the result of the determination by the second authentication means, to associate an image captured by the image capturing device with information about the user and record it in a memory.

[0007] The usability of the device can be improved while preventing an increase in the false acceptance rate in personal authentication of the device user.

[0008] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.

[0009] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments of the present invention, and are used, together with the description, to explain the principles of the present invention. An external view of a camera according to an embodiment. An external view of a camera according to an embodiment. A cross-sectional view of a camera according to an embodiment. A block diagram showing the configuration of a camera according to an embodiment. A diagram showing a field of view within a viewfinder. A diagram showing a field of view within a viewfinder. A diagram showing a field of view within a viewfinder. A diagram showing a field of view within a viewfinder. A diagram for explaining the principle of a gaze detection method. A diagram showing an eye image. A flowchart of gaze detection processing. A block diagram showing the functional configuration related to authentication processing of a camera according to an embodiment. A diagram showing a registered personal information table. A diagram showing a first authentication registration feature vector table. A diagram showing a second authentication registration feature vector table. A diagram showing an authentication status table. A flowchart of user registration processing. A flowchart of eye image acquisition processing. A flowchart of first authentication processing. A flowchart of first authentication processing. A flowchart of second authentication processing. A flowchart of second authentication processing. A flowchart of third-party use detection processing. A flowchart of first authentication invalidation processing. A flowchart of first authentication invalidation processing. A flowchart of first authentication invalidation processing. 10 is a flowchart illustrating an authentication status saving process.

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] An information processing device according to an embodiment has a biometric authentication function. The operation of the information processing device is controlled according to the result of the authentication process. The following describes the configuration and operation of an image capturing device, which is an example of such an information processing device.

[0012] [Configuration of the Photographing Device] FIGS. 1A-1B show the exterior of a camera 100 (digital still camera; interchangeable lens camera) that is a photographing device according to this embodiment. FIG. 1A is a front perspective view. FIG. 1B is a rear perspective view. As shown in FIG. 1A, the camera 100 has a photographing lens unit 100A and a camera housing 100B. The camera housing 100B is provided with a release button 101, which is an operation member that accepts photographing operations from the user (photographer). As shown in FIG. 1B, an eyepiece 102 (viewfinder) is provided on the rear of the camera housing 100B. The user looks into the eyepiece 102 to view a display device 214 (display panel) (described below) included within the camera housing 100B. The rear of the camera housing 100B also has operation members 103-105 that accept various operations from the user. For example, operation member 103 is a touch panel that accepts touch operations. Operation member 104 is an operation lever that can be pushed in any direction. The operation member 105 is a four-way key that can be pressed in each of four directions. The operation member 103 (touch panel) is equipped with a display panel (e.g., a liquid crystal panel). This display panel has a function of displaying images.

[0013] 2 is a cross-sectional view of the camera 100 taken along the YZ plane defined by the Y axis and Z axis shown in FIG.

[0014] The photographing lens unit 100A includes two lenses 201 and 202, an aperture 203, an aperture driver 204, a lens drive motor 205, a lens drive member 206, a photocoupler 207, a pulse plate 208, a mount contact 209, and a focus adjustment circuit 210. The lens drive member 206 includes a drive gear. The photocoupler 207 detects the rotation of the pulse plate 208, which is linked to the lens drive member 206, and transmits this rotation to the focus adjustment circuit 210. The focus adjustment circuit 210 drives the lens drive motor 205 based on information from the photocoupler 207 and information from the camera housing 100B (lens drive amount information). The focus adjustment circuit 210 drives the lens drive motor 205 to move the lens 201, thereby changing the focus position. The mount contact 209 is an interface between the photographing lens unit 100A and the camera housing 100B. Although two lenses 201 and 202 are shown in FIG. 2, more than two lenses may be included in photographing lens unit 100A.

[0015] The camera housing 100B includes an image sensor 211, a CPU 212, a memory unit 213, a display device 214, and a display device drive circuit 215. The image sensor 211 is disposed at the intended imaging plane of the photographing lens unit 100A. The CPU 212 is a central processing unit of a microcomputer. The CPU 212 controls the entire camera 100. The memory unit 213 stores various information. For example, the memory unit 213 stores images captured by the image sensor 211. The display device 214 displays various information on the screen (display surface) of the display device 214. For example, the display device 214 is a liquid crystal panel. The display device 214 can display the captured image (subject image) on the screen. The display device drive circuit 215 drives the display device 214. The user can view the screen of the display device 214 through the eyepiece 102.

[0016] The camera housing 100B further includes light sources 216a and 216b, a light splitter 217, a light receiving lens 218, and an eye imaging element 219. The light sources 216a and 216b are light sources for illuminating the eyeball 220 of a user looking through the viewfinder (eyepiece 102). The light sources 216a and 216b have traditionally been used in single-lens reflex cameras to detect the line of sight from the relationship between the pupil and the corneal reflection image of light. The light sources 216a and 216b are arranged around the eyepiece 102. For example, the light sources 216a and 216b are infrared light-emitting diodes. The light sources 216a and 216b can emit infrared light that is invisible to the user. An optical image of the illuminated eyeball 220 (eye optical image; an optical image formed by light emitted from the light sources 216 a and 216 b and reflected by the eyeball 220) passes through the eyepiece 102 and is reflected by the light splitter 217. The eye optical image is then formed on the eye imaging element 219 by the light receiving lens 218. The eye imaging element 219 has a configuration in which multiple photoelectric conversion elements (e.g., CCD or CMOS) are arranged two-dimensionally. The light receiving lens 218 positions the pupil of the eyeball 220 and the eye imaging element 219 in a conjugate imaging relationship. A gaze detection process, which will be described later, detects the gaze of the eyeball 220 from the position of the corneal reflection image in the eye optical image formed on the eye imaging element 219. For example, the gaze detection process obtains, as information related to the gaze, at least one of information indicating the gaze direction (the direction of the gaze) and information indicating the viewpoint (the position at which the gaze is fixed) on the screen of the display device 214. The viewpoint can be considered as the position where the user is looking. The viewpoint can also be considered as the line of sight.

[0017] 3 is a block diagram showing the electrical configuration within camera 100. Connected to CPU 212 are gaze detection circuit 301, photometry circuit 302, autofocus detection circuit 303, signal input circuit 304, display device drive circuit 215, and light source drive circuit 305. CPU 212 transmits signals via mount contacts 209 to focus adjustment circuit 210 disposed within photographing lens unit 100A and aperture control circuit 306 included in aperture drive unit 204 within photographing lens unit 100A. Memory unit 213 associated with CPU 212 has the function of storing image pickup signals from image pickup element 211 and eye image pickup element 219.

[0018] The gaze detection circuit 301 A / D converts the output of the eye imaging element 219 (an eye image obtained by capturing an image of the eye (eyeball 220)) when an optical image of the eye is formed on the eye imaging element 219. The gaze detection circuit 301 then transmits the conversion result to the CPU 212. The CPU 212 extracts feature points necessary for gaze detection from the eye image in accordance with gaze detection processing, which will be described later. The CPU 212 then detects the user's gaze from the positions of the feature points.

[0019] The photometry circuit 302 performs predetermined processing (e.g., amplification, logarithmic compression, and A / D conversion) on a signal obtained from the image sensor 211, which also functions as a photometry sensor, such as a luminance signal corresponding to the brightness of the subject field. The photometry circuit 302 then sends the processing result to the CPU 212 as subject field luminance information.

[0020] The autofocus detection circuit 303 A / D converts signals from multiple detection elements (multiple pixels) included in the image sensor 211 that are used for phase difference detection, and sends the converted signals to the CPU 212. The CPU 212 calculates the distance to the subject corresponding to each focus detection point from the signals from the multiple detection elements. This distance calculation method is known as image sensor phase difference AF. In this embodiment, there are 180 focus detection points on the image sensor. The 180 focus detection points correspond to the 180 focus detection point indicators 401 that exist in the viewfinder field of view (screen of the display device 214) shown in FIG. 4A .

[0021] Switches SW1 and SW2 are connected to the signal input circuit 304. Switch SW1 is a switch for starting the photographing preparation operation (e.g., photometry and distance measurement) of the camera 100. Switch SW1 is turned ON with the first stroke of the release button 101. Switch SW2 is a switch for starting the photographing operation. Switch SW2 is turned ON with the second stroke of the release button 101. ON signals from switches SW1 and SW2 are input to the signal input circuit 304 and transmitted to the CPU 212. Gaze detection may be started when switch SW1 is turned ON.

[0022] The light source drive circuit 305 drives the light sources 216a and 216b.

[0023] Furthermore, the operation members 103 to 105 are also connected to the CPU 212. When the user operates the operation members 103 to 105, the operation members 103 to 105 output operation signals corresponding to the user's operation to the CPU 212. The CPU 212 then performs processing (control) corresponding to the operation signals. For example, the CPU 212 can move a selection frame in a displayed menu in response to the operation signals.

[0024] FIG. 4A is a diagram showing the field of view within the viewfinder. FIG. 4A shows the display device 214 in operation (a state in which an image is displayed). As shown in FIG. 4A, the field of view within the viewfinder includes a focus detection area 400, 180 ranging point indices 401, and a field of view mask 402. Each of the 180 ranging point indices 401 is displayed at a position corresponding to a focus detection point on the imaging surface. The 180 ranging point indices 401 are also displayed superimposed on a through image (live view image) displayed on the display device 214. Of the 180 ranging point indices 401, the ranging point indices 401 that corresponds to the current viewpoint 411 (estimated position) is displayed highlighted, for example, with a frame.

[0025] 8A is a block diagram showing a functional configuration used for authentication by an information processing device according to an embodiment. Using this configuration, the camera 100 can perform personal authentication of the user.

[0026] In this embodiment, both a first authentication process and a second authentication process are used to authenticate a user. In the first authentication process, authentication is performed to verify that the user of a device (e.g., the camera 100) is a previously registered person. In this embodiment, the first authentication process is performed before the user takes a photograph using the camera 100. In addition, the second authentication process determines whether the user of the device (e.g., the camera 100) is the same person as the user determined to be registered in the first authentication process. Such second authentication process can be performed when a predetermined operation using the device is performed. Alternatively, such second authentication process may be performed using authentication information (e.g., biometric information) acquired when performing a predetermined operation using the device. In this embodiment, this predetermined operation is a photographing operation, and the second authentication process is performed during photographing. Note that, in this specification, "during photographing" refers to immediately before, during, or immediately after photographing. Then, the operation of the device can be controlled at least according to the result of the second authentication process. For example, the operation of the camera 100 is controlled to record the results of the first and second authentication processes together with the captured image. Furthermore, if the second authentication process fails, the operation of the camera 100 is controlled to record information indicating the authentication failure. In this embodiment, depending on the result of the second authentication process, the image captured by the camera 100 and the user information are associated with each other and recorded in memory.

[0027] In this embodiment, user authentication is performed using biometric information. For example, in the second authentication process, user authentication can be performed using biometric information. In this embodiment, the biometric information is an eye image. That is, the camera 100 authenticates whether the user is a previously registered person based on an image of the user's eyeball 220 looking through the viewfinder (eyepiece 102).

[0028] Each component will be described below. The imaging unit 813 is mainly realized by the eye imaging element 219. The other components shown in FIG. 8A can be realized by a processor such as the CPU 212 executing a program stored in a memory such as the memory unit 213. However, some or all of the functions of the camera 100 may be realized by dedicated hardware. The information processing device according to one embodiment of the present invention may be realized by a computer including a processor and a memory.

[0029] The camera 100 has an acquisition unit that acquires biometric information of a person. The biometric information acquired by the acquisition unit is used for biometric authentication. In this embodiment, an image acquisition unit 801 acquires the biometric information of a person. In this embodiment, the image acquisition unit 801 acquires an image of the eyeball 220 of a user looking through the viewfinder (eyepiece 102). Specifically, the image acquisition unit 801 acquires an eye image (eye image signal; electrical signal of the eye image) from the eye imaging element 219 via the gaze detection circuit 301. A specific example of the eye image acquisition process will be described later with reference to FIG. 10 .

[0030] The feature calculation unit 802 calculates feature quantities used for authentication from the biometric information. In this embodiment, the feature calculation unit 802 calculates a feature vector used for authentication from the eye image acquired by the image acquisition unit 801. A feature extractor can be used to calculate the feature vector. A neural network can be used as the feature extractor. For example, a convolutional neural network (CNN), which is a type of neural network, can be used. In processing using a CNN, abstracted information is extracted from an input image by repeatedly performing processing including convolution processing, activation processing, and pooling processing on the input image. In this case, a processing unit consisting of convolution processing, activation processing, and pooling processing is called a layer. Various activation processing methods are known. For example, a method called rectified linear unit (ReLU) may be used for activation processing. Various pooling processing methods are also known. For example, a technique called maximum pooling may be used for the pooling process. As the CNN, ResNet, which is introduced in non-patent literature (K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In ECCV, 2016), may be used. Alternatively, a neural network known as VisionTransformer (ViT) described in a non-patent document (Alexey Dosovitskiy, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.) may be used. The configuration of the neural network is not limited to these. Information indicating the structure and weights of the neural network used by the feature calculation unit 802 can be held by the memory unit 213 or the like.

[0031] The weights of the neural network used by the feature calculation unit 802 are obtained in advance through training. For training, various previously acquired eye images of people can be used. The neural network can be trained using a method such as ArcFace, which is shown in non-patent document (J. Deng, J. Guo, N. Xue, and S. Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In CVPR, 2019).

[0032] It is not essential to use an eye image or a neural network as a method for personal authentication. For example, other methods such as iris authentication (for example, the method described in Japanese Patent Laid-Open No. 8-504979) may be used. The method for personal authentication is not limited to a specific method.

[0033] The registration management unit 803 manages registration information of users of the device (camera 100 in this example). In this embodiment, the registration management unit 803 manages feature amounts of registered people and the names of the registered people in association with each other. In this embodiment, feature vectors of eye images are registered as feature amounts. The registration management unit 803 stores the information it manages in the memory unit 213. Here, the registration management unit 803 can manage, in association with each other, registration information used by the first authentication unit 805 to determine the user of the device and registration information used by the second authentication unit 806 to determine the user of the device.

[0034] The registration management unit 803 can manage registration information using, for example, the tables shown in Figures 8B-8D. Figure 8B shows a registered personal information table. Figure 8C shows a first authentication registered feature vector table. Figure 8D shows a second authentication registered feature vector table. A person ID is assigned to a registered person. Information about the same person is associated between these tables using this person ID. The registered personal information table records "name" information. The first authentication registered feature vector table records feature vectors (registered feature vectors) used by the first authentication unit 805 for authentication. The second authentication registered feature vector table records feature vectors (registered feature vectors) used by the second authentication unit 806 for authentication.

[0035] The registration unit 804 creates data to be registered in the registration management unit 803. A specific example of the user registration process performed by the registration unit 804 will be described later with reference to FIG.

[0036] The first authentication unit 805 performs a first authentication process to determine whether the user of the device (in this example, the camera 100) is a person who has been registered in advance. In this embodiment, the first authentication unit 805 authenticates whether the user is a person who has been registered in the registration management unit 803 when not taking a photograph. In this embodiment, the first authentication unit 805 can perform authentication based on a comparison of the user's features obtained during the first authentication process with the features registered in the registration management unit 803. The first authentication unit 805 can also control the first authentication state based on the results of the first authentication process. For example, in response to the first authentication unit 805 determining that the user is registered, the state management unit 810 (described later) can set the first authentication state to active. In this way, if the first authentication process is successful, the camera 100 enters the first authentication state (i.e., a state in which the first authentication process has been successful). A specific example of the first authentication process will be described later with reference to FIGS. 11A-11B.

[0037] The second authentication unit 806 performs second authentication processing to determine whether the device user is the same person as the user determined to be registered in the first authentication processing. In this embodiment, the second authentication unit 806 can perform authentication processing based on a comparison of the user's feature values ​​obtained during the second authentication processing with the feature values ​​registered in the registration management unit 803. In the second authentication processing, the second authentication unit 806 can perform biometric authentication processing using biometric information acquired via the image acquisition unit 801. In this embodiment, the second authentication unit 806 performs biometric authentication processing using the feature vector generated by the feature calculation unit 802 as the biometric information. The second authentication unit 806 can also perform the second authentication processing when a predetermined operation using the device is performed. The second authentication unit 806 can also perform biometric authentication processing using the biometric information acquired via the image acquisition unit 801 when the device performs the predetermined operation. In this embodiment, the second authentication unit 806 performs processing to authenticate whether the person authenticating the first authentication unit 805 is the person taking the photograph.

[0038] Furthermore, the second authentication unit 806 can control the second authentication state based on the result of the second authentication process. For example, in response to the second authentication unit 806 determining that the user is the same person as the user determined by the second authentication unit 806 to have been registered, the state management unit 810 (described later) can set the second authentication state to valid. In this way, if the second authentication process is successful, the camera 100 enters the second authentication state (i.e., a state in which the second authentication process has been successful). The second authentication unit 806 can control the second authentication state so that this second authentication state continues only while capturing images. For example, in response to the second authentication unit 806 determining that the user has finished capturing images, the state management unit 810 (described later) can set the second authentication state to invalid. A specific example of the second authentication process will be described later with reference to FIGS. 12A-12B.

[0039] In this embodiment, the authentication settings used by the first authentication unit 805 for the first authentication process are different from the authentication settings used by the second authentication unit 806 for the second authentication process. For example, the first authentication unit 805 can use authentication settings that result in a low false acceptance rate. On the other hand, the second authentication unit 806 can use authentication settings that result in a low false rejection rate. In this way, the second authentication unit 806 can perform authentication using a method that results in a lower false rejection rate than the first authentication unit 805. For example, when the first authentication unit 805 and the second authentication unit 806 use the same authentication method, a method of changing the similarity threshold can be used. The first authentication unit 805 can authenticate the user by comparing information acquired from the user with information registered for the user. Here, if the similarity of the information is higher than the threshold, the first authentication unit 805 can determine that the user is already registered. The second authentication unit 806 can also perform authentication using a similar method. For example, the second authentication unit 806 can perform biometric authentication processing by comparing biometric information associated with a user determined to be registered with biometric information acquired via the image acquisition unit 801. If the similarity of the information is higher than a threshold, the second authentication unit 806 can determine that the user is the same person as the user determined to be registered. A high threshold can be used in the first authentication processing, and a low threshold can be used in the second authentication processing. This configuration can reduce the false acceptance rate in the first authentication processing and the false rejection rate in the second authentication processing. Even if it is difficult to reduce both the false acceptance rate and the false rejection rate in authentication performed during device use, performing two-step authentication in this manner can reduce the false rejection rate during device use while preventing an increase in the false acceptance rate.

[0040] In one embodiment, the second authentication unit 806 determines the first authentication status. Then, when it is determined that the first authentication status is set to valid (i.e., when it is determined that the first authentication process is successful), the second authentication unit 806 performs the second authentication process (e.g., the biometric authentication process). With this configuration, when the later-described invalidation unit 808 invalidates the first authentication status, it is possible to prevent other people from using the device based on the result of the second authentication process.

[0041] The detection unit 807 detects suspicion of device use by a person other than the user determined to be registered by the first authentication unit 805. In this embodiment, the detection unit 807 detects that photography is being performed by a person (other person) other than the person authenticated by the first authentication unit 805. The detection unit 807 can detect suspicion of device use by a person other than the person when a predetermined condition indicating suspicion of device use by a person other than the person is satisfied.

[0042] For example, the detection unit 807 can detect suspicion of device use by another person based on the result of the second authentication process performed by the second authentication unit 806. The detection unit 807 can also detect suspicion of device use by another person based on the history of the second authentication process. For example, the detection unit 807 can detect suspicion of device use by another person based on a trend of authentication failures in the second authentication unit 806.

[0043] Specifically, the detection unit 807 can detect suspicion of device use by another person based on the number of failures in the second authentication process. The detection unit 807 may also detect suspicion of device use by another person based on the number of failures in the biometric authentication process. In one embodiment, the detection unit 807 detects suspicion of device use by another person when authentication failures by the second authentication unit 806 occur a predetermined number of times or more consecutively. The second authentication unit 806 may also detect suspicion of device use by another person based on the authentication score. For example, as described above, the second authentication unit 806 can perform biometric authentication processing by comparing authentication information (e.g., biometric information). At this time, the similarity of the compared authentication information can be used as the authentication score. That is, the second authentication unit 806 may detect suspicion of use of the imaging device by a different person based on the similarity of the compared authentication information. For example, when the authentication score is significantly low, the second authentication unit 806 can detect suspicion of device use by another person. The detection unit 807 may use these conditions in combination. Details of the process of detecting suspicion of device use by another person will be described later with reference to FIG. 13 . However, methods for detecting suspicion of device use by a third party are not limited to these methods.

[0044] The invalidation unit 808 can perform first authentication invalidation processing to set the first authentication state to invalid (i.e., switch to a state where the first authentication processing has not been successful) according to predetermined conditions. The invalidation unit 808 determines whether or not to invalidate the first authentication state in the first authentication state where the first authentication by the first authentication unit 805 has been successful. The conditions under which the invalidation unit 808 sets the first authentication state to invalid are not particularly limited. Examples of methods for invalidation determination are given below.

[0045] A first invalidation determination method is a method based on the elapsed time since the first authentication process was successful. The invalidation unit 808 can set the first authentication state to invalid based on the elapsed time since the first authentication process was set to valid. For example, when the elapsed time since the first authentication process was successful exceeds a predetermined lifetime, the invalidation unit 808 can invalidate the first authentication state. The invalidation unit 808 can also set the first authentication state to invalid based on the result of the second authentication process. For example, the invalidation unit 808 can change the lifetime based on the result of the second authentication process. Specifically, the invalidation unit 808 can extend the lifetime if the second authentication process is successful in the first authentication state. Conversely, the invalidation unit 808 can shorten the lifetime if the second authentication process fails. An example of an invalidation determination method based on the elapsed time will be described later with reference to FIG. 14A . However, the invalidation determination method is not limited to this example.

[0046] A second invalidation determination method is a method based on a change in the power state. The invalidation unit 808 can set the first authentication state to invalid based on a change in the power state of the device. For example, the invalidation unit 808 can set the first authentication state to invalid when the camera 100 is turned off or when the camera 100 enters sleep mode. Furthermore, the invalidation unit 808 may set the first authentication state to invalid when the camera 100 is turned on or when the camera 100 returns from sleep mode. With this configuration, even if the process of setting the first authentication state to invalid cannot be executed because the power was turned off due to a dead battery or the like, the first authentication state can be set to invalid when the power is turned on again. The same applies to sleep mode. An example of an invalidation determination method based on a change in the power state will be described later with reference to FIG. 14B . However, the invalidation determination method is not limited to this example.

[0047] A third invalidation determination method is a method based on the distance or connection status with another device. The invalidation unit 808 can invalidate the first authentication state based on the distance or connection status between the device (e.g., the camera 100) and the other device. The other device may be a device carried by the user. An example of the other device is a smartphone. For example, the user's smartphone can be connected to the camera 100 via Bluetooth. When the Bluetooth connection is terminated, the invalidation unit 808 can invalidate the first authentication state. Alternatively, the invalidation unit 808 can estimate the distance between the camera 100 and the other device based on the connection status. When the estimated distance exceeds a predetermined value, the invalidation unit 808 can invalidate the first authentication state. With this configuration, the first authentication state is invalidated when the user moves away from the camera 100, for example, by putting the camera 100 down. This prevents other people from using the camera 100. Note that an example of the other device is an RFID tag. The other device is not limited to these. An example of a method for determining invalidation based on the distance and connection state to other devices will be described later with reference to Fig. 14C, but the method for determining invalidation is not limited to this example.

[0048] A fourth invalidation determination method is a method based on an explicit invalidation operation input by the user. The invalidation unit 808 can set the first authentication state to invalid based on an input by the user to a device (e.g., the camera 100). For example, an item "Invalidate First Authentication" can be included in the operation menu. The user can select this item using the operation members 103 to 105. Alternatively, the camera 100 may be provided with a switch button such as an invalidation button. The user can press this button. When this operation is received, the invalidation unit 808 can invalidate the first authentication state. An example of an explicit invalidation operation input by the user will be described later with reference to FIG. 14D . However, the invalidation method is not limited to this example.

[0049] A fifth invalidation determination method is a method based on the detection result of the detection unit 807. The invalidation unit 808 can set the first authentication status to invalid when it is detected that a person other than the user determined to be registered is suspected of using the device. In this way, when the detection unit 807 detects that a third party is suspected of using the device, it is possible to invalidate the first authentication status.

[0050] The invalidation unit 808 can use a combination of multiple invalidation determination methods as described above. By using these methods in combination, the possibility of recognizing a different person as a specific registered person in the second authentication process can be further reduced. For example, the fourth method can prevent use by a different person through a conscious operation by the user. In addition, the first to third methods can prevent use of the device by a different person by invalidating the first authentication process when it is unlikely that the user authenticated in the first authentication process is using the device. Furthermore, the fifth method can prevent use of the device by a different person by invalidating the authentication status when it is suspected that the device is being used by a different person.

[0051] The execution unit 809 controls the operation of the device in accordance with the determination result by the second authentication unit 806. The execution unit 809 can control the operation of the device in accordance with the second authentication state. The execution unit 809 can further control the operation of the device in accordance with the determination result by the first authentication unit 805. Specifically, the execution unit 809 can control the operation of the device in accordance with the first authentication state. In this way, the execution unit 809 can execute processing in accordance with the first authentication state and the second authentication state.

[0052] In one embodiment, the execution unit 809 can perform a specific operation when both the first authentication status and the second authentication status are valid. The specific operation is not particularly limited. For example, the specific operation may be a photographing operation using the camera 100. In another embodiment, the execution unit 809 can record the first authentication status and the second authentication status. For example, the execution unit 809 can save records of the first authentication status and the second authentication status in association with a record of the result of the specific operation. For example, the execution unit 809 can save in the memory unit 213 a photographed image obtained by the photographing operation and records of the first authentication status and the second authentication status in association with each other.

[0053] The execution unit 809 includes a state management unit 810, a state saving unit 811, and a state display unit 812. The state management unit 810 can manage the first authentication state of the device. The state management unit 810 can also manage the second authentication state of the device. Furthermore, the state management unit 810 can manage information indicating whether or not there is suspicion of unauthorized use of the device. The state management unit 810 stores the managed information in the memory unit 213. For example, the state management unit 810 can store the authentication state table shown in FIG. 8E in the memory unit 213. In the authentication state table, the "first authentication state" indicates whether or not the device is in the first authentication state. The "first authentication state" takes one of two values: "authenticated," which indicates that the first authentication state is valid, and "unauthenticated," which indicates that the first authentication state is invalid. The "person ID" is the person ID of the user identified by the first authentication process. When the "first authentication state" is "unauthenticated," the "person ID" takes a value indicating an empty value, such as NULL. The "second authentication status" indicates whether the device is in the second authentication status. The "second authentication status" takes one of two values: "authenticated," which indicates that the second authentication status is valid, and "unauthenticated," which indicates that the first authentication status is invalid. The "presence or absence of use by another person" indicates whether the detection unit 807 has detected suspicion of use by another person. The "presence or absence of use by another person" takes one of two values: "yes" and "no." Specific examples of the update process for the authentication status table will be described later together with the explanations of the first authentication process (FIGS. 11A-11B), the second authentication process (FIGS. 12A-12B), and the first authentication invalidation process (FIGS. 14A-14D). The method for managing the authentication status table is not limited to the method using the table structure shown in FIG. 8E. For example, the authentication status table may be managed using a key-value structure or the like. Furthermore, the management format of this information is not limited to the above.

[0054] The imaging unit 813 records the image captured by the imaging element 211 in the memory unit 213. The imaging unit 813 may perform such a photographing operation in response to receiving a signal indicating that the release button 101 has been pressed by the user.

[0055] The state saving unit 811 performs control so that information indicating each of the first authentication state and the second authentication state is recorded in the memory. For example, the state saving unit 811 can record information indicating each of the first authentication state and the second authentication state managed by the state management unit 810 in the memory unit 213 in association with a captured image acquired by the imaging unit 813. The state saving unit 811 can record information indicating the authentication state and suspicion of use by another person as metadata of the captured image. One method for recording image metadata is known as C2PA, which is described in the non-patent document Coalition for Content Provenance and Authenticity (C2PA), "C2PA Specifications", <Technical Specifications Version 1.2>, [online], November 3, 2022, [retrieved January 23, 2023], Internet <URL: https: / / c2pa.org / specifications / specifications / 1.2 / specs / C2PA_Specification.html>. C2PA relates to a method for adding metadata to an image that indicates edits made to the image in order to authenticate the origin, history or provenance of the image. For this reason, the authentication status and the like may be recorded in accordance with C2PA. However, this information such as the authentication status may also be recorded by other methods. Alternatively, the image file and the metadata file may be recorded as separate files. Alternatively, the metadata may be managed in a database. However, the metadata recording method is not limited to these. A specific example of the authentication status saving process will be described later with reference to FIG. 15A.

[0056] The status display unit 812 notifies the authentication status of the device. For example, the status display unit 812 can separately notify the first authentication status and the second authentication status. The status display unit 812 may notify the authentication status of the device via the device. For example, the status display unit 812 may display the authentication status of the device on the device. In this embodiment, the status display unit 812 notifies the first authentication status on the camera 100. For example, the status display unit 812 can display the authentication status on the camera 100 based on the authentication status managed by the status management unit 810. When the first authentication status is "authenticating," the status display unit 812 can display "first authentication in progress" or the like on the display device 214 or the touch panel (operation member 103). The camera 100 may also be equipped with a lamp such as an LED lamp (not shown). The status display unit 812 can turn on the lamp when the first authentication status is "authenticating."

[0057] [User Registration Process] The user registration process in this embodiment will be described with reference to the flowchart in FIG. 9. This process is mainly performed by the registration unit 804. The CPU 212 can realize the operation of the registration unit 804. The user can start the registration process by operating the camera 100 at a time other than when taking a picture. For example, this process may be executed when the user operates the camera 100 to call this process from a menu or the like. For example, a menu screen (not shown) can be displayed on the touch panel (operation member 103) of the camera 100. The user can select an item to call this process by operating the operation members 103 to 105. Below, this process will be described step by step.

[0058] In S901, the registration unit 804 accepts input of personal information of the person to be registered. In this embodiment, a "name" is input. Specifically, a screen for inputting a name (not shown) is displayed on the touch panel (operation member 103). The user then operates the operation members 103 to 105 to input the name. When the user has finished inputting the name, the user notifies the user that the name input is complete by pressing a complete button or the like displayed on the screen.

[0059] In S902, the registration unit 804 instructs the user on how to register an eye image. Specifically, the registration unit 804 can cause the touch panel to display instructions instructing the user to look into the viewfinder. The registration unit 804 can cause the touch panel to display instructions instructing the user to look at an indicator in the viewfinder. In addition, the registration unit 804 may cause the touch panel to display instructions that are helpful for capturing a desirable eye image, such as not blinking and keeping the eyes wide open.

[0060] Next, in steps S903 to S911, the indices 421 to 425 shown in Fig. 4C are displayed in order on the display device 214. Then, the feature vectors obtained from the image of the user's eyes looking at these indices are stored in the registration management unit 803. The processing will be explained in order below.

[0061] In S903, the registration unit 804 displays the indicators on the display device 214. Specifically, only the indicator 421 shown in FIG. 4C is displayed, and the other indicators are not displayed. As an alternative method, all indicators may be displayed, while only the indicator 421 is highlighted in color. Other display methods that can inform the user that the indicator 421 is to be viewed may also be used. The indicator display method is not limited to these.

[0062] In S904, an image of the user's eye looking through the viewfinder (eyepiece 102) is acquired. The detailed processing in S904 will be described with reference to FIG. 10 . This processing is mainly performed by the image acquisition unit 801. The CPU 212 can realize the operation of the image acquisition unit 801.

[0063] In step S1001, a gaze detection process is executed. Details of the gaze detection process will be described later with reference to the flowchart in FIG.

[0064] In S1002, the image acquisition unit 801 determines whether an image suitable for authentication has been acquired. The image acquisition unit 801 can make this determination based on whether the gaze detection process was successful. For example, in the gaze detection process, the image acquisition unit 801 acquires an eye image (eye image signal; an electrical signal of the eye image) from the eye imaging element 219 via the gaze detection circuit 301. The image acquisition unit 801 then obtains the corneal reflection images of the light sources 216a and 216b and the coordinates of the pupil center observed on the eye image. The image acquisition unit 801 then obtains the user's gaze coordinates on the display device 214 from these coordinates. Therefore, if the coordinates of the pupil center, etc. cannot be detected, the gaze detection process fails. Therefore, in S703, which will be described later, if the coordinates of the pupil center, etc. cannot be obtained, the image acquisition unit 801 may determine that an image suitable for authentication has not been acquired.

[0065] In S1003, the image acquisition unit 801 performs processing control based on the determination result in S1002. If the image acquisition unit 801 determines in S1002 that an image suitable for authentication has been acquired, the process proceeds to S1004. Otherwise, the process proceeds to S1006.

[0066] In S1004, the image acquisition unit 801 acquires an eye image. Specifically, the image acquisition unit 801 acquires the eye image acquired in S702. Then, using the coordinates of the pupil-centered image c' acquired in S703, the image acquisition unit 801 crops an image of a certain size from the eye image so that the pupil-centered image c' is located at the center of the image. Furthermore, the image acquisition unit 801 resizes the acquired image to fit the input size of the neural network used by the feature calculation unit 802.

[0067] In S1005, the image acquisition unit 801 uses a flag or the like to record that the eye image has been successfully acquired.

[0068] In S1006, the image acquisition unit 801 performs a process of waiting for a predetermined time, such as several hundred milliseconds. This is expected to change the eye image obtained, leading to successful gaze detection.

[0069] In S1007, the image acquisition unit 801 determines whether or not there have been consecutive failures. A failure indicates that it has been determined in S1003 that an image suitable for authentication cannot be acquired. If the image acquisition unit 801 determines that there have been consecutive failures a predetermined number of times, the process proceeds to S1008. Otherwise, the process returns to S1001.

[0070] In S1008, the image acquisition unit 801 uses a flag or the like to record the fact that acquisition of the eye image has failed.

[0071] Returning now to the description of Fig. 9, in S905, the registration unit 804 determines whether or not the acquisition of the eye image in S904 was successful. The registration unit 804 can make this determination based on the flag recorded in S1005 or S1008. If the registration unit 804 determines that the acquisition of the eye image was successful, the process proceeds to S906. Otherwise, the process proceeds to S908.

[0072] In S906, a feature vector is extracted from the eye image. Specifically, the feature calculation unit 802 can extract a feature vector from the eye image acquired in S1004.

[0073] In S907, the registration unit 804 displays information indicating that the eye image corresponding to the displayed index has been successfully captured on the display device 214. For example, the display device 214 may display a message such as "Eye image captured successfully." Alternatively, the display device 214 may display an icon indicating success.

[0074] In S908, the registration unit 804 displays information indicating that capturing an eye image corresponding to the displayed index has failed on the display device 214. For example, the display device 214 may display a message such as "Failed to capture an eye image." Alternatively, the display device 214 may display an icon indicating the failure.

[0075] In S909, the registration unit 804 determines whether there are any indices that have not yet been displayed. For example, the registration unit 804 determines whether all of the indices 421 to 425 shown in FIG. 4C have already been displayed. If the registration unit 804 determines that there are any indices that have not yet been displayed, the process proceeds to S910. Otherwise, the process proceeds to S911.

[0076] In S910, the registration unit 804 performs processing to display the next index on the display device 214. For example, when the index 421 shown in Fig. 4C is displayed, the display device 214 can display the next index, index 422. In this way, the registration unit 804 selects the indexes 421 to 425 in order, and the display device 214 displays the selected index.

[0077] In S911, the registration management unit 803 registers the obtained information. The registration management unit 803 can save the obtained information in a registered personal information table, a first authentication registration feature vector table, and a second authentication registration feature vector table. The person IDs in the three tables are IDs for establishing relationships between the tables. Therefore, the same person ID is recorded in the three tables in association with information about the same person. The registration management unit 803 also adds the personal information (e.g., name) obtained in S901 to the registered personal information table shown in FIG. 8B. Furthermore, the registration management unit 803 records the feature vector obtained in S906 in the first authentication registration feature vector table shown in FIG. 8C and the second authentication registration feature vector table shown in FIG. 8D. Here, the information recorded in the first authentication registration feature vector table and the second authentication registration feature vector table may differ as follows.

[0078] In this embodiment, in the first authentication process (described later with reference to FIGS. 11A-11B ), the display device 214 displays an index, and authentication is performed based on an image of the user's eye when the user is looking at the index. Therefore, only the feature vector corresponding to the index displayed in the first authentication process can be saved in the first authentication registered feature vector table as a registered feature vector for the first authentication process. In this embodiment, the feature vector extracted from the eye image obtained when the index 421 was displayed is registered.

[0079] On the other hand, in the second authentication process (described later with reference to FIGS. 12A-12B ), the display device 214 displays an image obtained by the image sensor 211, and authentication is performed based on the eye image when the user is looking at this image. As such, in the second authentication process, no indices are displayed. Therefore, it is unclear where on the display device 214 the user is gazing. Therefore, in this embodiment, all feature vectors extracted in the process of FIG. 9 are stored in a second authentication registration feature vector table. That is, the second authentication registration feature vector table records each feature vector extracted from the eye image obtained when the indices 421 to 425 were displayed as a registration feature vector for the second authentication process.

[0080] In S912, the registration unit 804 notifies the user of the completion of registration via a display on the touch panel (operation member 103) or the display device 214.

[0081] 9, if the registration unit 804 detects that it has failed to acquire an eye image a predetermined number of times, the registration unit 804 can interrupt the user registration process. This process can prevent excessive repetition of the process when acquisition of an eye image in S904 is unsuccessful. Such exceptional processes can be added as appropriate.

[0082] [First Authentication Process] The first authentication process performed in this embodiment will be described with reference to the flowcharts of FIGS. 11A-11B. This process is mainly performed by the first authentication unit 805. The CPU 212 can realize the operation of the first authentication unit 805. The user can start the first authentication process by operating the camera 100 at a time other than when taking a picture. For example, this process may be executed when the user operates the camera 100 to call this process from a menu or the like. For example, a menu screen (not shown) can be displayed on the touch panel (operation member 103) of the camera 100. The user can select an item to call this process by operating the operation members 103 to 105. Below, this process will be described step by step.

[0083] In S1101, the first authentication unit 805 instructs the user on the authentication method. Specifically, the first authentication unit 805 can cause the touch panel (operation member 103) to display instructions indicating that the user should look into the viewfinder and see an indicator displayed on the display device 214. In addition, the first authentication unit 805 can cause the touch panel to display instructions that are useful for capturing a desirable eye image, such as not blinking, keeping the eyes wide open, and holding the camera firmly.

[0084] In S1102, the first authentication unit 805 displays an index on the display device 214. Specifically, the display device 214 can display only the index 421, as shown in FIG. 4D . This is because the feature vector obtained when the user was looking at the index 421 in the registration process described above is registered in the first authentication registration feature vector table. By displaying the image in this manner, similar eye images are obtained at the time of registration and authentication. This makes it easy to compare the eye images (or feature vectors) of both the users.

[0085] In S1103, an image of the user's eye looking through the finder (eyepiece 102) is acquired. Specifically, the eye image can be acquired by the process already described with reference to FIG.

[0086] In S1104, the first authentication unit 805 determines whether or not the acquisition of the eye image in S1103 was successful. The first authentication unit 805 can make this determination based on the flag recorded in S1005 or S1008. If the first authentication unit 805 determines that the acquisition of the eye image was successful, the process proceeds to S1106. Otherwise, the process proceeds to S1105.

[0087] In S1105, the first authentication unit 805 displays information indicating that capturing an eye image has failed on the display device 214. For example, the display device 214 may display a message such as "Failed to capture an eye image." Alternatively, the display device 214 may display an icon indicating the failure.

[0088] In S1106, a feature vector is extracted from the eye image. Specifically, the feature calculation unit 802 can extract a feature vector from the eye image acquired in S1103.

[0089] In S1107, the first authentication unit 805 acquires a registered feature vector for the first authentication process from the registration management unit 803. Specifically, the first authentication unit 805 can acquire a feature vector registered in a first authentication registered feature vector table shown in FIG.

[0090] In S1108, the first authentication unit 805 compares the feature vector obtained in S1106 with the registered feature vector obtained in S1107. The first authentication unit 805 then determines whether the registered feature vector matches the feature vector obtained in S1106. This determination can be made based on the similarity between the feature vectors. For example, the first authentication unit 805 can calculate the cosine similarity between the two feature vectors being compared. If the similarity calculated for the registered feature vector exceeds a predetermined threshold, the first authentication unit 805 can determine that the registered feature vector matches the feature vector obtained in S1106. In this case, the first authentication unit 805 identifies the person ID for the registered feature vector that matches the feature vector obtained in S1106.

[0091] In S1109, the first authentication unit 805 determines whether the first authentication process was successful. Specifically, the first authentication unit 805 can determine that the first authentication process was successful if the registered feature vector matches the feature vector obtained in S1106. At this time, the first authentication unit 805 has determined that the user of the camera 100 is a pre-registered person corresponding to the person ID identified in S1108. If the first authentication unit 805 determines that the first authentication process was successful, the process proceeds to S1110. Otherwise, the process proceeds to S1113.

[0092] In S1110, the state management unit 810 enables the first authentication state. For example, the state management unit 810 can update the "first authentication state" in the authentication state table shown in FIG. 8E to "authenticating (authentication successful state)." At this time, the state management unit 810 can disable the second authentication state. For example, the state management unit 810 can update the "second authentication state" in the authentication state table to "unauthenticated."

[0093] In S1111, the state management unit 810 registers information about the user determined to be registered by the first authentication unit 805. For example, the state management unit 810 can update the person ID in the authentication state table shown in Fig. 8E to the person ID identified in S1108.

[0094] In S1112, the status display unit 812 displays a message informing the user of successful authentication using the touch panel (operation member 103) or the display device 214 or the like.

[0095] In S1113, the state management unit 810 invalidates the first authentication state. The state management unit 810 can also invalidate the second authentication state. For example, the state management unit 810 can update the "first authentication state" and the "second authentication state" in the authentication state table shown in FIG. 8E to "unauthenticated."

[0096] In S1114, the state management unit 810 deletes the record of the user information determined by the first authentication unit 805 to be registered. For example, the state management unit 810 can delete the person ID recorded in the authentication state table shown in FIG. 8E. For example, a NULL value can be prepared as a value indicating that the person ID is empty. In this case, the state management unit 810 can overwrite the person ID with a value indicating that the person ID is empty.

[0097] In S1115, the status display unit 812 displays a message informing the user of the authentication failure using the touch panel (operation member 103) or the display device 214 or the like.

[0098] 11A-11B, if the first authentication unit 805 detects that it has failed to acquire an eye image a predetermined number of times, the first authentication unit 805 can interrupt the first authentication process. This process can prevent excessive repetition of the process when acquisition of an eye image in S1104 is unsuccessful. Such exceptional processes can be added as appropriate.

[0099] [Second Authentication Processing] The second authentication processing performed in this embodiment will be described with reference to the flowcharts of FIGS. 12A-12B. This processing is primarily performed by the second authentication unit 806. The CPU 212 can realize the operation of the second authentication unit 806. The second authentication processing can be performed after the first authentication processing. Specifically, the second authentication processing can be performed when the user looks through the viewfinder during photography. For this purpose, an eyepiece sensor (not shown) mounted on the camera 100 can detect when the user has brought their eye close to the viewfinder (eyepiece 102). This processing can be initiated when such an eyepiece sensor detects that the user has brought their eye close to the viewfinder. The eyepiece sensor may also detect when the skin around the user's eye comes into contact with the periphery of the eyepiece 102. Alternatively, the eyepiece sensor may detect the distance between the eyepiece 102 and the user's eye. If the detected distance is equal to or less than a predetermined value, it can be determined that the user is looking through the viewfinder. Alternatively, this process may start when it is detected that the release button 101 has been pressed down to the first stroke. Alternatively, the gaze detection process may be continued. Then, this process may start when the gaze detection process is successful. Below, this process will be explained step by step.

[0100] In S1201, the second authentication unit 806 determines whether the first authentication status is valid. The second authentication unit 806 can determine whether the first authentication status is valid, for example, based on whether the "first authentication status" in the authentication status table shown in FIG. 8E is "authenticated." The second authentication unit 806 further determines whether the user is continuing to capture images. The second authentication unit 806 can determine whether the user is continuing to capture images based on whether the user is keeping their eye close to the viewfinder. Whether the user is keeping their eye close to the viewfinder can be detected by an eye sensor. As described above, whether the user is continuing to capture images may also be determined by other methods, such as pressing the release button 101 or gaze detection processing. If the second authentication unit 806 determines that the first authentication status is valid and the user is continuing to capture images, the process proceeds to S1202. Otherwise, the process proceeds to S1218.

[0101] In S1202, an image of the user's eyes looking through the viewfinder (eyepiece 102) is acquired. Specifically, the image of the eyes can be acquired by the process already described with reference to FIG. 10. Note that if the gaze detection process has already been performed, the gaze detection in S1001 can be skipped. In this case, the image of the user's eyes can be acquired using the results of the gaze detection process that have already been obtained.

[0102] In S1203, the second authentication unit 806 determines whether or not the acquisition of the eye image in S1202 was successful. The second authentication unit 806 can make this determination based on the flag recorded in S1005 or S1008. If the second authentication unit 806 determines that the acquisition of the eye image was successful, the process proceeds to S1205. Otherwise, the process proceeds to S1204.

[0103] In S1204, the second authentication unit 806 displays information indicating that the eye image has not been captured on the display device 214. For example, the display device 214 may display an icon indicating the failure.

[0104] In S1205, a feature vector is extracted from the eye image. Specifically, the feature calculation unit 802 can extract a feature vector from the eye image acquired in S1202.

[0105] In S1206, the second authentication unit 806 acquires registered feature vectors for the second authentication process from the registration management unit 803. Specifically, the second authentication unit 806 acquires all registered feature vectors registered in the second authentication registered feature vector table shown in Fig. 8D.

[0106] In S1207, the second authentication unit 806 compares the feature vector obtained in S1205 with each of the registered feature vectors obtained in S1206. The second authentication unit 806 then determines whether any of the registered feature vectors matches the feature vector obtained in S1205. This determination can be made based on the similarity, or cosine similarity, between the feature vectors, as in S1108. If the similarity calculated for the registered feature vector exceeds a predetermined threshold, the second authentication unit 806 can determine that this registered feature vector matches the feature vector obtained in S1205.

[0107] In S1208, the second authentication unit 806 determines whether the second authentication process is successful. Specifically, the second authentication unit 806 can determine the success of the second authentication process if any registered feature vector matches the feature vector obtained in S1205. If the second authentication unit 806 determines that the second authentication process is successful, the process proceeds to S1209. Otherwise, the process proceeds to S1213.

[0108] In S1209, the state management unit 810 enables the second authentication state. For example, the state management unit 810 updates the "second authentication state" in the authentication state table shown in FIG. 8E to "authenticating."

[0109] In S1210, the status display unit 812 displays a message informing the user of successful authentication using the display device 214. For example, the display device 214 may display an icon indicating successful authentication.

[0110] In S1211, the second authentication unit 806 determines whether the user is continuing to take pictures. The determination of whether the user is continuing to take pictures can be performed in the same manner as in S1201. If the second authentication unit 806 determines that the user is continuing to take pictures, the determination process of S1211 is repeated. Otherwise, the process proceeds to S1212.

[0111] In both S1212 and S1213, the state management unit 810 invalidates the second authentication state. For example, the state management unit 810 can update the "second authentication state" in the authentication state table shown in FIG. 8E to "unauthenticated."

[0112] In S1214, the detection unit 807 performs processing to detect suspicion of use by another person. That is, the detection unit 807 determines whether or not the camera 100 is suspected to be used by a person other than the user determined to be registered in the first authentication processing. Details of this processing will be described later with reference to FIG. 13.

[0113] In S1215, the invalidation unit 808 determines whether the detection unit 807 detected suspicion of use by another person in S1214. If the invalidation unit 808 determines that suspicion of use by another person has been detected, the process proceeds to S1216. Otherwise, the process proceeds to S1217.

[0114] In S1216, the invalidation unit 808 performs processing to invalidate the first authentication status. For example, the invalidation unit 808 can control the status management unit 810 to update the "first authentication status" in the authentication status table shown in FIG. 8E to "unauthenticated." The invalidation unit 808 can also control the status management unit 810 to delete the person ID recorded in the authentication status table. By performing such processing, the first authentication status can be invalidated when use by another person is suspected.

[0115] In S1217, the status display unit 812 displays a message informing the user of the authentication failure using the display device 214 or the like. For example, the display device 214 may display an icon indicating the authentication failure. Note that, when the detection unit 807 detects suspicion of use by another person, the status display unit 812 may notify the user that suspicion of use of the device (camera 100 in this example) by a different person has been detected.

[0116] In S1218, the status display unit 812 updates the display on the display device 214. If image capture is no longer in progress, there is no need to display the authentication status. On the other hand, if image capture is in progress but the first authentication status has been disabled, the display device 214 can display a message, an icon, or the like indicating that the first authentication status has been disabled.

[0117] [Detection of use by another person] The detection of use by another person performed in this embodiment will be described with reference to the flowchart in Fig. 13. This process is mainly performed by the detection unit 807. The CPU 212 can realize the operation of the detection unit 807. This process is called from S1214 in Fig. 12B. Each step of this process will be described below.

[0118] In S1301, the detection unit 807 records the result of the second authentication process. In this embodiment, the unauthorized use detection process is performed when the second authentication process fails. Therefore, the detection unit 807 can record a history of failed second authentication processes. For example, the detection unit 807 can record the time of failure or the similarity between feature vectors calculated in the second authentication process. The detection unit 807 may record the maximum similarity among the similarities calculated for each of the multiple registered feature vectors.

[0119] In S1302, the detection unit 807 detects the suspicion of another person's use of the camera 100 based on the failure history recorded in S1301. The detection unit 807 can determine whether the failure history satisfies a predetermined condition. Examples of the predetermined condition have already been described. The detection unit 807 may, for example, detect the suspicion of another person's use of the device when a predetermined number of consecutive authentication failures occur. Furthermore, the detection unit 807 may detect the suspicion of another person's use of the camera 100 when, for example, the number of failures in the second authentication process within a predetermined time range up to the current time exceeds a threshold. The detection unit 807 may count multiple failures occurring during the same capture as a single failure. Furthermore, the detection unit 807 may count only the number of failures in which the similarity is below a predetermined threshold. Furthermore, the detection unit 807 may record the number of successes in the second authentication process in addition to the number of failures. For example, immediately after S1209, the detection unit 807 may record the success of the second authentication process. At this time, if the second authentication process is successful at least once during the same photographing, the detection unit 807 does not need to count failures during this photographing.

[0120] In S1303, the detection unit 807 determines whether or not the detection unit 807 detected suspicion of use of the camera 100 by another person in S1302. If the detection unit 807 detected suspicion of use by another person, the process proceeds to S1306. Otherwise, the process proceeds to S1304.

[0121] In S1304, the detection unit 807 determines whether the maximum similarity between the feature vectors obtained in S1207 is equal to or less than a predetermined threshold. In S1207, similarities are calculated for each of the multiple registered feature vectors associated with the same person ID. The detection unit 807 obtains the highest similarity among these similarities and determines whether this similarity is equal to or less than a predetermined threshold. Under poor shooting conditions, the calculated similarity decreases even when the subject's eye image is captured. However, the similarity calculated using the subject's eye image tends to be higher than when a different person's eye image is used. Therefore, a threshold can be set in advance to determine whether the image is a different person's eye image. Then, based on the similarity being equal to or less than the set threshold, the detection unit 807 determines that a different person is using the camera 100. If the detection unit 807 determines that the similarity is equal to or less than the threshold, the process proceeds to S1306. Otherwise, the process proceeds to S1305.

[0122] In S1305, the detection unit 807 records that there is no suspicion of use of the camera 100 by another person. Specifically, the detection unit 807 can turn off a flag recorded in the memory unit 213 indicating suspicion of use by another person.

[0123] In S1306, the detection unit 807 records that there is a suspicion that the camera 100 is being used by another person. Specifically, the detection unit 807 can turn on a flag recorded in the memory unit 213 indicating the suspicion of use by another person.

[0124] [First Authentication Disabling Process] The first authentication disabling process performed in this embodiment will be described with reference to the flowcharts of FIGS. 14A-14D. This process is mainly performed by the disabling unit 808. The CPU 212 can realize the operation of the disabling unit 808. As described above, the disabling unit 808 can set the first authentication status to invalid in accordance with predetermined conditions. Examples of the predetermined conditions have already been described. Below, as specific examples, processes according to each of four types of conditions will be described.

[0125] First, the first authentication invalidation process based on elapsed time will be described with reference to the flowchart of Fig. 14A. This process can be started periodically using a timer or the like.

[0126] In S1401, the invalidation unit 808 determines whether the first authentication status is valid. Specifically, the invalidation unit 808 can determine whether the "first authentication status" in the authentication status table shown in FIG. 8E is "authenticated" via the status management unit 810. If the invalidation unit 808 determines that the first authentication status is valid, the process proceeds to S1402. Otherwise, the process ends.

[0127] In S1402, the invalidation unit 808 calculates the elapsed time since the first authentication process was successful. The invalidation unit 808 can calculate the elapsed time since the "first authentication status" in the authentication status table was updated to "authenticating". The status management unit 810 can record the time when the "first authentication status" was updated to "authenticating". The invalidation unit 808 can calculate the difference between the time when the "first authentication status" was updated to "authenticating" and the current time as the elapsed time.

[0128] In S1403, the invalidation unit 808 detects the execution of the second authentication process. The invalidation unit 808 can detect whether the second authentication process has been executed between the previous first authentication invalidation process according to a timer or the like and the current first authentication invalidation process. If the invalidation unit 808 detects the execution of such second authentication process, the process proceeds to S1404. Otherwise, the process proceeds to S1407.

[0129] In S1404, the invalidation unit 808 determines whether the second authentication process detected in S1403 was successful. If the invalidation unit 808 determines that the second authentication process was successful, the process proceeds to S1405. Otherwise, the process proceeds to S1406.

[0130] In step S1405, the invalidation unit 808 adds a predetermined time to the lifetime. Note that the initial value of the lifetime is initialized to a predetermined value when the state management unit 810 updates the "first authentication state" to "authenticating."

[0131] In S1406, the invalidation unit 808 subtracts a predetermined time from the lifetime.

[0132] In S1407, the invalidation unit 808 determines whether the elapsed time calculated in S1402 has exceeded the lifetime. If the invalidation unit 808 determines that the elapsed time has exceeded the lifetime, the process proceeds to S1408. Otherwise, the process ends.

[0133] In S1408, in accordance with the instruction from the invalidation unit 808, the state management unit 810 updates the "first authentication state" in the authentication state table shown in Fig. 8E to "unauthenticated." At this time, the state management unit 810 can delete the person ID recorded in the authentication state table. In addition, the state management unit 810 can update the "second authentication state" in the authentication state table to "unauthenticated."

[0134] Next, the first authentication invalidation process based on a change in the power state will be described with reference to the flowchart in Fig. 14B. This process can be started when the power state of the camera 100 changes. For example, this process can be started when the camera 100 is turned on, turned off, goes into sleep mode, or returns from sleep mode.

[0135] In S1411, the invalidation unit 808 determines whether the first authentication status is valid. Specifically, the invalidation unit 808 can determine whether the "first authentication status" in the authentication status table shown in FIG. 8E is "authenticated" via the status management unit 810. If the invalidation unit 808 determines that the first authentication status is valid, the process proceeds to S1412. Otherwise, the process ends.

[0136] In S1412, the invalidation unit 808 determines whether the power state of the camera 100 has changed from ON to OFF or from ON to OFF. If the invalidation unit 808 detects such a change in the power state of the camera 100, the process proceeds to S1414. Otherwise, the process proceeds to S1413. Note that the invalidation unit 808 may determine whether there has been only one of a change from ON to OFF or a change from OFF to ON. Furthermore, if the invalidation unit 808 determines that at least one of a change from ON to OFF and a change from OFF to ON has occurred, the process may proceed to S1414.

[0137] In S1413, the invalidation unit 808 determines whether the camera 100 has transitioned to sleep mode or has returned from sleep mode. If the invalidation unit 808 detects such a change in the sleep mode of the camera 100, the process proceeds to S1414. Otherwise, the process ends. Note that the invalidation unit 808 may determine whether only one of a transition to sleep mode or a return from sleep mode has occurred. Alternatively, if the invalidation unit 808 determines that at least one of a transition to sleep mode and a return from sleep mode has occurred, the process may proceed to S1414.

[0138] In S1414, in accordance with the instruction from the invalidation unit 808, the state management unit 810 updates the "first authentication state" in the authentication state table shown in Fig. 8E to "unauthenticated." At this time, the state management unit 810 can delete the person ID recorded in the authentication state table. In addition, the state management unit 810 can update the "second authentication state" in the authentication state table to "unauthenticated."

[0139] Next, the first authentication invalidation process based on a change in the power state due to a change in the distance or connection status with another device will be described with reference to the flowchart in FIG. 14C. This process can be started periodically using a timer or the like. It is assumed that the camera 100 is already connected to a specific device that the user is carrying. As an example, a pairing process may be performed in advance so that the smartphone and camera 100 can be connected via Bluetooth.

[0140] In S1421, the invalidation unit 808 determines whether the first authentication status is valid. Specifically, the invalidation unit 808 can determine whether the "first authentication status" in the authentication status table shown in FIG. 8E is "authenticated" via the status management unit 810. If the invalidation unit 808 determines that the first authentication status is valid, the process proceeds to S1402. Otherwise, the process ends.

[0141] In S1422, the invalidation unit 808 determines whether the connection status with the specific device has deteriorated beyond a predetermined condition. The predetermined condition may be, for example, that the radio wave strength of the connection falls below a certain threshold. If the invalidation unit 808 determines that the connection status has deteriorated, the process proceeds to S1423. Otherwise, the process ends.

[0142] In S1423, in accordance with the instruction from the invalidation unit 808, the state management unit 810 updates the "first authentication state" in the authentication state table shown in Fig. 8E to "unauthenticated." At this time, the state management unit 810 can delete the person ID recorded in the authentication state table. In addition, the state management unit 810 can update the "second authentication state" in the authentication state table to "unauthenticated."

[0143] Next, the first authentication deactivation process based on an explicit deactivation operation input by the user will be described with reference to the flowchart in Fig. 14D. This process can be executed continuously while the camera 100 is activated.

[0144] In S1431, the invalidation unit 808 determines whether the first authentication status is valid. Specifically, the invalidation unit 808 can determine whether the "first authentication status" in the authentication status table shown in FIG. 8E is "authenticated" via the status management unit 810. If the invalidation unit 808 determines that the first authentication status is valid, the process proceeds to S1432. Otherwise, the process ends.

[0145] In S1432, the disabling unit 808 waits for a disabling operation by the user. The disabling operation may be, for example, the user pressing a switch button (not shown) installed on the camera 100. Alternatively, the disabling operation may be the selection of an item corresponding to the disabling operation in a menu displayed on a touch panel or the like. Examples of the disabling operation are not limited to these.

[0146] In S1433, the invalidation unit 808 determines whether or not a user's invalidation operation has been detected. If the invalidation unit 808 detects a user's invalidation operation, the process proceeds to S1434. Otherwise, the process ends.

[0147] In S1434, in accordance with the instruction from the invalidation unit 808, the state management unit 810 updates the "first authentication state" in the authentication state table shown in Fig. 8E to "unauthenticated." At this time, the state management unit 810 can delete the person ID recorded in the authentication state table. In addition, the state management unit 810 can update the "second authentication state" in the authentication state table to "unauthenticated."

[0148] [Authentication State Save Processing] The authentication state save processing in this embodiment will be described with reference to the flowchart in FIG. 15A. This processing is mainly performed by the state save unit 811. The CPU 212 can realize the operation of the state save unit 811. The authentication state save processing can be performed after the second authentication processing. Specifically, the authentication state save processing can be performed after a photographing operation by the camera 100. In this embodiment, the authentication state is saved along with the captured image taken by the user. Specifically, the state save unit 811 can generate an image file shown in FIG. 15B and save it in the memory unit 213. This processing can be started when the release button 101 is pressed down to the second stroke.

[0149] In S1501, the image capturing unit 813 captures an image. Specifically, the image capturing unit 813 performs an image capturing process of converting light received by the image sensor 211 into an electrical signal.

[0150] In S1502, the imaging unit 813 generates image data. Specifically, the imaging unit 813 generates image data 1530 by performing image processing such as development processing or encoding processing on the electrical signal obtained by the imaging processing in S1501.

[0151] In S1503, the state saving unit 811 generates photographer information 1521 about the photographer of the image data 1530. Specifically, the state saving unit 811 can acquire the authentication state table managed by the state management unit 810. The state saving unit 811 also acquires, from the registration management unit 803, personal information corresponding to the person ID indicated in the authentication state table. In this embodiment, the state saving unit 811 acquires "name" as the personal information. Note that if the person ID indicated in the authentication state table is "NULL," the state saving unit 811 can use "NULL" as the personal information (name). Then, the state saving unit 811 generates "name," "first authentication state," "second authentication state," and "whether or not another person has used the image" using the information included in the authentication state table. The state saving unit 811 generates photographer information 1521 including this information. Through this processing, the state saving unit 811 can record the user's personal information in association with the captured image in response to a successful second authentication process. Furthermore, the state saving unit 811 can control the saving operation so as not to record the user's personal information in association with the captured image in response to a failure in the second authentication process. Furthermore, the state saving unit 811 can control the recording in memory of information indicating that the detection unit 807 has detected that a different person has used the camera 100 in association with the captured image.

[0152] In S1504, the state saving unit 811 generates a hash value 1522 by applying a hash function to each of the image data 1530 and the photographer information 1521, which are binary data.

[0153] In S1505, the state saving unit 811 generates a digital signature 1523. The digital signature 1523 may include information indicating the signature value, the signer, and the signing date and time. The state saving unit 811 can generate the signature value by encrypting the hash value 1522 generated in S1504 using a private key prepared in advance. A public key paired with the private key used here can also be stored in the digital signature 1523. In this embodiment, information indicating the manufacturer of the camera 100 is stored as the signer. Note that, instead of information indicating the manufacturer, information indicating the model of the camera 100 may be used as the signer. Furthermore, the date and time at which the generation of the digital signature was completed is stored as the signing date and time.

[0154] In S1506, the state saving unit 811 assigns the photographer information 1521, the hash value 1522, and the digital signature 1523 to the image data 1530 as metadata 1520. In this way, the state saving unit 811 generates the image file 1510. If the image data 1530 represents a still image, the state saving unit 811 can generate the image file 1510 in JPEG format. If the image data 1530 represents a moving image, the state saving unit 811 can generate the image file 1510 in MPEG format.

[0155] In S1507, the state saving unit 811 records the image file 1510 in the memory unit 213. In this embodiment, the memory unit 213 is built into the camera 100. Alternatively, the memory unit 213 may be a storage medium that is physically detachably connected to the camera 100. In this case, the image file can be recorded in the storage medium. Furthermore, the camera 100 may be connected to a server having a memory via a wired and / or wireless communication path. In this case, the state saving unit 811 can transmit the image file 1510 to the server to record it in memory. For example, the state saving unit 811 can transmit a signal to the server so that the captured image and user information are associated and recorded in memory.

[0156] The following verification method can be used to confirm that a file has not been tampered with. First, a hash value is restored from the signature value using the public key. Next, the hash values ​​of the image data and photographer information are recalculated. If the restored hash value matches the recalculated hash value, it can be determined that the file has not been tampered with. On the other hand, if they do not match, it can be determined that the file has been tampered with.

[0157] This is because even if someone tries to tamper with the image data, the signature value is encrypted with a private key, so the person who tampered with the data cannot change the signature value. Therefore, if the image data has been tampered with, the hash value calculated from the image data will not match the restored hash value. Therefore, the above method can detect data tampering.

[0158] In the above embodiment, the hash value is stored in the image file. However, in this verification method, the hash value stored in the file is not used. Therefore, it is not necessary to store the hash value in the image file.

[0159] When shooting a video, video shooting can begin when the release button 101 is pressed down to the second stroke, and can end when the release button 101 is pressed down to the second stroke again. After that, video data is generated by the processing of S1501 to S1502. Furthermore, a video file is generated by the processing of S1503 and subsequent steps. In this case, the image data 1530 is video data, and the image file 1510 is a video file.

[0160] [Gaze Detection Processing] FIG. 5 is a diagram for explaining the principle of the gaze detection method. FIG. 5 shows a schematic diagram of an optical system for performing gaze detection. As shown in FIG. 5, light sources 216a and 216b are arranged approximately symmetrically with respect to the optical axis of light-receiving lens 218. Light sources 216a and 216b illuminate a user's eyeball 220. A portion of the light emitted from light sources 216a and 216b and reflected by eyeball 220 is collected on eye imaging element 219 via light-receiving lens 218. The upper part of FIG. 6 is a schematic diagram of an eye image captured by eye imaging element 219 (eye optical image projected onto eye imaging element 219). The lower part of FIG. 6 shows the output intensity of eye imaging element 219.

[0161] The gaze detection process performed in this embodiment will be described with reference to the flowchart in Fig. 7. In S701, the CPU 212 controls the light sources 216a and 216b via the light source drive circuit 305 to emit infrared light toward the user's eyeball 220. An optical image of the user's eye illuminated by the infrared light passes through the light receiving lens 218 and is formed on the eye image sensor 219. The eye image sensor 219 then photoelectrically converts the formed optical image of the eye. In this way, an electrical signal of the eye image that can be processed is obtained.

[0162] In S702 , the CPU 212 acquires an eye image (eye image signal; electrical signal of the eye image) from the eye imaging element 219 via the gaze detection circuit 301 .

[0163] Through the processing of S703 and S704, the CPU 212 acquires eye information relating to the position of the eyeball 220 relative to the finder from the eye image obtained in S702. In S703, the CPU 212 detects the coordinates of points corresponding to the corneal reflection images Pd and Pe of the light sources 216a and 216b and a point corresponding to the pupil center c from the eye image obtained in S702.

[0164] Infrared light emitted from light sources 216a and 216b illuminates cornea 501 of user's eyeball 220. At this time, corneal reflection images Pd and Pe formed by part of the infrared light reflected from the surface of cornea 501 are collected by light receiving lens 218 and formed on eye imaging element 219. In this way, corneal reflection images Pd' and Pe' in the eye image are obtained. Similarly, light beams from pupil edges 510a and 510b, which are the edges of pupil 502, are also formed on eye imaging element 219. In this way, pupil edge images a' and b' in the eye image are obtained.

[0165] The upper part of Figure 6 shows luminance information (luminance distribution) in region α of the eye image shown in the lower part of Figure 6. The horizontal direction of the eye image is the X-axis direction. The vertical direction of the eye image is the Y-axis direction. Figure 6 also shows the luminance distribution in the X-axis direction. The coordinates of the corneal reflection images Pd' and Pe' in the X-axis direction (horizontal direction) are designated Xd and Xe. The coordinates of the pupil edge images a' and b' in the X-axis direction are designated Xa and Xb. As shown in Figure 6, an extremely high level of luminance is obtained at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. Furthermore, in the region from coordinate Xa to coordinate Xb, which corresponds to the region of the pupil 502 (the region of the pupil image obtained when the light beam from the pupil 502 is focused on the eye image sensor 219), an extremely low level of luminance is obtained, except for coordinates Xd and Xe. A luminance intermediate between the two levels is obtained in the region of the iris 503 outside the pupil 502 (the region of the iris image outside the pupil image obtained by focusing the light beam from the iris 503). For example, a luminance intermediate between the two levels is obtained in a region where the X coordinate (coordinate in the X-axis direction) is greater than coordinate Xa and a region where the X coordinate is less than coordinate Xb.

[0166] Based on the luminance distribution shown in FIG. 6 , the coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the coordinates Xa and Xb of the pupil edge images a' and b' can be obtained. For example, the coordinates of extremely high luminance can be obtained as the coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. Furthermore, the coordinates of the outer edges of the region where luminance is extremely low can be obtained as the coordinates Xa and Xb of the pupil edge images a' and b'. When the rotation angle θx of the optical axis of the eyeball 220 relative to the optical axis of the light receiving lens 218 is small, the coordinate Xc of the pupil center image c' (center of the pupil image) obtained when the light beam from the pupil center 510c is focused on the eye imaging element 219 can be expressed as Xc ≒ (Xa + Xb) / 2. In other words, the coordinate Xc of the pupil center image c' can be calculated from the coordinates Xa and Xb of the pupil edge images a' and b'. By using this method, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupil center image c' can be calculated.

[0167] In S704, the CPU 212 calculates the imaging magnification β of the eye image. The imaging magnification β is determined by the position of the eyeball 220 relative to the light receiving lens 218. The imaging magnification β can be calculated as a function of the distance ΔP=Xe−Xd between the corneal reflection images Pd′ and Pe′.

[0168] In S705, the CPU 212 calculates the rotation angle of the optical axis of the eyeball 220 relative to the optical axis of the light receiving lens 218. The X coordinate of the midpoint of the corneal reflection images Pd and Pe and the X coordinate of the center of curvature 511 of the cornea 501 are approximately the same. Therefore, if the standard distance from the center of curvature 511 of the cornea 501 to the center c of the pupil 502 is Oc, the rotation angle θx of the eyeball 220 in the Z-X plane (plane perpendicular to the Y axis) can be calculated using the following formula (1). The rotation angle θy of the eyeball 220 in the Z-Y plane (plane perpendicular to the X axis) can also be calculated in the same way as the rotation angle θx. β×Oc×SINθx≈{(Xd+Xe) / 2}-Xc ... formula (1)

[0169] In S706, the CPU 212 reads the line-of-sight correction parameters stored in the memory unit 213. Specifically, the CPU 212 can read the parameters Ax, Bx, Ay, and By used in the following equations (2) and (3).

[0170] In S707, the CPU 212 estimates the user's gaze point on the screen of the display device 214 using the calculated rotation angles θx and θy. The coordinates (Hx, Hy) of the gaze point are defined as coordinates corresponding to the pupil center c. In this case, the coordinates (Hx, Hy) of the gaze point can be calculated using the following equations (2) and (3): Hx = m × (Ax × θx + Bx) (2) Hy = m × (Ay × θy + By) (3) In equations (2) and (3), the parameter m is a constant determined by the configuration of the optical system used to detect the gaze. The parameter m is a conversion coefficient that converts the rotation angles θx and θy into coordinates corresponding to the pupil center c on the screen of the display device 214. The parameter m is determined in advance and stored in the memory unit 213. The parameters Ax, Bx, Ay, and By are the gaze correction parameters described above.

[0171] The gaze correction parameter will now be described. For example, due to factors such as individual differences in the shape of the human eyeball, it may not be possible to estimate the gaze point with high accuracy. In this case, as shown in FIG. 4B , a discrepancy may occur between the actual gaze point 412 and the estimated gaze point 413. In FIG. 4B , the user is gazing at a person, but the camera 100 estimates that the user is gazing at the background. In this case, appropriate focus detection or adjustment may not be possible. The gaze correction parameter is a parameter for correcting such a discrepancy. The gaze correction parameter can be obtained by calibrating the gaze detection. Calibration can be performed, for example, by highlighting multiple indices at different positions on the screen of the display device 214, as shown in FIG. 4C . The user is instructed to look at these indices. A gaze detection operation is performed while the user is gazing at each indices. Then, the gaze correction parameter appropriate for the user can be calculated from the calculated positions of each gaze point (estimated positions) and the coordinates of each indices. Note that other indices may be used to indicate the position at which the user should gaze. For example, graphics may be displayed as the indices. The indicator may also be displayed by changing at least one of the brightness and color of the image (for example, the captured image).

[0172] [Effects of the Present Embodiment] By performing authentication at the time of shooting, it is possible to ensure that an image or video was taken by a specific person. On the other hand, to ensure that such authentication is not taken by someone else (that the person taking the photo is the person taking the photo), it is desirable to use settings that result in a low false acceptance rate. However, when using such settings, the false rejection rate often increases. In other words, even if the person taking the photo is the person taking the photo, there is a high possibility that authentication will fail at the time of shooting. In the use case of photography, there is no second opportunity to take the exact same photo. For example, the decisive moment in a sports or news scoop is fleeting. For professional photographers, the inability to authenticate at such a moment can be a problem. In other words, if authentication is not possible, it cannot be guaranteed that the person who captured the decisive moment is the photographer himself.

[0173] Therefore, in this embodiment, a first authentication process is performed when no image is captured. In the first authentication process, authentication is performed using settings that result in a low false acceptance rate. In addition, a second authentication process is performed when an image is captured. In the second authentication process, authentication is performed using settings that result in a low false rejection rate. According to this embodiment, it is possible to reduce the possibility that a false person will be authenticated as the true person by combining the first authentication process and the second authentication process, while reducing the possibility that the true person will not be authenticated in the second authentication process.

[0174] Note that the first authentication process uses settings that result in a low false acceptance rate, so the false rejection rate may be high. As a result, there is a possibility that the person in question will not be authenticated in the first authentication process. However, because the first authentication process can be performed when no photograph is being taken, authentication can be attempted again. Therefore, problems with use are unlikely to occur.

[0175] Furthermore, the second authentication process may increase the possibility of authenticating a different person as the user. On the other hand, in this embodiment, the first authentication state is invalidated according to various conditions. Such processing can reduce the possibility of authenticating a different person as the user. That is, if use by a different person is suspected in the second authentication process, the first authentication state is invalidated. Furthermore, the first authentication state can be invalidated based on the elapsed time since the first authentication process was successful, a change in the power state, a change in the distance or connection state to a peripheral device, or an explicit invalidation operation by the user. This configuration can reduce the possibility of use by a different person. Therefore, it is possible to further suppress authentication of a different person as the user in the second authentication process.

[0176] In this embodiment, the results of the first authentication process and the second authentication process are recorded separately in the metadata of the image file. This process ensures that even if the first authentication process is successful and the second authentication process fails, the success of the first authentication process is recorded in the metadata. Additionally, if the second authentication process is also successful, the success of both authentication processes is recorded so that it is clear. Recording many successful authentications in this way indicates the high degree of certainty that the photographer recorded took the photograph.

[0177] Furthermore, in this embodiment, information indicating the suspicion of use of the device (e.g., camera 100) by another person is recorded as metadata. With this configuration, if the first authentication process is successful but the second authentication process fails, it is possible to determine whether or not use by another person is suspected. By recording such information, it is possible to indicate the degree of certainty that the recorded photographer took the photograph.

[0178] [Modification] In the above embodiment, the registration management unit 803 managed only "name" as personal information. However, information other than name may be used as personal information. For example, if the camera is used within a company, an "employee number" assigned to an employee may be registered as personal information. Furthermore, account information such as an account name for any web service may be registered as personal information. In this case, if a user accesses and logs in to a web service, the account name may be registered as personal information. Furthermore, the web service may issue a token or the like upon successful access. In this case, the issued token can be registered as personal information. Furthermore, the state saving unit 811 may save this personal information as image metadata. The personal information used by the registration management unit 803 and the state saving unit 811 is not limited to these.

[0179] In the above embodiment, the first authentication unit 805 and the second authentication unit 806 use the same feature calculation unit 802 to perform authentication. Furthermore, the feature calculation unit 802 calculates features using the same process for the first authentication process and the second authentication process. However, the eye images used by the first authentication unit 805 for authentication and the eye images used by the second authentication unit 806 for authentication tend to differ from each other. For example, the eye images used by the first authentication unit 805 for authentication are captured while the user is aware of being authenticated. Therefore, eye images in which the photographer's eyes are wide open are often used. On the other hand, the eye images used by the second authentication unit 806 for authentication are captured while the user is taking a photo. Therefore, a wide variety of eye images, for example, eye images at various gaze angles, are likely to be used. Therefore, the neural network that calculates the features used by the first authentication unit 805 can be trained using eye images expected to be used in the first authentication process. On the other hand, the neural network that calculates the features used by the second authentication unit 806 can be trained using eye images expected to be used in the second authentication process. Such a configuration can further improve authentication accuracy.

[0180] Note that methods other than changing the threshold or model may be used to lower the false acceptance rate in the first authentication process and the false rejection rate in the second authentication process. For example, as described in Patent Document (JP 2022-182960 A), a method of improving recognition performance by changing the calculation method of feature vectors used during enrollment and verification may be used. In this way, the authentication methods used in the first authentication process and the second authentication process are not limited to the specific examples described above.

[0181] In the above embodiment, in the second authentication process, the status display unit 812 displays the authentication result on the display device 214 (S1210, S1217, and S1218). However, during shooting, the display device 214 displays the image captured by the image sensor 211. Therefore, the status display unit 812 may display the authentication result on the display device 214 as follows. For example, the display device 214 may display a display indicating whether the authentication was successful or failed at the edge of the screen. This display method reduces the likelihood that the display of the authentication result will interfere with shooting, allowing the user to concentrate more easily on shooting. Alternatively, the display position of the authentication result may be determined based on the user's gaze position. For example, the display device 214 may display the authentication result at a position far from the gaze position. The user's gaze position on the display device 214 can be obtained by the gaze detection process (S707) already described. As another example, multiple display positions, such as near the four corners of the display device 214, may be determined in advance. The display device 214 can then determine, from among the plurality of display positions, which position to display the authentication result at, based on the gaze position. For example, the display device 214 can display an icon or the like at the display position farthest from the gaze position among the plurality of display positions. With this configuration, the display position is fixed to a certain extent, making it easier for the user to understand the authentication result. However, the method of displaying the authentication result is not limited to the above example. Also, the authentication result does not need to be displayed. In this case, S1204 can be omitted.

[0182] In the above embodiment, only the second authentication status is displayed in S1210 and S1217. However, the first authentication status may be displayed at the same time. Similarly, both the first authentication status and the second authentication status may be displayed in S1218. Furthermore, in the above embodiment, these authentication statuses were not displayed when the second authentication process was started. However, when the second authentication process was started, for example, when the user looked through the viewfinder, the authentication status may already be displayed on the display device 214. The display of these authentication statuses can be performed by controlling the status display unit 812.

[0183] In the above embodiment, both the first authentication unit 805 and the second authentication unit 806 performed personal authentication using eye images. However, other authentication methods may be used. For example, other biometric authentication methods such as facial authentication, fingerprint authentication, or voice authentication may be used in the first authentication process. When performing facial authentication, a face-capturing camera may be installed on the back of the camera 100 to capture a user's facial image. For example, a face-recognition camera may be installed above the touch panel (operation member 103) to perform authentication when the user looks into the touch panel. Such a camera can be used to acquire a user's facial image. Then, authentication can be performed using such a facial image. When performing fingerprint authentication, a fingerprint sensor may be installed on the release button 101. Then, when the user places his / her finger on the release button, fingerprint authentication can be performed based on information acquired by the fingerprint sensor. When performing voice authentication, a microphone may be installed on the camera 100. Then, voice authentication can be performed based on a voiceprint extracted from the acquired voice.

[0184] Furthermore, personal authentication may be performed using a method other than biometric authentication. For example, personal authentication may be performed based on a password or PIN code entered by the user. Authentication may also be performed based on the connection status with another device. For example, the camera 100 and a smartphone may be paired via a Bluetooth connection. Then, when the camera 100 is connected to the smartphone via Bluetooth, the first authentication status may be considered to be valid. Authentication may also be performed using a mechanism such as FIDO authentication.

[0185] Other authentication methods may also be used for the second authentication process. Meanwhile, in the above-described embodiment, the second authentication process is performed during image capture. Therefore, it is convenient to use an authentication method that can be used during image capture for the second authentication process. For example, by using a fingerprint sensor installed on the release button 101, fingerprint authentication can be performed as the second authentication process during image capture. Furthermore, the user may perform image capture while viewing an image captured by the image sensor 211 displayed on the touch panel (operation member 103) without looking through the viewfinder. In this case, facial authentication can be performed as the second authentication process during image capture by using the user's facial image. Multiple authentication methods may also be used in combination. For example, the first authentication is determined to be successful when multiple authentication methods are used and authentication is successful with all of them. On the other hand, the second authentication is determined to be successful when authentication is successful with any one of the authentication methods. This eliminates the possibility of false acceptance in the first authentication. Additionally, the second authentication reduces the possibility of false rejection.

[0186] In the above embodiment, the second authentication state is controlled to be valid only during image capture. When facial authentication is used, image capture can be considered to be in progress while a face is being captured. When fingerprint authentication is used, image capture can be considered to be in progress while a finger is placed on the release button. However, the authentication method and the method of determining whether image capture is in progress are not limited to the above examples.

[0187] In the above embodiment, the invalidation unit 808 invalidated the first authentication status in accordance with a predetermined condition. On the other hand, after the first authentication status is invalidated, another person may repeat the first authentication process. In this case, there is a possibility that the first authentication process will be successful and the first authentication status will be invalidated repeatedly.

[0188] Therefore, the camera 100 may include a first authentication restriction unit (not shown). The first authentication restriction unit detects suspicion of unauthorized use by another person. Then, the first authentication restriction unit can temporarily restrict execution of the first authentication process in response to detecting the suspicion of unauthorized use. The first authentication restriction unit can detect the suspicion of unauthorized use in response to a predetermined condition. For example, the first authentication restriction unit can restrict the first authentication process by the first authentication unit 805 based on the number of times the first authentication unit 805 has set the first authentication state to invalid. That is, the predetermined condition may be that the first authentication process by the first authentication unit 805 is successful and the first authentication state is invalidated by the invalidation unit 808, both of which occur repeatedly within a predetermined period of time. Alternatively, some of the above-described first authentication invalidation determination conditions may be used as the predetermined condition. For example, the predetermined condition may be limited to the detection unit 807 detecting suspicion of unauthorized use of the camera 100 by another person. Alternatively, the predetermined condition may be that the invalidation of the first authentication state due to suspicion of unauthorized use by another person and the success of the first authentication process are repeated within a predetermined period of time. However, the method for detecting the suspicion of unauthorized use is not limited to the above-described method.

[0189] One method for restricting the execution of the first authentication process is to disable the user's operation to start the first authentication process. For example, when the user operates the camera 100 and calls the first authentication process from a menu, the item that calls the first authentication process can be disabled. Furthermore, after a certain period of time has passed, this state of restriction on the execution of the first authentication process can be lifted. However, the methods for restricting the execution of the first authentication process and lifting the restriction are not limited to the above methods.

[0190] This technique makes it difficult for a registered person to intentionally hand over the camera to another person and have that person take a photo. In particular, when an authentication process other than biometric authentication is used as the first authentication process, the first authentication process may be successful even when the registered person is not present. For example, the first authentication process can be successful by sharing the password or PIN used in the first authentication process or by handing over a smartphone paired with the camera 100 to another person. On the other hand, implementing the above-mentioned restrictions can prevent fraudulent use.

[0191] In the above embodiment, the second authentication process was performed when a predetermined operation, such as a photographing operation, was performed. Alternatively, information necessary for the second authentication process may be saved when a predetermined operation is performed, and the second authentication process may be performed thereafter. For example, an image of the user's eyes may be saved during photography, and the second authentication process may be performed using the saved eye image after photography. Even when using other types of biometric authentication (such as face authentication or fingerprint authentication), biometric information (such as face or fingerprint) may be saved and used for the second authentication process after photography. Furthermore, even when using the connection status with other devices, parameters of the connection status may be recorded during photography, and the connection status may be analyzed and authentication performed after photography. In this way, the second authentication process does not necessarily need to be performed during photography. Information necessary for the second authentication process may be acquired during photography, and authentication may be performed after photography, etc. This type of processing may be adopted when it is difficult to perform the second authentication process during photography due to resource constraints, such as speed.

[0192] If the second authentication process is performed later, the authentication process may not be completed in time for saving the image file. In this case, a state other than "authenticated" and "unauthenticated," such as "processing," can be defined as the second authentication status. If the second authentication process is not yet complete, information indicating "processing" can be saved as metadata. Then, when the result of the second authentication process is obtained, the metadata can be modified to indicate the result of the second authentication process.

[0193] In one embodiment, biometric authentication is used for the second authentication process. It is difficult for the user to enter a password or the like when taking a photo. Furthermore, if the user hands over the smartphone along with the camera to another person, the other person can pass authentication based on the connection status with other devices. On the other hand, biometric authentication can be performed without any special action for authentication. Furthermore, the biometric information used for biometric authentication can only be held by the person himself / herself. For this reason, in the above embodiment, biometric authentication is used as the second authentication process performed when taking a photo.

[0194] In the above embodiment, the registration management unit 803 manages the first authentication registration feature vector table and the second authentication registration feature vector table as separate tables. However, these tables may be integrated into a single table. This configuration can reduce duplication of information such as "feature vector 1" and improve memory efficiency. In this case, information indicating whether a record related to each registration feature vector is to be used in the first authentication process or the second authentication process can be recorded in the table. Based on this information, the registration feature vector to be used in the first authentication process and the second authentication process can be selected.

[0195] In the above embodiment, different registered feature vector groups are used in the first authentication process and the second authentication process. However, feature vector groups may be used in these processes. In this case, the registration management unit 803 does not need to manage a first authentication registered feature vector table and a second authentication registered feature vector table. The registration management unit 803 can manage feature vectors in a single table. However, the data management method in the registration management unit 803 is not limited to the above example.

[0196] In the above embodiment, one person is registered as the user of the camera 100. Therefore, when a different person is to be registered, the data held by the registration management unit 803 is erased and the registration is restarted. For example, when the user registration process is started, the data managed by the registration management unit 803 is deleted. Alternatively, when the user registration process is started, an invalid flag may be set for the data managed by the registration management unit 803. Data with an invalid flag attached will not be used in the subsequent first and second authentication processes.

[0197] In the above embodiment, the device (e.g., camera 100) includes a first authentication unit 805 that authenticates the user of the device. The device also includes a second authentication unit 806 that authenticates the user of the device when performing a predetermined operation using the device. The device also includes an execution unit 809 that controls the operation of the device depending on whether the same person is authenticated in both the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806. For example, if the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806 fails to authenticate the same person, the execution unit 809 can control the operation of the device to record information indicating the authentication failure. The execution unit 809 can also control the operation of the device to perform an operation to store information indicating that the same person is authenticated in both the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806. Furthermore, the execution unit 809 can notify, via the status display unit 812 or the like, that the same person has been authenticated both by the first authentication unit 805 and by the second authentication unit 806 .

[0198] In the above embodiment, when the first authentication status is valid, the second authentication unit 806 performs the second authentication process. Furthermore, the second authentication unit 806 can determine that the user of the device is the same person as the user determined by the first authentication unit 805 to be registered. This means that the same person was authenticated in both the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806. In response to this determination, the execution unit 809 stores the image data in the memory unit 213 together with metadata indicating that the second authentication status was valid. In response to this determination, the execution unit 809 also stores the image data in the memory unit 213 together with metadata including personal information of the authenticated user. On the other hand, a failure in the second authentication process means that the same person could not be authenticated in the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806. In such a case, the execution unit 809 can record information indicating that the authentication process failed. For example, in the above embodiment, the execution unit 809 stored the image data in the memory unit 213 together with metadata indicating that the second authentication status was invalid. In such a case, the execution unit 809 can store the image data in the memory unit 213 together with metadata including "NULL" as personal information. In this manner, the execution unit 809 can store the image data together with different metadata depending on whether the same person is authenticated in both the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806. Incidentally, the method for determining whether the user of the device is the same person as the user determined to be registered by the first authentication unit 805 is not limited to the above example.

[0199] For example, multiple users may be registered for the camera 100. In this case, each time the user registration process is performed, a different person ID and registration information corresponding to the person ID (e.g., personal information and feature vector) are registered in the registration management unit 803. At this time, additional processing may be performed to prevent duplicate registration by the same person. For example, if a name identical to an already registered name is entered during the user registration process, a notification may be sent that the user is already registered, and the process may end.

[0200] In this case, the first authentication process can determine which user corresponding to which person ID is using the camera 100. The first authentication unit 805 may identify the user based on a comparison of feature vectors. For example, in S1107, the first authentication unit 805 may acquire all feature vectors for all person IDs registered in the first authentication registered feature vector table. In addition, in S1108, the first authentication unit 805 may compare each of the multiple registered feature vectors with the feature vector obtained in S1106. Here, the similarity calculated for the multiple registered feature vectors may exceed a predetermined threshold. In this case, the first authentication unit 805 may identify the person ID for the registered feature vector that is most similar to the feature vector obtained in S1106. The first authentication unit 805 may determine that the user corresponding to the identified person ID is using the camera 100. Note that if the similarity calculated for the multiple registered feature vectors exceeds a predetermined threshold, the first authentication unit 805 may determine that the first authentication process has failed.

[0201] In this case, the registered feature vectors used in the second authentication process may be limited to those corresponding to the user identified in the first authentication process. That is, in S1206, the second authentication unit 806 can obtain only the record for the person ID identified in S1108 from the second authentication registered feature vector table. In S1207, the second authentication unit 806 can only compare the registered feature vectors included in the obtained record with the feature vector obtained in S1205. If a user is identified in a single-stage authentication process, each of multiple registered feature vectors is compared with the feature vector obtained for the user. On the other hand, the above configuration can reduce the number of registered feature vectors compared in the second authentication process. This can improve authentication accuracy. This is because the problem can be simplified from so-called 1:N identification to 1:1 verification. Furthermore, reducing the number of registered feature vectors to be compared can shorten processing time.

[0202] In the above embodiment, a pre-registered registered feature vector is used in the second authentication process. Alternatively, biometric information obtained when the first authentication process is successful may be used in the second authentication process. For example, feature vectors of eye images obtained when the first authentication process is successful, or before and after, can be recorded. In the second authentication process, the recorded feature vector can be used instead of the registered feature vector. That is, in the second authentication process, authentication may be performed by confirming the identity of the user's feature obtained when the first authentication process is successful and the user's feature obtained during the second authentication process (verification process). This method eliminates the need to register the feature for the second authentication process. This method is particularly effective when the second authentication process is performed in an environment different from that at the time of feature registration. This identity confirmation process may also be combined with the comparison process with registered feature data, as described above. In this case, the second authentication process is successful if either method of matching is successful (or if both methods of matching are successful). This method can reduce the false rejection rate or the false acceptance rate.

[0203] As another application of this authentication method, the following method can be adopted. In the first authentication process, authentication by PIN input and fingerprint authentication via the release button are performed. If the first authentication process is successful, a facial image of the user is simultaneously captured by the in-camera. Furthermore, the facial image is converted into feature quantities, and the obtained feature quantities are recorded. In the second authentication process, the identity of the feature quantities of the recorded facial image and the facial features of a re-acquired facial image of the user is confirmed. In this way, various forms can be adopted as a two-step authentication method.

[0204] On the other hand, it is not essential for the second authentication unit 806 to determine whether the device user is the same person as the user determined to be registered by the first authentication unit 805. For example, the second authentication unit 806 may authenticate the device user independently of the first authentication unit 805.

[0205] Furthermore, the second authentication unit 806 does not necessarily perform the second authentication process when the first authentication status is valid. For example, there may be cases where the first authentication process cannot be performed, such as when a photograph is suddenly required. Also, the user may forget to perform the first authentication process. Therefore, the second authentication process may be performed while the first authentication status is invalid. Specifically, in S1201, the process can be performed so that the process proceeds to S1202 not only when the first authentication status is valid, but also when the first authentication status is invalid.

[0206] Furthermore, when the first authentication status is invalid, a value indicating an empty state, such as NULL, is recorded in the person ID in the authentication status table shown in Fig. 8E. Therefore, when the second authentication process is successful while the first authentication status is invalid, in S1209, the person ID in the authentication status table can be updated so as to record the person ID identified in S1207.

[0207] In this case, in the authentication state saving process, "unauthenticated" is recorded as the "first authentication state" and "authenticating (authentication successful state)" is recorded as the "second authentication state" in the metadata. In this case, the person ID may be recorded in the metadata so that it is clear that the person was identified in the second authentication process. For example, the person ID identified in the first authentication process and the person ID identified in the second authentication process can be described as different items in the metadata.

[0208] Note that, when multiple people are registered in the camera 100, the second authentication unit 806 can acquire in S1206 only the registered feature vector for the person ID identified in the first authentication process, as described above. On the other hand, when the first authentication status is invalid, the second authentication unit 806 can acquire in S1206 the registered feature vectors for all person IDs recorded in the second authentication registered feature vector table. Furthermore, the second authentication unit 806 can perform matching in S1207 for each registered feature vector. In this case, the second authentication unit 806 can identify the person ID for the registered feature vector that is most similar to the feature vector obtained in S1205. The second authentication unit 806 can then determine that the user corresponding to the identified person ID is using the camera 100. Note that, when the similarity calculated for the multiple registered feature vectors exceeds a predetermined threshold, the second authentication unit 806 may determine that the second authentication process has failed.

[0209] In S1207, the second authentication unit 806 may use different similarity thresholds depending on whether the first authentication state is valid or invalid. The second authentication unit 806 may also use different authentication settings depending on whether the first authentication state is valid or invalid. For example, the second authentication unit 806 can change the model to be used. While 1:1 authentication and 1:N authentication present different problems, such a configuration allows the use of settings appropriate for each type of authentication.

[0210] In the above embodiment, the second authentication process is performed according to the person ID identified in the first authentication process. That is, in the second authentication process, it is determined whether the user is the same person as the user determined to be registered in the first authentication process. However, the person ID may be identified independently in the second authentication process without using the result of the first authentication process.

[0211] In this case, the "person ID of the first authentication process" and the "person ID of the second authentication process" can be recorded separately in the authentication state table of FIG. 8E managed by the state management unit 810. In this case, the person ID identified in the first authentication process and the person ID identified in the second authentication process can be recorded in their respective fields. Furthermore, in S1206, the second authentication unit 806 can acquire registered feature vectors for all person IDs recorded in the second authentication registered feature vector table. Furthermore, as described above, the second authentication unit 806 can identify the person ID of the person using the camera 100 by comparing each registered feature vector in S1207.

[0212] In this way, the first authentication process and the second authentication process may be performed independently. In this case, the execution unit 809 can control the operation of the device depending on whether the same person is authenticated in both the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806. For example, the execution unit 809 can store image data together with different metadata depending on whether the same person is authenticated in both the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806.

[0213] Furthermore, the state saving unit 811 can save authentication states such as the "person ID of the first authentication process," "person ID of the second authentication process," "first authentication state," and "second authentication state" separately in metadata. With this configuration, when using an image file, it is possible to confirm that the same person was authenticated in the first authentication process and the second authentication process, and to confirm that this person took the photograph.

[0214] Furthermore, if the "person ID of the first authentication process" and the "person ID of the second authentication process" are different, the state saving unit 811 may record "unauthenticated" as the "second authentication state." In this case, the state saving unit 811 may save personal information related to the "person ID of the first authentication process" as personal information (e.g., name) in the metadata. In this way, the process of checking whether the same person has been authenticated in the first authentication process and the second authentication process may be performed when the metadata is saved.

[0215] In the authentication state saving process in the above embodiment, the state saving unit 811 saves all of the contents of the authentication state table managed by the state management unit 810 as metadata. However, the state saving unit 811 does not need to save all of this content. Furthermore, the state saving unit 811 may process the contents of the authentication state table before saving. For example, the state saving unit 811 may save only the "name." In this case, the state saving unit 811 may save the "name" corresponding to the "person ID" only when both the "first authentication state" and the "second authentication state" are "authenticating (authentication successful state)." In other cases, the state saving unit 811 may save another value (for example, a value indicating "unknown") as the "name." In this case, the fact that the "name" is saved indicates that authentication was successful (i.e., both the "first authentication state" and the "second authentication state" are "authenticating").

[0216] As another example, the first authentication status and the second authentication status do not need to be stored separately. For example, the status storage unit 811 may record one "authentication status" item. For example, only when both the "first authentication status" and the "second authentication status" are "authenticated," the status storage unit 811 may record "authenticated" in the "authentication status" item. On the other hand, the status storage unit 811 may record "unauthenticated" in the "authentication status" item in other cases. The information stored in the authentication status storage process is not limited to the above example.

[0217] The timing of the second authentication process is not particularly limited. In addition to when taking a photograph, the second authentication process or the acquisition of biometric information for the second authentication process can be performed when performing a predetermined operation using the device, such as viewing an image. The timing of the first authentication process is also not particularly limited. The first authentication process can be performed before a predetermined operation, such as when the power is turned on or during a login process.

[0218] Furthermore, the method of controlling the device operation by the execution unit 809 is not limited to controlling the writing of information to the memory unit 213 as described above. For example, the execution unit 809 may execute a specific process by the device depending on the result of the determination by the second authentication unit 806. Furthermore, the execution unit 809 may execute a specific process by the device depending on whether the same person is authenticated in both the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806. For example, the execution unit 809 may control whether or not the device operates. Specifically, the execution unit 809 may permit the camera 100 to capture an image when the second authentication status is valid. Furthermore, the execution unit 809 may prohibit the camera 100 from capturing an image when the second authentication status is invalid. As another example, the execution unit 809 may execute a login process depending on whether the same person is authenticated in both the authentication by the first authentication unit 805 and the authentication by the second authentication unit 806.

[0219] In the above embodiment, the first authentication process and the second authentication process are performed in the same device. However, the first authentication process and the second authentication process may be performed in different devices. For example, the functions of the information processing device according to the above embodiment may be realized by an information processing system. Such an information processing system may be configured, for example, by multiple information processing devices connected via a network. Such an information processing device may be realized by a computer including a processor and a memory. That is, each of the information processing devices can realize at least some of the functions shown in FIG. 8A by the processor executing a program stored in the memory.

[0220] An information processing system according to one embodiment includes a head-mounted display (HMD) and a rental device that rents out the HMD. The rental device stores multiple HMDs. If a first authentication process in the rental device is successful, the HMD is rented to a user. Then, when the user wears the HMD, a second authentication process is performed. If the second authentication process is successful, the user can log in to the HMD and use the HMD.

[0221] Authentication information for the first authentication process can be obtained via a web service or the like. Specifically, a user requests rental of an HMD from an HMD rental service online. The user also pays the usage fee online. The user also registers a facial image online. Thereafter, the rental device performs a first authentication process using facial recognition. If the first authentication process identifies the user as the person who previously requested rental of the HMD, the rental device unlocks the HMD. The user can then take the HMD out of the rental device. At the same time, authentication information for the second authentication process is registered. For example, the user looks into a camera attached to the rental device that captures eye images. In this way, authentication information for the second authentication process based on the captured eye image is registered in the HMD. After that, when the user puts on the HMD, an eye-capturing camera attached to the HMD captures an image of the user's eyes. Then, for the second authentication process, the HMD compares the feature amounts of the acquired eye image of the user with the feature amounts of pre-registered eye images. If the second authentication process is successful, the user is permitted to log in to the HMD. In this embodiment, too, the operation of the HMD is controlled in response to the fact that the same person is authenticated in both the first authentication process and the second authentication process.

[0222] In such an embodiment, an information processing system having a rental device and an HMD has the components shown in FIG. 8A . This information processing system may further include an information processing device such as a server. In this case, the functions shown in FIG. 8A may be distributed among the rental device, the HMD, and the information processing device. For example, the rental device and the HMD may be communicably connected to the information processing device. In this example, the information processing device may provide the above-mentioned web service. This information processing device may also manage the authentication status.

[0223] In this manner, the first authentication process and the second authentication process may be implemented in different devices. Furthermore, the device used in this embodiment is not limited to a camera. Furthermore, the process executed in accordance with the first authentication status and the second authentication status is not limited to recording the authentication status along with the captured image. As in this example, a process such as logging in to a device may be performed in accordance with the first authentication status and the second authentication status.

[0224] In the above embodiment, an authentication method with a low false acceptance rate is used in the first authentication process, and an authentication method with a low false rejection rate is used in the second authentication process. However, this is not limited to this configuration. For example, an authentication method with a low false rejection rate may be used in the first authentication process, and an authentication method with a low false acceptance rate may be used in the second authentication process. Specifically, in the above example, the rental device can perform a simple and fast first authentication process using a facial image. On the other hand, the second authentication process in the HMD can perform more strict iris authentication by having the user gaze at a specific location for a certain period of time. Furthermore, the first authentication process and the second authentication process can be performed in the HMD. In this case, a fast first authentication process using an image of the eyes or face can be performed for logging in. Furthermore, the subsequent second authentication process can perform more strict iris authentication.

[0225] According to the above embodiment, user authentication is performed by combining the first authentication process and the second authentication process. Therefore, the settings of the second authentication process can be adjusted to improve the usability of the device. On the other hand, by using the first authentication process and the second authentication process in combination, an increase in the false acceptance rate due to adjustment of the settings of the authentication processes can be suppressed.

[0226] The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

[0227] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0228] This application claims priority based on Japanese Patent Application No. 2024-059142, filed April 1, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. An image capturing device comprising: an acquisition means for acquiring biometric information of a person; a first authentication means for determining whether a user of the image capturing device is a person who has been registered in advance; a second authentication means for determining whether the user of the image capturing device is the same person as a user determined to be registered by the first authentication means through biometric authentication processing using the biometric information acquired via the acquisition means; and a control means for controlling the recording of an image captured by the image capturing device in association with information about the user in memory according to the result of the determination by the second authentication means.

2. The photographing device according to claim 1, wherein the second authentication means performs the biometric authentication process when the photographing device performs a photographing operation.

3. The photographing device described in any one of claims 1 to 2, characterized in that the second authentication means performs the biometric authentication process using biometric information acquired via the acquisition means when the photographing device performs a photographing operation.

4. The photographing device according to any one of claims 1 to 3, wherein the second authentication means makes a determination using a method with a lower false rejection rate than the first authentication means.

5. An imaging device as described in any one of claims 1 to 4, characterized in that it is provided with a registration management means for associating and managing registration information used by the first authentication means to make a judgment about the user of the imaging device and registration information used by the second authentication means to make a judgment about the user of the imaging device.

6. The photographing device described in any one of claims 1 to 5, characterized in that the control means controls the captured image to be associated with information about the user and recorded in the memory when the second authentication means succeeds in the biometric authentication process.

7. An imaging device as described in any one of claims 1 to 6, further comprising a status management means for managing the first authentication status of the imaging device, wherein the status management means sets the first authentication status to valid in response to the first authentication means determining that the user has been registered.

8. The photographing device according to claim 7, wherein the second authentication means performs the biometric authentication process when it determines that the first authentication state is set to valid.

9. The photographing device according to any one of claims 7 to 8, further comprising a notification means for notifying the first authentication status on the photographing device.

10. The photographing device described in any one of claims 7 to 9, wherein the state management means manages the second authentication state of the photographing device, and the state management means sets the second authentication state to valid in response to the second authentication means succeeding in the biometric authentication process.

11. The photographing device according to claim 10, wherein the state management means sets the second authentication state to invalid upon completion of photographing using the photographing device.

12. The photographing device described in any one of claims 10 to 11, characterized in that the control means controls the recording of information indicating each of the first authentication state and the second authentication state in the memory in association with the photographed image.

13. The photographing device according to any one of claims 7 to 12, characterized in that the photographing device is provided with an invalidation means for setting the first authentication status to invalid in accordance with a predetermined condition.

14. The photographing device according to claim 13, wherein the invalidation means sets the first authentication status to invalid based on the amount of time that has elapsed since the first authentication status was set to valid.

15. The photographing device according to any one of claims 13 to 14, wherein the invalidation means sets the first authentication state to invalid based on a change in the power supply state of the photographing device.

16. The photographing device described in any one of claims 13 to 15, characterized in that the invalidation means sets the first authentication status to invalid based on the distance or connection status between the photographing device and another device.

17. The photographing device according to any one of claims 13 to 16, wherein the invalidation means sets the first authentication status to invalid based on an input by a user to the photographing device.

18. The photographing device described in any one of claims 13 to 17, characterized in that the invalidation means sets the first authentication status to invalid based on the detection of suspicion that the photographing device is being used by a person other than the user determined to be registered.

19. An imaging device as described in any one of claims 13 to 18, characterized in that it is provided with a limiting means for limiting the determination by the first authentication means based on the number of times the disabling means has set the first authentication status to invalid.

20. An imaging device as described in any one of claims 1 to 19, characterized in that it is provided with a detection means for detecting suspicion of use of the imaging device by a person other than the user determined to be registered by the first authentication means.

21. The photographing device according to claim 20, further comprising a notification means for notifying that the detection means has detected suspicion of use of the photographing device by the different person.

22. The photographing device described in any one of claims 20 to 21, characterized in that the control means controls the recording in the memory of information indicating that the detection means has detected suspicion of use of the photographing device by a different person, in association with the photographed image.

23. The photographing device according to any one of claims 20 to 22, wherein the detection means detects suspicion of use of the photographing device by a different person based on the number of times the biometric authentication process fails.

24. The photographing device described in any one of claims 20 to 23, characterized in that the second authentication means performs the biometric authentication process by comparing biometric information related to the user determined to be registered with biometric information acquired via the acquisition means, and the detection means detects suspicion of use of the photographing device by the different person based on the similarity of the biometric information compared by the second authentication means.

25. An imaging device according to any one of claims 1 to 24, characterized in that the memory is built into the imaging device or is physically detachably connected to the imaging device.

26. The photographing device described in any one of claims 1 to 25, characterized in that the photographing device is connected to a server having the memory via a wired and / or wireless communication path, and the control means transmits a signal to the server so that the photographed image and information about the user are associated and recorded in the memory.

27. A method of operating an imaging device, comprising the steps of: acquiring biometric information of a person; performing a first authentication process to determine whether the user of the imaging device is a person who has been registered in advance; determining, through biometric authentication processing using the acquired biometric information, whether the user of the imaging device is the same person as the user determined to be registered by the first authentication processing; and controlling, based on the result of the determination by the biometric authentication processing, so that an image captured by the imaging device and information about the user are associated and recorded in memory.

28. A program for causing a computer to function as the photographing device according to any one of claims 1 to 26.

Images