Photographing apparatus, and method for controlling photographing apparatus

Abstract

To provide a photographing apparatus that improves the accuracy of specifying a user of the photographing apparatus, and a method for manufacturing the photographing apparatus.SOLUTION: A photographing apparatus has: photographing means for photographing an eyeball image of a user of the photographing apparatus; first determination means that determines if the photographed eyeball image satisfies a first condition; second determination means that determines if the photographed eyeball image satisfies a second condition; first specification means that executes first specification processing for specifying the user of the photographing apparatus on the basis of a first eyeball image determined to satisfy the first condition; second specification means that executes second specification processing for specifying the user of the photographing apparatus on the basis of a second eyeball image determined to satisfy the second condition; and control means that performs control so that information on the user specified through the first specification processing and second specification processing and the image photographed by the photographing apparatus are recorded in a memory in association with each other.SELECTED DRAWING: Figure 19

Description

[Technical Field]

[0001] The present invention relates to an imaging device and a control method for an imaging device. [Background technology]

[0002] In information processing devices such as handheld cameras and head-mounted displays, users may be authenticated by comparing a pre-registered image with an image captured by a camera built into the information processing device for capturing images of the user. To properly perform image-based authentication, it is necessary to determine that the person appearing in the pre-registered image and the person appearing in the image captured by the camera are the same person. Patent Document 1 describes a method of pre-registering images with multiple different iris sizes, and then, during authentication, selecting a registered image with an iris size similar to that of the authentication image and calculating the similarity of the images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-518319 [Non-patent literature]

[0004] [Non-Patent Document 1] “MediaPipe: A Framework for Building Perception Pipelines”,[online],[Retrieved March 19, 2020],<https: / / arxiv.org / pdf / 1906.08172.pdf> Summary of the Invention [Problem to be solved by the invention]

[0005] Further improvement in the accuracy of identification is required to identify a user using an image of the user's eyeball captured by a photographing device.

[0006] An object of the present invention is to improve the accuracy of identifying a user of an image capturing device. [Means for solving the problem]

[0007] The present invention is characterized in that it is an image capturing device comprising: an image capturing means for capturing an eyeball image of a user of the image capturing device; a first judgment means for judging whether the eyeball image captured by the image capturing means satisfies a first condition; a second judgment means for judging whether the eyeball image captured by the image capturing means satisfies a second condition different from the first condition; a first identification means for executing a first identification process for identifying the user of the image capturing device based on the first eyeball image judged by the first judgment means to satisfy the first condition; a second identification means for executing a second identification process for identifying the user of the image capturing device based on the second eyeball image judged by the second judgment means to satisfy the second condition; and a control means for controlling the recording of information about the user identified by the first identification process and the second identification process in association with the image captured by the image capturing device in memory. [Effects of the Invention]

[0008] According to the present invention, it is possible to improve the accuracy of identifying a user of an image capturing device. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram showing the appearance of the imaging device. [Figure 2] FIG. 2 is a diagram showing the internal configuration of the imaging device. [Figure 3] FIG. 2 is a block diagram showing the hardware configuration of the imaging device. [Figure 4] FIG. 2 is a block diagram showing the functional configuration of the imaging device. [Figure 5] 10 is a flowchart showing a process executed when registering an eyeball image. [Figure 6] FIG. 10 is a diagram illustrating an example of a home screen. [Figure 7]FIG. 10 is a diagram illustrating an example of a start-up screen of an image registration application. [Figure 8] FIG. 10 is a diagram showing an example of a screen displayed during image acquisition. [Figure 9] FIG. 10 is a diagram showing an example of a screen notifying the user that a registration time has elapsed. [Figure 10] FIG. 10 is a diagram illustrating an example of a registration completion screen. [Figure 11] FIG. 10 is a diagram illustrating an example of a home screen. [Figure 12] FIG. 10 is a diagram illustrating an example of a registration incomplete screen. [Figure 13] 10 is a flowchart showing details of an eyeball image registration process. [Figure 14] FIG. 10 is a diagram illustrating an example of data used for image determination. [Figure 15] FIG. 10 is a diagram showing an example of a registered eyeball image. [Figure 16] FIG. 10 is a diagram illustrating an example of a camera image. [Figure 17] FIG. 10 is a diagram illustrating an example of an eyeball detection result. [Figure 18] FIG. 2 is a block diagram showing the functional configuration of the imaging device. [Figure 19] 10 is a flowchart showing processing executed when the photographing device is used in the second embodiment. [Figure 20] 11 is a flowchart showing processing executed when the photographing device is used in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0011] [Embodiment 1] In this embodiment, a method for registering an eyeball image depending on whether an image captured using a camera built into the finder of an image capturing device satisfies a first criterion or a second criterion will be described.

[0012] FIG. 1 is a diagram showing the appearance of an image capturing device 100 according to this embodiment. FIG. 1(a) is a front perspective view of the image capturing device 100, and FIG. 1(b) is a rear perspective view of the image capturing device 100. In this embodiment, the image capturing device 100 is configured such that an image capturing lens 100B is detachably attached to an image capturing device main body 100A, as shown in FIG. 1(a). The image capturing device 100 is provided with a release button 105, which is an operating member operated by a user when capturing an image of a subject. Note that, in this embodiment, a digital still camera will be described as an example of the image capturing device; however, the image capturing device is not limited to a digital still camera and may be an HMD (head-mounted display) or a data terminal device with a camera (e.g., a smartphone or tablet PC).

[0013] 1(b), a viewfinder 140 is provided on the back of the photographing device 100. The viewfinder 140 is arranged so that the user can look into a display element 110 (described later). A detection sensor 121 is also arranged to detect when the user looks into the viewfinder 140. An eyepiece 112 (FIG. 2) is arranged inside the viewfinder 140.

[0014] 1(b), a rear display device 141 and operation buttons 142 to 143, which are operation members for receiving various operations, are arranged on the rear surface of the imaging device 100. In this embodiment, the rear display device 141 is a touch panel display that also functions as an operation unit.

[0015] Fig. 2 is a cross-sectional view of the imaging device 100 taken along the YZ plane defined by the Y axis and Z axis shown in Fig. 1(a), and shows the internal configuration of the imaging device 100. In Fig. 1, Fig. 2, and Fig. 3 described later, corresponding parts are denoted by the same reference numerals. 2, photographic lens 100B is attached to device body 100A. In this embodiment, for the sake of convenience, photographic lens 100B is shown as being internally composed of two lenses 151 and 152, but in reality, it is composed of many more lenses.

[0016] The device body 100A includes an image sensor 102. An image of a subject transmitted through the photographing lens 100B is formed on the image sensor 102. The device body 100A also includes a CPU 103 that controls the entire photographing device 100, and a memory unit 104 that stores programs executed by the CPU 103 and various data. Under the control of the CPU 103, predetermined image processing is performed on the image signal obtained from the image sensor 102, thereby generating a photographed image. The photographed image is stored in the memory unit 104, etc.

[0017] The device main body 100A also includes a display element 110 and a display element drive circuit 111 that drives the display element 110. The display element 110 is configured with a liquid crystal or the like, and displays an image captured by the image sensor 102, etc. The eyepiece 112 is used to observe the image displayed on the display element 110 in an enlarged state.

[0018] Illumination light sources 113a-113b are light sources for illuminating user's eyeball 114, and are composed of infrared light-emitting diodes. They are arranged around eyepiece 112 and emit infrared light. The illuminated eyeball image and an image resulting from corneal reflection of illumination light sources 113a-113b pass through eyepiece 112, are reflected by light splitter 115, and are then focused by light-receiving lens 116 on eyeball imaging element 117, which is a two-dimensional array of photoelectric conversion elements such as CCDs. Light-receiving lens 116 is positioned so that the pupil of user's eyeball 114 and eyeball imaging element 117 have a conjugate imaging relationship. Under the control of CPU 103, predetermined image processing is performed on the image signal obtained from eyeball imaging element 117, and an image of eyeball 114 is generated. In this embodiment, two illumination light sources 113a-113b are arranged, but three or more illumination light sources may be arranged around the eyepiece 112.

[0019] On the other hand, the photographic lens 100B includes an aperture 161, an aperture drive device 162, a lens drive motor 163, a lens drive member 164, and a photocoupler 165. The lens drive member 164 is made up of gears and the like. The photocoupler 165 detects the rotation of a pulse plate 166 that rotates in conjunction with the lens drive member 164, and transmits this to a focus adjustment circuit 168.

[0020] The focus adjustment circuit 168 drives the lens drive motor 163 a predetermined amount based on information about the amount of rotation of the lens drive member 164 and information about the lens drive amount from the device body 100A, and adjusts the photographic lens 100B to a focused state. Note that the photographic lens 100B exchanges signals with the device body 100A via the mount contacts 147 of the device body 100A.

[0021] 3 is a block diagram showing the hardware configuration of the photographing device 100. The CPU 103 built into the device main body 100A is connected to the image sensor 102, memory unit 104, eye image sensor 117, and display element drive circuit 111. The CPU 103 reads and executes programs stored in the memory unit 104 to implement the processing of the flowcharts described below. The programs include an image registration application program described below. The memory unit 104 is composed of a ROM, a RAM, etc., and stores the photographed images from the image sensor 102, various data used in the eye image registration process described below, and registered eye images. The memory unit 104 also functions as a work area when the programs are executed.

[0022] Furthermore, release button 105, rear display device 141, and operation buttons 142 to 143 are connected to CPU 103 as an operation unit, and output operation information received from the user to CPU 103. Furthermore, rear display device 141 is connected to CPU 103, and displays captured images from image sensor 102 and various types of information under the control of CPU 103.

[0023] The display element 110 is connected to the display element drive circuit 111. The CPU 103 controls the display element drive circuit 111 to cause the display element 110 to display the captured image from the image pickup element 102 and various types of information.

[0024] The CPU 103 is also connected via the mount contacts 147 to a focus adjustment circuit 168 disposed within the photographic lens 100 B and an aperture control circuit 306 included in the aperture drive device 162 .

[0025] The CPU 103 is also connected to a communication I / F 310 and a storage medium I / F 311. The communication I / F 310 is an interface having at least one of wired communication and wireless communication modes, and communicates with an external device such as a server via a communication path corresponding to the communication mode used. The storage medium I / F 311 is an interface for connecting to a removable storage medium 312, such as an SD card slot. The storage medium 312 is a storage medium that is removable from the storage medium I / F 311, such as an SD card or USB memory. The storage medium 312 can also be used as a recording destination for captured images from the image sensor 102 and eyeball images registered in an eyeball image registration process described below.

[0026] Furthermore, a gaze detection circuit 301 is connected to CPU 103. Gaze detection circuit 301 A / D converts the image signal from eye image sensor 117 and transmits this image information to CPU 103. CPU 103 extracts each feature point of the eye image required for gaze detection in accordance with a predetermined algorithm described later, and further calculates the user's gaze direction from the position of each feature point.

[0027] 4 is a block diagram showing the functional configuration of the image capturing device 100. The image capturing device 100 functions as an eyeball capturing unit 401, a determination unit 402, and an eyeball image registration unit 403 by the CPU 103 reading and executing a program stored in the memory unit 104. The eyeball imaging unit 401 acquires an image using an image signal from the eyeball imaging element 117. In this embodiment, the eyeball imaging unit 401 continuously acquires images at predetermined time intervals until a certain time has elapsed, and provides the acquired images to the determination unit 402. The determination unit 402 determines whether the image acquired from the eyeball photographing unit 401 satisfies the first criterion or the second criterion. The eyeball image registration unit 403 registers an image that satisfies the first criterion as an eyeball image for first authentication, and an image that satisfies the second criterion as an eyeball image for second authentication. The eyeball image registration unit 403 associates the eyeball image for first authentication and the eyeball image for second authentication with information about a user currently using the image capturing device 100 (for example, a logged-in user), and stores them in the memory unit 104.

[0028] 5 is a flowchart showing the process executed by the image capturing device 100 according to this embodiment when registering an eyeball image. In the following description, each process (step) is prefixed with an S to omit the process (step) notation. When using the image capturing device 100 for the first time, the user of the image capturing device 100 registers an eyeball image as a preliminary preparation. After the eyeball image registration is complete, the user can be authenticated through a first authentication process and a second authentication process using the pre-registered eyeball image. The first authentication process is an authentication process using an eyeball image for first authentication, which will be described later. The second authentication process is an authentication process using an eyeball image for second authentication, which will be described later.

[0029] First, when the power of the photographing device 100 is turned on, the CPU 103 displays a home screen on the rear display device 141. Fig. 6 shows an example of the home screen. As shown in Fig. 6, an icon 600 for starting an image registration application and an icon 601 for starting to photograph a subject are displayed. The processing of this flowchart starts when the icon 600 for starting the image registration application is touched.

[0030] In S501, the CPU 103 starts the image registration application program stored in the memory unit 104 and displays a startup screen for the image registration application on the rear display device 141. Fig. 7 shows an example of the startup screen displayed in this step. As shown in Fig. 7, a message 701 is displayed that indicates that the image registration application has started and that instructs the user to look into the viewfinder.

[0031] In S502, CPU 103 determines whether or not it has detected someone peering into viewfinder 140, using the sensor value from detection sensor 121. CPU 103 waits in S502 until it detects someone peering into viewfinder 140, and if it detects someone peering into viewfinder 140, it proceeds to S503.

[0032] In S503, the CPU 103 displays a screen showing that an image is being acquired on the display element 110, which serves as an in-finder display device. FIG. 8 shows an example of the screen displayed in S503. As shown in FIG. 8, a message 801 is displayed instructing the user to look around. Here, "looking around" refers to the act of moving the line of sight so as to gaze at various positions within the image displayed on the display element 110. The screen shown in FIG. 8 continues to be displayed on the display element 110 while the registration process is being executed.

[0033] In S504, the CPU 103 acquires an image using an image signal from the eye image sensor 117. Hereinafter, the image acquired in this step will be referred to as a camera image. The CPU 103 continuously acquires camera images at predetermined time intervals while the registration process is being executed. In S505, the CPU 103 executes a registration process for the eyeball image. Details of the registration process executed in this step will be described later with reference to Fig. 13. The CPU 103 starts counting the time elapsed from the start of the registration process using a timer.

[0034] In S506, CPU 103 refers to the timer started in S505 and determines whether a certain time has elapsed since the start of the registration process. Here, it determines whether 10 seconds have elapsed, but another time may be set depending on the environment or use case. If CPU 103 determines that the certain time has elapsed, it proceeds to S507, and if it determines that the certain time has not elapsed, it returns to S504 and continues the registration process.

[0035] In S507, the CPU 103 displays a screen on the display element 110, which serves as an in-finder display device, notifying the user that the registration time has elapsed, and then proceeds to S508. Fig. 9 shows an example of the screen displayed in S507. As shown in Fig. 9, a message 901 is displayed instructing the user to move away from the viewfinder.

[0036] In S508, the CPU 103 determines whether or not the registration of the eyeball image is complete. If it is determined that the registration of the eyeball image is complete, the CPU 103 displays a registration completion screen on the rear display device 141. Fig. 10 shows an example of the registration completion screen. Thereafter, the processing of this flowchart ends.

[0037] When an icon 1000 in the registration completion screen shown in Fig. 10 is touched, the CPU 103 displays a home screen on the rear display device 141. Fig. 11 shows an example of a home screen that is displayed when the registration of an eyeball image is complete. As shown in Fig. 11, an icon 1100 for starting an image registration application and an icon 1101 for starting to photograph a subject are displayed. When the icon 1101 for starting to photograph a subject is touched, the CPU 103 switches the operation mode to a photographing mode.

[0038] On the other hand, if it is determined in S508 that the registration of the eyeball image is not complete, the CPU 103 displays a registration incomplete screen on the rear display device 141. FIG. 12 shows an example of the registration incomplete screen. As shown in FIG. 12, an icon 1200 for displaying a home screen and an icon 1201 for resuming image registration are displayed. When the icon 1201 is touched, the process transitions to S504, and the CPU 103 resumes the registration process. On the other hand, when the icon 1200 is touched, the CPU 103 displays the home screen on the rear display device 141. When the icon 1200 is touched in S508, the registration of the eyeball image is not complete, and therefore the home screen shown in FIG. 6 is displayed.

[0039] Next, the eyeball image registration process executed in S505 will be described in detail with reference to Fig. 13. Fig. 13 is a flowchart showing the eyeball image registration process. In S1301, the CPU 103 acquires one of the camera images acquired in S504. Fig. 16 shows an example of the camera image acquired in this step.

[0040] In S1302, CPU 103 determines whether or not an eyeball is captured in the camera image acquired in S1301. In this embodiment, CPU 103 detects the eyeball from the camera image. The method for eyeball detection may be the method described in Non-Patent Document 1, or another method may be used. If CPU 103 determines that eyeball detection was successful, the process proceeds to S1303, and if it determines that eyeball detection was unsuccessful, the process proceeds to S1307 without registering the image.

[0041] Fig. 17 shows examples of eyeball detection results. Fig. 17(a) shows eyeball cutout image 1702, which is cut out by detecting eyeball region 1701 from camera image 1700. Fig. 17(b) shows eyeball cutout image 1712, which is cut out by detecting eyeball region 1711 from camera image 1710. In this embodiment, if eyeball detection is successful, CPU 103 generates eyeball cutout images 1702 and 1712 from camera images 1700 and 1710 acquired in S1301, and subjects them to registration processing (S1305 and S1306).

[0042] In S1303, CPU 103 determines whether the camera image acquired in S1301 satisfies a first criterion. If CPU 103 determines that the camera image satisfies the first criterion, the process proceeds to S1305, and if CPU 103 determines that the camera image does not satisfy the first criterion, the process proceeds to S1304. In S1304, CPU 103 determines whether the camera image acquired in S1301 satisfies the second criterion. If CPU 103 determines that the camera image satisfies the second criterion, the process proceeds to S1306, and if CPU 103 determines that the camera image does not satisfy the second criterion, the process proceeds to S1307 without registering the image.

[0043] FIG. 14 shows an example of data used for image determination. Reference data 1400 indicates the conditions and the specified number of images to be registered (number of registered images) for each of the first and second references. Coordinate data 1401 indicates predetermined position coordinates in the camera image (five points: P0 to P4). Point P0 indicates the center of the camera image. Points P1 to P4 indicate the vicinity of the four corners of the camera image. The reference data 1400 and coordinate data 1401 are stored in the memory unit 104 or the like, and are read out and used by the CPU 103 during image determination. Note that, although this embodiment describes a method of performing a first determination based on the first reference and a second determination based on the second reference, three or more determinations may be performed using three or more references. The specified number of registered images may be one or more.

[0044] In this embodiment, the first criterion is a condition that the distance between point P0 on the camera image and the pupil center position is within a predetermined number of pixels (e.g., 10 pixels). Therefore, in S1303, a camera image in which the pupil center position is near point P0 is determined to satisfy the first criterion. Furthermore, the second criterion is a condition that the distance between points P1 to P4 on the camera image and the pupil center position is within a predetermined number of pixels (e.g., 10 pixels). Therefore, in S1304, a camera image in which the pupil center position is near points P1 to P4 is determined to satisfy the second criterion. Note that the predetermined number of pixels may be changed as appropriate depending on the resolution of the camera image, the detection accuracy of the pupil center, and the like. Furthermore, in this embodiment, the predetermined number of the first criterion is set to 1, and the predetermined number of the second criterion is set to 4, but this is not limited to this. The predetermined number of the first criterion and the predetermined number of the second criterion may be different or the same.

[0045] In S1305, the CPU 103 registers the cut-out image of the eyeball generated in S1302 as an eyeball image for first authentication. Then, the CPU 103 associates the registered eyeball image for first authentication with information about the user currently using the image capturing device 100 and stores the associated image in the memory unit 104. The process then proceeds to S1307. In S1306, the CPU 103 registers the cut-out image of the eyeball generated in S1302 as an eyeball image for second authentication. Then, the CPU 103 associates the registered eyeball image for second authentication with information about the user currently using the image capturing device 100 and stores the associated image in the memory unit 104. The process then proceeds to S1307.

[0046] 15 shows examples of registered eyeball images. Image 1501 is an example of an eyeball image for first authentication. Images 1502a, 1502b, 1502c, and 1502d are examples of eyeball images for second authentication.

[0047] In S1307, CPU 103 determines whether the number of images registered in S1305 or S1306 has reached the specified number for each of the first and second criteria. If CPU 103 determines that at least one of the first and second criteria has not reached the specified number, the process returns to S1301 and acquires the next camera image until a certain time has elapsed since the start of the registration process. If CPU 103 determines that both the first and second criteria have reached the specified number, the process determines that the registration of eyeball images has been completed and ends the process of this flowchart. In this embodiment, the specified number for the first criterion is set to one, and the specified number for the second criterion is set to four. Therefore, one image (image 1501) is registered as the eyeball image for the first authentication, and four images (images 1502a to 1502d) are registered as the eyeball images for the second authentication.

[0048] According to the first embodiment described above, an image captured using a camera built into the viewfinder of a photographing device is determined to satisfy either a first criterion or a second criterion, and images that satisfy the first criterion and images that satisfy the second criterion can be registered. As a result, when the photographing device is used, it becomes possible to identify the user of the photographing device by authentication using an image that satisfies the first criterion and authentication using an image that satisfies the second criterion. Therefore, the accuracy of identifying the user of the photographing device can be improved.

[0049] [Embodiment 2] In this embodiment, a method for authenticating a user of an imaging device by performing either a first authentication process using an image that satisfies a first criterion or a second authentication process using an image that satisfies a second criterion will be described. Hereinafter, the same components as those in the first embodiment will be assigned the same reference numerals and descriptions thereof will be omitted.

[0050] In this embodiment, the first authentication process is an authentication that is performed when the user starts using the image capturing device 100, and is an authentication that requires strictness. On the other hand, the second authentication process is an authentication that is performed when the user takes a photograph, such as when the operation of the release button 105 is accepted after the first authentication process has been successful, and is an authentication that requires more robustness (easiness of successful authentication) than strictness compared to the first authentication process.

[0051] In this embodiment, the first authentication process is so-called active authentication, in which the user cooperates for authentication. Therefore, with the user's cooperation, there is a high possibility that a high-quality image will be input during authentication. Therefore, according to the flowchart shown in FIG. 5, it is assumed that a high-quality eyeball image, in which the eyeball is centered and the eyelids are properly open, is registered as the eyeball image for the first authentication.

[0052] In this embodiment, the second authentication process is a so-called passive authentication in which the user does not cooperate for authentication. Therefore, it is unlikely that a high-quality image will be input during authentication, and the input images are likely to vary widely. Therefore, according to the flowchart shown in FIG. 5, various types of images that the user may take, such as eyeball position, gaze direction, and eyelid opening degree, are registered as eyeball images for second authentication.

[0053] Fig. 18 is a block diagram showing the functional configuration of the photographing device 100. The photographing device 100 functions as each unit shown in Fig. 18 by the CPU 103 reading and executing a program stored in the memory unit 104. Compared to Fig. 4, in this embodiment, a user identification unit 1804 and a photographed image recording unit 1805 are added.

[0054] The user identification unit 1804 inputs an image from the eyeball imaging unit 401 and performs a first authentication process by comparing the input image with a registered eyeball image for first authentication, and performs a second authentication process by comparing the input image with a registered eyeball image for second authentication. In this embodiment, the user identification unit 1804 selects and performs either the first authentication process or the second authentication process. If the first authentication process or the second authentication process is successful, the user identification unit 1804 identifies the user using user information linked to the eyeball image for first authentication or the eyeball image for second authentication. The first authentication process is an example of a first identification process. The second authentication process is an example of a second identification process.

[0055] The captured image recording unit 1805 associates the information of the user identified by the user identification unit 1804 with the captured image from the image sensor 102 and records the information in the memory unit 104 .

[0056] FIG. 19 is a flowchart showing the processing executed when the photographing device is used. In S1901, the CPU 103 determines whether the conditions for performing user authentication are met. If the CPU 103 determines that the conditions for performing user authentication are met, the process proceeds to S1902, and if the CPU 103 determines that the conditions for performing user authentication are not met, the process returns to S1901.

[0057] In S1902, the CPU 103 determines whether the conditions for the first authentication are met. In this embodiment, the determination that the conditions for the first authentication are satisfied may be based on the timing at which the use of the image capturing apparatus 100 is started, etc. For example, if a user starts the image capturing apparatus 100 and instructs execution of user authentication, it may be determined that the conditions for the first authentication are satisfied. Also, if a user instructs execution of user authentication after a predetermined time has elapsed since the first authentication process was successful, it may be determined that the conditions for the first authentication are satisfied. If the CPU 103 determines that the conditions for the first authentication are satisfied, the process proceeds to S1903. If the CPU 103 determines that the conditions for the first authentication are not satisfied, the process proceeds to S1904, assuming that the conditions for the second authentication are satisfied. Note that if it is determined that the conditions for the first authentication are satisfied, the CPU 103 may display a message on the rear display device 141 that includes content instructing the user to look into the viewfinder. In this embodiment, it is determined that the second authentication condition is met when, after the first authentication process has succeeded, an operation of the release button 105 is accepted. For example, this is the timing when an operation of the release button 105 is accepted after the icon 1101 is touched on the home screen shown in FIG. 11 to switch the operation mode to the shooting mode.

[0058] In S1903, CPU 103 inputs an image acquired using an image signal from eye image sensor 117, and performs the first authentication process by comparing the input image with the eye images for first authentication stored in memory unit 104. Specifically, CPU 103 compares the image features of the input image with the image features extracted from all eye images for first authentication, and determines that the people appearing in both images are the same (authentication successful) if the difference in the features is within a certain range.

[0059] Note that CPU 103 may perform matching using a plurality of input images during execution of the first authentication process. Furthermore, CPU 103 may display instructions to the user to gaze at a specific position on display element 110 as an in-finder display device during acquisition of the input images.

[0060] In S1904, CPU 103 inputs an image acquired using an image signal from eyeball image sensor 117, and performs second authentication processing by comparing the input image with eyeball images for second authentication stored in memory unit 104. Specifically, CPU 103 compares the image features of the input image with the image features extracted from all eyeball images for second authentication, and determines that the people appearing in both images are the same (authentication successful) if the difference in the features is within a certain range.

[0061] In S1905, the CPU 103 determines whether the first authentication process has been successful. If the CPU 103 determines that the first authentication process has been successful, the CPU 103 identifies the user using user information associated with the eyeball image used for the first authentication, and proceeds to S1907. If the CPU 103 determines that the first authentication process has failed, the CPU 103 ends the processing of this flowchart.

[0062] In S1906, CPU 103 determines whether the second authentication process has been successful. If CPU 103 determines that the second authentication process has been successful, it identifies the user using user information linked to the eyeball image used for the second authentication, and proceeds to S1907. If it determines that the second authentication process has failed, it ends the processing of this flowchart.

[0063] In S1907, the CPU 103 generates a captured image using an image signal from the image sensor 102 in accordance with the user's operation of the release button 105.

[0064] In S1908, CPU 103 associates the information of the user identified by the first authentication process or the second authentication process with the captured image generated in S1907 and records them in memory unit 104. Note that instead of memory unit 104, the information may be recorded in storage medium 312. CPU 103 may also access a server via communication I / F 310 and send a control signal so that the captured image and user information are recorded in association with each other. This server is, for example, a server device that stores and manages images captured by an imaging device. Thereafter, the processing of this flowchart ends.

[0065] In this embodiment, if the first authentication process or the second authentication process fails in S1905 and S1906, photography is not permitted. However, photography may be permitted and the captured image may be recorded but the user information may not be recorded.

[0066] According to the second embodiment described above, the accuracy of identifying the user of the photographing device can be improved by selectively using images that satisfy the first criterion and images that satisfy the second criterion for authentication depending on the situation in which the photographing device is used.

[0067] [Embodiment 3] In this embodiment, a method for authenticating a user of an imaging device by performing both a first authentication process using an image that satisfies a first criterion and a second authentication process using an image that satisfies a second criterion will be described. Hereinafter, the same components as those in the first and second embodiments will be assigned the same reference numerals and descriptions thereof will be omitted.

[0068] In this embodiment, the user identification unit 1804 first performs the first authentication process, and only if the first authentication process is successful does the user identification unit 1804 continue with the second authentication process. If both the first authentication process and the second authentication process are successful, the user identification unit 1804 identifies the user using user information linked to the eyeball image for the first authentication and the eyeball image for the second authentication.

[0069] FIG. 20 is a flowchart showing the processing executed when the photographing device is used. In S2001, CPU 103 determines whether or not the conditions for user authentication are met. If CPU 103 determines that the conditions for user authentication are met, the process proceeds to S2002, and if CPU 103 determines that the conditions for user authentication are not met, the process returns to S2001. In this embodiment, the conditions for user authentication include the timing at which use of image capturing apparatus 100 is started and the timing at which operation of release button 105 is accepted. Note that if it is determined that the conditions for user authentication are met, CPU 103 may display a message on rear display device 141 that includes content instructing the user to look into the viewfinder.

[0070] In S2002, CPU 103 inputs an image acquired using an image signal from eyeball image pickup element 117, and performs first authentication processing by comparing the input image with the eyeball image for first authentication stored in memory unit 104. This step is the same processing as S1903.

[0071] In S2003, CPU 103 determines whether the first authentication process has been successful. If CPU 103 determines that the first authentication process has been successful, it identifies the user using user information linked to the eyeball image used for the first authentication, and proceeds to S2004. If it determines that the first authentication process has failed, it ends the processing of this flowchart.

[0072] In S2004, CPU 103 inputs an image acquired using an image signal from eyeball image pickup element 117, and performs second authentication processing by comparing the input image with the eyeball image for second authentication stored in memory unit 104. This step is the same processing as S1904.

[0073] During the execution of the first authentication process and the second authentication process, CPU 103 may acquire multiple images and perform matching using the acquired multiple images. Furthermore, during image acquisition, CPU 103 may display on display element 110, which serves as an in-finder display device, instructions to the user to gaze at a specific position or instructions to look around as described in the first embodiment.

[0074] In S2005, CPU 103 determines whether the second authentication process has been successful. If CPU 103 determines that the second authentication process has been successful, it identifies the user using user information linked to the eyeball image used for the second authentication, and proceeds to S2006. If it determines that the second authentication process has failed, it ends the processing of this flowchart.

[0075] In S2006, the CPU 103 generates a captured image using an image signal from the image sensor 102 in accordance with the user's operation of the release button 105.

[0076] In S2007, CPU 103 associates the information of the user identified by the first authentication process and the second authentication process with the captured image generated in S2006 and records them in memory unit 104. Note that instead of memory unit 104, the information may be recorded in storage medium 312. CPU 103 may also access a server via communication I / F 310 and send a control signal so that the captured image and user information are recorded in association with each other. This server is, for example, a server device that stores and manages images captured by an imaging device. Thereafter, the processing of this flowchart ends.

[0077] In this embodiment, if the first authentication process or the second authentication process fails in S2003 and S2005, photography is not permitted, but photography may also be permitted and the captured image may be recorded but the user information may not be recorded.

[0078] According to the third embodiment described above, authentication is performed using an image that satisfies the first criterion and an image that satisfies the second criterion, thereby improving the accuracy of identifying the user of the image capturing device.

[0079] (Variation 1) In the above-described first embodiment, a method has been described in which a condition based on the pupil center position is used for image determination, but the condition used for image determination is not limited to this, and other conditions may also be used.

[0080] One example is a method using the size of infrared light reflecting areas on the pupil and iris regions. Because illumination light sources 113a-113b are installed inside viewfinder 140, infrared light reflecting areas exist on the pupil and iris regions in the camera image from eye image sensor 117. CPU 103 detects the infrared light reflecting areas on the pupil and iris regions and determines whether the detected areas satisfy a first criterion or a second criterion based on the size of the detected areas. For example, the first criterion may be that the detected area size is 10 pixels or more, and the second criterion may be that the detected area size is 5 pixels or more but less than 10 pixels. Furthermore, if no infrared light reflecting areas are detected on the pupil and iris regions, the image may not be registered.

[0081] Another example is a method that uses the gaze direction. CPU 103 determines whether the first criterion or the second criterion is met according to a condition based on the gaze direction calculated by gaze detection circuit 301. For example, gaze directions may be classified into nine directions (up, down, left, right, eight diagonal directions, and a frontal direction), and each criterion may be set using the nine directions. Another example is a method that uses the degree to which the eyelids are open.

[0082] Furthermore, image judgment may be performed using any combination of multiple conditions, such as conditions based on the pupil center position, conditions based on the size of the infrared light reflection area on the pupil region and iris region, and conditions based on the gaze direction.

[0083] (Variation 2) In the above-described first embodiment, a method for setting the first criterion and the second criterion has been described. However, the second criterion may be set in conjunction with an image that satisfies the first criterion. In this case, the CPU 103 first completes registration of only images that satisfy the first criterion. Next, the CPU 103 may compare the image features of a subsequently acquired image with the image features of the registered image, and determine that the second criterion is satisfied if the difference in the features is within a certain range. The image features and feature differences used here may be, for example, the image features and feature differences used in authentication processing.

[0084] (Variation 3) In the above-described first embodiment, the eyeball image is registered by the user instructing the start of image registration, but the eyeball image may be registered using an image acquired while the user is looking into the viewfinder 140 and photographing a subject. Alternatively, the eyeball image may be registered using an image acquired while the user is looking into the viewfinder 140 and gazing at a specific position. Alternatively, if the photographing device has a calibration function that collects images when the user gazes at a specific position, the eyeball image may be registered using an image acquired during calibration.

[0085] (Variation 4) In the first embodiment described above, an image satisfying the first criterion is registered as an eyeball image for first authentication, and an image satisfying the second criterion is registered as an eyeball image for second authentication. However, an image satisfying a different criterion may be selected and re-registered from the images satisfying the first criterion and the images satisfying the second criterion. For example, an image having a pupil region size equal to or larger than a certain value may be selected and re-registered from all of the images satisfying the first criterion and the second criterion. Furthermore, in order to use a combination of eyeball images that have been registered because they have satisfied specific criteria during authentication processing, each registered eyeball image may be associated with and stored with information indicating which criteria the image satisfied to be registered.

[0086] (Variation 5) In the above-described first embodiment, an eyeball image is registered, but information extracted from the eyeball image may also be registered. For example, data on image features used in authentication processing may be extracted from the eyeball image and registered.

[0087] (Variation 6) In the above-described first embodiment, a determination based on the first criterion and a determination based on the second criterion are successively performed on one image. However, a determination based on the first criterion may be performed on a series of images, and then a determination based on the second criterion may be performed on the series of images. Furthermore, CPU 103 may select, through a user operation, whether to register an eyeball image for the first authentication or to register an eyeball image for the second authentication. Furthermore, CPU 103 may display a selection screen for selecting a criterion to be used for image determination before starting image registration, and may accept an operation to select a criterion before starting image registration.

[0088] (Variation 7) In the second embodiment described above, all registered eyeball images are used in the authentication process. However, an integrated image in which multiple similar images are integrated into one image may also be used in the authentication process. Integrating registered eyeball images can reduce the calculation time for the authentication process. As a method for classifying and integrating registered eyeball images into similar images, for example, there is a method of performing clustering based on feature amounts obtained from the eyeball images and reducing the number of images so that there is only one image belonging to one class. The feature amounts here may be, for example, image features used in the authentication process.

[0089] Furthermore, in the first authentication process and the second authentication process, a priority order may be assigned to the registered eyeball images so that an eyeball image having an image feature that is likely to be input is used for matching in preference to an eyeball image having an image feature that is unlikely to be input. Furthermore, for example, when clustering is performed, a priority order may be assigned to the eyeball images and the integrated image according to the number of images belonging to the same class.

[0090] (Variation 8) In the above-described second embodiment, an example has been described in which the first authentication process is performed when the user starts using the image capture device 100, and the second authentication process is performed when the user takes a photograph, and eyeball images suitable for each authentication are registered in advance. The purposes of the authentication process here include ensuring the authenticity of images captured by the image capture device 100 and calling up modes set for each user of the image capture device 100. Furthermore, when an HMD is used as the image capture device, the purposes include logging in to a virtual environment such as the metaverse by the HMD user, and settling on an object that the HMD user is gazing at.

[0091] In the case of performing authentication with a low probability of false acceptance as the first authentication process, such as authenticating an image captured by the image capture device 100 or paying for an object gazed at by an HMD user, the first standard may be set to be stricter than the second standard. For example, the first standard may be set by combining multiple conditions, such as a condition based on the pupil center position, a condition based on the area size of the infrared reflection area detected from the pupil area and iris area, or a condition based on the gaze direction. Furthermore, the specified number of the first standard may be set to be smaller than the second standard.

[0092] In the case of performing authentication with a high identity acceptance rate as the second authentication process, such as calling up a mode set for each user of the imaging device or logging in to a virtual environment such as the metaverse of an HMD user, the second criteria may be set to be looser than the first criteria. Also, the specified number of the second criteria may be set to be larger than the first criteria.

[0093] (Variation 9) In the above-described second and third embodiments, the photographing device that registers the eyeball image and the photographing device that identifies the user using the registered eyeball image are described as being the same, but they may be different devices. When different devices are used, the photographing device that registers the eyeball image may store the registered eyeball image in a storage area of ​​an external server in association with user information. The photographing device that identifies the user may then obtain the eyeball image registered by the other device from the external server and perform authentication processing.

[0094] Furthermore, although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0095] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage media storing the program constitute the present invention.

[0096] The disclosure of each of the above-described embodiments includes the following configurations, methods, and programs. (Configuration 1) An imaging device, an image capturing means for capturing an image of the eyeball of a user of the photographing device; a first determination means for determining whether the eyeball image captured by the imaging means satisfies a first condition; a second determination means for determining whether the eyeball image captured by the image capturing means satisfies a second condition different from the first condition; a first identification means for executing a first identification process for identifying a user of the photographing device based on the first eyeball image determined by the first determination means to satisfy the first condition; a second identification means for executing a second identification process for identifying a user of the photographing device based on the second eyeball image determined by the second determination means to satisfy the second condition; a control means for controlling the recording of the user information identified by the first identification process and the second identification process in association with the image captured by the image capturing device in a memory; An imaging device comprising: (Configuration 2) 2. The photographing device according to configuration 1, wherein the memory is built into the photographing device or is physically detachably connected to the photographing device. (Configuration 3) the imaging device is connected to the server having the memory via a wired and / or wireless communication path; 3. The photographing device according to configuration 1 or 2, wherein the control means transmits a signal to the server so that the photographed image and the user information are recorded in association with each other in the memory. (Configuration 4) The photographing device described in any one of configurations 1 to 3, characterized in that the control means controls the user's information to be recorded in the memory when the user is successfully identified in either the first identification process or the second identification process. (Configuration 5) The photographing device described in any one of configurations 1 to 3, characterized in that the control means controls the user's information to be recorded in the memory when the user is successfully identified in both the first identification process and the second identification process. (Configuration 6) 5. The photographing device according to configuration 4, wherein the control means selects whether to execute the first specified process or the second specified process. (Configuration 7) the first identification means inputs an eyeball image captured by the imaging means, and executes the first identification process by comparing the input image with the first eyeball image; The photographing device described in any one of configurations 1 to 6, wherein the second identification means inputs an eyeball image photographed by the photographing means and performs the second identification process by comparing the input image with the second eyeball image. (Configuration 8) The photographing device described in any one of configurations 1 to 7, characterized in that while at least one of the judgment by the first judgment means and the judgment by the second judgment means is being executed, the control means controls the display device in the viewfinder to display content instructing the user to move their line of sight. (Configuration 9) 9. The photographing device according to any one of configurations 1 to 8, further comprising a registration means for registering the first eyeball image and the second eyeball image. (Configuration 10) further comprising a registration unit that registers a plurality of the first eyeball images and a plurality of the second eyeball images, 9. The photographing device according to any one of configurations 1 to 8, wherein the number of registered first eyeball images is different from the number of registered second eyeball images. (Configuration 11) further comprising a registration unit for registering a plurality of the first eyeball images or a plurality of the second eyeball images, Priorities are assigned to the plurality of images, respectively; 8. The photographing device according to configuration 7, wherein the first specifying means or the second specifying means checks the input image according to the priority order. (Configuration 12) the first condition is that the pupil center position in the eyeball image captured by the image capturing means is near the center of the image, 12. The photographing device according to any one of configurations 1 to 11, wherein the second condition is that the center position of the pupil in the eyeball image photographed by the photographing means is located away from the center of the image. (Configuration 13) Further, the device has an irradiation means for irradiating the eyeball with infrared light, the first condition and the second condition are that an infrared light reflecting area on a pupil area and an iris area in an eyeball image photographed by the photographing means has a predetermined size, 13. The photographing device according to any one of configurations 1 to 12, wherein the predetermined size differs between the first condition and the second condition. (Configuration 14) The first condition and the second condition are that the line of sight direction in the eyeball image captured by the image capturing means is in a predetermined direction, 14. The photographing device according to any one of configurations 1 to 13, wherein the predetermined direction differs between the first condition and the second condition. (Configuration 15) The photographing device described in any one of configurations 1 to 14, characterized in that the second judgment means judges that the second condition is satisfied when a difference in image features between the eyeball image photographed by the photographing means and the first eyeball image is within a certain range. (Configuration 16) the first identification means integrates similar images from among the first eyeball images determined by the first determination means to satisfy the first condition, and performs the first identification process based on the integrated image; The photographing device described in any one of configurations 1 to 15, characterized in that the second identification means integrates similar images among the second eye images determined by the second judgment means to satisfy the second condition, and performs the second identification process based on the integrated image. (method) A control method for an imaging device, comprising: a photographing step for photographing an eyeball image of a user of the photographing device; a first determination step of determining whether the eyeball image captured in the photographing step satisfies a first condition; a second determination step of determining whether the eyeball image captured in the photographing step satisfies a second condition different from the first condition; a first identification step of executing a first identification process for identifying a user of the photographing device based on the first eyeball image determined to satisfy the first condition in the first determination step; a second identification step of executing a second identification process for identifying a user of the photographing device based on the second eyeball image determined to satisfy the second condition in the second determination step; a control step of controlling the recording of the user information identified by the first identification process and the second identification process in a memory in association with the image captured by the image capturing device; 10. A method for controlling an imaging device, comprising: (program) The imaging device computer an image capturing means for capturing an image of the eyeball of a user of the photographing device; a first determination means for determining whether the eyeball image captured by the imaging means satisfies a first condition; a second determination means for determining whether the eyeball image captured by the image capturing means satisfies a second condition different from the first condition; a first identification means for executing a first identification process for identifying a user of the photographing device based on the first eyeball image determined by the first determination means to satisfy the first condition; a second identification means for executing a second identification process for identifying a user of the photographing device based on the second eyeball image determined by the second determination means to satisfy the second condition; a control means for controlling the recording of the user information identified by the first identification process and the second identification process in association with the image captured by the photographing device in a memory; A program characterized by functioning as

Claims

1. An imaging device, an image capturing means for capturing an image of the eyeball of a user of the photographing device; a first determination means for determining whether the eyeball image captured by the imaging means satisfies a first condition; a second determination means for determining whether the eyeball image captured by the image capturing means satisfies a second condition different from the first condition; a first identification means for executing a first identification process for identifying a user of the photographing device based on the first eyeball image determined by the first determination means to satisfy the first condition; a second identification means for executing a second identification process for identifying a user of the photographing device based on the second eyeball image determined by the second determination means to satisfy the second condition; a control means for controlling the recording of the user information identified by the first identification process and the second identification process in association with the image captured by the image capturing device in a memory; An imaging device comprising:

2. 2. The photographing device according to claim 1, wherein the memory is built into the photographing device or is physically detachably connected to the photographing device.

3. the imaging device is connected to the server having the memory via a wired and / or wireless communication path, 2. The photographing device according to claim 1, wherein the control means transmits a signal to the server so that the photographed image and the user information are recorded in association with each other in the memory.

4. The photographing device according to claim 1, characterized in that the control means controls the user's information to be recorded in the memory when the user is successfully identified in either the first identification process or the second identification process.

5. The photographing device according to claim 1, characterized in that the control means controls the user's information to be recorded in the memory when the user is successfully identified in both the first identification process and the second identification process.

6. 5. The photographing apparatus according to claim 4, wherein the control means selects whether to execute the first specific processing or the second specific processing.

7. the first identification means inputs an eyeball image captured by the imaging means, and executes the first identification process by comparing the input image with the first eyeball image; 2. The photographing device according to claim 1, wherein the second identification means inputs an eyeball image photographed by the photographing means and performs the second identification process by comparing the input image with the second eyeball image.

8. 2. The photographing device according to claim 1, wherein, while at least one of the first determination means and the second determination means is making a determination, the control means controls the display device in the viewfinder to display a message instructing the user to move their line of sight.

9. 2. The photographing apparatus according to claim 1, further comprising a registration unit for registering the first eyeball image and the second eyeball image.

10. further comprising a registration unit for registering a plurality of the first eyeball images and a plurality of the second eyeball images, 2. The photographing device according to claim 1, wherein the number of registered first eye images is different from the number of registered second eye images.

11. further comprising a registration unit for registering a plurality of the first eyeball images or a plurality of the second eyeball images, Priorities are assigned to the plurality of images, respectively; 8. The photographing apparatus according to claim 7, wherein the first specifying means or the second specifying means checks the input image in accordance with the priority order.

12. the first condition is that the pupil center position in the eyeball image captured by the image capturing means is near the center of the image, 2. The photographing apparatus according to claim 1, wherein the second condition is that the center of the pupil in the eyeball image photographed by the photographing means is located away from the center of the image.

13. Further, the device has an irradiation means for irradiating the eyeball with infrared light, the first condition and the second condition are that an infrared light reflection area on a pupil area and an iris area in an eyeball image captured by the image capturing means has a predetermined size, 2. The photographing apparatus according to claim 1, wherein the predetermined size differs between the first condition and the second condition.

14. the first condition and the second condition are that the line of sight direction in the eyeball image captured by the image capturing means is in a predetermined direction, 2. The photographing apparatus according to claim 1, wherein the predetermined direction differs between the first condition and the second condition.

15. The photographing device according to claim 1, wherein the second determination means determines that the second condition is satisfied when a difference in image features between the eyeball image photographed by the photographing means and the first eyeball image is within a certain range.

16. the first identification means integrates similar images from among the first eyeball images determined by the first determination means to satisfy the first condition, and performs the first identification process based on the integrated image; The photographing device described in claim 1, characterized in that the second identification means integrates similar images among the second eyeball images determined by the second judgment means to satisfy the second condition, and performs the second identification process based on the integrated image.

17. A control method for an imaging device, comprising: a photographing step for photographing an eyeball image of a user of the photographing device; a first determination step of determining whether the eyeball image captured in the photographing step satisfies a first condition; a second determination step of determining whether the eyeball image captured in the photographing step satisfies a second condition different from the first condition; a first identification step of executing a first identification process for identifying a user of the photographing device based on the first eyeball image determined to satisfy the first condition in the first determination step; a second identification step of executing a second identification process for identifying a user of the photographing device based on the second eyeball image determined to satisfy the second condition in the second determination step; a control step of controlling the recording of the user information identified by the first identification process and the second identification process in a memory in association with the image captured by the image capturing device; 10. A method for controlling an imaging device, comprising:

18. The imaging device computer an image capturing means for capturing an image of the eyeball of a user of the photographing device; a first determination means for determining whether the eyeball image captured by the imaging means satisfies a first condition; a second determination means for determining whether the eyeball image captured by the image capturing means satisfies a second condition different from the first condition; a first identification means for executing a first identification process for identifying a user of the photographing device based on the first eyeball image determined by the first determination means to satisfy the first condition; a second identification means for executing a second identification process for identifying a user of the photographing device based on the second eyeball image determined by the second determination means to satisfy the second condition; a control means for controlling the recording of the user information identified by the first identification process and the second identification process in association with the image captured by the image capturing device in a memory; A program characterized by functioning as

Images