Imaging device, imaging method, and computer program

Abstract

To provide an imaging device capable of improving the accuracy of personal identification based on eyeball images. [Solution] The imaging device includes an image sensor for capturing an image of a subject, a viewfinder for confirming the image of the subject, an image sensor for the eyeball for capturing an image of the user's eyeball while looking through the viewfinder, a state determination means for determining the state of the eyeball image, a user identification means for identifying the user based on the eyeball image, an assignment means for assigning user information relating to the user identified by the user identification means to the subject image captured by the image sensor while the user is looking through the viewfinder, and an assignment determination means for controlling the assignment of user information to the subject image if the state determination means determines that the state of the eyeball image does not meet predetermined conditions.

Description

Technical Field

[0001] The present invention relates to an imaging device, an imaging method, a computer program, and the like.

Background Art

[0002] In recent years, for copyright management and authenticity assurance, there has been a demand for a function to identify and record the photographer of an image or video. Patent Document 1 describes a method of performing personal authentication / identification by imaging the eyeball close to the eyepiece portion of the finder when using a camera, and attaching the information to the image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the method of Patent Document 1, when the user approaches the finder to place their eye close to it at the start of using the camera, the fineness of the video of the eye imaged by the imaging element and the size of the eye within the angle of view are not considered to change.

[0005] Therefore, depending on the imaging timing of the eyeball image, the eyeball image may be blurred, or the size of the eyeball image may change significantly. When performing personal identification using such an eyeball image, the identification accuracy may decrease.

[0006] One object of the present invention is to provide an imaging device capable of improving the accuracy of personal identification based on an eyeball image.

Means for Solving the Problems

[0007] The imaging device according to an embodiment of the present invention an imaging element that captures a subject image, A viewfinder for checking the image of the subject, An eye image sensor that captures an image of the user's eyeball as they look through the viewfinder, A state determination means for determining the state of the eyeball image, A user identification means for identifying the user based on the aforementioned eyeball image, A means for assigning user information relating to the user identified by the user identification means to the subject image captured by the image sensor while the user is looking through the viewfinder, If the state determination means determines that the state of the eyeball image does not meet predetermined conditions, the assignment determination means controls not to assign the user information to the subject image. It is characterized by having the following features. [Effects of the Invention]

[0008] According to the present invention, an imaging device capable of improving the accuracy of personal identification based on eyeball images can be realized. [Brief explanation of the drawing]

[0009] [Figure 1] (A) is a front perspective view of the imaging device in Embodiment 1 of the present invention, and (B) is a rear perspective view. [Figure 2] Figure 1(A) is a cross-sectional view of the camera housing 1B cut along the YZ plane formed by the Y and Z axes. [Figure 3] This is a functional block diagram showing an example configuration of the imaging device according to Embodiment 1. [Figure 4] This figure shows an example of the viewfinder field of view in Embodiment 1. [Figure 5] This figure illustrates the principle of the eyeball information detection method in Embodiment 1. [Figure 6] This flowchart shows an example of the processing of the imaging method using the personal identification unit in Embodiment 1. [Figure 7](A) is a diagram showing an example of an eyeball image projected onto the imaging device 17 for the eye, (B) of the same figure is a diagram showing an example of the luminance distribution in the area α of Fig. 7(A), and (C) is a diagram showing an example of the relationship between the distance z and the interval between the two reflected images. [Figure 8] This is a schematic diagram for explaining the method for determining the amount of temporal variation of the distance z from the eye to the finder in Embodiment 1. [Figure 9] This is a schematic diagram for explaining a configuration example of the personal identification unit 207. [Figure 10] This is a diagram showing an example of the correspondence table between the feature amounts stored in the feature amount storage unit 303 and the persons. [Figure 11] This is a diagram for explaining a configuration example of the CNN 302 that performs personal identification from two-dimensional image data. [Figure 12] This is a diagram for explaining a detailed example of the feature detection process on the feature detection cell surface and the feature integration process on the feature integration cell surface. [Figure 13] (A) and (B) are schematic diagrams for explaining an example of a head-mounted XR device as an imaging device according to Embodiment 2. [Figure 14] This is a flowchart showing an example of the personal identification process according to Embodiment 3. [Figure 15] This is a flowchart showing an example of the personal identification process according to Embodiment 4. [Figure 16] This is a diagram showing an example of the correspondence table between the feature amounts, the persons, and the distance z' according to Embodiment 4.

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each figure, the same members or elements are denoted by the same reference numerals, and duplicate explanations are omitted or simplified.

[0011] <Embodiment 1> Figs. 1(A) and 1(B) show an example of the appearance of a digital camera 1 having an eyeball information acquisition function and an individual identification function in Embodiment 1 of the present invention. Fig. 1(A) is a front perspective view, and Fig. 1(B) is a rear perspective view.

[0012] In this embodiment, as shown in Fig. 1(A), the digital camera 1 is composed of an interchangeable photographing lens 1A and a camera housing portion 1B as the camera body. Further, a release button 5, which is an operation member for receiving an imaging operation from the user, is arranged.

[0013] The release button 5 has switches SW1 and SW2 (not shown). SW1 is turned on by the first stroke of the release button 5 and is a switch for starting photometry, distance measurement, gaze detection operation, etc. of the camera. SW2 is turned on by the second stroke of the release button 5 and is a switch for starting the release operation.

[0014] As shown in Fig. 1(B), on the back of the digital camera 1, an eyepiece window frame 121 and an eyepiece lens 12 for the user to look into are arranged for a display element 10 (described later) included inside the camera. Further, a plurality of light sources 13a, 13b for illuminating the eyeball are arranged around the eyepiece lens 12.

[0015] The digital camera functions as an imaging device for taking images. Further, the eyepiece window frame 121 and the eyepiece lens 12 for the user to look into function as a finder for confirming an image of a subject. The digital camera of this embodiment can take still images and moving images of a subject. Also, in this embodiment, photographing and imaging are used in the same meaning.

[0016] Fig. 2 is a cross-sectional view of the camera housing portion 1B cut along the YZ plane formed by the Y-axis and the Z-axis shown in Fig. 1(A), showing an outline of the configuration of the digital camera 1. In Figs. 1 and 2, corresponding parts are denoted by the same numbers.

[0017] In Figure 2, 1A represents the photographic lens in an interchangeable lens camera. For convenience, Figure 2 shows two lenses, 101 and 102, inside the photographic lens 1A, but it may consist of three or more lenses.

[0018] 1B represents the housing of the camera body, and 2 is an image sensor for capturing subject images, which consists of, for example, a CCD or CMOS image sensor and is positioned on the intended imaging plane of the shooting lens 1A of the digital camera 1. In this embodiment, the image sensor 2 has a pixel configuration that can output two types of image signals with parallax in order to perform known image plane phase-detection autofocus.

[0019] The digital camera 1 includes a CPU 3, which acts as a computer to control the entire camera, and a memory unit 4 that records images captured by the image sensor 2. It also includes a display element 10 made of liquid crystal or the like for displaying the captured images, a display element drive circuit 11 that drives the display element, and an eyepiece lens 12 for observing the subject image displayed on the display element 10.

[0020] 13a to 13b are light sources consisting of infrared light-emitting diodes, etc., for illuminating the photographer's eyeball 14, and are arranged around the eyepiece lens 12. Based on the corneal reflection image of the photographer's (user's) eyeball illuminated by these light sources and the positional relationship of the pupil, the orientation of the eyeball (direction of gaze), etc., is detected.

[0021] The corneal reflection image of the eyeball 14, illuminated by light sources 13a to 13b, passes through the eyepiece lens 12, is reflected by the light splitter 15 consisting of a half mirror, and is imaged onto the light-receiving surface of the eyeball image sensor 17, such as a CMOS image sensor, by the light-receiving lens 16. The eyeball image sensor 17 then captures an image of the eyeball of the user (photographer) looking through the viewfinder.

[0022] Furthermore, the position of the pupil of the photographer's eyeball 14 and the position of the eyeball image sensor 17 are in a conjugate imaging relationship via the light-receiving lens 16. The positional relationship between the eyeball image (image of the pupil) formed on the eyeball image sensor 17 and the corneal reflection images of the light sources 13a to 13b is determined by a predetermined algorithm described later.

[0023] 111 is an aperture located inside the photographic lens 1A, 112 is an aperture drive device, 113 is a lens drive motor, and 114 is a lens drive component consisting of drive gears, etc. 115 is a photocoupler that detects the amount of rotation of a pulse plate 116 that is linked to the lens drive component 114 and transmits this to the focus adjustment circuit 118.

[0024] The focus adjustment circuit 118 drives the lens drive motor 113 by a predetermined amount based on information from the photocoupler 115 and information on the amount of lens drive from the camera, moving the photographic lens 101 to the focus position. 117 is a mount contact that serves as the interface between the camera and the lens.

[0025] Thus, the digital camera 1 as an imaging device includes at least an image sensor for capturing an image of a subject, a viewfinder for displaying the image of the subject, and an image sensor for the eyeball that captures an image of the user's eyeball as they look through the viewfinder.

[0026] Figure 3 is a functional block diagram showing an example configuration of an imaging device according to Embodiment 1. Components identical to those in Figure 2 are given the same numbers in Figure 3. Note that some of the functional blocks shown in Figure 3 are realized by having the CPU, which acts as a computer within the imaging device, execute computer programs stored in the memory, which acts as a storage medium.

[0027] However, some or all of these can be implemented in hardware. Hardware options include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs).

[0028] Furthermore, the functional blocks shown in Figure 3 do not necessarily have to be housed in the same enclosure; they may be composed of separate devices connected to each other via signal paths. The above explanation regarding Figure 3 also applies to Figure 9, which will be discussed later.

[0029] The CPU3 of the microcomputer built into the camera body is connected to an eyeball information detection circuit 201, a photometering circuit 202, an autofocus detection circuit 203, a signal input circuit 204, a display element drive circuit 11, and an illumination light source drive circuit 205.

[0030] Furthermore, the CPU 3 transmits signals via the focus adjustment circuit 118 located within the photographic lens, the aperture control circuit 206 included in the aforementioned aperture drive device 112, and the mount contact 117. The memory unit 4 attached to the CPU 3 stores imaging signals from the image sensor 2 and the eyeball image sensor 17, as well as gaze correction data to compensate for individual differences in gaze.

[0031] The eyeball information detection circuit 201 performs A / D conversion on the image signal of the eyeball 14 from the eyeball image sensor 17 and transmits this image information to the CPU 3. The CPU 3 extracts each feature point of the eyeball image necessary for eyeball information detection according to a predetermined algorithm described later, and further calculates the photographer's eyeball information from the position of each feature point.

[0032] The photometering circuit 202 uses the signal obtained from the image sensor 2, which also acts as a photometering sensor, to amplify the luminance signal output corresponding to the brightness of the field of view. After logarithmic compression and A / D conversion, it sends the field of view luminance information to the CPU 3.

[0033] The autofocus detection circuit 203 performs A / D conversion on the signal voltages from multiple pixels in the image sensor 2 that are capable of outputting two types of image signals with parallax for phase difference detection, and sends them to the CPU 3. Based on the two types of image signals with parallax, the CPU 3 calculates the distance to the subject corresponding to each focus detection point. This is a known technique known as image plane phase-detection autofocus. In this embodiment, for example, 180 focus detection points are provided on the imaging plane of the image sensor 2.

[0034] The signal input circuit 204 is connected to switches SW1 and SW2 of the release button 5, and the on and off signals of switches SW1 and SW2, respectively, are transmitted to the CPU 3. 207 is a personal identification unit, which is a unit for identifying the photographer (user) based on the eyeball image. The configuration of the personal identification unit 207 will be described later with reference to Figure 9.

[0035] Figure 4 shows an example of the viewfinder field of view in Embodiment 1, and shows the state in which the image of the subject is displayed on the display element 10. In Figure 4, 300 is the field of view mask and 400 is the focus detection area.

[0036] In this embodiment, the imaging surface of the image sensor 2 has, for example, 180 focus detection points, and in the viewfinder image in Figure 4, the distance measuring point targets 4001 to 4180 corresponding to these 180 focus detection points are superimposed on the subject image. In Figure 4, among these indicators, the indicator corresponding to the current estimated gaze point position is displayed in a frame as estimated gaze point A.

[0037] Figure 5 is a diagram illustrating the principle of the eyeball information detection method in Embodiment 1, and shows a schematic of the optical system for performing the eyeball information detection described in Figure 2. In Figure 5, 13a and 13b are light sources such as light-emitting diodes that irradiate the observer with infrared light.

[0038] Light sources 13a and 13b are arranged, for example, approximately symmetrically with respect to the optical axis of the light-receiving lens 16, and illuminate the observer's eyeball 14. A portion of the illumination light reflected by the eyeball 14 is focused by the light-receiving lens 16 onto the image sensor 17 for the eyeball. Note that 141 is the pupil, 142 is the cornea, a and b are the ends of the pupil 141, and c is the center of the pupil.

[0039] The personal identification unit 207 used in this embodiment will be described below with reference to Figures 6 to 12. Figure 6 is a flowchart showing an example of the processing of an imaging method using the personal identification unit in Embodiment 1. Note that the operation of each step in the flowchart of Figure 6 is performed sequentially by the CPU 3, etc., acting as a computer, executing a computer program stored in memory.

[0040] In Figure 6, for example, when switch SW2 is turned on and a photo is taken, the personal identification processing flow is started when the captured image is stored in the memory unit.

[0041] Alternatively, the processing flow shown in Figure 6 may be initiated when a sensor (not shown) detects the approach of the eye when the eye is brought close to the viewfinder in order to start imaging the subject. In this case, when the eye is brought close to the viewfinder in order to start imaging the subject, a determination operation will be performed by the state determination means described later.

[0042] In step S001 of Figure 6, the eyeball is illuminated by an illumination light source. That is, infrared light is shone towards the observer's eyeball 14 by light sources 13a and 13b.

[0043] The image of the observer's eyeball, illuminated by the infrared light described above, is formed on the eyeball image sensor 17 through the light-receiving lens 16. The eyeball image sensor 17 performs photoelectric conversion, and the eyeball image becomes available as an electrical signal from the eyeball image sensor 17.

[0044] Next, in step S002, the eyeball image signal obtained from the eyeball image sensor 17 is sent to the CPU 3, and in steps S003 to S004, eyeball information is calculated from the eyeball image signal obtained in step S002.

[0045] In other words, in step S003, the coordinates of the pupil center and the coordinates of the corneal reflection image are obtained. Specifically, from the information of the eyeball image signal obtained in step S002, the coordinates of the corneal reflection images Pd and Pe of light sources 13a and 13b, respectively, and the coordinates of the point corresponding to the pupil center c are determined, as shown in Figure 5.

[0046] Figure 7(A) shows an example of an eyeball image projected onto the eyeball image sensor 17, Figure 7(B) shows an example of the brightness distribution in region α of Figure 7(A), and Figure 7(C) shows an example of the relationship between distance z and the distance between two reflected images.

[0047] Infrared light emitted from light sources 13a and 13b illuminates the cornea 142 (Figure 5) of the observer's eyeball 14. Corneal reflection images Pd and Pe, formed by a portion of the infrared light reflected from the surface of the cornea 142, are focused by the light-receiving lens 16. These images are then formed on the eyeball image sensor 17 as points Pd' and Pe' as shown in the figure.

[0048] Similarly, light beams from the ends a and b of the pupil 141 (Figure 5) are also imaged onto the ocular image sensor 17. In Figure 7(A), the horizontal direction is the X-axis and the vertical direction is the Y-axis. Region α in Figure 7(A) is the region for measuring the brightness distribution in the X-axis direction (horizontal direction) of the images Pd' and Pe' formed by the corneal reflection images of light sources 13a and 13b (Figure 7(B)). In Figure 7(B), the X-axis coordinates (horizontal direction) of images Pd' and Pe' are denoted as Xd and Xe, respectively. Also, the X-axis coordinates of images a' and b' formed by light beams from the ends a and b (Figure 5) of pupil 14b are denoted as Xa and Xb, respectively.

[0049] In the example of the luminance distribution shown in Figure 7(B), a relatively very strong first level of luminance is obtained at coordinates Xd and Xe, which correspond to the images Pd' and Pe' formed by the corneal reflection images of light sources 13a and 13b, respectively. On the other hand, in the region between coordinates Xa and Xb, which corresponds to the area of ​​the pupil 141, a relatively very low second level of luminance is obtained, except for the coordinates Xd and Xe mentioned above.

[0050] In contrast, in the region corresponding to the area of ​​the iris 143 outside the pupil 141, which has an X-coordinate value smaller than Xa, and in the region with an X-coordinate value greater than Xb, a value between the first level and the second level is obtained.

[0051] Therefore, based on the luminance distribution for the above X coordinate, the X coordinates Xd and Xe of the images Pd' and Pe' formed by the corneal reflection images of light sources 13a and 13b, respectively, and the X coordinates Xa and Xb of the images a' and b' at the pupillary tip can be obtained.

[0052] Furthermore, as shown in Figure 5, when the rotation angle θx of the optical axis of the eyeball 14 with respect to the optical axis of the light-receiving lens 16 is small, the coordinates Xc of the pupil center c' imaged on the eyeball image sensor 17 can be approximated as Xc ≈ (Xa + Xb) / 2.

[0053] As described above, the X-coordinate Xc of the pupil center c' imaged on the ocular image sensor 17, and the X-coordinates Xd and Xe of the corneal reflection images Pd' and Pe' of the light sources 13a and 13b can be obtained.

[0054] Furthermore, in step S004, the distance z is obtained. The distance z is the distance from the photographer's (user's) eyeball to the viewfinder, and can be calculated, for example, from the distance between the two Purkinje images in the eyeball image in Figure 7(C).

[0055] The graph in Figure 7(C) shows the correlation between the distance ΔP between the two reflected images Pd and Pe formed by the two light sources 13a and 13b, respectively, and the distance z, which is the distance from the eyeball image sensor 17 to the eyeball 14. As shown in the graph in Figure 7(C), the distance z decreases monotonically and nonlinearly with increasing ΔP, so the distance z from the eyeball image sensor 17 to the eyeball 14 can be uniquely calculated based on ΔP.

[0056] In this embodiment, the correspondence between the distance z from the ophthalmic image sensor 17 to the eyeball 14, corresponding to the interval ΔP between the two reflected images Pd and Pe as described above, is stored in memory in advance in the form of a correlation table, for example.

[0057] Therefore, by reading that table, the distance z is obtained from the measured ΔP. However, this is not limited to this method; for example, the distance z may be calculated from ΔP using an approximate formula, or it may be measured using a measuring means for measuring the distance from the eyeball to the display surface.

[0058] Next, in step S005, it is determined whether the pupil position has been sufficiently detected. If the coordinates Xa and Xb of the pupil tip have been sufficiently detected to the extent necessary for pupil detection, the process proceeds to step S006. If they have not been detected, the process returns to step S001 and restarts from image acquisition.

[0059] This is because, for example, if the distance from the eyeball to the camera is far and the image of the eyeball is unclear, the difference in brightness between the pupil and the iris, as explained in Figure 7(B), may not be obtained, and the position of the pupillary edge may not be detected.

[0060] In other words, when photographing a subject with a camera, if the photographer does not bring their eye close enough to the viewfinder and attempts to identify an individual using a blurry image of the eyeball, the accuracy of the identification may decrease. Therefore, in step S005, based on the pupil position detection result, it is determined whether the image is clear enough to sufficiently detect the pupil tip. Here, step S005 functions as a state determination step (state determination means) for determining the clarity of the eyeball image.

[0061] Next, in step S006, it is determined whether the time-series fluctuation of distance z, obtained in step S004, has stabilized. If it is stable, the process proceeds to step S007; otherwise, it returns to step S001 and restarts from image acquisition. In other words, in step S006, the state determination means determines that the predetermined conditions are not met if the fluctuation of the distance from the eyeball to the finder within a predetermined time exceeds a predetermined threshold.

[0062] This is because, for example, if an eyeball image is acquired while the eye is being brought closer to the viewfinder, the size of the eyeball image will change significantly from a smaller state than a normal eyeball image, and this large size fluctuation may lead to a decrease in the accuracy of personal identification.

[0063] Figure 8 is a schematic diagram illustrating the method for determining the time variation of the distance z from the eyeball to the viewfinder in Embodiment 1. In step S006, as shown in Figure 8, the time-series variation of the distance z from the eye to the viewfinder is obtained, and as described later, it is determined whether the variation within a predetermined time is below a threshold value and whether the time series has stabilized.

[0064] The graph in Figure 8 shows elapsed time on the horizontal axis and the distance z from the eye to the viewfinder on the vertical axis. At time t=0, the distance z=z0. From this state, as time progresses, the eye is brought closer to the viewfinder, and the graph shows how the distance z decreases downwards.

[0065] As time progresses, around time t2, the decrease in distance z stops, and it takes on a nearly constant value of Z. This can be attributed to the fact that the area around the eye is in contact with the viewfinder frame, and further fluctuations in distance z cease.

[0066] The state where the variation in this distance z is almost eliminated is considered to be the eye position when the user holds the camera in their normal position. By using the eye image at this point for personal identification, it becomes possible to identify individuals using an eye image of a nearly constant size each time, thereby suppressing a decrease in identification accuracy.

[0067] In step S006, time-series data of distance z, as shown in Figure 8, is used to calculate the change in distance z Δz from the current time to a predetermined time Δt. When Δz falls below a predetermined threshold Zth, it is determined that the eyes have come as close as possible and the distance z has stabilized.

[0068] For example, if we focus on point A at time t1, the distance z at time t1 is z = Z12, and at a predetermined time Δt before time t1, the distance z is Z = Z11.

[0069] The amount of variation of z within this range is Δz1 = Z12 - Z11. Δz1 is the amount of variation when the graph is clearly sloping downwards, and since ΔZ1 > threshold Zth, it is determined that the graph is unstable.

[0070] Next, focusing on point B at time t2, the distance z at time t2 is z = z22, and at a predetermined time Δt prior to time t2, the distance z is z = z21. The amount of change in z within this range is Δz2 = Z22 - Z21.

[0071] As mentioned above, Δz2 is the range in which the graph around time t2 begins to take on an almost constant value, and since Δz2 ≤ threshold Zth, it is determined to be stable. Therefore, in step S006, if we enter region 2 after t2, it is determined that the size of the eyeball image has stabilized, that is, the time variation of distance z has stabilized, and we proceed to step S007. On the other hand, if we are in region 1 before t2, we return to step S001.

[0072] Here, step S006 functions as a state determination step (state determination means) that determines the temporal stability of the distance z as the state of the eyeball image.

[0073] Next, in step S007, the eyeball images acquired up to step S006 are input to the personal identification unit 207 to extract features for personal identification. That is, the user identification means identifies the user in step S007 when the position of the pupil of the user's eyeball is detected.

[0074] In step S008, the personal identification unit 207 performs personal identification using the features extracted in step S007. Here, step S008 functions as a user identification step (user identification means) that identifies the user based on an eyeball image.

[0075] Figure 9 is a schematic diagram illustrating an example configuration of the personal identification unit 207. 302 is a CNN (Convolutional Neural Network), 303 is a feature memory unit, 304 is a feature matching unit, and 305 is a personal information assignment unit.

[0076] In step S2008, the feature quantity extracted in step S007 and the multiple feature quantities stored in the feature quantity storage unit 303 are sequentially compared by the feature quantity matching unit 304. Then, among the multiple stored feature quantities, the feature quantity with a degree of agreement above a predetermined level and the highest degree of agreement is determined. The person possessing that feature quantity is identified as the personal identification result.

[0077] Figure 10 shows an example of a correspondence table between features and individuals stored in the feature memory unit 303. The feature memory unit 303, which stores a correspondence table between features and individuals as shown in Figure 10, is provided, for example, as part of the memory unit 4, and when performing the above-mentioned personal identification, a correspondence table like the one in Figure 10 is read from the feature memory unit 303.

[0078] In step S009, it is determined whether a feature with a degree of similarity above a predetermined level was found in step S008 and whether personal identification was successful. If successful, the process proceeds to step S010. If no feature with a degree of similarity above the predetermined level is found and personal identification is unsuccessful, the process returns to step S001 and restarts from image acquisition.

[0079] In other words, in step S009, the assignment determination means controls the system so that if the result of identification by the user identification means is not a predetermined result (i.e., not successful), it does not assign the user information to the subject image.

[0080] In step S010, the personal identification result is added to the image. That is, the user information (photographer information) identified by the personal identification operations up to step S090 is added to the captured image by the personal information addition unit 305, and the captured image is then stored in, for example, the image storage area of ​​the memory unit 4, after which the processing flow in Figure 6 is terminated.

[0081] Furthermore, the personal information attachment unit 305 adds (attaches) user information (photographer information) to the captured image by, for example, overlaying it as encrypted watermark data. Alternatively, it adds (attaches) user information (photographer information) to the image file of the captured image as encrypted metadata. The captured image with the added user information (photographer information) is then stored, for example, in the image storage area of ​​the memory unit 4.

[0082] Furthermore, the personal information assignment unit 305 compares the date and time of the eyeball image used to identify the user information (photographer information) with the date and time of the subject's image. If the two do not match or overlap, the user information (photographer information) is not assigned to the image. If the two do match or overlap, the user information (photographer information) is assigned to the image.

[0083] Furthermore, step S010 functions as an assignment step (assignment means) for assigning user information relating to the user identified by the user identification step (user identification means) to the subject image captured by the image sensor while the user is looking through the viewfinder.

[0084] As described above, in steps S001 to S010, the selection of images for personal identification and the timing of image capture are optimized based on the detection status of eyeball information, thereby maintaining a high level of personal identification accuracy.

[0085] Furthermore, steps S005 and S006 in the above description function as an assignment determination step (assignment determination) that controls the assignment of user information to the subject image if the state determination step determines that the state of the eyeball image does not meet predetermined conditions.

[0086] Next, we will explain the configuration example of the aforementioned CNN302 using Figures 11 and 12. Figure 11 is a diagram illustrating the configuration example of a CNN302 that performs personal identification from 2D image data.

[0087] In Figure 11, the processing flow is such that the input is taken from the left end and processing proceeds to the right. CNN302 consists of two layers, called the feature detection layer (S layer) and the feature integration layer (C layer), which are arranged hierarchically.

[0088] In CNN302, the S layer first detects the next features based on the features detected in the previous layer. The features detected in the S layer are then integrated in the C layer, and the detection results for that layer are sent to the next layer.

[0089] The S layer consists of feature detection cell surfaces, each detecting a different feature. The C layer consists of feature integration cell surfaces, pooling the detection results from the preceding feature detection cell surfaces. Hereafter, unless otherwise specified, feature detection cell surfaces and feature integration cell surfaces will be collectively referred to as feature surfaces. In this embodiment, the final nth output layer consists only of the S layer and does not use the C layer.

[0090] Figure 12 illustrates detailed examples of feature detection processing on the feature detection cell plane and feature integration processing on the feature integration cell plane. For example, the L-th layer feature detection cell plane (S layer) is composed of multiple feature detection neurons, and these feature detection neurons are connected to the preceding L-1 layer C layer in a predetermined structure.

[0091] For example, the L-th hierarchical feature integration cell plane (layer C) is composed of multiple feature integration neurons, and these feature integration neurons are connected to the S-layer of the same hierarchical level in a predetermined structure. Here, for example, in Figure 12, within the M-th cell plane of the L-th hierarchical S-layer, the output value of the feature detection neuron at position (ξ,ζ)

number

[0092] Furthermore, within the M-th cell plane of the L-th layer C, the output value of the feature integration neuron at position (ξ,ζ)

number

[0093] At that time, the connection coefficient of each neuron

number

number

[0094]

number

number

[0095] Furthermore, in Equation 1, f is an activation function, and any sigmoid function such as the logistic function or hyperbolic tangent function can be used; for example, it can be implemented using the tanh function. Also, the above

number

[0096] Equation 2 takes a simple linear sum without using an activation function. When an activation function is not used, as in Equation 2, the internal state of the neuron is considered.

number

number

[0097] Also, Equation 1

number

number

[0098] Next, we will explain ξ, ζ, u, v, and n in equations 1 and 2. The position (ξ,ζ) corresponds to the position coordinates in the input image, for example.

number

[0099] Furthermore, in Equation 2, n represents the nth cell surface of the L-1 layer C, and is called the integration target feature number. Basically, a sum-of-products operation is performed on all cell surfaces present in the L-1 layer C.

[0100] (u,v) is the relative position coordinate of the connection coefficient, and sum-of-products operations are performed within a finite range (u,v) depending on the size of the feature to be detected. This finite range of (u,v) is called the receptive field. The size of the receptive field is referred to as the receptive field size below and is expressed as the number of horizontal pixels × the number of vertical pixels in the connected range.

[0101] Also, in Equation 1, L=1, that is, in the very first S layer,

number

number

number

[0102] Incidentally, the distribution of neurons and pixels is discrete, and the feature numbers of the connected nodes are also discrete, so ξ, ζ, u, v, and n are not continuous variables but take discrete values. Here, ξ and ζ are non-negative integers, n is a natural number, and u and v are integers, all of which are within a finite range.

[0103] In Equation 1

number

[0104] The adjustment of this coupling coefficient distribution is the learning process, and in building a CNN, various test patterns are presented.

number

[0105] Next, in equation 2

number

[0106]

number

[0107] Here, (u,v) is given as a finite range, so, similar to the explanation of feature detection neurons, the finite range is called the receptive field, and the size of the range is called the receptive field size. This receptive field size can be set to an appropriate value depending on the size of the Mth feature in the Lth layer S.

[0108] In Equation 3, σ is the feature size factor, and it can be set to an appropriate constant depending on the receptive field size. Specifically, it is best to set it to a value such that the outermost value of the receptive field can be considered to be approximately 0.

[0109] In this embodiment of the CNN, the above-described calculations are performed at each layer, and in the final S layer, individual identification is performed to determine which user is using the device and the optimal gaze correction coefficient is applied to that individual.

[0110] In other words, in steps S005 to S006 of Figure 6, if the pupil position can be sufficiently detected and the time variation of the distance z from the eyeball to the finder is stable, the process proceeds to the calculation of individual features in step S007. That is, the selection of images for calculating individual features and the timing of the feature calculation are controlled.

[0111] Thus, in Embodiment 1, by simultaneously detecting eyeball information and taking the results into consideration to optimize the selection of images for personal identification and the timing of image capture, it is possible to provide an imaging device or personal information assignment device that can maintain a high level of accuracy in personal identification.

[0112] <Embodiment 2> In Embodiment 1, a digital camera 1 was given as an example of an imaging device, but it is not limited to this. In other words, any device that has an eyepiece close to the photographer's eye and uses an eyeball image sensor provided in the eyepiece to detect the photographer's eyeball image information and uses that eyeball image to detect eyeball information and perform personal identification is acceptable. For example, a head-mounted XR device as shown in Figure 13 may also be used.

[0113] Figures 13(A) and (B) are schematic diagrams illustrating an example of a head-mounted XR device as an imaging device according to Embodiment 2. XR stands for Extended Reality or Cross Reality.

[0114] The XR device in Figure 13 is a head-mounted display device 100 that acquires eyeball images independently for the left and right eyes and has means for detecting eyeball information and identifying individuals from these images. Figure 13(A) is a front perspective view of the head-mounted display device 100, and Figure 13(B) is a rear perspective view of the head-mounted display device 100.

[0115] 501 is a lens element, through which the user of the head-mounted display device views the outside world. 502 is a virtual image display element, which displays a virtual image superimposed on the field of view of the user's left and right eyes as they view the outside world through the optical system; this is a so-called see-through type head-mounted display device.

[0116] 503 is an illumination light source drive circuit, and 13a and 13b are light sources such as light-emitting diodes that irradiate infrared light onto the user (photographer), with each light source illuminating the user's eyeball. A portion of the illumination light reflected by the eyeball is focused onto the eyeball image sensor 17.

[0117] The 520 is an external imaging unit that captures images of the external environment in the direction the user (photographer) is facing. The external imaging unit includes an image sensor.

[0118] The head-mounted display device 100 described above will perform personal identification operations such as login operations after being worn. Also, similar to Embodiment 1, the personal identification unit will detect eyeball information and perform personal identification using an eyeball image captured via a light-receiving lens 16 by an eyeball image sensor 17 positioned directly in front of the eyeball.

[0119] During the process of putting on the head-mounted display device 100, the distance between the eyeball and the image sensor 17 for the eyeball located in the eyepiece changes, which may cause changes in the detail of the eyeball image and the appearance of the eye within the field of view.

[0120] However, in Embodiment 2, as in Embodiment 1, the accuracy of personal recognition can be improved by optimizing the selection of images for personal recognition and the timing of image capture based on the detection results of eyeball information.

[0121] <Embodiment 3> Figure 14 is a flowchart illustrating an example of personal identification processing according to Embodiment 3. The CPU 3 and other components of the computer execute a computer program stored in memory, sequentially performing each step in the flowchart of Figure 14. Note that steps with the same code numbers as those in Figure 6 in Figure 14 represent the same processing and therefore their explanation is omitted.

[0122] In the processing flow shown in Figure 14, feature vectors are calculated for all obtained eyeball images. Then, based on the detection status of the pupil position in each eyeball image and the time variation of the distance z from the eyeball to the finder, it is determined which features to use for individual identification.

[0123] In other words, as shown in Figure 14, by performing steps S007, S008, and S009 before the determination processing in steps S005 and S006, the process from feature extraction for personal identification to calculation of personal identification results is carried out in advance.

[0124] Thus, in Embodiment 3, the personal identification result has already been calculated by step S009, and in steps S005 and S006, a determination is made based on the obtained eyeball information to determine whether the calculated personal identification result should be adopted.

[0125] In other words, in this embodiment, the user identification means identifies the user, and if the pupil position of the user's eyeball is detected, it adopts the result of the user identification. If the eyeball information is an image that does not meet predetermined conditions, the system cannot proceed to step S010, and the identified user information cannot be written.

[0126] <Embodiment 4> In Embodiment 4, in addition to the correspondence between feature quantities and people as shown in Figure 10, the distance z' from the eye to the finder is also pre-associated with the person and recorded, and the information of distance z' is used to perform processing as shown in the flowchart of Figure 15.

[0127] Figure 15 is a flowchart showing an example of personal identification processing according to Embodiment 4, and Figure 16 is a diagram showing an example of a correspondence table between feature quantities, people, and distance z' according to Embodiment 4. The CPU 3 and other components of the computer execute the computer program stored in memory, which sequentially performs the operations of each step in the flowchart of Figure 15. In Figure 15, steps with the same code numbers as in Figures 6 and 14 represent the same processing, so their explanation is omitted.

[0128] In this embodiment, if it is determined that personal identification was successful in step S009, the process proceeds to step S011. In step S011, it is determined whether the difference between the distance z from the eye to the camera, obtained in step S004, and the distance z', which was registered along with the feature quantities when registering them in advance, as shown in Figure 16, falls within a predetermined range.

[0129] As shown in Figure 16, in this embodiment, when registering features in advance, not only the features and person as shown in Figure 10, but also the distance z' from the eye to the finder is calculated and associated with the person and recorded in the feature storage unit 303 in the memory unit 4.

[0130] Then, if the distance z from the eye to the camera, obtained in step S004, falls within a predetermined error range from the distance z' recorded in the feature memory unit 303, the process proceeds to the next step S010. Otherwise, the process returns to step S001 and restarts from image acquisition.

[0131] In other words, in step S011, the state determination means determines that the predetermined condition is not met if the difference between the distance from the user's eyeball to the finder and a predetermined distance registered in advance for each user is greater than or equal to a predetermined value, and returns to step S001.

[0132] Thus, in Embodiment 4, personal identification results are added to the image using an eyeball image taken at a distance z within a predetermined range relative to the distance z' from the eye to the finder, which was registered in advance. Therefore, a comparison between personal registration and use can be performed under the same image conditions (image clarity, eye size, distance z, etc.), enabling more accurate personal identification.

[0133] Next, in step S010, the identified user information (photographer information) is added to the captured image file, for example, by embedding it as encrypted metadata. Alternatively, it is added by superimposing it on the captured image as encrypted watermark data. The captured image is then stored in the memory unit 4, and the processing flow shown in Figure 15 is completed.

[0134] As described above, by comparing the detection results of eyeball information during device use with pre-registered eyeball information and optimizing the selection of personal identification results based on the results, a high level of personal identification accuracy can be maintained.

[0135] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible in accordance with the spirit of the present invention, and these are not excluded from the scope of the present invention. Furthermore, some of the above embodiments may be combined as appropriate.

[0136] Furthermore, the present invention includes, for example, a system that realizes the functions of the above embodiment using at least one processor such as a CPU, memory, and circuitry (e.g., an ASIC). Alternatively, multiple processors may be used for distributed processing.

[0137] Furthermore, in order to implement some or all of the control in the above embodiment, a computer program that implements the functions of the above embodiment may be supplied to the imaging device, etc., via a network or various storage media.

[0138] The computer (or CPU or MPU, etc.) in the imaging device may read and execute the program. In that case, the program and the storage medium storing the program constitute the present invention. The present invention includes the following combinations.

[0139] (Composition 1) An imaging device comprising: an image sensor for capturing an image of a subject; a viewfinder for confirming the image of the subject; an image sensor for capturing an image of the eyeball of a user looking through the viewfinder; a state determination means for determining the state of the eyeball image; a user identification means for identifying the user based on the eyeball image; an assignment means for assigning user information relating to the user identified by the user identification means to the subject image captured by the image sensor while the user is looking through the viewfinder; and an assignment determination means for controlling the assignment of user information to the subject image if the state determination means determines that the state of the eyeball image does not meet predetermined conditions.

[0140] (Configuration 2) The imaging apparatus according to configuration 1, characterized in that the assignment determination means controls not to assign the user information to the subject image if the result of the identification by the user identification means is not a predetermined result.

[0141] (Composition 3) The imaging device according to configuration 1 or 2, characterized in that the user identification means identifies the user when the position of the pupil of the user's eyeball is detected.

[0142] (Composition 4) The imaging device according to any one of configurations 1 to 3, characterized in that the user identification means identifies the user and, when the position of the pupil of the user's eyeball is detected, adopts the result of the user identification.

[0143] (Composition 5) The imaging apparatus according to any one of configurations 1 to 4, characterized in that the state determination means determines that the predetermined condition is not met if the variation in the distance from the eyeball to the viewfinder within a predetermined time period is greater than or equal to a predetermined threshold.

[0144] (Composition 6) The imaging device according to any one of configurations 1 to 5, characterized in that the state determination means performs the determination when the eye is brought close to the viewfinder in order to start imaging the subject.

[0145] (Composition 7) The imaging device according to any one of configurations 1 to 6, characterized in that the state determination means determines that the predetermined condition is not met when the difference between the distance from the user's eyeball to the viewfinder and a predetermined distance registered in advance for each user is greater than or equal to a predetermined value.

[0146] (method) An imaging method using an imaging device having an image sensor for capturing an image of a subject, a viewfinder for displaying an image of the subject, and an image sensor for capturing an image of the eyeball of a user looking through the viewfinder, the imaging method comprising: a state determination step for determining the state of the eyeball image; a user identification step for identifying the user based on the eyeball image; an assignment step for assigning user information relating to the user identified in the user identification step to the subject image captured by the image sensor while the user is looking through the viewfinder; and an assignment determination step for controlling the assignment of user information to the subject image if the state determination step determines that the state of the eyeball image does not meet predetermined conditions.

[0147] (program) A computer program for controlling each means of an imaging device described in any one of configurations 1 to 7 by computer. [Explanation of Symbols]

[0148] 1A: Shooting lens 1B: Camera housing 2: Image sensor 3:CPU 4: Memory section 10: Display elements 11: Display element driving circuit 12: Eyepiece 13a~b: Illumination light source 14:Eyeball 15: Light splitter 16: Light-receiving lens 17: Image sensor for the eye 111: Aperture 112: Aperture drive unit 113: Lens drive motor 114: Lens driving member 115: Photocoupler 116: Pulse plate 117: Mounting contact 141: Pupil 142:Cornea 143: Iris

Claims

1. An image sensor that captures the subject image, A viewfinder for checking the image of the subject, An eye image sensor that captures an image of the user's eyeball as they look through the viewfinder, A state determination means for determining the state of the eyeball image, A user identification means for identifying the user based on the aforementioned eyeball image, A means for assigning user information relating to the user identified by the user identification means to the subject image captured by the image sensor while the user is looking through the viewfinder, If the state determination means determines that the state of the eyeball image does not meet predetermined conditions, the assignment determination means controls not to assign the user information to the subject image. An imaging device characterized by having the following features.

2. The imaging apparatus according to claim 1, characterized in that the assignment determination means controls not to assign the user information to the subject image if the result of the identification by the user identification means is not a predetermined result.

3. The imaging device according to claim 1, characterized in that the user identification means identifies the user when the position of the pupil of the user's eyeball is detected.

4. The imaging device according to claim 1, wherein the user identification means identifies the user and, when the position of the pupil of the user's eyeball is detected, adopts the result of the user identification.

5. The imaging apparatus according to claim 1, characterized in that the state determination means determines that the predetermined condition is not met if the variation in the distance from the eyeball to the viewfinder within a predetermined time period exceeds a predetermined threshold.

6. The imaging apparatus according to claim 1, characterized in that the state determination means performs the determination when the eye is brought close to the viewfinder in order to start imaging the subject.

7. The imaging device according to claim 1, characterized in that the state determination means determines that the predetermined condition is not met when the difference between the distance from the user's eyeball to the viewfinder and a predetermined distance registered in advance for each user is greater than or equal to a predetermined value.

8. An imaging method using an imaging device having an image sensor for capturing an image of a subject, a viewfinder for displaying the image of the subject, and an image sensor for capturing an image of the user's eyeballs as they look through the viewfinder, The eyeball image state determination step, A user identification step, which identifies the user based on the aforementioned eyeball image, A step for assigning user information relating to the user identified in the user identification step to the subject image captured by the image sensor while the user is looking through the viewfinder, If the state determination step determines that the state of the eyeball image does not meet predetermined conditions, the grant determination step controls the system so that the user information is not granted to the subject image. An imaging method characterized by having the following features.

9. A computer program for controlling each means of an imaging apparatus according to any one of claims 1 to 7 by computer.

Images