Recording device, imaging device, control method, recording system

Abstract

To provide a device capable of detecting a line-of-sight that reflects the photographer's intention at the time of imaging on a still image and a moving image.SOLUTION: In an imaging apparatus, when a fixing operation is performed to fix a gazing point by an imaging operation by a user, CPU 3 controls to store a still image or a moving image displayed on a display element 10 by associating the same with line-of-sight detection information and gazing point fixing information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a device capable of detecting a line of sight.

Background Art

[0002] In recent years, the automation and intelligence of cameras have advanced, and technologies for acquiring and using information on the line-of-sight position of a photographer (user) have been proposed.

[0003] Patent Document 1 discloses a technique for adding information on the line-of-sight position of a photographer during shooting to an image and superimposing and displaying the trajectory of the shooting position on the captured image after shooting.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Further improvement is required for the information added to the image.

[0006] Therefore, an object of the present invention is to provide a recording device, an imaging device, a control method, and a recording system that can reflect the intention of a photographer.

Means for Solving the Problems

[0007] Therefore, the present invention provides an eyeball image sensor for capturing an image of a user's eyeball; a calculation means for calculating the position of the user's gaze point on a display unit multiple times based on the eyeball image captured by the eyeball image sensor; an operation means for receiving a first operation to confirm the position of the gaze point calculated by the calculation means; and the position of the gaze point calculated by the calculation means before the first operation is performed by the user on the operation means, and the position of the gaze point confirmed in response to the first operation being performed by the user on the operation means, after the first operation is performed. Using an image sensor separate from the aforementioned image sensor for the eyeball, the subject is captured photograph death It is characterized by having a first storage means that stores an image in association with it. [Effects of the Invention]

[0008] According to the present invention, the photographer's intentions can be reflected. [Brief explanation of the drawing]

[0009] [Figure 1] Schematic diagram of the imaging device to which the first embodiment of the present invention is applied. [Figure 2] Schematic diagram of the external appearance of the imaging device in the first embodiment of the present invention [Figure 3] Block diagram of an imaging device to which the first embodiment of the present invention is applied. [Figure 4] Explanatory diagram showing the viewfinder field of view in the first embodiment of the present invention [Figure 5] Diagram illustrating the principle of the gaze detection method in the first embodiment of the present invention. [Figure 6] Schematic diagram of the eyeball image projected onto the eyeball image sensor 17 [Figure 7] Outline Flow Routine for Eye-Tracking Detection [Figure 8] Outline routine for recording gaze point determination information and gaze detection information in the first embodiment of the present invention [Figure 9] Outline routine for recording gaze point determination information and gaze detection information in the second embodiment of the present invention [Figure 10]Schematic Routine for Fixation Point Determination Information and Gaze Detection Information Recording in the Third Embodiment of the Present Invention [Figure 11] Schematic Routine for Fixation Point Determination Information and Gaze Detection Information Recording in the Fifth Embodiment of the Present Invention [Figure 12] Schematic Routine for Fixation Point Determination Information and Gaze Detection Information Recording in the Sixth Embodiment of the Present Invention

Embodiments for Carrying Out the Invention

[0010] Hereinafter, preferred embodiments of the present invention will be described in detail based on the accompanying drawings.

[0011] Example 1 Description of the Configuration FIGS. 1 to 3 are diagrams showing the configuration diagrams of the camera of this example.

[0012] FIG. 2 shows the appearance of the digital still camera 1 in the present invention. FIG. (a) is a front perspective view, and FIG. (b) is a rear perspective view. In this example, the digital still camera 1 is composed of a photographing lens 1A and a housing portion 1B of the camera body, as shown in the front perspective view of FIG. (a). Further, a release button 5, which is an operation member for receiving an imaging operation from a photographer (user), is arranged.

[0013] Further, as shown in the rear perspective view of FIG. (b), an eyepiece 12 for the photographer to look through a display element (to be described later) included inside the camera is arranged on the back of the digital still camera 1. In addition, various operation members used for camera operations, such as an operation member α (touch panel-compatible liquid crystal), an operation member β (lever-type operation member), and an operation member γ (button-type cross key) indicated by 41 to, are arranged.

[0014] FIG. 1 is a cross-sectional view of the camera housing cut along the YZ plane formed by the Y-axis and the Z-axis illustrated in FIG. (a), and is an explanatory diagram showing the schematic configuration of the digital still camera 1 in the present invention. In FIGS. 1 and 2, corresponding parts are denoted by the same numbers.

[0015] In FIG. 1, 1A indicates a photographing lens in a lens-exchangeable camera. In this embodiment, for convenience, the inside of the photographing lens 1A is represented by two lenses 101 and 102, but as is well known, it is actually composed of a larger number of lenses. 1B indicates the housing portion of the camera body, and the configuration of the units included therein is as follows. 2 is an imaging element, which is disposed on the planned imaging surface of the photographing lens 1A of the digital still camera 1. The digital still camera 1 includes a CPU 3 that controls the entire camera and a memory unit 4 that records the image captured by the imaging element 2. Further, a display element 10 composed of a liquid crystal or the like for displaying the captured image, a display element driving circuit 11 for driving it, and an eyepiece lens 12 for observing the subject image displayed on the display element 10 are disposed. The above imaging element 2 and display element 10 correspond to the imaging element and display element in claim 1.

[0016] 13a to 13b are light sources for illuminating the eyeball 14 of the photographer for detecting the line-of-sight direction from the relationship between the reflection image due to the corneal reflection of the light source conventionally used in a single-lens reflex camera or the like and the pupil. They are composed of infrared light-emitting diodes and are disposed around the eyepiece lens 12. The illuminated eyeball image and the image due to the corneal reflection of the light sources 13a to 13b pass through the eyepiece lens 12, are reflected by the optical splitter 15, and are imaged on an eyeball imaging element 17 in which a two-dimensional array of photoelectric elements such as a CCD is arranged by the light-receiving lens 16. The light-receiving lens 16 positions the pupil of the photographer's eyeball 14 and the eyeball imaging element 17 in a conjugate imaging relationship. The line-of-sight direction is detected by a predetermined algorithm described later from the positional relationship between the image of the eyeball and the image due to the corneal reflection of the light sources 13a to 13b imaged on the eyeball imaging element 17. The above eyeball imaging element corresponds to the eyeball imaging element in claim 1.

[0017] 111 is an aperture located within the photographic lens 1, 112 is an aperture drive device, 113 is a lens drive motor, and 114 is a lens drive component consisting of drive gears, etc. 115 is a photocoupler that detects the rotation of a pulse plate 116 linked to the lens drive component 114 and transmits this information to the lens focus adjustment circuit 118. Based on this information and information on the amount of lens drive from the camera, the focus adjustment circuit 118 drives the lens drive motor 113 by a predetermined amount to move the photographic lens 1A to the focus position. 117 is a known mount contact that serves as an interface between the camera and the lens. 6 is an acceleration sensor built into the camera that detects camera panning. 119 is an acceleration sensor built into the lens that detects lens panning. For panning determination, which will be described later, either the acceleration sensor 6 built into the camera, the acceleration sensor 119 built into the lens, or both are used.

[0018] Figure 3 is a block diagram showing the electrical configuration built into the digital camera 1 in the earlier configuration, with components identical to those in Figure 1 being numbered the same. The central processing unit 3 of the microcomputer built into the camera body will be referred to as CPU3 below. The CPU3 is connected to the gaze detection circuit 201, the photometering circuit 202, the autofocus detection circuit 203, the signal input circuit 204, the display element drive circuit 11, the illumination light source drive circuit 205, and the acceleration sensor 6. Signals are also transmitted to the focus adjustment circuit 118 located in the photographic lens, the aperture control circuit 206 included in the aforementioned aperture drive device 112, and the acceleration sensor 119 via the mount contact 117 shown in Figure 1. The memory unit 4 attached to the CPU3 has the function of storing imaging signals from the image sensor 2 and the eyeball image sensor 17. The memory unit 4 is connected to the recording medium 5, and the CPU3 converts the imaging signals from the image sensor 2 stored in the memory unit 4 into captured images, which are then transferred to the recording medium 5.

[0019] The gaze detection circuit 201 performs A / D conversion on the output of the eyeball image formed from the eyeball image sensor 17 (CCD-EYE) and transmits this image information to the CPU 3. The CPU 3 extracts each feature point of the eyeball image necessary for gaze detection according to a predetermined algorithm described later, and further calculates the photographer's gaze from the position of each feature point.

[0020] The photometering circuit 202 uses the signal obtained from the image sensor 2, which also acts as a photometering sensor, to amplify the luminance signal output corresponding to the brightness of the field of view. After logarithmic compression and A / D conversion, it sends the field of view luminance information to the CPU 3.

[0021] The autofocus detection circuit 203 performs A / D conversion on the signal voltages from multiple pixels used for phase difference detection within the CCD of the image sensor 2 and sends them to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point from the signals of the multiple pixels. This is a known technique known as image plane phase difference AF. In this embodiment, as an example, it is assumed that there are 180 focus detection points on the image sensor corresponding to the locations shown in the viewfinder image in Figure 4.

[0022] A switch SW1 is connected to the signal input circuit 204, which turns ON with the first stroke of the release button 5 (not shown) and initiates the camera's metering, distance measurement, gaze detection, and the determination of the photographer's gaze position. A switch SW2 is also connected to the signal input circuit 204, which turns ON with the second stroke of the release button and initiates the release operation. The aforementioned signals are input to the signal input circuit 204 and transmitted to the CPU 3.

[0023] Furthermore, the operating components α (touch panel compatible LCD), β (lever-type operating component), and γ (button-type directional pad) shown in 41-43 above are configured to transmit their operation signals to the CPU 3.

[0024] Figure 4 shows the viewfinder field of view, indicating that the display element 10 is in operation.

[0025] In Figure 4, 300 is the field of view mask, 400 is the focus detection area, and 4001 to 4180 show 180 distance measuring point targets superimposed on the through image shown on the display element 10 at positions corresponding to multiple focus detection points on the imaging surface. Furthermore, among these indicators, the indicator corresponding to the current estimated gaze point position is displayed with a frame appearing, as shown for estimated gaze point A in the figure. Displaying a frame at the estimated gaze point position corresponds to displaying an indicator indicating the gaze point position estimated by the gaze point position estimation means on the display element, as described in claim 1. Also, the configuration in which the CPU 3 sends a signal to the display element 10 to display a frame at the estimated gaze point position corresponds to the gaze point position display means in claim 1.

[0026] Figure 5 is a diagram illustrating the principle of the gaze detection method, and corresponds to a summary diagram of the optical system used for gaze detection in Figure 1 mentioned above.

[0027] In Figure 5, 13a and 13b are light sources such as light-emitting diodes that emit infrared light insensitive to the observer. Each light source is positioned approximately symmetrically with respect to the optical axis of the light-receiving lens 16 and illuminates the observer's eyeball 14. A portion of the illumination light reflected by the eyeball 14 is focused by the light-receiving lens 16 onto the image sensor 17 for the eyeball.

[0028] The means for detecting eye gaze will be explained below using Figures 7 and 8.

[0029] <Explanation of eye-tracking operation> Figure 7 shows a schematic flowchart of the gaze detection process. In Figure 7, when the gaze detection routine starts, at S001, light sources 13a and 13b emit infrared light towards the observer's eyeball 14. The image of the observer's eyeball, illuminated by the infrared light, is formed on the eyeball image sensor 17 through the light-receiving lens 16. The eyeball image sensor 17 performs photoelectric conversion, and the eyeball image becomes processable as an electrical signal.

[0030] In step S002, the eyeball image signal obtained from the eyeball image sensor 17 as described above is sent to the CPU 3.

[0031] In step S003, the coordinates of the points corresponding to the corneal reflection images Pd, Pe and the pupil center c of the light sources 13a and 13b shown in Figure 5 are determined from the eyeball image signal information obtained in S002. The infrared light emitted from the light sources 13a and 13b illuminates the cornea 142 of the observer's eyeball 14. At this time, the corneal reflection images Pd, Pe, formed by a portion of the infrared light reflected from the surface of the cornea 142, are focused by the light-receiving lens 16. These are then imaged onto the eyeball image sensor 17 (points Pd', Pe' shown in the figure). Similarly, the light beams from the ends a and b of the pupil 141 are also imaged onto the eyeball image sensor 17. In Figure 6, Figure 6(a) shows an example of a reflection image obtained from the eyeball image sensor 17, and Figure 6(b) shows an example of brightness information obtained from the eyeball image sensor 17 in region α of the above image example. As shown in the figure, the horizontal direction is the X-axis and the vertical direction is the Y-axis. In this case, let Xd and Xe be the coordinates in the X-axis direction (horizontal direction) of the images Pd' and Pe' formed by the corneal reflections of light sources 13a and 13b. Also, let Xa and Xb be the coordinates in the X-axis direction of the images a' and b' formed by the light beams from the ends a and b of the pupil 14b. In the example of luminance information in (b), an extremely high level of luminance is obtained at positions Xd and Xe, which correspond to the images Pd' and Pe' formed by the corneal reflections of light sources 13a and 13b. In the region between coordinates Xa and Xb, which corresponds to the area of ​​the pupil 141, an extremely low level of luminance is obtained, except for the positions Xd and Xe. In contrast, in the region with an X coordinate value lower than Xa and an X coordinate value higher than Xb, which corresponds to the area of ​​the iris 143 outside the pupil 141, an intermediate value between the two types of luminance levels is obtained. From the information on the variation in brightness level relative to the above X-coordinate position, the X-coordinates Xd and Xe of the images Pd' and Pe' formed by the corneal reflection images of light sources 13a and 13b, and the X-coordinates Xa and Xb of the images a' and b' at the pupillary tip can be obtained. Furthermore, when the rotation angle θx of the optical axis of the eyeball 14 with respect to the optical axis of the light-receiving lens 16 is small, the coordinate Xc of the point corresponding to the pupillary center c (let's call it c') that is imaged on the eyeball image sensor 17 can be expressed as Xc ≈ (Xa + Xb) / 2. From the above, the X-coordinate of c' corresponding to the pupillary center that is imaged on the eyeball image sensor 17, and the coordinates of the corneal reflection images Pd' and Pe' of light sources 13a and 13b could be estimated.

[0032] Furthermore, in step S004, the imaging magnification β of the eyeball image is calculated. β is a magnification determined by the position of the eyeball 14 relative to the light-receiving lens 16, and can essentially be determined as a function of the interval (Xd-Xe) between the corneal reflection images Pd' and Pe'.

[0033] Furthermore, in step S005, the X-coordinate of the midpoint of the corneal reflection images Pd and Pe and the X-coordinate of the curvature center O of the cornea 142 are approximately coincident. Therefore, if we denote the standard distance from the curvature center O of the cornea 142 to the center c of the pupil 141 as Oc, then the rotation angle θ of the optical axis of the eyeball 14 in the ZX plane is... X teeth, β*Oc*SINθ X ≈{(Xd+Xe) / 2}-Xc This can be determined from the given relationship. Furthermore, in Figures 5 and 6, the rotation angle θ is shown when the observer's eyeball rotates in a plane perpendicular to the Y-axis. X The example shown illustrates how to calculate the rotation angle θy when the observer's eyeball rotates in a plane perpendicular to the X-axis.

[0034] In the previous step, the rotation angles θx and θy of the optical axis of the observer's eyeball 14 are calculated. In step S006, θx and θy are used to determine the position of the observer's line of sight on the display element 10 (the position of the point of fixation; hereinafter referred to as the fixation point). Assuming that the fixation point position corresponds to the coordinates (Hx, Hy) of the center c of the pupil 141 on the display element 10, Hx = m × (Ax × θx + Bx) Hy = m × (Ay × θy + By) This can be calculated as follows. At this time, the coefficient m is a constant determined by the configuration of the camera's viewfinder optical system, and is a conversion coefficient that converts the rotation angles θx and θy into position coordinates corresponding to the center c of the pupil 141 on the display element 10, and is assumed to be predetermined and stored in the memory unit 4. In addition, Ax, Bx, Ay, and By are gaze correction coefficients that correct for individual differences in the observer's gaze, and are obtained by performing a calibration operation and are assumed to be stored in the memory unit 4 before the gaze detection routine starts.

[0035] After calculating the coordinates (Hx, Hy) of the center c of the pupil 141 on the display element 10 as described above, in step S007, the above coordinates (hereinafter referred to as the gaze point position coordinates) and the time at which the eyeball image signal was acquired (hereinafter referred to as the gaze detection time) are stored in the memory unit 4, and the gaze detection routine is completed.

[0036] The above describes a method for acquiring the coordinates of the point of gaze on a display element using corneal reflection images from light sources 13a and 13b. However, the present invention is not limited to this method, and any method for acquiring the rotation angle of the eyeball from an captured eyeball image is applicable.

[0037] The above gaze detection routine corresponds to the gaze point position estimation means in claim 1.

[0038] <Explanation of the operation to determine the point of focus> The release button 5 is capable of a two-stage push operation. A half-press operation (hereinafter SW1) of the first stroke instructs AF operation, and a full-press operation (hereinafter SW2) of the second stroke releases the shutter. Before performing AF operation with SW1, the release button 5 includes a function to determine the point of focus by the photographer (hereinafter, gaze confirmation), and AF operation can be performed at the determined point of focus. The point of focus position information determined by gaze confirmation may be the coordinates of the point of focus, or the time when the determined point of focus was detected. It is sufficient that the timing of the detection of the point of focus coordinates by the gaze detection routine and the timing of the point of focus coordinates determined by the photographer can be distinguished. The point of focus position coordinates or confirmation timing determined by gaze confirmation are stored in the memory unit 4.

[0039] Furthermore, the operating members α to γ ​​of 41 to 43 may be assigned a line-of-sight determination function similar to the SW1 operation. The line-of-sight determination function corresponds to the point-of-view position determination means in claim 1.

[0040] <Explanation of the association process for still images or videos> Next, the gaze detection information recording means and the confirmed gaze point information recording means in claim 1 will be explained with reference to Figure 8.

[0041] When gaze detection is initiated, gaze detection is performed in step S101. The gaze detection in step S101 corresponds to steps S001 to S007. When the gaze point position is calculated in S101, the gaze point position coordinates and the gaze detection time are stored in the memory unit 4 in step S102, and the process proceeds to step S103. S102 corresponds to S008 and the gaze detection information recording means.

[0042] If the point of focus is determined in S103 by operating SW1 or by the photographer operating the operating members α to γ ​​of 41 to 43, the point of focus coordinates or line-of-sight detection time determined by the photographer's operation is stored in the memory unit 4 in step S104. S104 corresponds to the confirmed point of focus information recording means. If the point of focus is not determined in S103, the process proceeds to step S105.

[0043] S105 determines whether the current mode is still image shooting mode or video shooting mode. Switching between still image shooting mode and video shooting mode can be done using various operating components used to operate the camera. These various operating components include, for example, operating component α (touch panel compatible LCD), operating component β (lever-type operating component), and operating component γ (button-type directional key), as shown in Figures 2(b) 41-43.

[0044] If still image shooting mode is selected, the process proceeds to S106, where it is determined whether the shutter release was performed using SW2. If the shutter release was not performed, the process returns to S101 and the gaze detection is repeated. If the gaze point position determination operation in S103 is performed multiple times before reaching S105, the gaze point coordinates or gaze detection time at the time of the latest gaze point determination in the memory unit 4 are updated in S104.

[0045] If the release button is pressed in S106, the process proceeds to step S107. In S107, the CPU 3 records the gaze point position coordinates and gaze detection time from S102, and the gaze point coordinates or gaze detection time at the time of gaze point confirmation from S104, along with the captured still image data, onto the recording medium 5.

[0046] If video recording mode is detected in S105, the process proceeds to S108, where it is determined whether video recording has finished. If it has not finished, the process returns to S101 and the gaze detection is repeated. Similar to still image mode, if the gaze position determination operation in S103 is performed multiple times before reaching S105, the gaze position coordinates or gaze detection time in the memory unit 4 at the time of the most recent gaze position determination are updated in S104.

[0047] If video recording is not finished in S108, the process proceeds to step S109. In S109, the CPU 3 records the gaze point position coordinates and gaze detection time from S102, and the gaze point coordinates or gaze detection time at the time of gaze point confirmation from S104, along with the recorded video data, onto the recording medium 5.

[0048] Since steps S101 and S102 are repeated until gaze detection is complete, the coordinates of the gaze point and the gaze detection time are stored in the memory unit 4 as data from multiple past gaze detections. Therefore, when the release button is pressed in S106 or when video recording ends in S108, the results of multiple past gaze detections are recorded in association with the still image capture data.

[0049] <Example 2> In Example 2, as a modification of Example 1, Figure 9 illustrates a method for recording past gaze detection information, which is recorded in step S106 of Figure 8, onto the recording medium 5 in a way that avoids duplication.

[0050] In the aforementioned Embodiment 1, in S107 of Figure 8, the size that can be recorded in association with a still image is usually finite, so past gaze detection information is recorded in a fixed-length size. In this case, past gaze detection information up to the time of release by SW2 operation will be recorded a fixed number of times.

[0051] However, if eye-tracking detection information from multiple past instances is recorded a fixed number of times even during continuous shooting, the eye-tracking detection information from multiple past instances attached to the previous shooting data may be duplicated. For example, if 10 eye-tracking detection results can be attached to one shooting data, and the eye-tracking detection routine can be performed 5 times between continuous shots, then the eye-tracking detection information from the previous and current shooting will be duplicated 5 times. Therefore, the eye-tracking detection information from the past 5 instances in the current shooting image data can also be seen in the previous shooting image, resulting in wasted recording data. Considering the time required to transfer eye-tracking detection information from the memory unit 4 to the recording medium 5 and the recordable capacity of the recording medium 5, it is desirable to keep the information associated with the shooting data as small as possible.

[0052] Figure 9 is explained below.

[0053] Steps S201 to S206 in Figure 9 are equivalent to steps S101 to S106 in Figure 8, so their explanation is omitted. In step S206, the gaze detection information stored in step S202 and the gaze point confirmation information stored in step 204 are added to the captured data, and the process proceeds to step S207. In S207, the gaze point position coordinates and gaze detection times from multiple past instances stored in the memory unit 4 in S202 are erased.

[0054] As shown in Figure 9, when continuous shooting occurs, the gaze detection results detected in the next gaze detection routine are recorded in the cleared memory unit 4. Therefore, the multiple past gaze detection results added to the captured image at the next shutter release will not overlap with the multiple past gaze detection results recorded immediately before.

[0055] Example 2 corresponds to the imaging device described in claim 3.

[0056] <Example 3> In Example 3, as a modification of Examples 1 and 2, the method for storing the gaze point determination information stored in the memory unit 4 at S104 in Figure 8 or S204 in Figure 9 will be explained in Figure 10.

[0057] In Figure 8, S104, or Figure 9, S204, the gaze point coordinates or gaze detection time (hereinafter referred to as gaze confirmation information) stored in the memory unit 4 may overlap with the gaze detection information from multiple past instances stored in Figure 8, S102, or Figure 9, S202. As explained in Example 2, considering the gaze detection information transfer time from the memory unit 4 to the recording medium 5 and the recordable capacity of the recording medium 5, it is desirable to eliminate overlapping gaze detection information as much as possible and record it on the recording medium 5. For this reason, we consider minimizing the overlap of gaze confirmation information with past gaze detection information and adding only the data from which gaze confirmation information can be identified from past gaze detection information to the captured data.

[0058] Figure 10 illustrates the method for storing gaze determination information. Steps S301 to S303 are the same as steps S101 to 103 in Example 1, so their explanation is omitted.

[0059] If the gaze point position is determined in step S303, the process proceeds to step S304, where it is determined whether there is gaze point position information that matches the gaze point position information stored in S302 from multiple past gaze detections. If there is gaze point position information that matches the gaze point position information from multiple past gaze detections in S304, the process proceeds to step S305, where only the gaze detection time at the time of gaze confirmation is stored in the memory unit 4. If the gaze point position information does not match the gaze point position information from multiple past gaze detections in S304, the process proceeds to step S306, where both the gaze point coordinates and the gaze detection time at the time of gaze confirmation are stored in the memory unit 4. After each of steps S305 and S306 is performed, the process proceeds to step S307. Steps S307 to S311 are the same as steps S105 to S109 in Figure 8, so their explanation is omitted. Embodiment 3 corresponds to the imaging device described in claim 4.

[0060] <Example 4> In Example 4, the information recorded by the confirmed gaze point information recording means and the gaze detection information recording means will be described.

[0061] As explained in Example 2, considering the transfer time of gaze detection information from the memory unit 4 to the recording medium 5 and the recordable capacity of the recording medium 5, it is desirable to store as little information as possible in association with the captured data.

[0062] On the other hand, since the confirmed gaze point information to be recorded by the aforementioned confirmed gaze point information recording means is gaze point information at a timing determined by the photographer, there is a desire to record more detailed information about the gaze at the confirmed timing and provide it to a third party or use it for analysis. By recording more detailed information than gaze detection information as confirmed gaze information, the aim is to achieve both minimal information addition to the captured data and detailed information about the confirmed timing.

[0063] Specifically, the confirmed gaze point information and gaze detection information will be as follows, but this is not limited to these terms.

[0064] Confirmed gaze point information: Gaze point coordinates, gaze detection time, degree of gaze variability, blink information, gaze angular velocity, jump (large gaze jump to another subject) detection information. Eye-tracking detection information (retains information from multiple past instances): coordinates of the gaze point, time of eye-tracking detection. <Example 5> Example 5 will be explained using Figure 11.

[0065] As described in Embodiment 1, the fixation point position determination means is used to determine the position that the photographer is fixated on. The fixation point position determined by the fixation point position determination means is recorded on the recording medium 5 together with the shooting data by the fixed fixation point information recording means.

[0066] Since the operation by the gaze-point position determination means is different from the release timing, there is a possibility that the timing at which the gaze-point position determined by the gaze-point position determination means is detected and the timing at which the shooting data is generated may be significantly different. In this case, if the shooting scene changes by panning the camera between the time the gaze is determined and the shutter is released, the scene at the time of gaze determination will not be recorded as shooting data, rendering the gaze determination information useless. For this reason, when the camera is panned, the coordinates and detection time at the time of gaze determination are not recorded in the memory unit 4.

[0067] The flow is explained below in Figure 11.

[0068] Steps S401 to S404 are the same as those in Figure 8 of Example 1, steps S101 to S104, so their explanation is omitted.

[0069] Once step S404 is executed, the process proceeds to S405 to acquire acceleration. Acceleration is acquired from either the accelerometer 6 in the camera or the accelerometer 119 in the lens.

[0070] On the other hand, if the point of focus has not yet been determined, proceed to step S406. Steps S406 and S407 are equivalent to Figure 8, S105, and S106 of Example 1, so their explanation is omitted.

[0071] Step S408 determines whether panning has occurred since the gaze point was determined. Panning from gaze point determination to the present is determined using the acceleration obtained in S405, by checking whether the acceleration is greater than a threshold. If the acceleration is greater than the threshold, it is determined that panning has occurred since the gaze point was determined to the present, and the process proceeds to step S409.

[0072] In S409, only past gaze detection information is associated with still image data and recorded on recording medium 5.

[0073] On the other hand, if it is determined in S408 that the acceleration is less than the threshold, it is determined that no panning has occurred since the fixation point was determined, and the process proceeds to step S410. There, both the gaze detection information from multiple past instances and the fixation point determination information are associated with the still image data and recorded on the recording medium 5.

[0074] If video recording mode is detected in S406, the process proceeds to step S411 to determine whether recording has finished. If recording has not finished in S411, the process returns to S401 and repeats the gaze detection. If recording has finished in S411, the process proceeds to step S412.

[0075] Step S412 determines whether panning has occurred since the gaze point was determined. Panning from gaze point determination to the present is determined using the acceleration obtained in S405, by checking whether the acceleration is greater than a threshold. If the acceleration is greater than the threshold, it is determined that panning has occurred since the gaze point was determined to the present, and the process proceeds to step S413.

[0076] In S413, only past gaze detection information is associated with the video data and recorded on the recording medium 5.

[0077] On the other hand, if it is determined in S412 that the acceleration is less than the threshold, it is determined that no panning has occurred since the fixation point was determined, and the process proceeds to step S414. There, the gaze detection information from multiple past instances and the fixation point determination information are associated with the video data and recorded on the recording medium 5.

[0078] As described above, according to Example 5, meaningful point of focus position information can be recorded for the captured data.

[0079] <Example 6> Example 6 is a modification of Example 5 and corresponds to the imaging device described in claim 7. Example 6 explains whether or not to associate the gaze point determination information with the shooting data depending on whether or not panning is occurring between gaze point determination and shutter release.

[0080] On the other hand, in shooting techniques such as panning mode, where the camera is panned to bring the desired subject into the frame, there is a high probability that the relative position of the point of focus at the time of gaze determination and the subject will not change, even though panning is occurring. In this case, the point of focus determined by the aforementioned point of focus determination means is likely to be meaningful information for the shooting data, and therefore it is necessary to associate it with the shooting data.

[0081] Figure 12 is a flowchart illustrating the association of the gaze point position information with the shooting data when the gaze point position is determined in panning mode. The flowchart is explained below.

[0082] Steps S501 to S507 are the same as Figure 11 and S401 to S407 in Example 5, and step 512 is the same as Figure 11 and S412, so their explanation is omitted.

[0083] If the panning mode is determined in step S508, the process proceeds to step S512, where both the gaze detection information and gaze confirmation information from multiple past instances are associated with the still image data and recorded on the recording medium 5.

[0084] On the other hand, if the camera does not detect the panning mode, the process proceeds to step S509.

[0085] The panning mode can be set using various control elements used to operate the camera, such as control element α (touch panel compatible LCD), control element β (lever-type control element), and control element γ (button-type directional pad), as shown in Figures 2(b) 41-43.

[0086] Since steps S509 to S511 are the same as steps S408 to S410 described in Example 5, their explanation will be omitted.

[0087] If the panning mode is determined in step S513, the process proceeds to step S516, where both the gaze detection information and gaze confirmation information from multiple past instances are associated with the video data and recorded on the recording medium 5.

[0088] On the other hand, if it is not determined to be panning mode, the process proceeds to step S514.

[0089] Since sections S514 to S516 are the same as those described in Example 5 (S412 to S414), their explanation will be omitted.

[0090] As described above, according to Example 6, meaningful point of focus position information can be recorded for the captured data when using panning mode.

[0091] <Example 7> In Example 7, the gaze detection information recording means and the confirmed gaze location information recording means record the detection information recorded in the memory unit 4 in association with still image data or video data only when the gaze location position estimation means is active.

[0092] The enabled / disabled function for estimating the point of focus can be switched using various operating components used for camera operation. These various operating components include, for example, operating component α (touch panel compatible LCD), operating component β (lever-type operating component), and operating component γ (button-type directional pad), as shown in Figures 2(b) 41-43.

[0093] Although the above-described embodiment primarily focused on the camera configuration, the processing described in this embodiment can also be applied to other devices. The device could be, for example, a head-mounted display, or a recording system in which the configuration for gaze detection, image display, fixation point determination, and recording of images and other information are each implemented by separate devices.

[0094] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. [Explanation of symbols]

[0095] 3 CPU 4. Memory section 5. Recording media 10 display elements 14 Eyeball 17. Image sensor for the eye 119 Accelerometer

Claims

1. An eye image sensor that captures an image of the user's eyeball, A calculation means for calculating the position of the user's gaze point on the display unit multiple times based on the eyeball image captured by the eyeball image sensor, An operating means for receiving a first operation to determine the position of the point of focus calculated by the calculation means, A first storage means that stores the position of the point of fixation calculated by the calculation means before the first operation on the operating means by the user, and the position of the point of fixation determined in response to the first operation on the operating means by the user, in association with an image of the subject taken using an image sensor other than the eyeball image sensor after the first operation is performed. An imaging device characterized by having the following features.

2. The imaging device further has a second storage means separate from the first storage means, The second storage means is, The imaging apparatus according to claim 1, characterized in that it stores the position of the point of fixation calculated by the calculation means before the first operation on the operating means by the user, and the position of the point of fixation determined in response to the first operation on the operating means by the user.

3. The second storage means is, At the time before the first operation on the operating means is performed by the user, the position of the point of focus calculated by the calculation means before the first operation on the operating means is recorded. The imaging apparatus according to claim 2, characterized in that, at a timing after the first operation on the operating means by the user is performed, the position of the point of focus determined in response to the first operation on the operating means by the user is stored.

4. The imaging apparatus according to any one of claims 1 to 3, characterized in that the first operation on the operating means is further an operation to cause AF operation to be performed at the determined position of the point of focus.

5. The operating means is capable of receiving a second operation following the first operation. The imaging apparatus according to any one of claims 1 to 4, characterized in that the second operation on the operating means is an operation for releasing the shutter.

6. The first storage means is, The imaging apparatus according to claim 5, characterized in that the position of the point of focus calculated by the calculation means before the first operation on the operating means by the user, and the position of the point of focus determined in response to the first operation on the operating means by the user, are stored in association with the image captured by the second operation after the first operation is performed.

7. The first operation on the operating means is a half-press operation on the release button, The imaging apparatus according to claim 5 or 6, characterized in that the second operation on the operating means is a full press operation on the release button.

8. The second storage means further stores blink information, gaze angular velocity, and jump determination information in response to the first operation performed by the user on the operating means. The imaging apparatus according to claim 2, characterized in that the first storage means stores the blinking information, the line of sight angular velocity, and the jump determination information stored by the second storage means in association with the image taken after the first operation is performed.

9. A calculation step of calculating the position of the user's gaze point on the display unit multiple times based on the eyeball image captured by the eyeball image sensor that captures the user's eyeball image, A first storage step involves storing, in association with an image of the subject taken using an image sensor other than the eyeball image sensor after the first operation is performed, the position of the point of fixation calculated in the calculation step, before the first operation is performed on the operating means that receives the first operation for determining the position of the point of fixation calculated in the calculation step, and the position of the point of fixation determined in response to the first operation being performed on the operating means, A control method for an imaging device, characterized by having the following features.

Images