Line-of-sight detection device, imaging device, line-of-sight detection method, program, and storage medium

Abstract

To accurately correct a difference between a line-of-sight detection position and a position where a user actually gazes.SOLUTION: A line-of-sight detection apparatus according to a technique of the present disclosure includes: line-of-sight detection means for detecting a position of a line of sight of a user with respect to display means; subject detection means for detecting a subject from an image displayed in the display means; and correction value determination means for determining a correction value used by the line-of-sight detection means such that the position of the line of sight detected by the line-of-sight detection means matches a position of the subject detected by the subject detection means.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The technology disclosed in this case relates to a gaze detection device, an imaging device, a gaze detection method, a program, and a storage medium, and particularly relates to a technology for correcting the difference between the detected gaze position (detection position) and the position where the user is gazing (actual position).

Background Art

[0002] In recent years, the automation and intelligentization of imaging devices have advanced, and devices have been proposed that select positions in the space in the direction the user is looking, positions on the screen, etc. based on the gaze information (gaze information) of the user looking through the viewfinder without manually inputting the position. In addition, a technology for correcting the difference between the detected gaze position and the position where the user is actually gazing is shown in Patent Document 1.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in Patent Document 1, since the user himself designates the position where the user is actually gazing and corrects the detected gaze position based on the designated position information, the user has to perform the correction manually.

[0005] In addition, in Patent Document 2, when there is an error between the detected gaze position and the user's gazing position, a correction method for selecting an appropriate subject from candidates of subjects existing in the surroundings is shown, but the correction of the gaze position itself with an error is not performed. Hereinafter, the detected gaze position is referred to as the gaze detection position.

[0006] The object of the technology disclosed herein has been made in view of the above, and it is to provide an eye gaze detection device that accurately corrects the error between the position where the user is gazing and the eye gaze detection position.

Means for Solving the Problems

[0007] The eye gaze detection device according to the technology disclosed herein an eye gaze detection means for detecting the position of the user's eye gaze with respect to the display means, a subject detection means for detecting a subject from an image displayed on the display means, correction value determination means for determining a correction value used in the eye gaze detection means so that the position of the eye gaze detected by the eye gaze detection means matches the position of the subject detected by the subject detection means has and when a plurality of subjects are detected, the correction value determination means determines the correction value so that the position of the line of sight detected by the line of sight detection means matches any one of the plurality of subjects, and when the degree of coincidence between the position of the line of sight detected by the line of sight detection means and any one of the subjects does not satisfy a predetermined condition after the elapse of a predetermined time, the correction value determination means determines the correction value so that the position of the line of sight detected by the line of sight detection means matches a subject different from any one of the plurality of subjects

Figure 1

Effects of the Invention

[0008] According to the technology disclosed herein, it is possible to accurately correct the difference between the eye gaze detection position and the position where the user is actually gazing without the user manually performing correction processing.

Brief Description of the Drawings

[0009]

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

[0010] Hereinafter, preferred embodiments of the technology disclosed in this case will be described with reference to the drawings. However, the dimensions, materials, shapes, and relative arrangements of the components described below, etc., should be appropriately changed according to the configuration of the apparatus to which the invention is applied and various conditions. Therefore, it is not intended to limit the scope of this invention to the following description. In particular, well-known techniques or publicly known techniques in the relevant technical field can be applied to configurations and processes that are not particularly illustrated or described. Also, duplicate explanations may be omitted.

[0011] (First Embodiment) Hereinafter, the first embodiment of the technology disclosed in this case will be described.

[0012] <Description of Configuration> Figures 1A and 1B show the appearance of a camera 1 (digital still camera; interchangeable-lens camera) according to the first embodiment. Figure 1A is a front perspective view of the camera 1, and Figure 1B is a rear perspective view of the camera 1. As shown in Figure 1A, the camera 1 has a photographing lens unit 1A and a camera body 1B. A release button 5, which is an operation member for receiving a photographing operation from a user (photographer), is arranged on the camera body 1B. As shown in Figure 1B, an eyepiece lens 12 (eyepiece optical system) for the user to look into a display device 10 (electronic viewfinder), which will be described later and is included in the camera body 1B, is arranged on the rear surface of the camera body 1B. Note that the eyepiece optical system may include a plurality of lenses. Operation members 41 to 43 for receiving various operations from the user are also arranged on the rear surface of the camera body 1B. For example, the operation member 41 is a touch panel that receives touch operations, the operation member 42 is an operation lever that can be pushed down in each direction, and the operation member 43 is a four-way key that can be pushed in each of the four directions. The operation member 41 (touch panel) includes a display panel such as a liquid crystal panel and has a function of displaying an image on the display panel.

[0013] Figure 2 is a cross-sectional view of the camera 1 cut along the YZ plane formed by the Y-axis and the Z-axis shown in Figure 1A, showing a rough internal configuration of the camera 1.

[0014] As shown in FIG. 2, the imaging lens unit 1A includes two lenses 101 and 102, a diaphragm 111, a diaphragm driving unit 112, a lens driving motor 113, and a lens driving member 114. Further, the imaging lens unit 1A includes a photocoupler 115, a pulse plate 116, a mount contact 117, a focus adjustment circuit 118, etc. The lens driving member 114 is composed of a driving gear, etc. The photocoupler 115 detects the rotation of the pulse plate 116 interlocked with the lens driving member 114 and transmits it to the focus adjustment circuit 118. The focus adjustment circuit 118 drives the lens driving motor 113 based on the information from the photocoupler 115 and the information from the camera body 1B (information on the lens driving amount), and moves the lens 101 to change the focusing position. The mount contact 117 is an interface between the imaging lens unit 1A and the camera body 1B. For simplicity, two lenses 101 and 102 are shown, but actually, more than two lenses are included in the imaging lens unit 1A.

[0015] The camera body 1B includes an imaging element 2, a CPU 3, a memory unit 4, a display device 10, a display device driving circuit 11, etc. The imaging element 2 is arranged on the planned imaging plane of the imaging lens unit 1A. The CPU 3 is the central processing unit of a microcomputer and controls the entire camera 1. The memory unit 4 stores images captured by the imaging element 2, etc. The display device 10 is composed of liquid crystal, etc., and is a display means for displaying an image of the photographed subject, etc. on the display surface of the display device 10. The display device driving circuit 11 drives the display device 10. The user can view the image (such as the image captured by the imaging element 2) displayed on the display surface of the display device 10 through the eyepiece 12.

[0016] Inside the camera housing 1B, there are also included a light source 13a to 13f, a beam splitter 15, a light receiving lens 16, an eye imaging element 17, and the like. The light sources 13a to 13f are light sources for illuminating the user's eyeball 14. The light sources 13a to 13f have been conventionally used in a single-lens reflex camera or the like to detect the line-of-sight direction (the direction of the line of sight; the direction the user is looking) from the relationship between the reflected image (corneal reflex image; Purkinje image) due to the corneal reflex of light and the pupil. Specifically, the light sources 13a to 13f are infrared light-emitting diodes or the like that emit infrared light insensitive to the user, and are arranged around the eyepiece lens 12. The optical image of the illuminated eyeball 14 (eyeball image; an image formed by the reflected light emitted from the light sources 13a to 13f and reflected by the eyeball 14) passes through the eyepiece lens 12 and is reflected by the beam splitter 15. Then, the eyeball image is formed on the eye imaging element 17 in which a two-dimensional array of photoelectric elements such as a CCD or a CMOS is arranged by the light receiving lens 16. The light receiving lens 16 positions the pupil of the eyeball 14 and the eye imaging element 17 in a conjugate imaging relationship. Based on a predetermined algorithm described later, the line-of-sight direction of the eyeball 14 is detected from the position of the corneal reflex image in the eyeball image formed on the eye imaging element 17.

[0017] Figure 3 is a block diagram showing the electrical configuration inside the camera 1. Connected to the CPU 3 are a line-of-sight detection circuit 201, a photometry circuit 202, an autofocus detection circuit 203, a signal input circuit 204, a display device drive circuit 11, a light source drive circuit 205, and the like. Further, the CPU 3 transmits signals to the focus adjustment circuit 118 arranged inside the photographic lens unit 1A and the aperture control circuit 206 included in the aperture drive unit 112 inside the photographic lens unit 1A via the mount contact 117. The memory unit 4 associated with the CPU 3 has a function of storing imaging signals from the imaging element 2 and the eye imaging element 17, and a function of storing line-of-sight correction parameters for correcting individual differences in the line of sight, which will be described later.

[0018] The line-of-sight detection circuit 201 A / D-converts the output of the eye imaging element 17 (the eye image obtained by imaging the eye) in a state where the eye image is formed on the eye imaging element 17, and transmits the result to the CPU 3. The CPU 3 extracts feature points necessary for line-of-sight detection from the eye image according to a predetermined algorithm described later, and calculates the user's viewpoint (line-of-sight position; the position where the line of sight is directed; the position where the user is looking) on the display surface of the display device 10 from the positions of the feature points. Thereby, the CPU 3 detects the user's line-of-sight position with respect to the display device 10. In the following description, the detected line-of-sight position is referred to as the line-of-sight detection position.

[0019] The photometry circuit 202 performs amplification, logarithmic compression, A / D conversion, etc. on the signal obtained from the imaging element 2 which also serves as a photometry sensor, specifically, the luminance signal corresponding to the brightness of the subject, and sends the result to the CPU 3 as subject luminance information.

[0020] The autofocus detection circuit 203 A / D-converts the signal voltages from a plurality of detection elements (a plurality of pixels) included in the imaging element 2 and used for phase difference detection, and sends them to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point from the signals of the plurality of detection elements. This is a known technique called imaging surface phase difference AF. In the first embodiment, as an example, in the finder internal field of view (the field of view when looking through the finder), specifically, it is assumed that there are focus detection points at 180 locations on the imaging surface corresponding to 180 locations shown on the display surface of the display device 10.

[0021] The signal input circuit 204 is connected to a switch SW1 and a switch SW2. The switch SW1 is turned on by the first stroke of the release button 5, and is a switch for starting photometry, distance measurement, line-of-sight detection operations, etc. of the camera 1. The switch SW2 is turned on by the second stroke of the release button 5, and is a switch for starting the shooting operation. The ON signals from the switches SW1 and SW2 are input to the signal input circuit 204 and transmitted to the CPU 3. The subject detection circuit 207 can detect a person, an animal, or other specific subjects from the imaging signals from the imaging device 2 and the eyeball imaging device 17 stored in the memory unit 4.

[0022] FIG. 4 is a diagram for explaining an offset, which is the difference between the position where the user is actually gazing at a subject existing within the angle of view while looking through the viewfinder and the line-of-sight detection position detected by the CPU 3. The user is gazing at the subject 401 within the angle of view, but the line-of-sight detection circuit 201 detects the line-of-sight detection position 402 indicated by the solid line frame as the user's line-of-sight position due to a detection error. At this time, the position offset 403 becomes the offset between the actually gazed position and the detected position. In the present embodiment, the purpose is to automatically detect and correct this offset. Also, in offset correction, the position of the subject detected from the image by the subject detection circuit 207 is used.

[0023] FIG. 5A is a diagram for explaining a method of correcting the line-of-sight detection position after a detected subject used for offset correction in the present embodiment is determined. FIG. 5B is a diagram for explaining a method of determining which detected subject to use for correction when there are a plurality of candidates for the detected subject used for offset correction.

[0024] Here, with reference to FIGS. 5A and 5B and the flowchart of FIG. 6, the processing executed by the CPU 3 of the camera 1 in the present embodiment will be described. The CPU 3 controls the operations of components such as each circuit in the camera 1, and realizes the functions of the line-of-sight detection means, the subject detection means, and the correction value determination means in the following processing. It is assumed that the display device 10 does not display the dotted line indicating the line-of-sight detection position 501 and the subject detection position 502 illustrated in FIG. 5A, the arrow indicating the offset 503, and the solid line indicator indicating the corrected line-of-sight detection position 504. Similarly, it is assumed that the display device 10 does not display the dotted line of the area surrounding the subjects 508 to 510, the arrow indicating the maximum offset radius 506, and the dotted circle indicator indicating the maximum offset area 507 illustrated in FIG. 5B. However, the display device 10 may be configured to display at least one or more of these indicators.

[0025] In step S601, the CPU 3 determines the degree of coincidence between the user's line-of-sight detection position with respect to the display device 10 and the position of the detected subject existing in the captured image detected by the subject detection circuit 207. When the line-of-sight detection position is corrected by the following processing, the processing of this step is executed with respect to the corrected line-of-sight detection position. In this determination of the degree of coincidence, the degree of coincidence indicating how much the positions of the line-of-sight detection position and the detected subject coincide within a predetermined fixed period is calculated, and the calculated degree of coincidence is compared with a threshold value. Here, as an example of the degree of coincidence between the line-of-sight detection position and the position of the detected subject, the distance between the center of the area indicating the line-of-sight detection position and the center of the area indicating the position of the detected subject can be mentioned. In this case, the smaller the distance, the higher the degree of coincidence. Note that as long as the degree of coincidence can calculate the degree of approximation between the line-of-sight detection position and the position of the detected subject using a well-known technique, any calculation method and calculation result may be adopted. Also, here, when there are a plurality of detected subject candidates in the image, it is assumed that the degree of coincidence between the position of any one detected subject and the line-of-sight detection position may be calculated. As an example, the CPU 3 selects the position of the detected subject with the shortest distance from the line-of-sight detection position in the image and calculates the degree of coincidence. The CPU 3 determines that the calculated degree of coincidence is greater than the threshold value (S601: NO), the process proceeds to step S609. Further, when the degree of coincidence is equal to or less than the threshold value (S601: YES), the CPU 3 proceeds the process to step S602.

[0026] In step S602, the CPU 3 specifies a detected subject that exists within a region determined based on the line-of-sight detection position detected by the line-of-sight detection circuit 201 among the detected subjects detected by the subject detection circuit 207. As shown in FIG. 5B, the line-of-sight detection position of the user detected by the line-of-sight detection circuit 201 before the correction value of the line-of-sight detection position described later is updated is the line-of-sight detection position 505 before correction. The region within a circle having a radius of a maximum offset radius 506 determined in advance from the center of the line-of-sight detection position 505 before correction is the maximum offset region 507. The CPU 3 determines whether or not there is a detected subject detected by the subject detection circuit 207 within the maximum offset region 507. Note that the maximum offset region is an example of a predetermined region within the image.

[0027] The maximum offset radius 506 is determined in advance based on information regarding the maximum offset generated by the line-of-sight detection circuit 201. The maximum offset radius can be calculated, for example, by a function that calculates an appropriate maximum offset radius based on statistics regarding the distance between the line-of-sight detection positions detected by the line-of-sight detection circuit 201 in the past and the positions of the detected subjects being gazed at by the user. In the present embodiment, it is assumed that three subjects 508 to 510 exist as detected subjects within the maximum offset region 507. Subject 508 is a bird, subject 509 is a dog, and subject 510 is a person. Note that since the dog subject 510 and the bird subject 511 existing within the angle of view are outside the maximum offset region 507, they are excluded from the detected subjects. Further, the CPU 3 excludes detected subjects within the maximum offset region 507 that are associated with attribute information indicating that they have been used for calculating the offset. Note that this attribute information is information associated with each subject in step S605 described later. Thereby, even if the correction value is updated in step S607 described later and the degree of coincidence becomes equal to or less than the threshold value in step S601, that is, when an incorrect detected subject is specified, the same incorrect detected subject is prevented from being specified again in step S602.

[0028] Next, in step S603, the CPU 3 determines whether or not one or more detected subjects (the detected subjects specified in step S602) detected by the subject detection circuit 207 exist within the maximum offset region 507. If the CPU 3 determines that no detected subject exists within the maximum offset region 507 (S603: N), the process ends. If the CPU 3 determines that a detected subject exists within the maximum offset region 507 (S603: Y), the process proceeds to step S604.

[0029] In step S604, the CPU 3 determines a subject that is a candidate for the line-of-sight position from a plurality of detected subjects existing within the maximum offset area 507 based on the shooting mode used for shooting, the distance between the line-of-sight detection position and the detected subject, and the like. Here, the shooting mode is a mode for performing shooting, and a shooting mode (various parameters corresponding to the subject) corresponding to the subject is set by the user operating the camera 1 to select it. The shooting modes include, for example, a shooting mode for shooting a person and a shooting mode for shooting a bird. Note that the information indicating the shooting mode can also be said to be information indicating which type of subject (such as a person or a bird) among the types of detected subjects is to be prioritized. For example, in the case of a shooting mode for shooting a person, a subject corresponding to a person among the plurality of detected subjects is preferentially determined as the detected subject. In the examples shown in FIGS. 5A and 5B, it is assumed that the user has selected a shooting mode for shooting a bird. The CPU 3 comprehensively determines the selected shooting mode, the detected subject closest to the line-of-sight detection position, and the like, and here, selects the subject 508 as the candidate subject for the line-of-sight position.

[0030] Next, in step S605, the CPU 3 associates the subject 508 selected as the candidate subject for the line-of-sight position with attribute information indicating that the subject 508 has been used for the process of calculating the offset. The CPU 3 stores the attribute information in the memory unit 4.

[0031] Next, in step S606, the CPU 3 calculates an offset, which is an error in line-of-sight detection, from the difference between the line-of-sight detection position detected by the line-of-sight detection circuit 201 and the position of the candidate subject for the line-of-sight position selected in step S604. The details of this process will be described with reference to FIG. 5A.

[0032] As shown in FIG. 5A, there are a pre-correction line-of-sight detection position 501 indicated by a dotted line frame and a subject detection position 502 indicated by a dotted line frame, and the positional difference between the pre-correction line-of-sight detection position 501 and the subject detection position 502 is an offset 503. Also, the subject detection position 502 corresponds to the position of the subject 508 in FIG. 5B. As an example, the CPU 3 calculates, as an offset, the distance between the center of the area of the dotted line frame of the line-of-sight detection position 501 and the center of the area of the dotted line frame of the subject detection position 502.

[0033] Next, in step S607, the CPU 3 determines a correction value used when the line-of-sight detection circuit 201 detects the line-of-sight detection position of the user of the camera 1 by using the offset 503 calculated in step S606, and updates the existing correction value.

[0034] By updating the correction value in step S607, the CPU 3 corrects the line-of-sight detection position 501 that the line-of-sight detection circuit 201 detected before correction using the offset to a new position 504 (post-correction line-of-sight detection position) by the updated correction value. In the example of FIG. 5A, the pre-correction line-of-sight detection position 501 indicated by the dotted line frame is changed to the post-correction line-of-sight detection position 504 indicated by the solid line frame. As a result, the post-correction line-of-sight detection position 504 comes to match the subject detection position 502, and in the display device 10, the post-correction line-of-sight detection position 504 is displayed as the line-of-sight detection position.

[0035] Next, when the process of step S607 is completed, the CPU 3 advances the process to step S608. In step S608, the CPU 3 determines whether or not a predetermined time has elapsed. When the predetermined time has elapsed (S608: Y), the CPU 3 returns the process to step S601 and executes the process of step S601 again. By returning the process from step S607 to step S601 after the elapse of the predetermined time, in the example of FIG. 5B, if the subject that the user was gazing at was not the subject 508, the degree of coincidence may become equal to or less than the threshold value in S601. Reasons for the degree of coincidence becoming equal to or less than the threshold value include that the line-of-sight detection position does not follow the subject 508. Then, in step S602, the CPU 3 refers to the attribute information associated in step S605, excludes the subject 508 from the detected subjects, and then repeats the above process for the other detected subjects to update a new correction value. Here, as the predetermined time used for the determination in step S608, for example, the time during which an average user continues to gaze at a subject can be adopted.

[0036] According to the present embodiment, without the user manually correcting the line-of-sight detection position, an error, which is the deviation between the line-of-sight detection position and the gazing position, for each user is calculated based on the difference between the line-of-sight detection position and the position of a subject within a circle having the maximum offset radius in the image. Then, by using the calculated error, the error between the line-of-sight detection position and the position that the user is actually gazing at can be automatically corrected. Note that there may be individual differences in the error. In that case, the CPU 3 may store the correction value calculated by the above process for each user in the memory unit 4.

[0037] (Second Embodiment) Next, a second embodiment of the present disclosure will be described with reference to the drawings. In the following description, the same reference numerals are given to the same configurations as those in the first embodiment, and detailed descriptions thereof are omitted.

[0038] FIG. 7 is a diagram schematically showing the calculation of correction values used when the gaze detection circuit 201 in the present embodiment detects the gaze detection position of the user of the camera 1. In the example shown in FIG. 7, there are subjects bird 711, 712, dog 713, 714 existing in the image displayed by the display device 10, and it is assumed that the user is gazing at the dog 714. In FIG. 7, the range indicated by the subject detection frame 702 for the detected subject (dog 714) detected by the subject detection circuit 207 is larger than a predetermined size. Here, the predetermined size of the subject occupying the angle of view may be determined as appropriate. In this case, even if an attempt is made to specify an offset based on the difference between the gaze detection position 701 before correction and the position of the detected subject, the camera 1 cannot specify which part of the subject 714 within the subject detection frame 702 the user is gazing at, so the offset cannot be specified. For example, depending on the characteristics of the user and the situation at the time of shooting, etc., the gaze position by the user may be different, such as the gaze candidate position 703 which is the eye part of the dog that is the subject 714, the gaze candidate position 704 corresponding to the center of the dog's body, the gaze candidate position 705 which is the front paw part of the dog, etc. Therefore, in the present embodiment, the CPU 3 executes the following processing to accurately specify which part of the subject 714 the user is gazing at and appropriately update the correction value of the gaze detection position.

[0039] Here, with reference to FIGS. 8A to 8D and the flowchart of FIG. 9, the processing executed by the CPU 3 of the camera 1 in the present embodiment will be described. In FIGS. 8A to 8D, for convenience of explanation, the subjects 811 to 813 are shown omitted.

[0040] In step S901, the degree of coincidence between the gaze detection position detected by the gaze detection circuit 201 and the position of the detected subject existing in the captured image detected by the subject detection circuit 207, or the gaze candidate position, is determined. Details of the gaze candidate position will be described later with reference to FIG. 10.

[0041] The processes of steps S902 to S905 are the same as the processes of steps S602 to 605 in the flowchart of FIG. 6, so the details are omitted. Next, in step S910, the CPU 3 determines whether the size of the subject that is a candidate for the line-of-sight position is equal to or greater than a predetermined threshold value. Here, as an example, the size of the subject is the size of the subject detection frame of the subject within the angle of view determined using a well-known technique. In the example shown in FIG. 7, when the size of the subject detection frame 702 is equal to or greater than a certain threshold value, it is considered inappropriate to narrow down the gaze position of the user to one position for the subject. When the CPU 3 determines that the size of the subject is equal to or greater than the threshold value (S910: YES), the process proceeds to step S911. Also, when the CPU 3 determines that the size of the subject is less than the predetermined threshold value (S910: NO), the process proceeds to step S906. When the CPU 3 proceeds with the process to step S906, it performs the same process as in the first embodiment. Since the processes of steps S907, S908, and S909 are the same as the processes of steps S607, S608, and S609 in FIG. 6, the details of the processes are omitted here.

[0042] In step S911, according to the line-of-sight candidate position table, the position of the line-of-sight candidate in the subject that is a candidate for the line-of-sight position is selected. Here, the position of the line-of-sight candidate and the line-of-sight candidate position table will be described with reference to FIGS. 10A and 10B. FIG. 10A is an example of the line-of-sight candidate position table, and FIG. 10B shows the specific position of the line-of-sight candidate in the subject. As an example, it is assumed that the data of the line-of-sight candidate position table is stored in the memory unit 4 of the camera 1. The CPU 3 refers to the line-of-sight candidate position table stored in the memory unit 4 and executes the process of step S911.

[0043] In FIG. 10B, the subject detection frame 1001 is a rectangular frame that surrounds the entire subject 714 detected by the subject detection circuit 207. The eye detection position 1002, the center of gravity position 1003, The front foot position 1004 and the rear foot position 1005 are positions that are within the subject detection frame 1001 and serve as candidates for the line-of-sight positions that the user may be gazing at. Note that the eye detection position 1002, the center-of-gravity position 1003, the front foot position 1004, and the rear foot position 1005 are examples of the characteristic positions of the detected subject. Also, the positions that serve as candidates for the line-of-sight position may be appropriately set according to the type of the detected subject. As the type of the subject, categories for classifying the subject such as animals, people, vehicles, plants, and structures can be adopted.

[0044] The eye detection position 1002 is detected by the organ detection process of the subject detection circuit 207 when the subject is a living being such as an animal or a person. Also, the center-of-gravity position 903 is detected as the geometric center-of-gravity position of the subject detection frame 901 by the subject detection circuit 207. Note that the subject detection frame 901 is an example of the subject area. Here, the subject detection frame 901 is a rectangular area surrounding the subject, but it may be an area surrounding only a part of the subject, not limited to the whole subject, or an area of any shape, not limited to a rectangle. And not only the center-of-gravity position of the subject detection frame 901, but positions that can be specified according to the shape of the subject detection frame 901 may be regarded as candidates for the detection position. The front foot position 1004 and the rear foot position 1005 are detected by the organ detection process of the subject detection circuit 207 and specific processes such as the positional relationship between the front foot position 1004 and the rear foot position 1005 within the subject detection frame 1001. Note that the organ detection process by the subject detection circuit 207, the detection process of the geometric center-of-gravity position of the subject detection frame, and specific processes such as the positional relationship between the front foot position 1004 and the rear foot position 1005 are realized using well-known techniques, so the details of the processes are omitted here.

[0045] As shown in FIG. 10A, in the line-of-sight candidate position table, for each of the four line-of-sight positions of the eye detection position 1002, the center-of-gravity position 1003, the front foot position 1004, and the rear foot position 1005 detected by the subject detection circuit 207 for the subject 714, the priority of the detection position candidates is set. In the example shown in FIG. 10A, the smaller the priority value, the higher the priority as a detection position candidate. Therefore, according to the line-of-sight candidate position table in FIG. 10A, the priority (priority) is lower in the order of the eye position, the center-of-gravity position, the front foot position, and the rear foot position of the subject 714. For this reason, the CPU 3 refers to the line-of-sight candidate position table, first uses the eye position of the subject 714 as a candidate for the user's line-of-sight position, and updates the correction value for line-of-sight detection.

[0046] The processing of the following steps will be described with reference to FIGS. 8A to 8D. FIG. 8A assumes a case where the user is gazing at the eyes of the subject 714, and FIG. 8B assumes a case where the user is gazing at the center-of-gravity position of the subject 714. First, the processing executed by the CPU 3 in the case of FIG. 8A will be described.

[0047] In the case of FIG. 8A, the detected subject 714 is surrounded by the subject detection frame 801. Also, it is assumed that the user is gazing at the gaze position 802 indicated by the dotted line frame corresponding to the eyes of the subject 714. The line-of-sight detection position detected by the line-of-sight detection circuit 201 before correction is the line-of-sight detection position 803 before correction indicated by the solid line frame. Also, the eyes of the subject 714 are within the maximum offset region 804 within the circle centered on the line-of-sight detection position 803 before correction, and in the line-of-sight position candidate table in FIG. 10A, the eyes have the highest priority (priority = 1).

[0048] Therefore, in step S911, the CPU 3 refers to the line-of-sight candidate position table and selects the eye detection position 805 of the subject 714 detected by the subject detection circuit 207 as the line-of-sight candidate position to be used for the correction process of the line-of-sight detection position 803. Then, in step S912, the CPU 3 calculates the offset 806 from the difference in the positions of the line-of-sight detection position 803 before correction and the eye detection position 805.

[0049] Next, in step S913, the CPU 3 updates the correction value for gaze detection of the gaze detection circuit 201 using the offset 806 calculated in step S911. As a result, in the camera 1, the gaze detection position detected by the gaze detection circuit 201 is corrected by the offset 806, and the corrected gaze detection position is obtained. As a result, in the camera 1, the gaze detection position detected by the gaze detection circuit 201 is corrected by the offset 806, and the corrected gaze detection position is obtained.

[0050] Next, in step S914, the CPU 3 determines whether or not a predetermined time has elapsed. When the predetermined time has elapsed (S914: YES), the CPU 3 advances the process to step S915.

[0051] Next, in step S915, the CPU 3 determines the degree of coincidence between the gaze detection position corrected using the correction value updated in step S913 and the gaze candidate position 805. This determination is a process of determining how much the corrected gaze detection position and the gaze candidate position coincide within a predetermined fixed time. Note that the coincidence of their positions includes not only the case where they completely coincide, but also the case where they are strictly separated but can be regarded as approximately coinciding.

[0052] The details of this process will be described with reference to FIG. 8C. FIG. 8C shows an example of a state in which the dog, which is the subject 714 shown in FIG. 8A, has moved, its orientation has been reversed left and right, and its posture has changed. In the present embodiment, when the state of the subject changes, the validity of the correction value updated in step S913 is determined based on the degree of coincidence between the corrected gaze detection position and the gaze candidate position in each state.

[0053] In the example shown in FIG. 8C, when the user is gazing at the eyes of the subject 714, the detected subject 714 is surrounded by a rectangular subject detection frame 807. Also, the user continues to gaze at the gaze position 808 indicated by the dotted line frame corresponding to the eyes of the subject 714 from the state of FIG. 8A. The gaze detection position detected by the gaze detection circuit 201 is the gaze detection position 809 indicated by the dotted line frame. Also, the position where the offset 810 (the same as the offset 806 in FIG. 8A) is applied using the correction value updated in step S913 is the corrected gaze detection position 811 indicated by the solid line frame.

[0054] Also, in the example of FIG. 8C, the gaze candidate position is the eye detection position 812 detected by the subject detection circuit 207. Therefore, even when the state of the subject 714 changes from the state of the subject 714 in FIG. 8A to the state of the subject 714 in FIG. 8C, the three positions of the gaze position 808 at which the user is gazing, the corrected gaze detection position 811, and the eye detection position 812 match. In step S915, the CPU 3 calculates the degree of coincidence between the corrected gaze detection position 811 and the eye detection position 812. As an example, the CPU 3 calculates the degree of coincidence based on the distance between the center of the region of the gaze detection position 811 indicated by the dotted line frame and the center of the region of the eye detection position 812 indicated by the dotted line frame. In this case, the CPU 3 calculates the degree of coincidence such that the degree of coincidence increases as the distance between the center of the region of the gaze detection position 811 and the center of the region of the eye detection position 812 becomes shorter. Note that since this degree of coincidence indicates the degree to which the two positions match and can be calculated using well-known techniques, a detailed description of the calculation of the degree of coincidence will be omitted.

[0055] Next, in step S916, the CPU 3 determines whether or not the degree of coincidence calculated in step S914 is equal to or greater than a predetermined threshold value. If the degree of coincidence is equal to or greater than the predetermined threshold value (S916: YES), the CPU 3 ends the processing of the flowchart in FIG. 9. If the degree of coincidence is less than the predetermined threshold value (S916: NO), the processing proceeds to step S917. In the example shown in FIG. 8C, the degree of coincidence is equal to or greater than the threshold value, and the CPU 3 ends the processing of the flowchart in FIG. 9.

[0056] Next, the processing executed by the CPU 3 in the case of FIG. 8B will be described. In the case of FIG. 8B, the detected subject 714 is surrounded by the subject detection frame 813. Also, it is assumed that the user is gazing at the gaze position 814 indicated by the dotted line frame corresponding to the centroid position of the subject 714. Before correction, the gaze detection position detected by the gaze detection circuit 201 is the pre-correction gaze detection position 815 indicated by the solid line frame.

[0057] Also, the centroid position of the subject 714 is within the maximum offset region 816 that is within the circle centered on the pre-correction gaze detection position 815. Further, in the gaze position candidate table of FIG. 10A, the eye part (priority = 1) with a higher priority than the centroid position (priority = 2) is also within the region 816. Therefore, in step S911, the CPU 3 refers to the gaze candidate position table and selects the eye part detection position 817 of the subject 714 detected by the subject detection circuit 207 as the gaze candidate position to be used for the correction process of the gaze detection position 815.

[0058] Next, in step 912, the CPU 3 calculates the offset 819 from the difference in the positions between the pre-correction gaze detection position 815 and the eye part detection position 817. Next, in step S913, the CPU 3 updates the correction value for gaze detection of the gaze detection circuit 201 using the offset 819 calculated in step S911. As a result, in the camera 1, the position where the gaze detection position detected by the gaze detection circuit 201 is corrected by the offset 819 becomes the post-correction gaze detection position.

[0059] Next, in step S914, the CPU 3 determines whether or not a predetermined time has elapsed. When the predetermined time has elapsed (S914: YES), the CPU 3 advances the process to step S915.

[0060] Then, in step S915, the CPU 3 calculates the degree of coincidence between the gaze detection position corrected using the correction value updated in step S913 and the gaze candidate position 817, and in step S916, determines the calculated degree of coincidence using a threshold value.

[0061] In the case of FIG. 8B, the position where the user is gazing is the gaze position 814 corresponding to the center-of-gravity position of the subject 714. Therefore, the offset to be adopted in the correction process is the offset 818 corresponding to the difference in position between the pre-correction gaze detection position 815 and the gaze candidate position 814 of the center-of-gravity position. However, since the CPU 3 selects the eye part of the subject 714 as the gaze candidate position 817 according to the priority of the gaze candidate position table, the correction value is updated based on the offset 819 instead of the offset 818.

[0062] FIG. 8D shows an example of a state in which the dog, which is the subject 714 shown in FIG. 8B, has moved, its orientation has been reversed left and right, and its posture has changed. In the example shown in FIG. 8D, it is the case where the user is gazing at the center-of-gravity position of the subject 714, and the detected subject 714 is surrounded by a rectangular subject detection frame 820. Also, the user continues to gaze at the gaze position 821 indicated by the dotted line frame corresponding to the center-of-gravity position of the subject 714 from the state of FIG. 8B. Also, the gaze detection position detected by the gaze detection circuit 201 is the gaze detection position 823 indicated by the dotted line frame. However, the position to which the offset 824 (the same as the offset 819 in FIG. 8B) is applied using the correction value updated in step S913 is the corrected gaze detection position 825 indicated by the solid line frame.

[0063] At this time, the gaze position candidate is the eye part detection position 826 detected by the subject detection circuit 207. Therefore, when changing from the state of the subject 714 in FIG. 8B to the state of the subject 714 in FIG. 8D, the three positions of the gaze position 821 at which the user is gazing, the corrected gaze detection position 825, and the eye part detection position 826 are not in agreement with each other. For this reason, in step S915, the CPU 3 calculates the degree of agreement between the corrected gaze detection position 825 and the eye part detection position 826, and in step S916, the CPU 3 determines that the degree of agreement is less than a predetermined threshold value (S916: NO).

[0064] As a result, the CPU 3 advances the process from step S916 to step S917. Ste In step S917, the CPU 3 refers to the line-of-sight candidate position table and determines whether there is an unused line-of-sight candidate position that was not selected in step S911. If the CPU 3 determines that there is an unused line-of-sight candidate position (S917: YES), the process proceeds to step S911. If the CPU 3 determines that there is no unused line-of-sight candidate position (S917: NO), the process returns to step S901.

[0065] Through the above processing, the eye part (priority = 1) is selected as the line-of-sight candidate position, and the processing from step S911 to step S916 is executed. However, in the line-of-sight candidate position table of FIG. 10A, the center-of-gravity position, the front foot position, and the rear foot position (priorities = 2 to 4) are unused as line-of-sight candidate positions. Therefore, the CPU 3 returns the process from step S917 to step S911, and in step S911, the position with the highest priority among the unused line-of-sight candidate positions (in this case, the center-of-gravity position) is selected as the line-of-sight candidate position.

[0066] With reference to FIGS. 11A and 11B, the processing when the user is gazing at the center-of-gravity position of the subject 714 and the CPU 3 selects the center-of-gravity position of the subject 714 as the line-of-sight candidate position will be described. Note that the examples in FIGS. 11A and 11B correspond to the examples in FIGS. 8A and 8B, respectively.

[0067] In step S912, the CPU 3 calculates an offset 1106 from the difference between the pre-correction line-of-sight detection position 1103 and the center-of-gravity detection position 1105. Next, in step S913, the CPU 3 updates the correction value for line-of-sight detection of the line-of-sight detection circuit 201 using the offset 1106 calculated in step S912. As a result, in the camera 1, the position where the line-of-sight detection position detected by the line-of-sight detection circuit 201 is corrected by the offset 1106 becomes the post-correction line-of-sight detection position.

[0068] Next, in step S914, the CPU 3 determines whether a predetermined time has elapsed. When the predetermined time has elapsed (S914: YES), the CPU 3 proceeds with the process to step S915.

[0069] Next, in step S915, the CPU 3 calculates the degree of coincidence between the gaze detection position corrected using the correction value updated in step S913 and the gaze candidate position 1105, and determines the calculated degree of coincidence using a threshold value. FIG. 11B shows an example of a state in which the dog, which is the subject 714 shown in FIG. 11A, has moved, its orientation has been reversed left and right, and its posture has changed.

[0070] In the example shown in FIG. 11B, the user is gazing at the center-of-gravity position of the subject 714, and the detected subject 714 is surrounded by a rectangular subject detection frame 1107. Also, the user continues to gaze at the gaze position 1108 indicated by the dotted line frame corresponding to the center-of-gravity position of the subject 714 from the state of FIG. 11A. The gaze detection position detected by the gaze detection circuit 201 is the gaze detection position 1109 indicated by the dotted line frame. Also, the position to which the offset 1110 (the same as the offset 1106 in FIG. 11A) is applied using the correction value updated in step S913 is the corrected gaze detection position 1111 indicated by the solid line frame.

[0071] Also, in the example of FIG. 11B, the gaze candidate position is the eye detection position 1112 detected by the subject detection circuit 207. Therefore, even when the state of the subject 714 changes from the state of the subject 714 in FIG. 11A to the state of the subject 714 in FIG. 11B, the three positions of the gaze position 1108 at which the user is gazing, the corrected gaze detection position 1111, and the eye detection position 1112 match.

[0072] In step S914, the CPU 3 determines whether or not a predetermined time has elapsed. When the predetermined time has elapsed (S914: YES), the CPU 3 advances the process to step S915. Then, in step S915, the CPU 3 calculates the degree of coincidence between the corrected gaze detection position 1111 and the eye detection position 1112.

[0073] Next, in step S916, the CPU 3 determines whether the degree of coincidence calculated in step S913 is equal to or greater than a predetermined threshold value. In the example shown in FIG. 11B, the degree of coincidence is equal to or greater than the threshold value (S916: YES), and the CPU 3 ends the processing of the flowchart in FIG. 9.

[0074] In the present embodiment, even when the subject being gazed at by the user within the angle of view of the camera 1 is larger than a predetermined size, the offset between the gaze detection position by the camera 1 and the position within the subject actually being gazed at by the user can be corrected accurately. As a result, in the camera 1, it becomes possible to accurately align the gaze detection position after correction with the user's gaze position.

[0075] The above is the description of the embodiment related to the technology disclosed in this case. However, the above embodiment merely exemplarily explains the configuration example of the technology disclosed in this case. The technology disclosed in this case is not limited to the above specific forms, and various modifications are possible within the scope of its technical idea. For example, in steps S604 and S904, when determining the subject that is a candidate for the gaze position, instead of based on the shooting mode, the distance between the gaze detection position and the detected subject, etc., the camera 1 may be configured to preferentially select the subject specified by the user in advance. As a method of designating in advance the subject that is a candidate for the gaze position for the camera 1, there is a method of storing the identification information (personal information) of the user including the information designating the subject in the memory unit 4. Thereby, the CPU 3 can select, for each user, the subject that is an appropriate candidate for the gaze position based on the identification information stored in the memory unit 4.

[0076] Also, in the above embodiment, the CPU 3 selects the position of the gaze candidate based on the priority of the gaze position candidate table stored in the memory unit 4. Instead of or in addition to this, the CPU 3 may select the position of the gaze candidate based on information about the subject including the size of the detected subject, etc., and image information such as the contrast of the image of the detected subject.

[0077] Also, in the above embodiment, the priority of the line-of-sight position candidate table stored in the memory unit 4 may be changed according to the relative size of the subject area with respect to the angle of view in the image displayed by the display device 10. Further, the priority of the line-of-sight position candidate table may be changed based on information indicating the tendency of the feature position to be gazed at. For example, in the above embodiment, when the information indicates that the area around the front paws of a dog tends to be gazed at, the CPU 3 first selects the position of the area around the front paws of the subject 714 as the line-of-sight candidate position. Also, the information indicating the tendency of the feature position to be gazed at may be common information among a plurality of users or may be user-specific information. For example, the identification information (individual information) of the user including the information indicating the tendency of the feature position gazed at by the user may be stored in the memory unit 4 in advance. Thereby, the CPU 3 changes the priority of the line-of-sight position candidate table using the identification information. Then, the CPU 3 can select the position of the feature of the subject that the user tends to gaze at as the line-of-sight candidate position by selecting the line-of-sight candidate position based on the changed priority. Note that, as an example of the user's identification information (individual information), personal authentication means for authenticating the user is provided in the camera 1, and the personal authentication data used when the user authentication by the personal authentication means is successful can be mentioned. Also, since the personal authentication means can be realized using well-known techniques, detailed description thereof is omitted here.

[0078] Also, in the above embodiment, when reselecting the position of the detected subject in S601, the line-of-sight detection circuit 201 may select the position of the detected subject based on the moving direction of the user's line of sight (line-of-sight detection position) in addition to or instead of the distance from the line-of-sight detection position. For example, in step S601, the CPU 3 in the image displayed on the display device 10 prioritizes and selects the position of the subject moving in substantially the same direction as the moving direction of the user's line of sight as the position of the detected subject. Thereby, the subject gazed at by the user and the detected subject selected in step S601 can be made to match more accurately.

[0079] Further, in the above-described embodiment, at least one position such as the line-of-sight detection position, the subject detection position, the position of the line-of-sight candidate, etc. may be displayed on the display device 10. For example, the display device 10 can display each position by a solid-line or dotted-line rectangle as illustrated in the above description. Also, in the above-described embodiment, the detected subject is not limited to the illustrated animals, and may be other subjects such as a person, an automobile, a motorcycle, a train, etc., or these subjects may be mixed.

[0080] Further, in the above-described embodiment, instead of the CPU 3, the line-of-sight detection circuit 201 may be configured to detect the user's line-of-sight position with respect to the display device 10.

[0081] <<Other Embodiments>> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiment to a system or device via a network or a storage medium, and causing one or more processors in the computer of the system or device to read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Description of Reference Numerals

[0082] 1 Camera, 3 CPU, 10 Display Device, 201 Line-of-Sight Detection Circuit, 207 Subject Detection Circuit

Claims

1. A line-of-sight detection means for detecting the position of a user's line of sight with respect to a display means; A subject detection means for detecting a subject from an image displayed on the display means; A correction value determination means for determining a correction value used in the line-of-sight detection means so that the position of the line of sight detected by the line-of-sight detection means matches the position of the subject detected by the subject detection means and having, wherein the correction value determination means when a plurality of subjects are detected, determines the correction value so that the position of the line of sight detected by the line-of-sight detection means matches any one of the plurality of subjects; after a lapse of a predetermined time, when the degree of coincidence between the position of the line of sight detected by the line-of-sight detection means and any one of the subjects does not satisfy a predetermined condition, determines the correction value so that the position of the line of sight detected by the line-of-sight detection means matches a subject different from any one of the plurality of subjects A line-of-sight detection device characterized by the above.

2. The line-of-sight detection device according to claim 1, wherein when a plurality of subjects are detected from the image, the correction value determination means determines the correction value so that the position of the line of sight detected by the line-of-sight detection means matches any one of the subjects existing in a predetermined region of the image among the plurality of subjects.

3. The line-of-sight detection device according to claim 2, wherein when a plurality of types of subjects are detected from the image, the correction value determination means selects a subject corresponding to the type indicated by the information from the plurality of subjects according to information indicating which type of subject to prioritize.

4. The line-of-sight detection device according to claim 2 or 3, wherein the correction value determination means selects any one of the plurality of subjects based on information specifying the subject used for determining the correction value.

5. The line-of-sight detection device according to claim 4, wherein the correction value determination means selects a subject determined based on the position of the line of sight detected by the line-of-sight detection means.

6. The case where the degree of coincidence does not satisfy the predetermined condition means the case where the degree of coincidence is less than a predetermined threshold value. The line-of-sight detection device according to any one of claims 1 to 5.

7. The subject detection means detects a characteristic position of the detected subject, and the correction value determination means determines the correction value so that the position of the line of sight detected by the line-of-sight detection means matches the characteristic position. The line-of-sight detection device according to any one of claims 1 to 6, characterized in that...

8. The subject detection means detects an organ of the subject when the detected subject is a living being, and detects the position of the detected organ as the characteristic position. The line-of-sight detection device according to claim 7, characterized in that...

9. The subject detection means detects the characteristic position according to the shape of the subject area of the detected subject. The line-of-sight detection device according to claim 7 or 8, characterized in that...

10. The subject detection means uses the geometric centroid position of the subject area as the characteristic position. The line-of-sight detection device according to claim 9, characterized in that...

11. When a plurality of characteristic positions are detected, the correction value determination means selects any one of the plurality of characteristic positions as the characteristic position that aligns the position of the line of sight detected by the line-of-sight detection means according to the priority of each of the plurality of characteristic positions. The line-of-sight detection device according to any one of claims 7 to 10, characterized in that...

12. The priority of each of the plurality of characteristic positions is changed according to the relative size of the subject area with respect to the angle of view in the image. The line-of-sight detection device according to claim 11, characterized in that...

13. The priority of each of the plurality of characteristic positions is changed according to the contrast of the image. The line-of-sight detection device according to claim 11 or 12, characterized in that...

14. The correction value determination means... When detecting a plurality of characteristic positions of the detected subject, the correction value is determined so that the position of the line of sight detected by the line-of-sight detection means matches any one of the plurality of characteristic positions. After the elapse of a predetermined time, when the degree of coincidence between the position of the line of sight detected by the line-of-sight detection means and any one of the characteristic positions is less than a predetermined threshold value, the correction value is determined so that the position of the line of sight detected by the line-of-sight detection means matches a characteristic position different from any one of the plurality of characteristic positions. The line-of-sight detection device according to any one of claims 7 to 13, characterized in that...

15. The correction value determination means determines the correction value so that the position of the line of sight detected by the line-of-sight detection means matches the characteristic position of the subject that the user tends to gaze at. The line-of-sight detection device according to any one of claims 7 to 14, characterized in that...

16. The line-of-sight detection device according to any one of claims 1 to 15, wherein the display means displays an index indicating the position of the line of sight of the user detected by the line-of-sight detection means within the image.

17. The line-of-sight detection device according to any one of claims 1 to 16, wherein the display means displays an index indicating the position of the subject detected by the subject detection means within the image.

18. The line-of-sight detection device according to claim 16 or 17, wherein the display means is an electronic viewfinder through which the user can bring the eyes into contact and which displays the index.

19. An imaging means, The line-of-sight detection device according to any one of claims 1 to 18 and having wherein the line-of-sight detection device controls to display the image captured by the imaging means on the display means, and detects the line of sight of the eye looking at the image. This is a characteristic imaging device.

20. A line-of-sight detection step of detecting the position of the user's line of sight with respect to the display means, A subject detection step of detecting a subject from the image displayed on the display means, A correction value determination step of determining a correction value used in the line-of-sight detection step so that the position of the line of sight detected by the line-of-sight detection step matches the position of the subject detected by the subject detection step and having The correction value determination step is When a plurality of subjects are detected, the correction value is determined so that the position of the line of sight detected by the line-of-sight detection step matches any one of the plurality of subjects, After a lapse of a predetermined time, when the degree of coincidence between the position of the line of sight detected by the line-of-sight detection step and any one of the subjects does not satisfy a predetermined condition, the position of the line of sight detected by the line-of-sight detection step is different from any one of the plurality of subjects. The correction value is determined so as to match the subject. This is a characteristic line-of-sight detection method.

21. A program for causing a computer to function as each means of the line-of-sight detection device according to any one of claims 1 to 18.

22. A computer-readable storage medium storing a program for causing a computer to function as each means of the line-of-sight detection device according to any one of claims 1 to 18.

Images