Electronic device and control method

Abstract

To improve convenience in selecting a subject on the basis of a location of an observer's line of sight.SOLUTION: An electronic device has display means for displaying a captured image, line of sight information acquisition means for acquiring line of sight information such as line of sight location coordinates and line of sight location reliability of an observer observing the display means, limited area calculation means for calculating a first limited area in the captured image by using the line of sight information, first line of sight selection means for selecting the first limited area calculated by the limited area calculation means, and second line of sight selection means for selecting a subject by using the line of sight information out of the first limited area selected by the first line of sight selection means.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an electronic device capable of detecting the line of sight of an observer observing a display unit. [Background technology]

[0002] There are known technologies for detecting the gaze position of a user. Patent Document 1 discloses a technology for detecting the gaze position of a user looking at a display unit and displaying a mark indicating the detected gaze position of the user on the display unit, thereby assisting the user in selecting an item displayed on the display unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2020-119093 Summary of the Invention [Problem to be solved by the invention]

[0004] However, depending on the arrangement of the target to be selected and the accuracy of detecting the gaze position, for example, the target intended by the user may not necessarily be selected.

[0005] Therefore, an object of the present invention is to provide an electronic device and a control method thereof that improve the convenience of selecting an object based on a gaze position. [Means for solving the problem]

[0006] The present invention provides a display device for displaying a captured image, an acquisition device for acquiring line-of-sight information including a line-of-sight position of an observer observing the display device, and a display control device for controlling a display based on the line-of-sight information. a selection means for selecting a subject based on the line-of-sight information, control The means, based on the line of sight information, The aforementioned A limited area of ​​the captured image The limited area is divided into a plurality of areas and displayed as a display that distinguishes it from other areas. Highlight Control it so that The selection means selects a The aforementioned The apparatus is characterized by being configured to select a subject based on information about the viewer's line of sight. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the convenience of selecting an object based on a gaze position. [Brief explanation of the drawings]

[0008] [Figure 1] A block diagram showing the configuration of an imaging device including an electronic device. [Figure 2] FIG. 1 is a diagram showing the correspondence between the pupil plane of a pixel of an imaging device and a photoelectric conversion unit. [Figure 3] FIG. 1 is a diagram showing the correspondence between the pupil plane and the aperture of a pixel of an imaging device. [Figure 4] Diagram showing the configuration of the eye-gaze input operation unit [Figure 5] Diagram showing the configuration of the eye-gaze input operation unit [Figure 6] Flowchart explaining a method for selecting a gaze position [Figure 7] Diagram explaining how to roughly and finely adjust the gaze position [Figure 8] 1 is a main flowchart illustrating a gaze detection processing method for an electronic device according to a first embodiment, gaze position state determination, and determination results. [Figure 9] 1 is a flowchart illustrating a gaze state determination process according to a first embodiment. [Figure 10] FIG. 10 is a diagram for explaining stop determination of the gaze state determination process according to the first embodiment. [Figure 11] FIG. 10 is a diagram illustrating a stop determination process of the gaze state determination process according to the first embodiment. [Figure 12] FIG. 10 is a diagram illustrating a movement start determination process in the gaze state determination process according to the first embodiment. [Figure 13] FIG. 10 is a diagram illustrating the movement start determination process of the gaze state determination process according to the first embodiment. [Figure 14]1 is a main flowchart illustrating a gaze detection processing method and a gaze position correction method of an imaging device according to an embodiment. [Figure 15] A diagram explaining a method for correcting gaze position [Figure 16] 10 is a flowchart illustrating focus detection, line-of-sight detection, and image capturing operations of an electronic device according to a second embodiment. [Figure 17] 10 is a flowchart illustrating a photographing subroutine according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] First Embodiment Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0010] [Explanation of the configuration of electronic devices] FIG. 1 is a block diagram showing the configuration of an imaging device. Devices including imaging devices are also referred to as electronic devices. In FIG. 1, lens unit 150 is a lens unit equipped with an interchangeable photographic lens. Lens 103 is usually composed of multiple lenses, but for simplicity, only a single lens is shown here. Communication terminal 6 is a communication terminal that allows lens unit 150 to communicate with digital camera 100, and communication terminal 10 is a communication terminal that allows digital camera 100 to communicate with lens unit 150. Lens unit 150 communicates with system controller 50 via communication terminals 6 and 10, controls aperture 102 via aperture drive circuit 2 using an internal lens system control circuit 4, and adjusts focus by displacing the position of lens 103 via AF drive circuit 3.

[0011] The shutter 101 is a focal plane shutter that can freely control the exposure time of the imaging unit 22 under the control of the system control unit 50. The imaging unit 22 is an imaging element composed of a CCD, CMOS element, or the like that converts an optical image into an electrical signal. The A / D converter 23 converts an analog signal into a digital signal. The A / D converter 23 is used to convert the analog signal output from the imaging unit 22 into a digital signal. The signal obtained from the imaging unit 22 is used not only for imaging but also for exposure control and focus detection control. The imaging unit 22 has pixels in which a photoelectric conversion unit is divided for each microlens. By dividing the photoelectric conversion unit, the entrance pupil is divided, and a phase difference detection signal can be obtained from each photoelectric conversion unit. In addition, an imaging signal can be obtained by adding the signals from the divided photoelectric conversion units.

[0012] Such pixels have the advantage that they can be used as both focus detection pixels and imaging pixels.

[0013] The image processing unit 24 performs predetermined pixel interpolation, resizing such as reduction, and color conversion processing on data from the A / D converter 23 or data from the memory control unit 15. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained arithmetic results. This allows TTL (through-the-lens) AF (autofocus) processing, AE (autoexposure) processing, and EF (flash pre-flash) processing to be performed. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data, and TTL AWB (auto white balance) processing based on the obtained arithmetic results.

[0014] The output data from the A / D converter 23 is written into the memory 32 via the image processing unit 24 and the memory control unit 15, or directly via the memory control unit 15. The memory 32 stores image data obtained by the imaging unit 22 and converted into digital data by the A / D converter 23, as well as image data to be displayed on the display unit 28 serving as display means in this embodiment. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio.

[0015] The memory 32 also serves as a memory (video memory) for image display. The D / A converter 19 converts the image display data stored in the memory 32 into an analog signal and supplies it to the display unit 28. In this way, the display image data written to the memory 32 is displayed by the display unit 28 via the D / A converter 19. The display unit 28 displays the image according to the analog signal from the D / A converter 19 on a display such as an LCD. The digital signal that has been A / D converted once by the A / D converter 23 and stored in the memory 32 is converted to analog by the D / A converter 19 and sequentially transferred to and displayed on the display unit 28, thereby functioning as an electronic viewfinder and providing a through-image display (live view display). The display unit 28 may be provided with an electronic viewfinder that can be viewed through an eyepiece (not shown), or may be provided with a display on the back of the digital camera 100. Alternatively, both an electronic viewfinder and a display on the back may be provided.

[0016] The nonvolatile memory 56 is an electrically erasable and recordable memory, such as an EEPROM. The nonvolatile memory 56 stores constants, programs, etc. for the operation of the system control unit 50. The programs referred to here are programs for executing various flowcharts described later in this embodiment.

[0017] The system control unit 50 controls the entire digital camera 100. The system control unit 50 includes the line-of-sight display means, line-of-sight state determination means, line-of-sight information determination means, subject detection means, and line-of-sight acquisition availability determination means of this embodiment. Each process of this embodiment, described below, is realized by executing a program stored in the nonvolatile memory 56. Reference numeral 52 denotes a system memory, which uses RAM. The system memory 52 stores constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 56, and other data. The system control unit also controls display by controlling the memory 32, D / A converter 19, display unit 28, and other components. The display unit 28 may be an EVF through which the user peers, or a TFT LCD that allows touch operation and other operations but does not require peering. The eye sensor installed in the EVF also controls the display unit 28 based on information indicating whether the user's eye is in focus or not.

[0018] The system timer 53 is a timekeeping unit that measures the time used for various controls and the time of a built-in clock.

[0019] The power switch 72 is an operating member for switching the power of the digital camera 100 on and off.

[0020] The mode selector switch 60, the first shutter switch 62, the second shutter switch 64, and the operation unit 70 are operation units for inputting various operation instructions to the system control unit 50. Note that the line of sight acquiring means and line of sight determining means of this embodiment are included in this operation unit 70.

[0021] The mode selector switch 60 switches the operating mode of the system control unit 50 to one of still image recording mode, video shooting mode, playback mode, etc. Modes included in the still image recording mode include auto shooting mode, auto scene determination mode, manual mode, aperture priority mode (Av mode), and shutter speed priority mode (Tv mode). Other modes include various scene modes, program AE mode, and custom mode, which are shooting settings for different shooting scenes. The mode selector switch 60 can directly switch to one of these modes included in the menu button. Alternatively, after switching to the menu button with the mode selector switch 60, another operating member can be used to switch to one of these modes included in the menu button. Similarly, the video shooting mode may also include multiple modes.

[0022] The first shutter switch 62 is turned on and generates a first shutter switch signal SW1 when the shutter button 61 provided on the digital camera 100 is pressed halfway (a shooting preparation command) during operation. The first shutter switch signal SW1 starts operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing.

[0023] The second shutter switch 64 is turned on when the shutter button 61 is fully pressed (photographing instruction) and generates a second shutter switch signal SW2. The system control unit 50 starts a series of photographing processing operations, from reading out a signal from the imaging unit 22 to writing image data to the recording medium 200, in response to the second shutter switch signal SW2.

[0024] Each operating member of the operating unit 70 is assigned a function appropriate for each situation by selecting and operating various function icons displayed on the display unit 28, and acts as various function buttons. The function buttons include, for example, an end button, a back button, an image forward button, a jump button, a filter button, and an attribute change button. For example, when the menu button is pressed, a menu screen on the display unit 28 on which various settings can be made is displayed. The user can intuitively make various settings using the menu screen displayed on the display unit 28, the four directional buttons (up, down, left, and right), and the SET button.

[0025] The operation unit 70 is an input unit that accepts operations from the user and includes various operation members, such as electronic buttons and a cross key for selecting a menu, selecting a mode, and playing back captured moving images.

[0026] The power supply control unit 80 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, etc., and detects whether a battery is installed, the type of battery, and the remaining battery power. The power supply control unit 80 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the required voltage to each unit, including the recording medium 200, for the required period of time.

[0027] The power supply unit 30 is composed of primary batteries such as alkaline batteries or lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries or Li batteries, an AC adapter, etc. The recording medium I / F 18 is an interface with a recording medium 200 such as a memory card or hard disk. The recording medium 200 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, a magnetic disk, etc.

[0028] The communication unit 54 is connected wirelessly or via a wired cable and transmits and receives video and audio signals. The communication unit 54 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 54 can transmit images (including through images) captured by the imaging unit 22 and images recorded on the recording medium 200, and can also receive image data and various other information from external devices.

[0029] The orientation detection unit 55 detects the orientation of the digital camera 100 with respect to the direction of gravity. Based on the orientation detected by the orientation detection unit 55, it is possible to determine whether an image captured by the imaging unit 22 was captured with the digital camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the orientation detected by the orientation detection unit 55 to the image file of the image captured by the imaging unit 22, or rotate and record the image. The orientation detection unit 55 can be an acceleration sensor, a gyro sensor, or the like.

[0030] [Explanation of gaze detection configuration] In this embodiment, a gaze input operation unit 701 is provided as one of the operation units 70. The gaze input operation unit 701 is a gaze acquisition means of this embodiment and is an operation member for detecting which part of the display unit 28 the user's gaze is looking at. FIG. 4 shows an example of the gaze input operation unit 701. FIG. 4 shows a configuration that realizes a method disclosed in Patent Document 1, in which the rotation angle of the optical axis of the user's eyeball 501a looking into the viewfinder field is detected and the user's gaze is detected from the detected rotation angle. A live view display image captured through the lens unit 100 is displayed on the display unit 28. 701a denotes an image sensor, 701b denotes a light receiving lens, 701c denotes a dichroic mirror, 701d denotes an eyepiece, and 701e denotes an illumination light source. Infrared light is projected onto the eyeball 501a by the illumination light source 701e. The infrared light reflected by the eyeball 501a is reflected by the dichroic mirror 701c and captured by the image sensor 701a. The captured eyeball image is converted into a digital signal by an A / D converter (not shown) and transmitted to the system control unit 50. The system control unit 50, which serves as a gaze acquisition unit, extracts the pupil area and the like from the captured eyeball image and calculates the user's gaze.

[0031] Note that the gaze input operation unit 701 is not limited to this method, and may instead be a method of capturing an image of both of the user's eyes and detecting the gaze. FIG. 5 shows an example of the gaze input operation unit 701 different from that shown in FIG. 4. In FIG. 5, a live view display image captured through the lens unit 100 is displayed on the display unit 28 provided on the back of the digital camera 100. In FIG. 5, a camera 701f is provided on the back of the digital camera 100 to capture an image of the user's face 500 observing the display unit 28. In FIG. 5, the angle of view captured by the camera 701f is indicated by a dotted line. Light is projected onto the user's face from an illumination light source 701e (not shown), and an image of the user's eyeballs is acquired by the camera 701f. This allows the user's gaze to be calculated. Note that the gaze input operation unit 701 is not limited to this method, and may be any other configuration that can detect which part of the display unit 28 the user is gazing at.

[0032] [How to select gaze position] Hereinafter, the line-of-sight selection method in the first embodiment will be described with reference to FIGS.

[0033] Fig. 6 is a main flowchart illustrating a gaze selection method for an electronic device according to the first embodiment. Fig. 6 shows the operation performed when a user performs a gaze determination operation while viewing an image displayed on the display unit 28, which is a display means, and is realized mainly by the system control unit 50. Note that, although the present embodiment will be described using the block diagram of the imaging device in Fig. 1, similar operations are possible in an electronic device that has at least the system control unit 50, the display unit 28, and the operation unit 70.

[0034] In S1, the system control unit 50 determines the state of the gaze position, which is gaze detection information used in gaze position selection. The gaze detection, gaze position state determination, and display will be described later with reference to FIGS. 8 to 13.

[0035] In S2, the system control unit 50 determines whether the user has selected to perform two-stage gaze position selection, i.e., coarse gaze adjustment or fine gaze adjustment. If the user has selected to perform two-stage gaze position selection, the system proceeds to step S6 to perform gaze position selection using coarse gaze adjustment and fine gaze adjustment. If the user has selected not to perform two-stage gaze position selection, the system proceeds to step S3 to not perform two-stage gaze position selection.

[0036] In S3, the system control unit 50 corrects the gaze position based on the result of the state determination and display process in S1. A method for calculating the gaze correction position by correcting the gaze position will be described later with reference to FIGS. 14 and 15.

[0037] In S4, the system control unit 50 determines whether or not to determine the gaze position using the gaze correction position calculated in S3. If the user selects to determine the gaze position, the gaze position is determined and the main flow ends. If the user selects not to determine the gaze position, the process returns to S2 to determine the gaze position state again, and steps S2 to S4 are repeated.

[0038] Next, a case will be described in which the user selects to perform two-stage gaze position selection in S1. In this case, the gaze position selection is performed by coarse and fine gaze adjustment, and the process proceeds to step S6.

[0039] In S6, the system control unit 50 further selects an operation mode for the gaze position selection method of this embodiment. There are two operation modes to select. One is a stability-priority mode, which is used when stability is to be prioritized over responsiveness in gaze position calculation. The other is a responsiveness-priority mode, which is used when responsiveness is to be prioritized over stability in gaze position calculation.

[0040] If the user selects the stability priority mode in S6, the process proceeds to S7. If the user selects the quick response priority mode, the process proceeds to S11. Next, each of the operation modes will be described.

[0041] In S7, the system control unit 50 selects a first limited area within the shooting screen to roughly determine (coarsely adjust) the gaze position. Furthermore, since S7 is a process that is performed after it is determined in S6 that the user prioritizes stability of the gaze position, the system control unit 50 performs a smoothing process on the user's gaze position coordinates. The size of the first limited area is changed according to the smoothing score in this smoothing process.

[0042] The user may also change the score for the smoothing process by setting the sensitivity of the gaze position. In this case, if the sensitivity is set high, the smoothing score will be lower than if it is set low.

[0043] Furthermore, depending on the result of the determination of whether the user's gaze position has stopped, performed in S1, the first limited area may be reduced in size from a size that is normally set. The method for selecting this first limited area will be described later. After the gaze position is roughly adjusted, the gaze position is determined to have started moving a second number of times or a fifth number of times, which is greater. After the first limited area is selected, the process proceeds to S8.

[0044] In S8, the system control unit 50 corrects the gaze position using the gaze correction position calculation means of this embodiment based on the results of the state determination and display processing in S1, similar to S3 described above. After the gaze position has been corrected, the process proceeds to S9.

[0045] In S9, the system control unit 50 turns off the movement start determination that was performed in S1 prior to fine-tuning the gaze position in S10. This is because quick response can be prioritized in coarse adjustment of the gaze position. After turning off the movement start determination, the process proceeds to S10. Note that the movement start determination may be left on. This allows for more precise gaze position selection.

[0046] In S10, the system control unit 50 precisely determines (fine-tunes) the gaze position from the first limited area selected in S7. At this time, a determination is made as to whether the user's gaze position is to be stopped. In S10, the user selected priority for gaze position stability in S6. Therefore, the display of the gaze position is stopped by making a determination on the gaze position for the second determination number or a third determination number, which is greater than the second determination number, in this embodiment. Then, the position determined by the fine adjustment of the gaze position is determined as the gaze position selected by the user, and the main flow ends.

[0047] On the other hand, if the user selects the quick response priority mode in S6, the process proceeds to S11.

[0048] In S11, the system control unit 50 sets the gaze position display stop determination and movement start determination performed in S1 to OFF in order to prioritize responsiveness to gaze position information. By setting this to OFF, the display dwell time associated with the gaze position display stop determination and movement start determination is reduced.

[0049] In S12, the system control unit 50 performs coarse adjustment of the gaze position in the same manner as in S7 described above. After the coarse adjustment of the gaze position, the gaze position movement start determination is performed the second number of determinations in this embodiment, or the sixth number of determinations, which is greater. After the coarse adjustment of the gaze position is performed in S12, the process proceeds to S13.

[0050] In S13, the system control unit 50 executes fine adjustment of the gaze position, which is the second gaze selection means of the present invention, as in S10 described above. In this gaze position stop determination, the display of the gaze position is stopped by performing a determination at the gaze position the second number of determinations of this embodiment, or a larger number of determinations on the basis of the number of determinations of the base 4. The position determined by the fine adjustment of the gaze position is then determined as the gaze position selected by the user, and the main flow ends.

[0051] [Explanation of gaze detection, state determination, and display] 8 is a sub-flowchart of S1, which is a display process and a method for determining the state of a gaze position when a user performs a gaze determination operation while looking at an image displayed on the display unit 28, which is a display means. FIG. 8 is realized mainly by the system control unit 50. Note that, although the present embodiment will be described using the block diagram of the imaging device in FIG. 1, similar operations are possible in electronic devices that have at least the system control unit 50, the display unit 28, and the operation unit 70.

[0052] In S21, the system control unit 50 displays the image data on the display unit 28. The image to be displayed is, for example, a reduced image that matches the resolution of the display unit 28. While the image is being displayed, it is displayed at a predetermined frame rate, and the user can check the displayed image through the display unit 28.

[0053] In S22, the system control unit 50 determines whether or not gaze information can be acquired using the gaze acquisition availability determination means of this embodiment. In the case of an eye sensor installed in the EVF of the display unit 28, whether or not gaze information can be acquired is determined based on information on whether the user has their eyes in close contact with the EVF. If the user is looking into the EVF, the eye sensor determines that the user has their eyes in close contact with the EVF, and the gaze input operation unit 701, which is the gaze acquisition means of this embodiment, determines that gaze information can be acquired. On the other hand, if the user is not looking into the EVF, the eye sensor determines that the user has their eyes in close contact with the EVF, and the gaze input operation unit 701, which is the gaze acquisition means of this embodiment, determines that gaze information cannot be acquired. In addition, in the case of a TFT liquid crystal display of the display unit 28, the gaze input operation unit 701 also determines whether or not gaze information can be acquired.

[0054] If it is determined in S22 that gaze information can be acquired, the system control unit 50 starts gaze detection in S23. From S24 onwards, the gaze input operation unit 701 acquires and stores at predetermined time intervals the position on the display unit 28 at which the user is observing (gaze position) in association with the displayed image that the user was observing. Also, if it is determined in S22 that gaze information cannot be acquired, the gaze information at the determined timing is not acquired or stored.

[0055] In S24, it is determined whether the number of times that gaze information was acquired and accumulated in S23 is equal to or greater than a predetermined number N. The predetermined number N is determined by the number of times that gaze state determination can be performed in S25, which follows. If it is determined in S22 that acquisition is not possible, the information is not acquired or accumulated, and is therefore not counted toward the predetermined number N in S24. If it is determined in S23 that the number of times that gaze information was acquired and accumulated is equal to or greater than the predetermined number N, it is determined that gaze state determination in S5, which is the gaze state determination means of this embodiment, is possible, and the process proceeds to S25. If it is determined in S23 that the number of times that gaze information was acquired and accumulated is less than the predetermined number N, it is determined that gaze state determination in S15, which is the gaze state determination means of this embodiment, is impossible, and the process skips S25 and proceeds to S26.

[0056] In S25, the gaze state determination is performed as the gaze state determination means of this embodiment. The gaze state determination in S25 will be described later with reference to the gaze state determination sub-flow diagram in FIG.

[0057] In S26, the gaze position information acquired in S23 or S25 is used to display the gaze position on the display unit 28. If it is determined in S24 that the gaze accumulation count is equal to or greater than the predetermined count N, the gaze position acquired in S25 is used. If it is determined in S24 that the gaze accumulation count is less than the predetermined count N, the gaze position acquired in S23 is used.

[0058] In S27, it is determined whether or not the user has performed a gaze confirmation operation. If the user has performed a gaze confirmation operation, the display of the gaze position on the display unit 28 is stopped, and this gaze position state determination and display processing flow is terminated. Note that the display of the gaze position on the display unit 28 may be continued, or the display method may be changed. Also, if the user has not performed a gaze confirmation operation, the process proceeds to S12, in which the gaze acquisition possibility determination means of this embodiment determines whether or not gaze information can be acquired, and new gaze information is acquired. Note that if a gaze confirmation operation is performed in S27, the number of times gaze information has been acquired and accumulated in S13, and gaze position change information ΔS j After the series of steps is completed, the process proceeds to the main flow S2 in FIG.

[0059] [Explanation of gaze state determination] The series of operations in the subflow of FIG. 9 is also realized mainly by the system control unit 50. S1501 is the gaze position information S acquired in S23 of FIG. i Using this, the gaze position change information ΔS j In S14 of FIG. 8, it is determined that the number of times is equal to or greater than the predetermined number N, and the gaze position change information ΔS j Among them, ΔS corresponding to the horizontal position change jH , ΔS corresponding to the vertical position change jV , ΔS corresponding to the position change between the two points jrare calculated using the following formulas (1), (2), and (3), respectively.

[0060]

number

[0061] In step S1502, the first determination count JN1 of this embodiment and the first threshold value Th1 of this embodiment are used to calculate the gaze position change information ΔS j The relationship between the predetermined number of times N and the number of times of determination JN1 is expressed by the following formula (4).

[0062]

number

[0063] In this embodiment, JN1 is set to a fixed value of 5 times, but may be changed depending on the conditions. j includes the judgment time, and if it is judged that the first judgment number JN1 times in the past is equal to or less than the first threshold value Th1 in this embodiment, the display of the gaze position displayed on the display unit 28 is stopped (does not move). For this purpose, the process proceeds to S1505. Also, the gaze position change information ΔS acquired in S1501 j includes the judgment time, and if it has been judged to be equal to or greater than the first threshold value Th1 of this embodiment at least once during the past first judgment count JN1 of this embodiment, the judgment will be made using the second threshold value of this embodiment, and the process proceeds to S1503.

[0064] Here, the stillness determination method and calculation of the display fixed position for stilling the gaze position display performed in S1502 will be described with reference to FIGS. 10 and 11. FIG. 10 shows an example in which a captured video or a video being captured is displayed on the display unit 28, and the user performs a gaze confirmation operation on a specific subject with their gaze. A method for keeping the gaze information display still using the first determination count JN1 and the first threshold value Th1 of this embodiment will be described. FIG. 10 shows excerpts of images displayed on the display unit 28 at times t1 (FIG. 10(a)), t2 (FIG. 10(b)), and t3 (FIG. 10(c)) from the left. Sm represents the gaze information display, and Trg represents the subject the user is following with their gaze.

[0065] 10(a) shows the moment when a subject Trg on a bicycle moves in from the left side of the screen. The image also shows the user recognizing the subject Trg and beginning to make eye contact with him. At this moment, the gaze information display Sm does not yet match the subject Trg, and the gaze information display Sm is scattered as indicated by the dashed lines.

[0066] Figure 10(b) shows the moment when the subject Trg slows down the speed of his bicycle to a stop. It also shows the subject Trg and the line-of-sight information display Sm at time t2, after time t1 in Figure 10(a). At this moment, the line-of-sight information display Sm begins to roughly align with the subject Trg, but as indicated by the dashed lines, the line-of-sight information display Sm varies.

[0067] Figure 10(c) shows the moment when subject Trg stops his bicycle and stands. It also shows subject Trg and the line-of-sight information display Sm at time t3, after time t2 in Figure 10(b). At this moment, the line-of-sight information display Sm is aligned with subject Trg, and the line-of-sight information display Sm is displayed stationary by using the first determination count JN1 and the first threshold value Th1 of this embodiment.

[0068] Next, the series of steps in FIG. 10 is shown in FIG. 11 as the gaze position change information ΔS jThis will be explained using the first determination count JN1 of this embodiment and the first threshold value Th1 of this embodiment. Fig. 11 explains the stillness determination using the first determination count JN1 of this embodiment and the first threshold value Th1 of this embodiment. Note that in Fig. 11, the first determination count JN1 of this embodiment is set to 5 times, but it is not limited to this number and may be changed depending on the user's line of sight characteristics, the viewing state of the display unit 28, or manual operation by the user.

[0069] FIG. 11(a) shows the gaze position change information ΔS j Among them, ΔS corresponding to the horizontal position change jH The two solid horizontal lines represent the threshold value Th1 corresponding to the horizontal position change amount, out of the first threshold value Th1 of this embodiment. 1H The gray hatched portion represents the number of determinations JN1 corresponding to the horizontal position change amount, out of the first number of determinations JN1 of this embodiment. 1H represents.

[0070] FIG. 11(b) shows the gaze position change information ΔS j Among them, ΔS corresponding to the vertical position change jV The two solid horizontal lines represent the threshold value Th1 corresponding to the vertical position change amount, out of the first threshold value Th1 of this embodiment. 1V The gray hatched portion represents the number of determinations JN1 corresponding to the horizontal position change amount, out of the first number of determinations JN1 of this embodiment. 1V represents.

[0071] FIG. 11(c) shows the gaze position change information ΔS j Among them, ΔS corresponding to the position change between two points jr The two solid horizontal lines represent the threshold value Th1 corresponding to the position change amount between two points, out of the first threshold value Th1 of this embodiment. 1r The gray hatched portion represents the number of determinations JN1 corresponding to the amount of position change between two points, out of the first number of determinations JN1 of this embodiment. 1r represents.

[0072] FIG. 11(a) shows the gaze position change information ΔS jH This is an explanation of the case where the stillness determination of the gaze information display is performed in the time t1 to the time t2. jH is gradually decreasing but still corresponds to the threshold value Th 1H After that, the gaze position change information ΔS for the horizontal position change amount from time t2 to time t3 is jH is the threshold value Th corresponding to the horizontal position change amount. 1H The horizontal position change amount is within the threshold Th 1H In this embodiment, the number of determinations JN corresponding to the amount of change in horizontal position is counted up. 1H In order to explain this as five times, from time t3, which is the fifth time, the line of sight information display on the display unit 28 is changed to a fixed value so as to be displayed statically. The fixed value is the number of determinations JN corresponding to the horizontal position change amount. 1H The average value of the horizontal position change amount is used as a fixed value. 1H It may also be the last value of

[0073] FIG. 11(b) shows the gaze position change information ΔS jV This is an explanation of the case where the stillness determination of the gaze information display is performed. jV is a threshold Th that gradually decreases but still corresponds to the vertical position change amount. 1V After that, the gaze position change information ΔS jV is the threshold value Th corresponding to the vertical position change amount. 1V The vertical position change amount is converged as time passes. 1V In this embodiment, the number of determinations JN corresponding to the amount of change in vertical position is 1VIn order to explain this as five times, from time t3, which is the fifth time, the line of sight information display on the display unit 28 is changed to a fixed value so as to be displayed statically. The fixed value is the fixed position of the display, and the number of determinations JN corresponds to the vertical position change amount. 1V The average value of the number of judgments JN corresponding to the vertical position change is used as a fixed value. 1V It may also be the last value of

[0074] FIG. 11(c) shows the gaze position change information ΔS jr This is an explanation of the case where the stillness determination of the gaze information display is performed with the above-mentioned ΔS jH and ΔS jV Unlike ΔS jr Since is a positive value, the threshold value Th corresponding to the position change between two points is 1r is only a positive value. The gaze position change information ΔS for the position change amount between two points from time t1 to time t2 jr is gradually decreasing but still remains at the threshold Th corresponding to the position change between two points. 1r After that, the gaze position change information ΔS for the amount of position change between the two points from time t2 to time t3 is jr is the threshold value Th corresponding to the position change amount between two points. 1r The value of the threshold Th corresponding to the position change amount between the two points from time t2 is 1r In this embodiment, the number of determinations JN corresponding to the amount of position change between the two points is 1r In order to explain this as five times, from time t3, which is the fifth time, the line of sight information display on the display unit 28 is changed to a static display with a fixed value. The fixed value is a fixed position of the display, and the number of determinations JN corresponds to the amount of position change between two points. 1r The average position of the two points is set as a fixed value. 1r The position may be determined from the last value of

[0075] As described above, by statically displaying the line-of-sight information display in accordance with changes in line-of-sight information, it is possible to provide an electronic device that improves display quality and convenience.

[0076] In step S1503, the second determination count JN1 of this embodiment and the second threshold value Th1 of this embodiment are used to calculate the gaze position change information ΔS j The relationship between the predetermined number of times N and the number of times of determination JN2 is expressed by the following formula (5).

[0077]

number

[0078] In this embodiment, JN2 is set to a fixed value of 5 times, but may be changed depending on the conditions. j includes the judgment time, and if it is judged that the second judgment number JN2 times in the past is equal to or greater than the second threshold value Th2 in this embodiment, the display of the gaze position displayed on the display unit 28 is operated (moved). To do this, the process proceeds to S1504 to calculate the movement start position. Also, the gaze position change information ΔS acquired in S1501 j includes the judgment time, and if it is judged to be equal to or less than the second threshold value Th2 of this embodiment at least once during the past second judgment number JN2 of this embodiment, the display is not changed from the previous display. i The process proceeds to S1506 to determine the line-of-sight display position to be displayed on the display unit 28 using the above.

[0079] Here, the method of determining whether or not a movement has started and the calculation of the fixed position of the display performed in S1503 for starting to move the gaze position display will be described with reference to FIGS. 12 and 13. FIG. 12 shows an example in which a captured video or a video being captured is displayed on the display unit 28, and the user performs a gaze confirmation operation on a specific subject with their gaze. A method of fixing the gaze information display using the second determination count JN2 and the second threshold value Th2 of this embodiment will be described. FIG. 12 shows excerpts of images displayed on the display unit 28 at times t1' (FIG. 12(a)), t2' (FIG. 12(b)), and t3' (FIG. 12(c)) from the left. Sm indicates the gaze information display as in FIG. 12, and Trg' indicates a subject different from that in FIG. 12 that the user is following with their gaze.

[0080] FIG. 12(a) shows the moment when the subject, a car Trg', is about to move from the left side of the screen. The user continues to recognize the car Trg' and their gazes are fixed on it. At this moment, the gaze information display Sm is aligned with the car Trg', and the gaze information display Sm is still due to the processing of S1502 described above. Note that the gaze information display Sm may be operating before the processing of S1502 is executed.

[0081] Figure 12(b) shows the moment when the subject Trg' starts to move and picks up speed. Also, Figure 12(a) shows the subject Trg' and the line-of-sight information display Sm at time t2', after time t1'. At this moment, the line-of-sight information display Sm is beginning to diverge from the subject Trg', but the line-of-sight information display Sm is stationary due to the processing of S1502 described above. Note that the line-of-sight information display Sm may be operating before the processing of S1502 is executed.

[0082] Figure 12(c) shows the moment when the subject Trg', a car, is stopped. Also, Figure 12(c) shows the subject Trg' and the gaze information display Sm at time t3', after time t2' in Figure 12(b). At this moment, the gaze information display Sm matches the subject Trg', and the gaze information display Sm is displayed in a dynamic manner by using the second determination count JN2 of this embodiment and the second threshold value Th2 of this embodiment, and is displayed with variations as indicated by the dashed lines.

[0083] Next, the series of steps in FIG. 12 are shown in FIG. 13 as the gaze position change information ΔS j This will be explained using the second determination number JN2 of this embodiment and the second threshold value Th1 of this embodiment. Fig. 13 explains the stillness determination using the second determination number JN2 of this embodiment and the second threshold value Th2 of this embodiment. Note that in Fig. 13, the second determination number JN2 of this embodiment is set to 5 times, but it is not limited to this number and may be changed depending on the user's line of sight characteristics, the viewing state of the display unit 28, or manual operation by the user.

[0084] FIG. 13(a) shows the gaze position change information ΔS j Among them, ΔS corresponding to the horizontal position change jH The two solid horizontal lines represent the threshold value Th2 corresponding to the horizontal position change amount, out of the second threshold values ​​Th2 of this embodiment. 2H The gray hatched portion represents the number of determinations JN corresponding to the amount of change in horizontal position, out of the second number of determinations JN2 of this embodiment. 2H represents.

[0085] FIG. 13(b) shows the gaze position change information ΔS j Among them, ΔS corresponding to the vertical position change jV The two solid horizontal lines represent the threshold value Th2 corresponding to the vertical position change amount, out of the second threshold values ​​Th2 of this embodiment. 2V The gray hatched portion represents the number of determinations JN corresponding to the amount of change in horizontal position, out of the second number of determinations JN2 of this embodiment. 2Vrepresents.

[0086] 13(c) is the gaze position change information ΔS at time t'. j Among them, ΔS corresponding to the position change between two points jr The two solid horizontal lines represent the threshold value Th2 corresponding to the amount of position change between two points, out of the second threshold value Th2 of this embodiment. 2r The gray hatched portion represents the number of determinations JN corresponding to the amount of position change between two points, out of the second number of determinations JN2 in this embodiment. 2r represents.

[0087] FIG. 13(a) shows the gaze position change information ΔS jH This is an explanation of the case where the stillness determination of the gaze information display is performed at time t1' to time t2'. jH is gradually increasing, and the threshold value Th corresponding to the horizontal position change amount is 1H After that, the gaze position change information ΔS for the horizontal position change amount from time t2' to time t3' is jH is the threshold value Th corresponding to the horizontal position change amount. 2H The horizontal position of the object Trg′ is changed from time t2′ to time t3′. 2H In this embodiment, the number of determinations JN corresponding to the amount of change in horizontal position is counted up. 2H For the explanation, the number of times is set to five, and from the fifth time t3', the line-of-sight information display on the display unit 28 is changed to be displayed in motion. The value for displaying is the fixed position of the display, and the number of determinations JN corresponding to the amount of change in horizontal position is 2H The average value of the horizontal position change amount is used as a fixed value. 2H The last value of the horizontal position change information ΔS jH is the threshold value Th corresponding to the horizontal position change amount. 2HEven if the horizontal position change amount is within 1 / 2, the line of sight information display may be changed to an operational display in the following cases: 2H If the movement is continuous and always in the same direction, the line of sight information display may be changed to an animated display.

[0088] FIG. 13(b) shows the gaze position change information ΔS jV This is an explanation of the case where the stillness determination of the gaze information display is performed at time t1' to time t2'. jV is roughly constant, and always corresponds to the threshold value Th 2V At this time, the vertical position change amount is lower than the threshold value Th 2V Therefore, the eye gaze position change information ΔS jV In this case, it is determined that the line-of-sight information display on the display unit 28 remains statically displayed as a fixed value even at time t3'.

[0089] FIG. 13(c) shows the gaze position change information ΔS jr This is an explanation of the case where the stillness determination of the gaze information display is performed with the above-mentioned ΔS jH and ΔS jV Unlike ΔS jr Since is a positive value, the threshold value Th corresponding to the position change between two points is 2r The gaze position change information ΔS for the amount of position change between two points from time t1' to time t2' is jr is gradually increasing, but is still the threshold value Th corresponding to the position change between two points. 2r After that, the gaze position change information ΔS for the amount of position change between the two points from time t2' to time t3' jr is the threshold value Th corresponding to the position change amount between two points. 2r The change in the position between the two points from time t2' is within the threshold value Th 2rIn this embodiment, the number of determinations JN corresponding to the amount of position change between the two points is counted up. 2r In order to explain this as five times, the line of sight information display on the display unit 28 is changed to be displayed in motion from time t3', which is the fifth time. The value to be displayed is the fixed position of the display, and the number of determinations JN corresponding to the amount of position change between two points is 2r The average position of the two points is set as a fixed value. 2r The position may be determined from the last value of

[0090] As described above, by dynamically displaying the line-of-sight information display in response to changes in line-of-sight information, it is possible to provide an electronic device that improves display quality and convenience.

[0091] 12 and 13, the amount of change in gaze position ΔS j However, it is also possible to determine whether the gaze position is stationary or moving depending on whether the direction of the change is within a predetermined range. For example, ±45 degrees can be selected as the predetermined range. This makes it possible to determine whether the gaze position is changing with variation.

[0092] S1504 is performed when it is determined in S1503 that the value is equal to or greater than the second threshold value Th2 of this embodiment.The operation position of the gaze display is calculated, and the process proceeds to S1506.

[0093] S1505 is performed when it is determined in S1502 that the value is equal to or less than the first threshold value Th1 of this embodiment. The stationary position of the gaze display is calculated, and the process proceeds to S1506. After this series of steps is completed, the process proceeds to S16 in the main flow of FIG. 8.

[0094] [Explanation of gaze detection and dynamic calibration method] Hereinafter, a gaze position correction method according to the present embodiment will be described with reference to Figs. 14 and 15. Fig. 14 is a sub-flowchart illustrating a gaze detection processing method and a gaze position correction method for an imaging device according to the present embodiment. Fig. 14 shows the operation when a user performs a gaze determination operation and implements a focus adjustment means while looking at an image displayed on the display unit 28, which is the display means of the present embodiment, and is realized mainly by the system control unit 50. Note that, although the present embodiment will be described using the block diagram of the imaging device in Fig. 1, similar operations are possible in a device having at least the system control unit 50, the display unit 28, and the operation unit 70.

[0095] Steps S31 to S33 are the same as steps S21 to S23.

[0096] In S34, the subject detection means of this embodiment is used to acquire subject position information, which is the position information of the subjects displayed on the display unit 28, the number of detected subjects, the type of detected subjects, and the size of the detected subjects. Rating information for the captured image is also acquired. The rating information may be a value determined by the camera based on focus information or the like, or a manual value determined by the user. This subject information is updated each time the scene changes, such as by panning the camera. The update timing may be each time the scene changes, or, during continuous shooting, only when a change occurs, based on information indicating whether the subject information acquired for the previous frame has been updated. After acquiring this subject information, the process proceeds to S35.

[0097] In S35, the scene determination information acquisition means of this embodiment is used to acquire camera operation information at the time of shooting the scene displayed on the display unit 28 and subject operation information displayed on the display unit. Information acquired as camera operation information at the time of shooting includes panning information acquired using a gyro sensor or the like, lens information such as lens focal length and zoom position, shutter settings such as single shot / continuous shot, and camera setting information such as Tv and ISO. Information acquired as subject operation information also includes the in-plane movement speed v of the subject on the display unit 28. t , the movement vector V t , past movement trajectory information P t After acquiring the camera operation information and subject operation information, the process proceeds to S36.

[0098] In S36, it is determined whether or not the user has performed a gaze confirmation operation. If the user has performed a gaze confirmation operation, the display of the gaze position on the display unit 28 is stopped, and the process proceeds to S37 so that the gaze position can be corrected by the gaze correction position calculation means of this embodiment. Note that the display of the gaze position on the display unit 28 may be continued, or the display method may be changed. Furthermore, if the user has not performed a gaze confirmation operation, the process proceeds to S2, in which it is determined whether or not gaze information can be acquired by the scene determination information acquisition means of this embodiment, and new gaze information is acquired.

[0099] In S37, the gaze correction position calculation means of this embodiment is executed to calculate a correction value for the gaze position. Here, the gaze position correction means of the gaze correction position calculation means of this embodiment will be explained using FIG. 15. FIG. 15 shows various scenes on the display unit 28 when the gaze correction position calculation means of this embodiment is used. EP is a gaze display that displays the gaze position of the user, and the gaze position coordinates on the display unit 28 displayed at this time are (x eye ,y eye ) where Trg is the detected object, Area_dT represents the range of Trg, and the Trg coordinates are (x trg ,y trg ) and CAL eyerepresents the gaze position value, which is divided into a horizontal position correction value and a vertical position correction value,

[0100]

number

[0101] This can be expressed as equation (1).

[0102] FIG. 15(a) shows the moment when the user adjusts his / her gaze position to the flying airplane and fixes his / her gaze. In FIG. 15(a), the gaze position coordinate (x eye ,y eye ) is the Trg coordinate (x trg ,y trg ) is displayed at a different position from the Trg coordinate (x trg ,y trg ) cannot be acquired, it is determined that dynamic calibration will not be performed.

[0103] First, the gaze reliability information acquired in S32 is used as gaze information. According to this gaze reliability information, the gaze gain Gain eye The value of Gain changes when the gaze reliability information is low. eye <1, the acquired gaze position coordinate (x eye ,y eye ) is likely to have a large error for some reason, so it cannot be used as is. On the other hand, if the gaze reliability information is high, Gain eye =1 is the acquired gaze position coordinate (x eye ,y eye ) can be used.

[0104] Next, the subject gain Gain at the time of calculating the gaze position correction value is calculated based on the detected subject type, subject size, and rating result as subject information. Trg If the subject is considered difficult to photograph based on the subject type and size, the value of Gain Trg <1. Also, if the subject is considered to be easy to photograph based on the subject type and size, change the subject gain to Gain <1. Trg Change the subject gain to =1.

[0105] Finally, as scene determination information, the scene determination gain Gain sceane The value changes depending on the difficulty of the scene. When the scene difficulty is high based on the camera operation information and subject movement information acquired in S35 as scene determination information, Gain sceane <1, and change the scene determination gain. Also, if the scene difficulty level is low based on the camera operation information and subject movement information acquired in S35, sceane =1 and change the scene determination gain.

[0106] From these gains, the gaze position correction value in this embodiment is calculated using the distance between the two points (ΔX', ΔY') as follows:

[0107]

number

[0108] This can be calculated from equation (3).

[0109] Similarly, in Figure 15(b), the line of sight reliability is high and Gain eye = 1, the subject is also a stationary subject, so it is easy and Gain Trg =1, because the scene difficulty is low sceane = 1. Eye gaze position correction value CAL eye is the gaze position coordinate (x eye ,y eye ) is the Trg coordinate (x trg ,y trg ) is considered to be the exact difference in coordinates.

[0110] In addition, in FIG. 15(c), the gaze position coordinate (x eye ,y eye In this case, the object (coffee) in the Trg coordinate (x trg ,y trg ) cannot be acquired, it is determined that dynamic calibration will not be performed. eye Then, the process moves to S38 to correct the gaze position.

[0111] In S38, the gaze position correction value CAL calculated in S37 is eye The eye gaze position is corrected by the eye gaze position correction value CAL. eye has horizontal and vertical position correction values, so each coordinate is calculated as the gaze position correction value CAL eye After the calculation, the process proceeds to S4 in the main flow.

[0112] [Coarse and fine adjustment of gaze position] Next, coarse adjustment and fine adjustment of the gaze position will be described with reference to FIG. 7. The processing described in FIG. 7 is realized by the system control unit 50 executing a predetermined program or the like. FIG. 7 shows the process of accurately specifying a subject frame, AF frame, etc. for Target_A based on the user's gaze position. In this embodiment, a case where the user wants to accurately display an AF frame for Target_A will be described. In this embodiment, a limited area, which will be described later, is highlighted to make it easier for the user to select a frame.

[0113] Eyeposition in Fig. 7(a) represents the user's gaze position. Target_A and Target_B in Fig. 7 represent people (subjects) standing next to each other. TargetArea_A and TargetArea_B are subject detection frames for Target_A and Target_B displayed using subject information acquired by the system control unit 50 as subject detection means. Note that, although the subject here is described using people standing next to each other as an example, it may be an animal, an object, a landscape, etc. instead of a person.

[0114] Figure 7 shows the state starting from state (a), followed by state (b) where coarse adjustment of the gaze position is performed. In state (c), fine adjustment of the gaze position is performed from within the first limited area selected in state (b). Finally, in state (d), an AF frame is displayed for Target_A, the target desired by the user.

[0115] First, the coarse adjustment of the gaze position will be explained using Figures 7(a), (b1), and (b2). When the user's gaze position is displayed at the location of EyePosition in Figure 7(a), the gaze input operation unit 701 in Figure 1 executes coarse adjustment and determination of the gaze position. As a result, the system control unit 50 performs control so that the first limited area EyeArea shown in Figures 7(b1) and (b2) is displayed according to the user's gaze position coordinates and gaze position reliability.

[0116] The first limited area EyeArea is an area determined using the coordinates of the user's gaze position and is a part of the captured image. In this case, gaze position coordinates acquired in one go may be used, or statistical data such as a moving average of multiple points may be used. If the reliability of the user's gaze position is low, a second limited area larger than the first limited area may be set. Even if the reliability of the user's gaze position is low, if a stop determination was made when the gaze input selection unit 701 confirmed the position, the first limited area may remain instead of the second limited area.

[0117] Figures 7(b1) and (b2) show examples of different display methods for the first limited area EyeArea. Figure 7(b1) shows an example in which an area is set that includes Target_A and Target_B, with the EyePosition specified in Figure 7(a) at the center. Figure 7(b2) shows an example in which an area (quadrant) that includes the EyePosition when the screen is divided into four is set.

[0118] Next, FIGS. 7(c1), (c21), and (c22), which are examples of fine adjustment of the gaze position, will be described starting from FIG. 7(b1). FIG. 7(c1) shows an example in which the first limited area selected in FIG. 7(b1) is enlarged and displayed. After the user's gaze position is more accurately aligned with Target_A from the enlarged display image, the fine adjustment of the gaze position is confirmed by the gaze input scanning unit 701. As a result, an AF frame (AF Area) is accurately displayed for Target_A, as shown in FIG. 7(d).

[0119] FIG. 7(c21) shows a vector position specification that detects whether the user's gaze position has moved to the left or right division, starting from the EyePosition position when the first limited area EyeArea was set in FIG. 7(b1). FIG. 7(c21) shows an example of a case where a subject is not detected within the EyeArea in FIG. 7(b1). If a subject is not detected within the EyeArea in FIG. 7(b1) as in FIG. 7(c21), the EyeArea is narrowed down by dividing the EyeArea into two parts in the left and right direction. Then, subject detection is performed again within the divided EyeArea that includes the EyePosition, and AFArea is set for Target_A as shown in FIG. 7(d) based on the detection information of the detected Target_A. Note that this embodiment shows an example of fine-tuning the gaze position, but is not limited to dividing into two. Division may be performed not only horizontally but also vertically. Furthermore, if the priority of a subject is determined by subject detection after coarse adjustment of the gaze position, AFArea may be set by prioritizing detection of the priority subject.

[0120] FIG. 7(c22) shows an example of a case where a subject is detected in the EyeArea in FIG. 7(b1). In FIG. 7(c22), the EyePosition position when the first limited area EyeArea was set in FIG. 7(b1) is used as the starting point, and a subject in the direction to which the user's gaze position has moved can be selected in combination with subject detection information within the EyeArea. When a subject is detected in the EyeArea in FIG. 7(b1) as in FIG. 7(c22), TargetArea_A and Target_B are displayed using the subject information acquired by subject detection. FIG. 7(c22) shows a state where the user's gaze position has moved to the right side of the screen, with the EyePosition located to the right of the EyeArea. Then, when the gaze position is confirmed using the gaze input operation unit 701, the EyePosition is located within TargetArea_A, and therefore AFArea is set to Target_A in combination with the detection information of Target_A, as shown in FIG. 7(d). In this embodiment, the case where EyePosition exists within the subject detection frame has been described, but the determination method is not limited to this, and a determination area may be set inside or outside the subject detection frame. Also, although the case where there are two subjects has been described, the subject is not limited to two people, and a detectable subject other than a person (such as an animal or a vehicle) may also be used.

[0121] Finally, the selection method of FIG. 7(c3) will be described. In FIG. 7(c3), the user's gaze position is moved and then confirmed by the gaze input operation unit 701, and the EyeArea, which is the quadrant selected in FIG. 7(b2), is further divided into four, thereby setting the AFArea as shown in FIG. 7(d). Note that in this embodiment, the method of dividing into four has been described, but this is not limited to this as long as the display allows the areas to be distinguished from one another. For example, the area may be divided into more than nine parts, or may be limited to only the left-right direction or only the vertical direction. Furthermore, although the AFArea is set as shown in FIG. 7(d) by finely adjusting the gaze position once, fine adjustment of the gaze position may be repeated until the setting is complete.

[0122] As described above, by performing dynamic calibration correction of the gaze position using gaze information, subject information, and scene determination information, it is possible to provide an imaging device that can always perform gaze detection with high accuracy.

[0123] Furthermore, by making the display of the gaze position still or moving depending on the amount of change in the gaze position, it is possible to provide an electronic device that improves display quality and convenience.

[0124] Furthermore, by performing coarse and fine adjustment of the gaze position, the gaze position of the user can be set more accurately.

[0125] In this embodiment, the line of sight information, subject information, and scene determination information are all used, but correction may be performed using at least one of them. Also, the calibration data may be reset depending on whether the observer's eye is close to or far from the viewfinder.

[0126] In addition, in the present embodiment, a method for determining whether the user is still or moving based on the amount of change in gaze position has been described, but the determination may also be made using the range of change angle in gaze position information, vector information, etc. Furthermore, the determination may also be made using information such as gyro information of an electronic device in addition to gaze position information.

[0127] Furthermore, in this embodiment, the value of the first threshold Th1 used in the horizontal position, vertical position, and inter-point position has been described as being the same, but it is possible to change it to a different value, and it may also be changed to a different value for each.

[0128] <Second embodiment> Next, a second embodiment will be described with reference to Figures 16 and 17. In this embodiment, a method for displaying the gaze position when combined with a focus detection device will be described.

[0129] [Explanation of focus detection operation] FIG. 2 shows the pixel configuration according to this embodiment and the correspondence between the pupil plane and the photoelectric conversion unit. Reference numeral 201 denotes the photoelectric conversion unit, 253 denotes the pupil plane, 251 denotes a microlens, and 252 denotes a color filter. FIG. 2 shows two photoelectric conversion units 201: a photoelectric conversion unit 201a (first focus detection pixel) and a photoelectric conversion unit 201b (second focus detection pixel). In the photoelectric conversion unit 201a, light passing through the pupil plane indicated by 253a is incident on the photoelectric conversion unit 201a. In the photoelectric conversion unit 201b, light passing through the pupil plane indicated by 253b is incident on the photoelectric conversion unit 201b. This enables focus detection based on signals obtained from the photoelectric conversion unit 201a and the photoelectric conversion unit 201b. An imaging signal can be generated by adding the signals obtained from the photoelectric conversion unit 201a and the photoelectric conversion unit 201b.

[0130] In this embodiment, by providing the pixels shown in FIG. 2 over the entire screen area of ​​the imaging unit 22, it becomes possible to focus on any subject appearing on the screen by detecting the phase difference.

[0131] Although the present embodiment will be described using the focus detection method described above, the focus detection method is not limited to this. For example, the image capture unit 22 may be provided with dedicated focus detection pixels shown in FIG. 3 (described later) to perform focus detection. Furthermore, the image capture unit 22 may not be provided with focus detection pixels, but may be provided with only image capture pixels, and focus detection may be performed using a contrast method.

[0132] FIG. 3 shows the configuration of a pixel dedicated to focus detection and the correspondence between the pupil plane and the photoelectric conversion unit. Unlike FIG. 2, FIG. 3 shows a pixel dedicated to focus detection. The shape of the pupil plane 253 is determined by the aperture 254. Furthermore, in order to detect only light that has passed through the pupil plane 253, it is necessary to provide a paired pixel, a pixel that detects light from the pupil plane on the right side (not shown in FIG. 3), to acquire a focus detection signal. By providing the focus detection pixels and imaging pixels shown in FIG. 3 over the entire screen area in the imaging unit 22, it becomes possible to focus on any subject that appears on the screen by phase difference detection.

[0133] The digital camera 100 described above is capable of taking pictures using center single-point AF or face AF. Center single-point AF means performing AF on a single point at the center of the shooting screen. Face AF means performing AF on a face within the shooting screen detected by the face detection function. The face detection function will now be described. The system control unit 50 sends image data for which face detection is to be performed to the image processing unit 24. Under the control of the system control unit 50, the image processing unit 24 applies a horizontal band-pass filter to the image data. Also, under the control of the system control unit 50, the image processing unit 24 applies a vertical band-pass filter to the processed image data. These horizontal and vertical band-pass filters detect edge components from the image data.

[0134] The system control unit 50 then performs pattern matching on the detected edge components to extract candidate eyes, noses, mouths, and ears. The system control unit 50 then determines, from the extracted candidate eyes, those that satisfy preset conditions (e.g., the distance between the two eyes, the inclination, etc.) as eye pairs and narrows down the candidate eyes to only those with eye pairs. The system control unit 50 then associates the narrowed-down candidate eyes with other features (nose, mouth, ears) that form the corresponding face, and detects faces by passing them through a preset non-face condition filter. The system control unit 50 outputs the face information according to the face detection result and ends the processing. At this time, feature quantities such as the number of faces are stored in the system memory 52. ​​The method for realizing the face detection function is not limited to the above-described method, and the number, size, features, etc. of faces may also be detected using a known machine learning method. Furthermore, the type of subject is not limited to human faces, and animals, vehicles, etc. may also be detected.

[0135] As described above, image analysis of image data displayed in live view or playback mode can be performed to extract features of the image data and detect subject information. In this embodiment, face information is used as an example of subject information, but subject information can also include various other types of information such as red-eye detection, eye detection, blink detection, and smile detection.

[0136] Furthermore, face AE, face FE, and face WB can be performed simultaneously with face AF. Face AE optimizes the exposure of the entire screen according to the brightness of the detected face. Face FE adjusts the flash light centered on the detected face. Face WB optimizes the WB of the entire screen according to the color of the detected face.

[0137] [Explanation of gaze detection processing method, gaze position state determination and determination results] Next, a gaze detection processing method, gaze position state determination, and determination results of an electronic device according to a second embodiment of the present invention will be described with reference to Fig. 16. Fig. 16 is a flowchart for explaining focus detection, gaze detection, and shooting operations of the electronic device of the present embodiment. Fig. 16 shows operations during live view shooting in which shooting is performed from a live view state (video shooting state) such as a shooting standby state, and is realized mainly by the system control unit 50.

[0138] In S041, under the control of the system control unit 50, the imaging unit 22 is driven to acquire imaging data. The acquired imaging data is not for recording, as will be described later, but for detection and display, so an image with a smaller size than the recorded image is acquired. In S041, an image with sufficient resolution for focus detection, subject detection, or live view display is acquired. Here, since this is a drive operation for capturing video for live view display, shooting is performed using a so-called electronic shutter, which accumulates and reads charge for a period corresponding to the frame rate for live view display. The live view display performed here allows the user to check the shooting range and shooting conditions, and may be, for example, 30 frames per second (shooting interval 33.3 ms) or 60 frames per second (shooting interval 16.6 ms).

[0139] In S042, the system control unit 50 acquires focus detection data obtained from the first and second focus detection pixels included in the focus detection area from the imaging data obtained in S041. The system control unit 50 also adds the output signals of the first and second focus detection pixels to generate an imaging signal, and the image processing unit 24 acquires image data by applying color interpolation processing and the like. In this way, image data and focus detection data can be acquired with a single imaging operation. Note that if the imaging pixels and the first and second focus detection pixels are configured as separate pixels, image data is acquired by performing interpolation processing and the like on the focus detection pixels.

[0140] In S43, the system control unit 50 generates an image for live view display using the image processing unit 24 based on the image data obtained in S042, and displays it on the display unit 28. Note that the image for live view display is, for example, a reduced image matched to the resolution of the display unit 28, and the image processing unit 24 can also perform reduction processing when generating the image data in S042. In this case, the system control unit 50 displays the image data obtained in S042 on the display unit 28. As described above, since shooting and display are performed at a predetermined frame rate during live view display, the user can adjust the composition and exposure conditions during shooting via the display unit 28.

[0141] Furthermore, as described above, in this embodiment, it is possible to detect a human face, an animal, or the like as a subject. In S043, a frame or the like indicating the area of ​​the detected subject is also displayed in conjunction with the start of live view display. Note that, as in main flow S1 of the first embodiment, this embodiment may display an already acquired image instead of a live view image.

[0142] Hereinafter, steps S044 to S049 of this embodiment are the same as steps S12 to S17 of the first embodiment, and therefore the description thereof will be omitted.

[0143] If Sw1 is not detected as being on (or is detected as being off) in S050, the system control unit 50 proceeds to S058, where it determines whether the main switch included in the operation unit 70 has been turned off. On the other hand, if Sw1 is detected as being on in S050, the system control unit 50 proceeds to S051, where it sets the focus detection area to be focused on and performs focus detection. Here, the focus detection area is set using the gaze position whose detection began in S048. The gaze position detected in S048 may have an error with respect to the subject position intended by the user due to various factors.

[0144] In S051, the focus detection area is set using gaze position information that has been processed as described below. At this time, the gaze position and the center position of the focus detection area may be aligned. Furthermore, if there are candidates for the focus detection area determined by other means, such as a subject detection means, the gaze position may be associated with the detected subject area, and the closest subject area may be selected as the focus detection area. From S051 onwards, the setting of the focus detection area using the gaze position information and the focus detection process are repeatedly executed each time an image is captured. Here, the focus detection process will be described.

[0145] The defocus amount and direction are calculated for each focus detection area using focus detection data corresponding to the set focus detection area. In this embodiment, the system control unit 50 generates image signals for focus detection, calculates the amount of deviation (phase difference) between the focus detection signals, and calculates the amount of defocus and direction from the calculated amount of deviation.

[0146] Shading correction and filter processing are performed on the first and second focus detection signals obtained from the set focus detection area as image signals for focus detection, reducing the difference in light intensity between the pair of signals and extracting spatial frequency signals for phase difference detection.Next, shift processing is performed to relatively shift the first and second focus detection signals after filter processing in the pupil division direction, and a correlation amount that indicates the degree of signal coincidence is calculated.

[0147] The kth first focus detection signal after filter processing is denoted by A(k), the second focus detection signal is denoted by B(k), and the range of number k corresponding to the focus detection area is denoted by W. Furthermore, if the shift amount due to shift processing is denoted by S1 and the shift range of shift amount S1 is denoted by Γ1, the correlation amount COR is calculated by equation (6).

[0148]

number

[0149] By shifting the shift amount S1, the kth first focus detection signal A(k) and the k-S1th second focus detection signal B(k-S1) are matched and subtracted to generate a shift subtraction signal. The absolute value of the generated shift subtraction signal is calculated, and the sum of the number k is taken within the range W corresponding to the focus detection area to calculate the correlation amount COR(S1). If necessary, the correlation amount calculated for each row may be added across multiple rows for each shift amount.

[0150] Next, the real-valued shift amount that minimizes the correlation amount is calculated from the correlation amount using sub-pixel calculations, and is set as the image shift amount p1. The calculated image shift amount p1 is then multiplied by a conversion coefficient K1 that corresponds to the image height of the focus detection area, the F-number of the imaging lens (imaging optical system), and the exit pupil distance to determine the detected defocus amount.

[0151] In S052, the system control unit 50 drives the lens based on the defocus amount detected in the selected focus detection area. If the detected defocus amount is smaller than a predetermined value, it is not necessarily necessary to drive the lens.

[0152] In S053, the process performs the acquisition of the detection and display image performed in S041, the live view display performed in S043, and the focus detection process performed in S046. The detected subject area and gaze position information, as described above, are superimposed on the live view display. The process performed in S053 may be performed in parallel with the lens drive in S052. The focus detection area may also be changed to correspond to the obtained gaze position in accordance with the continuously updated live view display. After the focus detection process is completed, the process proceeds to S054, where the system controller 50 detects the on / off state of the second shutter switch 64 (Sw2), which indicates the start of shooting. The release (shooting trigger) switch, which is one of the operation units 70, can detect two levels of on / off depending on the amount of depression, and the aforementioned Sw2 corresponds to the second level of on / off of the release (shooting trigger) switch. If the system controller 50 does not detect that Sw2 is on in S054, it returns to S050 and detects the on / off state of Sw1.

[0153] Furthermore, subject detection according to this embodiment is performed from the detection image acquired in S041, and information on the detected subject position and range is acquired. In this embodiment, the acquired detected subject information and the gaze position information acquired in S048 are used to change the gaze position information. The acquired detected subject information is compared with the gaze position information, and if the gaze position is included within the detected subject range the first determination number of times, the stillness determination described above is performed, and the gaze information display on display unit 28 is displayed as still. Note that the determination range does not have to be within the detected subject range, but may be, for example, a range near the detected subject range. Furthermore, rather than displaying the gaze information display as still, it is also possible to not display it at all.

[0154] In this embodiment, the predetermined number of times N, the first determination number JN1, and the second determination number JN2 can be changed depending on the selected AF mode. AF modes include one-shot AF, which performs the aforementioned AF control only once when Sw1 detection is detected in S020, and servo AF, which continues to perform AF control while Sw1 detection is ongoing. First, when one-shot AF is set, it is considered that it is often used, for example, when the composition is already determined and a still subject is photographed at low speed. Therefore, in this embodiment, the first determination number and second determination number are set to be larger than when servo AF is set, and the first and second threshold values ​​are set to larger values. This allows fine adjustment of the gaze position display relative to the subject, enabling good gaze display.

[0155] On the other hand, when the servo AF mode is set, it is considered that it is often used, for example, when photographing a medium to fast-moving subject that requires framing operation. Therefore, the first determination count and the second determination count in this embodiment are set to be smaller than when the one-shot AF is set, and the first threshold and the second threshold are set to small values. This allows for quick action display, making it possible to display the line of sight well even for fast-moving subjects.

[0156] It should be noted that instead of changing the settings according to the AF mode as described above, the user may freely set the first number of determinations, the second number of determinations, and the first and second threshold values ​​of this embodiment.

[0157] If Sw2 is detected as being on in S054, the system control unit 50 proceeds to S055 and determines whether to record images. In this embodiment, image acquisition during continuous shooting is switched between processing for recording images and processing for image capture / display and focus detection. Switching may be alternated, or, for example, image capture / display and focus detection may be performed once every three shots. This allows for highly accurate focus detection without significantly reducing the number of images taken per unit time.

[0158] If it is determined in S055 that image recording is to be performed, the process proceeds to S056, where a photography subroutine is executed. Details of the photography subroutine will be described later. Once the photography subroutine is executed in S056, the process returns to S054, where it is determined whether Sw2 is on, i.e., whether a continuous shooting instruction has been issued.

[0159] If it is determined in S055 that image capture / display and focus detection are to be performed, the process proceeds to S057, where image capture / display and focus detection processing during continuous shooting are performed. The image capture / display and focus detection processing during continuous shooting are the same as those performed in S053. The difference is that the display period, display update rate (interval), and display delay of the captured images in S057 are different from those in S53, depending on the frame rate of continuous shooting and the process of generating recorded images. The system control unit 50, which functions as a display unit, controls these displays. As described above, the obtained gaze position information is used to set the focus detection area and associate it with the detected subject area. Details will be described later. Once image capture / display and focus detection processing during continuous shooting are performed in S057, the process returns to S054, where it is determined whether Sw2 is turned on, i.e., whether a continuous shooting instruction has been issued. If Sw1 is not turned on (or turned off) in S050 and the main switch is detected to be off in S058, focus detection and shooting operations are terminated. On the other hand, if it is not detected in S58 that the main switch is off, the process returns to S042, and image data and focus detection data are acquired.

[0160] [Explanation of the shooting subroutine] Next, the photographing subroutine executed in S026 of Fig. 16 will be described in detail with reference to the flowchart shown in Fig. 17. The series of operations in this subroutine is also realized mainly by the system control unit 50.

[0161] In S0161, the system control unit 50 executes exposure control processing to determine the shooting conditions (shutter speed, aperture value, and shooting sensitivity). This exposure control processing can be performed by the system control unit 50 based on luminance information of the image data, but any known technology can be used. In this embodiment, details such as the timing of obtaining the image data used in the exposure control processing will be described later. In S0161, the operation of the shutter 101 is controlled based on the determined aperture value and shutter speed. Furthermore, the system control unit 50 causes charge accumulation to occur during the period in which the imaging unit 22 is exposed through the shutter 101.

[0162] When the exposure period ends, in step S0162, the system control unit 50 performs image readout for high-resolution still image capture, i.e., readout of all pixels. The system control unit 50 also performs image readout from either the first focus detection pixels or the second focus detection pixels. The signals readout from the focus detection pixels are used to detect the focus state of the subject during image playback. Therefore, the amount of read data for the recorded image for high-resolution still image capture may be reduced by limiting the area or setting a low resolution for all pixels. If a signal from the first or second focus detection pixel is available, the difference with the image for high-resolution still image capture can be calculated to calculate the other focus detection signal. In this embodiment, to prioritize the S / N ratio of the image signal for high-resolution still image capture, the image signal for high-resolution still image capture and one focus detection signal are read out and recorded, and the other focus detection signal is calculated by calculation. Subsequent image-related processing is performed on the image signal for high-resolution still image capture and one focus detection signal.

[0163] In S0163, the system control unit 50 performs defective pixel correction processing on the read image data using the image processing unit 24. In S0164, the system control unit 50 applies image processing such as demosaic (color interpolation), white balance processing, gamma correction (tone correction), color conversion processing, edge enhancement processing, and encoding processing to the image data after the defective pixel correction using the image processing unit 24. In S0165, the system control unit 50 records the image signal for high-pixel still image shooting and one of the focus detection signals in the memory 32 as an image data file.

[0164] In S0166, the system control unit 50 records the characteristic information of the camera body in the memory 32 and the memory in the system control unit 50, in association with the captured image recorded in S0165. Examples of the characteristic information of the camera body include the following information: Shooting conditions (aperture value, shutter speed, sensitivity, etc.) Information about the image processing applied by the image processing unit 24, Light-receiving sensitivity distribution information of the imaging pixels and focus detection pixels of the imaging unit 22, - Information on vignetting of the shooting light beam within the camera body, Distance information from the mounting surface of the camera body and the lens unit to the imaging unit 22, ·Manufacturing error information

[0165] Note that, since the light sensitivity distribution information of the imaging pixels and focus detection pixels of the imaging unit 22 is information dependent on the on-chip microlenses ML and the photoelectric conversion units PD, information regarding these components may also be recorded. The light sensitivity distribution information is information regarding sensitivity according to a position at a predetermined distance on the optical axis from the imaging element. The light sensitivity distribution information may also be information regarding changes in sensitivity with respect to the angle of incidence of light.

[0166] In S0167, the system control unit 50 records lens unit characteristic information in memory 32 and in a memory within the system control unit 50, corresponding to the captured image recorded in S0165. Examples of lens unit characteristic information include exit pupil information, frame information, focal length and F-number information at the time of capture, aberration information, manufacturing error information, and subject distance information associated with the focus lens position at the time of capture. In S0168, the system control unit 50 records image-related information related to the captured image in memory 32 and in a memory within the system control unit 50. The image-related information may include information related to the focus detection operation before capture, subject movement information, and information related to the accuracy of the focus detection operation.

[0167] In S0169, the system control unit 50 displays a preview of the captured image on the display unit 28. This allows the user to easily check the captured image. The image used for the preview display in S0169 is intended for easy image checking, so the various processes performed in S0163 and S0164 do not need to be performed. If these various processes are not performed, the preview display in S0169 can be performed in parallel with the processes from S0163 onwards, thereby further reducing the time lag from exposure to display.

[0168] When the process of S0169 is completed, the system control unit 50 ends the photographing subroutine of S026 and proceeds to the process of S024 of the main routine. In this embodiment, during continuous shooting, the acquired images are displayed whether the images are recorded in the photographing subroutine of S026 or when the image capturing / display and focus detection process during continuous shooting is performed in S027.

[0169] As described above, by making the display of the gaze position still or moving depending on the amount of change in the gaze position, it is possible to provide an electronic device that improves display quality and convenience.

[0170] <Other Examples> In the above-described first embodiment, the case where the eye gaze position is coarsely adjusted and then finely adjusted is illustrated, but only one of these may be performed.

[0171] In the above-described embodiment, a method for determining whether a still state or a moving state is determined based on the AF mode and subject detection has been described. The processing of the above-described embodiment may be changed depending on the shooting mode (moving image, still image, etc.) and the operating method of the shooting device (whether shooting using the viewfinder or the rear LCD screen, etc.).

[0172] Furthermore, the order of steps performed in the operations described using flowcharts in the above embodiments can be changed as appropriate to achieve the same purpose.

[0173] The present invention can also be realized by supplying a program for realizing one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0174] In addition, although the present embodiment has been described using a digital camera as the device for taking photographs, the processing of the present embodiment may also be realized by other devices such as a smartphone, a personal computer (PC), a tablet computer, or a head-mounted display. [Explanation of symbols]

[0175] 50 System control section

Claims

1. a display means for displaying the captured image; an acquisition means for acquiring line-of-sight information including a line-of-sight position of an observer observing the display means; a display control means for controlling a display based on the line-of-sight information; a selection means for selecting a subject based on the line-of-sight information, the display control means controls, based on the line-of-sight information, to highlight a limited area that is a part of the captured image by dividing the limited area into a plurality of areas and displaying the divided area as the display that distinguishes the limited area from other areas; The electronic device is characterized in that the selection means selects a subject based on information about the viewer's line of sight with respect to the display.

2. 2. The electronic device according to claim 1, wherein the limited area is an area that includes the line of sight of the viewer.

3. 3. The electronic device according to claim 1, wherein the display unit displays the captured image by dividing it into a plurality of areas including the limited area.

4. the line-of-sight information includes a reliability of the line-of-sight position of the viewer acquired by the acquisition means, The electronic device according to any one of claims 1 to 3, characterized in that the area displayed by the display control means as the limited area when the reliability is low is larger than the area displayed by the display control means as the limited area when the reliability is high.

5. A method for controlling an electronic device having a display means for displaying a captured image, comprising: an acquiring step of acquiring line-of-sight information including a line-of-sight position of an observer observing the display means; a display control step of controlling a display based on the line-of-sight information; a selection step of selecting a subject based on the line-of-sight information, In the display control step, based on the line-of-sight information, the limited area that is a part of the captured image is controlled to be highlighted by dividing the limited area into a plurality of areas and displaying the divided areas as the display that distinguishes the limited area from other areas, The method for controlling an electronic device, wherein the selecting step selects a subject based on information about the viewer's line of sight with respect to the display.

Images