Control device and control method thereof

Abstract

To improve display quality and convenience of line-of-sight display.SOLUTION: A control device provided herein comprises acquisition means for acquiring information corresponding to a point of gaze of an observer, and control means for providing control to display an indicator corresponding to the point of gaze on display means. The control means does not change the display position of the indicator if change in the point of gaze is less than a first threshold for a first consecutive number of times.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a technique for detecting and displaying the gaze position of a viewer. [Background technology]

[0002] There is known a technique for detecting the gaze position of a viewer. Patent Document 1 discloses a technique for detecting the gaze position of a viewer looking at a display means and displaying an index indicating the detected gaze position of the viewer on the display means. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2020-119093 Summary of the Invention [Problem to be solved by the invention]

[0004] According to Patent Document 1, the detected gaze position can be visually confirmed, which improves user convenience. If the viewer's gaze position information is used as is, an unintended change in the viewer's gaze position may be reflected in the display, which may result in a decrease in display quality and convenience.

[0005] An object of the present invention is to provide a control device and a control method that can improve the display quality and convenience in displaying the gaze position. [Means for solving the problem]

[0006] In order to achieve the above object, the present invention comprises an acquisition means for acquiring information corresponding to a position at which an observer gazes, and a control means for controlling a display means to display an index corresponding to the position at which the observer gazes acquired by the acquisition means, wherein the control means controls the display of the index to be statically displayed without moving when a change in the position at which the observer gazes acquired by the acquisition means is smaller than a first threshold value a first number of times in succession, and controls the display of the statically displayed index to be changed to a moving display when a change in the position at which the observer gazes acquired by the acquisition means is larger than a second threshold value a second number of times in succession. and when the change in the position of the viewer's gaze acquired by the acquisition means is within the second threshold value and the change in the position occurs in a predetermined direction the second number of times in succession, the still-displayed indicator is controlled to be moved and changed to a moving display. The present invention is characterized in that it is configured to: [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the display quality and convenience in displaying the gaze position. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing a configuration according to a first embodiment; [Figure 2] FIG. 10 is a diagram showing the correspondence between the pupil plane of a pixel and a photoelectric conversion unit of an image pickup device according to a second embodiment. [Figure 3] FIG. 10 is a diagram showing the correspondence between the pupil plane and the aperture of a pixel of an image pickup device according to a second embodiment. [Figure 4] FIG. 1 is a diagram illustrating an example of a configuration of an eye-gaze input operation unit according to a first embodiment. [Figure 5] FIG. 10 is a diagram showing another example of the configuration of the eye-gaze input operation unit according to the first embodiment; [Figure 6] 1 is a main flowchart illustrating a gaze detection processing method for an electronic device according to a first embodiment, gaze position state determination, and determination results. [Figure 7] 1 is a flowchart illustrating a gaze detection process according to a first embodiment. [Figure 8] FIG. 10 is a diagram illustrating the gaze state determination process according to the first embodiment. [Figure 9]FIG. 1 is a diagram illustrating a stillness determination according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating a line-of-sight determination operation according to the first embodiment. [Figure 11] FIG. 1 is a diagram illustrating operation determination according to the first embodiment. [Figure 12] 10 is a flowchart illustrating focus detection, line-of-sight detection, and shooting operations according to a second embodiment. [Figure 13] 10 is a flowchart illustrating a photographing subroutine according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] First Embodiment Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0010] [Electronic device configuration] FIG. 1 is a block diagram showing the configuration of an imaging device including an electronic device according to an embodiment of the present invention. In FIG. 1, lens unit 150 is a lens unit equipped with an interchangeable photographic lens. Lens 103 is usually composed of multiple lenses, but for simplicity, only a single lens is shown here. Communication terminal 6 is a communication terminal through which lens unit 150 communicates with digital camera 100, and communication terminal 10 is a communication terminal through which digital camera 100 communicates with lens unit 150. Lens unit 150 communicates with system controller 50 via communication terminals 6 and 10, controls aperture 102 via aperture drive circuit 2 using an internal lens system control circuit 4, and adjusts focus by displacing the position of lens 103 via AF drive circuit 3.

[0011] The shutter 101 is a focal plane shutter that can freely control the exposure time of the imaging unit 22 under the control of the system control unit 50. The imaging unit 22 is an imaging element composed of a CCD, CMOS element, or the like that converts an optical image into an electrical signal. The A / D converter 23 converts an analog signal into a digital signal. The A / D converter 23 is used to convert the analog signal output from the imaging unit 22 into a digital signal. The signal obtained from the imaging unit 22 is used not only for imaging but also for exposure control and focus detection control. The imaging unit 22 has pixels in which a photoelectric conversion unit is divided for each microlens. By dividing the photoelectric conversion unit, the entrance pupil is divided, and a phase difference detection signal can be obtained from each photoelectric conversion unit. In addition, an imaging signal can be obtained by adding the signals from the divided photoelectric conversion units.

[0012] Such pixels have the advantage that they can be used as both focus detection pixels and imaging pixels.

[0013] The image processing unit 24 performs predetermined pixel interpolation, resizing such as reduction, and color conversion processing on data from the A / D converter 23 or data from the memory control unit 15. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained arithmetic results. This allows TTL (through-the-lens) AF (autofocus) processing, AE (autoexposure) processing, and EF (flash pre-flash) processing to be performed. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data, and TTL AWB (auto white balance) processing based on the obtained arithmetic results.

[0014] The output data from the A / D converter 23 is written into the memory 32 via the image processing unit 24 and the memory control unit 15, or directly via the memory control unit 15. The memory 32 stores image data obtained by the imaging unit 22 and converted into digital data by the A / D converter 23, and image data to be displayed on the display unit 28 as display means. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio.

[0015] The memory 32 also serves as a memory (video memory) for image display. The D / A converter 19 converts the image display data stored in the memory 32 into an analog signal and supplies it to the display unit 28. In this way, the display image data written to the memory 32 is displayed by the display unit 28 via the D / A converter 19. The display unit 28 displays the image according to the analog signal from the D / A converter 19 on a display such as an LCD. The digital signal that has been A / D converted once by the A / D converter 23 and stored in the memory 32 is converted to analog by the D / A converter 19 and sequentially transferred to and displayed on the display unit 28, thereby functioning as an electronic viewfinder and providing a through-image display (live view display). The display unit 28 may be provided with an electronic viewfinder that can be viewed through an eyepiece (not shown), or may be provided with a display on the back of the digital camera 100. Alternatively, both an electronic viewfinder and a display on the back may be provided.

[0016] The nonvolatile memory 56 is an electrically erasable and recordable memory, such as an EEPROM. The nonvolatile memory 56 stores constants, programs, etc. for the operation of the system control unit 50. The programs referred to here are programs for executing various flowcharts described later in this embodiment.

[0017] The system control unit 50 controls the entire digital camera 100. The system control unit 50 includes a line-of-sight display unit, a line-of-sight state determination unit, a line-of-sight information determination unit, a subject detection unit, and a line-of-sight acquisition determination unit. Each process described below in this embodiment is realized by executing a program stored in the nonvolatile memory 56. Reference numeral 52 denotes a system memory, which uses RAM. The system memory 52 stores constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 56, and other data. The system control unit also controls display by controlling the memory 32, the D / A converter 19, the display unit 28, and other components. The display unit 28 includes an EVF through which the user looks and a TFT LCD screen that can be touched and operated without looking into the display. The eye sensor installed in the EVF also controls the display unit 28 based on information indicating whether the user's eye is in focus or not.

[0018] The system timer 53 is a timekeeping unit that measures the time used for various controls and the time of a built-in clock.

[0019] The power switch 72 is an operating member for switching the power of the digital camera 100 on and off.

[0020] The mode selector switch 60, the first shutter switch 62, the second shutter switch 64, and the operation unit 70 are operation units for inputting various operation instructions to the system control unit 50. Note that the line of sight acquiring means and the line of sight determining means are included in this operation unit 70.

[0021] The mode selector switch 60 switches the operating mode of the system control unit 50 to one of still image recording mode, video shooting mode, playback mode, etc. Modes included in the still image recording mode include auto shooting mode, auto scene determination mode, manual mode, aperture priority mode (Av mode), and shutter speed priority mode (Tv mode). Other modes include various scene modes, program AE mode, and custom mode, which are shooting settings for different shooting scenes. The mode selector switch 60 can directly switch to one of these modes included in the menu button. Alternatively, after switching to the menu button with the mode selector switch 60, another operating member can be used to switch to one of these modes included in the menu button. Similarly, the video shooting mode may also include multiple modes.

[0022] The first shutter switch 62 is turned on and generates a first shutter switch signal SW1 when the shutter button 61 provided on the digital camera 100 is pressed halfway (a shooting preparation command) during operation. The first shutter switch signal SW1 starts operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing.

[0023] The second shutter switch 64 is turned on when the shutter button 61 is fully pressed (photographing instruction) and generates a second shutter switch signal SW2. The system control unit 50 starts a series of photographing processing operations, from reading out a signal from the imaging unit 22 to writing image data to the recording medium 200, in response to the second shutter switch signal SW2.

[0024] Each operating member of the operating unit 70 is assigned a function appropriate for each situation by selecting and operating various function icons displayed on the display unit 28, and acts as various function buttons. The function buttons include, for example, an end button, a back button, an image forward button, a jump button, a filter button, and an attribute change button. For example, when the menu button is pressed, a menu screen on the display unit 28 on which various settings can be made is displayed. The user can intuitively make various settings using the menu screen displayed on the display unit 28, the four directional buttons (up, down, left, and right), and the SET button.

[0025] The operation unit 70 is an input unit that accepts operations from the user and includes various operation members, such as electronic buttons and a cross key for selecting a menu, selecting a mode, and playing back captured moving images.

[0026] The power supply control unit 80 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, etc., and detects whether a battery is installed, the type of battery, and the remaining battery power. The power supply control unit 80 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the required voltage to each unit, including the recording medium 200, for the required period of time.

[0027] The power supply unit 30 is composed of primary batteries such as alkaline batteries or lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries or Li batteries, an AC adapter, etc. The recording medium I / F 18 is an interface with a recording medium 200 such as a memory card or hard disk. The recording medium 200 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, a magnetic disk, etc.

[0028] The communication unit 54 is connected wirelessly or via a wired cable and transmits and receives video and audio signals. The communication unit 54 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 54 can transmit images (including through images) captured by the imaging unit 22 and images recorded on the recording medium 200, and can also receive image data and various other information from external devices.

[0029] The orientation detection unit 55 detects the orientation of the digital camera 100 with respect to the direction of gravity. Based on the orientation detected by the orientation detection unit 55, it is possible to determine whether an image captured by the imaging unit 22 was captured with the digital camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the orientation detected by the orientation detection unit 55 to the image file of the image captured by the imaging unit 22, or rotate and record the image. The orientation detection unit 55 can be an acceleration sensor, a gyro sensor, or the like.

[0030] [Gaze detection configuration] In this embodiment, a gaze input operation unit 701 is provided as one of the operation units 70. The gaze input operation unit 701 is a gaze acquisition means, and is an operation member for detecting at which point on the display unit 28 the user's gaze is directed. Fig. 4 shows an example of the gaze input operation unit 701.

[0031] FIG. 4 illustrates a configuration for implementing a method disclosed in Patent Document 1, in which the rotation angle of the optical axis of a user's eyeball 501a looking into the viewfinder field is detected and the user's line of sight is detected from the detected rotation angle. A live view image captured through the lens unit 100 is displayed on the display unit 28. Reference numeral 701a denotes an image sensor, 701b denotes a light-receiving lens, 701c denotes a dichroic mirror, 701d denotes an eyepiece, and 701e denotes an illumination light source. The illumination light source 701e projects infrared light onto the eyeball 501a. The infrared light reflected by the eyeball 501a is reflected by the dichroic mirror 701c and captured by the image sensor 701a. The captured eyeball image is converted into a digital signal by an A / D converter (not shown) and transmitted to the system control unit 50. The system control unit 50, which serves as a line of sight acquisition means, extracts the pupil area and other information from the captured eyeball image to calculate the user's line of sight.

[0032] Note that the gaze input operation unit 701 is not limited to this method, and may instead be a method of capturing an image of both of the user's eyes and detecting the gaze. FIG. 5 shows an example of the gaze input operation unit 701 different from that shown in FIG. 4. In FIG. 5, a live view display image captured through the lens unit 100 is displayed on the display unit 28 provided on the back of the digital camera 100. In FIG. 5, a camera 701f is provided on the back of the digital camera 100 to capture an image of the user's face 500 observing the display unit 28. In FIG. 5, the angle of view captured by the camera 701f is indicated by a dotted line. Light is projected onto the user's face from an illumination light source 701e (not shown), and an image of the user's eyeballs is acquired by the camera 701f. This allows the user's gaze to be calculated. Note that the gaze input operation unit 701 is not limited to this method, and may be any other configuration that can detect which part of the display unit 28 the user is gazing at.

[0033] [Gaze detection processing] Hereinafter, the gaze detection processing method, gaze position state determination, and display method based on the determination result in the first embodiment will be described with reference to FIGS.

[0034] Fig. 6 is a main flowchart illustrating a gaze detection processing method for an electronic device according to the first embodiment, gaze position state determination, and the determination result. Fig. 6 shows the operation when a user performs a gaze determination operation while looking at an image displayed on the display unit 28, which is a display means, and is realized mainly by the system control unit 50. Note that, although the present embodiment will be described using the block diagram of the imaging device in Fig. 1, similar operations are possible in an electronic device having at least the system control unit 50, the display unit 28, and the operation unit 70.

[0035] In S1, the system control unit 50 displays the image data on the display unit 28. The image to be displayed is, for example, a reduced image that matches the resolution of the display unit 28. While the image is being displayed, it is displayed at a predetermined frame rate, and the user can check the displayed image through the display unit 28.

[0036] In S2, the system control unit 50 determines whether or not gaze information can be acquired using the gaze acquisition availability determination means. If the eye sensor is installed in the EVF of the display unit 28, it determines whether or not gaze information can be acquired based on information on whether the user has their eyes in close contact with the EVF. If the user is looking into the EVF, the eye sensor determines that the user has their eyes in close contact with the EVF, and the gaze input operation unit 701, which is the gaze acquisition means, determines that gaze information can be acquired. On the other hand, if the user is not looking into the EVF, the eye sensor determines that the user has their eyes in close contact with the EVF, and the gaze input operation unit 701, which is the gaze acquisition means, determines that gaze information cannot be acquired. It also determines whether or not gaze information can be acquired by the gaze input operation unit 701 for the TFT liquid crystal of the display unit 28.

[0037] If it is determined in S2 that gaze information can be acquired, the system control unit 50 starts gaze detection in S3. From S4 onwards, the gaze input operation unit 701 acquires and stores at predetermined time intervals the position on the display unit 28 at which the user is observing (gaze position) in association with the displayed image the user was observing. Also, if it is determined in S2 that gaze information cannot be acquired, the gaze information at the determined timing is not acquired or stored.

[0038] In S4, it is determined whether the number of times gaze information has been acquired and accumulated in S3 is equal to or greater than a predetermined number N. The predetermined number N is determined by the number of times that gaze state determination can be performed in S5 at a later stage. If it is determined in S2 that acquisition is not possible, the information is not acquired or accumulated, and is therefore not counted as the predetermined number N in S4. If it is determined in S3 that the number of times gaze information has been acquired and accumulated is equal to or greater than the predetermined number N, it is determined that gaze state determination by S5, the gaze state determination means, is possible, and the process proceeds to S5. If it is determined in S3 that the number of times gaze information has been acquired and accumulated is less than the predetermined number N, it is determined that gaze state determination by S5, the gaze state determination means, is not possible, and the process skips S5 and proceeds to S6.

[0039] In S5, the system control unit 50 performs a gaze state determination process, which will be described in detail later.

[0040] In S6, the gaze position information acquired in S3 or S5 is used to display the gaze position on display unit 28. If it is determined in S4 that the gaze accumulation count is equal to or greater than the predetermined count N, the gaze position acquired in S5 is used. If it is determined in S4 that the gaze accumulation count is less than the predetermined count N, the gaze position acquired in S3 is used.

[0041] In S7, it is determined whether or not the user has performed a gaze confirmation operation. If the user has performed a gaze confirmation operation, the display of the gaze position on the display unit 28 is stopped, and this gaze position state determination and display processing flow is terminated. Note that the display of the gaze position on the display unit 28 may be continued, or the display method may be changed. Also, if the user has not performed a gaze confirmation operation, the process proceeds to S2, in which the gaze acquisition possibility determination means determines whether or not gaze information can be acquired, and new gaze information is acquired. Note that if a gaze confirmation operation is performed in S7, the number of times gaze information has been acquired and accumulated in S3 and gaze position change information ΔS are j Reset.

[0042] [Gaze position status determination] The gaze state determination in S5 will be described with reference to the gaze state determination sub-flow diagram in Fig. 7. The series of operations in this sub-flow is also realized mainly by the system control unit 50.

[0043] S501 is the gaze position information S acquired in S3 of FIG. i Using this, the gaze position change information ΔS j In S4 of FIG. 6, it is determined that the number of times is equal to or greater than the predetermined number N, and the gaze position change information ΔS j Among them, ΔS corresponding to the horizontal position change jH , ΔS corresponding to the vertical position change jV , ΔS corresponding to the position change between the two points jr ΔS jr are calculated using the following formulas (1), (2), and (3), respectively. S i ={x i , y i} ΔS jH =x j+1 -x j (j=1~N-1) (1) ΔS jV =y j+1 -y j (j=1~N-1) (2)

[0044]

number

[0045] In step S502, the first determination count JN1 and the first threshold value Th1 are used to calculate the gaze position change information ΔS j The relationship between the predetermined number of times N and the number of times of determination JN1 is as follows: N≧JN1(4) The relationship is expressed by the above formula (4). In this embodiment, JN1 is set to a fixed value of 5 times, but it may be changed depending on the conditions. In S502, the gaze position change information ΔS acquired in S501 is jincludes the judgment time, and if it is judged that the first judgment count JN1 times in the past is equal to or less than the first threshold value Th1, the process proceeds to S505 to freeze (not move) the display of the gaze position displayed on the display unit 28. Also, the gaze position change information ΔS acquired in S501 j includes the judgment time, and if it has been judged to be equal to or greater than the first threshold value Th1 at least once during the past first judgment count JN1 times, the judgment is made using the second threshold value, and the process proceeds to S503.

[0046] In step S503, the second determination count JN1 and the second threshold value Th1 are used to calculate the gaze position change information ΔS j The relationship between the predetermined number of times N and the number of times of determination JN2 is as follows: N ≧ JN2(5) The relationship is expressed by the above formula (5). In this embodiment, JN2 is set to a fixed value of 5 times, but this may be changed depending on the conditions. In S503, the gaze position change information ΔS j includes the judgment time, and judges whether the second judgment count JN2 times in the past consecutively is equal to or greater than the second threshold value Th2. If it is judged that the second judgment count JN2 times is equal to or greater than the second threshold value Th2, the process proceeds to S504 to calculate a movement start position for operating (moving) the display of the gaze position displayed on the display unit 28. In addition, the gaze position change information ΔS acquired in S501 is used to calculate the movement start position for operating (moving) the display of the gaze position displayed on the display unit 28. j includes the judgment time, and if it is judged to be equal to or less than the second threshold Th2 even once during the past second judgment number JN2 times, the display is not changed from the previous display. i The process proceeds to S506 to determine the line-of-sight display position to be displayed on the display unit 28 using the above.

[0047] S504 is performed if it is determined in S503 that the value is equal to or greater than the second threshold value Th2. The operation position of the line-of-sight display is calculated, and the process proceeds to S506.

[0048] S505 is performed when it is determined in S502 that the value is equal to or less than the first threshold value Th1. The stationary position of the gaze display is calculated, and the process proceeds to S506. After completing this series of steps, the process proceeds to S6 in the main flow of FIG. 6.

[0049] In S506, the gaze display position is determined.

[0050] [Determining whether to fix the display position and calculating the display position] Here, the determination method for freezing the gaze position display and calculation of the fixed display position performed in S502 will be described with reference to Figures 8 and 9. Figure 8 shows an example in which a captured video or a video being captured is displayed on display unit 28 and the user performs a gaze confirmation operation on a specific subject with their gaze, and describes a method for freezing the gaze information display using a first determination count JN1 and a first threshold value Th1. Figure 8 shows excerpts of images displayed on display unit 28 at times t1 (Figure 8(a)), t2 (Figure 8(b)), and t3 (Figure 8(c)) from the left. Sm represents the gaze information display, and Trg represents the subject that the user is following with their gaze.

[0051] 8(a) shows the moment when a subject Trg on a bicycle moves in from the left side of the screen. The image also shows the user recognizing the subject Trg and beginning to make eye contact with him. At this moment, the gaze information display Sm does not yet match the subject Trg, and the gaze information display Sm is scattered as indicated by the dashed lines.

[0052] Figure 8(b) shows the moment when the subject Trg slows down the speed of his bicycle to a stop. It also shows the subject Trg and the gaze information display Sm at time t2, after time t1 in Figure 8(a). At this moment, the gaze information display Sm begins to roughly align with the subject Trg, but as indicated by the dashed lines, the gaze information display Sm varies.

[0053] FIG. 8(c) shows the moment when the subject Trg stops the bicycle and stands. Also, FIG. 8(b) shows the subject Trg and the line-of-sight information display Sm at time t3, after time t2. At this moment, the line-of-sight information display Sm matches the subject Trg, and the line-of-sight information display Sm is displayed stationary by using the first determination number JN1 and the first threshold value Th1. Next, the series of steps in FIG. 8 will be compared with the line-of-sight position change information ΔS in FIG.j The following description will be given using the first determination count JN1 and the first threshold value Th1. Fig. 9 explains the still state determination using the first determination count JN1 and the first threshold value Th1. Note that, although Fig. 9 shows the first determination count JN1 as five times, it is not limited to this number and may be changed depending on the user's line of sight characteristics, the viewing state of the display unit 28, or manual operation by the user.

[0054] FIG. 9(a) shows the gaze position change information ΔS j Among them, ΔS corresponding to the horizontal position change jH The two solid horizontal lines represent the threshold value Th1 corresponding to the horizontal position change amount. 1H The gray hatched area represents the number of determinations JN1 corresponding to the amount of change in horizontal position. 1H represents.

[0055] FIG. 9(b) shows the gaze position change information ΔS j Among them, ΔS corresponding to the vertical position change jV The two horizontal solid lines represent the threshold value Th1 corresponding to the vertical position change amount. 1V The gray hatched area represents the number of determinations JN1 corresponding to the amount of change in horizontal position. 1V represents.

[0056] FIG. 9(c) shows the gaze position change information ΔS j Among them, ΔS corresponding to the position change between two points jr The two horizontal solid lines represent the threshold value Th1 corresponding to the position change amount between two points. 1r The gray hatched area represents the number of judgments JN1 corresponding to the amount of position change between two points, out of the first number of judgments JN1. 1r represents.

[0057] FIG. 9(a) shows the gaze position change information ΔS jHThis is an explanation of the case where the stillness determination of the gaze information display is performed in the time t1 to the time t2. jH is gradually decreasing but still corresponds to the threshold value Th 1H After that, the gaze position change information ΔS for the horizontal position change amount from time t2 to time t3 is jH is the threshold value Th corresponding to the horizontal position change amount. 1H The horizontal position change amount is within the threshold Th 1H In this embodiment, the number of determinations JN corresponding to the amount of change in horizontal position is counted up. 1H In order to explain this as five times, from time t3, which is the fifth time, the line of sight information display on the display unit 28 is changed to a fixed value so as to be displayed statically. The fixed value is the number of determinations JN corresponding to the horizontal position change amount. 1H The average value of the horizontal position change amount is used as a fixed value. 1H It may also be the last value of

[0058] FIG. 9(b) shows the gaze position change information ΔS jV This is an explanation of the case where the stillness determination of the gaze information display is performed. jV is a threshold Th that gradually decreases but still corresponds to the vertical position change amount. 1V After that, the gaze position change information ΔS jV is the threshold value Th corresponding to the vertical position change amount. 1V The vertical position change amount is converged as time passes. 1V In this embodiment, the number of determinations JN corresponding to the amount of change in vertical position is 1VIn order to explain this as five times, from time t3, which is the fifth time, the line of sight information display on the display unit 28 is changed to a fixed value so as to be displayed statically. The fixed value is the fixed position of the display, and the number of determinations JN corresponds to the vertical position change amount. 1V The average value of the number of judgments JN corresponding to the vertical position change is used as a fixed value. 1V It may also be the last value of

[0059] FIG. 9(c) shows the gaze position change information ΔS jr This is an explanation of the case where the stillness determination of the gaze information display is performed with the above-mentioned ΔS jH and ΔS jV Unlike ΔS jr Since is a positive value, the threshold value Th corresponding to the position change between two points is 1r is only a positive value. The gaze position change information ΔS for the position change amount between two points from time t1 to time t2 jr is gradually decreasing but still remains at the threshold Th corresponding to the position change between two points. 1r After that, the gaze position change information ΔS for the amount of position change between the two points from time t2 to time t3 is jr is the threshold value Th corresponding to the position change amount between two points. 1r The value of the threshold Th corresponding to the position change amount between the two points from time t2 is 1r In this embodiment, the number of determinations JN corresponding to the amount of position change between the two points is 1r In order to explain this as five times, from time t3, which is the fifth time, the line of sight information display on the display unit 28 is changed to a static display with a fixed value. The fixed value is a fixed position of the display, and the number of determinations JN corresponds to the amount of position change between two points. 1r The average position of the two points is set as a fixed value. 1r The position may be determined from the last value of

[0060] As described above, by statically displaying the line-of-sight information display in response to changes in line-of-sight information, it is possible to provide an electronic device that improves display quality and convenience.

[0061] [Determining whether to change the display position and calculating the display position] Here, the determination method for freezing the gaze position display and calculation of the fixed display position performed in S503 will be described with reference to Figures 10 and 11. Figure 10 shows an example in which a captured video or a video being captured is displayed on display unit 28 and the user performs a gaze confirmation operation on a specific subject with their gaze. A method for freezing the gaze information display using a second determination count JN2 and a second threshold Th2 will be described. Figure 10 shows excerpts of images displayed on display unit 28 at times t1' (Figure 10(a)), t2' (Figure 10(b)), and t3' (Figure 10(c)) from the left. Sm indicates the gaze information display as in Figure 8, and Trg' indicates a subject different from that in Figure 8 that the user is following with their gaze.

[0062] FIG. 10(a) shows the moment when the subject, a car Trg', is about to move from the left side of the screen. The user continues to recognize the car Trg' and their gazes are fixed on it. At this moment, the gaze information display Sm is aligned with the car Trg', and due to the processing of S502 described above, the gaze information display Sm is still. Note that the gaze information display Sm may be operating before the processing of S502 is executed.

[0063] Figure 10(b) shows the moment when the subject Trg' starts to move and picks up speed. It also shows the subject Trg' and the line-of-sight information display Sm at time t2', after time t1' in Figure 10(a). At this moment, the line-of-sight information display Sm is beginning to diverge from the subject Trg', but the line-of-sight information display Sm is stationary due to the processing of S502 described above. It should be noted that the line-of-sight information display Sm may be operating before the processing of S502 is executed.

[0064] Figure 10(c) shows the moment when the subject Trg', a car, is stopped. It also shows the subject Trg' and the line-of-sight information display Sm at time t3', after time t2' in Figure 10(b). At this moment, the line-of-sight information display Sm matches the subject Trg', and the line-of-sight information display Sm is displayed in a dynamic manner using the second determination count JN2 and the second threshold value Th2, resulting in a variation in the display as shown by the dashed lines.

[0065] Next, the series of steps in Figure 10 are shown in Figure 11. j The following description will be given using the second determination count JN2 and the second threshold value Th1. Fig. 11 explains the stillness determination using the second determination count JN2 and the second threshold value Th2 of the present invention. Note that, although Fig. 11 shows the second determination count JN2 as five times, it is not limited to this number and may be changed depending on the user's line of sight characteristics, the viewing state of the display unit 28, or manual operation by the user.

[0066] FIG. 11(a) shows the gaze position change information ΔS at time t'. j Among them, ΔS corresponding to the horizontal position change jH The two solid horizontal lines represent the second threshold value Th2, which corresponds to the horizontal position change amount. 2H The gray hatched area represents the number of determinations JN2 corresponding to the horizontal position change amount. 2H represents.

[0067] FIG. 11(b) shows the gaze position change information ΔS j Among them, ΔS corresponding to the vertical position change jV The two horizontal solid lines represent the second threshold Th2, which corresponds to the vertical position change amount. 2V The gray hatched area represents the number of determinations JN2 corresponding to the horizontal position change amount. 2V represents.

[0068] FIG. 11(c) shows the gaze position change information ΔSj Among them, ΔS corresponding to the position change between two points jr The two horizontal solid lines represent the second threshold Th2, which corresponds to the amount of position change between two points. 2r The gray hatched area represents the number of judgments JN2 corresponding to the amount of position change between two points. 2r represents.

[0069] FIG. 11(a) shows the gaze position change information ΔS jH This is an explanation of the case where the stillness determination of the gaze information display is performed at time t1' to time t2'. jH is gradually increasing, and the threshold value Th corresponding to the horizontal position change amount is 1H After that, the gaze position change information ΔS for the horizontal position change amount from time t2' to time t3' is jH is the threshold value Th corresponding to the horizontal position change amount. 2H The horizontal position of the object Trg′ is changed from time t2′ to time t3′. 2H In this embodiment, the number of determinations JN corresponding to the amount of change in horizontal position is counted up. 2H For the explanation, the number of times is set to five, and from the fifth time t3', the line-of-sight information display on the display unit 28 is changed to be displayed in motion. The value for displaying is the fixed position of the display, and the number of determinations JN corresponding to the amount of change in horizontal position is 2H The average value of the horizontal position change amount is used as a fixed value. 2H The last value of the horizontal position change information ΔS jH is the threshold value Th corresponding to the horizontal position change amount. 2H Even if the horizontal position change is within the range, the number of judgments JN 2H If the movement is always continuous in the same direction, the following may be done: That is, the line-of-sight information display may be changed to an animated display.

[0070] FIG. 11(b) shows the gaze position change information ΔS jV This is an explanation of the case where the stillness determination of the gaze information display is performed at time t1' to time t2'. jV is roughly constant, and always corresponds to the threshold value Th 2V At this time, the vertical position change amount is lower than the threshold value Th 2V Therefore, the eye gaze position change information ΔS jV In this case, it is determined that the line-of-sight information display on the display unit 28 remains statically displayed as a fixed value even at time t3'.

[0071] FIG. 11(c) shows the gaze position change information ΔS jr This is an explanation of the case where the stillness determination of the gaze information display is performed with the above-mentioned ΔS jH and ΔS jV Unlike ΔS jr Since is a positive value, the threshold value Th corresponding to the position change between two points is 2r The gaze position change information ΔS for the amount of position change between two points from time t1' to time t2' is jr is gradually increasing, but is still the threshold value Th corresponding to the position change between two points. 2r After that, the gaze position change information ΔS for the amount of position change between the two points from time t2' to time t3' jr is the threshold value Th corresponding to the position change amount between two points. 2r The change in the position between the two points from time t2' is within the threshold value Th 2r In this embodiment, the number of determinations JN corresponding to the amount of position change between the two points is counted up. 2r In order to explain this as five times, the line of sight information display on the display unit 28 is changed to be displayed in motion from time t3', which is the fifth time. The value to be displayed is the fixed position of the display, and the number of determinations JN corresponding to the amount of position change between two points is 2rThe average position of the two points is set as a fixed value. 2r The position may be determined from the last value of

[0072] As described above, by dynamically displaying the line-of-sight information display in response to changes in line-of-sight information, it is possible to provide an electronic device that improves display quality and convenience.

[0073] 10 and 11, the amount of change in gaze position ΔS j However, it is also possible to determine whether the gaze position is stationary or moving depending on whether the direction of the change is within a predetermined range. For example, ±45 degrees can be selected as the predetermined range. This makes it possible to determine whether the gaze position is changing with variation.

[0074] As described above, by making the display of the gaze position still or moving depending on the amount of change in the gaze position, it is possible to provide an electronic device that improves display quality and convenience. In this embodiment, a method for determining whether the display is still or moving depending on the amount of change in the gaze position has been described, but the determination may also be made using the range of change angle of the gaze position information, vector information, etc. Furthermore, the determination may also be made using information such as gyro information of the electronic device in addition to the gaze position information.

[0075] In addition, in this embodiment, the value of the first threshold Th1 used for the horizontal position, vertical position, and position between two points has been described as being the same, but it is possible to change it to a different value, and furthermore, it may be changed to a different value for each.

[0076] <Second embodiment> Next, a second embodiment will be described with reference to Figures 12 and 13. In this embodiment, a method for displaying the gaze position when combined with a focus detection device will be described.

[0077] [Explanation of focus detection operation] FIG. 2 shows the pixel configuration according to this embodiment and the correspondence between the pupil plane and the photoelectric conversion unit. Reference numeral 201 denotes the photoelectric conversion unit, 253 denotes the pupil plane, 251 denotes a microlens, and 252 denotes a color filter. FIG. 2 shows two photoelectric conversion units 201: a photoelectric conversion unit 201a (first focus detection pixel) and a photoelectric conversion unit 201b (second focus detection pixel). In the photoelectric conversion unit 201a, light passing through the pupil plane indicated by 253a is incident on the photoelectric conversion unit 201a. In the photoelectric conversion unit 201b, light passing through the pupil plane indicated by 253b is incident on the photoelectric conversion unit 201b. This enables focus detection based on signals obtained from the photoelectric conversion unit 201a and the photoelectric conversion unit 201b. An imaging signal can be generated by adding the signals obtained from the photoelectric conversion unit 201a and the photoelectric conversion unit 201b.

[0078] In this embodiment, by providing the pixels shown in FIG. 2 over the entire screen area of ​​the imaging unit 22, it becomes possible to focus on any subject appearing on the screen by detecting the phase difference.

[0079] Although the present embodiment will be described using the focus detection method described above, the focus detection method is not limited to this. For example, the image capture unit 22 may be provided with dedicated focus detection pixels shown in FIG. 3 (described later) to perform focus detection. Furthermore, the image capture unit 22 may not be provided with focus detection pixels, but may be provided with only image capture pixels, and focus detection may be performed using a contrast method.

[0080] FIG. 3 shows the configuration of a pixel dedicated to focus detection and the correspondence between the pupil plane and the photoelectric conversion unit. Unlike FIG. 2, FIG. 3 shows a pixel dedicated to focus detection. The shape of the pupil plane 253 is determined by the aperture 254. Furthermore, in order to detect only light that has passed through the pupil plane 253, it is necessary to provide a paired pixel, a pixel that detects light from the pupil plane on the right side (not shown in FIG. 3), to acquire a focus detection signal. By providing the focus detection pixels and imaging pixels shown in FIG. 3 over the entire screen area in the imaging unit 22, it becomes possible to focus on any subject that appears on the screen by phase difference detection.

[0081] The digital camera 100 described above is capable of taking pictures using center single-point AF or face AF. Center single-point AF means performing AF on a single point at the center of the shooting screen. Face AF means performing AF on a face within the shooting screen detected by the face detection function.

[0082] The face detection function will now be described. The system control unit 50 sends image data for which face detection is to be performed to the image processing unit 24. Under the control of the system control unit 50, the image processing unit 24 applies a horizontal band-pass filter to the image data. Also, under the control of the system control unit 50, the image processing unit 24 applies a vertical band-pass filter to the processed image data. These horizontal and vertical band-pass filters detect edge components from the image data.

[0083] The system control unit 50 then performs pattern matching on the detected edge components to extract candidate eyes, noses, mouths, and ears. The system control unit 50 then determines, from the extracted candidate eyes, those that satisfy preset conditions (e.g., the distance between the two eyes, the inclination, etc.) as eye pairs and narrows down the candidate eyes to only those with eye pairs. The system control unit 50 then associates the narrowed-down candidate eyes with other features (nose, mouth, ears) that form the corresponding face, and detects faces by passing them through a preset non-face condition filter. The system control unit 50 outputs the face information according to the face detection result and ends the processing. At this time, feature quantities such as the number of faces are stored in the system memory 52. ​​The method for realizing the face detection function is not limited to the above-described method, and the number, size, features, etc. of faces may also be detected using a known machine learning method. Furthermore, the type of subject is not limited to human faces, and animals, vehicles, etc. may also be detected.

[0084] As described above, image analysis of image data displayed in live view or playback mode can be performed to extract features of the image data and detect subject information. In this embodiment, face information is used as an example of subject information, but subject information can also include various other types of information such as red-eye detection, eye detection, blink detection, and smile detection.

[0085] Furthermore, face AE, face FE, and face WB can be performed simultaneously with face AF. Face AE optimizes the exposure of the entire screen according to the brightness of the detected face. Face FE adjusts the flash light centered on the detected face. Face WB optimizes the WB of the entire screen according to the color of the detected face.

[0086] Next, a gaze detection processing method, gaze position state determination, and determination results of an electronic device according to a second embodiment will be described with reference to Fig. 12. Fig. 12 is a flowchart for explaining focus detection, gaze detection, and shooting operations of the electronic device according to this embodiment. Fig. 12 shows the operation during live view shooting, in which shooting is performed from a live view state (video shooting state) such as a shooting standby state, and is realized mainly by the system control unit 50.

[0087] In S11, under the control of the system control unit 50, the imaging unit 22 is driven to acquire imaging data. The acquired imaging data is not for recording, as will be described later, but for detection and display, so an image with a smaller size than the recorded image is acquired. In S11, an image with sufficient resolution for focus detection, subject detection, or live view display is acquired. Here, since the drive operation is for capturing video for live view display, so image capture is performed using a so-called electronic shutter, which accumulates and reads charge for a period of time corresponding to the frame rate for live view display. The live view display performed here allows the photographer to check the shooting range and shooting conditions, and may be, for example, 30 frames per second (shooting interval 33.3 ms) or 60 frames per second (shooting interval 16.6 ms).

[0088] In S12, the system control unit 50 acquires focus detection data obtained from the first and second focus detection pixels included in the focus detection area from the imaging data obtained in S11. The system control unit 50 also adds the output signals of the first and second focus detection pixels to generate an imaging signal, and the image processing unit 24 acquires image data by applying color interpolation processing and the like. In this way, image data and focus detection data can be acquired with a single imaging operation. Note that if the imaging pixels and the first and second focus detection pixels are configured as separate pixels, image data is acquired by performing interpolation processing and the like on the focus detection pixels.

[0089] In S13, the system control unit 50 generates an image for live view display using the image processing unit 24 based on the image data obtained in S12, and displays the image on the display unit 28. Note that the image for live view display is, for example, a reduced image matched to the resolution of the display unit 28, and the image processing unit 24 can also perform reduction processing when generating the image data in S12. In this case, the system control unit 50 displays the image data obtained in S12 on the display unit 28. As described above, during live view display, images are captured and displayed at a predetermined frame rate, allowing the photographer to adjust the composition and exposure conditions during capture through the display unit 28. Also, as described above, in this embodiment, it is possible to detect subjects such as human faces and animals. In S13, a frame or the like indicating the area of ​​the detected subject is also displayed in conjunction with the start of live view display. Note that, as in the main flow S1 of the first embodiment, an acquired image may be displayed instead of a live view image.

[0090] Since steps S14 to S19 are the same as steps S2 to S7 in the first embodiment, the explanation will be omitted.

[0091] If Sw1 is not detected as being on (or is detected as being off) in S20, the system control unit 50 proceeds to S28, where it determines whether the main switch included in the operation unit 70 has been turned off. On the other hand, if Sw1 is detected as being on in S20, the system control unit 50 proceeds to S21, where it sets the focus detection area to be focused on and performs focus detection. Here, the focus detection area is set using the gaze position whose detection began in S18. The gaze position detected in S18 may contain errors due to various factors relative to the position of the subject intended by the photographer.

[0092] In S21, the focus detection area is set using gaze position information that has been processed as described below. At this time, the gaze position and the center position of the focus detection area may be aligned. Furthermore, if there are candidates for the focus detection area determined by other means, such as a subject detection means, the gaze position may be associated with the detected subject area, and the closest subject area may be selected as the focus detection area. After S21, the setting of the focus detection area using the gaze position information and the focus detection process are repeatedly performed each time an image is captured. Here, the focus detection process will be described.

[0093] The defocus amount and direction are calculated for each focus detection area using focus detection data corresponding to the set focus detection area. In this embodiment, the system control unit 50 generates image signals for focus detection, calculates the amount of deviation (phase difference) between the focus detection signals, and calculates the amount of defocus and direction from the calculated amount of deviation.

[0094] Shading correction and filter processing are performed on the first and second focus detection signals obtained from the set focus detection area as image signals for focus detection, reducing the difference in light intensity between the pair of signals and extracting spatial frequency signals for phase difference detection.Next, shift processing is performed to relatively shift the first and second focus detection signals after filter processing in the pupil division direction, and a correlation amount that indicates the degree of signal coincidence is calculated.

[0095] The kth first focus detection signal after filter processing is denoted by A(k), the second focus detection signal is denoted by B(k), and the range of number k corresponding to the focus detection area is denoted by W. Furthermore, if the shift amount by shift processing is denoted by s1 and the shift range of shift amount s1 is denoted by Γ1, the correlation amount COR is calculated by equation (6).

[0096]

number

[0097] By shifting the shift amount s1, the k-th first focus detection signal A(k) and the k-s1-th second focus detection signal B(k-s1) are matched and subtracted to generate a shift subtraction signal. The absolute value of the generated shift subtraction signal is calculated, and the sum of the number k is taken within the range W corresponding to the focus detection area to calculate the correlation amount COR(s1). If necessary, the correlation amount calculated for each row may be added across multiple rows for each shift amount.

[0098] Next, the real-valued shift amount that minimizes the correlation amount is calculated from the correlation amount using sub-pixel calculations, and is set as the image shift amount p1. The calculated image shift amount p1 is then multiplied by a conversion coefficient K1 that corresponds to the image height of the focus detection area, the F-number of the imaging lens (imaging optical system), and the exit pupil distance to determine the detected defocus amount.

[0099] In S22, the system control unit 50 drives the lens based on the defocus amount detected in the selected focus detection area. If the detected defocus amount is smaller than a predetermined value, it is not necessarily necessary to drive the lens.

[0100] In S23, the acquisition and live view display of the detection and display images performed in S1, as well as the focus detection processing performed in S6, are performed. The live view display also includes information on the detected subject area and gaze position, as described above, superimposed on it. The processing performed in S23 may be performed in parallel with the lens drive in S22. The focus detection area may also be changed to correspond to the obtained gaze position in accordance with the continuously updated live view display. After the focus detection processing is completed, the processing proceeds to S24, where the system controller 50 detects the on / off state of the second shutter switch 64 (Sw2), which indicates an instruction to start shooting. The release (shooting trigger) switch, which is one of the operation units 70, can detect two levels of on / off depending on the amount of depression, and the above-mentioned Sw2 corresponds to the second level of on / off of the release (shooting trigger) switch. If the system controller 50 does not detect that Sw2 is on in S24, it returns to S20 and detects the on / off state of Sw1.

[0101] Furthermore, subject detection is performed from the detection image acquired in S1, and information on the detected subject position and range is acquired. In this embodiment, the acquired detected subject information and the gaze position information acquired in S18 are used to change the gaze position information. The acquired detected subject information is compared with the gaze position information, and if the gaze position is included within the detected subject range a first number of times, the stillness determination described above is performed, and the gaze information display on display unit 28 is displayed as still. Note that the determination range does not have to be within the detected subject range, but may be, for example, a range near the detected subject range. Furthermore, instead of displaying the gaze information display as still, it may be hidden.

[0102] In this embodiment, the predetermined number of times N, the first determination number JN1, and the second determination number JN2 can be changed depending on the selected AF mode. AF modes include one-shot AF, which performs the aforementioned AF control only once when Sw1 detection is detected in S20, and servo AF, which continues to perform AF control while Sw1 detection is ongoing. First, when one-shot AF is set, it is considered that it is often used, for example, when the composition is already decided and a still subject is photographed at low speed. Therefore, the first determination number and the second determination number are set to be larger than when servo AF is set, and the first threshold and the second threshold are set to larger values. This allows fine adjustment of the gaze position display relative to the subject, enabling good gaze display.

[0103] On the other hand, when the Servo AF mode is set, it is considered that the mode is often used, for example, when photographing a medium to fast-moving subject that requires framing operations. Therefore, the first determination count and the second determination count are set to fewer values ​​than when the One-Shot AF mode is set, and the first threshold and the second threshold are set to smaller values. This allows for quick action display, making it possible to provide good line-of-sight display even for fast-moving subjects.

[0104] It should be noted that instead of changing the settings according to the AF mode as described above, the user may freely set the first determination count, the second determination count, the first threshold value, and the second threshold value.

[0105] If Sw2 is detected as on in S24, the system control unit 50 proceeds to S25 and determines whether to record images. In this embodiment, image acquisition during continuous shooting is switched between processing for recording images and processing for image capture / display and focus detection. Switching may be alternated, or, for example, image capture / display and focus detection may be performed once every three shots. This allows for highly accurate focus detection without significantly reducing the number of images taken per unit time.

[0106] If it is determined in S25 that image recording is to be performed, the process proceeds to S26, where a photography subroutine is executed. The photography subroutine will be described in detail later. Once the photography subroutine is executed in S26, the process returns to S24, where it is determined whether Sw2 is on, i.e., whether a continuous shooting instruction has been issued.

[0107] If it is determined in S25 that imaging / display and focus detection are to be performed, the process proceeds to S27, where imaging / display and focus detection processing during continuous shooting are performed. The imaging / display and focus detection processing during continuous shooting is the same as that performed in S23. The difference is that the display period, display update rate (interval), and display delay of the images captured in S27 are different from those in S23, depending on the frame rate of continuous shooting and the process of generating recorded images. The system control unit 50, which functions as display means, controls these displays. As described above, the obtained gaze position information is used to set the focus detection area and associate it with the detected subject area. Details will be described later. Once imaging / display and focus detection processing during continuous shooting are performed in S27, the process returns to S24, where it is determined whether Sw2 is turned on, i.e., whether a continuous shooting instruction has been issued. If Sw1 is not turned on (or turned off) in S20 and the main switch is detected to be off in S28, focus detection and shooting operations are terminated. On the other hand, if it is not detected in S28 that the main switch is off, the process returns to S12, and image data and focus detection data are acquired.

[0108] Next, the photographing subroutine executed in S26 of Fig. 12 will be described in detail with reference to the flowchart shown in Fig. 13. The series of operations in this subroutine is also realized mainly by the system control unit 50.

[0109] In S2601, the system control unit 50 executes exposure control processing to determine the shooting conditions (shutter speed, aperture value, and shooting sensitivity). This exposure control processing can be performed by the system control unit 50 based on luminance information of the image data, but any known technology can be used. In this embodiment, details such as the timing of obtaining the image data used in the exposure control processing will be described later. In S2601, the operation of the shutter 101 is controlled based on the determined aperture value and shutter speed. Furthermore, the system control unit 50 causes charge accumulation to occur during the period in which the imaging unit 22 is exposed through the shutter 101.

[0110] When the exposure period ends, in S2602, the system controller 50 performs image readout for high-resolution still image capture, i.e., readout of all pixels. The system controller 50 also performs image readout from either the first focus detection pixels or the second focus detection pixels. The signals readout from the focus detection pixels are used to detect the focus state of the subject during image playback. Therefore, the amount of read data for the recorded image for high-resolution still image capture may be reduced by limiting the area or setting a low resolution for all pixels. If a signal from the first focus detection pixel or the second focus detection pixel is available, the difference with the image for high-resolution still image capture can be calculated to calculate the other focus detection signal. In this embodiment, to prioritize the S / N ratio of the image signal for high-resolution still image capture, the image signal for high-resolution still image capture and one focus detection signal are read out and recorded, and the other focus detection signal is calculated by calculation. Subsequent image-related processing is performed on the image signal for high-resolution still image capture and one focus detection signal.

[0111] In S2603, the system control unit 50 performs defective pixel correction processing on the read image data using the image processing unit 24. In S2604, the system control unit 50 applies image processing such as demosaic (color interpolation), white balance processing, gamma correction (tone correction), color conversion processing, edge enhancement processing, and encoding processing to the image data after the defective pixel correction using the image processing unit 24. In S2605, the system control unit 50 records the image signal for high-resolution still image shooting and one of the focus detection signals in the memory 32 as an image data file.

[0112] In S2606, the system control unit 50 records the characteristic information of the camera body in the memory 32 and in the memory within the system control unit 50, in association with the captured image recorded in S2605. Examples of the characteristic information of the camera body include the following information: Shooting conditions (aperture value, shutter speed, sensitivity, etc.) Information about the image processing applied by the image processing unit 24, Light-receiving sensitivity distribution information of the imaging pixels and focus detection pixels of the imaging unit 22, - Information on vignetting of the shooting light beam within the camera body, Distance information from the mounting surface of the camera body and the lens unit to the imaging unit 22, ·Manufacturing error information

[0113] Note that, since the light sensitivity distribution information of the imaging pixels and focus detection pixels of the imaging unit 22 is information dependent on the on-chip microlenses ML and the photoelectric conversion units PD, information regarding these components may also be recorded. The light sensitivity distribution information is information regarding sensitivity according to a position at a predetermined distance on the optical axis from the imaging element. The light sensitivity distribution information may also be information regarding changes in sensitivity with respect to the angle of incidence of light.

[0114] In S2607, the system control unit 50 records lens unit characteristic information in memory 32 and in a memory within the system control unit 50, corresponding to the captured image recorded in S2605. Examples of lens unit characteristic information include exit pupil information, frame information, focal length and F-number information at the time of capture, aberration information, manufacturing error information, and subject distance information associated with the focus lens position at the time of capture. In S2608, the system control unit 50 records image-related information regarding the captured image in memory 32 and in a memory within the system control unit 50. The image-related information may include information regarding the focus detection operation before capture, subject movement information, information related to the accuracy of the focus detection operation, and the like.

[0115] In S2609, the system control unit 50 displays a preview of the captured image on the display unit 28. This allows the photographer to easily check the captured image. The image used for the preview display in S2609 is intended for easy image confirmation, so the various processes performed in S2603 and S2604 do not need to be performed. If these various processes are not performed, the preview display in S2609 can be performed in parallel with the processes from S2603 onwards, thereby further reducing the time lag from exposure to display.

[0116] When the process of S2609 is completed, the system control unit 50 ends the photographing subroutine of S26 and proceeds to the process of S24 of the main routine. In this embodiment, during continuous shooting, the acquired images are displayed whether the images are recorded in the photographing subroutine of S26 or when the image capturing / display and focus detection process during continuous shooting is performed in S27.

[0117] As described above, by making the display of the gaze position still or moving depending on the amount of change in the gaze position, it is possible to provide an electronic device that improves display quality and convenience. In this embodiment, a method for determining whether to determine whether the gaze position is still or moving depending on the AF mode and subject detection has been described, but this may be changed depending on the shooting mode (moving image, still image, etc.) and the operating method of the shooting device (whether shooting using a viewfinder or a rear LCD screen, etc.).

[0118] In addition, although the embodiments have been described as examples in which the present invention is implemented in a digital camera, the present invention may be applied to any device that performs gaze detection, such as a head-mounted display, a smartphone, or a PC.

[0119] Furthermore, the order of steps performed in the operations described using flowcharts in the above embodiments can be changed as appropriate to achieve the same purpose.

[0120] The present invention can also be configured such that a program that realizes one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium. The present invention can also be realized by one or more processors in a computer of the system or device reading and executing the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions. [Explanation of symbols]

[0121] 50 System control section

Claims

1. an acquisition means for acquiring information corresponding to a position at which an observer is gazing; a control unit that controls a display unit to display an index corresponding to the position of gaze of the viewer acquired by the acquisition unit, The control means When the change in the gaze position of the viewer acquired by the acquisition means is smaller than a first threshold value a first number of times in succession, the indicator is displayed as a still image without moving; A control device characterized by controlling the statically displayed indicator to move and change it to an animated display when the change in the position of the observer's gaze acquired by the acquisition means is greater than a second threshold value a second number of times consecutively, and controlling the statically displayed indicator to move and change it to an animated display when the change in the position of the observer's gaze acquired by the acquisition means is within the second threshold value but the change in the position occurs the second number of times consecutively in a predetermined direction.

2. 2. The control device according to claim 1, wherein the acquisition means acquires the position of the viewer's gaze at a predetermined time interval.

3. A control device as described in claim 1 or 2, characterized in that when the change in the position of gaze of the observer acquired by the acquisition means occurs continuously in a predetermined direction, it is when there are at least two changes within a range corresponding to the predetermined direction.

4. further comprising a subject detection means for detecting the position of a subject; The control device according to any one of claims 1 to 3, characterized in that the control means controls the display of the index to be displayed statically without moving when the position at which the observer is gazing, acquired by the acquisition means, approximately coincides with the position of the subject.

5. further comprising a subject detection means for detecting the position of a subject; 5. The control device according to claim 1, wherein the control means controls the indicator not to be displayed when the position of the subject being observed by the observer, which is acquired by the acquisition means, substantially coincides with the position of the subject.

6. 6. The control device according to claim 1, wherein the control means controls so that at least one of the first number of times, the first threshold value, the second number of times, and the second threshold value is changed in accordance with at least one of a change in horizontal position, a change in vertical position, and a change in position between two points.

7. The optical system further includes a first driving means for driving the focus lens and a second driving means for repeatedly driving the focus lens, 7. The control device according to claim 1, wherein when the second driving means is selected, the first threshold value and the second threshold value are smaller than when the first driving means is selected.

8. 8. The control device according to claim 7, wherein when the second driving means is selected, the first number of times and the second number of times are smaller than when the first driving means is selected.

9. 9. The control device according to claim 1, wherein the control means controls to change at least one of the first number of times and the second number of times in accordance with at least one of the line of sight characteristics of the viewer, the viewing state of the display means, and manual operation by the viewer.

10. an acquisition step of acquiring information corresponding to a position where an observer is gazing; a control step of controlling a display unit to display an index corresponding to the position of gaze of the viewer acquired by the acquisition step, In the control step, When the change in the gaze position of the viewer acquired in the acquiring step is smaller than a first threshold value a first number of times in succession, the indicator is displayed as a still image without being moved; A control method for a control device, characterized in that if the change in the position of the observer's gaze acquired by the acquisition step is greater than a second threshold value a second number of times consecutively, the statically displayed indicator is controlled to move and change to an animated display, and if the change in the position of the observer's gaze acquired by the acquisition step is within the second threshold value but the change in the position occurs the second number of times consecutively in a predetermined direction, the statically displayed indicator is controlled to move and change to an animated display.

11. A program for causing a computer to execute the control method for a control device according to claim 10.

12. A computer-readable storage medium having recorded thereon a program for causing a computer to execute the control method for a control device according to claim 10.

Images