Image processing device, control method for image processing device, and program
The image processing device adjusts zoom positions based on user gaze to re-capture subjects that have deviated from the view, effectively reducing missed photographing opportunities.
Patent Information
- Application Number
- JP2020069707
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-02-05
- Filing Date
- 2020-04-08
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-04-08
AI Technical Summary
Existing imaging apparatuses struggle to re-capture a subject that has deviated from the shooting angle of view, making it difficult to apply techniques for subject re-detection, as the subject may temporarily disappear, leading to lost photographing opportunities.
An image processing device that includes a display means for displaying images captured through an optical system, with a control means that adjusts the zoom position based on the user's gaze position, zooming out to a wider angle and then back to a telephoto angle when instructed, and adjusts the zoom speed based on gaze time.
Enables easy re-capture of subjects that have moved out of the shooting angle of view, reducing the loss of photographing opportunities by ensuring the subject remains within the frame.
Smart Images

Figure 0007580935000001 
Figure 0007580935000002 
Figure 0007580935000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an imaging apparatus having a shooting assist function for recapturing a subject when the subject is out of the shooting angle of view. [Background technology]
[0002] Conventionally, imaging devices with a shooting assist function that makes it easy to recapture a subject that has moved out of a narrow shooting angle of view, such as in telephoto shooting, are known. For example, when a photographer starts shooting assist via an operating member in an attempt to recapture a subject that has moved out of the shooting angle of view, the camera automatically zooms out to the wide-angle side to capture the subject, and when the photographer then cancels the shooting assist, the camera automatically zooms in to the telephoto side (framing assist).
[0003] Patent Document 1 discloses a technique for recognizing a subject intended by a photographer based on information on the gaze position of the photographer looking through a viewfinder. Patent Document 2 discloses a technique for controlling focus based on the variation in gaze position and distance information. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2004-8323 A [Patent Document 2] Patent No. 5828070 Summary of the Invention [Problem to be solved by the invention]
[0005] However, when the subject moves out of the photographing angle of view and needs to be recaptured, the subject temporarily disappears, making it difficult to apply the techniques disclosed in Patent Documents 1 and 2.
[0006] Therefore, an object of the present invention is to provide an image processing device, an imaging device, a control method for an image processing device, and a program that can easily recapture a subject that has moved out of the shooting angle of view, thereby reducing the loss of shooting opportunities. [Means for solving the problem]
[0007] According to one aspect of the present invention, there is provided an image processing apparatus comprising: a display unit for displaying an image captured through an optical system; View The user's gaze position detection do detection and a control means for changing a zoom position of an image displayed on the display means, wherein the control means, when an instruction to start assist control is given, zooms out the image displayed on the display means from a first zoom position to a second zoom position that is wider than the first zoom position, and, when an instruction to stop the assist control is given, zooms out the zoomed-out image to the detection By means detection Based on the gaze position thus determined, the camera zooms in from the second zoom position to a third zoom position that is more telephoto than the second zoom position. and changing a speed of zooming in from the second zoom position to the third zoom position based on a gaze time at the gaze position. do. Other objects and features of the present invention will be described in the following embodiments. Effect of the Invention
[0008] According to the present invention, it is possible to provide an image processing device, an imaging device, a control method for an image processing device, and a program that can easily recapture a subject that has moved out of the shooting angle of view, thereby reducing the loss of shooting opportunities. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a block diagram of an imaging device according to first to third embodiments. [Diagram 2] 1 is a cross-sectional view of an imaging device according to first to third embodiments. [Diagram 3]3 is an explanatory diagram of the principle of a gaze detection method in each embodiment. [Figure 4] 5A to 5C are diagrams illustrating an eyeball image projected onto an eyeball imaging element and an output intensity of the eyeball imaging element in each embodiment. [Diagram 5] 1 is a flowchart of a gaze detection method in each embodiment. [Figure 6] 4 is a flowchart of a control method according to the first embodiment. [Figure 7] 2 is a schematic diagram of a display screen of a display unit in the first embodiment. FIG. [Figure 8] 10 is a flowchart of a control method according to a second embodiment. [Figure 9] 13 is a flowchart of a control method according to a third embodiment. [Figure 10] 13 is a schematic diagram of a display screen of a display unit in the second and third embodiments. FIG. [Figure 11] FIG. 13 is a block diagram of an imaging device according to a fourth embodiment. [Figure 12] 13 is a flowchart of a control method according to a fourth embodiment. [Figure 13] FIG. 13 is a schematic diagram of a display screen of a display unit in a fourth embodiment. [Figure 14] FIG. 13 is a schematic diagram illustrating the appearance of an imaging device according to a fifth embodiment. [Figure 15] FIG. 13 is a block diagram showing the electrical configuration of an imaging apparatus according to a fifth embodiment. [Figure 16] FIG. 13 is an explanatory diagram of a field of view within a finder in the fifth embodiment.
[0010] [Figure 17] FIG. 13 is a diagram showing a change in the display angle of view in the fifth embodiment. [Figure 18] 13 is a flowchart of a shooting assistance process according to the fifth embodiment. [Figure 19] 13 is a flowchart of a control method according to the fifth embodiment. [Figure 20] FIG. 23 is a diagram showing a change in the display angle of view in the sixth embodiment. [Figure 21] 13 is a flowchart of a control method according to the sixth embodiment. [Figure 22] FIG. 13 is a diagram showing a change in the display angle of view in the seventh embodiment. [Figure 23] 13 is a flowchart of a control method according to the seventh embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.
[0012] (First embodiment) First, an imaging device (image processing device) in a first embodiment of the present invention will be described with reference to Figures 1 and 2. Figure 1 is a block diagram of a digital camera (imaging device) 100. Figure 2 is a cross-sectional view of the digital camera 100. Note that in Figures 1 and 2, corresponding parts are denoted by the same numbers.
[0013] In FIG. 1, the imaging optical unit 101 has an optical system (imaging optical system) including a plurality of lens groups including a focus lens and an anti-vibration lens, and an aperture. The imaging optical unit 101 also has an aperture control circuit 211 and a focus adjustment circuit 212. When taking a photograph, the imaging optical unit 101 performs focus adjustment by the focus adjustment circuit 212, exposure adjustment by the aperture control circuit 211, and blur correction, and forms an optical image on the imaging element 102. The imaging element 102 has a photoelectric conversion function for converting an optical image into an electrical signal (analog image signal), and is composed of a CCD sensor, a CMOS sensor, or the like. The A / D conversion unit 103 converts the analog image signal from the imaging element 102 into a digital image signal. The converted image data is input to the image processing unit 104 at the subsequent stage. The bus 116 is a system bus mainly for transmitting control signals for each block from the CPU 114, etc. The bus 117 is a data bus mainly for transferring image data.
[0014] The CPU 114 is composed of a microcomputer or the like that controls the entire image processing device, and issues operation instructions to each functional block to execute various control processes. The CPU 114 also performs calculations required for various control processes. The CPU 114 controls the image sensor 102, the image processing unit 104, the data transfer unit 105, the memory control unit 106, the non-volatile memory control unit 108, the recording media control unit 110, the display control unit 112, and the operation unit (operation means) 115 via the bus 116. The CPU 114 also controls the line of sight detection circuit 120, the photometry circuit 202, the auto focus detection circuit 203, and the light source drive circuit 205 via the bus 116. The microcomputer that constitutes the CPU 114 realizes each process of this embodiment by executing a program recorded in the ROM 109. The CPU 114 also controls the lens and aperture of the imaging optical unit 101 to acquire information such as focal length. The CPU 114 also has an estimation unit 114a and a control unit 114b. The estimation means 114a estimates the gaze position of the photographer on the display unit 113, using the line-of-sight detection circuit 120 and the eye image pickup element 121. The control means 114b changes the focal length of the imaging optical unit 101.
[0015] The data transfer unit 105 is composed of a plurality of DMACs (Direct Memory Access Controllers) that transfer data. The DRAM (memory) 107 is a storage unit that stores data, and has a storage capacity sufficient to store a predetermined number of still images, a predetermined length of moving image, audio data, constants for the operation of the CPU 114, programs, and the like. The memory control unit 106 writes data to and reads data from the DRAM 107 in response to an instruction from the CPU 114 or the data transfer unit 105. The non-volatile memory control unit 108 writes data to and reads data from a ROM (non-volatile memory) 109 in response to an instruction from the CPU 114. The ROM 109 is an electrically erasable and recordable memory, and an EEPROM or the like is used. The ROM 109 stores constants, programs, and the like for the operation of the CPU 114.
[0016] The image processing unit 104 is composed of various image processing units and a buffer memory, and performs appropriate magnification chromatic aberration correction, development processing, noise reduction processing, geometric transformation, enlargement / reduction processing (resizing processing), etc. on the image data. The image processing unit 104 also includes an imaging correction unit that performs appropriate pixel correction, black level correction, shading correction, and scratch correction on the image data converted by the A / D converter 103.
[0017] The recording medium 111 is a recording medium such as an SD card, and is controlled by the recording media control unit 110 to record image data and read out the recorded data. The display unit (display means) 113 is made up of a liquid crystal display or an electronic viewfinder, and is controlled by the display control unit 112 to display various image data (such as images captured via the imaging optical unit 101) and menu screens transferred from the image processing unit 104. In addition, before shooting a still image or during video shooting, the image data input from the A / D conversion unit 103 is processed in real time and displayed.
[0018] The operation unit 115, which is an operation means, includes switches, buttons, a touch panel, etc. that are operated by the photographer, and is used for operations such as turning the power on / off and the shutter on / off. A dedicated button for implementing the shooting assist function is provided, and pressing this button starts the shooting assist function.
[0019] The light source driving circuit 205 drives the light source (illumination light source) 210 (210a, 210b) for illuminating the eyeball of the photographer. The gaze detection circuit 120 detects the gaze direction based on the positional relationship between the eyeball imaged on the eyeball image sensor 121 and the image of the light source 210 due to the corneal reflection.
[0020] In Fig. 2, 1A is a photographing lens (lens device) in a lens-interchangeable camera, and corresponds to the imaging optical unit 101 in Fig. 1. In this embodiment, for convenience, the photographing lens 1A is shown to have two lenses 2101 and 2102 inside, but in reality it is composed of a number of lenses. 1B is a camera housing unit (camera body). The image sensor 102 is disposed on a planned imaging plane of the photographing lens 1A of the digital camera 100. The eyepiece lens 12 is disposed to observe the subject image displayed on the display unit 113.
[0021] Reference numerals 210a and 210b denote light sources for illuminating the photographer's eyeball 14 to detect the gaze direction from the relationship between the pupil and the reflection image of the light source due to the corneal reflection. The light sources 210a and 210b are made of infrared light-emitting diodes and are arranged around the eyepiece 12. The illuminated eyeball image and the image due to the corneal reflection of the light sources 210a and 210b pass through the eyepiece 12, are reflected by the light splitter 15, and are imaged by the light receiving lens 16 on the eyeball image sensor 121, which is a two-dimensional array of photoelectric elements such as CCDs. The light receiving lens 16 positions the pupil of the photographer's eyeball 14 and the eyeball image sensor 121 in a conjugate image-forming relationship. The CPU 114 controls the gaze detection circuit 120, and detects the gaze direction from the positional relationship between the eyeball imaged on the eyeball image sensor 121 and the image due to the corneal reflection of the light sources 210a and 210b, using a predetermined algorithm described later.
[0022] Reference numeral 2111 denotes an aperture provided in the photographing lens 1, 211 denotes an aperture control circuit, 213 denotes a lens drive motor, and 214 denotes a lens drive member consisting of a drive gear and the like. Reference numeral 215 denotes a photocoupler, which detects the rotation of a pulse plate 216 linked to the lens drive member 214 and transmits the rotation (rotation information) to the focus adjustment circuit 212. Based on the rotation information and information on the lens drive amount from the camera side, the focus adjustment circuit 212 drives the lens drive motor 213 by a predetermined amount to move the photographing lens 1A to a focal point position. Reference numeral 217 denotes a mount contact that serves as an interface between the photographing lens 1A and the camera body 1B. The operation unit 115 includes operation members such as a touch panel compatible liquid crystal, a photographing assist button, and a button-type cross key. The operation unit 115 is used for control by a photographing assist operation described later. In this embodiment, the digital camera (imaging device) 100 is a photographing lens (interchangeable lens) 1A that can be attached to and detached from a camera housing (camera body) 1B, but this embodiment can also be applied to an imaging device in which the camera body and the photographing lens are integrated.
[0023] Next, the gaze detection method in this embodiment will be described with reference to Fig. 3 to Fig. 5. Fig. 3 is an explanatory diagram of the principle of the gaze detection method, and an explanatory diagram of an optical system for performing gaze detection. In Fig. 2, light sources 210a and 210b are light sources such as light-emitting diodes that emit infrared light that is insensitive to the observer. The light sources 210a and 210b are disposed approximately symmetrically with respect to the optical axis of the light receiving lens 16, and illuminate the observer's eyeball 14. A part of the illumination light reflected by the eyeball 14 is collected by the light receiving lens 16 onto the eyeball image sensor 121.
[0024] Fig. 4(a) is a schematic diagram of an eyeball image projected onto the eyeball image sensor 121, and Fig. 4(b) is an output intensity diagram of the CCD in the eyeball image sensor 121. Fig. 5 is a flowchart of a gaze detection routine (gaze detection method). The gaze detection routine of Fig. 5 is executed by a gaze point position estimation means (estimation means) including the CPU 114 and the gaze detection circuit 120.
[0025] 5, when the gaze detection routine starts, first, in step S501, the CPU 114 controls the light source drive circuit 205 to drive the light sources 210a and 210b. The light sources 210a and 210b emit infrared light toward the observer's eyeball 14 under the control of the CPU 114. An image of the observer's eyeball illuminated by the infrared light from the light sources 210a and 210b is formed on the eyeball imaging element 121 through the light receiving lens 16 and is photoelectrically converted by the eyeball imaging element 121, so that the eyeball image can be processed as an electrical signal (eyeball image signal). Next, in step S502, the eyeball image signal obtained from the eyeball imaging element 121 is sent to the CPU 114.
[0026] Next, in step S503, the CPU 114 obtains the coordinates of the corneal reflection images Pd, Pe of the light sources 210a, 210b and the point corresponding to the pupil center c shown in Fig. 3 based on the information of the eyeball image signal obtained in step S502. The infrared light emitted from the light sources 210a, 210b illuminates the cornea 142 of the observer's eyeball 14. At this time, the corneal reflection images Pd, Pe formed by a part of the infrared light reflected by the surface of the cornea 142 are collected by the light receiving lens 16 and imaged on the eyeball image sensor 121 (points Pd', Pe' in the figure). Similarly, the light beams from the ends a, b of the pupil 141 are also imaged on the eyeball image sensor 121.
[0027] FIG. 4(a) shows an example of a reflected image obtained from the eye image sensor 121. FIG. 4(b) shows an example of luminance information obtained from the eye image sensor 121 in the region α of the example image in FIG. 4(a). As shown in FIG. 4(a), the horizontal direction is the X axis, and the vertical direction is the Y axis. In this case, the coordinates in the X axis direction (horizontal direction) of the images Pd', Pe' formed by the corneal reflection images of the light sources 210a, 210b are Xd, Xe. Also, the coordinates in the X axis direction of the images a', b' formed by the light beams from the ends (pupil ends a, b) of the pupil 14b are Xa, Xb, respectively. In the example of luminance information in FIG. 4(b), a high level of luminance is obtained at positions Xd, Xe corresponding to the images Pd', Pe' formed by the corneal reflection images of the light sources 210a, 210b. A low level of luminance is obtained in the area between coordinates Xa and Xb, except for positions Xd and Xe, which corresponds to the area of pupil 141. In contrast, intermediate values between the two luminance levels mentioned above are obtained in areas having X coordinate values lower than Xa and areas having X coordinate values higher than Xb, which correspond to the area of iris 143 outside pupil 141. Based on information on the variation in luminance level with respect to the X coordinate position, it is possible to obtain the X coordinates Xd and Xe of images Pd' and Pe' formed by the corneal reflection images of light sources 210a and 210b, and the X coordinates Xa and Xb of the images a' and b' of the pupil edge.
[0028] Furthermore, when the rotation angle θx of the optical axis of eye 14 relative to the optical axis of light receiving lens 16 is small, the coordinate Xc of a point (assumed to be c') corresponding to pupil center c imaged on eye image sensor 121 can be expressed as Xc ≈ (Xa + Xb) / 2. This makes it possible to estimate the X coordinate of c' corresponding to the pupil center imaged on eye image sensor 121 and the corneal reflection images Pd', Pe' of light sources 210a, 210b.
[0029] 5, CPU 114 calculates the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position of eyeball 14 relative to light receiving lens 16, and can be obtained substantially as a function of the distance (Xd-Xe) between corneal reflection images Pd', Pe'.
[0030] Next, in step S505, the biaxial eyeball rotation angle (rotation angles θx, θy) causes the X coordinate of the midpoint of the corneal reflection images Pd, Pe to approximately coincide with the X coordinate of the center of curvature O of the cornea 142. For this reason, if the standard distance from the center of curvature O of the cornea 142 to the center c of the pupil 141 is Oc, then the rotation angle θx of the optical axis of the eyeball 14 in the ZX plane is expressed by the following formula (1).
[0031] θx=β*Oc*SINθx≒{(Xd+Xe) / 2}-Xc … (1) Also, although FIGS. 3 and 4 show an example of calculating the rotation angle θx when the observer's eyeball rotates in a plane perpendicular to the Y axis, the method of calculating the rotation angle θy when the observer's eyeball rotates in a plane perpendicular to the X axis is similar.
[0032] When the rotation angles θx, θy of the optical axis of the observer's eyeball 14 are calculated in step S505, the process proceeds to step S506. In step S506, the CPU 114 reads the correction coefficient data stored in the DRAM (memory) 107. Next, in step S507, the CPU 114 uses the rotation angles θx, θy to determine the position of the observer's line of sight (position of the point of gaze (point of gaze)) on the display unit 113. If the position of the point of gaze is set to coordinates (Hx, Hy) corresponding to the center c of the pupil 141 on the display unit 113, the coordinates Hx, Hy are expressed by the following formulas (2) and (3), respectively.
[0033] Hx = m × (Ax × θx + Bx) … (2) Hy = m × (Ay × θy + By) … (3) In this case, the coefficient m is a constant determined by the configuration of the viewfinder optical system of the digital camera 100, i.e., a conversion coefficient for converting the rotation angles θx, θy into position coordinates corresponding to the center c of the pupil 141 on the display unit 113, and is determined in advance and stored in the DRAM 107. Also, Ax, Bx, Ay, and By are line-of-sight correction coefficients for correcting individual differences in the line of sight of the observer, which are obtained by performing a calibration operation described later and stored in the DRAM 107 before the line-of-sight detection routine starts.
[0034] After calculating the coordinates (Hx, Hy) of the center c of the pupil 141 on the display unit 113 as described above, in step S508, the CPU 114 stores the calculated coordinates in the DRAM 107 and ends the gaze detection routine of Fig. 5. The CPU 114 also measures how long the gaze position remained in a certain area, and stores this in the DRAM 107 as gaze time.
[0035] In this embodiment, a method for acquiring the gaze point coordinates on the display element using the corneal reflection images of the light sources 210a and 210b is shown, but this is not limited to the above method, and a method for acquiring the eyeball rotation angle from a captured eyeball image may also be used.
[0036] Next, a control method in this embodiment will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a flowchart of the control method in this embodiment. Each step in Fig. 6 is mainly executed by CPU 114. Fig. 7 is a schematic diagram of a display screen of display unit 113 in this embodiment.
[0037] When the flow of Fig. 6 is started, the subject 1403 of Fig. 7(a) is displayed on the display unit 113. First, in step S601, the CPU 114 determines whether or not the shooting assist button of the operation unit 115 has been pressed and a shooting assist operation (shooting assist control) has started. If the shooting assist operation has not started, step S601 is repeated. On the other hand, if the shooting assist operation has started, the process proceeds to step S602.
[0038] In step S602, the CPU 114 controls the lens drive motor 213 to perform a zoom-out operation to a predetermined wide-angle angle of view as shown in FIG. 6(b). That is, the control means 114b zooms out the image displayed on the display unit 113 from a first focal length to a second focal length that is wider than the first focal length. The focal length when performing the zoom-out operation can be set to a desired value by the photographer. 1412 in FIG. 6(b) indicates a field of view frame (photographing field of view) at the focal length at the start of the photographing assist control. The field of view frame 1412 at the focal length at the start of the photographing assist control is displayed on the display unit 113 in a smaller size according to the zoom-out operation. 1411 in FIG. 6(b) is a gaze position display frame that includes the gaze point position (gaze position) calculated by the gaze detection routine in FIG. 5. The gaze position display frame 1411 is displayed in the same size as the field of view frame 1412 at the focal length at the start of the photographing assist control. In FIG. 6( b ), the photographer's gaze point is located around the subject 1402 .
[0039] Next, in step S603, the CPU 114 determines whether the photographing assist button has been released and the photographing assist operation has been released (stopped). If the photographing assist operation has not been released, step S603 is repeated. On the other hand, if the photographing assist button has been released, the process proceeds to step S604. In step S604, the CPU 114 controls the lens drive motor 213 to perform a zoom-in operation to the maximum focal length at which the gaze position display frame 1411 is included within the angle of view, as shown in FIG. 6(c). That is, the control unit 114b zooms in the image zoomed out in step S602 (zoomed-out image) from the second focal length to a third focal length that is more telephoto than the second focal length, based on the gaze position estimated by the estimation unit 114a.
[0040] As described above, in this embodiment, the control means 114b zooms out the image displayed on the display unit 113 from the first focal length to the second focal length that is wider than the first focal length. Then, the control means 114b zooms in the zoomed-out image from the second focal length to the third focal length that is more telephoto than the second focal length, based on the gaze position estimated by the estimation means 114a.
[0041] Preferably, the control unit 114b zooms in the zoomed-out image to the third focal length within the range in which the gaze position is displayed on the display unit 113. Also preferably, the display unit 113 displays a frame (gaze position display frame 1411) that includes the gaze position on the display unit 113. More preferably, this frame is the same size as the field of view frame of the first focal length. Also preferably, when the control unit 114b starts the shooting assist control, it zooms out the image displayed on the display unit 113 from the first focal length to the second focal length. And, when the control unit 114b stops the shooting assist operation, it zooms in the zoomed-out image from the second focal length to the third focal length. More preferably, when the control unit 114b receives a release command for the shooting assist control based on the operation of the operation unit 115 by the photographer, it stops the shooting assist control.
[0042] According to this embodiment, when the shooting assist operation is cancelled, a zoom-in operation is performed to a frame including the photographer's gaze point position. This prevents the subject to be photographed from moving out of the angle of view due to the zoom-in operation even when the center point position moves due to an unexpected movement of the subject or when the gaze point position moves to a different subject, and reduces the loss of shooting opportunities.
[0043] Second embodiment Next, a second embodiment of the present invention will be described with reference to Fig. 8 and Fig. 10. Fig. 8 is a flowchart of a control method in this embodiment. Fig. 10 is a schematic diagram of a display screen of display unit 113 in this embodiment. Note that the configuration of the digital camera in this embodiment is similar to that of digital camera 100 described in the first embodiment, and therefore description thereof will be omitted.
[0044] First, in step S801, CPU 114 determines whether or not the photographing assist button of operation unit 115 has been pressed and a photographing assist operation (photographing assist control) has started. If the photographing assist operation has not started, step S801 is repeated. On the other hand, if the photographing assist operation has started, the process proceeds to step S802. CPU 114 also measures the gaze time at the gaze point position.
[0045] In step S802, the CPU 114 controls the lens drive motor 213 to perform a zoom-out operation to a predetermined wide-angle angle of view, as in the first embodiment. Fig. 8(a) shows an example of a screen displayed on the display unit 113 after the zoom-out operation. In Fig. 8(a), 1000 is a field of view frame of the focal length at the start of shooting assist control, 1001 is the optical center position of the lens, and 1002 is a gaze position display frame that includes the gaze point position calculated by the gaze detection routine of Fig. 5.
[0046] Next, in step S803, the CPU 114 judges whether the field of view frame 1000 of the focal length at the start of the photographing assist control and the gaze position display frame 1002 overlap. In this embodiment, when the area of the overlapping region of the field of view frame 1000 and the gaze position display frame 1002 is 70% or more of the area of each frame, it is judged that the frames overlap, but this is not limited to this. FIG. 10B shows a state in which the field of view frame 1000 of the focal length at the start of the photographing assist control and the gaze position display frame 1002 overlap by the photographer panning the digital camera 100 to the right without changing the gaze position (the position of the gaze position display frame 1002). If the field of view frame 1000 and the gaze position display frame 1002 do not overlap in step S803, step S803 is repeated. On the other hand, if it is judged that the field of view frame 1000 and the gaze position display frame 1002 overlap, the process proceeds to step S804.
[0047] In step S804, the CPU 114 cancels (stops) the photographing assist operation. That is, the control unit 114b stops the photographing assist operation when it is determined that the field of view frame 1000 of the first focal length and the frame 1002 including the gaze position overlap. In addition to the condition for canceling the photographing assist operation, that is, that the field of view frame 1000 of the focal length at the start of photographing assist control and the gaze position display frame 1002 overlap, a determination condition may be added that the photographing assist button on the operation unit 115 has been released from being pressed. By adding this determination condition, the timing for canceling the photographing assist operation can be determined to be more consistent with the photographer's intention.
[0048] Next, in step S805, the CPU 114 controls the lens driving motor 213 to perform a zoom-in operation to the focal length at the start of the photographing assist control as shown in FIG. 10(c). The CPU 114 may also control the lens driving motor 213 to increase the driving speed according to the length of the measured gaze time, thereby increasing the zoom-in speed after the photographing assist operation is released. That is, the control unit 114b may change the speed at which the zoom-in is performed from the second focal length to the third focal length based on the gaze time at the gaze position. By controlling the zoom-in speed, when the photographer clearly identifies the subject to be photographed and concentrates on it, the gaze time becomes longer, and it becomes possible to zoom in on the subject to be photographed quickly. On the other hand, when the subject to be photographed is not fixed and the gaze moves to a surrounding subject, etc., and the gaze time becomes shorter, the zoom-in to the focal length at the start of the photographing assist control becomes slower, so that the surrounding subjects can be observed for a longer period of time, and it becomes possible to quickly respond to an unexpected change in the subject, etc.
[0049] According to this embodiment, it is possible to quickly determine the timing to cancel the shooting assist when the field of view frame 1000 of the focal length at the start of shooting assist control overlaps with the gaze position display frame 1002, thereby reducing the loss of shooting opportunities.
[0050] (Third embodiment) Next, a third embodiment of the present invention will be described with reference to Figs. 9 and 10. Fig. 9 is a flowchart of a control method in this embodiment. Fig. 10 is a schematic diagram of a display screen of display unit 113 in this embodiment. Note that the configuration of the digital camera in this embodiment is similar to that of digital camera 100 described in the first embodiment, and therefore a description thereof will be omitted. Also, the flowchart in Fig. 9 includes the same processes as those in the flowchart in Fig. 8 described in the second embodiment, and therefore a duplicate description of the same processes will be omitted.
[0051] Step S901 is the same as step S801 in Fig. 8. After determining in step S901 whether to start the capture assist operation, the process proceeds to step S902. In step S902, the CPU 114 starts measuring the time until the field of view frame 1000 of the focal length at the start of capture assist control overlaps with the gaze position display frame 1002.
[0052] The subsequent steps S903 and S904 are similar to steps S802 and S803, respectively, in Fig. 8. If it is determined in step S904 that the viewing angle frame 1000 of the focal length at the start of the shooting assist control overlaps with the gaze position display frame 1002, the process proceeds to step S905. In step S905, the CPU 114 completes the time measurement started in step S902. The subsequent step S906 is similar to step S804 in Fig. 8.
[0053] Next, in step S907, the CPU 114 controls the lens drive motor 213 to perform a zoom-in operation to the focal length at the start of the photographing assist control, as shown in Fig. 10(c). The CPU 114 also changes the drive speed of the lens drive motor 213 according to the length of the measurement time until the viewing position display frame 1002 overlaps with the field of view frame 1000 of the focal length at the start of the photographing assist control (first focal length), and changes the zoom-in speed after the photographing assist operation is released. That is, if the measurement time is short, the zoom-in speed is increased, and if the measurement time is long, the zoom-in speed is decreased.
[0054] As described above, in this embodiment, the control unit 114b stops the shooting assist control when the gaze time at the gaze position exceeds a predetermined time. Preferably, the control unit 114b changes the speed of zooming in from the second focal length to the third focal length based on the time until the distance between the field of view frame 1000 of the first focal length and the gaze position display frame 1002 becomes smaller than the predetermined distance.
[0055] According to this embodiment, if the photographer clearly identifies the subject to be photographed and quickly points the digital camera 100 at it, it is possible to quickly zoom in on the subject to be photographed. On the other hand, if the photographer does not identify the subject to be photographed and slowly points the digital camera 100 at it, the zoom in to the focal length at the start of the photographing assist control is slower, and the surrounding subjects can be observed for a longer period of time. As a result, it is possible to quickly respond to a sudden change in the subject, and the loss of photographing opportunities can be reduced.
[0056] (Fourth embodiment) Next, an imaging device (image processing device) according to a fourth embodiment of the present invention will be described. The imaging device according to this embodiment has a sub-imaging unit that captures wide-angle images to assist framing, in addition to a main imaging unit for recording. Note that the configuration of the imaging device according to this embodiment and a part of the retained shooting area display process are the same as those of the first embodiment, and only the different parts will be described, and the description of the same parts will be omitted.
[0057] FIG. 11 is a block diagram of a digital camera (imaging device) 500 in this embodiment. The same elements as those in the first embodiment are denoted by the same reference numerals, and detailed description here is omitted. In FIG. 11, the main camera imaging optical unit 501, which mainly performs telephoto shooting, is equipped with a plurality of lens groups including a focus lens and an anti-vibration lens, and an aperture. When shooting, the main camera imaging optical unit 501 performs focus adjustment by the focus adjustment circuit 5212, exposure adjustment by the aperture control circuit 5211, and blur correction, and forms an optical image on the main camera imaging element 502. The main camera imaging element 502 has a photoelectric conversion function that converts an optical image into an electrical signal (analog image signal), and is composed of a CCD sensor, a CMOS sensor, or the like. The main camera A / D conversion unit 503 converts the analog image signal from the main camera imaging element 502 into a digital image signal. The converted image data is input to the image processing unit 104 at the subsequent stage.
[0058] Digital camera 500 also includes sub-camera imaging optical unit 504, sub-camera image sensor 505, and sub-camera A / D conversion unit 506, which have similar functions for the purpose of performing wide-angle photography. The converted image data is input to image processing unit 104 at the subsequent stage. Like main camera imaging optical unit 501, sub-camera imaging optical unit 504 includes focus adjustment circuit 5214 and aperture control circuit 5213. Note that in this embodiment, the elements subsequent to image processing unit 104 are the same as those in the first embodiment described with reference to FIG. 1.
[0059] Next, a control method in this embodiment will be described with reference to Fig. 12 and Fig. 13. Fig. 12 is a flowchart of the control method in this embodiment. Fig. 13 is a schematic diagram of the display screen of display unit 113 in this embodiment.
[0060] First, in step S1201, the CPU 114 determines whether or not the shooting assist button of the operation unit 115 has been pressed and a shooting assist operation (shooting assist control) has started. If the shooting assist operation has not started, step S1201 is repeated. On the other hand, if the shooting assist operation has started, the process proceeds to step S1202.
[0061] In step S1202, CPU 114 switches the display screen of display unit 113 to an image (wide-angle image) captured by a sub-camera capable of capturing wider-angle images than the main camera, as shown in Fig. 13(a). In Fig. 13(a), 1303 indicates the capturing angle of view of the main camera, and 1302 indicates a gaze position display frame including the gaze point position calculated by the gaze detection routine in Fig. 5.
[0062] Next, in step S1203, the CPU 114 judges whether the shooting angle of view 1303 of the main camera and the gaze position display frame 1302 overlap. In this embodiment, when the area of the overlapping region of the shooting angle of view 1303 of the main camera and the gaze position display frame 1302 is 70% or more of the area of each frame, it is judged that the shooting angle of view 1303 of the main camera and the gaze position display frame 1302 overlap, but this is not limited to this. FIG. 13(b) shows a state in which the shooting angle of view 1303 of the main camera and the gaze position display frame 1302 overlap by the photographer panning the digital camera 100 to the right without changing the gaze position (the position of the gaze position display frame 1302). If the shooting angle of view 1303 of the main camera and the gaze position display frame 1302 do not overlap in step S1203, step S1203 is repeated. On the other hand, if it is determined in step S1203 that the main camera's shooting angle of view 1303 overlaps with the gaze position display frame 1302, the process proceeds to step S1204.
[0063] In step S1204, CPU 114 cancels (stops) the shooting assist operation. In addition to the shooting assist operation cancellation condition of whether the shooting angle of view 1303 of the main camera overlaps with the gaze position display frame 1002, a determination condition may be added that the shooting assist button on operation unit 115 has been released from being pressed. By adding this determination condition, the timing to cancel the shooting assist operation can be determined more precisely as intended by the photographer. Next, in step S1205, CPU 114 switches the display screen of display unit 113 to an image captured by the main camera (telephoto angle of view) on the telephoto side of the sub camera, as shown in FIG. 13(c).
[0064] As described above, in this embodiment, the imaging device 500 has a first imaging element (main camera imaging element 502) and a second imaging element (sub camera imaging element 505). The first imaging element photoelectrically converts an optical image formed through a first optical system (main camera imaging optical unit 501) to obtain a first image. The second imaging element photoelectrically converts an optical image formed through a second optical system (sub camera imaging optical unit 504) to obtain a second image with a wider angle than the first image. The control means 114b displays the second image on the display unit 113. Then, the control means 114b determines whether or not the field of view frame (main camera shooting field of view 1303) of the first image displayed superimposed on the second image overlaps with a frame (gazing position display frame 1302) including the gaze position in the second image. When the field of view frame of the first image overlaps with the frame including the gaze position, the control means 114b switches the second image displayed on the display unit 113 to the first image.
[0065] According to this embodiment, it is possible to quickly determine the timing to cancel the shooting assist when the shooting angle of view 1303 of the main camera overlaps with the gaze position display frame 1002, thereby reducing the loss of shooting opportunities.
[0066] Fifth embodiment Next, a fifth embodiment of the present invention will be described with reference to Figs. 14 to 19. Fig. 14 is a schematic external view of a digital camera (imaging device) 100 in this embodiment. Fig. 14(a) shows a front perspective view of the digital camera 100, and Fig. 14(b) shows a rear perspective view. In this embodiment, the digital camera 100 is configured to include a photographing lens (lens device) 1A and a camera housing section (camera body) 1B, as shown in Fig. 14(a). The camera housing section 1B is provided with a release button 5 that constitutes a part of an operation section 115 that receives an image capturing operation from a photographer. As shown in Fig. 14(b), an eyepiece 12 is provided on the rear surface of the digital camera 100, through which the photographer peers into the display section 113. The digital camera 100 is also provided with an operation member α (touch panel compatible liquid crystal 41), an operation member β (lever type operation member 42), and an operation member γ (button type cross key 43) that are used for camera operation, which will be described later, as part of the operation section 115. The operation members α, β, and γ are used, for example, when manually controlling the movement of the estimated gaze frame position, which will be described later. Fig. 1 corresponds to a cross-sectional view of the digital camera 100 cut along the YZ plane formed by the Y axis and the Z axis in Fig. 14(a). Note that in Fig. 1 and Fig. 14, corresponding parts are denoted by the same numbers.
[0067] Fig. 15 is a block diagram showing the electrical configuration of the digital camera 100. In Fig. 15, the same components as those in Fig. 1 are given the same reference numbers. The CPU 114 is connected to the line-of-sight detection circuit 120, the photometry circuit 202, the autofocus detection circuit 203, the signal input circuit 204, the display control unit 112, the light source drive circuit 205, the tracking circuit 207, the recognition circuit 208, and the image processing circuit 209. The CPU 114 also transmits signals to the focus adjustment circuit 212 and the aperture control circuit 211 arranged in the photographic lens 1A via the mount contact 217. The memory unit 4 associated with the CPU 114 has a function of storing image signals from the image sensor 102 and the eye image sensor 121, and a function of storing line-of-sight correction data for correcting individual differences in the line of sight, which will be described later.
[0068] The gaze detection circuit 120 A / D converts the output of the eyeball image formed on the eyeball image sensor 121 (CCD-EYE) and sends this image information to the CPU 114. The CPU 114 extracts each feature point of the eyeball image required for gaze detection according to a predetermined algorithm described later, and further calculates the gaze of the photographer from the position of each feature point. Based on the signal obtained from the image sensor 102, which also serves as a photometry sensor, the photometry circuit 202 amplifies the luminance signal output corresponding to the brightness of the field, performs logarithmic compression and A / D conversion, and sends it to the CPU 114 as field luminance information.
[0069] The auto focus detection circuit 203 A / D converts signal voltages from a plurality of pixels used for phase difference detection, which are included in the CCD in the image sensor 102, and sends the converted signal to the CPU 114. The CPU 114 calculates the distance to the subject corresponding to each focus detection point from the signals from the plurality of pixels. This is a well-known technique known as image plane phase difference AF. In this embodiment, as an example, it is assumed that there are 180 focus detection points on the image plane corresponding to the locations shown in the viewfinder field image in FIG. 16.
[0070] The tracking circuit 207 is a circuit that inputs an image and tracks a subject under the control of the CPU 114, and transmits information on a tracking frame of the image information to the CPU 114. The tracking process is performed by, for example, calculating the similarity between two images using SAD (Sum Of Absolute Difference). The tracking circuit 207 may use a tracking process other than SAD. The recognition circuit 208 is a circuit that recognizes a subject from an input image, and performs, for example, face detection of a person or animal detection. The image processing circuit 209 is composed of various image processing units and a buffer memory, and performs appropriate processing such as magnification chromatic aberration correction, development processing, noise reduction processing, geometric transformation, and resizing such as enlargement and reduction on the input image. In addition, the image processing circuit 209 includes a correction unit and the like that appropriately performs pixel correction, black level correction, shading correction, and scratch correction on the image data.
[0071] A switch SW1 that is turned on by the first stroke of the release button 5 to start the camera's photometry, distance measurement, line of sight detection, etc., and a switch SW2 that is turned on by the second stroke of the release button to start the release operation are connected to the signal input circuit 204. The above signals are input to the signal input circuit 204 and transmitted to the CPU 114. The operation members α (touch panel compatible liquid crystal 41), β (lever type operation member 42), and γ (button type cross key 43) are configured to transmit their operation signals to the CPU 114, and in response to this, movement operation control of the estimated gaze frame position, which will be described later, is performed.
[0072] Fig. 16 is an explanatory diagram of the viewfinder field of view, showing a state in which the display unit 113 is operating. In Fig. 16, 300 denotes a field mask, 400 denotes a focus detection area, and 4001 to 4180 denote 180 ranging point targets displayed on the through image shown on the display unit 113, superimposed at positions corresponding to a plurality of focus detection points on the imaging surface. Among these targets, the target corresponding to the current estimated gaze point position is displayed with a frame appearing, as in the case of estimated gaze point A in the figure. The gaze detection method of this embodiment is as described in the first embodiment with reference to Figs. 3 to 5.
[0073] The calibration operation will be described with reference to Fig. 16. As described above, in the gaze detection routine, the rotation angles θx, θy of the eyeball are obtained from the eyeball image, and the gaze point position is estimated by performing a calculation to convert the pupil center position into a corresponding position on the display unit 113.
[0074] However, due to factors such as individual differences in the shape of the human eyeball, unless the photographer adjusts the values of the gaze correction coefficients Ax, Ay, Bx, and By to appropriate values, a discrepancy will occur between position B where the photographer is actually gazing and the calculated estimated gaze point C, as shown in FIG. 16(b). In the above example, even though the photographer wants to gaze at the person at position B, the camera erroneously estimates that the person is gazing at the background, making it impossible to perform appropriate focus detection and adjustment. Therefore, before capturing an image with the camera, a calibration operation must be performed to obtain the appropriate correction coefficient values for the user and store them in the camera.
[0075] Conventionally, the calibration work is performed by highlighting multiple indices at different positions in the viewfinder field of view before imaging, as shown in Fig. 16(c), and having the observer look at the indices. It is known as a publicly known technique to perform a gaze point detection flow when gazing at each target, and to find the appropriate value of the coefficient from the calculated multiple estimated gaze point coordinates and the position of each target coordinate.
[0076] Next, the shooting assist process will be described with reference to FIGS.
[0077] Fig. 17 is a diagram showing a change in the display angle of view in this embodiment, and shows a case where the display angle of view is automatically zoomed out and zoomed in during shooting (when shooting assistance is performed). Fig. 18 is a flowchart of the shooting assistance process. Each step in Fig. 18 is mainly executed by the CPU 114.
[0078] In FIG. 17(a), 800 is an image captured by the image sensor 102 in the case of electronic zoom, and in the case of optical zoom, a shooting scene outside the angle of view captured by the image sensor 102 is shown. 801 is a display image displayed on the display unit 113. 802 is a subject that the photographer wants to shoot. The dotted frame 803 in FIG. 17(b) indicates a warning display, and the frame of the image displayed on the display unit 113 is displayed, for example, in a blinking red color. FIG. 17(a) shows an example of shooting a play at a sports day, and shows a state in which the shooting subject 802 has moved out of the display unit 113 due to camera shake. FIG. 17(b) shows a state in which the shooting assist processing flow described later detects that the shooting subject 802 has been lost, and the outermost corner of the display is displayed in a blinking manner to inform the photographer in advance that the display angle of view should be changed to the wide-angle side. FIG. 17(c) shows a state in which the display angle of view is widened after the warning display in FIG. 17(b). In Fig. 17(c), the subject 802 is found, but the angle of view is gradually widened until the subject is found. Fig. 17(d) shows the state in which the photographer returns the subject 802 to the center position of the display unit 113, thereby returning the angle of view to that of Fig. 17(a).
[0079] Next, a specific shooting assist process will be described with reference to Fig. 18. Note that in this embodiment, the zoom operation will be described as optical zoom, but is not limited to this. When the zoom operation is electronic zoom, the same operation can be achieved by cropping the image in the image processing circuit 209 and changing the angle of view. This is also true for the first to fourth embodiments described above.
[0080] When the photographing assist process is started, CPU 114 performs a photographing subject loss determination process in step S100. The photographing subject loss determination process will be described later with reference to FIG. 19, and by performing this process, it is determined whether or not the subject has been lost. Next, in step S101, CPU 114 determines whether or not the photographer has lost sight of the subject based on the result of the photographing subject loss determination process. If it is determined that the photographer has lost sight of the subject, the process proceeds to step S102. On the other hand, if it is determined that the photographer has not lost sight of the subject (the photographer has captured the photographing subject), the process ends without doing anything.
[0081] Next, in step S102, the CPU 114 determines whether or not a warning display mode is set when the angle of view is automatically changed by the shooting assist function, which is set in advance in the camera menu. If the warning display mode is set, the process proceeds to step S103. On the other hand, if the warning display mode is not set, the process proceeds to step S105.
[0082] In step S103, CPU 114 displays a warning on display unit 113 for threshold (c) seconds, as shown in FIG. 17(b). Next, in step S104, CPU 114 performs the process of determining whether the subject has been lost sight of again, as in step S100. Next, in step S105, CPU 114 performs the same determination as in step S101, or determines whether the maximum angle of view has been reached. If it is determined that the subject has been lost sight of, or that the maximum angle of view is reached, the process proceeds to step S106. On the other hand, if it is determined that the subject has not been lost sight of (the photographer has captured the subject to be photographed) and that the maximum angle of view is not reached, the process ends without doing anything.
[0083] In step S106, the CPU 114 controls the lens drive motor 213 to zoom out to display a predetermined angle of view, as shown in FIG. 17(c). Next, in step S107, the CPU 114 performs the process of determining whether the subject has been lost sight of again, as in step S100. Next, in step S108, the CPU 114 performs the same determination as in step S101. If it is determined that the subject has been lost sight of, the process returns to step S102. That is, if the subject has been lost sight of, steps S102 to S106 are repeated to gradually widen the angle of view until the subject is found. This is to avoid the need to search for many subjects when the angle of view is widened to the maximum in one go. On the other hand, if it is determined that the subject has not been lost sight of, the process proceeds to step S109.
[0084] In step S109, CPU 114 controls lens drive motor 213 to zoom in to an angle of view that includes the gaze area. Next, in step S110, CPU 114 determines whether the angle of view is the angle before zooming out. If it is determined that the angle of view has returned to the angle before zooming out, the process ends. The display form when the process ends is as shown in FIG. 17(d). On the other hand, if the angle of view is not the angle before zooming out, the process proceeds to step S111. In step S111, CPU 114 determines whether the photographer panned the camera to capture subject 802 in the center of the screen. If it is determined that the camera has been panned, the process returns to step S109. On the other hand, if it is not determined that the camera has been panned, the process proceeds to step S112. The determination in step S111 may be made based on whether subject 802 has been positioned toward the center of the screen.
[0085] In step S112, CPU 114 determines whether the angle of view has remained the same for at least threshold (d) time, or whether the release button 5 has been pressed and a photographing operation has been performed. If the angle of view has remained the same for at least threshold (d) time, or a photographing operation has been performed, the process ends. On the other hand, if the angle of view has not remained the same for at least threshold (d) time, and a photographing operation has not been performed, the process returns to step S111. The display form when the process ends is as shown in FIG. 17(d).
[0086] Next, the photographing subject loss-of-sight determination process in this embodiment will be described with reference to Fig. 19. Fig. 19 is a flowchart of the control method (photographing subject loss-of-sight determination process) in this embodiment.
[0087] First, in step S200, CPU 114 resets and then starts a timer for measuring the number of times the gaze point moves within a threshold (a) time. Next, in step S201, CPU 114 determines whether or not there is a gaze point detected in the gaze detection routine. If there is a gaze point, the process proceeds to step S202. On the other hand, if there is no gaze point, the process proceeds to step S203.
[0088] In step S202, the CPU 114 determines whether or not the gaze point determined in step S201 has moved within a threshold (e) seconds. If the gaze point has moved, the process proceeds to step S203. On the other hand, if the gaze point has not moved, the process proceeds to step S206.
[0089] In step S203, CPU 114 determines whether or not it has been within a threshold (a) seconds since the first gaze point was detected. If it has been within a threshold (a) seconds since the first gaze point was detected, the process proceeds to step S204. On the other hand, if it has not been within a threshold (a) seconds since the first gaze point was detected, the process returns to step S200.
[0090] In step S204, CPU 114 determines whether the gaze point has moved threshold (b) times or more. If the gaze point has moved threshold (b) times or more, the process proceeds to step S205. On the other hand, if the gaze point has not moved threshold (b) times or more, the process returns to step S201. In step S205, CPU 114 determines that the subject has been lost, and ends the process. In step S206, CPU 114 determines that a subject is present, and ends the process.
[0091] As described above, in the process of determining whether or not a subject has been lost from sight in the present embodiment, the loss of sight determination is performed based on whether or not the gaze point has moved within a certain area within a certain specified time. According to the present embodiment, it is possible to detect whether or not a subject has been lost from sight, thereby reducing the possibility of missing the timing to capture a photograph.
[0092] Sixth embodiment Next, a sixth embodiment of the present invention will be described with reference to Fig. 20 and Fig. 21. This embodiment differs from the fifth embodiment in that a gaze point and a result of recognizing a subject by the recognition circuit 208 are used.
[0093] FIG. 20 is a diagram showing a change in the display angle of view in this embodiment, and shows the same scene as FIG. 17. A face frame 1000 in FIG. 20(a) shows a face frame for the face of the subject recognized by the recognition circuit 208. FIG. 20(a) also shows that a face frame is displayed on the subject of the display image 801, although no number is attached. Note that in this embodiment, a face frame is described, but a subject frame detected by the recognition circuit 208 may be displayed. A face frame 1002 in FIG. 20(b) shows a face frame that the photographer has already gazed upon with a dotted line, and a face frame 1001 is a face frame that the photographer is not gazing upon. Note that the gazed upon face frame 1002 is shown with a dotted line, but any display form may be used as long as the photographer can recognize the determination of the gazed upon frame and the ungained frame, and for example, the gazed upon face frame may be erased. FIG. 20(c) is a diagram when the display angle of view is widened, similar to FIG. 17(c).
[0094] Next, the photographing subject loss-of-sight determination process in this embodiment will be described with reference to Fig. 21. Fig. 21 is a flowchart of the control method (photographing subject loss-of-sight determination process) in this embodiment.
[0095] First, in step S300, CPU 114 displays a face or subject frame. Note that if a face or subject frame is displayed before the start of the shooting assist process, step S300 does not need to be performed. Next, in step S301, CPU 114 sets the face or subject frame near the gaze point detected in the gaze detection routine as the determination target in step S302. Next, in step S302, CPU 114 determines whether or not the face or subject frame is being gazed at. Note that a face frame within a threshold (f) pixels from the center of the gazed frame is likely to be seen at the same time, so it is also possible to consider it as being gazed at. If the face or subject frame is being gazed at, the process proceeds to step S303. On the other hand, if the face or subject frame is not being gazed at, the process proceeds to step S304.
[0096] In step S303, CPU 114 determines whether the gaze point has moved within a threshold (e) seconds. The threshold (e) may be changed according to the size of the face or subject frame. If the gaze point has moved within the threshold (e) seconds, the process proceeds to step S304. On the other hand, if the gaze point has not moved within the threshold (e) seconds, the process proceeds to step S307.
[0097] In step S304, CPU 114 changes the display form of the face or subject frame before the gaze point moves, that is, the frame targeted in step S301, to face frame 1002 in FIG. 20(b). Next, in step S305, CPU 114 determines whether or not all of the faces or subject frames on the image displayed on display unit 113 have been confirmed. The number of times to gaze at one face or subject frame may be changed depending on the size of the face or subject frame. This is to prevent the possibility that the photographer may have seen the subject once if it is small, but may have missed it even though it was originally a photographed subject. If all of the faces or subject frames on the display screen have been confirmed, proceed to step S306. On the other hand, if there is a face or subject frame that has not been confirmed, return to step S301. In step S306, CPU 114 determines that the subject has been lost, and ends the process. In step S307, CPU 114 determines that there is a photographed subject, and ends the process.
[0098] As described above, in the photographing subject loss judgment process of this embodiment, the subject loss judgment is performed depending on whether or not the photographer is gazing at the subject frame. According to this embodiment, it is possible to detect the loss of the judged subject and reduce the possibility of missing the timing of photographing. In addition, by displaying the face frame and making it easy to judge whether or not the photographer is gazing at the face frame, it is possible to prevent the photographer from checking the same subject multiple times.
[0099] Seventh embodiment Next, a seventh embodiment of the present invention will be described with reference to Figures 22 and 23. This embodiment differs from the sixth embodiment in that the subject is a moving object.
[0100] FIG. 22 is a diagram showing a change in the display angle of view in this embodiment. FIG. 22(a) shows the display screen of the display unit 113 when a certain time is cut out when a racing car (subject) moves from the upper right to the lower left. Frames are displayed at the four corners of each subject as the subject frame 1300 recognized by the recognition circuit 208. FIG. 22(b) shows a state in which the photographed subject 802 is determined not to exist in FIG. 22(a) and the angle of view of the image displayed on the display unit 113 is widened, and the photographed subject 802 comes into the angle of view as a result of the widening of the angle of view. In addition, the subject frame 1302 shows the face frame that the photographer has already gazed upon with a dotted line. Note that the gazed upon subject frame 1302 is shown with a dotted line, but any display form may be used as long as the photographer can recognize the determination between the gazed upon frame and the ungaze upon frame, and for example, the gazed upon frame may be erased. FIG. 22(c), like FIG. 17(d), shows a state where the camera is zoomed in to an angle of view in which the subject 802 is completely captured or an angle of view before zooming out, in this case the angle of view of FIG. 22(a).
[0101] Next, the photographing subject loss-of-sight determination process in this embodiment will be described with reference to Fig. 23. Fig. 23 is a flowchart of the control method (photographing subject loss-of-sight determination process) in this embodiment.
[0102] First, in step S400, CPU 114 displays a subject frame. Note that, if a subject frame is displayed beforehand before the start of the shooting assist process, step S400 does not need to be performed. Next, in step S401, CPU 114 sets a subject near the gaze point detected in the gaze detection routine as a determination target in step S402. Next, in step S402, CPU 114 calculates a motion vector (motion information) of the subject from the previous frame and the current frame by a block matching method or the like. Note that, the motion vector calculation may use a different method. Next, in step S403, CPU 114 calculates a motion vector of the subject from the previous frame and the current frame of the image used in the gaze detection by the same method as step S402. Note that, the motion vector calculation may use a method different from that of step S402.
[0103] Next, in step S404, CPU 114 determines whether the motion vector of the object calculated in step S402 is the same as the motion vector of the gaze point calculated in step S403. If the motion vector of the object and the motion vector of the gaze point are the same, the process proceeds to step S405. On the other hand, if the motion vector of the object and the motion vector of the gaze point are different, the process proceeds to step S406.
[0104] In step S405, CPU 114 determines that the moving subject is being gazed at, and proceeds to step S407. In step S406, CPU 114 determines that the moving subject is not being gazed at, and proceeds to step S408. In step S407, CPU 114 determines whether the gaze point has moved within threshold (e) seconds. Note that the threshold (e) may be changed depending on the size of the subject frame. If the gaze point has moved within threshold (e) seconds, proceeds to step S408. On the other hand, if the gaze point has not moved within threshold (e) seconds, proceeds to step S411.
[0105] In step S408, CPU 114 changes the display form of the subject frame before the gaze point is moved, that is, the subject frame targeted in step S401, to subject frame 1302 in FIG. 22(b). Next, in step S409, CPU 114 determines whether or not all of the subject frames on the image displayed on display unit 113 have been confirmed. The number of times to gaze at one subject frame may be changed depending on the size of the subject frame. This is to prevent the possibility that the photographer may see the subject once if it is small, but may miss the subject even though it was originally photographed. If all of the subject frames on the display screen have been confirmed, proceed to step S410. On the other hand, if there is a subject frame that has not yet been confirmed, return to step S401. In step S410, CPU 114 determines that the subject has been lost, and ends the process. In step S411, CPU 114 determines that the photographed subject exists, and ends the process.
[0106] As described above, in the photographing subject loss judgment process, the motion vector of the subject frame and the motion vector of the gaze point are used to judge whether or not the subject (moving object) is being gazed at, and the subject loss judgment is made according to the movement of the gaze point. According to this embodiment, it is possible to detect the loss of the judged subject and reduce the possibility of missing the photographing timing.
[0107] As described above, in the fifth to seventh embodiments, the control means (CPU 114) changes the angle of view when it is determined that the gaze position has changed. Preferably, the control means changes the angle of view when it is determined that the gaze position has changed a threshold number of times or more within a threshold time. Also preferably, the control means changes the angle of view by cropping the image or by driving the optical system to change the imaging position.
[0108] In the fifth to seventh embodiments, when the control means determines that the gaze position has been changed, the control means displays a warning on the display means (display unit 113). Preferably, when the control means determines that the gaze position has been changed a threshold number of times or more within a threshold time, the control means displays a warning.
[0109] Preferably, the image processing device has a recognition means (recognition circuit 208) for recognizing an object from an image. The control means changes the angle of view when it is determined that the gaze position is the position of the object recognized by the recognition means and that the gaze position has been changed a threshold number of times or more within a threshold time. Also preferably, the control means displays a warning when it is determined that the gaze position is the position of the object recognized by the recognition means and that the gaze position has been changed a threshold number of times or more within a threshold time. Also preferably, the control means displays object recognition information on the display means for each of the first object and the second object recognized by the recognition means. Then, the control means changes the display method of the object recognition information when the gaze position is each of the positions of the first object and the second object recognized by the recognition means. More preferably, the control means changes the display method of the object recognition information for the first object when it is determined that the gaze time of the first object is equal to or less than the threshold and that the gaze position has moved from the first object to the second object. Preferably, the control means determines whether or not the gaze position is the position of the subject recognized by the recognition means based on the movement information of the subject recognized by the recognition means and the movement information of the gaze position. That is, when the degree of coincidence between the movement information of the subject and the movement information of the gaze position is equal to or greater than a threshold, the control means determines that the gaze position is the position of the subject recognized by the recognition means.
[0110] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.
[0111] The image processing device of each embodiment uses information on the gaze position as a condition for determining whether to cancel the shooting assist function, and can perform zoom-in control to a focal length at which the gaze position is included within the angle of view after the shooting assist function is cancelled. Therefore, according to each embodiment, it is possible to provide an image processing device, an imaging device, a control method, and a program that can easily recapture a subject that has fallen out of the shooting angle of view, thereby reducing the loss of shooting opportunities.
[0112] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]
[0113] 100 Imaging device (image processing device) 113 Display section (display means) 114a Estimation means 114b Control means
Claims
1. A display means for displaying an image captured via an optical system; A detection means for detecting a gaze position of a user looking at the display means; a control means for changing a zoom position of an image displayed on the display means, The control means When a command to start assist control is received, the image displayed on the display means is zoomed out from a first zoom position to a second zoom position that is wider than the first zoom position; When a command to stop the assist control is received, the zoom-out image is zoomed in from the second zoom position to a third zoom position that is more telephoto than the second zoom position, based on the gaze position detected by the detection means; 2. An image processing device comprising: an image processing apparatus which changes a speed of zooming in from the second zoom position to the third zoom position based on a gaze time at the gaze position.
2. The image processing apparatus according to claim 1 , wherein the control means zooms in the zoomed-out image to the third zoom position within a range in which the gaze position is displayed on the display means.
3. The control means displays a frame including the gaze position on the display means, 3. The image processing device according to claim 1, wherein the frame has a size corresponding to an angle of view corresponding to the first zoom position.
4. 4. The image processing device according to claim 1, wherein the control means determines that an instruction to stop the assist control has been issued when it determines that the field of view frame of the first zoom position overlaps with a frame containing the gaze position.
5. 5. The image processing apparatus according to claim 1, wherein the control unit determines that an instruction to stop the assist control has been issued based on an operation of an operating unit by a user.
6. 6. The image processing device according to claim 1, wherein the control means changes a speed of zooming in from the second zoom position to the third zoom position based on a time required for a distance between a field of view frame corresponding to the first zoom position and a frame containing the gaze position to become smaller than a predetermined distance.
7. A display means for displaying an image captured via an optical system; A control method for an image processing device having a detection means for detecting a gaze position of a user looking at the display means, and a control means for changing a zoom position of an image displayed on the display means, receiving an instruction to start assist control, zooming out the image displayed on the display means from a first zoom position to a second zoom position that is wider than the first zoom position; receiving an instruction to stop the assist control, zooming in a zoomed-out image from the second zoom position to a third zoom position that is more telephoto than the second zoom position, based on the gaze position detected by the detection means; and changing a speed of zooming in from the second zoom position to the third zoom position based on a gaze time at the gaze position.
8. A program for causing a computer to execute the control method according to claim 7.
Citation Information
Patent Citations
Emergency gas shut down apparatus
JP1983028070A
Electronic zoom system
JP1993130477A
Camera
JP1995151950A
Optical device with visual axis function
JP2004008323A
Imaging apparatus and control method thereof
JP2014103489A
