Electronic device, control method of electronic device, and computer readable medium

Abstract

The disclosure provides an electronic device, a control method of an electronic device, and a computer readable medium. The electronic device according to the disclosure includes a reading unit configured to read an image that is a photographed image and gaze information associated with the image from a recording medium, a display control unit configured to control a display unit, and a control unit configured to control such that the image read by the reading unit and displayed on the display unit by the display control unit is enlarged or reduced with a position based on the gaze information read by the reading unit as a center when the image is enlarged or reduced.

Description

Technical Field

[0001] This invention relates to electronic devices, methods for controlling electronic devices, and computer-readable media. Background Technology

[0002] In recent years, increased photographic speeds have enabled the acquisition of large numbers of images in a short period. However, when processing a large number of images, image selection becomes difficult. Therefore, the demand for efficient image verification is increasing. For example, Japanese Patent Application Publication 2009-177345 discloses a technique for magnifying an image centered on a focal point.

[0003] According to the technology disclosed in Japanese Patent Application Publication No. 2009-177345, the focus area can be easily identified. However, since the area of ​​interest to the photographer is not always in focus, it is not easy to identify the area of ​​interest. For example, after displaying the focus area in a magnified manner, an operation is required to change the display area, which takes time and effort. Summary of the Invention

[0004] This invention provides a technique that allows photographers to easily identify areas of interest.

[0005] The present invention provides, in a first aspect, an electronic device comprising: a reading unit configured to read from a recording medium an image as a captured image and gaze information associated with the image; a display control unit configured to control a display unit; and a control unit configured to, when an image read by the reading unit and displayed on the display unit by the display control unit is magnified or reduced, such that the image is magnified or reduced centered on the position of the gaze information read by the reading unit.

[0006] In a second aspect, the present invention provides a method for controlling an electronic device, comprising: a reading step for reading an image as a captured image and gaze information associated with the image from a recording medium; a display control step for controlling a display unit; and a control step for controlling, when the image read in the reading step and displayed on the display unit in the display control step is magnified or reduced, such that the image is magnified or reduced with the position centered on the gaze information read in the reading step.

[0007] In a third aspect, the present invention provides a computer-readable medium storing a program that enables a computer to be used as a unit of an electronic device.

[0008] Other features of the invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0009] Figure 1A and Figure 1B This is an external view of the camera according to the first embodiment;

[0010] Figure 2 This is a cross-sectional view of the camera according to the first embodiment;

[0011] Figure 3 This is a block diagram of a camera according to the first embodiment;

[0012] Figure 4 This is a diagram showing the view within the viewfinder according to the first embodiment;

[0013] Figure 5 This is a diagram illustrating the principle of the field-of-view detection method according to the first embodiment;

[0014] Figure 6A This is a diagram showing an eye image according to the first embodiment;

[0015] Figure 6B This is a diagram showing the brightness distribution of an eye image according to the first embodiment;

[0016] Figure 7 This is a flowchart of the gaze detection operation according to the first embodiment;

[0017] Figure 8 This is a flowchart of the recording process according to the first embodiment;

[0018] Figure 9 This is a flowchart of the reproduction process (enlargement / reduction process) according to the first embodiment;

[0019] Figures 10A to 10C This is a flowchart of the reference point setting process according to the first embodiment;

[0020] Figure 11 This is a flowchart of the recording process according to the second embodiment;

[0021] Figure 12A and Figure 12B This is a flowchart of the reference point setting process according to the second embodiment;

[0022] Figure 13 This is a flowchart of a modified example of the recording process according to the second embodiment;

[0023] Figure 14A and Figure 14B This is a flowchart of the reference point setting process according to the second embodiment;

[0024] Figure 15 This is a flowchart of the reproduction process (enlargement / reduction process) according to the third embodiment;

[0025] Figure 16A and Figure 16B This is a flowchart of the reference point setting process according to the third embodiment;

[0026] Figure 17 This is a flowchart of the recording process according to the fourth embodiment;

[0027] Figure 18A and Figure 18B This is a flowchart of the process for setting the judgment range according to the fourth embodiment;

[0028] Figure 19 This is a flowchart of the rating process according to the fourth embodiment;

[0029] Figure 20 This is a flowchart of the recording process according to the fifth embodiment;

[0030] Figures 21A to 21C This is a flowchart of the process for setting the judgment range according to the fifth embodiment;

[0031] Figure 22 This is a flowchart of a modified example of the recording process according to the fifth embodiment; and

[0032] Figure 23A and Figure 23B This is a flowchart of the process for setting the judgment range according to the fifth embodiment. Detailed Implementation

[0033] First Embodiment

[0034] In the following description, a first embodiment of the invention will be described with reference to the accompanying drawings.

[0035] Configuration description

[0036] Figure 1A and Figure 1B The appearance of the camera 1 (digital still camera, lens interchangeable camera) according to the first embodiment is shown. Figure 1A It is a front stereoscopic view. Figure 1B It is a rear stereoscopic image. For example... Figure 1A As shown, the camera 1 includes a photographic lens unit 1A and a camera housing 1B. A release button 5 is arranged in the camera housing 1B, which is an operating component for receiving photographic operations from the user (photographer). Figure 1BAs shown, an eyepiece 12 (eyepiece optical system) is arranged in the rear surface of the camera housing 1B, through which the user observes the display device 10 (display panel) included in the camera housing 1B (described later). Note that the eyepiece optical system may include multiple lenses. Operating members 41 to 43 for receiving various operations from the user are also arranged in the rear surface of the camera housing 1B. For example, operating member 41 is a touch panel that receives touch operations, operating member 42 is an operating lever that can be pressed in various directions, and operating member 43 is a four-way key that can be pressed in each of four directions. Operating member 41 (touch panel) includes a display panel such as a liquid crystal panel and has the function of displaying images on the display panel.

[0037] Figure 2 It is along the way by Figure 1A The diagram shows a cross-sectional view of camera 1 cut by the YZ plane formed by the Y-axis and Z-axis, and also shows the approximate internal configuration of camera 1.

[0038] The camera lens unit 1A includes two lenses 101 and 102, an aperture 111, an aperture drive unit 112, a lens drive motor 113, a lens drive member 114, an optocoupler 115, a pulse plate 116, a mounting contact 117, and a focus adjustment circuit 118. The lens drive member 114 is composed of drive gears, etc. The optocoupler 115 detects the rotation of the pulse plate 116, which is linked to the lens drive member 114, and transmits this rotation to the focus adjustment circuit 118. The focus adjustment circuit 118 drives the lens drive motor 113 and moves the lens 101 to change the focus position based on information from the optocoupler 115 and information from the camera housing 1B (information related to the amount of lens drive). The mounting contact 117 is the interface between the camera lens unit 1A and the camera housing 1B. Note that for simplicity, only two lenses 101 and 102 are shown, but the camera lens unit 1A actually includes two or more lenses.

[0039] The camera housing 1B includes an image sensor 2, a CPU 3, a memory unit 4, a display device 10, and a display device drive circuit 11. The image sensor 2 is arranged on the surface of the camera lens unit 1A where an image is to be formed. The CPU 3 is the central processing unit of the microcomputer and controls the camera 1 as a whole. The memory unit 4 stores images captured by the image sensor 2. The display device 10 is composed of a liquid crystal or the like and displays the captured image (subject image) on the screen of the display device 10. The display device drive circuit 11 drives the display device 10. This allows the user to view the screen of the display device 10 through the eyepiece 12.

[0040] The camera housing 1B also includes light sources 13a and 13b, a beam splitter 15, a light-receiving lens 16, and an eye-mounted imaging element 17. Light sources 13a and 13b are conventionally used in single-lens reflex cameras to detect the direction of gaze (line of sight) based on the relationship between the reflected image (corneal reflection image) generated by light reflection from the cornea and the pupil, and to illuminate the user's eyeball 14. Specifically, light sources 13a and 13b are infrared light-emitting diodes or the like that that emit imperceptible infrared light to the user, and are arranged around the eyepiece 12. The optical image of the illuminated eyeball 14 (eye image; the image formed by light emitted from light sources 13a and 13b and reflected by the eyeball 14) passes through the eyepiece 12 and is reflected by the beam splitter 15. The eye image is then formed by the light-receiving lens 16 on the eye-mounted imaging element 17, which has rows of photoelectric elements such as a CCD arranged in two dimensions. The light-receiving lens 16 positions the pupil of the eyeball 14 and the eye-mounted imaging element 17 in a conjugate imaging relationship. According to a predetermined algorithm described later, the gaze direction (gaze position; viewpoint on the screen of the display device 10) of the eyeball 14 is detected from the position of the corneal reflection image formed in the eye image formed on the eye-mounted imaging element 17.

[0041] Figure 3 This is a block diagram showing the electrical configuration inside the camera 1. The CPU 3 is connected to the gaze detection circuit 201, metering circuit 202, autofocus detection circuit 203, signal input circuit 204, display device drive circuit 11, and light source drive circuit 205, etc. Furthermore, the CPU 3 sends signals via mounting contacts 117 to the focus adjustment circuit 118 arranged within the camera lens unit 1A and the aperture control circuit 206 included in the aperture drive unit 112 within the camera lens unit 1A. The memory unit 4 attached to the CPU 3 has the function of storing imaging signals from the imaging element 2 and the eye-mounted imaging element 17. The CPU 3 converts the imaging signals stored in the memory unit 4 into photographic images and transmits the photographic images to the recording medium 5. The accelerometer 6 detects the magnitude of the acceleration applied to the camera housing 1B and sends the result to the CPU 3. The accelerometer 119 detects the magnitude of the acceleration applied to the camera lens unit 1A and sends the result to the CPU 3.

[0042] The gaze detection circuit 201 performs A / D conversion on the output of the eye-image sensor 17 (an eye image obtained by photographing the eye) when the eye image is formed on the eye-image sensor 17 (CCD-EYE), and sends the result to the CPU 3. The CPU 3 extracts the feature points required for gaze detection according to a predetermined algorithm described later, and calculates the user's gaze (viewpoint on the screen of the display device 10) based on the position of the feature points.

[0043] The light metering circuit 202 amplifies, logarithmically compresses, or performs A / D conversion on the signal (specifically, the brightness signal corresponding to the brightness of the field of view) obtained from the camera element 2, which is also used as a light metering sensor, and sends the result as field of view information to the CPU 3.

[0044] The automatic focus detection circuit 203 performs A / D conversion on the signal voltages from multiple detection elements (multiple pixels) included in the CCD of the imaging element 2 for phase difference detection, and sends the resulting signal voltages to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point based on the signals from the multiple detection elements. This is a technique known as imaging plane phase difference AF. In the first embodiment, as an example, it is assumed that the focus detection point exists at the point where the subject is located. Figure 4 The 180 spots shown in the viewfinder image (screen of display device 10) correspond to the 180 spots on the camera surface.

[0045] Signal input circuit 204 is connected to switch SW1 and switch SW2. Switch SW1 is turned on by releasing the first stroke of button 5 and is used to start metering, ranging, or gaze detection operations of camera 1. Switch SW2 is turned on by releasing the second stroke of button 5 and is used to start the photography operation. The ON signals from switches SW1 and SW2 are input to signal input circuit 204 and sent to CPU 3.

[0046] The light source driving circuit 205 drives light sources 13a and 13b.

[0047] The image processing circuit 207 applies predetermined image processing to image data to generate signals or image data, or to acquire and / or generate various types of information. The image processing circuit 207 may be, for example, a dedicated hardware circuit (such as an ASIC designed to implement a specific function), or it may be configured to implement a specific function when a processor such as a DSP performs software processing.

[0048] Here, the image processing applied by the image processing circuit 207 includes preprocessing, color interpolation, correction, detection, and data processing. Preprocessing includes signal amplification, reference level adjustment, and defective pixel correction. Color interpolation is a process used to interpolate the values ​​of color components not included in the image data; it is also called demosaicing. Correction includes white balance adjustment, processing for correcting image brightness, processing for correcting optical aberrations of the camera lens unit 1A, and processing for color correction. Detection includes processing for detecting and tracking feature regions (e.g., facial regions, human body regions, or object regions) or processing for recognizing people. Data processing includes scaling, encoding and decoding, and head information generation. Note that the above processing exemplifies image processing that can be performed by the image processing circuit 207, but is not limited to image processing performed by the image processing circuit 207.

[0049] Figure 4 This is a diagram showing the viewfinder's field of view and the state of operation of the display device 10 (the state of displaying an image). For example... Figure 4 As shown, the viewfinder's field of view includes a focus detection area 400, 180 ranging point indicators 401, and a viewfinder mask 402. Each of the 180 ranging point indicators 401 is superimposed on the through-view image (live-view image) displayed on the display device 10, so that it is displayed at the position corresponding to the focus detection point on the imaging surface. Furthermore, the ranging point indicator 401 corresponding to the current viewpoint A (estimated position) among the 180 ranging point indicators 401 is highlighted by a frame, etc.

[0050] Description of gaze detection operation

[0051] Will use Figure 5 , Figure 6A , Figure 6B and Figure 7 To describe the gaze detection operation. Figure 5 This diagram illustrates the principle of gaze detection methods and is a schematic diagram of the optical system used for gaze detection. Figure 5 As shown, light sources 13a and 13b are arranged substantially symmetrically with respect to the optical axis of the light-receiving lens 16 and illuminate the user's eyeball 14. A portion of the light emitted from light sources 13a and 13b and reflected by the eyeball 14 is converged onto the eye-mounted imaging element 17 by the light-receiving lens 16. Figure 6A This is a schematic diagram of an eye image captured by the eye-mounted camera element 17 (an image of the eyeball projected onto the eye-mounted camera element 17). Figure 6B This is a graph showing the output intensity of the CCD of the eye-mounted imaging element 17. Figure 7 A schematic flowchart of the gaze detection operation is shown.

[0052] When the gaze detection operation begins, Figure 7 In step S701, light sources 13a and 13b emit infrared light toward the user's eyeball 14. The image of the user's eyeball illuminated by the infrared light is formed on the eye-mounted imaging element 17 via the light-receiving lens 16, and is photoelectrically converted by the eye-mounted imaging element 17. Thus, a processable eye image electrical signal is obtained.

[0053] In step S702, the gaze detection circuit 201 sends the eye image (eye image signal; eye image electrical signal) obtained from the eye camera element 17 to the CPU 3.

[0054] In step S703, CPU 3 calculates the coordinates of the points corresponding to the corneal reflection images Pd and Pe of light sources 13a and 13b and the pupil center c based on the eye image obtained in step S702.

[0055] Infrared light emitted from light sources 13a and 13b illuminates the cornea 142 of the user's eyeball 14. At this time, corneal reflection images Pd and Pe, formed by a portion of the infrared light reflected from the surface of the cornea 142, are converged by the light-receiving lens 16 and formed on the eye-viewing image element 17 to be transformed into corneal reflection images Pd' and Pe' in the eye image. Similarly, images of the light beams from the ends a and b of the pupil 141 are also formed on the eye-viewing image element 17 to be transformed into pupil end images a' and b' in the eye image.

[0056] Figure 6B Show Figure 6A Brightness information (brightness distribution) in region α of the eye image. Figure 6B In this example, with the horizontal direction of the eye image defined as the X-axis and its vertical direction defined as the Y-axis, the brightness distribution along the X-axis is shown. In the first embodiment, the coordinates of the corneal reflection images Pd' and Pe' in the X-axis direction (horizontal direction) are represented as Xd and Xe, and the coordinates of the pupillary end images a' and b' in the X-axis direction are represented as Xa and Xb. Figure 6BAs shown, extremely high levels of brightness are obtained at coordinates Xd and Xe of the corneal reflective images Pd' and Pe'. Extremely low levels of brightness are obtained in the region from coordinates Xa to Xb, corresponding to the region of pupil 141 (the region of the pupil image obtained when the image of the light beam from pupil 141 is formed on the eye's imaging element 17), except for coordinates Xd and Xe. Furthermore, in the region of iris 143 outside pupil 141 (the region of the iris image outside the pupil image obtained when the image of the light beam from iris 143 is formed), an intermediate brightness between the two types of brightness is obtained. Specifically, an intermediate brightness between the two types of brightness is obtained in the region where the X-coordinate (coordinate in the X-axis direction) is less than coordinate Xa and in the region where the X-coordinate is greater than coordinate Xb.

[0057] from Figure 6B The brightness distribution shown allows us to obtain the X coordinates Xd and Xe of the corneal reflection images Pd′ and Pe′, and the X coordinates Xa and Xb of the pupillary images a′ and b′. Specifically, we can obtain the coordinates of the corneal reflection images Pd′ and Pe′ with extremely high brightness, and the coordinates of the pupillary images a′ and b′ with extremely low brightness. Furthermore, when the rotation angle θx of the optical axis of the eyeball 14 relative to the optical axis of the light-receiving lens 16 is small, the coordinates Xc of the pupillary center image c′ (the center of the pupil image) obtained when the image of the light beam from the pupil center c is formed on the eye's imaging element 17 can be expressed as Xc≈(Xa+Xb) / 2. That is, the coordinates Xc of the pupillary center image c′ can be calculated based on the X coordinates Xa and Xb of the pupillary images a′ and b′. In this way, the coordinates of the corneal reflection images Pd′ and Pe′ and the coordinates of the pupillary center image c′ can be estimated.

[0058] In step S704, CPU 3 calculates the imaging magnification β of the eye image. The imaging magnification β is determined by the position of the eyeball 14 relative to the light-receiving lens 16. The imaging magnification β can be calculated as a function of the interval (Xd-Xe) between the corneal reflective images Pd′ and Pe′.

[0059] In step S705, CPU 3 calculates the rotation angle of the optical axis of eyeball 14 relative to the optical axis of light-receiving lens 16. The X-coordinate of the midpoint between the corneal reflective images Pd and Pe is almost identical to the X-coordinate of the center of curvature O of cornea 142. Therefore, when the normal distance from the center of curvature O of cornea 142 to the center c of pupil 141 is defined as Oc, the rotation angle θx of eyeball 14 in the ZX plane (the plane perpendicular to the Y-axis) can be calculated according to Equation 1. The rotation angle θy of eyeball 14 in the ZY plane (the plane perpendicular to the X-axis) can also be calculated using the same method as used to calculate the rotation angle θx.

[0060] β×Oc×SINθx≈{(Xd+Xe) / 2}-Xc...(Formula 1)

[0061] In step S706, CPU 3 uses the rotation angles θx and θy calculated in step S705 to calculate (estimate) the user's viewpoint (gaze position; the position when the user is turning his / her eyes) on the screen of display device 10. When the gaze position (coordinates of the viewpoint) (Hx, Hy) indicates the coordinates corresponding to the pupil center c, the gaze position (Hx, Hy) can be calculated according to the following formulas 2 and 3.

[0062] Hx=m×(Ax×θx+Bx)…(Equation 2)

[0063] Hy=m×(Ay×θy+By)…(Equation 3)

[0064] In Equations 2 and 3, the parameter m is a constant set according to the configuration of the viewfinder optical system of the camera 1 (such as the light-receiving lens 16), and is a conversion coefficient that converts the rotation angles θx and θy into coordinates corresponding to the pupil center c on the screen of the display device 10. The parameter m is predetermined and stored in the memory unit 4. The parameters Ax, Bx, Ay, and By are gaze correction parameters used to correct for individual differences in gaze, and are obtained through a calibration operation. Before the gaze detection operation begins, the parameters Ax, Bx, Ay, and By are stored in the memory unit 4.

[0065] In step S707, CPU 3 stores the gaze position (Hx, Hy) in memory unit 4 and ends the gaze detection operation.

[0066] Description of AF processing based on gaze position

[0067] Camera 1 has an AF (autofocus) function based on the photographer's gaze position. When the photographer's gaze position is detected, camera 1 adjusts the focal length of the camera lens unit 1A to achieve focus around the gaze position.

[0068] However, in AF processing based on gaze position (AF operation), there are situations where the photographer's gaze position and the focus position are inconsistent. For example, in AF processing with subject detection, camera 1 detects subjects around the photographer's gaze position and focuses on the detected subjects. However, when another subject exists around the subject the photographer is viewing (the area where the gaze position has been detected), there are cases where camera 1 detects a different subject and focuses on that other subject. Furthermore, in AF processing without subject detection, camera 1 detects areas that can be focused from the periphery of the gaze position (such as high-contrast areas) and focuses on the detected areas. However, there are cases where the area that camera 1 can focus on is far from the photographer's gaze position. In such cases, the captured image may not be focused on the photographer's gaze position.

[0069] The user (the photographer in this embodiment) displays the image in a magnified manner to confirm, for example, that focus has been achieved. Therefore, in the first embodiment, the camera 1 is controlled such that when the image is displayed in a magnified or reduced manner, the image is magnified or reduced with the position centered on the photographer's gaze information. This allows the user to easily confirm whether focus has been achieved in the area of ​​focus for the photographer.

[0070] Record processing

[0071] Figure 8 This is a flowchart illustrating the recording process used to associate gaze information and images with each other. (Refer to...) Figure 8 Describe the recording process. Recording process begins when the power to camera 1 is turned on.

[0072] In step S801, the CPU 3 detects the gaze position of the user who is viewing the image captured by the camera element 2 and displayed on the display device 10.

[0073] In step S802, CPU 3 records gaze information in memory unit 4. The gaze information recorded in memory unit 4 indicates the gaze position and the time when the gaze position was detected.

[0074] In step S803, CPU 3 determines whether a release (operation of switch SW2) has been performed. If a release has not been performed, CPU 3 returns to step S801 and repeats the gaze position detection. If a release has been performed, CPU 3 proceeds to step S804.

[0075] In step S804, the CPU 3 records the gaze information recorded in the memory unit 4 and the image captured by the camera element 2 in association on the recording medium 5. The gaze information recorded on the recording medium 5 indicates multiple gaze positions detected after the camera 1 is powered on and before it is released. When the camera 1 is powered off, the CPU 3 ends the recording process.

[0076] Reproduction processing

[0077] Next, we will refer to Figure 9 This describes a part of the reproduction process used to reproduce the image recorded on recording medium 5. Figure 9 This is a flowchart of a portion of the playback process according to the first embodiment. The CPU 3 can control the live view image captured by the imaging element 2 or the image recorded on the recording medium 5 for display on the display device 10 or the rear display. The rear display is a display panel provided in the operating member 41. When a signal instructing the playback of an image recorded on the recording medium 5 is received, the CPU 3 reads the image from the recording medium 5 and copies the image to the memory unit 4. The CPU 3 displays the copied image to the memory unit 4 on the display device 10 or the rear display. Note that the playback process can be performed by a device separate from the camera 1 (such as a personal computer). The user in the playback process may be the same as or different from the user in the recording process.

[0078] Figure 9 This is a flowchart of the zoom-in / zoom-out process for zooming in / out on the image displayed on the display device 10 or the rear display. The zoom-in / zoom-out process begins when the user performs an operation to zoom in / out on the image displayed on the display device 10 or the rear display (zoom-in / zoom-out operation). Note that the user can use any one of the operation members 41 to 43 to perform the zoom-in / zoom-out operation.

[0079] In step S901, when the CPU 3 receives a zoom-in / zoom-out operation from the user, it reads (acquires) gaze information recorded on the recording medium 5 in association with the image being displayed, and copies this gaze information to the memory unit 4. The gaze information recorded on the recording medium 5 indicates multiple gaze positions detected before release.

[0080] In step S902, CPU 3 performs setting processing for the center position (reference point) for zooming in or out based on the gaze information obtained in step S901. The reference point setting process will be described in detail later.

[0081] In step S903, CPU 3 zooms in or out on the image displayed on display device 10 or the rear display, centered on the reference point determined in step S902. By zooming in on the image centered on a position based on gaze information, the user can easily confirm, for example, whether the focus at the intended location is achieved or whether the facial expression of the subject is in good condition. When the CPU 3 receives an instruction from the user to end the zoomed-in or zoomed-out display, it ends the zoomed-in or zoomed-out display.

[0082] Reference point setting processing

[0083] Next, we will refer to Figure 10A Detailed description in Figure 9 The reference point setting process is performed in step S902. Figure 10A This is a flowchart illustrating an example of the reference point setting process according to the first embodiment.

[0084] exist Figure 10A In step S901, the CPU 3 sets the latest (immediately before release) gaze position among the multiple gaze positions indicated by the gaze information obtained in step S901 as the reference point (step S1001). Note that the gaze position indicates the coordinates (Hx, Hy) on the display device 10 corresponding to the center c of the pupil 141. By setting the latest gaze position as the reference point, the user can easily identify the subject area being focused on immediately before release.

[0085] Figure 10B This is a flowchart illustrating a variation of the reference point setting process according to the first embodiment. Figure 10B In the process, CPU 3 sets the reference point based on the fluctuations in the gaze position.

[0086] In step S1011, CPU 3 calculates the fluctuation (variance) of multiple gaze positions (Hx, Hy) indicated by the gaze information obtained in step S901. Note that, in addition to the variance, only the value used to evaluate the fluctuation of the gaze position can be used. Furthermore, CPU 3 can calculate the fluctuation of either the X or Y coordinate of the gaze position.

[0087] In step S1012, CPU 3 determines whether the fluctuation calculated in step S1011 is less than the third threshold. When the fluctuation is less than the third threshold, CPU 3 proceeds to step S1013, and when the fluctuation is at least the third threshold, CPU 3 proceeds to step S1014.

[0088] In step S1013, CPU 3 obtains the average position of multiple gaze positions indicated by the gaze information obtained in step S901, and sets the average position as a reference point. When the fluctuation of the gaze position is less than the third threshold as described above, CPU 3 controls the image to be zoomed in or out centered on the average position of the gaze position.

[0089] In step S1014, the CPU 3 sets the focus position of the image displayed on the display device 10 or the rear display as a reference point. Note that the CPU 3 can set the center position of the image as the reference point. For example, when the focus position of the image is acquired through AF operation, the CPU 3 sets the focus position as the reference point. When the focus position of the image is not acquired, as in the case of manual focusing, the CPU 3 sets the center position of the image as the reference point. When AF operation is performed but the focus position of the image is not acquired, the CPU 3 sets the center position of the image as the reference point. When the fluctuation of the gaze position is at least the third threshold as described above, the CPU 3 controls the image to be magnified or reduced centered on the focus position or center position of the image.

[0090] Here, the effect of processing based on fluctuations in gaze position will be described. The human eye is characterized by slight eye movements even when gazing at a point (involuntary eye movements), and its gaze position is not always set at the same coordinates. Furthermore, it is assumed that the user glanced at other subjects during the photography process. Therefore, there is a possibility that each gaze position differs from the position of the subject that needs to be zoomed in or out for confirmation. Moreover, when the fluctuation in gaze position exceeds a predetermined value, there is a possibility that the average position of the gaze position differs from the position of the subject that needs to be zoomed in or out for confirmation. Therefore, by taking into account the detected fluctuations in gaze position when setting a reference point, the probability that the position of the subject that needs to be zoomed in or out for confirmation by the user equals the reference point can be increased.

[0091] Figure 10C This is a flowchart illustrating a variation of the reference point setting process according to the first embodiment. Here, it is assumed that an image recorded on the recording medium 5 is displayed on the display device 10. Furthermore, it is assumed that a gaze detection operation is performed to obtain gaze information while the user views the image displayed on the display device 10 (the image recorded on the recording medium 5). Figure 10C The settings processing includes processing using gaze information (current gaze information) related to the user who is viewing the display device 10.

[0092] In step S1021, CPU 3 determines whether the gaze information recorded in memory unit 4 is set to be used. This allows the user to use operation components 41 to 43 to make settings related to whether or not the gaze information recorded in memory unit 4 is used. When the gaze information recorded in memory unit 4 is set to be used, CPU 3 proceeds to step S1022. When the gaze information recorded in memory unit 4 is set to be unused, CPU 3 proceeds to step S1023.

[0093] In step S1022, CPU 3 sets a reference point based on the gaze information recorded in memory unit 4. For example, CPU 3 based on... Figure 10A or Figure 10B The method shown is for setting a reference point.

[0094] In step S1023, CPU 3 determines whether the user is in contact with eyepiece 12. This determination is based on, for example, whether an eye image of the user's eyes has been acquired. If an eye image has been acquired, CPU 3 determines that the user is in contact with eyepiece 12 and proceeds to step S1024. If an eye image has not yet been acquired, CPU 3 determines that the user is separated from eyepiece 12 and proceeds to step S1025. Note that when... Figure 10B In step S1012, if the fluctuation is determined to be at least the third threshold, CPU 3 can perform control to switch to... Figure 10C Step S1023.

[0095] In step S1024, the CPU 3 sets a reference point based on the position of the gaze information (e.g., the gaze position indicated by the current gaze information) related to the user viewing the image displayed on the display device 10. When the gaze information recorded in the memory unit 4 is acquired in a state where it is set to be unused as described above, the CPU 3 controls the image to be zoomed in or out centered on the position of the gaze information. This control allows the user to easily identify the area of ​​the subject he / she is currently focusing on in the image displayed on the display device 10.

[0096] In step S1025, the CPU 3 sets the focus position of the image displayed on the display device 10 as a reference point. Note that, like... Figure 10B As in step S1014, the CPU 3 can set the center position of the image as a reference point. When the gaze information recorded in the memory unit 4 is set to unused as described above and no gaze information is acquired, the CPU 3 controls the image to be magnified or reduced centered on the focus position or center position of the image.

[0097] Note that in the first embodiment, gaze information indicating multiple gaze details is recorded. However, a single gaze position can be recorded and set as a reference point. Therefore, the image can be zoomed in or out with simpler processing.

[0098] As described above, in the first embodiment, when the image recorded on the recording medium 5 is magnified or reduced, the camera 1 performs display control to magnify or reduce the image centered on the position of the gaze information recorded on the recording medium 5. Therefore, this allows the user to easily confirm, for example, the realization of focus at the position intended by the user (photographer), and reduces the time and effort required for selecting images.

[0099] Second Embodiment

[0100] In the second embodiment, unlike the first embodiment, the camera 1 has a receiving function that receives an instruction from the user to fix the gaze position. Furthermore, the recording process according to the second embodiment includes a process for determining whether the captured image meets predetermined conditions. Hereinafter, the differences from the first embodiment will be primarily described.

[0101] Record processing

[0102] Reference Figure 11 The recording process according to the second embodiment is described. Figure 11 This is a flowchart illustrating the recording process according to the second embodiment. Recording processing begins when the camera 1 is powered on. The processing of steps S1101 and S1102 is... Figure 8 The processing steps S801 and S802 are the same.

[0103] In step S1103, CPU 3 determines whether it has received an instruction from the user to determine the gaze position. If an instruction to determine the gaze position has been received, CPU 3 proceeds to step S1104. If an instruction to determine the gaze position has not been received, CPU 3 proceeds to step S1106.

[0104] Here, the function of determining the gaze position will be described. The release button 5 is configured as a two-stage push-button switch. This allows the user to indicate AF operation via a first stroke, or so-called half-press (switch SW1), and to release via a second stroke, or so-called full-press (switch SW2). Furthermore, the release button 5 has a receiving function for receiving an indication from the user that the gaze position has been determined at a specific location (gaze position determination). This allows the user to determine the gaze position via, for example, the operation of switch SW1, and to indicate AF operation based on the determined position. Note that in the second embodiment, the operation of determining the gaze position is assigned to the operation of switch SW1, but it could also be assigned to the operations of operating members 41 to 43.

[0105] In step S1104, CPU 3 records the gaze information at the time of gaze position determination in step S1103 on memory unit 4. The gaze information at the time of gaze position determination indicates the gaze position acquired by CPU 3 when it has received the instruction to determine the gaze position from the user, and the time when CPU 3 detected the gaze position. Note that CPU 3 continues to perform gaze detection even after receiving the instruction to determine the gaze position, and records multiple gaze information entries, including gaze positions other than the determined position, on memory unit 4. The recording of multiple gaze information entries on memory unit 4 makes the gaze information at the time of gaze position determination identifiable.

[0106] In step S1105, the CPU 3 calculates the acceleration of the camera 1 using at least one of the acceleration sensors included in the camera housing 1B and the acceleration sensor 119 included in the camera lens unit 1A, and records the acceleration in the memory unit 4.

[0107] In step S1106, CPU 3 determines whether a release has been performed (switch SW2 operation). If a release has not been performed, CPU 3 returns to step S1101 and repeats the detection of gaze position and the recording of gaze information. If a release has been performed, CPU 3 proceeds to step S1107. Note that when repeating the recording of gaze information, CPU 3 records the gaze information in different areas of memory unit 4 and records gaze information indicating multiple positions. Note that CPU 3 can update and record gaze information in the same area of ​​memory unit 4.

[0108] In step S1107, CPU 3 determines whether the gaze information at the time of gaze position determination has been recorded in memory unit 4. If the gaze information at the time of gaze position determination has not been recorded in memory unit 4 (i.e., when CPU 3 has not yet received the instruction to determine the gaze position in step S1103), CPU 3 proceeds to step S1108. If the gaze information at the time of gaze position determination has been recorded in memory unit 4 (i.e., when CPU has received the instruction to determine the gaze position in step S1103), CPU 3 proceeds to step S1109.

[0109] In step S1108, the CPU 3 records the gaze information recorded on the memory unit 4 and the image captured by the camera element 2 in association on the recording medium 5.

[0110] In step S1109, CPU 3 determines whether the captured image meets predetermined conditions. Figure 11In the recording process shown, a predetermined condition is met when the movement of camera 1 exceeds a first threshold during image capture. The movement of camera 1 is calculated based on the average or instantaneous acceleration after CPU 3 receives the instruction to determine the gaze position in step S1103 and before release in step S1106. When at least one of the average and instantaneous accelerations exceeds the first threshold, CPU 3 determines that panning has been performed (the predetermined condition is met) and proceeds to step S1110. When both the average and instantaneous accelerations are not greater than the first threshold, CPU 3 determines that panning has not been performed (the predetermined condition is not met) and proceeds to step S1111. Here, the case where the instantaneous acceleration exceeds the first threshold is, for example, when there is a time interval during the period from the determination of the gaze position to release where the instantaneous acceleration exceeds the first threshold. The case where the instantaneous acceleration exceeds the first threshold can also be when there is a period of time longer than a predetermined length during the period from the determination of the gaze position to release where the instantaneous acceleration exceeds the first threshold.

[0111] In step S1110, the CPU 3 records gaze information, captured images, and judgment information indicating whether the images meet predetermined conditions on the recording medium 5 in a mutually correlated manner. In step S1110, the CPU 3 records panning execution information on the recording medium 5 as information indicating that panning has been performed (meeting predetermined conditions). The gaze information recorded in step S1110 may or may not include gaze information when the gaze position is determined.

[0112] In step S1111, CPU 3 records gaze information, captured images, and judgment information in a correlated manner on recording medium 5. In step S1111, CPU 3 records panning not executed information (indicating that a predetermined condition has not been met) as judgment information. The gaze information recorded in step S1111 includes gaze information at the time of gaze determination. When recording on recording medium 5 is completed through any one of steps S1108, S1110, and S1111, CPU 3 ends the recording process.

[0113] Reference point setting processing

[0114] Because the image reproduction process read from the recording medium 5 in the second embodiment is different from that in the first embodiment... Figure 9 The reproduction process shown is the same, so its description will be omitted. Next, refer to... Figure 12A Detailed description when reproduction is achieved Figure 11 The recording process shown is in the image being recorded. Figure 9 The reference point setting process is performed in step S902. Figure 12A This is a flowchart illustrating an example of the reference point setting process according to the second embodiment.

[0115] exist Figure 12A In step S1201, the CPU 3 determines whether the judgment information (panning execution information or panning non-execution information) is recorded on the recording medium 5 in association with the image displayed on the display device 10 or the rear display. When judgment information is recorded, the CPU 3 reads the judgment information from the recording medium 5 and proceeds to step S1202. When no judgment information is recorded, the CPU 3 proceeds to step S1203. Note that in step S1201, the CPU 3 can determine whether the gaze information at the time of gaze position determination is recorded on the recording medium 5 in association with the image.

[0116] In step S1202, CPU 3 determines whether the judgment information read from the recording medium 5 is panning execution information or panning non-execution information. When the judgment information is panning execution information, CPU 3 proceeds to step S1203. When the judgment information is panning non-execution information, CPU 3 proceeds to step S1204.

[0117] In step S1203, the CPU 3 sets the focus position of the image displayed on the display device 10 or the rear display to the center position (reference point) for zooming in or out. Note that, like Figure 10B As in step S1014, the CPU 3 can set the center position of the image as the reference point. Here, when panning is performed before release after the CPU 3 has received an instruction to determine the gaze position, there is a possibility that the subject the user was focusing on at the time the gaze position was determined may not be recorded in the image. When panning is performed before release as described above, there is a possibility that the gaze position at the time the gaze position was determined may differ from the position of the subject that needs to be zoomed in or out by the user for confirmation. Therefore, when recording panning execution information in association with the image, the CPU 3 sets the focus position or center position of the image as the reference point, rather than setting the gaze position at the time the gaze position was determined as the reference point.

[0118] In step S1204, CPU 3 sets the gaze position determined in step S1103 as a reference point. As described above, when panning inaction information is recorded in association with the image, CPU 3 controls the image to be zoomed in or out centered on the gaze position when the specified (gaze position determined) instruction is received. Therefore, this allows the user to easily identify the subject area where the photographer has determined the gaze position.

[0119] Figure 12B This is a variation of the reference point setting process. Figure 12B The processing of steps S1211, S1212 and S1214 and Figure 12A The processes in steps S1201, S1202, and S1204 are the same.

[0120] In step S1213, CPU 3 sets the latest gaze position among the multiple gaze positions indicated by the gaze information obtained in step S901 as the reference point. As described above, when panning execution information is recorded in association with an image, CPU 3 can set the latest gaze position as the reference point.

[0121] Note that CPU 3 can be based on Figure 10A or Figure 10B The method shown is used to set the reference point. For example, there exists a situation where the CPU 3 determines, based on gaze information recorded in association with an image displayed on the display device 10 or the rear display, that panning will be performed before a specific time but not after that specific time. In this case, the CPU 3 can, taking gaze information from the period after the specific time (the period during which panning is not performed) as the object, set the reference point according to... Figure 10A or Figure 10B The method shown is for setting a reference point.

[0122] Record processing

[0123] Next, we will refer to Figure 13 A variation of the record processing according to the second embodiment is described. Figure 13 It is a flowchart of the record processing. Figure 13 The processing of steps S1301 to S1304 and steps S1306 to S1308 and Figure 11 The processes in steps S1101 to S1104 and steps S1106 to S1108 are the same.

[0124] In step S1305, CPU 3 uses image processing circuit 207 to detect a subject (such as a face or human body) in the area surrounding the gaze position determined in step S1303. CPU 3 continues subject detection until release and records the position of the detected subject (coordinates on display device 10 or rear display) in memory unit 4. Note that when an operation to determine the gaze position is received, and then an operation to determine the gaze position is received at a new position after cancellation processing, CPU 3 can detect a subject in the area surrounding the new position. When no operation to determine the gaze position is received, CPU 3 may or may not detect a subject.

[0125] In step S1309, CPU 3 determines whether the captured image meets predetermined conditions. Figure 13In the recording process shown, when the movement (movement amount) of the subject during image capture is greater than a second threshold, a predetermined condition is met. Based on the movement amount (movement distance) of the subject before release in step S1306 after the CPU 3 has received the instruction to determine the gaze position in step S1303, the movement amount of the subject is calculated. When the movement amount of the subject is greater than the second threshold, the CPU 3 determines that the movement amount is large (meeting the predetermined condition) and proceeds to step S1310. When the movement amount of the subject is not greater than the second threshold, the CPU 3 determines that the movement amount is small (not meeting the predetermined condition) and proceeds to step S1311. Note that the CPU 3 can change the second threshold according to the field of view (i.e., the focal length of the attached camera lens unit 1A).

[0126] In step S1310, the CPU 3 records the gaze information, the captured image, and the judgment information in a correlated manner on the recording medium 5. In step S1310, the CPU 3 records information indicating a large amount of movement of the subject as judgment information. The gaze information recorded in step S1310 may or may not include gaze information when the gaze position is determined.

[0127] In step S1311, CPU 3 records gaze information, captured images, and judgment information in a correlated manner on recording medium 5. In step S1311, CPU 3 records information indicating minimal movement of the subject as judgment information. The gaze information recorded in step S1311 includes gaze information at the time of gaze determination. When recording is completed through any one of steps S1308, S1310, and S1311, CPU 3 terminates the recording process.

[0128] Reference point setting processing

[0129] Next, we will refer to Figure 14A and Figure 14B Detailed description when reproduction is achieved Figure 13 The recording process shown is in the image being recorded. Figure 9 The reference point setting process is performed in step S902. Figure 14A and Figure 14B This is a flowchart illustrating an example of the reference point setting process according to the second embodiment. Note that... Figure 14A The processing of steps S1403 and S1404 and Figure 12A The processing of steps S1203 and S1204 is the same, and Figure 14B The processing of steps S1413 and S1414 and Figure 12B The processing of steps S1213 and S1214 is the same. Furthermore, Figure 14B The processing of steps S1411 and S1412 and Figure 14AThe processing of steps S1401 and S1402 is the same.

[0130] exist Figure 14A In step S1401, the CPU 3 determines whether judgment information (information indicating whether the amount of movement of the subject is large or small) is recorded on the recording medium 5 in association with the image displayed on the display device 10 or the rear display. When judgment information is recorded, the CPU 3 proceeds to step S1402, and when no judgment information is recorded, the CPU 3 proceeds to step S1403. Note that in step S1401, the CPU 3 can determine whether the gaze information at the time of gaze position determination is recorded on the recording medium 5 in association with the image.

[0131] In step S1402, CPU 3 determines whether the judgment information read from the recording medium 5 indicates a large amount of movement of the subject or a small amount of movement. When the judgment information indicates a large amount of movement of the subject, CPU 3 proceeds to step S1403, and when the judgment information indicates a small amount of movement of the subject, CPU 3 proceeds to step S1404.

[0132] like Figure 14A and Figure 14B As shown, when the movement of the subject is less than the second threshold, the CPU 3 sets the gaze position at the time of gaze position determination as the reference point. On the other hand, when the movement of the subject is greater than the second threshold, the CPU 3 sets the focus position of the image, the center position of the image, or the latest gaze position as the reference point. This is because there is a possibility that if the subject moves significantly after the CPU 3 has received the instruction to determine the gaze position but before releasing it, the gaze position at the time of gaze position determination may differ from the position of the subject that needs to be zoomed in or out by the user for confirmation.

[0133] In the second embodiment, when an image is recorded on the recording medium 5, the CPU 3 determines whether the image meets predetermined conditions and records the determination result and the image in association on the recording medium 5. Note that the CPU 3 can determine whether the image meets predetermined conditions when reproducing the image. For example, when setting a reference point based on the movement of the camera 1 during image capture, the CPU 3 records acceleration-related information and the image in association on the recording medium 5. The CPU 3 can determine whether the movement of the imaging device exceeds a first threshold based on the acceleration-related information recorded on the recording medium 5 when reproducing the image, and set the reference point based on the determination result.

[0134] As described above, in the second embodiment, when enlarging or reducing the image recorded on the recording medium 5, if the image meets predetermined conditions, the camera 1 is controlled to enlarge or reduce the image centered on the image's focus position, the image's center position, or the latest gaze position. If the image does not meet the predetermined conditions, the camera 1 is controlled to enlarge or reduce the image centered on the gaze position obtained when the user receives an instruction to determine the gaze position. Therefore, this makes it easier for the user to confirm the realization of focus at the position intended by the user (photographer).

[0135] Third Embodiment

[0136] The first and second embodiments describe the processing for magnifying or reducing images recorded on the recording medium 5. The third embodiment will describe the processing for magnifying or reducing images captured by the imaging element 2 (live view images, images of the subject in real time).

[0137] Reproduction processing

[0138] Reference Figure 15 This describes a part of the reproduction process for reproducing the live view image captured by the imaging element 2. In the third embodiment, it is assumed that the live view image is displayed on the display device 10. Figure 15 This is a flowchart of a portion of the reproduction process according to the third embodiment, and a flowchart of a zoom-in / zoom-out process for zooming in or out of a live view image. The zoom-in / zoom-out process begins when a user performs an operation to zoom in or out of the live view image displayed on the display device 10 (zoom-in / zoom-out operation).

[0139] In step S1501, the CPU 3 detects the gaze position of the user who is viewing the live view image captured by the camera element 2 and displayed on the display device 10, and obtains gaze information.

[0140] In step S1502, CPU 3 performs a setting process for the reference point (center position) used to set the zoom in or zoom out based on the gaze information obtained in step S1501. The reference point setting process will be described in detail later.

[0141] In step S1503, CPU 3 zooms in or out on the image displayed on display device 10, centered on the reference point determined in step S1502. By zooming in on the image centered on a position based on gaze information, the user can easily confirm, for example, whether the focus at the intended location is achieved or whether the facial expression of the subject is in good condition, before releasing the image. When the CPU 3 receives an instruction from the user to end the zoomed-in or zoomed-out display, it ends the zoomed-in or zoomed-out display.

[0142] Reference point setting processing

[0143] Next, we will refer to Figure 16A Detailed description in Figure 15 The reference point setting process is performed in step S1502. Figure 16A This is a flowchart illustrating an example of the reference point setting process according to the third embodiment. Here, it is assumed that during the period when the user views the live view image displayed on the display device 10, a gaze detection operation is performed before receiving the zoom-in / zoom-out operation, and the gaze information is recorded on the memory unit 4.

[0144] exist Figure 16A In step S1601, CPU 3 determines whether the gaze position has been correctly detected in step S1501. This determination is made by comparing, for example, at least one gaze position prior to the zoom-in / zoom-out operation (the previous gaze position) with the gaze position acquired in step S1501. The previous gaze position may, for example, be a gaze position detected at a predetermined time prior to receiving the zoom-in / zoom-out operation. When the deviation between the previous gaze position and the gaze position acquired in step S1501 is not greater than a predetermined threshold, CPU 3 determines that the gaze position has been correctly detected; and when the deviation is greater than the predetermined threshold, CPU 3 determines that the gaze position has not been correctly detected. When the gaze position has been correctly detected, CPU 3 proceeds to step S1602; and when the gaze position has not been correctly detected, CPU 3 proceeds to step S1603.

[0145] Note that in step S1601, CPU 3 can determine whether a gaze position has been detected in step S1501. This determination, related to whether a gaze position has been detected, is performed, for example, by detecting the corneal reflection images Pd and Pe in step S1501 or calculating the pupil center c. As described above, CPU 3 can perform processing branches without relying on comparisons with previous gaze positions. When a gaze position has been detected, CPU 3 proceeds to step S1602, and when a gaze position has not been detected, CPU 3 proceeds to step S1603.

[0146] In step S1602, CPU 3 sets the gaze position indicated by the gaze information acquired in step S1501 as the center position (reference position) for zooming in or out. As described above, CPU 3 controls the zooming in or out of the image centered on the position of gaze information related to the user viewing the image displayed on display device 10 (e.g., the gaze position indicated by the current gaze information). This control allows the user to easily identify the area of ​​the subject they are currently focusing on in the image displayed on display device 10. Note that CPU 3 may set the reference point based on the acquired gaze information, for example, the average position of the gaze position over a predetermined time period.

[0147] In step S1603, the CPU 3 sets the center position of the image displayed on the display device 10 as a reference point. When it is determined in step S1601 that the gaze position has not been correctly detected, the CPU 3 sets the center position of the image as a reference point, thereby preventing the image from being zoomed in or out centered on an area outside the user's intended area. Note that in step S1603, when the focus position of the image is obtained, the CPU 3 can set the focus position of the image as a reference point.

[0148] Figure 16B This is a variation of the reference point setting process. In step S1611, CPU 3 determines whether the gaze position detected by the gaze detection operation is set to be displayed on the display device 10. Note that this allows the user to use the user's operation components 41 to 43 to make settings related to whether the gaze position is displayed. Furthermore, when the gaze position is displayed, it can be displayed on the display device 10 by, for example, a gaze pointer. When the gaze position is set to be displayed, CPU 3 proceeds to step S1612, and when the gaze position is set not to be displayed, CPU 3 proceeds to step S1615.

[0149] In step S1612, CPU 3 determines whether the user is in contact with eyepiece 12. This determination is made, for example, by acquiring an image of the user's eyes. When it is determined that the user is in contact with the eyepiece, CPU 3 proceeds to step S1613, and when it is determined that the user is separated from the eyepiece, CPU 3 proceeds to step S1614. Note that in step S1612, CPU 3 may determine, as in step S1601, whether the gaze position was correctly detected in step S1501.

[0150] In step S1613, CPU 3 sets the gaze position indicated by the gaze information obtained in step S1501 as the reference point.

[0151] In step S1614, CPU 3 sets the center position of the image displayed on display device 10 as a reference point. Similarly, in step S1615, CPU 3 sets the center position of the image displayed on display device 10 as a reference point. Note that in steps S1614 and S1615, when the focus position of the image is obtained, CPU 3 can set the focus position of the image as a reference point.

[0152] Here, the effect of processing based on whether or not the gaze position is displayed will be described. When the gaze position is not displayed on the display device 10 via a gaze indicator or the like, when the image is zoomed in or out centered on a position based on gaze information, the user may not understand the center of the zoom or reduction and may experience discomfort. Therefore, when the gaze position is set not to be displayed, the CPU 3 controls the zooming or reduction of the image to be centered on, for example, the center position of the image (rather than the position based on gaze information). This reduces discomfort.

[0153] Note that the third embodiment describes processing for magnifying or reducing the live view image displayed on the display device 10. However, the present invention can be applied to processing for magnifying or reducing the live view image displayed on the rear display. For example, when a user changes the display on the display device 10 to the rear display midway through taking a photo, the CPU 3 controls the magnification or reduction of the live view image based on the gaze information displayed on the display device 10. Therefore, this allows the user to view and confirm the area of ​​interest on the display device 10 on the rear display.

[0154] In the third embodiment, when the live view image captured by the imaging element 2 is magnified or reduced, the camera 1 is controlled to magnify or reduce the live view image centered on a position based on gaze information related to the user (photographer) viewing the live view image. Therefore, this allows the user to easily confirm the focus of the live view image, etc.

[0155] Fourth embodiment

[0156] Unlike the first to third embodiments, the recording process according to the fourth embodiment includes a process for rating the focus of the image recorded on the recording medium 5. Therefore, this allows the user to easily confirm the focus achievement when reviewing the image recorded on the recording medium 5.

[0157] Record processing

[0158] Reference Figure 17 The recording process according to the fourth embodiment is described. Figure 17This is a flowchart of the recording process according to the fourth embodiment. Recording processing begins when the power to camera 1 is turned on. The processing of steps S1701 and S1702 is... Figure 8 The processing steps S801 and S802 are the same.

[0159] In step S1703, CPU 3 determines whether an instruction to perform AF operation has been received (whether the switch SW1 operation of release button 5 has been performed). If the switch SW1 operation has not been performed, CPU 3 returns to step S1701 and repeats the detection of gaze position and recording of gaze information. If the switch SW1 operation has been performed, CPU 3 proceeds to step S1704.

[0160] In step S1704, the CPU 3 uses the autofocus detection circuit 203 to perform focus detection for detecting the distance to the subject detected based on gaze information.

[0161] In step S1705, CPU 3 calculates a lens drive amount, which is the driving amount of lens 101, based on the distance to the subject detected in step S1704. CPU 3 sends the calculated lens drive amount to focus adjustment circuit 118. Focus adjustment circuit 118 moves lens 101 to the focal position via lens drive member 114 based on the received lens drive amount.

[0162] In step S1706, CPU 3 determines whether a release (switch SW2 operation) has been performed. If a release has not been performed, CPU 3 returns to step S1701 and repeats the processing of steps S1701 to S1705. If a release has been performed, CPU 3 proceeds to step S1707.

[0163] In step S1707, CPU 3 performs setting processing for setting a judgment range (the range for judging focus) for the captured image. The judgment range is set based on the gaze information recorded in memory unit 4. The judgment range setting processing will be described in detail later.

[0164] In step S1708, CPU 3 calculates an evaluation value of the focus of the judgment range set in step S1707. CPU 3 calculates the focus evaluation value based on, for example, the contrast evaluation value of the judgment range. The contrast evaluation value is, for example, the sum of the contrast values ​​of multiple regions constituting the judgment range. Note that CPU 3 may calculate the focus evaluation value based on the amount of blur calculated using the point spread function (PSF). Note that the focus evaluation value is not limited to such a value, but may use only the value used to evaluate whether the image is in focus.

[0165] In step S1709, CPU 3 performs rating processing to rate the captured image based on the focus evaluation value calculated in step S1708. The rating processing will be described in detail later.

[0166] In step S1710, CPU 3 records the captured image and the rating result from step S1709 on the recording medium 5 in a correlated manner. When the power to camera 1 is turned off, CPU 3 terminates the recording process.

[0167] Setting the range of judgment

[0168] Next, we will refer to Figure 18A Detailed description in Figure 17 The judgment range setting process is performed in step S1707. Figure 18A This is a flowchart illustrating an example of the setting process for the judgment range according to the fourth embodiment.

[0169] exist Figure 18A In step S1801, the CPU 3 sets the latest gaze position among the multiple gaze positions indicated by the gaze information recorded in the memory unit 4 in step S1702 as the center position (reference point) of the judgment range.

[0170] In step S1802, the CPU 3 sets a predetermined range centered on a reference point as a judgment range for the captured image. Note that the CPU 3 can change the size of the judgment range based on information such as the focal length of the camera lens unit 1A or the camera settings. Furthermore, the size of the judgment range can be set by the user.

[0171] Figure 18B This is a variation of the setting and processing of the judgment range. Figure 18B In this process, CPU 3 sets a reference point based on the fluctuations in the gaze position recorded in memory unit 4. Note that the processing in step S1815 is related to... Figure 18A The process of step S1802 is the same.

[0172] In step S1811, CPU 3, according to, for example Figure 10B The method shown in step S1011 is used to obtain the fluctuation of the gaze position.

[0173] In step S1812, CPU 3 determines whether the fluctuation calculated in step S1811 is less than the third threshold. When the fluctuation is less than the third threshold, CPU 3 proceeds to step S1813, and when the fluctuation is at least the third threshold, CPU 3 proceeds to step S1814.

[0174] In step S1813, CPU 3 obtains the average position of multiple gaze positions indicated by the gaze information recorded in memory unit 4 in step S1702, and sets the average position as a reference point. As described above, when the fluctuation of the gaze position is less than the third threshold, CPU 3 sets the judgment range with the average position of the gaze position as the center.

[0175] In step S1814, CPU 3 sets the focus position of the captured image as a reference point. Note that CPU 3 can also set the center position of the image as the reference point. As described above, when the fluctuation of the gaze position is at least a third threshold, CPU 3 sets the judgment range centered on the focus position or center position of the image.

[0176] Rating Processing

[0177] Next, we will refer to Figure 19 Detailed description in Figure 17 The rating process is carried out in step S1709. Figure 19 This is a flowchart of the rating process according to the fourth embodiment. Figure 19 The method described below will serve as an example for rating captured images into four levels based on focus-based evaluation values.

[0178] In step S1901, CPU 3 determines that... Figure 17 The evaluation value of the focus calculated in step S1708 is at least the threshold A1. Note that in the fourth embodiment, a larger evaluation value indicates a higher focus within the judgment range, i.e., the realization of focus in the region centered on the gaze position. When the evaluation value is at least the threshold A1, CPU 3 proceeds to step S1902, and when the evaluation value is less than the threshold A1, CPU 3 proceeds to step S1903.

[0179] In step S1902, CPU 3 rates the captured image as three stars. Note that the number of stars is an indicator. That is, a larger number indicates higher focus, while a smaller number indicates lower focus.

[0180] In step S1903, when the evaluation value is at least threshold A2, CPU 3 proceeds to step S1904, and when the evaluation value is less than threshold A2, CPU 3 proceeds to step S1905. Note that threshold A2 is a value less than threshold A1.

[0181] In step S1904, CPU 3 rates the captured image as two stars.

[0182] In step S1905, when the evaluation value is at least threshold A3, CPU 3 proceeds to step S1906, and when the evaluation value is less than threshold A3, CPU 3 proceeds to step S1907. Note that threshold A3 is a value less than threshold A2.

[0183] In step S1906, CPU 3 rates the captured image as one star.

[0184] In step S1907, the CPU 3 rates the captured image as zero stars. After rating the image as any one of zero stars to three stars, the CPU 3 ends the rating routine.

[0185] Note that in Figure 19 The example described is a four-level rating system for focus evaluation, but the rating can be assigned to any level. Furthermore, the CPU 3 can set the focus evaluation value itself as the rating value. Additionally, in the fourth embodiment, focus calculation and rating are performed when the image captured by the imaging element 2 is recorded on the recording medium 5, but focus calculation and rating can also be performed when the image recorded on the recording medium 5 is reproduced.

[0186] As described above, in the fourth embodiment, camera 1 sets a judgment range for determining the focus of an image based on gaze information related to the user. Camera 1 rates the image based on the focus within the judgment range and records the image and the rating result in association. By referring to the rating result, the user can easily confirm the achievement of focus at the location intended by the user (photographer). Furthermore, when selecting an image based on the achievement of focus, the user can select an image without spending time and effort to zoom in for confirmation.

[0187] Fifth Embodiment

[0188] In the fifth embodiment, unlike the fourth embodiment, camera 1 has a receiving function for receiving an instruction to determine the gaze position at a specific location. Furthermore, camera 1 sets a judgment range considering whether an instruction for determining the gaze position has been received and whether the image meets predetermined conditions. Hereinafter, the differences from the fourth embodiment will be primarily described.

[0189] Record processing

[0190] Reference Figure 20 The recording process according to the fifth embodiment is described. Figure 20 This is a flowchart of the recording process. Note that in the recording process according to the fifth embodiment, the parameters related to recording are omitted. Figure 11 or Figure 17 The record processing shown is a common process. Specifically, Figure 20The processing of steps S2001 and S2002 and Figure 17 The processing of steps S1701 and S1702 is the same. Figure 20 Step S2005 processing and Figure 11 The process of step S1105 is the same. Figure 20 The processing of steps S2006 to S2009 and steps S2011 to S2013 and Figure 17 The processes in steps S1703 to S1706 and steps S1708 to S1710 are the same.

[0191] In step S2003, CPU 3 determines whether an instruction to determine the gaze position has been received from the user. In the fifth embodiment, this allows the user to provide an instruction for determining the gaze position using, for example, any of the operating components 41 to 43. When an instruction to determine the gaze position has been received, CPU 3 proceeds to step S2004, and when no instruction to determine the gaze position has been received, CPU 3 proceeds to step S2006.

[0192] In step S2004, CPU 3 records the gaze information at the time of gaze position determination in step S2003 on memory unit 4. Note that even after receiving the instruction to determine the gaze position in step S2003, CPU 3 continues to perform gaze detection operations and records multiple gaze information entries, including gaze positions other than the determined position, on memory unit 4. The recording of multiple gaze information entries on memory unit 4 makes the gaze information at the time of gaze position determination discernible.

[0193] In step S2010, CPU 3 performs the setting process for the judgment range. The setting process for the judgment range will be described in detail later.

[0194] Setting the range of judgment

[0195] Next, we will refer to Figure 21A Detailed description in Figure 20 The judgment range setting process is performed in step S2010. Figure 21A This is a flowchart illustrating an example of the setting process for the judgment range according to the fifth embodiment.

[0196] exist Figure 21A In step S2101, CPU 3 determines whether the gaze information when the gaze position is determined is recorded in memory unit 4. If the gaze information when the gaze position is determined is not recorded, CPU 3 proceeds to step S2102, and if the gaze information when the gaze position is determined is recorded, CPU 3 proceeds to step S2103.

[0197] In step S2102, the CPU 3 sets the latest (immediately before release) gaze position among the multiple gaze positions indicated by the gaze information recorded in the memory unit 4 as the center position (reference point) of the judgment range. Note that in step S2102, the CPU 3 can set the focus position of the captured image as the reference point.

[0198] In step S2103, CPU 3 sets the gaze position determined in step S2003 as a reference point. As described above, when the gaze information at the time of gaze position determination is recorded on memory unit 4, CPU 3 sets a judgment range centered on the gaze position at the time the gaze position determination instruction is received. By rating the focus of the image based on the judgment range set therein, the user can easily confirm whether the subject area (where the photographer has determined the gaze position) has been focused.

[0199] In step S2104, as in step S1802 of Figure 18, the CPU 3 sets a judgment range centered on the reference point for the captured image.

[0200] Figure 21B This is a variation of the setting and processing of the judgment range. Figure 21B In the process, the handling of steps S2111 and steps S2113 to S2115 is similar to... Figure 21A The processing of steps S2101 and steps S2102 to S2104 is the same. Figure 21B In the process, CPU 3 sets the reference point based on whether the image meets the predetermined conditions.

[0201] In step S2112, CPU 3 determines whether the captured image meets predetermined conditions. In step S2112, the image... Figure 11 As shown in step S1109, CPU 3 determines whether the captured image meets predetermined conditions. That is, in Figure 21B In the process, when the movement of camera 1 exceeds a first threshold during image capture, a predetermined condition is met. Based on the average or instantaneous acceleration of the camera 1 before release in step S2009 after the CPU 3 has received the instruction for determining the gaze position in step S2003, the movement of camera 1 is calculated. The CPU 3 proceeds to step S2113 if it determines that panning has been performed (the predetermined condition is met), and proceeds to step S2114 if it determines that panning has not been performed (the predetermined condition is not met).

[0202] Figure 21C This is a variation of the setting and processing of the judgment range. Figure 21C In the process, the steps S2121, S2122, S2124, and S2125 are processed and Figure 21BThe processing of steps S2111, S2112, S2114 and S2115 is the same. Figure 21C and Figure 21B The difference lies in the reference point set in step S2123 when the captured image meets predetermined conditions. In step S2123, CPU 3 sets the focus position of the captured image as the reference point. Note that CPU 3 can also set the center position of the image as the reference point.

[0203] As described above, when a pan is performed after the CPU 3 has received an instruction to determine the gaze position but before release, the CPU 3 sets the judgment range centered on the latest gaze position, the focus position of the image, or the center position of the image. This is because there is a possibility that the gaze position at the time of gaze position determination during panning differs from the position of the subject for which focus confirmation is required by the user.

[0204] Record processing

[0205] Next, we will refer to Figure 22 A variation of the record processing according to the fifth embodiment is described. Figure 22 This is a flowchart of the record processing. Note that... Figure 22 The processing of steps S2201 to S2204, S2206 to S2209, and S2211 to S2213 Figure 20 The processes of steps S2001 to S2004, S2006 to S2009, and S2011 to S2013 are the same.

[0206] In step S2205, CPU 3 uses image processing circuit 207 to detect a subject (such as a face and human body) in the area surrounding the gaze position determined in step S2203. CPU 3 continues subject detection until release and records the detected subject position (coordinates on display device 10 or rear display) in memory unit 4. Note that CPU 3, based on, for example... Figure 13 The method shown in step S1305 is used to detect the subject.

[0207] In step S2210, CPU 3 performs the setting process for the judgment range. The setting process for the judgment range will be described in detail later.

[0208] Setting the range of judgment

[0209] Next, we will refer to Figure 23A Detailed description in Figure 22 The judgment range setting process is performed in step S2210. Figure 23A This is a flowchart illustrating an example of the setting process for the judgment range according to the fifth embodiment. Figure 23AThe processing of steps S2301 and S2303 to S2305 and Figure 21B The processing of steps S2111 and steps S2113 to S2115 is the same.

[0210] In step S2302, CPU 3 determines whether the captured image meets predetermined conditions. In step S2302, the image... Figure 13 As shown in step S1309, CPU 3 determines whether the captured image meets predetermined conditions. That is, in Figure 23A In the process, when the movement (movement amount) of the subject during image capture exceeds a second threshold, a predetermined condition is met. Based on the movement amount (movement distance) of the subject before release in step S2209 after the CPU 3 has received the instruction to determine the gaze position in step S2203, the movement amount of the subject is calculated. When it is determined that the movement amount of the subject is large (meeting the predetermined condition), the CPU 3 proceeds to step S2303, and when it is determined that the movement amount is small (not meeting the predetermined condition), the CPU 3 proceeds to step S2304.

[0211] Figure 23B This is a variation of the setting and processing of the judgment range. Figure 23B In the process, the steps S2311, S2312, S2314, and S2315 are processed and Figure 23A The processing steps S2301, S2302, S2304 and S2305 are the same. Figure 23B and Figure 23A The difference lies in the reference point set in step S2313 when the image meets predetermined conditions. In step S2313, the CPU 3 sets the focus position of the captured image as the reference point.

[0212] As described above, when the subject moves significantly after the CPU 3 has received the instruction to determine the gaze position but before release, the CPU 3 sets the judgment range based on the latest gaze position, the image's focus position, or the image's center position. This is because there is a possibility that when the subject moves significantly, the gaze position at the time of gaze position determination differs from the position of the subject that requires user confirmation of focus.

[0213] As described above, in the fifth embodiment, when the captured image meets predetermined conditions, camera 1 sets a judgment range for determining the focus of the image, centered on the latest gaze position, the image's focus position, or the image's center position. When the captured image does not meet the predetermined conditions, camera 1 sets the judgment range centered on the gaze position at which the user receives the instruction to determine the gaze position. This setting increases the possibility of determining the image's focus at the position intended by the user (photographer). Therefore, this makes it easier for the user to confirm whether focus has been achieved at the position intended by him / her.

[0214] According to this disclosure, photographers can easily identify areas of interest.

[0215] Other embodiments

[0216] The present invention has been described in detail above based on preferred embodiments. However, the present invention is not limited to the defined embodiments and includes various modes without departing from its spirit. Some embodiments described above can be suitably combined together.

[0217] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0218] Furthermore, the above embodiments describe the application of the present invention to digital still cameras. However, the present invention is not limited to this example and is applicable to any electronic device capable of receiving gaze input during image recording. For example, the present invention is applicable to tablet terminals or smartphones. Furthermore, the above embodiments describe an example of gaze information and image recording and image reproduction performed by the same device. However, image recording and reproduction can be performed by different devices.

[0219] While the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be accorded the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An electronic device comprising: The reading unit is configured to read from the recording medium an image as a captured image, gaze information associated with the image, and judgment information associated with the image; The display control unit is configured to control the display unit; as well as A control unit is configured to control the magnification or reduction of an image read by the reading unit and displayed on the display unit by the display control unit. The gaze information includes the gaze position when the user receives a specific instruction for determining the gaze position. The judgment information indicates whether the movement of the camera device during the period from receiving the specific instruction until the image is captured exceeds a first threshold, and The control unit performs control such that: If the judgment information indicates that the movement of the camera device exceeds the first threshold, the image is magnified or reduced centered on the focal position of the image. If the judgment information indicates that the movement of the camera device is not greater than the first threshold, the image is magnified or reduced with the gaze position at the time of receiving the specific instruction as the center.

2. The electronic device according to claim 1, further comprising: The first acquisition unit is configured to acquire an image as a captured image; The second acquisition unit is configured to acquire gaze information relating to a user who is viewing the display unit; as well as A recording unit is configured to record the image acquired by the first acquisition unit, the gaze information acquired by the second acquisition unit, and the judgment information in association on the recording medium.

3. The electronic device according to claim 2, further comprising: The receiving unit is configured to receive from the user an instruction for setting whether to use gaze information recorded on the recording medium.

4. The electronic device according to claim 3, wherein, When the gaze information recorded on the recording medium is set to be unused, and the image read by the reading unit and displayed on the display unit by the display control unit is magnified or reduced, The control unit performs control such that: When the second acquisition unit acquires the gaze information, the image is zoomed in or out based on the position of the gaze information. If the second acquisition unit does not acquire the gaze information, the image is magnified or reduced based on the focus position or the center position of the image.

5. The electronic device according to any one of claims 2 to 4, wherein, When the image acquired by the first acquisition unit and displayed on the display unit by the display control unit is magnified or reduced, the control unit controls the image to be magnified or reduced with the position based on the gaze information acquired by the second acquisition unit as the center.

6. The electronic device according to claim 5, wherein, When the image acquired by the first acquisition unit and displayed on the display unit by the display control unit is zoomed in or zoomed out... The control unit performs control such that: When the second acquisition unit acquires the gaze information, the image is zoomed in or out based on the position of the gaze information. If the second acquisition unit does not acquire the gaze information, the image is magnified or reduced based on the focus position or the center position of the image.

7. The electronic device according to claim 5, further comprising: The second receiving unit is configured to receive from the user an instruction to set whether the gaze position based on the gaze information acquired by the second acquiring unit is displayed on the display unit.

8. The electronic device according to claim 7, wherein, When the image acquired by the first acquisition unit and displayed on the display unit by the display control unit is zoomed in or zoomed out... The control unit performs control such that: When the viewing position is set not to be displayed on the display unit, the image is magnified or reduced with the focal position or center position of the image as the center.

9. A method for controlling an electronic device, comprising: The reading step is used to read from the recording medium an image as a captured image, gaze information associated with the image, and judgment information associated with the image; The display control steps are used to control the display unit; as well as A control step, used to control the image read in the reading step and displayed on the display unit in the display control step to be magnified or reduced. The gaze information includes the gaze position when the user receives a specific instruction for determining the gaze position. The judgment information indicates whether the movement of the camera device during the period from receiving the specific instruction until the image is captured exceeds a first threshold, and Control is performed in the control step such that: If the judgment information indicates that the movement of the camera device exceeds the first threshold, the image is magnified or reduced centered on the focal position of the image. If the judgment information indicates that the movement of the camera device is not greater than the first threshold, the image is magnified or reduced with the gaze position at the time of receiving the specific instruction as the center.

10. A computer-readable storage medium storing a program that, when executed by a computer, implements the control method of the electronic device according to claim 9.

11. A computer program product comprising a program that, when executed by a computer, implements the control method of the electronic device according to claim 9.

Images