electronic machinery

Abstract

To provide a technique to prevent an object not intended by a user from being selected as a target to allow selection of a target according to the intention of the user.SOLUTION: An electronic apparatus of the present invention has: first detection means that detects a line-of-sight position that is a position a user sees; second detection means that detects an object and acquires object information that is information related to the object; receiving means that can receive a specific operation from the user; selection means that, in response to the specific operation, selects the object the user sees as a target based on the line-of-sight position detected by the first detection means; and recording means that, in response to the specific operation, records object information on the target selected by the selection means in a storage unit. When the specific operation is not performed, the selection means selects the target based on the object information stored by the storage unit.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to electronic devices such as imaging devices, and more particularly to a technique for selecting a target object from one or more objects. [Background technology]

[0002] An imaging device (camera) has been proposed that can detect a subject and identify the type of the subject (for example, a person, an animal, a vehicle, etc.) Also proposed as such a camera is a camera that selects a main subject from the detected subjects, which is the target of focus adjustment, exposure adjustment, etc.

[0003] When a captured image (image taken) contains multiple subjects, the subject that is closest to the center of the captured image or the subject with the largest image size (size on the captured image) is generally selected as the main subject.

[0004] However, a user (photographer) may not necessarily want to select a subject near the center of a captured image or a subject with a large image size as the main subject. For example, a user may want to capture a photograph with a main subject, such as a person, positioned at the edge of the captured image. In such a case, if a large subject, such as a train, is captured in the center of the captured image, the large subject will be selected as the main subject, contrary to the user's intention. While a user can manually select a main subject, this is time-consuming.

[0005] Cameras have also been proposed that can estimate the user's viewpoint (the position where the user is looking) in a captured image and select a subject near the viewpoint as the main subject in response to the user's viewpoint confirmation operation.Even with these cameras, selecting a subject located at the edge of the captured image or a small subject as the main subject requires the user to perform the time-consuming operation to confirm the point of gaze every time the shooting scene changes.

[0006] Patent Document 1 discloses a technique in which a user stores information about subjects photographed in the past and selects a main subject based on that information. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent Publication No. 2021-32927 Summary of the Invention [Problem to be solved by the invention]

[0008] However, with the technology disclosed in Patent Document 1, when a captured image contains multiple subjects, not only information about the main subject but also information about subjects other than the main subject is stored. Even if only information about the main subject is stored, if the main subject is erroneously determined, information about a subject unintended by the user will be stored. As a result, there is a risk that a subject unintended by the user will be selected as the main subject based on information about subjects captured in the past. This problem arises not only when selecting a main subject from one or more subjects, but also in various cases when selecting an object from one or more objects.

[0009] The present invention aims to provide a technology that prevents an object unintended by the user from being selected as a target object and enables the user to select a target object that matches the user's intention. [Means for solving the problem]

[0010] The electronic device of the present invention comprises: On the display surface of the display device, a first detection means for detecting a gaze position, which is a position where a user is looking; From the image displayed on the display device Detect an object and Categories Object information as a second detection means for acquiring the gaze position detected by the first detection means in response to the specific operation; a reception means for receiving a specific operation from the user; One displayed within a given range froma selection means for selecting an object as a target object; and a recording means for recording object information of the target object selected by the selection means in a storage unit in response to the specific operation, When the specific operation is performed a plurality of times, a plurality of pieces of object information corresponding to the plurality of specific operations are recorded in the storage unit, The selection means is configured to select a After that, the specific operation is performed. Not yet period based on the object information stored in the storage unit, The priority of the object detected by the second detection means is changed from a default value, and the object with the highest priority after the change is selected. the object as It is characterized by selecting. [Effects of the Invention]

[0011] According to the present invention, it is possible to prevent an object unintended by the user from being selected as the target object, and to select a target object that matches the user's intention. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of the appearance of a camera. [Figure 2] FIG. 2 is a diagram illustrating an example of the internal configuration of a camera. [Figure 3] FIG. 2 is a block diagram showing an example of an electrical configuration within the camera. [Figure 4] FIG. 2 is a diagram showing an example of a field of view within a finder. [Figure 5] FIG. 2 is a diagram illustrating an example of an optical system for detecting a line of sight. [Figure 6] FIG. 2 is a diagram illustrating an example of an eye image and its luminance information. [Figure 7] 10 is a flowchart illustrating an example of a gaze detection operation. [Figure 8] 10 is a flowchart showing an example of a main subject selection operation. [Figure 9] 10 is a flowchart illustrating an example of a priority determination operation based on type information. [Figure 10] 10 is a flowchart illustrating an example of a priority determination operation based on identification information. [Figure 11] FIG. 4 is a diagram showing an example of subject information stored in a memory unit. [Figure 12]FIG. 10 is a diagram for explaining a specific example of the operation of the camera. [Figure 13] FIG. 10 is an external view of another electronic device. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0014] <Configuration example> 1(A) and 1(B) show an example of the appearance of a camera 1 (digital still camera; interchangeable lens camera) according to this embodiment. FIG. 1(A) is a front perspective view, and FIG. 1(B) is a rear perspective view. As shown in FIG. 1(A), the camera 1 has a photographing lens unit 1A and a camera housing 1B. A release button 5, which is an operation member (operation receiving unit) that can receive photographing operations from a user (photographer), is disposed on the camera housing 1B. As shown in FIG. 1(B), an eyepiece lens 12 (eyepiece optical system) is disposed on the rear of the camera housing 1B, through which the user peers to view a display device 10 (display panel) (described below) contained within the camera housing 1B. The eyepiece optical system may include multiple lenses. Operation members 41 to 43 that can receive various operations from the user are also disposed on the rear of the camera housing 1B. For example, operation member 41 is a touch panel that can accept touch operations, operation member 42 is an operation lever that can be pushed down in each direction, and operation member 43 is a four-way key that can be pushed in each of four directions. Operation member 41 (touch panel) is equipped with a display panel such as a liquid crystal panel, and has the function of displaying images on the display panel.

[0015] FIG. 2 is a cross-sectional view of the camera 1 taken along the YZ plane defined by the Y axis and Z axis shown in FIG. 1(A), and shows an example of the internal configuration of the camera 1. In FIG.

[0016] The photographing lens unit 1A includes two lenses 101 and 102, an aperture 111, an aperture driver 112, a lens drive motor 113, a lens drive member 114, a photocoupler 115, a pulse plate 116, a mount contact 117, and a focus adjustment circuit 118. The lens drive member 114 is comprised of a drive gear and other components, and the photocoupler 115 detects the rotation of the pulse plate 116, which is linked to the lens drive member 114, and transmits this information to the focus adjustment circuit 118. The focus adjustment circuit 118 drives the lens drive motor 113 based on information from the photocoupler 115 and information from the camera housing 1B (lens drive amount information), thereby moving the lens 101 and changing the focus position. The mount contact 117 is an interface between the photographing lens unit 1A and the camera housing 1B. Note that for simplicity, two lenses 101 and 102 are shown, but in reality, more than two lenses are included in the photographing lens unit 1A.

[0017] The camera housing 1B contains an image sensor 2, a CPU 3, a memory unit 4, a display device 10, a display device drive circuit 11, and the like. The image sensor 2 is located at the intended imaging plane of the photographing lens unit 1A. The CPU 3 is the central processing unit of the microcomputer, and controls the entire camera 1. The memory unit 4 stores images captured by the image sensor 2, etc. The display device 10 is composed of a liquid crystal or the like, and displays the captured image (subject image), etc. on the display surface of the display device 10. The display device drive circuit 11 drives the display device 10. A user can view the image (such as the image captured by the image sensor 2) displayed on the display surface of the display device 10 through an eyepiece 12.

[0018] The camera housing 1B also contains light sources 13a and 13b, a beam splitter 15, a light-receiving lens 16, and an ocular imaging element 17. The light sources 13a and 13b are light sources for illuminating the user's eyeball 14. The light sources 13a and 13b have been used in single-lens reflex cameras and the like to detect the gaze direction (the direction in which the user is looking) from the relationship between the pupil and the image of light reflected from the cornea (corneal reflection image; Purkinje image). Specifically, the light sources 13a and 13b are infrared light-emitting diodes or the like that emit infrared light that is insensitive to the user, and are arranged around the eyepiece 12. The optical image of the illuminated eyeball 14 (the eyeball image; the image formed by light emitted from the light sources 13a and 13b and reflected by the eyeball 14) passes through the eyepiece 12 and is reflected by the beam splitter 15. The eyeball image is then formed by light receiving lens 16 on ocular imaging element 17, which is a two-dimensional array of photoelectric elements such as CCD or CMOS. Light receiving lens 16 positions the pupil of eyeball 14 and ocular imaging element 17 in a conjugate imaging relationship. Using a predetermined algorithm described later, the gaze direction of eyeball 14 is detected from the positional relationship between the pupil and the corneal reflection image in the eyeball image formed on ocular imaging element 17.

[0019] 3 is a block diagram showing an example of the electrical configuration within camera 1. CPU 3 is connected to a gaze detection circuit 201, an image processing circuit 202, an autofocus detection circuit 203, a signal input circuit 204, a display device drive circuit 11, a light source drive circuit 205, and the like. CPU 3 also transmits signals to a focus adjustment circuit 118 disposed within photographing lens unit 1A and to an aperture control circuit 206 included in an aperture drive unit 112 within photographing lens unit 1A via mount contacts 117. Memory unit 4 associated with CPU 3 has the function of storing image capture signals from image sensor 2 and eye image sensor 17, and the function of storing gaze correction parameters that correct for individual differences in gaze, as described below.

[0020] The gaze detection circuit 201 A / D converts the output of the eye imaging element 17 (eye image obtained by capturing an image of the eye) when an eyeball image is formed on the eye imaging element 17, and transmits the result to the CPU 3. The CPU 3 extracts feature points required for gaze detection from the eye image according to a predetermined algorithm described later, and calculates the user's viewpoint (gaze position; position at which the gaze is fixed; position at which the user is looking) on ​​the display surface of the display device 10 from the positions of the feature points.

[0021] The image processing circuit 202 performs various processes based on the image captured by the image sensor 2. For example, the image processing circuit 202 performs amplification, logarithmic compression, A / D conversion, etc. on a signal obtained from the image sensor 2, which also functions as a photometric sensor; specifically, a luminance signal corresponding to the brightness of the subject field, and sends the resulting field luminance information to the CPU 3. The field luminance information is information indicating the luminance of each pixel in the captured image. The image processing circuit 202 also detects a specific subject (specific object) from the image captured by the image sensor 2, and sends the resulting subject information and subject area information to the CPU 3. The subject information (object information) is information related to the detected subject, and the subject area information (object area information) is information indicating the area of ​​the detected subject. The subject information and subject area information can also be obtained by analyzing the field luminance information.

[0022] The autofocus detection circuit 203 A / D converts signal voltages from multiple detection elements (multiple pixels) included in the image sensor 2 and used for phase difference detection, and sends the converted signal to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point from the signals from the multiple detection elements. This is a well-known technique known as image plane phase difference AF. In this embodiment, as an example, it is assumed that there are focus detection points at each of the 180 locations on the image plane corresponding to the field of view within the viewfinder (the field of view when looking through the viewfinder) in FIG. 4, specifically the 180 locations on the image plane corresponding to the 180 locations shown on the display surface of the display device 10.

[0023] Switches SW1 and SW2 are connected to the signal input circuit 204. Switch SW1 is turned on by the first stroke (half-press) of the release button 5, and is used to start photometry and focus detection (distance measurement) of the camera 1. Switch SW2 is turned on by the second stroke (full press) of the release button 5, and is used to start a photographing operation. The ON signals from switches SW1 and SW2 are input to the signal input circuit 204 and transmitted to the CPU 3. Photometry and focus detection are performed within a set photometry area or focus detection area, or within the area of ​​a selected (determined) main subject. For example, in photometry, the image processing circuit 202 determines the brightness of the target area from field brightness information, and in focus detection, the autofocus detection circuit 203 determines the distance to a subject present in the target area.

[0024] The light source drive circuit 205 drives the light sources 13a and 13b.

[0025] 4 is a diagram showing an example of the field of view within the viewfinder, and shows a state in which the display device 10 is operating (a state in which an image is displayed). As shown in FIG. 4, the field of view within the viewfinder includes a focus detection area 400, 180 ranging point indices 401, a field of view mask 402, and the like. Each of the 180 ranging point indices 401 is displayed superimposed on a through image (live view image) displayed on the display device 10 so as to be displayed at a position corresponding to a focus detection point on the imaging surface. Furthermore, of the 180 ranging point indices 401, the ranging point indices 401 that corresponds to the current viewpoint A (estimated position) is displayed highlighted with a frame or the like.

[0026] <Example of gaze detection operation> The gaze detection method will be described using Figures 5, 6(A), 6(B), and 7. Figure 5 is a diagram for explaining the principle of the gaze detection method, and shows an example of an optical system for performing gaze detection. As shown in Figure 5, light sources 13a and 13b are arranged approximately symmetrically with respect to the optical axis of light receiving lens 16 and illuminate user's eyeball 14. A portion of the light emitted from light sources 13a and 13b and reflected by eyeball 14 is collected by light receiving lens 16 onto ocular imaging element 17. Figure 6(A) is a diagram showing an example of an eye image captured by ocular imaging element 17 (eyeball image projected onto ocular imaging element 17), and Figure 6(B) is a diagram showing an example of the output intensity of ocular imaging element 17. Figure 7 is a flowchart showing an example of gaze detection operation.

[0027] The gaze detection operation starts, for example, when the camera 1 is activated, and is repeated. When the gaze detection operation starts, in step S701 of FIG. 7, the light sources 13a and 13b emit infrared light toward the user's eyeball 14. The image of the user's eyeball illuminated by the infrared light is received by the light receiving lens 16. The light passes through 16 and is focused onto the ocular image sensor 17, where it is photoelectrically converted, thereby obtaining an electrical signal of the eye image that can be processed.

[0028] In step S702, the line-of-sight detection circuit 201 sends the eye image (eye image signal; electric signal of the eye image) obtained from the eye imaging device 17 to the CPU 3.

[0029] In step S703, the CPU 3 detects the coordinates of the corneal reflection images Pd and Pe of the light sources 13a and 13b and the point corresponding to the pupil center c from the eye image obtained in step S702.

[0030] Infrared light emitted from light sources 13a and 13b illuminates cornea 142 of user's eyeball 14. At this time, corneal reflection images Pd and Pe formed by part of the infrared light reflected from the surface of cornea 142 are collected by light receiving lens 16 and formed on ocular imaging element 17 as corneal reflection images Pd' and Pe' in the eye image. Similarly, light beams from edges a and b of pupil 141 are also formed on ocular imaging element 17 as pupil edge images a' and b' in the eye image.

[0031] FIG. 6B shows luminance information (luminance distribution) of region α in the eye image of FIG. 6A. In FIG. 6B, the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In this embodiment, the X-axis (horizontal) coordinates of corneal reflection images Pd' and Pe' are defined as Xd and Xe, and the X-axis coordinates of pupil edge images a' and b' are defined as Xa and Xb. As shown in FIG. 6B, an extremely high level of luminance is obtained at the X-coordinates Xd and Xe of corneal reflection images Pd' and Pe'. In the region from X-coordinate Xa to X-coordinate Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained when the light beam from the pupil 141 is focused on the ocular imaging element 17), an extremely low level of luminance is obtained, except for the X-coordinates Xd and Xe. A luminance intermediate between the two types of luminance is obtained in the region of iris 143 outside pupil 141 (the region of the iris image outside the pupil image obtained by focusing the light beam from iris 143). Specifically, a luminance intermediate between the two types of luminance is obtained in a region where the X coordinate (coordinate in the X-axis direction) is smaller than X coordinate Xa and a region where the X coordinate is larger than X coordinate Xb.

[0032] From the luminance distribution shown in FIG. 6B, the X-coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X-coordinates Xa and Xb of the pupil edge images a' and b' can be obtained. Specifically, the coordinates of the corneal reflection images Pd' and Pe' can be obtained as the coordinates of extremely high luminance, and the coordinates of the pupil edge images a' and b' can be obtained as the coordinates of extremely low luminance. Furthermore, when the rotation angle θx of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16 is small, the X-coordinate Xc of the pupil center image c' (center of the pupil image) obtained when the light beam from the pupil center c is focused on the ocular imaging element 17 can be expressed as Xc ≒ (Xa + Xb) / 2. In other words, the X-coordinate Xc of the pupil center image c' can be calculated from the X-coordinates Xa and Xb of the pupil edge images a' and b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupil center image c' can be estimated.

[0033] In step S704, CPU 3 calculates the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position of eyeball 14 relative to light receiving lens 16, and can be calculated using a function of the distance (Xd-Xe) between corneal reflection images Pd' and Pe'.

[0034] In step S705, CPU 3 calculates the rotation angle of the optical axis of eyeball 14 relative to the optical axis of light receiving lens 16 as information on the line of sight of eyeball 14. The X coordinate of the midpoint between corneal reflection images Pd and Pe and the X coordinate of the center of curvature O of cornea 142 approximately coincide. Therefore, if the standard distance from the center of curvature O of cornea 142 to the center c of pupil 141 is Oc, then the rotation angle θx of eyeball 14 in the ZX plane (plane perpendicular to the Y axis) can be calculated using the following equation 1. The rotation angle θy of eyeball 14 in the ZY plane (plane perpendicular to the X axis) can also be calculated using a method similar to that for calculating rotation angle θx. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc (Formula 1)

[0035] In step S706, CPU 3 uses the rotation angles θx, θy calculated in step S705 to estimate the user's viewpoint on the display surface of display device 10. If the viewpoint coordinates (Hx, Hy) are coordinates corresponding to the pupil center c, the viewpoint coordinates (Hx, Hy) can be calculated by the following equations 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0036] Parameter m in equations 2 and 3 is a constant determined by the configuration of the viewfinder optical system (light receiving lens 16, etc.) of camera 1, and is a conversion coefficient that converts rotation angles θx and θy into coordinates corresponding to the pupil center c on the display surface of display device 10. Parameter m is determined in advance and stored in memory unit 4. Parameters Ax, Bx, Ay, and By are gaze correction parameters that correct for individual differences in gaze, and are acquired by performing a calibration operation. Parameters Ax, Bx, Ay, and By are stored in memory unit 4 before the gaze detection operation begins.

[0037] In step S707, CPU 3 stores the coordinates (Hx, Hy) of the viewpoint in memory unit 4 in association with the time when the eye image was acquired, and ends the gaze detection operation.

[0038] Although an example using a corneal reflection image has been described, the method of gaze detection is not limited to this, and any method of detecting the gaze based on an eye image may be used. Also, although an example in which the coordinates (Hx, Hy) of the viewpoint are obtained as the result (final result) of gaze detection has been described, any information related to the gaze (gaze information), such as rotation angles θx, θy, may be obtained as the result of gaze detection.

[0039] <Example of main subject selection operation> The main subject selection operation will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the main subject selection operation. The main subject selection operation starts, for example, when camera 1 is started up, and is repeated. The main subject selection operation includes a gaze detection operation. When a main subject (target object) is selected by the main subject selection operation, camera 1 performs focus adjustment (including focus detection) and exposure adjustment (including metering) on ​​the main subject.

[0040] When the main subject selection operation starts, in step S801, CPU 3 detects the user's viewpoint on the display surface of display device 10 by the above-described line-of-sight detection operation. When an image captured by imaging element 2 is displayed on the display surface of display device 10, the user's viewpoint on the display surface of display device 10 can also be considered as the user's viewpoint in the image captured by imaging element 2.

[0041] In step S802, the CPU 3 controls the image processing circuit 202 to acquire subject information and subject area information, which are the results of detecting a specific subject from an image captured by the image sensor 2. The specific subject is, for example, a specific type of subject, such as a person, an animal, or a vehicle. Individual subjects of a specific type may be treated as specific subjects without distinction, or individual subjects of a specific type may be distinguished and treated as specific subjects. The subject information is information related to the detected subject, such as type information indicating the type of the detected subject and identification information (e.g., an identifier) ​​for identifying the detected subject. The subject detection method is not particularly limited, and for example, a trained model obtained by learning using an image of a specific subject may be used. A specific subject may be detected based on the color, shape, etc. of the specific subject using the above method.

[0042] In step S803, CPU 3 determines whether or not the user has performed a viewpoint confirmation operation (specific operation). If CPU 3 determines that a viewpoint confirmation operation has been performed, it proceeds to step S806, and if it determines that a viewpoint confirmation operation has not been performed, it proceeds to step S804. The viewpoint confirmation operation is an operation for confirming the viewpoint, and is an operation for communicating the user's intended viewpoint to camera 1. The viewpoint confirmation operation is performed using, for example, operation members 41 to 43. The viewpoint confirmation operation may be the first stroke (half-press) of release button 5. In that case, after the processing of step S806, photometry, focus detection (distance measurement), etc. of camera 1 begin.

[0043] In step S804, the CPU 3 executes a priority determination operation. The priority determination operation is an operation for determining the priority of the subject detected in step S802. The priority determination operation will be described later with reference to FIG.

[0044] In step S805, CPU 3 selects, from among the subjects detected in step S802, the subject with the highest priority determined in step S804 as the main subject.

[0045] In step S806, CPU 3 selects the subject the user is looking at as the main subject based on the viewpoint detected in step S801. Specifically, CPU 3 selects, from the subjects detected in step S802, a subject that exists near the viewpoint (within a predetermined range from the viewpoint) as the main subject. If there are multiple subjects near the viewpoint, it is preferable to select the subject closest to the viewpoint as the main subject. If the subject detected in step S802 is not near the viewpoint, CPU 3 controls image processing circuit 202 to detect the subject that exists at the viewpoint (a subject not detected in step S802) as the main subject. For example, CPU 3 detects, as the main subject, an area consisting of the pixel at the viewpoint and pixels having a color similar to that of the pixel at the viewpoint. The predetermined range defining the vicinity of the viewpoint may be a fixed range or may be a range that can be set (changed) by the user using operation members 41 to 43, etc. The predetermined range may be automatically switched in response to changes in the angle of view, i.e., the focal length of photographing lens unit 1A.

[0046] In step S807, CPU 3 records the subject information of the main subject selected in step S806 in memory unit 4 (storage unit) in association with the current time. Here, the subject information may be type information indicating the type of the main subject. The type in the type information may be, for example, "person" or "animal." "Animals" may be further subdivided into "dog," "cat," "bird," etc. The subject information may be identification information for identifying the main subject, such as "person A," "person B," or "dog C." Furthermore, if the main subject is selected when the subject detected in step S802 is not present near the viewpoint, information such as "other" may be recorded as the subject information, or nothing may be stored.

[0047] <Example of priority determination operation> The priority determination operation (step S804 in FIG. 8) when the subject information is type information will be described with reference to FIG. 9. FIG. 9 is a flowchart showing an example of the priority determination operation when the subject information is type information.

[0048] When the priority determination operation starts, in step S901, the CPU 3 initializes the priorities of all the subjects detected in step S802 of FIG. 8 to default values.

[0049] In step S902, the CPU 3 deletes the object information that has been recorded for a predetermined time or more from the past object information stored in the memory unit 4. Memory unit 4 will only store subject information recorded up to a predetermined time ago. Here, it is preferable that the predetermined time is not too long, for example, about several minutes to a day. Since a user does not necessarily continue to photograph the same type of subject, if too old subject information remains, there is a risk that when the user photographs a different type of subject, the main subject will not match the user's intention. If the predetermined time is set appropriately, the main subject can match the user's intention even when the user photographs a different type of subject. The predetermined time may be a fixed time, or may be a time that the user can set (change) using operation members 41 to 43, etc.

[0050] The CPU 3 may delete subject information so that the memory unit 4 stores no more than a predetermined number of pieces of subject information. As with the predetermined time period described above, it is preferable that the predetermined number is not too large. The predetermined number may be a fixed number, or may be a number that can be set (changed) by the user using the operation members 41 to 43, etc. The CPU 3 may delete all subject information stored in the memory unit 4 in response to a power-off operation. The CPU 3 may selectively or collectively delete subject information stored in the memory unit 4 in response to an operation performed by the user using the operation members 41 to 43, etc.

[0051] In step S903, CPU 3 performs settings for performing the processes of steps S904 to S909 for each subject type indicated by the subject information stored in memory unit 4. That is, CPU 3 performs settings for repeatedly performing the processes of steps S904 to S909 the number of times corresponding to the subject types indicated by the subject information stored in memory unit 4.

[0052] In step S904, CPU 3 determines whether the number of object information of object type i (processing target) stored in memory unit 4 is equal to or greater than a predetermined number. If it is determined that the number of object information of object type i stored in memory unit 4 is equal to or greater than the predetermined number, CPU 3 proceeds to step S905. If it is determined that the number of object information of object type i stored in memory unit 4 is less than the predetermined number, CPU 3 proceeds to step S910.

[0053] In step S905, CPU 3 increases the priority of subject type i. At this time, CPU 3 may increase the priority by a predetermined amount of change that is independent of the number of subject information items of subject type i stored in memory unit 4. CPU 3 may increase the priority by a larger amount of change as the number of subject information items of subject type i stored in memory unit 4 increases.

[0054] In step S906, CPU 3 determines whether the subject type for which memory unit 4 stores the most subject information is subject type i. If it is determined that the subject type for which memory unit 4 stores the most subject information is subject type i, CPU 3 proceeds to step S907. If it is determined that the subject type for which memory unit 4 stores the most subject information is not subject type i, CPU 3 proceeds to step S908.

[0055] In step S907, the CPU 3 increases the priority of the object type i.

[0056] In step S908, CPU 3 determines whether or not subject information of subject type i has been recorded in memory unit 4 within a predetermined time from the present. If it is determined that subject information of subject type i has been recorded in memory unit 4 within a predetermined time from the present, CPU 3 proceeds to step S909. If it is determined that subject information of subject type i has not been recorded in memory unit 4 within a predetermined time from the present, CPU 3 proceeds to step S910.

[0057] In step S909, the CPU 3 increases the priority of the object type i. The PU 3 may increase the priority by a predetermined amount of change that is independent of the time when the subject information of the subject type i was last recorded in the memory unit 4. The CPU 3 may increase the priority by a larger amount of change as the time when the subject information of the subject type i was last recorded in the memory unit 4 becomes closer to the present.

[0058] The CPU 3 may increase the priority of the subject type i when there is subject information of the subject type i that is within a predetermined rank when counting from the time closest to the present when it was recorded in the memory unit 4. In other words, the CPU 3 may increase the priority of the subject type i when there is subject information of the subject type i among a predetermined number of subject information of the subject type i that is recorded in the memory unit 4 from the time closest to the present.

[0059] In step S910, CPU 3 determines whether or not the processes of steps S904 to S909 have been performed for all subject types indicated by the subject information stored in memory unit 4. If it is determined that the processes of steps S904 to S909 have been performed for all subject types, CPU 3 proceeds to step S911. If it is determined that there is a subject type for which the processes of steps S904 to S909 have not been performed, CPU 3 updates subject type i and proceeds to step S904.

[0060] When proceeding to step S911, CPU 3 applies the priority of the subject type determined in steps S904 to S909 to subjects of that subject type to all subjects detected in step S802 of Fig. 8. For example, CPU 3 applies the priority of the subject type "animal" determined in steps S904 to S909 to animals detected in step S802.

[0061] In step S911, CPU 3 initializes the priority of subjects detected in step S802 that exist outside a predetermined range from the viewpoint detected in step S801 to a default value. This allows the user's intention to be more accurately reflected in the subject priority. Specifically, the change in priority based on the subject information stored in memory unit 4 (application of the priority determined in steps S904 to S909) is effective only for subjects that exist within a predetermined range from the viewpoint detected in step S801. As a result, the priority of subjects that exist near the current viewpoint and are of the same type as the main subject selected in response to a previous viewpoint confirmation operation (the subject that existed near the viewpoint at the time of the previous viewpoint confirmation operation) becomes higher than the default value. Here, the predetermined range may be a fixed range or may be a range that the user can set (change) using operation members 41 to 43, etc. The predetermined range may be automatically switched in response to changes in the angle of view, i.e., the focal length of photographing lens unit 1A.

[0062] In step S912, CPU 3 increases the priority of subjects detected in step S802 that exist within a predetermined range from the viewpoint detected in step S801 (near the viewpoint). At this time, CPU 3 may increase the priority of the subject by a predetermined amount that is independent of the distance between the subject and the viewpoint. CPU 3 may increase the priority of the subject by a larger amount the closer the subject is to the viewpoint. The predetermined range may be a fixed range, or may be a range that can be set (changed) by the user using operation members 41 to 43, etc. The predetermined range may be automatically switched in response to changes in the angle of view, i.e., the focal length of photographing lens unit 1A.

[0063] In step S913, the CPU 3 increases the priority of the subject detected in step S802 that exists within a predetermined range from the center of the image captured by the image sensor 2. At this time, the CPU 3 may increase the priority of the subject by a predetermined amount that does not depend on the distance between the subject and the center of the image. The CPU 3 increases the priority of the subject by a predetermined amount that does not depend on the distance between the subject and the center of the image. The priority of the subject may be increased by a certain amount of change. The predetermined range may be a fixed range, or may be a range that can be set (changed) by the user using operation members 41 to 43, etc. The predetermined range may be automatically switched according to changes in the angle of view, i.e., the focal length of photographing lens unit 1A.

[0064] In step S914, CPU 3 increases the priority of any of the subjects detected in step S802 whose image size (size on the image captured by image sensor 2) is equal to or larger than a predetermined size. At this time, CPU 3 may increase the priority of the subject by a predetermined amount independent of the image size. CPU 3 may increase the priority of the subject by a larger amount as the image size increases. The predetermined size may be a fixed size, or may be a size that can be set (changed) by the user using operation members 41 to 43, etc. The predetermined size may be automatically changed in accordance with changes in the angle of view, i.e., the focal length of photographing lens unit 1A.

[0065] In steps S904 to S909, the priority is increased based on past subject information stored in memory unit 4, but in steps S911 to S914, the priority is increased based on the current viewpoint, subject position, subject size, and the like.

[0066] The priority determination operation (step S804 in FIG. 8) when the subject information is identification information will be described with reference to FIG. 10. FIG. 10 is a flowchart showing an example of the priority determination operation when the subject information is identification information.

[0067] Steps S1001 and S1002 are similar to steps S901 and S902 in FIG.

[0068] In step S1003, the CPU 3 performs settings for performing the processes of steps S1004 to S1009 for each subject identified by the subject information stored in the memory unit 4. In other words, the CPU 3 performs settings for repeatedly performing the processes of steps S1004 to S1009 for the number of subjects identified by the subject information stored in the memory unit 4.

[0069] In step S1004, CPU 3 determines whether the number of pieces of subject information for subject i (processing target) stored in memory unit 4 is equal to or greater than a predetermined number. If it is determined that the number of pieces of subject information for subject i stored in memory unit 4 is equal to or greater than the predetermined number, CPU 3 proceeds to step S1005. If it is determined that the number of pieces of subject information for subject i stored in memory unit 4 is less than the predetermined number, CPU 3 proceeds to step S1010.

[0070] In step S1005, CPU 3 increases the priority of subject i. At this time, CPU 3 may increase the priority by a predetermined amount of change that is independent of the amount of subject information for subject i stored in memory unit 4. CPU 3 may increase the priority by a larger amount of change as the amount of subject information for subject i stored in memory unit 4 increases.

[0071] In step S1006, CPU 3 determines whether or not the subject for which memory unit 4 stores the most subject information is subject i. If it is determined that the subject for which memory unit 4 stores the most subject information is subject i, CPU 3 proceeds to step S1007. If it is determined that the subject for which memory unit 4 stores the most subject information is not subject i, CPU 3 proceeds to step S1008.

[0072] In step S1007, the CPU 3 increases the priority of the subject i.

[0073] In step S1008, the CPU 3 calculates the object information of the object i from the present time up to a predetermined time ago. If it is determined that the subject information of subject i has been recorded in memory unit 4 within the predetermined time from the present, CPU 3 proceeds to step S1009. If it is determined that the subject information of subject i has not been recorded in memory unit 4 within the predetermined time from the present, CPU 3 proceeds to step S1010.

[0074] In step S1009, CPU 3 increases the priority of subject i. At this time, CPU 3 may increase the priority by a predetermined amount of change that is independent of the time when the subject information of subject i was last recorded in memory unit 4. CPU 3 may increase the priority by a larger amount of change as the time when the subject information of subject i was last recorded in memory unit 4 is closer to the present.

[0075] The CPU 3 may increase the priority of the subject i when there is subject information of the subject i whose time of recording in the memory unit 4 is within a predetermined rank, counting from the time closest to the present. In other words, the CPU 3 may increase the priority of the subject i when there is subject information of the subject i among a predetermined number of subject information whose time of recording in the memory unit 4 is counted from the time closest to the present.

[0076] In step S1010, CPU 3 determines whether or not the processes of steps S1004 to S1009 have been performed for all of the subjects indicated by the subject information stored in memory unit 4. If it is determined that the processes of steps S1004 to S1009 have been performed for all of the subjects, CPU 3 proceeds to step S1011. If it is determined that there is a subject for which the processes of steps S1004 to S1009 have not been performed, CPU 3 updates subject i and proceeds to step S1004.

[0077] 9, the priority of the subject type was determined, and therefore, when proceeding to step S911, a process of applying the priority of the subject type to the subject of that subject type was necessary. In steps S1004 to S1009, the priority of each subject, such as person A, person B, dog C, etc., is determined, and therefore, such a process is not necessary.

[0078] Steps S1011 to S1014 are the same as steps S1011 to S1014 in FIG.

[0079] The number of possible priorities (number of priority levels) is not particularly limited. A score may be calculated as the priority. An upper limit may be set for the priority, and the priority may not be set to a value higher than the upper limit. Furthermore, the amount of change in priority may be different for each judgment condition.

[0080] 9 and 10 may be modified as appropriate. For example, although the process proceeds to step S910 if the determination condition in step S904 in Fig. 9 is not satisfied, the process may proceed to step S906. The same applies to the case where the determination condition in step S1004 in Fig. 10 is not satisfied. At least one of the determination conditions shown in Figs. 9 and 10 may be omitted, or another determination condition based on past subject information stored in memory unit 4 may be added.

[0081] <Example of subject information stored in the memory unit> 11(A) to 11(D) are diagrams showing examples of subject information stored in memory unit 4. FIGS. 11(A) and 11(B) show an example where the subject information is type information, and FIGS. 11(C) and 11(D) show an example where the subject information is identification information. For simplicity of explanation, it is assumed here that the maximum number (upper limit number) of subject information stored in memory unit 4 is 7. The predetermined number in step S904 of FIG. 9 and step S1004 of FIG. 10 is 3, and the memory unit 4 stores three or more pieces of subject information for a subject type or subject with high priority. 11(A) to 11(D), object types or objects for which the memory unit 4 stores three or more pieces of object information are shown in gray. Furthermore, for times a1 to a7, b1 to b7, c1 to c7, and d1 to d7, the smaller the number, the closer the time to the present. Times a1, a2, b1, b2, c1, c2, d1, and d2 are times from the present to a predetermined time before step S908 and step S1008, and times a3 to a7, b3 to b7, c3 to c7, and d3 to d7 are times that are more than a predetermined time before the present. The priority is increased by one. Furthermore, only the object information stored in the memory unit 4 is considered, and the current viewpoint, object position, object size, etc. are not considered. That is, only steps S903 to S910 and steps S1003 to S1010 are considered, and steps S911 to S914 and steps S1011 to S1014 are not considered.

[0082] In FIG. 11(A), people is the only object type for which memory unit 4 stores three or more pieces of object information, so 1 is added to the priority of people (step S905). Also, because memory unit 4 stores the most object information of people, 1 is added to the priority of people (step S907). Furthermore, because the time a1 when object information of people was last recorded in memory unit 4 is a time from the present to a predetermined time ago, 1 is added to the priority of people (step S909). As a result, the priority of people becomes 3, and the priorities of the others become 0.

[0083] In FIG. 11(B), the object types for which memory unit 4 stores three or more pieces of object information are people and animals, so 1 is added to each of the priority of people and the priority of animals (step S905). Also, the object type for which memory unit 4 stores the most object information is people, so 1 is added to the priority of people (step S907). Furthermore, the time b1 at which object information for people was last recorded in memory unit 4 is a time a predetermined time before the present, so 1 is added to the priority of people (step S909). On the other hand, the time b3 at which object information for animals was last recorded in memory unit 4 is a time more than a predetermined time before the present, so the priority of animals is not increased in step S909. As a result, the priority of people becomes 3, the priority of animals becomes 1, and the other priorities become 0.

[0084] In FIG. 11(C), person A is the only subject for which memory unit 4 stores three or more pieces of subject information, so 1 is added to the priority of person A (step S1005). Also, because memory unit 4 stores the most subject information of person A, 1 is added to the priority of person A (step S1007). Furthermore, because the time c1 at which subject information of person A was last recorded in memory unit 4 is a time from the present to a predetermined time ago, 1 is added to the priority of person A (step S1009). As a result, the priority of person A becomes 3, and the priorities of the others become 0.

[0085] 11(D), there is no subject for which the memory unit 4 stores three or more pieces of subject information. As a result, the priority of each subject is 0. In other words, there is no difference in priority between subjects based on the subject information stored in the memory unit 4.

[0086] <Example of operation> A specific example of the operation of the camera 1 will be described with reference to FIG.

[0087] In the state (a) of FIG. 12, a subject 12A (person) and a subject 12B (dog) are captured in an image captured by the image sensor 2, and the subjects 12A and 12B are detected. Here, it is assumed that the user wants to select the subject 12B as the main subject. However, in a typical center priority or size priority, the subject 12A, which is close to the center of the image and is large in size on the image, is selected as the main subject, and a main subject frame (selection frame) is displayed around the subject 12A. It ends up like this.

[0088] State (b) is the state after the user performs a viewpoint confirmation operation while looking at subject 12B, and subject 12B located near the viewpoint is selected as the main subject, and a main subject frame is displayed around subject 12B.

[0089] State (c) is the state after the user has changed the orientation of camera 1, for example. Subjects 12A and 12B are not visible in the image, and only subject 12C (flower) is visible. Here, it is assumed that camera 1 cannot detect any plants (obviously, neither subject 12A nor subject 12B is detected). Therefore, the main subject is not selected (the state in which subject 12B was selected as the main subject is canceled), and the main subject frame is not displayed.

[0090] State (d) is the state after the user changes the image composition to the same composition as state (a) by, for example, changing the orientation of camera 1. As in state (a), subject 12A is selected as the main subject, and a main subject frame is displayed around subject 12A.

[0091] In a conventional camera, no matter how many times the operations in states (b) to (d) are repeated, in the composition of states (a) and (d), subject 12A is selected as the main subject before the viewpoint confirmation operation is performed.

[0092] On the other hand, in camera 1 according to this embodiment, each time the user performs a viewpoint confirmation operation in state (b), subject information related to subject 12B is recorded in memory unit 4, and the priority of subject 12B is increased. Then, after performing the operations of states (b) to (d) a predetermined number of times or more, after changing the composition from state (c) to the same composition as state (a), there is no transition to state (d), and subject 12B is selected as the main subject even before the viewpoint confirmation operation is performed, as in state (e).

[0093] <Summary> As described above, according to this embodiment, in response to a viewpoint confirmation operation, the subject the user is looking at is selected as the main subject, and subject information about the main subject is recorded in the memory unit. If a viewpoint confirmation operation is not performed, the main subject is selected based on the subject information stored in the memory unit. For example, if a viewpoint confirmation operation is not performed, the subject whose subject information is stored in the memory unit is selected as the main subject. This makes it possible to prevent a subject unintended by the user from being selected as the main subject, and to select a main subject that matches the user's intention.

[0094] The above-described embodiment is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configuration of the above-described embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0095] For example, although an example of viewing a subject through an EVF (electronic viewfinder) has been described, the present invention can also be applied to viewing a subject through an OVF (optical viewfinder). The present invention can be applied to devices other than imaging devices (cameras), such as viewing three-dimensional computer graphics. The present invention can be applied to various electronic devices that can detect the viewpoint.

[0096] <Examples of application to other electronic devices> 13A is an external view of a notebook personal computer 1310 (notebook PC). In FIG. 13A, an imaging unit 1315 for capturing an image of a user looking at a display unit 1311 of the notebook PC 1310 is connected to the notebook PC 1310. The image capture unit 1315 captures the captured image. The laptop PC 1310 then detects the user's viewpoint based on the captured image. The present invention is also applicable to the laptop PC 1310.

[0097] Fig. 13(B) is an external view of a smartphone 1320. In Fig. 13(B), the smartphone 1320 detects the viewpoint of a user looking at a display unit 1322 of the smartphone 1320 based on the imaging result of an in-camera 1321 (front camera). The present invention is also applicable to the smartphone 1320. Similarly, the present invention is also applicable to various tablet terminals.

[0098] FIG. 13(C) is an external view of a game console 1330. In FIG. 13(C), a head-mounted display 1335 (HMD) that displays a VR (Virtual Reality) image of a game on a display unit 1336 is connected to the game console 1330. The HMD 1335 has a camera 1337 that captures an image of the eyes of a user wearing the HMD 1335, and the game console 1330 acquires the captured image from the HMD 1335. The game console 1330 then detects the user's viewpoint based on the captured image. The present invention is also applicable to the game console 1330. The components of the game console 1330 may be built into the HMD 1335. Therefore, the present invention is also applicable to the HMD 1335. Just as the present invention is applicable to viewing a VR image displayed on an HMD, the present invention is also applicable to viewing an AR (Augmented Reality) image displayed on the lens portion of a glasses-type wearable terminal or a real object beyond the lens portion. Just as the present invention is applicable to VR technology and AR technology, the present invention is also applicable to other xR technologies such as MR (Mixed Reality) technology and SR (Substitutional Reality) technology.

[0099] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0100] 1: Camera 3: CPU 5: Release button 201: Gaze detection circuit 202: Image processing circuit

Claims

1. A first detection means for detecting a gaze position, which is the position at which a user is looking, on a display surface of a display device; a second detection means for detecting an object from the image displayed on the display device and acquiring a category of the object as object information; a receiving means capable of receiving a specific operation from the user; a selection means for selecting, in response to the specific operation, one object displayed within a predetermined range from the line-of-sight position detected by the first detection means as a target object; a recording means for recording object information of the target object selected by the selecting means in a storage unit in response to the specific operation; and When the specific operation is performed a plurality of times, a plurality of pieces of object information corresponding to the plurality of specific operations are recorded in the storage unit, The selection means, after the specific operation is performed, during a period in which the specific operation is not performed, changing a priority of the object detected by the second detection means from a default value based on the object information stored in the storage unit; The object with the highest priority after the change is selected as the target object. An electronic device characterized by:

2. The object information is a category according to the type of object.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

3. The object information is an individual category for each object.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

4. The selection means increases the priority of an object for which the number of pieces of object information stored in the storage unit is equal to or greater than a first number.

4. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

5. The selection means increases the priority of an object having the largest number of pieces of object information stored in the storage unit.

5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

6. The selecting means increases the priority of an object whose object information has been recorded in the storage unit within a first time period from the present.

6. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

7. The selecting means increases the priority of an object related to object information that is within a predetermined rank when counted from the time closest to the present when recorded in the storage unit.

6. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

8. The storage unit stores object information recorded from the present time until a second time ago.

8. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

9. The storage unit stores a second number or less of object information.

9. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

10. further comprising an imaging means; The second detection means detects an object from the image captured by the imaging means.

10. The electronic device according to claim 1.

11. 11. The electronic device according to claim 10, wherein the selection means assigns a higher priority to an object that exists within a first range from the center of the image.

12. The selection means assigns a higher priority to an object whose size on the image is equal to or larger than a predetermined size.

12. The electronic device according to claim 10 or 11.

13. The selection means increases the priority of an object that exists within a second range from the line-of-sight position detected by the first detection means.

13. The electronic device according to claim 1.

14. The selection means changes the priority based on the object information stored in the storage unit for objects present within a second range from the line-of-sight position detected by the first detection means.

14. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

15. A first detection step of detecting a gaze position, which is the position at which a user is looking, on a display surface of a display device; a second detection step of detecting an object from the image displayed on the display device and acquiring a category of the object as object information; a receiving step of receiving a specific operation from the user; a selection step of selecting, in response to the specific operation, an object displayed within a predetermined range from the line-of-sight position detected in the first detection step, as a target object; a recording step of recording object information of the target object selected in the selection step in a storage unit in response to the specific operation; and When the specific operation is performed a plurality of times, a plurality of pieces of object information corresponding to the plurality of specific operations are recorded in the storage unit, In the selecting step, after the specific operation is performed, During this period, changing a priority of the object detected in the second detection step from a default value based on the object information stored in the storage unit; The object with the highest priority after the change is selected as the target object. A method for controlling an electronic device.

16. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 14.

17. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 14.

Images