Electronic devices, control methods, and programs

Abstract

To provide an electronic apparatus that can improve the responsiveness of operations to select and switch a main subject using detection of a user's viewpoint.SOLUTION: An electronic apparatus of the present disclosure has: detection means that detects a user's viewpoint in an image picked up by imaging means and displayed on display means; tracking means that tracks a plurality of subjects in the image; determination means that, based on the user's viewpoint, determines the user's degree of attention to each of the plurality of subjects; and selection means that selects any one of the plurality of subjects as a main subject. The selection means selects the main subject based on the degree of attention to each of the plurality of subjects.SELECTED DRAWING: Figure 2

Description

Technical Field

[0006] , , , , ,

[0001] The present invention relates to an electronic device, a control method, and a program, and particularly to subject switching control in an electronic device.

Background Art

[0002] In recent years, digital cameras have become more mirrorless, and technologies are used that detect the perspective of a user looking through the finder without manually operating the cross keys, multi-controller, etc., display a perspective pointer based on the position information, and enable the user to specify an arbitrary position.

[0003] Patent Document 1 discloses a technique related to the selection and switching of a main subject that is the target of automatic focus (AF) control using position designation by the line of sight of a photographer who is a user.

[0004] Also, Patent Document 2 discloses a technique for displaying a perspective pointer at an averaged position over time in order to mitigate the influence on the characteristic of the human eye called fixation microsaccades, in which the eye makes fine movements even when fixating on a certain point.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the prior art disclosed in the aforementioned patent documents, the position of the viewpoint pointer is displayed on a time-averaged basis to mitigate the effects of fixation micro-movements, which may impair the responsiveness of viewpoint-based position selection. As a result, in the selection and switching of the main subject, which is the target of autofocus (AF) control using viewpoint-based position selection, there is a possibility of losing shooting opportunities due to timing delays.

[0007] Therefore, the object of the present invention is to provide an electronic device that enables improved responsiveness of the selection and switching operation of the main subject using user viewpoint detection. [Means for solving the problem]

[0008] To achieve the above objective, the electronic device relating to this disclosure includes: detection means for detecting the user's viewpoint in an image captured by an imaging means and displayed on a display means; tracking means for tracking a plurality of subjects in the image; determination means for determining the degree of the user's attention to each of the plurality of subjects based on the user's viewpoint; and selection means for selecting one of the plurality of subjects as the main subject, wherein the selection means Of the aforementioned multiple subjects, the distance from the user's viewpoint in the image is before Note Based on size Subjects that are below the threshold distance, The main subject As a candidate subject to be switched choice To make it possible Includes electronic equipment characterized by doing so. Furthermore, the electronic device relating to this disclosure includes: detection means for detecting the user's viewpoint in an image displayed on a display means; determination means for determining the degree of attention the user is paying to each of a plurality of objects in the image based on the user's viewpoint; and selection means for selecting an object based on the degree of attention each of the plurality of objects is paying to. The selection means then makes available for selection, among the plurality of objects, an object whose distance from the user's viewpoint in the image is less than or equal to the threshold distance based on the level of attention, as a candidate object to be switched. Includes electronic equipment characterized by the following features.

[0009] Furthermore, the control method for electronic equipment according to this disclosure includes a detection step of detecting the user's viewpoint in an image captured by an imaging means and displayed on a display means; a tracking step of tracking a plurality of subjects in the image; a determination step of determining the degree of the user's attention to each of the plurality of subjects based on the user's viewpoint; and a selection step of selecting one of the plurality of subjects as the main subject, wherein the selection step is Of the aforementioned multiple subjects, the distance from the user's viewpoint in the image is before Note Based on size Below the threshold distance The subject to be, The main subject As a candidate subject to be switched choice To make it possible The invention includes a control method characterized by the following: [Effects of the Invention]

[0010] According to this disclosure, responsiveness can be improved by controlling the selection and switching of the main subject for AF control based on the degree of attention given to each subject, which is calculated from the user's viewpoint and the area of ​​the subject recognized by image processing. Furthermore, according to this disclosure, the loss of shooting opportunities can be reduced. [Brief explanation of the drawing]

[0011] [Figure 1] External view of a digital camera according to one embodiment [Figure 2] Block diagram showing the configuration of a digital camera according to one embodiment. [Figure 3] Cross-sectional view of the housing of a digital camera according to one embodiment. [Figure 4] Summary diagram explaining the principle of the gaze detection method. [Figure 5] Schematic diagram of the eyeball image projected onto the image sensor and output intensity diagram at the image sensor. [Figure 6] A flowchart illustrating the gaze detection process in one embodiment. [Figure 7] Flowchart showing the switching control process for the main subject in the first embodiment. [Figure 8] A schematic diagram showing the switching control process of the main subject in the first embodiment. [Figure 9] Schematic diagram showing an example of display of degree of attention in one embodiment [Figure 10] Flowchart showing the switching control process of the main subject in the second embodiment [Figure 11] Schematic diagram showing the switching control process of the main subject in the second embodiment [Figure 12] Flowchart showing the switching control process of the main subject in the third embodiment [Figure 13] Schematic diagram showing the switching control process of the main subject in the third embodiment [Figure 14] Flowchart showing the switching control process of the main subject in the fourth embodiment [Figure 15] Schematic diagram showing the switching control process of the main subject in the fourth embodiment

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the present disclosure is not limited to the following embodiments, and can be appropriately changed without departing from the gist thereof. In the drawings described below, those having the same function are denoted by the same reference numerals, and the description thereof may be omitted or simplified.

[0013] (First Embodiment) (第1の実施形態) Figures 1A and 1B show the external appearance of a digital camera 100, which is an electronic device according to the first embodiment. The digital camera 100 is an example of an imaging device that captures images of a subject. Figure 1A is a front perspective view, and Figure 1B is a rear perspective view. As shown in Figure 1A, the camera 1 has a photographic lens 100A and a housing 100B. A release button 5, which is an operating member that receives imaging operations from the user (photographer), is located in the housing 100B. As shown in Figure 1B, an eyepiece window frame 6 and an eyepiece lens 203 (eyepiece optical system) are located on the back of the camera housing 100B, allowing the user to look into the display unit 113 contained within the camera housing 100B. The eyepiece window frame 6 surrounds the eyepiece lens 203 and protrudes outward (towards the back) relative to the eyepiece lens 203. The eyepiece optical system may include multiple lenses. The rear of the camera housing 100B also features an operation section 115 consisting of operation members 115a, 115b, and 115c that receive various operations from the user. For example, operation member 115a is a touch panel that accepts touch operations, operation member 115b is an operation lever that can be pushed down in each direction, and operation member 115c is a four-way key that can be pressed in each of the four directions. Operation member 115a (touch panel) is equipped with a display panel such as an LCD panel and has the function of displaying images on the display panel.

[0014] Figure 2 is a block diagram of a digital camera 100, which is an electronic device according to the first embodiment of the present invention. In Figure 2, the imaging optical unit 101 includes a plurality of lens groups, including a focusing lens and an image stabilization lens, and an aperture. During shooting, the imaging optical unit 101 adjusts the focus with the focus control unit 118, adjusts the exposure with the aperture control unit 119, corrects blur, etc., and forms an optical image on the image sensor 102. The image sensor 102 has a photoelectric conversion function that converts the optical image into an electrical signal (analog image signal). The image sensor 102 is composed of, for example, a CCD (Charged Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. 102 is equipped with dedicated pixels and multiple independent photodiodes within each pixel for on-sensor phase-detection autofocus. The A / D conversion unit 103 converts the analog image signal from the image sensor unit 102 into a digital image signal. The converted image data is input to the subsequent image processing unit 104.

[0015] Bus 116 is a system bus mainly for transmitting control signals from the CPU 114 to each block, and bus 117 is a data bus mainly for transferring image data. The CPU 114 is composed of a microcomputer that controls the entire digital camera 100, issues operation instructions to each functional block, and executes various control processes. The CPU 114 also performs calculations necessary for various control processes. The CPU 114 controls the image processing unit 104, data transfer unit 105, memory control unit 106, non-volatile memory control unit 108, recording media control unit 110, display control unit 112, operation unit 115, image sensor 102, etc. via the system bus 116. The CPU 114 realizes each process of the digital camera control method described below by executing the program recorded in ROM 109. Furthermore, CP U114 controls the lens and aperture of the imaging optics unit 101 and acquires information such as the focal length.

[0016] The data transfer unit 105 is composed of multiple DMACs (Direct Memory Access Controllers) that perform data transfer. The DRAM (memory) 107 is a memory that stores data. It has sufficient storage capacity to store a predetermined number of still images, a predetermined amount of video footage, audio data, constants for the operation of the CPU 114, programs, etc. The memory control unit 106 writes data to and reads data from the DRAM 107 in response to instructions from the CPU 114 or the data transfer unit 105.

[0017] The non-volatile memory control unit 108 writes and reads data from the ROM (non-volatile memory) 109 in response to instructions from the CPU 114. The ROM 109 is an electrically erasable and recordable memory, and EEPROM or similar is used. Constants for the operation of the CPU 114, programs, etc. are stored in the ROM 109.

[0018] The image processing unit 104 consists of various image processing units and buffer memory, and performs processing on image data such as chromatic aberration correction, development processing, noise reduction processing, geometric transformation, and resizing. The image processing unit 104 also detects, recognizes, and tracks a subject from the developed and processed image, and includes an imaging correction unit that appropriately performs pixel correction, black level correction, shading correction, and scratch correction on the image data converted by the A / D converter 103.

[0019] The recording medium 111 is a recording medium such as an SD card, and is controlled by the recording medium control unit 110 to record image data and read recorded data.

[0020] The display unit 113 consists of a liquid crystal display and an electronic viewfinder, and is controlled by the display control unit 112. It displays various image data and menu screens transferred from the image processing unit 104. In addition, before taking still images and during video recording, it processes and displays image data input from the A / D conversion unit 103 in real time.

[0021] The operating unit 115, which is the means of operation, includes switches, buttons, touch panels, etc., operated by the user, and is used for operations such as turning the power on / off and turning the shutter on / off. The eyeball image sensor 121 forms an optical image of the user's eyeball as they look through the viewfinder and outputs image data to the gaze detection unit 120. The gaze detection unit 120 detects the direction of the user's gaze based on the image input from the eyeball image sensor 121. Details of the gaze detection operation will be described later.

[0022] The illumination light source 123 and the illumination light source drive unit 124 are an infrared light source for line-of-sight detection and a drive unit that controls it. The infrared light is directed towards the user's eyeball, and its reflected image is formed by the eyeball image sensor 120.

[0023] The attention detection unit 125 calculates the user's level of interest in each subject based on the user's gaze data detected by the gaze detection unit 120 and the subject area information recognized by the image processing unit 104, and outputs it as the attention level.

[0024] The autofocus detection unit 122 calculates the lens drive amount for the focusing region output by the gaze detection unit 120 and the image processing unit 104, and instructs the focus control unit 118 to control the drive of the focusing lens. The lens drive amount is calculated, for example, using an image plane phase difference method based on the focus detection image acquired by the image sensor 102.

[0025] Figure 3 is a cross-sectional view of the housing of the digital camera 100 according to the first embodiment, and its configuration This is an explanatory diagram illustrating the concept in general terms. In Figures 1 and 2, corresponding parts are indicated by the same number.

[0026] In Figure 3, the photographic lens 100A is a lens for an interchangeable lens camera. For convenience in this embodiment, the inside of the photographic lens 100A is represented by two lenses, 205 and 206, but in reality, it may be composed of many more lenses. The housing 100B is the camera body, and the image sensor 102 is positioned at the planned imaging plane of the photographic lens 100A of the digital camera 100. The eyepiece lens 203 is a lens for observing the subject image displayed on the display unit 113.

[0027] In the housing 100B, light sources 123a and 123b, each consisting of infrared light-emitting diodes, are arranged around the eyepiece 203 to illuminate the user's eyeball 204 in order to detect the direction of line of sight from the relationship between the reflected image from the corneal reflection of the light source and the pupil. The illuminated eyeball image and the images from the corneal reflection of the light sources 123a and 123b pass through the eyepiece 203, are reflected by the light divider 202, and are imaged onto the eyeball image sensor 121, which has a two-dimensional arrangement of photoelectric elements such as a CCD, by the light-receiving lens 201. The light-receiving lens 201 is positioned so that the pupil of the user's eyeball 204 and the eyeball image sensor 121 are in a conjugate imaging relationship. The direction of line of sight is detected by a predetermined algorithm, described later, from the positional relationship between the eyeball imaged on the eyeball image sensor 121 and the images from the corneal reflection of the light sources 123a and 123b.

[0028] The photographic lens 100A is equipped with an aperture 207, an aperture control unit 119, a lens drive member 210 consisting of a drive gear, and a lens drive motor 211. The photographic lens 100A is also equipped with a photocoupler 209 that detects the rotation of a pulse plate 208 linked to the lens drive member 210 and transmits this information to the focus control unit 118. Based on the information from the photocoupler 209 and the lens drive amount information from the camera, the focus control unit 118 drives the lens drive motor 211 by a predetermined amount to move the focus lens 205 to the focus position. The photographic lens 100A also has a mount contact 212 that serves as an interface between the camera and the lens.

[0029] Figure 4 is an explanatory diagram illustrating the principle of the gaze detection method and is a summary diagram of the optical system for performing gaze detection. In Figure 3, the light sources 123a and 123b are light-emitting diodes, etc., that emit infrared light insensitive to the user. Each light source 123a and 123b is positioned approximately symmetrically with respect to the optical axis of the light-receiving lens 201 and illuminates the observer's eyeball 204. A portion of the illumination light reflected by the eyeball 204 is focused by the light-receiving lens 201 onto the image sensor 121 for the eyeball.

[0030] Figure 5 shows a schematic diagram of the eyeball image projected onto the eyeball image sensor 121 and a diagram of the output intensity of the CCD in the eyeball image sensor 121. In the figure, the upper diagram is a schematic diagram of the eyeball image, and the lower diagram is a diagram of the output intensity of the CCD. Figure 6 shows a flowchart of the gaze detection process executed by the CPU 114 of the digital camera 100 in this embodiment. The gaze detection process in this embodiment will be described below using Figures 4 to 6.

[0031] <Explanation of eye-tracking operation> When the gaze detection routine starts, in step S601, the CPU 114 emits infrared light towards the user's eyeball 204 using light sources 123a and 123b. The image of the user's eyeball, illuminated by the infrared light, is formed on the eyeball image sensor 121 through the light-receiving lens 201, and the eyeball image sensor 121 performs photoelectric conversion, making the eyeball image processable as an electrical signal.

[0032] In step S602, the CPU 114 sends the eyeball image signal obtained from the eyeball image sensor 121 to the gaze detection unit 120.

[0033] In step S603, the CPU 114 uses the information from the eyeball image signal obtained in step S602 to determine the coordinates of the points corresponding to the corneal reflection images Pd and Pe and the pupil center c of the light sources 123a and 123b shown in Figure 4. The infrared light emitted from the light sources 123a and 123b illuminates the cornea 301 of the user's eyeball 204. At this time, the corneal reflection images Pd and Pe, formed by a portion of the infrared light reflected from the surface of the cornea 301, are focused by the light-receiving lens 201 and imaged onto the eyeball image sensor 121 (points Pd' and Pe' in the figure). Similarly, the light beams from the ends a and b of the pupil 302 are also imaged onto the eyeball image sensor 121.

[0034] Figure 5 shows an example of a reflected image obtained from the ophthalmic image sensor 121 (upper figure) and an example of luminance information obtained from the ophthalmic image sensor 121 in region α of the said image example (lower figure). As shown in the figure, the horizontal direction is the X-axis and the vertical direction is the Y-axis. In this case, the coordinates in the X-axis direction (horizontal direction) of the images Pd' and Pe' formed by the corneal reflected images of light sources 123a and 123b are denoted as Xd and Xe, respectively. Also, the coordinates in the X-axis direction of the images a' and b' formed by the light beams from the ends a and b of the pupil 302 are denoted as Xa and Xb, respectively. In the example of luminance information in Figure 5, an extremely high level of luminance is obtained at positions Xd and Xe corresponding to the images Pd' and Pe' formed by the corneal reflected images of light sources 123a and 123b. In the region between coordinates Xa and Xb, which corresponds to the pupil 302, an extremely low level of brightness is obtained, except for the positions Xd and Xe. In contrast, in the region corresponding to the iris 401 outside the pupil 302, which has an X coordinate value lower than Xa and an X coordinate value higher than Xb, an intermediate value between the two types of brightness levels is obtained. From the information on the variation of brightness levels with respect to the above X coordinate positions, the X coordinates Xd and Xe of the images Pd' and Pe' formed by the corneal reflection images of the light sources 123a and 123b, and the X coordinates Xa and Xb of the images a' and b' at the pupillary end can be obtained. Furthermore, when the rotation angle θx of the optical axis of the eyeball 204 with respect to the optical axis of the light-receiving lens 201 is small, the coordinate Xc of the point corresponding to the pupillary center c (let's call it c') that is imaged on the eyeball image sensor 121 can be expressed as Xc ≈ (Xa + Xb) / 2. From the above, the X coordinate of c', which corresponds to the pupil center formed on the image sensor 121 for the eyeball, and the coordinates of the corneal reflection images Pd' and Pe' of the light sources 123a and 123b can be estimated.

[0035] Next, in step S604, the CPU 114 calculates the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position of the eyeball 204 relative to the light-receiving lens 201, and can be essentially calculated as a function of the interval (Xd-Xe) between the corneal reflection images Pd' and Pe'.

[0036] Furthermore, in step S605, the X-coordinate of the midpoint of the corneal reflection images Pd and Pe and the X-coordinate of the curvature center O of the cornea 301 are approximately coincident. Therefore, if Oc is the standard distance between the curvature center O of the cornea 301 and the center c of the pupil 302, the rotation angle θx of the optical axis of the eyeball 204 in the ZX plane can be calculated using the following equation (1). β×Oc×sinθx≒{(Xd+Xe) / 2}-Xc···(1)

[0037] Furthermore, while Figures 4 and 5 show examples of calculating the rotation angle θx when the user's eyeball rotates in a plane perpendicular to the Y-axis, the method for calculating the rotation angle θy when the user's eyeball rotates in a plane perpendicular to the X-axis is similar.

[0038] Once the rotation angles θx and θy of the optical axis of the user's eyeball 204 are calculated, in step S506, the CPU 114 uses θx and θy to determine the position of the user's line of sight (the position of the point of fixation; hereinafter referred to as the fixation point) on the display unit 113. The fixation point position can be determined as the coordinates (Hx, Hy) corresponding to the center c of the pupil 302 on the display unit 113 using the following equations (2) and (3). Hx = m × (Ax × θx + Bx) ···(2) Hy = m × (Ay × θy + By) ... (3) Here, the coefficient m is a constant determined by the configuration of the camera's viewfinder optical system. It is a conversion coefficient that transforms the rotation angles θx and θy into position coordinates corresponding to the center c of the pupil 302 on the display unit 113, and is predetermined and stored in memory 107. In addition, Ax, Bx, Ay, and By are gaze correction coefficients that compensate for individual differences in the user's gaze. These are obtained by performing a calibration operation and are stored in memory 107 before the gaze detection routine shown in Figure 6 starts.

[0039] As described above, after calculating the coordinates (Hx, Hy) of the center c of the pupil 302 on the display unit 113, in step S607, the CPU 114 applies a time-series filter that averages the calculated coordinates over a predetermined period in order to mitigate the effect of coordinate fluctuations due to fixation tremors. Then, in step S608, the CPU 114 stores the above coordinates in memory 107 and terminates the gaze detection routine. The CPU 114 also measures how long the gaze remained in a certain area and stores the time spent at the gaze position in each area as the gaze time in memory 107.

[0040] The above process is one example of a method for obtaining the gaze point coordinates on a display element using corneal reflection images from light sources 123a and 123b. However, any method that can obtain the eyeball rotation angle from the captured eyeball image can be applied to this embodiment.

[0041] Figure 7 shows a flowchart of the main subject selection and switching control process performed by the digital camera 100 according to this embodiment, and Figure 8 is a diagram schematically illustrating the main subject selection and switching control process of this embodiment. Figure 9 is a diagram showing an example of the display of the user's attention level. The process performed in Figure 7 will be described below with reference to Figures 7 to 9.

[0042] When the user starts shooting, the CPU 114 starts the process of selecting and switching the main subject, and two processes, step S701 and step S704, are executed in parallel.

[0043] In step S701, the image sensor 102 acquires an image of the shooting area including the main subject, and outputs the acquired image to the image processing unit 104 via the A / D conversion unit 103. Once the acquisition and output of the image of the shooting area is complete in step S701, the process proceeds to step S702.

[0044] In step S702, the image processing unit 104 recognizes subjects in the live view image generated for display on the display unit 113 in the viewfinder. The subject recognition process estimates specific subject regions using, for example, deep learning, i.e., a multilayer neural network including trained convolutional layers. Alternatively, without using deep learning, prominent regions in the image may be calculated from texture, color, histogram, etc., and estimated as subject regions. If there are multiple subjects in the image, multiple subject region estimation results will be output. In the initial frame, all subjects are estimated, and in subsequent frames, results corresponding to newly framed subjects that are not being tracked are output. For example, as shown in Figure 8A, if subjects 801, 803, and 805 are present in the live view image, their respective subject regions 802, 804, and 806 are output as estimation results. Once the subject regions are estimated by the subject recognition process in step S702, the process proceeds to step S703.

[0045] In step S703, the image processing unit 104 functions as a tracking means, tracking multiple subjects output in step S702, and outputting the subject tracking result to the CPU 114. The subject tracking process estimates the region in the current frame that corresponds to the subject region to be tracked in past frames, for example, using deep learning. More specifically, it has a function to extract feature points and the feature quantities contained in the feature points for the subject region, and a function to associate the extracted feature points between frames. That is, the position and size of the subject region to be tracked in the current frame are estimated from the feature points of the current frame that are associated with the feature points of the subject region to be tracked in past frames. In addition, texture, color, histogram, etc. are used without using deep learning. Alternatively, the system may estimate the subject area in the current frame that corresponds to the subject area being tracked in past frames. Subject tracking is considered lost if the subject area being tracked in past frames cannot be found in the current frame due to the subject going out of frame, etc. Furthermore, for subjects newly recognized in step S702 in the current frame, the estimation result of the subject recognition process is output directly as the subject tracking result.

[0046] For example, as shown in Figure 8A, if subjects 801, 803, and 805 are present in the live view image, the corresponding subject regions 802, 804, and 806 in the previous frame can be estimated, allowing the system to continue recognizing them as the same subject across multiple frames. In step S703, when the subject tracking process estimates each subject region in the current frame, the results are also output to the attention level calculation unit 125, and the process waits for the completion of step S705 before proceeding to step S706.

[0047] Meanwhile, in step S704, the eyeball image sensor 121 acquires an image of the user's eyeball illuminated by the illumination light source 123 and outputs the image data to the gaze detection unit 120. Once the user's eyeball image is acquired in step S704, the process proceeds to step S705.

[0048] In step S705, the gaze detection unit 120 functions as a detection means and detects the user's gaze using the gaze detection routine described above. The gaze detection unit 120 then calculates the viewpoint coordinates in the live view image displayed on the display unit 113 in the viewfinder and displays the viewpoint pointer (for example, 808 in Figure 8A). The gaze detection unit 120 also outputs the viewpoint coordinates to the CPU 114 and the attention level calculation unit 125. Once the user's viewpoint coordinates are calculated in step S705, the process waits for the completion of step S703 and proceeds to step S706.

[0049] In step S706, the attention calculation unit 125 functions as a determination means and determines the user's level of attention to each of multiple subjects based on the user's viewpoint. Specifically, the attention calculation unit 125 calculates the user's level of attention for each subject from the subject area information in the current frame output by the image processing unit 104 in step S703 and the viewpoint coordinates output by the gaze detection unit 120 in step S705. The level of attention for each subject is the degree to which the detected user's viewpoint is included within the area of ​​the tracked subject. This degree can be determined by the period during which the viewpoint coordinates remained within the subject area (for example, the number of samples used to calculate the viewpoint coordinates, the number of corresponding display frames, the count value of the number of times it remained for a predetermined time, or the ratio to the total value for all subjects). For subjects that are determined to be lost in step S703, the level of attention is discarded. The calculated level of attention for each subject is output to the CPU 114. The level of attention information is also superimposed on the live view image displayed on the display unit 113 in the viewfinder, depending on the settings (for example, 809 in Figure 8A).

[0050] The attention level information may be displayed as follows: as shown in Figure 9A, it may be divided into several levels (for example, 5 levels) based on a preset threshold and displayed as a numerical value; as shown in Figure 9B, the attention level information may be displayed as a numerical value as is; or as shown in Figure 9C, the level values ​​may be displayed on a meter. Alternatively, the level values ​​may be represented by the color and thickness of the frame indicating the subject area, or by multiple lines. In this way, the display unit 113 can display the attention level of multiple determined subjects using at least one of numerical values, shapes, or colors. Once the attention level for each subject is calculated in step S706, the process proceeds to step S707.

[0051] In step S707, the CPU 114 determines whether the user has performed a switching operation for the main subject to be AF controlled by operating the operating unit 115, which is an operating means. Here, as an example of a switching operation for the main subject, the user can use at least one of the touch panel 115a, the operating lever 115b, or the four-way key 115c to determine the main subject after switching. The following actions are performed. If the decision to switch the main subject is made (S707:YES), the process proceeds to step S708. If the decision to switch the main subject is not made (S707:NO), the process proceeds to step S709.

[0052] In step S709, the CPU 114 determines whether the distance between the viewpoint coordinates and the coordinates of the subject (for example, the center coordinates of the subject area) is less than or equal to a specific threshold distance. As shown in Figure 8A, when the viewpoint pointer, which indicates the user's viewpoint, moves from position 807 to position 808, the viewpoint coordinates are included within the region 810, which indicates that the viewpoint coordinates are within a specific threshold distance range from the subject 803 in Figure 8B. This determines that the viewpoint coordinates and the subject coordinates are less than or equal to a specific threshold distance. The region 810, which indicates the specific threshold distance from the subject 803, is calculated to be the same size as the subject area 804 when the attention level of the subject 803 is zero, and to increase as the attention level increases. For the sake of explanation, although the region 810 is shown as being displayed on the display unit 113 in the figure, the region 810 is not displayed on the display unit 113 in the viewfinder, and the digital camera 100 internally holds information about the region 810.

[0053] Furthermore, since switching the main subject basically involves switching from the main subject to a subject other than the main subject, in step S709, the attention level of the main subject 801 is treated as zero. If it is determined in step S709 that the viewpoint coordinates and the subject coordinates are less than or equal to a certain distance (S709: YES), the process proceeds to step S710. If it is determined that the viewpoint coordinates and the subject coordinates are greater than a certain distance (S709: NO), the process proceeds to step S711.

[0054] In step S710, the CPU 114 functions as a selection means and activates subjects whose distance from the viewpoint coordinates is less than or equal to a certain distance as targets for main subject switching (selection state in which they can be determined as the main subject). The CPU 114 then outputs a command to the display control unit 112 to display a notification to the user of this state. The display control unit 112 functions as a display control means according to the command from the CPU 114 and displays multiple subjects in the image in a selectable state. Specifically, for example, it displays a double frame 811 surrounding a subject as shown in Figure 8B, which is a different display from the thick frame 802 of the main subject 801 that is currently the target of AF control, and the dashed frame 806 that indicates the area of ​​subjects other than the main subject. Once the activation display for the main subject as a target for switching is made in step S710, the process proceeds to step S711.

[0055] Meanwhile, in step S708, the CPU 114 sets the subject selected by the user when the main subject switching decision operation is performed as the main subject, and resets the attention level of all subjects to their initial values. The display of attention level information is also reset to its initial value of zero. Once the main subject switching process is completed in step S708, the process proceeds to step S711.

[0056] In step S711, the CPU 114 instructs the autofocus detection unit 122 to perform focus control on the main subject area (802 or 812). The autofocus detection unit 122 calculates the lens drive amount and instructs the focus control unit 118 to perform drive control of the focusing lens. Once the focus control unit 118 completes the focus control on the main subject area, the process proceeds to step S712.

[0057] In the examples in Figures 8A to 8C, when subject 801 is set as the main subject, and the viewpoint pointer 808 indicating the user's viewpoint position moves into area 810, a notification (display of frame 811) is given that subject 803 can be selected as the main subject. Furthermore, AF control is performed on subject 801 until the user makes a decision to switch subject 803 to the main subject. When the user makes a decision to switch the main subject to subject 803, the main subject switches from subject 801 to subject 803, as shown in Figure 8C, and subject 80 AF control is performed on 3. In addition, the display unit 113 displays a dashed frame 813 indicating subjects other than the main subject and a thick frame 812 indicating the main subject for the subject area.

[0058] Then, in step S712, the CPU 114 determines whether the user has finished shooting. If the user turns off the power of the digital camera 100 or cancels the AF operation (S712: YES), the CPU 114 terminates the main subject selection and switching control process. If shooting continues (S712: NO), the CPU 114 returns to steps S701 and S704 and repeats the process described in steps S701 to S712.

[0059] As explained above, the range of subjects selected as candidates for the main subject is expanded based on the level of attention each subject receives. Then, among multiple subjects, the subject that receives more attention in the image, i.e., the subject that attracts the user's attention more, is displayed with higher priority than other subjects. The display control unit 120 distinguishes and displays, among multiple subjects, the second subject that the user has looked at for a longer time than the first subject, as a candidate for switching to the main subject, rather than the first subject. Therefore, among multiple subjects, the subject that the user has looked at for a longer time than the first subject is more likely to be selected. In addition, it is possible to select a subject as a target for switching to the main subject even when the viewpoint pointer is not completely over the subject area. This improves the responsiveness of the main subject selection and switching operation when the main subject switching operation is performed by specifying the position with the user's gaze and confirming with a button, thereby reducing the loss of shooting opportunities.

[0060] (Second embodiment) Next, a digital camera, which is an electronic device according to the second embodiment, will be described with reference to Figures 10 and 11. In the second embodiment, instead of enabling subjects whose distance from the viewpoint coordinates is less than or equal to a certain distance as targets for main subject switching, the position of the viewpoint pointer is moved. This makes it possible to select subjects for switching even when the viewpoint pointer is not completely over the subject area.

[0061] The configuration of the digital camera according to this embodiment is the same as that of the first embodiment, and some of the main subject selection and switching control processing is the same as that of the first embodiment. In the following description, we will mainly describe the parts that differ from the first embodiment, and the same components and processes will be denoted by the same reference numerals, and detailed explanations will be omitted.

[0062] Figure 10 is a flowchart showing the main subject selection and switching control process of the digital camera 100 according to this embodiment, and Figure 11 is a schematic diagram illustrating the main subject selection and switching control process of this embodiment. The process performed in Figure 10 will be described below with reference to Figures 10 and 11.

[0063] When the user starts shooting, the CPU 114 starts the process of selecting and switching the main subject, and two processes, steps S701 and S704, are executed in parallel. Steps S701 to S709 are the same as in the first embodiment.

[0064] Next, in step S1001, the CPU 114 moves the viewpoint pointer to a coordinate that overlaps with a subject whose distance from the viewpoint coordinate is less than or equal to a specific distance. Then, the CPU 114 puts the subject that overlaps with the viewpoint pointer into a state where it can be selected as the target for switching the main subject, and outputs a command to the display control unit 112 to display a notification to the user that this state has been reached. Here, the notification to the user is, for example, a display that changes the position of the viewpoint pointer 808 to a position that overlaps with the subject 1101, as shown in Figure 11B. Note that the viewpoint point in Figure 11A The display format for the movement of the viewpoint pointer 808 from the position of TA 808 to the position of viewpoint pointer 808 in Figure 11B can be any display format. For example, the display of viewpoint pointer 808 in Figure 11A may be instantly switched to the display in Figure 11B, or the movement display may be such that the viewpoint pointer 808 in Figure 11A is attracted to the position of viewpoint pointer 808 in Figure 11B. When the viewpoint coordinates and the display position of the viewpoint pointer are changed in step S1001, the process proceeds to step S711. The processing in steps S711 and S712 is the same as in the first embodiment.

[0065] As explained above, by applying a range of viewpoint coordinate movement determined based on the level of attention each subject receives, the viewpoint pointer moves even when it is not over a subject, making it possible to select it as the main subject to switch. This improves the responsiveness of the main subject selection and switching operation when the main subject switching operation is performed by specifying the position with the user's gaze and confirming with a button, thereby reducing the loss of shooting opportunities.

[0066] (Third embodiment) Next, a digital camera according to a third embodiment will be described with reference to Figures 12 and 13. In the third embodiment, an example of the selection and switching process for the main subject will be described when the decision to select the subject is made by gaze detection rather than by button operation.

[0067] The configuration of the digital camera according to this embodiment is the same as that of the first embodiment, and some of the main subject selection and switching control processing is the same as that of the first embodiment. In the following description, we will mainly describe the parts that differ from the first embodiment, and the same components and processes will be denoted by the same reference numerals, and detailed explanations will be omitted.

[0068] Figure 12 is a flowchart showing the main subject selection and switching control process of the digital camera 100 according to this embodiment, and Figure 13 is a schematic diagram illustrating the main subject selection and switching control process of this embodiment. The process performed in Figure 12 will be described below with reference to Figures 12 and 13.

[0069] When the user starts shooting, the CPU 114 starts the process of selecting and switching the main subject, and two processes, steps S701 and S704, are executed in parallel. Steps S701 to S706 are the same as in the first embodiment.

[0070] Next, in step S1201, the CPU 114 determines whether the viewpoint coordinates are located within a subject area other than the main subject. As a specific example, as shown in Figure 13A, the position of the viewpoint pointer indicating the user's viewpoint moves from the position of viewpoint pointer 807 to the position of viewpoint pointer 1301 within the subject area 804. At this time, the CPU 114 determines that the user's viewpoint coordinates are located within the subject area 704 of subject 703. If it is determined that the viewpoint coordinates are located within a subject area other than the main subject (S1201: YES), the process proceeds to step S1202. If it is determined that the viewpoint coordinates are not located within a subject area other than the main subject (S1201: NO), the process proceeds to step S711.

[0071] In step S1202, the CPU 114 reads the user's gaze time stored in memory 107 in step S705 and determines whether the gaze time is greater than or equal to a specific time. For example, as shown in Figure 13B, after the viewpoint pointer 1301 moves into the subject area 804, the viewpoint pointer 1301 remains within the subject area 704 of the subject 703 for a specific time or longer. At this time, the CPU 114 determines that the gaze time is greater than or equal to a specific time and determines that a decision operation to switch the main subject has been performed. Here, the specific time, which is the threshold for making the gaze determination, is, for example, a threshold time that is set in advance when the attention level of the subject 703 is zero. This time is set to be shorter as the attention level increases, and the attention level increases as the attention level increases. The shorter the image, the shorter the time required for gaze detection. If it is determined that the gaze time is longer than a certain time (S1202: YES), the process proceeds to step S708. If it is determined that the gaze time is shorter than a certain time (S1202: NO), the process proceeds to step S711. The processing in steps S708 and S711 and S712 is the same as in the first embodiment.

[0072] As explained above, by setting the gaze detection threshold time based on the level of attention each subject receives, the time it takes for a subject to be selected as the main subject to change decreases for subjects that receive more attention. This improves the responsiveness of the main subject selection and switching operation when the main subject switching operation is performed by specifying the position with the user's gaze and making a decision based on gaze detection, thereby reducing the loss of shooting opportunities.

[0073] (Fourth embodiment) Next, a digital camera according to the fourth embodiment will be described with reference to Figures 14 and 15. In the fourth embodiment, an example of the main subject selection and switching process will be described when the decision to select a subject is made by button operation, rather than by processing using the user's viewpoint position.

[0074] The configuration of the digital camera according to this embodiment is the same as that of the first embodiment, and some of the main subject selection and switching control processing is the same as that of the first embodiment. In the following description, we will mainly describe the parts that differ from the first embodiment, and the same components and processes will be denoted by the same reference numerals, and detailed explanations will be omitted.

[0075] Figure 14 is a flowchart showing the main subject selection and switching control process of the digital camera 100 according to this embodiment, and Figure 15 is a diagram schematically illustrating the main subject selection and switching control process of this embodiment. The process performed in Figure 14 will be described below with reference to Figures 14 and 15.

[0076] When the user starts shooting, the CPU 114 starts the process of selecting and switching the main subject, and two processes, steps S701 and S704, are executed in parallel. Steps S701 to S708 are the same as in the first embodiment.

[0077] Next, in step S1401, the CPU 114 determines whether the user has performed an operation to select a subject for switching the main subject to be AF controlled by using a specific operation such as the directional keys on the control unit 115. Here, the subject selection operation is an operation to switch the selection among the subjects that are candidates for the main subject displayed on the display unit 113. In this step, it is not determined whether an operation to confirm the selected subject as the main subject has been performed; this determination is made in step S707. If it is determined that the user has performed a subject selection operation (S1401: YES), the process proceeds to step S1402. If it is determined that the user has not performed a subject selection operation (S1401: NO), the process proceeds to step S711.

[0078] In step S1402, the CPU 114, in response to the user's selection operation, selects subjects in order of importance as candidates for switching the main subject, and outputs a command to the display control unit 112 to display a notification to the user of the subject selection status. The notification to the user is, for example, a double-framed display 1401 as shown in Figure 15B, and is different from the thick frame 802 that indicates the main subject 801 that is the target of AF control, and the dashed frame 806 that indicates the subject area other than the main subject. Also, Figure 15B shows the display state of the display unit 113 when the user has performed a subject selection operation once, and it shows that the subject 803, which is the most important subject other than the main subject 801, has been selected as the target for switching the main subject. After the selection status of the subject as a subject candidate for the main subject is displayed in step S1402, the process proceeds to step S711. Steps S711 and S712 are the same as in the first embodiment.

[0079] Figure 15D shows the state after the user has performed another subject selection operation from the state shown in Figure 15B. More specifically, after the process in step S1402 is executed and the display state of the display unit 113 becomes as shown in Figure 15B, the process proceeds to step S711 and AF control is performed on the main subject 801. Then, the process returns from step S712 to steps S701 and S704, and the processes in steps S701 to S706 are executed, and the process proceeds from step S707 to step S1401. Then, because the user has performed another subject selection operation, the process proceeds from step S1401 to step S1402. In step S1402, the display state of the display unit 113 becomes as shown in Figure 15D, with subject 805, which is the next most important subject after subject 803 other than the main subject 801, being selected as the target for main subject switching. Furthermore, if there is no subject other than the main subject that is the next most important subject after the currently selected subject, the display control may be configured to re-select the subject other than the main subject that is the most important subject. In the examples in Figures 15A to 15D, if the user performs another subject selection operation from the state in Figure 15D where subject 805 is selected as the target for switching to the main subject, the display state of the display unit 113 will return to the state shown in Figure 15B.

[0080] As explained above, subjects that are candidates for switching to the main subject are selected and displayed in a display order based on the level of attention each subject receives, so that subjects with a higher level of attention are given priority as subjects to be switched to the main subject. In other words, the display control unit 120 displays the subjects that are candidates for switching from among multiple subjects in a way that distinguishes the level of attention each subject candidate receives. This makes it possible to improve the responsiveness of the main subject selection and switching operation when the main subject switching operation is performed by the user's button operation for subject selection and confirmation, and reduces the loss of shooting opportunities.

[0081] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

[0082] Although the embodiments described above illustrate the application of the present invention to a digital camera, the electronic devices to which the present invention can be applied are not limited to digital cameras. For example, the present invention can be applied to personal computers, smartphones, tablet terminals, head-mounted displays, smart glasses, and the like. Furthermore, the images to be displayed are not limited to those captured by an imaging device. For example, the present invention can also be applied when selecting an item from a screen (image, such as a menu screen or desktop screen) that contains multiple items (objects).

[0083] Furthermore, for example, in the first embodiment, an example of a gaze detection method was described assuming that the user is looking through the viewfinder, but a gaze detection method when the user is looking at the display on the rear panel may also be used. Also, although the explanation assumed that subjects other than the main subject were the target of the main subject switching and treated the attention level of the main subject as zero, this is not limited to this. The main subject may also be treated in the same way as subjects other than the main subject.

[0084] Furthermore, in the third embodiment, an example was described in which the target of the selection and switching operation of the main subject is a subject other than the main subject. However, the main subject may also be treated as a processing target in the same way as other subjects, so that the process is executed to re-select the main subject. This makes it possible to control the so-called re-grabbing of the subject.

[0085] Furthermore, each functional unit in each of the above embodiments (each modified example) may or may not be individual hardware. The functions of two or more functional units may be implemented by common hardware. Each of the multiple functions of a single functional unit may be implemented by individual hardware. Two or more functions of a single functional unit may be implemented by common hardware. In addition, each functional unit may or may not be implemented by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. The functions of at least some of the functional units of the device may be implemented by the processor reading and executing the control program from the memory.

[0086] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0087] This embodiment includes the following configurations and methods. (Configuration 1) A detection means for detecting the user's viewpoint in an image captured by an imaging means and displayed on a display means, Tracking means for tracking multiple subjects in the aforementioned image, A determination means for determining the degree of attention the user is paying to each of the multiple subjects, based on the user's perspective, A selection means for selecting one of the aforementioned multiple subjects as the main subject. It has, The selection means selects the main subject based on the degree of attention each of the multiple subjects receives. An electronic device characterized by the following features. (Configuration 2) The electronic device according to Configuration 1, characterized in that the determination means determines the degree to which the user's viewpoint detected by the detection means is included within the area of ​​the subject being tracked by the tracking means as the degree of attention. (Configuration 3) The electronic device according to Configuration 2, characterized in that the degree is the number of images sampled or displayed frames in which the user's viewpoint detected by the detection means is within the area of ​​the subject, or the number of times the user's viewpoint detected by the detection means remained within the area of ​​the subject for a predetermined time. (Configuration 4) The electronic device according to any one of Configurations 1 to 3, characterized in that the selection means prioritizes selecting the subject with the highest degree of attention from among the plurality of subjects. (Configuration 5) The electronic device according to any one of Configurations 1 to 4, wherein the selection means selects the main subject based on the degree of attention each subject has to receive, excluding the subject selected as the main subject from among the plurality of subjects. (Configuration 6) The electronic device according to any one of Configurations 1 to 5, characterized in that the determination means returns the attention level of the subject being tracked to its initial value when the subject being tracked by the tracking means is no longer present in the image. (Configuration 7) The electronic device according to any one of Configurations 1 to 6, further comprising a display control means that displays the degree of attention of the plurality of subjects determined by the determination means on the display means using at least one of numerical values, figures, and colors. (Configuration 8) Further comprising an operating means for receiving operations by the user, The electronic device according to any one of configurations 1 to 7, characterized in that, when the switching of the main subject is performed by specifying a position from the user's viewpoint in the image and an operation received by the operating means, the selection means determines a candidate subject to become the main subject based on the distance between the subject and the user's viewpoint and the degree of attention. (Configuration 9) The electronic device according to Configuration 8, characterized in that the selection means determines as a subject candidate a subject whose distance from the subject to the user's viewpoint is less than or equal to a threshold distance determined according to the degree of attention. (Configuration 10) The electronic device according to Configuration 8 or 9, further comprising a notification means for notifying the user that the main subject can be switched by the operating means. (Configuration 11) Further comprising an operating means for receiving operations by the user, The electronic device according to any one of configurations 1 to 7, characterized in that, when the switching of the main subject is performed by specifying a position from the user's viewpoint in the image and an operation received by the operating means, the selection means determines the display position of the user's viewpoint in the image based on the distance between the subject and the user's viewpoint and the degree of attention. (Configuration 12) The electronic device according to Configuration 11, characterized in that the selection means determines a position where the distance between the subject and the user's viewpoint is less than or equal to a threshold distance determined according to the degree of attention, as the display position of the user's viewpoint. (Configuration 13) The electronic device according to any one of Configurations 1 to 7, wherein when the switching of the main subject is performed by specifying a position from the user's viewpoint and determining the gaze from the user's viewpoint, the selection means determines the gaze time used for the gaze determination based on the degree of attention the subject receives. (Configuration 14) An operating means for receiving operations by the user, The system further includes a display control means that displays in a manner that allows for the distinction of candidate subjects to be switched among the aforementioned plurality of subjects, The electronic device according to any one of configurations 1 to 7, characterized in that when the switching of the main subject is performed by an operation received by the operating means, the display control means displays the subject candidates to be switched, distinguishing the level of attention for each subject candidate. (Configuration 15) A display control means that allows selection of multiple subjects in an image captured by the imaging means and displayed on the display means, A selection means for selecting one of the multiple subjects displayed as selectable, It has, Of the multiple subjects, the subject that the user views for a second time, which is longer than the first time, is more likely to be selected by the selection means than the subject that the user views for a first time. An electronic device characterized by the following features. (Composition 16) A display control means that displays an image captured by an imaging means on a display means, A detection means for detecting the user's viewpoint in the image displayed on the display means, A selection means for selecting one of the aforementioned multiple subjects as the main subject, An operating means for receiving operations by the user, It has, The operation received by the operation means allows for the switching of the main subject selected by the selection means. The display control means is characterized in that it displays, in a manner that distinguishes, among the plurality of subjects, a second subject that the user has viewed for a longer period of time than the first subject, rather than the first subject, as a candidate for switching to the main subject. (Configuration 17) A detection means for detecting the user's viewpoint in the image displayed on the display means, A determination means for determining the degree of attention the user is paying to each of the multiple objects in the image, based on the user's viewpoint. A selection means for selecting an object based on the degree of attention each of the aforementioned multiple objects, An electronic device characterized by having the following features. (Method 1) A detection step of detecting the user's viewpoint in an image captured by an imaging means and displayed on a display means, A tracking step that tracks multiple subjects in the aforementioned image, A determination step that determines the degree of attention the user is paying to each of the multiple subjects based on the user's perspective, A selection step of selecting one of the aforementioned multiple subjects as the main subject. Includes, The selection step involves selecting the main subject based on the degree of attention each of the multiple subjects receives. A method for controlling electronic equipment characterized by the following features. (Program 1) A program that causes a computer to execute the control method described in Method 1. [Explanation of Symbols]

[0088] 100 Digital camera, 104 Image processing unit, 114 CPU, 120 Eye-tracking unit, 125 Attention level calculation unit

Claims

1. A detection means for detecting the user's viewpoint in an image captured by an imaging means and displayed on a display means, Tracking means for tracking multiple subjects in the aforementioned image, A determination means for determining the degree of attention the user is paying to each of the multiple subjects, based on the user's perspective, A selection means for selecting one of the aforementioned multiple subjects as the main subject. It has, The selection means makes it possible to select, from among the plurality of subjects, a subject whose distance from the user's viewpoint in the image is less than or equal to the threshold distance based on the level of attention, as a candidate subject to be switched as the main subject. An electronic device characterized by the following features.

2. The electronic device according to claim 1, characterized in that the determination means determines the degree to which the user's viewpoint detected by the detection means is included within the area of ​​the subject being tracked by the tracking means as the degree of attention.

3. The electronic device according to claim 2, characterized in that the degree is the number of image samples or display frames in which the user's viewpoint detected by the detection means is located within the area of ​​the subject, or the number of times the user's viewpoint detected by the detection means remained within the area of ​​the subject for a predetermined time.

4. The electronic device according to claim 1 or 2, characterized in that the selection means prioritizes selecting the subject with the highest level of attention from among the plurality of subjects as a candidate subject.

5. The selection means is characterized in that, among the multiple subjects, excluding the subject selected as the main subject, a subject whose distance from the user's viewpoint in the image is less than or equal to the threshold distance can be selected as the main subject (1 or 2). The electronic devices described above.

6. The electronic device according to claim 1 or 2, characterized in that the determination means returns the attention level of the subject being tracked to its initial value when the subject being tracked by the tracking means is no longer present in the image.

7. The electronic device according to claim 1 or 2, further comprising a display control means for displaying the degree of attention of the plurality of subjects determined by the determination means on the display means using at least one of numerical values, figures, and colors.

8. The system further includes an operating means for receiving operations from the user, The electronic device according to claim 1 or 2, characterized in that the switching of the main subject is performed by specifying a position from the user's viewpoint in the image and an operation received by the operating means.

9. The electronic device according to claim 8, further comprising a notification means for notifying the user that the main subject can be switched by the aforementioned operating means.

10. The system further includes an operating means for receiving operations from the user, The electronic device according to claim 1 or 2, characterized in that, when the switching of the main subject is performed by specifying a position from the user's viewpoint in the image and an operation received by the operating means, the selection means determines the display position of the user's viewpoint in the image based on the distance between the subject and the user's viewpoint and the degree of attention.

11. The electronic device according to claim 10, characterized in that the selection means determines a position where the distance between the subject and the user's viewpoint is less than or equal to a threshold distance determined according to the degree of attention, as the display position of the user's viewpoint.

12. The electronic device according to claim 1 or 2, wherein, when the switching of the main subject is performed by specifying a position from the user's viewpoint and determining the gaze from the user's viewpoint, the selection means determines the gaze time used for the gaze determination based on the degree of attention the subject receives.

13. An operating means for receiving operations by the user, The system further includes a display control means for displaying the aforementioned candidate subjects in a distinguishable manner. The electronic device according to claim 1 or 2, characterized in that when the switching of the main subject is performed by an operation received by the operating means, the display control means displays the subject candidates in a manner that distinguishes the degree of attention for each subject candidate.

14. A detection means for detecting the user's viewpoint in an image displayed on a display means, A determination means for determining the degree of attention the user is paying to each of the multiple objects in the image, based on the user's viewpoint. A selection means for selecting an object based on the degree of attention each of the aforementioned multiple objects, It has, The selection means makes available for selection, among the plurality of objects, an object whose distance from the user's viewpoint in the image is less than or equal to the threshold distance based on the level of attention, as a candidate object to be switched. An electronic device characterized by the following features.

15. A detection step for detecting the user's viewpoint in an image captured by an imaging means and displayed on a display means, A tracking step that tracks multiple subjects in the aforementioned image, A determination step that determines the degree of attention the user is paying to each of the multiple subjects based on the user's perspective, A selection step of selecting one of the aforementioned multiple subjects as the main subject. Includes, The selection step makes it possible to select, from among the multiple subjects, a subject whose distance from the user's viewpoint in the image is less than or equal to the threshold distance based on the level of attention, as a candidate subject to be switched to the main subject. A method for controlling electronic equipment characterized by the following features.

16. A program for causing a computer to execute the control method described in claim 15.

Images