Electronic device and control method for electronic device

Abstract

To provide an electronic device capable of adjusting the responsiveness of a visual line position detected with respect to a user's visual line considering the response speed of a user's visual line to an object.SOLUTION: The electronic device has first acquisition means for detecting a user's visual line and acquiring a user's visual line location in an image displayed on a display unit, second acquisition means for acquiring an object location in the image, and adjusting means for adjusting the visual line responsiveness, which is the responsiveness in which the user's visual line movement is reflected in the visual line location on the basis of the amount of displacement of the amount of movements between the amount of movement of the visual line location and the amount of movement of the object location, both moving in a predetermined time, or the amount of displacement in the movement time between the amount of movement time of the visual line location and the amount of movement time of the object's location, both moving by a predetermined distance.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to an electronic device capable of detecting a gaze position and a method for controlling the electronic device. [Background technology]

[0002] In recent years, cameras have become increasingly automated and intelligent. For example, a technology has been proposed that recognizes the subject intended by the user based on the line of sight of the user looking through the viewfinder and controls the focus, without the need to manually input the position of the subject (object). When a camera detects the user's line of sight, there may be a discrepancy between the user's intended line of sight and the position of the user's line of sight recognized by the camera, resulting in the camera not being able to focus on the subject intended by the user.

[0003] In response to this, Patent Document 1 discloses a technology that displays an index in the viewfinder and corrects detection errors in the line of sight of an observer gazing at the index. Patent Document 2 discloses a technology that adjusts focus based on tracking operation parameters that are set according to the movement of the subject, the movement of the imaging device, etc., and continues to track the subject appropriately. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-008323 [Patent Document 2] Japanese Patent Application Publication No. 2019-020716 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology of Patent Document 1, the detected gaze position is corrected without taking into consideration the gaze reaction speed to an object. Since the gaze reaction speed differs depending on the user, if the responsiveness of the gaze position detected to the user's gaze is made uniform, there is a possibility that the subject intended by the user will not be in focus.

[0006] The technology in Patent Document 2 does not take into account the reaction speed of the line of sight, so when multiple subjects are included within the camera's field of view, there may be a delay in identifying the subject to be photographed by moving the line of sight, making it difficult for the user to focus on the intended subject in a timely manner.

[0007] Therefore, an object of the present invention is to provide an electronic device that can adjust the responsiveness of the gaze position detected in response to the user's gaze, taking into account the reaction speed of the user's gaze to an object. [Means for solving the problem]

[0008] In order to achieve the above object, the present invention provides: a first acquisition means for detecting a user's line of sight and acquiring a line of sight position of the user in an image displayed on a display unit; a second acquisition means for acquiring a position of an object in the image; a display control means for controlling the display unit to display an index indicating the gaze position acquired by the first acquisition means in the image; an adjustment means for adjusting the responsiveness of the movement of the indicator to the movement of the user's line of sight; a control means for controlling the adjustment by the adjustment means based on a deviation between the amount of movement of the line of sight position that moves in a predetermined time and the amount of movement of the position of the object, or a deviation between the movement time of the line of sight position that moves a predetermined distance and the movement time of the position of the object; and When the deviation amount of the movement amount or the deviation amount of the movement time is greater than a predetermined threshold, the control means (1) The object is moving and controlling the adjusting means so as to increase the responsiveness. (2) When the object is not moving, the response is reduced. the adjusting means of Control The electronic device is characterized by the above. [Effects of the Invention]

[0009] According to the present invention, it is possible to adjust the responsiveness of the gaze position detected in response to the user's gaze, taking into account the reaction speed of the user's gaze. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is an external view of a camera according to an embodiment of the present invention; [Figure 2] 1 is a schematic diagram illustrating the configuration of a camera according to an embodiment. [Figure 3] FIG. 1 is a block diagram of a camera according to an embodiment. [Figure 4] FIG. 1 is a schematic diagram illustrating the configuration of a head-mounted display according to an embodiment. [Figure 5] FIG. 2 is a diagram showing a field of view within a finder according to the embodiment. [Figure 6] 1A and 1B are diagrams for explaining the principle of a visual field detection method according to an embodiment. [Figure 7] FIG. 2 is a diagram showing an eye image according to an embodiment. [Figure 8] 10 is a flowchart of a gaze detection operation. [Figure 9] 10A and 10B are diagrams illustrating a case where gaze responsiveness is set lower than standard. [Figure 10] 10A and 10B are diagrams illustrating a case where gaze responsiveness is set higher than standard. [Figure 11] 10 is a flowchart of an operation for changing gaze responsiveness. [Figure 12] 10 is a flowchart of an operation for changing gaze responsiveness using a threshold value. [Figure 13] 10A and 10B are diagrams illustrating examples of setting information of gaze responsiveness for each shooting scene. [Figure 14] FIG. 10 is an external view showing an example of another electronic device to which the present invention can be applied. DETAILED DESCRIPTION OF THE INVENTION

[0011] Gaze position detection technology can be used not only to recognize the subject the user is gazing at with a camera, but also in other technologies such as xR (a collective term for virtual reality (VR), augmented reality (AR), mixed reality (MR), and substitute reality (SR)). For example, a head-mounted display (HMD) can determine the position at which the user is gazing on a video image, and can clearly depict the area around the viewpoint (gaze position) while blurring the surrounding area away from the viewpoint. Simulating depth of field on a video image based on the viewpoint enhances the user's sense of immersion and contributes to providing a more comfortable VR experience. Therefore, it is desirable for the detected gaze position to more accurately track the subject (object) intended by the user.

[0012] However, the reaction speed of a user's gaze to an object may vary depending on age, physical condition, etc., and if the responsiveness of the gaze position detected to the gaze is uniform, the accuracy of tracking the gaze position to the user's intended object may decrease. The present invention can improve the accuracy of tracking the gaze position to an object by adjusting the responsiveness of the gaze position detected to the user's gaze, taking into account the reaction speed of the user's gaze to the object. Hereinafter, the responsiveness of the detected gaze position to the user's actual gaze is also referred to as gaze responsiveness. Gaze responsiveness is the responsiveness with which the movement of the user's gaze is reflected in the gaze position. Note that in the following embodiments, the object is described as a subject of a camera, but the present invention can also be applied to objects in an image displayed on an HMD or the like.

[0013] <<Embodiment>> Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0014] <Configuration explanation> 1(a) and 1(b) show the appearance of a camera 1 as an electronic device according to an embodiment. FIG. 1(a) is a front perspective view, and FIG. 1(b) is a rear perspective view. As shown in FIG. 1(a), the camera 1 has a photographing lens unit 1A and a camera housing 1B. A release button 5, which is an operating member that accepts image capture operations from a user (photographer), is disposed on the camera housing 1B. As shown in FIG. 1(b), an eyepiece window frame 121 and an eyepiece lens 12 (eyepiece optical system) are disposed on the back of the camera housing 1B, through which the user looks into a display device 10 (display panel) (described later) contained within the camera housing 1B. The eyepiece window frame 121 is It surrounds the lens 12 and protrudes outward (toward the rear side) of the camera housing 1B relative to the eyepiece lens 12. The eyepiece optical system may include multiple lenses. On the rear side of the camera housing 1B, operation members 41 to 43 that accept various operations from the user are also arranged. For example, operation member 41 is a touch panel that accepts touch operations, operation member 42 is an operation lever that can be pushed down in each direction, and operation member 43 is a four-way key that can be pressed in each of four directions. Operation member 41 (touch panel) is equipped with a display panel such as a liquid crystal panel and has the function of displaying images on the display panel.

[0015] FIG. 2 is a cross-sectional view of the camera 1 taken along the YZ plane defined by the Y axis and Z axis shown in FIG. 1(a), and shows the general internal configuration of the camera 1. As shown in FIG.

[0016] The photographing lens unit 1A includes two lenses 101 and 102, an aperture 111, an aperture driver 112, a lens drive motor 113, a lens drive member 114, a photocoupler 115, a pulse plate 116, a mount contact 117, and a focus adjustment circuit 118. The lens drive member 114 is comprised of a drive gear and other components, and the photocoupler 115 detects the rotation of the pulse plate 116, which is linked to the lens drive member 114, and transmits this information to the focus adjustment circuit 118. The focus adjustment circuit 118 drives the lens drive motor 113 based on information from the photocoupler 115 and information from the camera housing 1B (lens drive amount information), thereby moving the lens 101 and changing the focus position. The mount contact 117 is an interface between the photographing lens unit 1A and the camera housing 1B. Note that for simplicity, two lenses 101 and 102 are shown, but in reality, more than two lenses are included in the photographing lens unit 1A.

[0017] The camera housing 1B contains an image sensor 2, a CPU 3, a memory unit 4, a display device 10, a display device drive circuit 11, etc. The image sensor 2 is disposed at the intended imaging plane of the photographing lens unit 1A. The CPU 3 is the central processing unit of the microcomputer, and controls the entire camera 1. The memory unit 4 stores images captured by the image sensor 2, etc. The display device 10 is composed of a liquid crystal display or the like, and displays the captured image (subject image), etc. on the screen (display surface) of the display device 10. The display device drive circuit 11 drives the display device 10.

[0018] The camera housing 1B also contains light sources 13a and 13b, a beam splitter 15, a light-receiving lens 16, an eye image sensor 17, and the like. The light sources 13a and 13b are used to detect the gaze direction from the relationship between the pupil and the image of light reflected by the cornea (corneal reflection image), and are used to illuminate the user's eyeball 14. Specifically, the light sources 13a and 13b are infrared light-emitting diodes or the like that emit infrared light that is insensitive to the user, and are arranged around the eyepiece 12. An optical image of the illuminated eyeball 14 (eye image; an image formed by light emitted from the light sources 13a and 13b and reflected by the eyeball 14) passes through the eyepiece 12 and is reflected by the beam splitter 15. The eye image is then formed by the light-receiving lens 16 on the eye image sensor 17, which is a two-dimensional array of photoelectric elements such as a CCD or CMOS. The light receiving lens 16 positions the pupil of the eyeball 14 and the ocular imaging element 17 in a conjugate imaging relationship. Using a predetermined algorithm described later, the line of sight direction of the eyeball 14 (the viewpoint on the screen of the display device 10) is detected from the position of the corneal reflection image in the eyeball image formed on the ocular imaging element 17.

[0019] 3 is a block diagram showing the electrical configuration within the camera 1. Connected to the CPU 3 are a line-of-sight detection circuit 201, a photometry circuit 202, an autofocus detection circuit 203, a signal input circuit 204, a display device drive circuit 11, a light source drive circuit 205, and the like. The CPU 3 also transmits signals via mount contacts 117 to a focus adjustment circuit 118 disposed within the photographing lens unit 1A and an aperture control circuit 206 included in an aperture drive section 112 within the photographing lens unit 1A. A memory section 4 attached to the CPU 3 has a function of storing image pickup signals from the image sensor 2 and the eye image sensor 17, and a function of storing line-of-sight correction parameters for correcting individual differences in the line of sight, which will be described later. Has.

[0020] The gaze detection circuit 201 A / D converts the output of the eye image pickup element 17 (CCD-EYE) when an eyeball image is formed on the eye image pickup element 17 (eye image obtained by capturing an image of the eye), and transmits the result to the CPU 3. The CPU 3 extracts feature points required for gaze detection from the eye image according to a predetermined algorithm described later, and calculates the user's gaze (the viewpoint on the screen of the display device 10) from the positions of the feature points.

[0021] The photometry circuit 202 amplifies, logarithmically compresses, and A / D converts the signal obtained from the image sensor 2, which also functions as a photometry sensor, specifically the luminance signal corresponding to the brightness of the field, and sends the result to the CPU 3 as field luminance information.

[0022] The autofocus detection circuit 203 A / D converts signal voltages from multiple detection elements (multiple pixels) used for phase difference detection, which are included in the CCD in the image sensor 2, and sends the converted signal to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point from the signals from the multiple detection elements. This is a well-known technique known as image plane phase difference AF. In this embodiment, as an example, it is assumed that there are focus detection points at 180 locations on the image plane corresponding to the 180 locations shown in the viewfinder field of view image (screen of the display device 10) in FIG. 5(a). The CPU 3 acquires lens type information, optical information, and the like, thereby determining the focusable range of the photographing lens unit 1A.

[0023] Furthermore, CPU 3 has a function of selecting a main subject (object) from among the objects within the angle of view and acquiring the position of the selected subject. For example, if the subject is a person, CPU 3 can acquire the area of ​​the person included within the angle of view as a candidate area for the subject from the image obtained by image sensor 2 using a known technology for recognizing a face area or an eye area. Note that if multiple subjects are detected, CPU 3 may select the main subject based on the distance from the center of the angle of view, the size of the subject, etc.

[0024] Furthermore, if an image sensor 2 capable of image plane phase difference AF is used, the CPU 3 can select an object located closer to the user than the background area as the subject based on the distance measurement results in the area within the angle of view.

[0025] Furthermore, the CPU 3 may estimate the subject type by inputting the image obtained from the image sensor 2 into a learning model generated by deep learning such as a convolution neural network (CNN). The CPU 3 can select an estimated object of a specific type, such as a person or an animal, as the subject.

[0026] The method of selecting the subject is not limited to the above example, and any method can be used as long as it can select an object to be the subject from among objects within the angle of view. For example, the CPU 3 can select an object of a type set by the camera 1, a moving object, etc. as the subject.

[0027] Once the CPU 3 has selected the main subject, it can compare the image of the previous frame with the image of the current frame based on the image information obtained from the image sensor 2, and continuously acquire the position of the selected subject using known pattern matching techniques, etc.

[0028] A switch SW1 that is turned on by the first stroke of the release button 5 to start photometry, distance measurement, line of sight detection, etc. of the camera 1, and a switch SW2 that is turned on by the second stroke of the release button 5 to start a photographing operation are connected to the signal input circuit 204. The ON signals from the switches SW1 and SW2 are input to the signal input circuit 204 and sent to the CPU 3.

[0029] The light source drive circuit 205 drives the light sources 13a and 13b.

[0030] Operation members 41 to 43 are configured to transmit their respective operation signals to CPU 3, and control the movement of the detected (acquired) viewpoint in accordance with the received operation signals.

[0031] The configuration for detecting the line of sight of an xR head-mounted display (hereinafter referred to as HMD 400) to which the present invention is applicable will be described with reference to Fig. 4. Fig. 4 is a schematic diagram of the HMD 400. Note that Fig. 4 shows a schematic internal configuration of the HMD 400 on the YZ plane as viewed from the left eye side, but the HMD 400 has a similar configuration on the right eye side as on the left eye side.

[0032] 4, the configuration included inside the housing 403 of the HMD 400 will be described. The HMD 400 includes a CPU 428 and a memory unit 429. The CPU 428 controls the entire HMD 400, and the memory unit 429 records video information.

[0033] The HMD 400 also has a display 406, a display drive circuit 424, and an eyepiece 404 arranged inside a housing 403. The display 406 is configured with a liquid crystal display or the like for displaying images. The display drive circuit 424 drives the display 406. The eyepiece 404 is used by the user to observe the image displayed on the display 406.

[0034] The image sensor 425, aperture mechanism 426, and focus mechanism 427 function as a camera that captures images of the outside. This function allows the user to check the outside while wearing the HMD 400. The image information captured by the image sensor 425 may be replaced with an internal image depending on the use, or may be displayed on the display 406 in combination with the internal image.

[0035] The illumination light source 420 is a light source that irradiates light onto the eyeball 401 for gaze detection. The illumination light source 420 includes, for example, a plurality of infrared light emitting diodes, and is arranged around the eyepiece 404. An image of the eyeball illuminated with light and an image formed by corneal reflection of the light source pass through the eyepiece 404, are reflected by a light splitter 421, and are formed by a light receiving lens 422 on an eyeball image sensor 423, which is made up of a two-dimensionally arranged array of photoelectric elements such as CMOS.

[0036] The light receiving lens 422 positions the pupil of the user's eyeball 401 and the eyeball image sensor 423 in a complementary imaging relationship. The gaze direction is detected according to a predetermined algorithm, which will be described later, from the positional relationship between the eyeball image formed on the eyeball image sensor 423 and the image resulting from the corneal reflection of the illumination light source 420. The illumination light source 420, the light splitter 421, the light receiving lens 422, and the eyeball image sensor 423 function as a gaze detector. The memory unit 429 stores image signals from the image sensor 425 and the eyeball image sensor 423, gaze correction data, etc.

[0037] FIG. 5(a) is a diagram showing the field of view within the viewfinder, and shows the state in which the display device 10 is operating (the state in which an image is displayed). As shown in FIG. 5(a), the field of view within the viewfinder includes a focus detection area 500, 180 ranging point indices 501, a field of view mask 502, and the like. Each of the 180 ranging point indices 501 is displayed superimposed on a through image (live view image) displayed on the display device 10 so as to be displayed at a position corresponding to a focus detection point on the imaging surface. Furthermore, of the 180 ranging point indices 501, the ranging point indices 501 that corresponds to the user's current viewpoint A (gaze position) in the through image is displayed highlighted with a frame or the like.

[0038] <Explanation of gaze detection operation> Using Figures 6, 7(a), 7(b), and 8, we will explain the gaze detection method (gaze detection algorithm). FIG. 6 is a diagram illustrating the principle of the gaze detection method, and is a schematic diagram of an optical system for performing gaze detection. As shown in FIG. 6, light sources 13a and 13b are arranged approximately symmetrically with respect to the optical axis of light-receiving lens 16, and illuminate user's eyeball 14. A portion of the light emitted from light sources 13a and 13b and reflected by eyeball 14 is collected by light-receiving lens 16 onto ocular imaging element 17. FIG. 7(a) is a schematic diagram of an eye image captured by ocular imaging element 17 (eyeball image projected onto ocular imaging element 17), and FIG. 7(b) is a diagram illustrating the output intensity of the CCD in ocular imaging element 17. FIG. 8 is a schematic flowchart of the gaze detection operation.

[0039] 8, light sources 13a and 13b emit infrared light toward user's eyeball 14. An image of the user's eyeball illuminated by the infrared light is formed on ocular imaging element 17 through light receiving lens 16 and is photoelectrically converted by ocular imaging element 17. As a result, a processable electrical signal of the eye image is obtained.

[0040] In step S802, the line-of-sight detection circuit 201 sends the eye image (eye image signal; electric signal of the eye image) obtained from the eye imaging device 17 to the CPU 3.

[0041] In step S803, the CPU 3 obtains the coordinates of the points corresponding to the corneal reflection images Pd and Pe of the light sources 13a and 13b and the pupil center c from the eye image obtained in step S802.

[0042] Infrared light emitted from light sources 13a and 13b illuminates cornea 142 of user's eyeball 14. At this time, corneal reflection images Pd and Pe formed by part of the infrared light reflected from the surface of cornea 142 are collected by light receiving lens 16 and formed on ocular imaging element 17 as corneal reflection images Pd' and Pe' in the eye image. Similarly, light beams from edges a and b of pupil 141 are also formed on ocular imaging element 17 as pupil edge images a' and b' in the eye image.

[0043] FIG. 7(b) shows luminance information (luminance distribution) of region α' in the eye image of FIG. 7(a). In FIG. 7(b), the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In this embodiment, the X-axis (horizontal) coordinates of corneal reflection images Pd', Pe' are set to Xd, Xe, and the X-axis coordinates of pupil edge images a', b' are set to Xa, Xb. As shown in FIG. 7(b), an extremely high level of luminance is obtained at the coordinates Xd, Xe of the corneal reflection images Pd', Pe'. In the region from coordinate Xb to coordinate Xa, which corresponds to the region of the pupil 141 (the region of the pupil image obtained when the light beam from the pupil 141 is focused on the ocular imaging element 17), an extremely low level of luminance is obtained except for coordinates Xd, Xe. A luminance intermediate between the two types of luminance is obtained in the region of iris 143 outside pupil 141 (the region of the iris image outside the pupil image obtained by focusing the light beam from iris 143). Specifically, a luminance intermediate between the two types of luminance is obtained in the region where the X coordinate (coordinate in the X-axis direction) is smaller than coordinate Xb and the region where the X coordinate is larger than coordinate Xa.

[0044] From the luminance distribution shown in FIG. 7(b), the X-coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X-coordinates Xa and Xb of the pupil edge images a' and b' can be obtained. Specifically, the coordinates of the corneal reflection images Pd' and Pe' can be obtained as the coordinates of extremely high luminance, and the coordinates of the pupil edge images a' and b' can be obtained as the coordinates of extremely low luminance. Furthermore, when the rotation angle θx of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16 is small, the coordinate Xc of the pupil center image c' (center of the pupil image) obtained when the light beam from the pupil center c is focused on the ocular imaging element 17 can be expressed as Xc ≒ (Xa + Xb) / 2. In other words, the coordinate Xc of the pupil center image c' can be calculated from the X-coordinates Xa and Xb of the pupil edge images a' and b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupil center image c' can be estimated.

[0045] In step S804, the CPU 3 calculates the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position of the eyeball 14 relative to the light receiving lens 16, and is used to calculate the corneal reflection images Pd' and Pe'. It can be calculated using a function of the interval (Xd-Xe).

[0046] In step S805, CPU 3 calculates the rotation angle of the optical axis of eyeball 14 relative to the optical axis of light receiving lens 16. The X coordinate of the midpoint between corneal reflection images Pd and Pe and the X coordinate of the center of curvature O of cornea 142 approximately coincide. Therefore, if the standard distance from the center of curvature O of cornea 142 to the center c of pupil 141 is Oc, then the rotation angle θx of eyeball 14 in the ZX plane (plane perpendicular to the Y axis) can be calculated using the following equation 1. The rotation angle θy of eyeball 14 in the ZY plane (plane perpendicular to the X axis) can also be calculated using a method similar to that for calculating rotation angle θx. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc (Formula 1)

[0047] In step S806, CPU 3 uses the rotation angles θx, θy calculated in step S805 to determine (acquire) the user's viewpoint (gaze position; the position where the user is looking) on ​​the screen of display device 10. If the coordinates (Hx, Hy) of the viewpoint are coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the viewpoint can be calculated by the following equations 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0048] Parameter m in equations 2 and 3 is a constant determined by the configuration of the viewfinder optical system (light receiving lens 16, etc.) of camera 1, and is a conversion coefficient that converts rotation angles θx and θy into coordinates corresponding to pupil center c on the screen of display device 10. Parameter m is determined in advance and stored in memory unit 4. Parameters Ax, Bx, Ay, and By are gaze correction parameters that correct for individual differences in gaze, and are acquired by performing a calibration operation, which will be described later. Parameters Ax, Bx, Ay, and By are stored in memory unit 4 before the gaze detection operation begins.

[0049] In step S807, CPU 3 stores the coordinates (Hx, Hy) of the viewpoint in memory unit 4, and ends the gaze detection operation. Note that, although Fig. 8 shows an example in which the rotation angle of the eyeball is obtained using the corneal reflection images of light sources 13a and 13b, and the coordinates of the viewpoint on display device 10 are obtained, this is not limiting. A method of obtaining the rotation angle of the eyeball from the eyeball image may also be a method of measuring the gaze from the pupil center position.

[0050] <Calibration procedure explanation> As described above, the gaze detection operation obtains the rotation angles θx, θy of the eyeball 14 from the eye image, and converts the position of the pupil center c into a position on the screen of the display device 10 by coordinate conversion, thereby detecting the viewpoint.

[0051] However, due to factors such as individual differences in the shape of the human eyeball, the viewpoint may not be detected with high accuracy. Specifically, if the gaze correction parameters Ax, Ay, Bx, and By are not adjusted to values ​​appropriate for the user, a discrepancy will occur between the actual viewpoint B and the detected viewpoint C, as shown in Figure 5(b). In Figure 5(b), the user is gazing at a person, but camera 1 mistakenly detects that the user is gazing at the background, resulting in an inability to perform appropriate focus detection and adjustment.

[0052] Therefore, before the camera 1 captures an image, it is necessary to perform a calibration operation to obtain viewpoint correction parameters suitable for the user and store them in the camera 1.

[0053] Conventionally, calibration work has been performed by highlighting multiple indices at different positions on the screen of the display device 10 before capturing an image, as shown in FIG. 5(c), and having the user look at the indices. A known technique involves performing a gaze detection operation when the user gazes at each indices, and determining viewpoint correction parameters suitable for the user from the detected (acquired) multiple viewpoints (gaze positions) and the coordinates of each indices. Note that the method for displaying the indices is not particularly limited as long as it suggests the position where the user should look. The indices may be displayed as graphics, or the indices may be displayed by changing the brightness or color of an image (such as a captured image).

[0054] Control of gaze responsiveness to the user's gaze will be described with reference to Figs. 9 to 11. Gaze responsiveness is the responsiveness of a change in the detected position of the gaze to a change in the direction of the user's actual gaze. Figs. 9 and 10 are diagrams illustrating an example of a gaze index (an index indicating a detected gaze position) when a subject moves. Camera 1 performs display control to indicate the gaze position in an image by the gaze index in response to the movement of the user's gaze. Gaze responsiveness can also be said to be the responsiveness of the movement of the gaze index to the movement of the user's gaze.

[0055] Fig. 9 is a diagram illustrating a case where gaze responsiveness is set lower than standard. Fig. 9(a) shows a gaze index 901 before a subject 910 starts moving. Fig. 9(b) shows a gaze index 902 while the subject 910 is moving. Fig. 9(c) shows a gaze index 903 after the subject 910 has moved a predetermined time or a predetermined distance.

[0056] In Fig. 9(a), the subject 910 has not yet started moving, and the gaze indicator 901 is overlapping the position of the subject 910. In Fig. 9(b), even though the subject 910 has started moving, the gaze responsiveness is set low, so the gaze indicator 902 does not track the position of the subject 910. In Fig. 9(c), after the subject 910 has completed moving for a predetermined time or a predetermined distance, the gaze indicator 903 tracks the position of the subject 910 with a delay.

[0057] If the gaze responsiveness is set lower than standard, the indicator indicating the user's gaze position may not track the position of the subject 910 while the subject 910 is moving. If the movement of the indicator indicating the user's gaze position lags behind the subject 910, the focus of the camera 1 may not be aligned with the subject 910 intended by the user.

[0058] 10A and 10B are diagrams illustrating a case where the gaze responsiveness is set higher than the standard. It is assumed that the subject 1010 does not move during the time that passes from FIG. 10A to FIG. 10C.

[0059] In FIG. 10(a), the gaze index 1001 is overlapped with the subject 1010. In FIG. 10(b), even though the subject 1010 is not moving, the gaze index 1002 is displayed at a position displaced from the subject 1010. Since the human eye makes minute vibrations (hereinafter referred to as fixational eye movement) even when looking at one point, if the gaze responsiveness is set high, the gaze index 1002 will move due to the influence of fixational eye movement. Even after that, the influence of fixational eye movement remains, as shown in FIG. 10(c), and the gaze index 1003 will be displayed at a displaced position.

[0060] When the gaze responsiveness is set higher than standard, even if the subject 1010 is stationary, the influence of fixational eye movement may cause the index indicating the user's gaze position to deviate from the position of the subject 1010. If the index indicating the user's gaze position deviates from the position of the subject 1010, the focus of the camera 1 may not be aligned with the subject 1010 intended by the user.

[0061] In order to suppress the influence of fixational eye movement explained in Figure 10, we applied a filter such as Infinite Impulse Response (IIR) to the amount of time-dependent change in the gaze point (gaze position). By applying an IIR filter, gaze responsiveness can be suppressed. The IIR filter calculates the output discrete-time signal y[n] by multiplying the past N outputs y[n-1],...,y[nN] and the current and past N inputs x[n],...,x[nN] by preset coefficients. Since the IIR filter is a well-known technique, a detailed description will be omitted. The coefficients of the IIR filter correspond to the filter parameters in this embodiment, and the filter characteristics are determined by the parameters. By changing the filter parameters, the CPU 3 can suppress the influence of fixational eye movement and change the gaze responsiveness.

[0062] The present invention is not limited to the above filters, and any time-domain filter that suppresses temporal fluctuations at the current time can be applied. For example, the CPU 3 may suppress gaze responsiveness using a filter that performs moving averaging. Specifically, the viewpoint positions in the last five frames are defined as X(t-5), X(t-4), X(t-3), X(t-2), and X(t-1), and the viewpoint position in the current frame is defined as X(t). The focus position is obtained as a time-domain moving average, such as X'(t) = {X(t-5) + X(t-4) + X(t-3) + X(t-2) + X(t-1) + X(t)} / 5. Even if a sudden viewpoint change occurs at the X-coordinate X(t) of the current frame, the influence of the viewpoint change is reduced at the focus position X'(t) by taking the time-domain moving average of the viewpoint positions X(t-5) to X(t-1) in the last five frames.

[0063] In this way, by applying a time-direction filter to the amount of change in the viewpoint position, the CPU 3 can suppress the influence of unintentional reflexive eye movements by the user on the detected eye position (viewpoint).

[0064] The operation of changing the gaze responsiveness will be described with reference to Fig. 11. The processing of each step in the flowchart shown in Fig. 11 is executed by the CPU 3 of the camera housing 1B.

[0065] In step S1101, the CPU 3 determines whether or not the user is looking at the display device 10 of the camera housing 1B. Whether or not the user is looking at the display device 10 can be determined, for example, by detecting the eye coming close to and moving away from the viewfinder using a sensor (not shown). If it is determined that the user is looking at the display device 10, the process proceeds to step S1102. If it is determined that the user is not looking at the display device 10, the process of step S1101 is repeated until the user's line of sight is detected.

[0066] In step S1102, the CPU 3 acquires the gaze position using the gaze detection algorithm described above. The gaze position here is the position detected on the display device 10 as the position at which the user is looking. The display device 10 displays an image captured by the image sensor 2, and the CPU 3 can determine which subject the user is looking at. The gaze position here is the gaze position that has been filtered in the time direction, such as an IIR filter. The CPU 3 can display the acquired gaze position, for example, as gaze index 901 or gaze index 902 shown in FIG. 9.

[0067] Note that the CPU 3 may omit the process of step S1101 and determine whether or not the gaze position has been acquired after step S1102. If it is determined that the gaze position has been acquired, the process proceeds to step S1103, and if not, the process proceeds to step S1102.

[0068] In step S1103, CPU 3 detects the subject (object) that the user is looking at based on the image captured by image sensor 2 and the line of sight position acquired in step S1102, and acquires the position of the detected subject.

[0069] In step S1104, the CPU 3 changes the parameters of the IIR filter. The CPU 3 changes the parameters of the IIR filter based on, for example, the amount of deviation ΔX between the amount of movement of the gaze position that moves in a predetermined time and the amount of movement of the subject (object).

[0070] The larger the deviation ΔX, the less the gaze position and the subject position coincide. For this reason, as the deviation ΔX and the subject's movement amount increase, the filter parameters are set so that the gaze responsiveness increases. Also, as the deviation ΔX increases and the subject's movement amount decreases, the filter parameters are set so that the gaze responsiveness decreases. Also, as the deviation ΔX decreases, the filter parameters are no longer changed. Note that the parameter information changes uniquely with respect to the deviation ΔX.

[0071] As described above, according to the above embodiment, camera 1 can adjust gaze responsiveness by changing filter parameters, thereby suppressing shifts in gaze position due to fixational eye movement, while increasing gaze responsiveness and enabling the gaze position to follow the movement of the subject.

[0072] <Variation 1> 11 shows a method for changing gaze responsiveness by changing the filter characteristics (parameters) based on the displacement amount ΔX. In Modification 1, the camera 1 controls the filter characteristics by comparing the displacement amount ΔX with a threshold value.

[0073] The operation of changing the gaze responsiveness using a threshold value will be described with reference to Fig. 12. The processing of each step in the flowchart shown in Fig. 12 is executed by the CPU 3 of the camera housing 1B. The processing from step S1201 to step S1203 is the same as the processing from step S1101 to step S1103 in Fig. 11, and therefore description thereof will be omitted.

[0074] In step S1204, CPU 3 determines whether the deviation ΔX between the amount of movement of the gaze position that moves over a predetermined time period and the amount of movement of the subject (object) is greater than a predetermined threshold Xth. The predetermined threshold Xth is set according to gaze responsiveness, which is the responsiveness with which the movement of the user's gaze is reflected in the gaze position. The higher the gaze responsiveness, the larger the value set as the predetermined threshold Xth.

[0075] If the deviation ΔX between the amount of movement of the gaze position and the amount of movement of the subject does not exceed the predetermined threshold Xth, the gaze responsiveness is deemed appropriate and the processing in Fig. 12 ends. If the deviation ΔX between the amount of movement of the gaze position and the amount of movement of the subject exceeds the predetermined threshold Xth, the processing proceeds to step S1205.

[0076] In step S1205, CPU 3 determines whether the subject is moving. If the deviation amount ΔX exceeds a predetermined threshold Xth and the subject is moving, it can be determined that the user's line of sight is a line of sight movement to track the position of the subject, and that the line of sight reaction speed has decreased. On the other hand, if the deviation amount ΔX exceeds a predetermined threshold Xth and the subject is not moving, it can be determined that the user's line of sight is fixational eye movement. That is, in steps S1206 and S1207, the eye movement responsiveness is adjusted based on whether the user's line of sight is fixational eye movement or eye movement.

[0077] For example, if the amount of movement of the subject in a predetermined time is greater than a predetermined amount, CPU 3 can determine that the subject is moving. If the subject is moving, for example, if the state changes from Figure 9(a) to Figure 9(b), processing proceeds to step S1206. If the subject is not moving, for example, if the state changes from Figure 10(a) to Figure 10(b), processing proceeds to step S1207.

[0078] In step S1206, CPU 3 increases the gaze responsiveness. In step S1206, since the deviation ΔX is greater than the predetermined threshold Xth and the subject is moving, CPU 3 sets the gaze responsiveness higher than the current value, thereby making it possible to make the gaze position follow the movement of the subject and suppress the deviation ΔX.

[0079] In step S1207, CPU 3 lowers the gaze responsiveness. In step S1207, since the deviation amount ΔX is greater than the predetermined threshold Xth and the subject is not moving, CPU 3 sets the gaze responsiveness lower than the current value, thereby suppressing deviation between the gaze position and the subject position due to fixational eye movement.

[0080] As described above, according to Modification 1, camera 1 controls the filter characteristics and changes the gaze responsiveness by comparing the deviation amount ΔX with a threshold value. Camera 1 also changes the gaze responsiveness by determining whether the subject is moving. When the subject is moving, the deviation amount ΔX increases as the gaze reaction speed slows. Furthermore, when the subject is not moving, deviation amount ΔX may increase due to the influence of fixational eye movement or reflexive gaze movement. Therefore, camera 1 can adjust the gaze responsiveness by taking into account the reaction speed of the user's gaze to the subject.

[0081] <Variation 2> The gaze responsiveness may be adjusted by setting appropriate parameters according to the type of subject to be photographed or the type of photographic scene. In Modification 2, camera 1 has a function of determining the type of subject and the type of photographic scene, and adjusts the gaze responsiveness by applying appropriate filter characteristics (parameters) based on the determined type information. This allows camera 1 to appropriately track the gaze position of the subject intended by the user according to the type of subject or photographic scene.

[0082] The adjustment of gaze responsiveness based on the type of subject and the type of shooting scene may be performed when shooting starts, when the subject or shooting scene changes, or at predetermined intervals (for example, every 5 minutes). The type of subject and the type of shooting scene may be determined by analyzing brightness information of the image, or may be determined based on the setting information of the camera 1.

[0083] An example of setting gaze responsiveness according to the type of photographic scene will be described with reference to Fig. 13. Fig. 13 is a diagram showing an example of setting information of gaze responsiveness for each photographic scene. Fig. 13 describes setting gaze responsiveness for each photographic scene related to sports.

[0084] The gaze responsiveness and parameters are determined in advance for each photographic scene and are stored in a memory area (memory unit 4 in FIGS. 2 and 3) of camera 1. The information on the gaze responsiveness and parameters for each photographic scene stored in memory unit 4 is read out when the subject or photographic scene is identified, and is used to adjust the gaze responsiveness.

[0085] The setting information of the gaze responsiveness includes information on a photographed scene 1301, gaze responsiveness 1302, and parameters 1303, and each piece of information is stored in the memory unit 4 in association with each other. The photographed scene 1301 is information indicating a scene detected by the CPU 3. The gaze responsiveness 1302 is information indicating the setting of the gaze responsiveness in the photographed scene 1301. The parameters 1303 are specific filter characteristics or parameter setting values ​​in the photographed scene 1301.

[0086] When the gaze response is set using an IIR filter, the information of the parameter 1303 corresponds to a setting value that determines the cutoff frequency of the IIR filter. When a filter in the time direction is used, the information can be the number of frames of data to be used, i.e., the number of frames from the most recent frame to be used.

[0087] In the example of FIG. 13 , gaze responsiveness 1302 is defined in three levels: “high,” “medium,” and “low.” Parameter 1303 indicates “K1,” “K2,” and “K3” as identifiers for identifying the parameter values ​​corresponding to each gaze responsiveness 1302. When the photographed scene 1301 is “soccer,” the direction and amount of movement of the players are random, so the deviation ΔX tends to be large, and the gaze responsiveness is set to “high.” In contrast, when the photographed scene 1301 is “athletics” or “skating,” the direction and amount of movement of the players are predictable, so the deviation ΔX is likely to be smaller than in “soccer,” and the gaze responsiveness is set to “low.” In this way, by changing the gaze responsiveness according to the type of photographed scene, CPU 3 can cause the gaze position to appropriately track the position of the subject.

[0088] The gaze detection interval may be changed depending on the gaze responsiveness. For example, by shortening the gaze detection interval when the gaze responsiveness is increased and lengthening the gaze detection interval when the gaze responsiveness is decreased, it is possible to reduce power consumption when the gaze responsiveness is decreased.

[0089] Furthermore, in Modification 2, an example has been described in which the gaze responsiveness is set according to the shooting scene, but this is not limiting, and the CPU 3 may change the gaze responsiveness according to the type of subject, such as a photographic subject. For example, for subjects whose moving direction changes randomly and rapidly, such as children and dogs, the deviation amount ΔX tends to be large, and the gaze responsiveness needs to be set higher in order to make the gaze position follow the movement of the subject.

[0090] In contrast, when the subject moves in a constant direction and does not change much, such as an adult or a horse, the user can predict the subject's movement, so the deviation ΔX tends to be small. In this case, CPU 3 is required to reduce the gaze responsiveness and ignore the deviation caused by reflexive gaze movement, thereby stably tracking the gaze position with the subject's movement.

[0091] As described above, in Modification 2, camera 1 adjusts gaze responsiveness according to the type of subject being photographed. Because gaze responsiveness is changed according to the characteristics of the movement of the subject, camera 1 can adjust the gaze position to follow the movement of the subject and achieve stable focusing on the subject.

[0092] <Variation 3> The gaze responsiveness may be adjusted by setting appropriate parameters for each user. In Modification 3, the CPU 3 adjusts the gaze responsiveness according to information specific to the user (hereinafter also referred to as user information) such as age or gaze reaction speed to a subject.

[0093] The speed of gaze response to a subject tends to decrease with age. For this reason, the CPU 3 may adjust the gaze responsiveness according to information such as the user's age that is stored in advance in the memory unit 4.

[0094] The CPU 3 may also adjust the gaze responsiveness according to the speed at which the user's gaze responds to the subject. The gaze response speed to the subject can be evaluated, for example, during a calibration process. The CPU 3 may evaluate the speed at which the user's gaze responds when the indicator is moved during the calibration process.

[0095] The adjustment of the gaze responsiveness based on the user information may be performed, for example, when the image capture is started. The adjustment of the gaze responsiveness based on the user information may also be performed at predetermined intervals (for example, every 30 minutes) in consideration of changes in the user's physical condition or changes in the photographer. stomach.

[0096] Furthermore, the CPU 3 may adjust the gaze responsiveness by combining the modification 3 with the modification 2. For example, when setting the gaze responsiveness according to the type of subject or the type of shooting scene, the CPU 3 may set the filter parameters by multiplying the filter parameters by a weighting coefficient according to the user information. Specifically, when the gaze response speed to the subject is slower than the standard speed, the CPU 3 weights the filter parameters so that the gaze responsiveness is high.

[0097] As described above, in variant example 3, gaze responsiveness is adjusted according to information such as gaze reaction speed, which differs for each user, so that camera 1 can appropriately adjust gaze responsiveness to suit each user.

[0098] In the above embodiment, the deviation amount ΔX is described as the deviation amount between the movement amount of the line of sight position that moves in a predetermined time and the movement amount of the object position, but is not limited to this. The deviation amount ΔX may be the deviation amount between the movement time of the line of sight position that moves a predetermined distance and the movement time of the object position.

[0099] As described above, according to the embodiment and each modification, the camera 1 having a gaze detection function can adjust the responsiveness of gaze detection in consideration of the reaction speed of the user's gaze to a subject.

[0100] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0101] <<Application examples to other electronic devices>> 14(a) is an external view of a notebook personal computer 1410 (notebook PC). In FIG. 14(a), an imaging unit 1415 that captures an image of a user looking at a display unit 1411 of the notebook PC 1410 is connected to the notebook PC 1410, and the notebook PC 1410 acquires the imaging results from the imaging unit 1415. The notebook PC 1410 then detects the user's line of sight based on the imaging results. The imaging unit 1415 may perform the line of sight detection. The present invention is also applicable to the notebook PC 1410 and the imaging unit 1415.

[0102] Fig. 14(b) is an external view of a smartphone 1420. In Fig. 14(b), the smartphone 1420 detects the line of sight of a user looking at a display unit 1422 of the smartphone 1420 based on the imaging results of an in-camera 1421 (front camera). The present invention is also applicable to the smartphone 1420. Similarly, the present invention is also applicable to various tablet terminals.

[0103] FIG. 14(c) is an external view of a game console 1430. In FIG. 14(c), a head-mounted display 1435 (HMD) that displays a VR (Virtual Reality) image of a game on a display unit 1436 is connected to the game console 1430. The HMD 1435 has a camera 1437 that captures an image of the eyes of a user wearing the HMD 1435, and the game console 1430 acquires the captured image from the HMD 1435. The game console 1430 then detects the user's line of sight based on the captured image. The HMD 1435 may perform the line of sight detection. The present invention is applicable to the game console 1430 and the HMD 1435. Just as the present invention is applicable to viewing VR images displayed on an HMD, the present invention is also applicable to viewing AR (Augmented Reality) images displayed on the lens portion of a glasses-type wearable terminal. Just as the present invention is applicable to VR technology and AR technology, the present invention is also applicable to other xR technologies such as MR (Mixed Reality) technology and SR (Substitutional Reality) technology.

[0104] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0105] 1: Camera 3: CPU

Claims

1. a first acquisition means for detecting a user's line of sight and acquiring a line of sight position of the user in an image displayed on a display unit; a second acquisition means for acquiring a position of an object in the image; a display control means for controlling the display unit to display an index indicating the gaze position acquired by the first acquisition means in the image; an adjustment means for adjusting the responsiveness of the movement of the indicator to the movement of the user's line of sight; a control means for controlling the adjustment by the adjustment means based on a deviation between the amount of movement of the line of sight position that moves in a predetermined time and the amount of movement of the position of the object, or a deviation between the movement time of the line of sight position that moves a predetermined distance and the movement time of the position of the object; and When the deviation amount of the movement amount or the deviation amount of the movement time is greater than a predetermined threshold, the control means (1) when the object is moving, controlling the adjusting means to increase the responsiveness; (2) When the object is not moving, the adjusting means is controlled to lower the responsiveness. An electronic device characterized by:

2. 2. The electronic device according to claim 1, wherein the control means controls the adjustment means not to adjust the responsiveness when the deviation amount of the movement amount or the deviation amount of the movement time is not greater than the predetermined threshold value.

3. When the deviation amount of the movement amount or the deviation amount of the movement time is greater than the predetermined threshold value, the control means (1) if the object is moving, it is determined that the line of sight movement is for tracking the position of the object, and the adjusting means is controlled to increase the responsiveness; (2) If the object is not moving, the adjustment means is controlled to determine that the movement of the user's line of sight is fixational eye movement and to lower the responsiveness.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

4. The first acquisition means shortens the detection interval of the user's line of sight when the responsiveness is increased, and lengthens the detection interval of the user's line of sight when the responsiveness is decreased.

4. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

5. a correction unit for acquiring a correction parameter for correcting a deviation between the gaze position of the user and the gaze position acquired by the first acquisition unit; The adjustment means sets the responsiveness using the gaze position corrected by the correction parameter.

5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

6. The adjusting means adjusts the responsiveness using an IIR (Infinite Impulse Response) filter or a filter that performs a moving average in the time direction.

6. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

7. a first acquisition step of detecting a user's line of sight and acquiring a line of sight position of the user in an image displayed on a display unit; a second acquiring step of acquiring a position of an object in the image; a display control step of controlling the display unit to display an index indicating the gaze position acquired in the first acquisition step in the image; an adjustment step of adjusting responsiveness of the movement of the indicator to the movement of the user's line of sight; a control step of controlling the adjustment in the adjusting step based on a deviation amount between the movement amount of the line of sight position that moves in a predetermined time and the movement amount of the position of the object, or a deviation amount between the movement time of the line of sight position that moves a predetermined distance and the movement time of the position of the object, In the control step, when the deviation amount of the movement amount or the deviation amount of the movement time is greater than a predetermined threshold value, (1) When the object is moving, the responsiveness is controlled to be high; (2) When the object is not moving, the response is controlled to be low. A method for controlling an electronic device.

8. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 6.

Images