Electronic apparatus, method of controlling the same, and storage medium

Abstract

A shake or a delay occurs in an image displayed based on the line-of-sight position.SOLUTION: An information processing apparatus comprising: an acquisition unit configured to acquire line-of-sight information including a line-of-sight position of a user viewing a display unit; and a correction unit configured to correct the line-of-sight position acquired by the acquisition unit by filter processing. An electronic apparatus comprising: a state determination unit configured to determine at least two or more line-of-sight states; and a control unit configured to perform control to execute processing for displaying an image based on a line-of-sight position corrected by a correction unit, wherein the control unit performs control to change a parameter of filter processing used for correction by the correction unit based on the line-of-sight state determined by the state determination unit.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an electronic device, a control method thereof, and a program. [Background technology]

[0002] In recent years, electronic devices that use user gaze information as a user interface have been used in various fields. In such electronic devices, a gaze pointer is displayed at the gaze position, allowing the user to visually confirm the results of gaze detection. However, if the gaze detection results are used directly to display the gaze pointer, the gaze pointer will shake significantly if there is significant fixational eye movement or if there is significant variance in the gaze detection results, resulting in an uncomfortable display for the user.

[0003] Patent Document 1 describes an electronic device that uses processed gaze information generated by processing (averaging, etc.) detected gaze positions to display the gaze position. Patent Document 2 also discloses a processing device that corrects gaze information by a Kalman filter using a low-order polynomial regression equation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-132272 [Patent Document 2] Japanese Patent Application Publication No. 2023-4678 Summary of the Invention [Problem to be solved by the invention]

[0005] Even if processed gaze information generated by processing such as averaging, as in the electronic device disclosed in the above-mentioned Patent Document 1, is used to display the gaze position, the shaking of the gaze pointer displayed at the gaze position may be noticeable.

[0006] Furthermore, when a Kalman filter is used as in the processing device disclosed in Patent Document 2, a large delay may occur in the gaze pointer that is displayed at the gaze position based on the filter processing result.

[0007] Therefore, an object of the present invention is to suppress shaking and delays in images displayed based on the gaze position. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, the electronic device of the present invention comprises an acquisition means for acquiring gaze information including the gaze position of a user looking at a display means, a correction means for correcting the gaze position acquired by the acquisition means by filter processing, a state discrimination means for discriminating at least two or more gaze states depending on the gaze speed and gaze movement amount included in the gaze information acquired by the acquisition means, and a control means for controlling the execution of a process to display an image based on the gaze position corrected by the correction means, and is characterized in that the control means controls the parameters of the filter processing used for correction by the correction means to be changed based on the gaze state discriminated by the state discrimination means. [Effects of the Invention]

[0009] According to the present invention, shaking and delay of an image displayed based on the gaze position can be suppressed. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a perspective view showing an example of the appearance of a digital still camera 1, which is an example of an imaging device according to the present embodiment. [Figure 2] 1 is a cross-sectional view showing an example of the configuration of a digital still camera 1, which is an example of an imaging device according to the present embodiment. [Figure 3] 1 is a block diagram illustrating an example of the configuration of a digital still camera 1, which is an example of an imaging device according to the present embodiment, focusing on the electrical circuitry. [Figure 4]3A and 3B are diagrams showing examples of finder images observed through an eyepiece 12 according to the present embodiment. [Figure 5] FIG. 2 is a diagram illustrating the principle of gaze detection. [Figure 6] 1A is a schematic diagram of an eyeball image formed by the light receiving lens 16, and FIG. 1B is a schematic diagram of the luminance distribution in the region α. [Figure 7] 10 is a flowchart illustrating a gaze detection process according to the present embodiment. [Figure 8] 10 is a flowchart illustrating a process of determining a state of the line of sight and a process of determining a process by a Kalman filter suitable for each state according to the present embodiment. [Figure 9] 10 is a flowchart of a jump determination process according to the present embodiment. [Figure 10] 10 is a flowchart of a tracking / fixation determination process according to the present embodiment. [Figure 11] FIG. 2 is a schematic diagram for explaining line-of-sight movement according to the present embodiment. [Figure 12] 10 is a flowchart of Kalman filter processing according to the present embodiment. [Figure 13] 10 is an example of a display screen for setting the strength of a Kalman filter by a user according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

[0012] <Description of appearance> FIG. 1 is a perspective view showing an example of the appearance of a digital still camera (hereinafter referred to as "camera") 1 according to this embodiment, where FIG. 1(a) is a perspective view seen from the front side, and FIG. 1(b) is a perspective view seen from the back side.

[0013] 1, an XYZ Cartesian coordinate system is defined as the camera coordinate system, with the optical axis of lens unit 1A as the Z axis, the vertical axis perpendicular to the Z axis as the Y axis, and the axis perpendicular to the Z and Y axes as the X axis. Note that the origin of the camera coordinate system may be, for example, the intersection of the imaging plane and the optical axis, but is not limited to this.

[0014] Camera 1 has camera body 1B and lens unit 1A that can be attached to or detached from camera body 1B. Release button 5 is an operating member that accepts image capture instructions from the user. Operating members such as release button 5 will hereinafter be referred to as the "operating unit." An eyepiece 12 is located on the back of camera 1, allowing the user to look into a display element (described later) contained within camera 1. The user can view a field of view image by looking into eyepiece 12.

[0015] <Configuration explanation> Fig. 2 is a cross-sectional view showing an example of the configuration of camera 1, which is an example of an imaging device according to this embodiment. Fig. 2 is a cross-sectional view of camera 1 cut along the YZ plane formed by the Y axis and Z axis shown in Fig. 1. In Fig. 1 and Fig. 2, corresponding components are denoted by the same numbers.

[0016] When lens unit 1A is attached to camera body 1B, lens unit 1A and camera body 1B are electrically connected via mount contacts 117. Power is supplied to lens unit 1A from camera body 1B via mount contacts 117. In addition, the circuitry within lens unit 1A and CPU 3 of camera body 1B can communicate via mount contacts 117.

[0017] Lens unit 1A has a movable lens 101 and a fixed lens 102. Although each is shown as a single lens in Fig. 2, in reality it is made up of multiple lenses. Here, movable lens 101 is assumed to be a focus lens, but other movable lenses such as a magnification lens or a blur correction lens may also be included.

[0018] Movable lens 101 is supported by lens drive member 114 and driven in the optical axis direction (left and right in the drawing) by lens drive motor 113. Rotation of pulse plate 116, which is linked to lens drive member 114, is detected by photocoupler 115 and output to focus adjustment circuit 118. Focus adjustment circuit 118 can detect the amount and direction of drive of movable lens 101 based on the output of photocoupler 115. When the amount and direction of drive of movable lens 101 are instructed by CPU 3 of camera body 1B, focus adjustment circuit 118 controls the operation of lens drive motor 113 based on the output of photocoupler 115.

[0019] In the camera body 1B, the image sensor 2 is a CCD image sensor or a CMOS image sensor. The image sensor 2 has a plurality of pixels arranged two-dimensionally, each of which is provided with one microlens, one color filter, and one or more photoelectric conversion units. In this embodiment, each pixel is provided with multiple photoelectric conversion units, and a signal can be read out from each photoelectric conversion unit. By configuring the pixels in this manner, a captured image, a parallax image pair, and an image signal for phase-difference AF can be generated from the signal read out from the image sensor 2. The image sensor 2 converts the optical image formed by the lens unit 1A into a pixel signal group (analog image signal) through photoelectric conversion by the multiple pixels. In this embodiment, the image sensor 2 also has an A / D conversion function, converting the analog image signal into digital image data and outputting it.

[0020] The memory unit 4 has a non-volatile memory (ROM) and a volatile memory (RAM). The CPU 3 loads a program stored in the ROM into the RAM and executes it, thereby controlling the operation of the camera body 1B and the lens unit 1A and realizing the functions of the camera. The memory unit 4 also includes a recording medium (such as a memory card) for recording image data and audio data obtained by shooting. The CPU 3 controls the operation of the focus adjustment circuit 118 and the aperture drive unit 112 via the mount contact 117.

[0021] The nonvolatile memory of the memory unit 4 may be rewritable. The nonvolatile memory stores programs executed by the CPU 3, various setting values, image data of a GUI (Graphical User Interface), gaze correction data for correcting individual differences in gaze, and the like.

[0022] The display element 10 is an LCD (Liquid Crystal Display) or an organic EL display panel, and displays captured images such as live view images, menu screens, various types of information, and the like.

[0023] Display element drive circuit 11 drives display element 10 under the control of CPU 3. Since display element 10 is provided inside camera body 1B, an eyepiece 119 is provided for observing display element 10 from outside camera body 1B. Eyepiece 119 is provided with eyepiece lens 12 and illumination light sources 13a to 13f for detecting the line of sight. Eyepiece 119 also is provided with optical splitter 15 for capturing an image of the eye, light receiving lens 16, and eye image sensor 17.

[0024] Illumination light sources 13a to 13f are a plurality of infrared LEDs provided around eyepiece lens 12, and illuminate eyeball 14 of a user looking through the eyepiece with infrared light. An eyeball image obtained by reflecting the infrared light from illumination light sources 13a to 13f on eyeball 14 is reflected by light splitter 15 and captured by eyeball imaging element 17 via light receiving lens 16 provided above. Light receiving lens 16 positions the pupil of user's eyeball 14 and eyeball imaging element 17 in a conjugate imaging relationship. Eyeball imaging element 17 has a plurality of pixels arranged two-dimensionally and is configured to capture an image using infrared light. The number of pixels in eyeball imaging element 17 may be fewer than the number of pixels in imaging element 2. The line of sight of eyeball 14 can be detected based on the positional relationship between the corneal reflection and the pupil in the eyeball image obtained by eyeball imaging element 17.

[0025] <Block diagram explanation> 3 is a block diagram illustrating an example of the configuration of the camera 1 of this embodiment, focusing on the electrical circuits. Connected to the CPU 3 are a line-of-sight detection circuit 201, a photometry circuit 202, an autofocus detection circuit 203, an operation unit 204, a display element drive circuit 11, an illumination light source drive circuit 205, and a liquid crystal display unit 120. In addition, a focus adjustment circuit 118 and an aperture control circuit 206 (included in the aperture drive unit 112) provided in the photographic lens 1 are electrically connected to the CPU 3 via mount contacts 117.

[0026] The gaze detection circuit 210 A / D converts the analog image signal of the eyeball image obtained from the eyeball image sensor 17 and transmits it as digital image data to the CPU 3. The CPU 3 detects feature points required for gaze detection from the digital image data of the eyeball image according to a known algorithm, and detects the position of the user's gaze from the position of each feature point.

[0027] The photometry circuit 202 generates brightness information as a predetermined evaluation value for exposure control based on image data obtained from the image sensor 2 and outputs it to the CPU 3. The CPU 3 performs automatic exposure control (AE) processing based on the brightness information and determines the shooting conditions. For example, the shooting conditions for still image shooting are the shutter speed, aperture value, and sensitivity. The CPU 3 controls the aperture value (opening size) of the aperture 111 of the photographing lens 1 based on the determined shooting conditions. The CPU 3 also controls the operation of the mechanical shutter within the camera body 1B.

[0028] The autofocus detection circuit 203 generates an image signal for phase-difference AF based on image data obtained from the image sensor 2 and outputs it to the CPU 3. The CPU 3 calculates the defocus amount based on the phase difference of the image signal for phase-difference AF. This is a well-known technique known as image-surface phase-difference AF. In this embodiment, as an example, it is assumed that there are 180 focus detection points on the image surface corresponding to the locations shown in the viewfinder image (described below) in FIG. 4, but this is not limiting.

[0029] The operation unit 204 is a collective term for a plurality of input devices (buttons, switches, dials, etc.) that can be operated by the user, including the previously described release button 5. When the CPU 3 detects an operation of an input device, it executes processing according to the detected operation.

[0030] The release button 5 has a first shutter switch (SW1) that turns on when pressed halfway, and a second shutter switch (SW2) that turns on when pressed all the way. When the CPU 3 detects that SW1 is turned on, it executes preparatory operations for still image capture. These preparatory operations include AE ​​processing and AF processing. When the CPU 3 detects that SW2 is turned on, it executes still image capture and recording operations according to the shooting conditions determined by the AE processing.

[0031] The illumination light source drive circuit 205 controls the light emission operations of the illumination light sources 13a to 13f under the control of the CPU 3.

[0032] The liquid crystal display unit 120 displays information according to a signal from the CPU 3 on a display such as an LCD or an organic EL display.

[0033] Fig. 4 is a diagram showing an example of a viewfinder image according to this embodiment. Here, the viewfinder image is an image displayed on the display element 10, with various indicators superimposed on it. The user can observe the viewfinder image of Fig. 4 through the eyepiece 12. Fig. 4 is a diagram showing the viewfinder field, with the display element 10 in operation.

[0034] In Fig. 4, 300 denotes a field of view mask, 400 denotes an index indicating the range where focus detection is possible, and 4001 to 4180 denote 180 indexes (AF frames) displayed at positions corresponding to points where focus detection is possible (focus detection points). Of these AF frames, the AF frame corresponding to the current gaze position is highlighted as an estimated gaze position A. Here, the AF frame A displayed highlighted in Fig. 4 is an image displayed based on the gaze position.

[0035] <Explanation of gaze detection processing> The gaze detection process will be described with reference to FIGS.

[0036] Fig. 5 is a diagram illustrating the principle of gaze detection. Illumination light sources 13a to 13f are arranged approximately symmetrically with respect to the optical axis of light receiving lens 16, and irradiate infrared light onto user's eyeball 14. Only illumination light sources 13a and 13b are shown in Fig. 5. Light receiving lens 16 forms an eyeball image on the imaging surface of eyeball imaging element 17 using the infrared light reflected by eyeball 14.

[0037] FIG. 6(a) is a schematic diagram of an eyeball image formed by light receiving lens 16, and FIG. 6(b) is a schematic diagram of the luminance distribution in region α of FIG. 6(a).

[0038] FIG. 7 is a flowchart of the gaze detection process according to this embodiment. The gaze detection process can be executed, for example, when it is detected that an object is approaching the eyepiece 12. The proximity of an object to the eyepiece 12 can be detected using any known method, such as using a proximity sensor provided near the eyepiece 12. The gaze detection operation may be started in response to a user instruction via the operation unit 204. The process in FIG. 7 is executed by the CPU 3 controlling each unit.

[0039] In S701, CPU 3 turns on one or more of illumination light sources 13a to 13f via illumination light source drive circuit 205. For convenience, it is assumed here that illumination light sources 13a and 13b shown in FIG. 5 are turned on. This causes infrared light to be emitted from illumination light sources 13a and 13b toward the outside of camera body 1B. The infrared light is reflected by the eyeball of the user looking through eyepiece 12, and is further reflected by light splitter 15 before entering light receiving lens 16.

[0040] In S702, CPU 3 performs imaging using eyeball image sensor 17. An eyeball image formed by light receiving lens 16 is converted into an image signal by eyeball image sensor 17. The image signal is A / D converted by gaze detection circuit 201 and input to CPU 3 as eyeball image data.

[0041] In S703, the CPU 3 obtains the coordinates of the corneal reflection images Pd' and Pe' of the illumination light sources 13a and 13b and the coordinates of the image point c' of the pupil center c from the eyeball image data acquired in S702. The eyeball image acquired by the eyeball imaging element 17 includes the reflection images Pd' and Pe' corresponding to the images Pd and Pe of the illumination light sources 13a and 13b that are reflected on the cornea 142 (FIG. 6(a)).

[0042] 6(a), the horizontal direction is the X-axis and the vertical direction is the Y-axis. Here, the X-axis coordinates of the centers of the reflected images Pd' and Pe' of illumination light sources 13a and 13b included in the eyeball image are designated Xd and Xe. Furthermore, the X-axis coordinates of the images a' and b' of pupil edges a and b, which are the edges of pupil 141, are designated Xa and Xb.

[0043] 6(b), the luminance at coordinates Xd and Xe corresponding to the reflected images Pd' and Pe' of illumination light sources 13a and 13b is much higher than the luminance at other positions. On the other hand, the luminance in the range from coordinates Xa to Xb corresponding to the region of pupil 141 is much lower except for coordinates Xd and Xe. In addition, in the range of coordinates smaller than Xa and larger than Xb corresponding to the region of iris 143 outside pupil 141, the luminance is intermediate between the luminance of the reflected images of the illumination light sources and the luminance of the pupil.

[0044] Based on these brightness level characteristics in the X-axis direction, CPU 3 can detect, from the eyeball image, the X-axis coordinates Xd and Xe of the reflected images Pd' and Pe' of illumination light sources 13a and 13b, and the X-axis coordinates Xa and Xb of the images a' and b' of pupil edges a and b. Furthermore, in applications such as this embodiment, the rotation angle θx of the optical axis of eyeball 14 relative to the optical axis of light receiving lens 16 is relatively small. In such cases, the X-axis coordinate Xc of the image c' of pupil center c in the eyeball image can be expressed as Xc ≒ (Xa + Xb) / 2. In this way, CPU 3 can determine, from the eyeball image, the coordinates of the reflected images Pd' and Pe' of illumination light sources 13a and 13b, and the X-axis coordinate of the image c' of pupil center c.

[0045] In S704, CPU 3 calculates the imaging magnification β of the eyeball image. β is a magnification determined by the position of eyeball 14 relative to light receiving lens 16, and can be calculated as a function of the distance (Xd-Xe) between the reflected images Pd' and Pe' of the illumination light source.

[0046] In S705, the CPU 3 calculates the rotation angle of the eyeball. The X-axis coordinate of the midpoint of the images Pd and Pe of the illumination light source on the cornea 142 approximately coincides with the X-axis coordinate of the center of curvature O of the cornea 142. Therefore, if the standard distance from the center of curvature O of the cornea 142 to the center c of the pupil 141 is Oc, the rotation angle θx of the optical axis of the eyeball 14 in the ZX plane can be found from the relational expression β*Oc*SINθx≒{(Xd+Xe) / 2}-Xc.

[0047] 5 and 6 show an example of calculating the rotation angle θx in a plane perpendicular to the Y axis, but the rotation angle θy in a plane perpendicular to the X axis can also be calculated in a similar manner. In this way, the CPU 3 determines the rotation angles θx and θy of the eyeballs. The gaze position can be calculated from the rotation angles of the eyeballs.

[0048] In S706, the CPU 3 acquires a correction coefficient from the memory unit 4. The correction coefficient is a coefficient that corrects for individual differences in the user's gaze. The correction coefficient is generated by a calibration operation and stored in the memory unit 4 before starting the gaze detection process. When the memory unit 4 stores correction coefficients for multiple users, the correction coefficient corresponding to the current user is used at any timing, for example, by inquiring of the user.

[0049] In S707, the CPU 3 uses the rotation angles θx and θy of the eyeball calculated in S705 to calculate the user's gaze coordinates (gaze position) on the display element 10. Furthermore, assuming that the user's gaze position is the coordinates (Hx, Hy) corresponding to the center c of the pupil 141 on the display element 10, it can be calculated by the formulas Hx=m×(Ax×θx+Bx) and Hy=m×(Ay×θy+By).

[0050] Here, coefficient m is a conversion coefficient that converts rotation angles θx and θy into coordinates corresponding to the center c of the pupil 141 on the display element 10, and is determined by the characteristics of the eyepiece lens 12 of the viewfinder optical system of the camera. Coefficient m can be stored in advance in memory unit 4. Furthermore, Ax, Bx, Ay, and By are correction coefficients acquired in S706.

[0051] In S708, CPU 3 determines whether the acquired results of the gaze detection process are valid values. Here, the gaze position (Hx, Hy), the interval (Xd-Xe) between the reflected images Pd' and Pe', and the interval (Xa-Xb) between the pupil edge images a' and b' are acquired as the results of the gaze detection process. For example, in a typical human eye, the size of the pupil is 2 to 6 mm, and the size of the cornea is approximately 12 mm in diameter. When the predetermined threshold is set to a value determined based on the size of the pupil and cornea of ​​the human eye, if the value of the interval (Xa-Xb) between the pupil edge images a' and b' and the value of the interval (Xd-Xe) between the reflected images Pd' and Pe' acquired as the results of the gaze detection process are greater than the predetermined threshold, the CPU 3 determines that the values ​​are valid. If the values ​​are equal to or less than the predetermined threshold, the CPU 3 determines that the values ​​are valid. If the values ​​are determined to be invalid, the CPU 3 proceeds to S711. If the values ​​are determined to be valid, the CPU 3 proceeds to S709.

[0052] Furthermore, when the predetermined threshold is set to a value determined based on the size of the EVF screen used, if the value of the gaze position (Hx, Hy) obtained from the gaze detection result is greater than the predetermined threshold, it is determined to be an invalid value. If it is equal to or less than the predetermined threshold, it is determined to be a valid value. If it is determined to be an invalid value, the process proceeds to S711, and if it is determined to be a valid value, the process proceeds to S709. Note that if any one of the indexes of the value of the distance (Xa-Xb) between the pupil edge images a', b', the value of the distance (Xd-Xe) between the reflection images Pd', Pe', and the gaze position (Hx, Hy) is greater than the predetermined threshold, the process proceeds to S711.

[0053] In S709, CPU 3 compares the average value of the results of the past gaze detection processes with the value of the result of the gaze detection process at the current time, and determines whether the value is abnormal based on the comparison result. Specifically, the CPU 3 determines whether the value (change rate) obtained by dividing the difference between the average value of the results of the past gaze detection processes and the result of the current gaze detection process by the average value of the results of the past gaze detection processes is abnormal. If the change rate obtained from the result of the gaze detection process is greater than a predetermined threshold, where the predetermined threshold is determined based on the speed of a person's pupil contraction and the speed of eye movement, the change is determined to be abnormal. If the change rate is equal to or less than the predetermined threshold, the change is determined to be not abnormal. If the change (value) is determined to be abnormal, the process proceeds to S711. If the change (value) is not determined to be abnormal, the process proceeds to S710. The values ​​to be compared are the coordinates of the reflection images Pd', Pe', the coordinates of the image c' of the pupil center c, and the gaze position (Hx, Hy), which are included in the result of the gaze detection process. If any one of the coordinates of the reflected images Pd', Pe', the coordinates of the image c' of the pupil center c, and the gaze position (Hx, Hy) is greater than a predetermined threshold, the process proceeds to S711.

[0054] In S710, the CPU 3 performs Kalman filter processing, which will be described later. The calculated gaze position on the display element 10 is affected by gaze detection errors and fixational eye movement. If the detected gaze position is directly displayed on the display element 10 as the gaze position, the fluctuation of the gaze position displayed on the display element 10 increases proportionally when the gaze detection variability is large. As a result, when changing the AF frames 4001 to 4003 and 4180 shown in FIG. 4 based on the gaze position, it may not be possible to change to the desired AF frame. Furthermore, when displaying an image such as a gaze pointer based on the gaze position, the display may be unpleasant for the user. Therefore, it is necessary to perform Kalman filter processing on the calculated gaze position (Hx, Hy) to suppress the variability in gaze detection and suppress the fluctuation of the gaze position.

[0055] In S711, the CPU 3 performs interpolation processing. For example, in the interpolation processing in S711, linear interpolation is performed using the past gaze position (Hx, Hy) to predict the current gaze position. Then, the predicted gaze position is set as the current gaze position.

[0056] In S712, the CPU 3 records the gaze position after the Kalman filter processing or the interpolation processing in the memory unit 4, and ends the gaze detection processing.

[0057] As described above, when the gaze position is detected and stored in the memory unit 4, the CPU 3 displays an gaze pointer indicating the gaze position on the display element 10, or highlights the AF frame corresponding to the gaze position, based on the gaze position information stored in the memory unit 4.

[0058] CPU 3 converts the gaze position stored in memory unit 4 into coordinates corresponding to center c of pupil 141 on display element 10, using coefficient m determined by the characteristics of eyepiece lens 12 of the camera's viewfinder optical system. CPU 3 then generates an image centered on the converted gaze coordinates (gaze position), and displays the image on display element 10. The image displayed here includes a gaze pointer indicating the gaze position, and highlighting of subject detection frames, AF frames, and setting items determined based on the gaze position.

[0059] Furthermore, based on the detected gaze position, the focus detection point (AF frame) used by the auto focus detection circuit 201 is changed, and AF processing is performed in response to a user operation.

[0060] <Filter processing explanation> Figures 8 to 10 are flowcharts for explaining in detail the filter processing of S710 in Figure 7. Specifically, these are flowcharts for performing processing using a Kalman filter suitable for the gaze state. Gaze behavior can be roughly classified into three states (fixation state, following state, and jumping state), and the optimal filter processing changes for each state.

[0061] The fixation state represents a state in which the subject is gazing at one point. The pursuit state represents a state in which the subject is slowly following an object with the eyes. The jump state represents a state in which the gaze is shifted significantly, i.e., a state in which saccadic eye movement is occurring. Figure 8 is a flowchart showing the process of determining the gaze state and the Kalman filter processing appropriate for each state.

[0062] In S801, the CPU 3 performs a jump determination process, which will be described with reference to FIG.

[0063] In S802, the CPU 3 determines whether or not it is in a jumping state based on the result of the jumping determination process. If it is determined that it is in a jumping state, the process proceeds to S807, and if not, the process proceeds to S803.

[0064] In S803, CPU 3 updates the time series information by adding information about the gaze position at the current time that is determined not to be in a jumping state to information about past gazes (time series information). That is, CPU 3 stores time series information including information about the gaze position at the current time in memory unit 4. When updating the time series information, if the storage capacity of memory unit 4 is full, information about the oldest gaze position is deleted, and information about the gaze position at the current time is stored as time series information. Here, the time series information specifically includes the gaze position (Hx, Hy), the coordinates of the reflected images Pd', Pe', and the coordinates of the image c' of the pupil center c after gaze detection processing, or after filter processing or interpolation processing if these processing have already been performed.

[0065] In S804, the CPU 3 performs a tracking / fixation determination process, which will be described with reference to FIG.

[0066] In S805, the CPU 3 performs a model change process to change the model of the Kalman filter by switching the parameters used in the Kalman filter based on the determined line of sight state. Here, the CPU 3 initializes the variables used in the Kalman filter and performs a model change process to change the coefficient G of the state space model set in the processing by the Kalman filter based on the line of sight state. t , F tThe variables and coefficients used in the Kalman filter are explained in Figure 12.

[0067] In S806, the CPU 3 performs processing using a Kalman filter.

[0068] In S807, the CPU 3 resets information (time-series information) relating to gaze positions prior to the current gaze position determined to be in a jumping state. That is, the CPU 3 deletes the time-series information stored in the memory unit 4, and returns to a state in which no time-series information is stored.

[0069] <Jump Judgment Processing> 9 is a flowchart illustrating the jump determination process, which uses multiple frames older than the current frame.

[0070] In S901, the CPU 3 determines whether or not the jump determination flag is set. If it is determined that the jump determination flag is not set, the process proceeds to S902. If it is determined that the jump determination flag is set, the process proceeds to S910. The jump determination flag being set means that it has been determined that there is a high possibility that a jump state has occurred in a past frame. The jump determination flag not being set means that no past frames have been determined to be in a jump state.

[0071] In S902, CPU 3 compares the gaze position after the previous Kalman filter processing or interpolation processing with the gaze position at the current time to calculate the gaze movement amount. In this case, since there are no past frames that have been determined to be in a jumping state, the gaze position after the Kalman filter processing or interpolation processing, which is less susceptible to the influence of noise and gaze detection errors, is used.

[0072] In S903, the CPU 3 determines whether the amount of line-of-sight movement is equal to or greater than a predetermined threshold, and if it determines that the amount of line-of-sight movement is equal to or greater than the predetermined threshold, proceeds to S905, and if not, proceeds to S904.

[0073] In S904, the CPU 3 determines that the state is not a jumping state, and ends the jumping determination process.

[0074] In S905, the CPU 3 sets the jump frame count=1 and the jump determination in progress flag to True.

[0075] In S906, the CPU 3 determines whether or not the number of jump frames is equal to or greater than a predetermined threshold, and if it determines that the number of jump frames is equal to or greater than the predetermined threshold, proceeds to S907, and if not, proceeds to S909.

[0076] In S907, the CPU 3 sets the jump frame count to 0 and the jump determination flag to False.

[0077] In S908, the CPU 3 determines that the state is a jumping state.

[0078] In S909, the CPU 3 determines that the state is not a jumping state.

[0079] In S910, CPU 3 compares the gaze position at the previous time before Kalman filter processing or interpolation processing with the gaze position at the current time to calculate the gaze movement amount. In this case, because it was determined in the past frame that there was a high possibility of a jump state, it is necessary to confirm whether the determination that there was a high possibility of a jump state was correct. Therefore, when calculating the gaze movement amount, the gaze position at the previous time before Kalman filter processing or interpolation processing, which includes the effects of noise and gaze detection errors, is used.

[0080] In S911, the CPU 3 determines whether the amount of line-of-sight movement is equal to or greater than a predetermined threshold, and if it determines that the amount of line-of-sight movement is equal to or greater than the predetermined threshold, proceeds to S913, and if not, proceeds to S912.

[0081] In S912, CPU 3 adds 1 to the jump frame count and proceeds to S906. If a jump has been made, the amount of gaze movement will be a small value. Therefore, if the amount of gaze movement is less than a predetermined threshold, it is determined that a jump has been made, and proceeds to S906.

[0082] In S913, the CPU 3 sets the jump frame count to 0 and the jump determination flag to False. If the gaze movement amount is equal to or greater than a predetermined threshold, it is possible that a jump state has been erroneously determined due to noise or gaze detection error, and the process proceeds to S914.

[0083] In S914, the CPU 3 determines that the state is not a jumping state.

[0084] <Follow-up and fixation determination processing> FIG. 10 is a flowchart illustrating the tracking / fixation determination process in S804 of FIG.

[0085] In S1001, the CPU 3 calculates the line-of-sight velocity using the line-of-sight position (Hx, Hy) from the time-series information stored in the memory unit 4.

[0086] In S1002, the CPU 3 calculates the pupil velocity using the coordinates of the image c' of the pupil center c from the time-series information.

[0087] In S1003, the CPU 3 calculates the corneal velocity using the coordinates of the reflected images Pd' and Pe' from the time-series information.

[0088] In S1004, CPU 3 determines whether the gaze velocity, pupil velocity, and corneal velocity are each equal to or greater than their respective thresholds. If all of the gaze velocity, pupil velocity, and corneal velocity are equal to or greater than their respective thresholds, the process proceeds to S1006. If at least one of the gaze velocity, pupil velocity, and corneal velocity is not equal to or greater than the threshold, the process proceeds to S1005. If the gaze position changes, the pupil and cornea positions also change, so taking all of the gaze velocity, pupil velocity, and corneal velocity into consideration is less susceptible to the influence of noise and gaze detection errors than taking only the gaze velocity into consideration.

[0089] In S1005, the CPU 3 determines that the state is a fixation state, and ends the tracking / fixation determination process.

[0090] In S1006, CPU 3 determines whether the radial velocity is a positive value greater than or equal to 0. If the radial velocity is a positive value greater than or equal to 0, the process proceeds to S1010, and if the radial velocity is a negative value less than 0, the process proceeds to S1007.

[0091] In S1007, CPU 3 determines whether the current line of sight is a minus direction tracking state. If it is determined that it is a minus direction tracking state, the process proceeds to S1009, and if it is determined that it is not a minus direction tracking state, the process proceeds to S1008.

[0092] 11 is a schematic diagram for explaining gaze movement according to this embodiment. As shown in FIG. 11(a), when a direction change occurs during gaze tracking, if gaze detection can be performed at a sufficient frame rate, the gaze velocity approaches 0, and the timing at which a fixation state occurs can be detected. Therefore, even when a state transition in which the pursuit movement switches between positive (plus direction) and negative (minus direction) is detected as shown in FIG. 11(b), a fixation state will occur when switching between the plus direction pursuit state and the minus direction pursuit state.

[0093] In S1008, the CPU 3 determines that the state is a plus direction tracking state, and ends the tracking / fixation determination process.

[0094] In S1009, the CPU 3 determines that the state is a fixation state, and ends the tracking / fixation determination process.

[0095] In S1010, CPU 3 determines whether the current line of sight is in a plus direction tracking state. If it is determined that the line of sight is in a plus direction tracking state, the process proceeds to S1012, and if it is determined that the line of sight is not in a minus direction tracking state, the process proceeds to S1011.

[0096] In S1011, the CPU 3 determines that the state is a minus direction tracking state, and ends the tracking / fixation determination process.

[0097] In S1012, the CPU 3 determines that the state is a fixation state, and ends the tracking / fixation determination process.

[0098] <About Kalman filtering> FIG. 12 is a flowchart of the Kalman filter process.

[0099] The Kalman filter is a method for calculating the current gaze position using past observations through a statistical method. The state space model is defined as shown in Equation (1). x t =G t x t-1 +w t y t =F t x t +v t …(1)

[0100] where x t is the state vector, y t is the observation vector, G t , F t is the known coefficient, w t is the modeling error, v t represents the observation error, and w t and v t are known w t and v t is a normal white noise with mean 0 and variance.

[0101] Here, the known value of G t , F t changes depending on the line of sight.

[0102] In S1201, the CPU 3 calculates the prior estimate a at time t. t is calculated using equation (2). a t =G t m t-1 …(2)

[0103] where m t-1 is the state estimate at time t-1. m t-1 The element of includes the gaze position after filtering or interpolation one time before.

[0104] In S1202, the CPU 3 calculates the calculated prior estimate a t Using the a priori error covariance matrix R t is calculated using equation (3).

[0105]

number

[0106] where C t-1 is the a posteriori error covariance matrix at time t-1.

[0107] In S1203, the Kalman gain K is calculated using equation (4). t Calculate.

[0108]

number

[0109] In S1204, the state estimate m t Calculate. m t =a t +K t (H t -F t a t ) …(5)

[0110] where H t is the gaze position at time t before filtering or interpolation. t The elements of contain the gaze position after filtering or interpolation.

[0111] In S1206, the CPU 3 calculates the a posteriori error covariance matrix at time t using equation (6), and ends the Kalman filter processing. C t =R t -K t F t R t …(6)

[0112] Coefficient G of the state space model set in processing by the Kalman filter t , F t is determined by the gaze state. When the gaze state is a fixation state, the gaze is fixed on one point, so the continuous prediction model is used to estimate that the predicted value is the same as the current gaze position. In this case, the coefficient G t , F t When the gaze state is a tracking state or a jumping state, it is assumed that the gaze is moving at a constant speed, and either the constant speed motion model or the constant speed motion model is used. In this case, the coefficient G t , F t is expressed as equation (7).

[0113]

number

[0114] Here, dt is the time difference between time t and time t-1.

[0115] In this embodiment, the state space model is set as described above, but this is not limitative, and a model including, for example, an acceleration term may also be used.

[0116] As the values ​​of Wt and Vt in equation (3) or (4) increase, the cutoff frequency takes on a lower value. A lower cutoff frequency means that only signals in a lower frequency band are allowed to pass, and therefore the strength of the Kalman filter becomes stronger. On the other hand, as the values ​​of Wt and Vt in equation (3) or (4) decrease, the cutoff frequency takes on a higher value. A higher cutoff frequency means that signals in a higher frequency band are also allowed to pass, and therefore the strength of the Kalman filter becomes weaker.

[0117] Increasing the strength of the Kalman filter reduces fluctuations in the image displayed in response to changes in gaze position, but increases the delay between the change in gaze position and the image being displayed based on the gaze position. Therefore, the user can switch between prioritizing reducing fluctuations in the image displayed in response to changes in gaze position and reducing the delay between the change in gaze position and the image being displayed based on the gaze position, by changing the user's settings.

[0118] FIG. 13 is an example of a display screen for a user to set the strength of the Kalman filter according to this embodiment. This screen allows the user to adjust the sensitivity of the gaze pointer movement. When the user operates the operation unit 204 to display the gaze pointer sensitivity setting screen, the screen shown in FIG. 13 is displayed on the LCD display unit 120. When the user operates the operation unit 204 and moves the selection bar on the display screen from 0 to the minus side, the strength of the Kalman filter is weakened, and the delay can be reduced. In this case, the fluctuation of the gaze pointer at the displayed gaze position relative to changes in gaze position increases. On the other hand, when the selection bar is moved from 0 to the plus side, the strength of the Kalman filter is strengthened, and the fluctuation of the gaze pointer at the gaze position can be reduced. In this case, the delay between the change in gaze position and the display of an image indicating the gaze position increases.

[0119] If a jump state is determined when the user has set a small delay, the observed value, state estimate m, and a posteriori error covariance matrix up until the jump state is determined are reset.

[0120] <Other embodiments> The various controls described above as being performed by CPU3 may be performed by a single piece of hardware, or the entire device may be controlled by multiple pieces of hardware (e.g., multiple processors or circuits) sharing the processing.

[0121] Furthermore, although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0122] The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage media storing the program constitute the present invention.

[0123] The disclosure of this embodiment includes the following configurations and methods.

[0124] (Configuration 1) acquisition means for acquiring line-of-sight information including a line-of-sight position of a user looking at the display means; a correction means for correcting the gaze position acquired by the acquisition means through a filter process; a state determination means for determining at least two or more gaze states according to a gaze speed and a gaze movement amount included in the gaze information acquired by the acquisition means; and control means for controlling the execution of a process for displaying an image based on the gaze position corrected by the correction means, the control means controls the correction means to change parameters of the filtering process used for correction based on the line-of-sight state determined by the state determination means. An electronic device characterized by:

[0125] (Configuration 2) 2. The electronic device according to configuration 1, wherein the filtering process is a process using a Kalman filter.

[0126] (Configuration 3) 3. The electronic device according to configuration 1 or 2, wherein the control means controls the Kalman filter so as to change a model based on the line-of-sight state determined by the state determination means.

[0127] (Configuration 4) The electronic device described in any one of configurations 1 to 3, wherein the state determination means determines that the state is one of a jumping state, a fixation state, and a tracking state based on the gaze velocity and the gaze movement amount in the gaze information acquired by the acquisition means.

[0128] (Configuration 5) The electronic device according to any one of configurations 1 to 4, wherein the control means controls the Kalman filter to change the model to a sustained prediction model when the state determination means determines that the state is a fixation state.

[0129] (Configuration 6) The electronic device according to any one of configurations 1 to 4, characterized in that when the state determination means determines that the state is a following state or a jumping state, the control means switches the model of the Kalman filter to either a uniform motion model or a uniform motion model.

[0130] (Configuration 7) The electronic device includes: The system further includes a setting unit for setting an image to be displayed based on the gaze position of the user, The control means 7. The electronic device according to any one of configurations 1 to 6, wherein the strength of the filter is controlled to be set in accordance with the state of the setting by the setting means.

[0131] (Configuration 8) 8. The electronic device according to configuration 7, wherein the setting is a setting regarding a delay until the image is displayed at a position based on the gaze position acquired by the acquisition means.

[0132] (Configuration 9) 9. The electronic device according to configuration 8, wherein the stronger the strength of the filter, the greater the delay until the image is displayed at a position based on the gaze position acquired by the acquisition means.

[0133] (Configuration 10) The electronic device according to configuration 7, characterized in that when the state determination means determines that the device is in a jumping state, the electronic device controls to display an image based on the gaze position without changing the value of the gaze position acquired by the acquisition means.

[0134] (Configuration 11) The electronic device described in any one of configurations 1 to 10, wherein the control means controls to display a gaze pointer or a highlighted AF frame at a position based on the gaze position corrected by the correction means.

[0135] (Configuration 12) an acquisition step of acquiring gaze information including a gaze position of a user looking at the display means; a correction step of correcting the gaze position acquired by the acquisition means by filtering; a state determining step of determining at least two or more gaze states according to a gaze speed and a movement amount based on the gaze position acquired in the acquiring step; a control step of controlling to execute a process of displaying an image based on the gaze position corrected in the correction step, the control step controls to change a parameter of a filter process used for correction by the correction step based on the line-of-sight state determined by the state determination means. 10. A method for controlling an electronic device comprising:

[0136] (Configuration 13) 13. A program for causing a computer to execute the method for controlling an electronic device according to claim 12.

[0137] (Configuration 14) A computer-readable storage medium having recorded thereon a program for causing a computer to execute the method for controlling an electronic device according to configuration 12.

Claims

1. acquisition means for acquiring line-of-sight information including a line-of-sight position of a user looking at the display means; a correction means for correcting the gaze position acquired by the acquisition means through a filter process; a state determination means for determining at least two or more gaze states according to a gaze speed and a gaze movement amount included in the gaze information acquired by the acquisition means; and control means for controlling the execution of a process for displaying an image based on the gaze position corrected by the correction means, the control means controls the correction means to change parameters of the filtering process used for correction based on the line-of-sight state determined by the state determination means. An electronic device characterized by:

2. 2. The electronic device according to claim 1, wherein the filtering process is a process using a Kalman filter.

3. 2. The electronic device according to claim 1, wherein the control means controls the Kalman filter so as to change a model based on the line-of-sight state determined by the state determination means.

4. 2. The electronic device according to claim 1, wherein the state determination means determines whether the state is one of a jumping state, a fixation state, or a tracking state based on the gaze velocity and the gaze movement amount in the gaze information acquired by the acquisition means.

5. 4. The electronic device according to claim 3, wherein the control means controls the Kalman filter to change the model to a sustained prediction model when the state determination means determines that the state is a fixation state.

6. 4. The electronic device according to claim 3, wherein the control means switches the model of the Kalman filter to either a uniform motion model or a uniform motion model when the state determination means determines that the state is a following state or a jumping state.

7. The electronic device includes: The system further includes a setting unit for setting an image to be displayed based on the gaze position of the user, The control means 2. The electronic device according to claim 1, wherein the strength of the filter is controlled to be set in accordance with the state of the setting by the setting means.

8. 8. The electronic device according to claim 7, wherein the setting is a setting regarding a delay until the image is displayed at a position based on the gaze position acquired by the acquisition means.

9. 9. The electronic device according to claim 8, wherein the stronger the strength of the filter, the greater the delay until the image is displayed at the position based on the gaze position acquired by the acquisition means.

10. The electronic device according to claim 7, characterized in that when the state determination means determines that the jumping state is present, the electronic device is controlled to display an image based on the gaze position without changing the value of the gaze position acquired by the acquisition means.

11. 2. The electronic device according to claim 1, wherein the control means controls the display of a gaze pointer or a highlighted AF frame at a position based on the gaze position corrected by the correction means.

12. an acquisition step of acquiring gaze information including a gaze position of a user looking at the display means; a correction step of correcting the gaze position acquired by the acquisition means by filtering; a state determining step of determining at least two or more gaze states according to a gaze velocity and a movement amount based on the gaze position acquired in the acquiring step; a control step of controlling to execute a process of displaying an image based on the gaze position corrected in the correction step, the control step controls to change a parameter of a filter process used for correction by the correction step based on the line-of-sight state determined by the state determination means.

10. A method for controlling an electronic device comprising:

13. A program for causing a computer to execute the method for controlling an electronic device according to claim 12.

14. A computer-readable storage medium having recorded thereon a program for causing a computer to execute the electronic device control method according to claim 12.

Images