Gaze estimation system, gaze estimation method, and computer program

Abstract

To provide a visual line estimation system, a visual line estimation method, and a computer program which properly detect movement of eyes of a target person.SOLUTION: A visual line estimation system 10 according to the present invention has a display control unit 101 for display a fixation point gazed by a target person so that a first period in which the fixation point moves at a predetermined speed and a second period in which the fixation point stops at a predetermined period of time are repeatedly switched, movement detection means for detecting movement of eyes of the target person from an image of the target person, a pursuit determination unit 110 for determining, based on the movement of the eyes, whether or not the eyes of the target person follow the fixation point every time the fixation point stops, and an impersonation detection unit 201 for detecting impersonation as the target person based on the determination result of the pursuit determination unit 110.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present disclosure relates to the technical fields of a gaze estimation system, a gaze estimation method, and a computer program that estimate the gaze of a subject. [Background technology]

[0002] Known systems of this type detect the gaze of a subject. For example, Patent Document 1 discloses a technology in which predetermined visual stimulus information is displayed on a display and the gaze position is estimated from the visual stimulus information and the subject's eye movement. Patent Document 2 discloses a technology in which a detection process for detecting the gaze, such as a corneal reflex method, is performed using each of images captured at a predetermined frame rate. Patent Document 3 discloses a technology in which a gaze detection process is performed using an image of a user received from an imaging device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-024608 [Patent Document 2] Japanese Patent Application Laid-Open No. 2017-107546 [Patent Document 3] Japanese Patent Application Publication No. 2018-015218 Summary of the Invention [Problem to be solved by the invention]

[0004] The method of asking a subject to gaze at a fixed gaze point not only places a heavy burden on the subject, but also may induce unconscious gaze wandering. The above-mentioned patent documents do not adequately address gaze wandering, and there is room for improvement.

[0005] An object of the present disclosure is to provide a gaze estimation system, a gaze estimation method, and a computer program for solving the above-mentioned problems. [Means for solving the problem]

[0006] One aspect of the gaze estimation system of this disclosure includes a display control means for displaying a gaze point at which a subject is gazing so that it moves repeatedly between a first period in which it moves at a predetermined speed and a second period in which the gaze point is stopped for a predetermined period; a detection means for detecting the eye movement of the subject from an image of the subject; a tracking determination means for determining whether the subject's eyes are tracking the gaze point each time the gaze point stops based on the eye movement; and an impersonation detection means for detecting impersonation of the subject based on the determination result of the tracking determination means.

[0007] One aspect of the gaze estimation method disclosed herein displays a gaze point that a subject is gazing at, which alternates between a first period in which it moves at a predetermined speed and a second period in which the gaze point is stopped for a predetermined period; detects the subject's eye movement from an image of the subject; determines whether the subject's eyes are following the gaze point each time the gaze point stops based on the eye movement; and detects impersonation of the subject based on the determination result of whether the subject's eyes are following the gaze point.

[0008] One aspect of the computer program of this disclosure operates a computer to display a point of gaze at which a subject is gazing, repeatedly moving between a first period in which the point of gaze moves at a predetermined speed and a second period in which the point of gaze is stopped for a predetermined period, detect eye movement of the subject from an image of the subject, determine based on the eye movement whether the subject's eyes are following the point of gaze each time the point of gaze stops, and detect impersonation of the subject based on the determination result of whether the subject's eyes are following the point of gaze. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing the overall configuration of a gaze estimation system according to a first embodiment. [Figure 2] 1 is a block diagram showing a hardware configuration of a gaze estimation system according to a first embodiment. [Figure 3] 4 is a flowchart showing the flow of operations of the gaze estimation system according to the first embodiment. [Figure 4] 10 is a conceptual diagram (part 1) showing a display mode of a gaze point by the gaze estimation system according to the second embodiment. FIG. [Figure 5] FIG. 10 is a conceptual diagram (part 2) illustrating a display mode of a gaze point by the gaze estimation system according to the second embodiment. [Figure 6] FIG. 10 is a conceptual diagram (part 3) showing a display mode of a gaze point by the gaze estimation system according to the second embodiment. [Figure 7] FIG. 11 is a conceptual diagram (part 4) illustrating a display mode of a gaze point by the gaze estimation system according to the second embodiment. [Figure 8] 10 is a conceptual diagram (part 1) showing a display mode of a trajectory of a gaze point by the gaze estimation system according to the second embodiment. [Figure 9] FIG. 10 is a conceptual diagram (part 2) illustrating a display mode of a trajectory of a gaze point by the gaze estimation system according to the second embodiment. [Figure 10] FIG. 10 is a conceptual diagram (part 3) illustrating a display mode of a trajectory of a gaze point by the gaze estimation system according to the second embodiment. [Figure 11] FIG. 10 is a block diagram showing the overall configuration of a gaze estimation system according to a third embodiment. [Figure 12] 11 is a flowchart showing the flow of operations of the gaze estimation system according to the third embodiment. [Figure 13] FIG. 10 is a block diagram showing the overall configuration of a gaze estimation system according to a fourth embodiment. [Figure 14] 10 is a flowchart showing the flow of operations of the gaze estimation system according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of a gaze estimation system, a gaze estimation method, and a computer program will be described with reference to the drawings.

[0011] First Embodiment The gaze estimation system according to the first embodiment will be described with reference to FIGS. 1 to 3. FIG.

[0012] (System Configuration) First, the overall configuration of the gaze estimation system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the overall configuration of the gaze estimation system according to the first embodiment.

[0013] As shown in FIG. 1, the gaze estimation system 10 according to the first embodiment includes a display control unit 101, a motion detection unit 102, and a tracking determination unit 110 as functional blocks for realizing its functions.

[0014] The display control unit 101 is configured to be able to display a point of gaze on a display device having, for example, a display. The display control unit 101 controls the display so that the point of gaze moves along a predetermined movement path. More specifically, the display control unit 101 controls each parameter related to the movement path, movement speed, size, color, etc. of the point of gaze and the behavior and display mode of the point of gaze. Specific display examples of the point of gaze will be described in detail later.

[0015] The motion detection unit 102 estimates the eye movement of the subject from an image of the subject (i.e., a person gazing at the gaze point). The motion detection unit 102 may acquire an image of the subject from, for example, a camera or the like installed in the vicinity of a display device on which the gaze point is displayed. The motion detection unit 102 may, for example, detect a facial region of the subject from the image of the subject and detect eye movement from the image of the facial region. The motion detection unit 102 may also estimate the subject's line of sight (e.g., which position on the display unit 20 the subject is gazing at) from the eye movement in the image of the subject. Note that a more specific method of detecting eye movement can be appropriately adopted from existing technology, and therefore a detailed description thereof will be omitted here.

[0016] The tracking determination unit 110 is configured to be able to determine whether the subject's eyes are tracking the point of gaze based on the relationship between the movement of the point of gaze controlled by the display control unit 101 and the eye movement detected by the movement detection unit 102. For example, the tracking determination unit 110 may determine that the subject's eyes are tracking the point of gaze when the subject's eyes are moving to follow the movement of the point of gaze. However, there is a certain amount of time delay (a discrepancy due to a delayed reaction) between the movement of the point of gaze and the movement of the subject. Therefore, the tracking determination unit 110 may determine the tracking taking such a time delay into consideration. Furthermore, if there is no discrepancy at all between the movement of the point of gaze and the movement of the subject's eyes, it may be determined that the subject's eyes are not tracking the point of gaze (for example, determining that some kind of fraud is occurring).

[0017] (Hardware configuration) Next, the hardware configuration of the viewpoint position estimation device 10 according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the hardware configuration of the viewpoint position estimation device according to the first embodiment.

[0018] 2, the gaze estimation system 10 according to the first embodiment includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, and a storage device 14. The gaze estimation system 10 may further include an input device 15 and an output device 16. The CPU 11, the RAM 12, the ROM 13, the storage device 14, the input device 15, and the output device 16 are connected via a data bus 17. Note that the gaze estimation system 10 may include a plurality of each of the CPU 11, the RAM 12, the ROM 13, the storage device 14, the input device 15, and the output device 16.

[0019] The CPU 11 loads a computer program. For example, the CPU 11 is configured to load a computer program stored in at least one of the RAM 12, the ROM 13, and the storage device 14. Alternatively, the CPU 11 may load a computer program stored in a computer-readable storage medium using a storage medium reading device (not shown). The CPU 11 may acquire (i.e., load) the computer program from a device (not shown) located outside the gaze estimation system 10 via a network interface. The CPU 11 controls the RAM 12, the storage device 14, the input device 15, and the output device 16 by executing the loaded computer program. In particular, in this embodiment, when the CPU 11 executes the loaded computer program, functional blocks are realized within the CPU 11 for controlling the display of a gaze point, estimating the eye movement of a subject, and determining tracking (see FIG. 1).

[0020] The RAM 12 temporarily stores computer programs executed by the CPU 11. The RAM 12 temporarily stores data that is temporarily used by the CPU 11 while the CPU 11 is executing the computer programs. The RAM 12 may be, for example, a D-RAM (Dynamic RAM).

[0021] The ROM 13 stores computer programs executed by the CPU 11. The ROM 13 may also store fixed data. The ROM 13 may be, for example, a P-ROM (Programmable ROM).

[0022] The storage device 14 stores data that is to be saved long-term by the gaze estimation system 10. The storage device 14 may operate as a temporary storage device for the CPU 11. The storage device 14 may include, for example, at least one of a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device.

[0023] The input device 15 is a device that receives input instructions from a user of the gaze estimation system 10. The input device 15 may include, for example, at least one of a keyboard, a mouse, and a touch panel.

[0024] The output device 16 is a device that outputs information related to the gaze estimation system 10 to the outside. For example, the output device 16 may be a display device (for example, a display) that can display information related to the gaze estimation system 10.

[0025] (Operation flow) Next, the flow of operations of the viewpoint position estimation device 10 according to the first embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the flow of operations of the viewpoint position estimation device according to the first embodiment.

[0026] 3, during operation of the viewpoint position estimation device 10 according to the first embodiment, the display control unit 101 first starts display control of the point of gaze (step S11). Note that the display control of the point of gaze may be started, for example, by a terminal operation by the subject, or may be started automatically upon detecting that the subject is present in the vicinity of a display device or the like on which the point of gaze is displayed.

[0027] When the display control of the gaze point is started, the movement detection unit 102 acquires an image of the subject (step S12). Then, the movement detection unit 102 detects the movement of the subject's eyes from the acquired image of the subject (step S13).

[0028] Next, the tracking determination unit 110 determines whether the subject's eyes are tracking the point of gaze based on the relationship between the movement of the point of gaze controlled by the display control unit 101 and the eye movement detected by the movement detection unit 102 (step S14).

[0029] (Technical Effects) Next, an example of the technical effect obtained by the viewpoint position estimation device 10 according to the first embodiment will be described.

[0030] 1 to 3, the viewpoint position estimation device 10 according to the first embodiment can detect the movement of the subject's eyes by having the subject gaze at a gaze point. In particular, since the gaze point is controlled to move, it is possible to suppress the subject's wandering gaze compared to, for example, a case where the subject is asked to gaze at a stationary gaze point.

[0031] In this embodiment, whether the subject is following the gaze point is determined from the detected eye movement. Therefore, it is possible to determine whether the subject is normally gazing at the moving gaze point. This determination result can be used not only to determine whether the eye movement can be appropriately detected, but also for processing to correct the subject's gaze position or processing to detect impersonation, as in the second and third embodiments described below.

[0032] Second Embodiment The gaze estimation system according to the second embodiment will be described with reference to Fig. 4 to Fig. 10. The second embodiment differs from the first embodiment described above only in some operations (mainly operations related to displaying the stop point), and is otherwise generally similar. Therefore, the following will describe in detail the parts that differ from the first embodiment, and will omit explanations of other overlapping parts as appropriate.

[0033] (System Configuration) The configuration of the gaze estimation system according to the second embodiment may be the same as that of the gaze estimation system according to the first embodiment (see FIG. 1), and therefore a description thereof will be omitted. Also, the hardware configuration of the gaze estimation system according to the second embodiment may be the same as that of the gaze position estimation system 10 according to the first embodiment (see FIG. 2), and therefore a description thereof will be omitted.

[0034] (Operation flow) The operation flow of the gaze estimation system 10 according to the second embodiment may be the same as the operation flow of the gaze estimation system 10 according to the first embodiment (see FIG. 3), and therefore a description thereof will be omitted. However, in the gaze estimation system 10 according to the second embodiment, the display control unit 101 controls the display mode of the gaze point or the display mode of the gaze point trajectory as follows.

[0035] (Display mode of gaze point) The display mode of the gaze point in the gaze estimation system 10 according to the second embodiment will be described with reference to Figs. 4 to 7. Fig. 4 is a conceptual diagram (part 1) showing the display mode of the gaze point by the gaze estimation system according to the second embodiment. Fig. 5 is a conceptual diagram (part 2) showing the display mode of the gaze point by the gaze estimation system according to the second embodiment. Fig. 6 is a conceptual diagram (part 3) showing the display mode of the gaze point by the gaze estimation system according to the second embodiment. Fig. 7 is a conceptual diagram (part 4) showing the display mode of the gaze point by the gaze estimation system according to the second embodiment.

[0036] As shown in FIG. 4, the gaze point X moves on the display surface of the display unit 20, alternately moving and stopping along a predetermined trajectory. More specifically, the gaze point X moves along a predetermined straight line, stops at the end point of the line for a predetermined period (e.g., several seconds), moves again along another straight line from there, and stops at the end point of the line for a predetermined period, repeating this action. In this way, the subject can easily gaze at the gaze point X. It is preferable that the gaze point X moves so as to pass through all points on the display surface of the display unit 20. Here, the shape of the gaze point X is, for example, an ellipse, and the position of the gaze point X is, for example, the center position of the gaze point X. Note that the shape and position of the gaze point X are not limited to this.

[0037] The subject is asked to follow the movement of the gaze point X with his / her eyes. In order to have the subject follow the gaze point X, specific instructions may be output to the subject before display control of the gaze point X is started. For example, instructions such as text to follow the gaze point X may be displayed on the display unit 20. Alternatively, display control may be performed so that the subject naturally follows the gaze point X with his / her eyes. For example, the gaze point X may be displayed as a character, object, or the like that attracts the subject's interest.

[0038] The moving speed and size of the gaze point X are set in advance. However, the moving speed of the gaze point X may be changed as appropriate depending on the subject's reaction. For example, for a subject whose eye movement delay relative to the movement of the gaze point is relatively large, the movement of the gaze point X may be slowed down. Alternatively, the size of the gaze point X may be changed as appropriate depending on the subject's reaction. For example, for a subject whose gaze position varies relatively greatly, the size of the gaze point X may be increased. Furthermore, both the moving speed and size of the gaze point X may be changed as appropriate depending on the subject's reaction. Control to change the moving speed of the gaze point X or control to change the size of the gaze point X may be performed using the result immediately after display control of the gaze point X is started (for example, the result measured when moving on the initial straight line).

[0039] As shown in FIG. 5, the fixation point X may be controlled to flash at predetermined intervals while moving (in other words, to be highlighted). This makes it easy for the subject to gaze at the fixation point X. Furthermore, by having the fixation point X flash at regular intervals, the subject can more easily predict the movement of the fixation point X. This makes it easy to reduce the time delay of the parallax (i.e., the delay in the movement of the line of sight relative to the movement of the fixation point) to a certain value. Note that even in the display mode shown in FIG. 5, the fixation point X may alternate between moving and stationary states, as described in FIG. 4.

[0040] As shown in FIG. 6, the fixation point X may be controlled so as to alternate between a state in which it is visible while moving (X1 in the figure) and a state in which it is invisible (X2 in the figure). Even in this case, it is possible to easily get the subject to gaze at the fixation point X. Furthermore, by periodically alternating between a state in which the fixation point X is visible and a state in which it is invisible, the subject can more easily predict the movement of the fixation point X. This makes it easy to reduce the time delay of the parallax to a certain value. Note that, even in the display mode shown in FIG. 6, the fixation point X may alternate between a moving state and a stationary state, as described in FIG. 4.

[0041] As shown in FIG. 7, the fixation point X may be controlled so as to alternate between a state in which it is displayed large while moving (X3 in the figure) and a state in which it is displayed small (X4 in the figure). Even in this case, it is possible to easily get the subject to fixate on the fixation point X. Furthermore, the periodic change in size of the fixation point X makes it easier for the subject to predict the movement of the fixation point X. This makes it easy to reduce the time delay of the parallax to a certain value. Note that even in the display mode shown in FIG. 7, the fixation point X may alternate between a moving state and a stationary state, as described in FIG. 4.

[0042] Furthermore, in addition to the gaze point X, a number for counting may be displayed. For example, this number may count up from a preset initial value to an upper limit value, or may count down from a preset initial value to a lower limit value. The number may be counted over time (for example, every second). Furthermore, the number may be counted from the time the gaze point X starts to move until it stops, or may be counted every time it flashes, or may be counted when the position of the gaze point X and the position of the eye are continuously within a predetermined distance.

[0043] (Technical effect of the display mode of the point of gaze) Next, an example of the technical effect obtained by the viewpoint position estimation device 10 according to the display mode of the point of interest of the second embodiment will be described.

[0044] According to the viewpoint position estimation device 10 of the second embodiment, the display mode of the gaze point is controlled as shown in Figs. 4 to 7. This makes it easier to get the subject to gaze at the gaze point X. The display modes shown in Figs. 4 to 7 may be combined as appropriate. For example, the gaze point X, which moves so as to alternately move and stop along a predetermined trajectory as shown in Fig. 4, may be controlled to flash at predetermined intervals while moving as shown in Fig. 5.

[0045] (Display mode of gaze point locus) Next, the display mode of the trajectory of the gaze point X will be described with reference to Fig. 8 to Fig. 10. Fig. 8 is a conceptual diagram (part 1) showing the display mode of the trajectory of the gaze point by the gaze estimation system according to the second embodiment. Fig. 9 is a conceptual diagram (part 2) showing the display mode of the trajectory of the gaze point by the gaze estimation system according to the second embodiment. Fig. 10 is a conceptual diagram (part 3) showing the display mode of the trajectory of the gaze point by the gaze estimation system according to the second embodiment.

[0046] As shown in Fig. 8, the movement trajectory of the gaze point X may be displayed. In this way, the subject can know how the gaze point X has moved up to now or how the gaze point X will move in the future. As a result, the subject can more easily predict the movement of the gaze point X.

[0047] 9 and 10, the movement trajectory of the fixation point X may be displayed in different ways depending on whether it is a movement trajectory that has already been moved or a movement trajectory that will move from now on. Specifically, as shown in Fig. 9, the movement trajectory that has already been moved may be displayed as a solid line, while the movement trajectory that will move from now on may be displayed as a dotted line. Alternatively, as shown in Fig. 10, the movement trajectory that has already been moved may be displayed normally, while the movement trajectory that will move from now on may not be displayed (i.e., only the movement trajectory that has already been moved may be displayed).

[0048] (Technical effect of the display mode of the trajectory of the gaze point) Next, an example of the technical effect obtained by the viewpoint position estimation device 10 according to the display mode of the trajectory of the point of interest of the second embodiment will be described.

[0049] According to the viewpoint position estimation device 10 of the second embodiment, the display mode of the trajectory of the gaze point is controlled as shown in Figs. 8 to 10. This makes it easier for the subject to predict the movement of the gaze point X, which makes it easier to reduce the time delay of the parallax to a certain value. An example of the technical effect of reducing the time delay of the parallax to a certain value will be described in detail in the third embodiment described later.

[0050] Third Embodiment A gaze estimation system according to a third embodiment will be described with reference to Figs. 11 and 12. In the third embodiment, an example will be described in which the gaze estimation system functions as a system for correcting (calibrating) a viewpoint position. Note that the third embodiment differs only in part from the first and second embodiments described above in terms of configuration and operation, and is otherwise generally similar. Therefore, the following will describe in detail the parts that differ from the first and second embodiments, and will omit a description of the other overlapping parts as appropriate.

[0051] (System Configuration) First, the overall configuration of the gaze estimation system according to the third embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing the overall configuration of the gaze estimation system according to the third embodiment. Note that the hardware configuration of the gaze estimation system according to the third embodiment may be the same as the hardware configuration of the gaze position estimation system 10 according to the first embodiment (see Fig. 2), and therefore the description thereof will be omitted.

[0052] As shown in FIG. 11 , the gaze estimation system 10 according to the third embodiment is connected to a display unit 20 and an imaging unit 30. The display unit 20 is a display placed at a position where a subject, whose gaze position is to be estimated, can be seen, and displays a gaze point for estimating the gaze position. The display control of the gaze point on the display unit 20 is performed by the gaze estimation system 10. The imaging unit 30 is a camera installed in the periphery of the display unit 20, and is placed at a position where it can capture an image of the subject (particularly an image around the face). The image of the subject captured by the imaging unit 30 is configured to be output to the gaze estimation system 10.

[0053] The gaze estimation system 10 according to the third embodiment includes, in addition to the components of the first embodiment (see FIG. 1), a parallax calculation unit 103, a statistical processing unit 104, a time delay calculation unit 105, a deviation amount calculation unit 106, a correction value integration unit 107, and a viewpoint correction output unit 108.

[0054] The disparity calculation unit 103 calculates disparity, which is the difference between the position of the gaze point controlled by the display control unit 101 and the viewpoint position of the subject estimated by the motion detection unit 102. The disparity calculated by the disparity calculation unit 103 is calculated as a value including random statistical errors and time delays. The disparity calculation unit 103 may be realized as a functional block having the above-mentioned functions in, for example, the CPU 11 (see FIG. 1).

[0055] The statistical processing unit 104 is configured to be able to execute statistical processing for removing statistical errors contained in the disparity calculated by the disparity calculation unit 103. Specifically, the statistical processing unit 104 removes statistical errors by time-averaging the disparity calculated by the disparity calculation unit 103 over a fixed period of time. Note that the above-described statistical processing is merely an example, and other statistical processing may be used to remove statistical errors. The statistical processing unit 104 may be realized as a functional block having the above-described functions in, for example, the CPU 11 (see FIG. 1 ).

[0056] The time delay calculation unit 105 is configured to be able to calculate the time delay included in the parallax calculated by the parallax calculation unit 103. The time delay calculation unit 105 calculates the time delay of the parallax based on the parallax at least when the point of interest is moving. The time delay calculation unit 105 may be realized as a functional block having the above-mentioned functions in, for example, the CPU 11 (see FIG. 1).

[0057] The deviation amount calculation unit 106 calculates the deviation amount between the viewpoint position estimated by the motion detection unit 102 and the true value of the viewpoint position (i.e., the position where the subject is actually looking) based on the disparity from which statistical errors have been removed by the statistical processing unit 104 and the time delay calculated by the time delay calculation unit 105. The deviation amount calculation unit 106 may be realized as a functional block having the above-mentioned functions in, for example, the CPU 11 (see FIG. 1).

[0058] The correction value integrating unit 107 calculates a correction value for the viewpoint position estimated by the motion detecting unit 102 (in other words, a correction amount for reducing the deviation of the viewpoint position) based on the deviation amount calculated by the deviation amount calculating unit 106. The correction value integrating unit 107 also performs an integration process on the correction values ​​calculated at multiple locations on the display unit 20, and generates a correction formula for correcting the viewpoint position. By using this correction formula, it becomes possible to correct the viewpoint position even for points for which no correction value has actually been calculated. The correction value integrating unit 107 may be realized as a functional block having the above-mentioned functions in, for example, the CPU 11 (see FIG. 1).

[0059] The viewpoint correction output unit 108 outputs the correction formula generated by the correction value integration unit 107 as information for calibrating the viewpoint position. Note that the viewpoint correction output unit 108 may have a function of storing the generated correction formula and correcting and outputting the viewpoint position (i.e., a function of outputting the corrected viewpoint position). The viewpoint correction output unit 108 may be realized as a functional block having the above-described functions in, for example, the CPU 11 (see FIG. 1).

[0060] (Operation flow) Next, the flow of operations of the gaze estimation system 10 according to the third embodiment will be described with reference to Fig. 12. Fig. 12 is a flowchart showing the flow of operations of the gaze estimation system according to the third embodiment.

[0061] 12, during operation of the gaze estimation system 10 according to the third embodiment, first, the display control unit 101 starts display control of the gaze point on the display unit 20 (step S101). Note that the display control of the gaze point may be started, for example, by a terminal operation by the subject, or may be started automatically when the presence of the subject near the display unit 20 is detected.

[0062] When the display control of the gaze point is started, the motion detection unit 102 acquires an image of the subject from the imaging unit 30 (step S102). Then, the motion detection unit 102 estimates the gaze point position of the subject from the acquired image of the subject (step S103).

[0063] Next, the disparity calculation unit 103 calculates the disparity, which is the difference between the position of the gaze point and the estimated viewpoint position (step S104). After the disparity is calculated, the statistical processing unit 104 performs statistical processing to remove statistical errors contained in the disparity (step S105).

[0064] Thereafter, the time delay calculation unit 105 calculates the time delay occurring in the disparity (step S106). Then, the deviation amount calculation unit 106 calculates the amount of deviation between the estimated viewpoint position and the actual viewpoint position based on the disparity from which the statistical error has been removed and the time delay amount of the disparity (step S107).

[0065] Once the deviation amount is calculated, it is determined whether or not the display control of the gaze point by the display control unit 101 has ended (step S108). That is, it is determined whether or not the gaze point has repeatedly moved and stopped along a predetermined path and has completed its movement to the end point. If it is determined that the display control has not ended (step S108: NO), the process is repeated again from step S102. As a result, deviation amounts are calculated at multiple locations along the movement path of the gaze point.

[0066] If it is determined that the display control has ended (step S108: YES), the tracking determination unit 110 determines whether the eyes of the subject have been tracking the gaze point (step S109). Note that the tracking determination unit 110 may determine whether the eyes of the subject are tracking the gaze point before the display control ends (i.e., while the gaze point is displayed). For example, the tracking determination unit 110 may determine whether the eyes of the subject are tracking the gaze point every time the gaze point X stops. Note that if it is determined that the eyes of the subject are not tracking the gaze point (step S109: NO), the subsequent processes are omitted and the series of operations ends. This is because if the eyes of the subject are not tracking the gaze point, appropriate correction of the gaze position cannot be performed.

[0067] On the other hand, if it is determined that the subject's eyes have been tracking the gaze point (step S109: YES), the correction value integrating unit 107 calculates multiple correction values ​​from the deviation amounts calculated at multiple locations and performs an integration process on the multiple correction values ​​(step S109). That is, a correction formula for the viewpoint position is calculated based on the multiple correction values. Then, the viewpoint correction output unit 108 outputs the correction formula generated by the correction value integrating unit 107 as information for performing calibration of the viewpoint position (step S110).

[0068] (Viewpoint position correction) Next, correction of the viewpoint position by the gaze estimation system 10 according to the second embodiment will be specifically described. In the following, it is assumed that the gaze point X is displayed in the display mode as shown in FIG. 6 described in the second embodiment.

[0069] If the gaze position estimated by the motion detection unit 102 of the gaze estimation system 10 according to this embodiment is “Xgaze,est”, the correction value is “ΔXcalib”, and the statistical error is “ε”, the true value of the subject's gaze position, “Xgaze,true”, can be expressed by the following equation (1).

[0070] TIFF0007806863000001.tif33170

[0071] When the parallax does not include statistical errors, the above formula (1) can also be expressed as the following formula (2). TIFF0007806863000002.tif33170 In this case, it is possible to ignore "ε" in the following explanation and perform the calculations.

[0072] Furthermore, if the position of the fixation point X is "Xc" and the time delay is "ΔXdelay", it can also be expressed as in the following equation (3).

[0073] TIFF0007806863000003.tif29170

[0074] Furthermore, from the above equations (1) and (3), the disparity "Xgaze,est-Xc" calculated by the disparity calculation unit 103 can be expressed by the following equation (4).

[0075] TIFF0007806863000004.tif28170

[0076] Here, the statistical error ε can be removed by statistical processing executed by the statistical processing unit 104. As a result, it becomes possible to calculate the correction value ΔXcalib using Xgaze,est-Xc calculated by the disparity calculation unit 103 and the time delay ΔXdelay calculated by the time delay calculation unit 105.

[0077] The correction value ΔXcalib is calculated using the parallax at least when the gaze point X is moving. When the gaze point X is moving, the time delay ΔXdelay becomes small if a sufficient amount of time has passed since the gaze point X began to move. Specifically, when the gaze point moves at a constant speed, the subject is able to predict the movement of the gaze point, so the time delay becomes small and approaches a constant value. This time delay can be estimated, for example, from the viewpoint position when the gaze point X stops, or the elapsed time until the viewpoint settles at the stopped position. Therefore, using the parallax when the gaze point X is moving makes it possible to more easily and accurately calculate the correction value ΔXcalib. Note that the parallax when the gaze point is moving may be calculated, for example, at the midpoint of the line along which the gaze point X moves.

[0078] The correction value ΔXcalib is calculated at multiple locations on the display unit 20. Then, the correction value integration unit 107 integrates the correction values ​​ΔXcalib integrated at multiple locations to generate a correction formula for the viewpoint position. The correction formula is generated as, for example, the following formula (5) including predetermined coefficients A and b.

[0079] TIFF0007806863000005.tif42170

[0080] The correction value ΔXcalib may be calculated by integrating not only the correction value calculated when the gaze point X is moving, but also the correction value calculated when the gaze point X is stationary.

[0081] Furthermore, the position Xc of the gaze point X may be the position on the circumference of the gaze point X that is closest to the viewpoint position Xgaze,est (i.e., the intersection of the line segment connecting the viewpoint position Xgaze,est and the center position of the gaze point X with the circumference of the gaze point X).

[0082] In the above example, a case was described in which disparity calculation, disparity statistical processing, disparity time delay calculation, and deviation amount calculation (i.e., the processing of steps S104 to S107 in Figure 5) are performed sequentially, but all of these processes do not necessarily have to be performed.

[0083] For example, the viewpoint position may be corrected by calculating the parallax, which makes it possible to perform corrections that reduce the influence of the parallax.

[0084] Alternatively, the viewpoint position may be corrected by performing parallax statistical processing, which makes it possible to perform correction that reduces the influence of statistical errors in parallax.

[0085] Alternatively, the viewpoint position may be corrected by calculating a parallax time delay, which makes it possible to perform a correction that reduces the influence of the parallax time delay.

[0086] Alternatively, the viewpoint position may be corrected by calculating the amount of deviation, which makes it possible to perform correction that reduces the influence of the amount of deviation of the viewpoint position.

[0087] Furthermore, at least two of the processes of disparity calculation, disparity statistical processing, disparity time delay calculation, and deviation amount calculation may be performed in combination.

[0088] (Technical Effects) Next, an example of the technical effect obtained by the gaze estimation system 10 according to the third embodiment will be described.

[0089] 11 and 12, the gaze estimation system 10 according to the third embodiment can calculate a correction value for correcting the estimated gaze position (in other words, a value for calibration) by having the subject gaze at the gaze point X. In particular, in this embodiment, the correction value is calculated using the parallax when the gaze point X is moving, thereby reducing the effect of the time delay ΔXdelay and making it possible to obtain correction values ​​over a wider range than when the gaze point X is stationary. As a result, it becomes possible to accurately estimate where the subject is actually looking.

[0090] <Fourth embodiment> Next, a gaze estimation system 10 according to a fourth embodiment will be described with reference to FIGS. 13 and 14. In the fourth embodiment, an example will be described in which the gaze estimation system 10 functions as a system for determining whether a target person is impersonating another person. Note that the third embodiment differs only in part of the configuration and operation from the first to third embodiments already described, and the remaining parts are generally the same. Therefore, the following will describe in detail the parts that differ from the first to third embodiments, and will omit a description of the other overlapping parts as appropriate.

[0091] (System Configuration) First, the overall configuration of the gaze estimation system 10 according to the fourth embodiment will be described with reference to Fig. 13. Fig. 13 is a block diagram showing the overall configuration of the gaze estimation system according to the fourth embodiment. Note that the hardware configuration of the gaze estimation system according to the fourth embodiment may be the same as the hardware configuration of the gaze position estimation system 10 according to the first embodiment (see Fig. 2), and therefore the description thereof will be omitted.

[0092] 13, the gaze estimation system 10 according to the fourth embodiment includes, in addition to the components of the gaze estimation system 10 according to the third embodiment (see FIG. 1), a spoofing detection unit 201 and a determination result output unit 202. More precisely, the gaze estimation system 10 according to the fourth embodiment includes the spoofing detection unit 201 and the determination result output unit 202 instead of the correction value integrating unit 107 and the viewpoint correction output unit 108 according to the third embodiment.

[0093] The spoofing detection unit 201 is configured to detect spoofing (i.e., fraudulent behavior using a video or the like) based on the determination result of the following determination unit 110. For example, if the subject is actually present in front of the imaging unit 30, the deviation amount is calculated as a value having a reasonable magnitude due to a time delay. On the other hand, if the subject is not actually present in front of the imaging unit 30 and is pointing a video or the like captured by the subject toward the imaging unit 30, the time delay specific to humans does not occur, and the deviation amount is calculated as an extremely small value. Therefore, the following determination unit 110 can determine whether the subject is properly following the gaze point X by comparing the calculated deviation amount with a predetermined threshold. Therefore, the spoofing detection unit can detect spoofing based on the determination result of the following determination unit 110. Specifically, the following determination unit 110 determines that the subject is properly following the gaze point X when the calculated deviation amount is greater than a predetermined threshold. In this case, the spoofing detection unit 201 detects that spoofing is occurring. On the other hand, if the calculated deviation amount is smaller than a predetermined threshold, the following determination unit 110 determines that the target person is not properly following the gaze point X. In this case, the spoofing detection unit 201 does not detect that spoofing is occurring. The spoofing detection unit 201 may be realized as a functional block having the above-mentioned functions in, for example, the CPU 11 (see FIG. 1).

[0094] The determination result output unit 202 is configured to be able to output the determination result obtained by the spoofing detection unit 201. The determination result output unit 202 may output only the result indicating whether or not spoofing is occurring, or may execute a predetermined operation (for example, an alert operation) when spoofing is detected. The determination result output unit 202 may be realized as a functional block having the above-described functions in the CPU 11 (see FIG. 1), for example.

[0095] (Operation flow) Next, the flow of operations of the gaze estimation system 10 according to the fourth embodiment will be described with reference to Fig. 14. Fig. 14 is a flowchart showing the flow of operations of the gaze estimation system according to the fourth embodiment.

[0096] 14, when the gaze estimation system 10 according to the fourth embodiment operates, the processes of steps S101 to S108 are first executed, as in the first embodiment. That is, the subject is asked to follow the gaze point X with his / her eyes, and the amount of deviation between the estimated gaze position and the true value is calculated from the image captured at that time.

[0097] Particularly in the fourth embodiment, when it is determined that the display control of the gaze point has ended (step S108: YES), the tracking determination unit 110 determines whether or not the eyes of the subject have been tracking the gaze point (step S109). Then, when it is determined that the eyes of the subject have not been tracking the gaze point (step S109: NO), the spoofing detection unit 201 detects that spoofing is occurring (step S201). On the other hand, when it is determined that the eyes of the subject have been tracking the gaze point (step S109: YES), the spoofing detection unit 201 does not detect that spoofing is occurring (step S202). Then, the determination result output unit 202 outputs the determination result by the spoofing determination unit 201 (step S202).

[0098] In the fourth embodiment, the spoofing detection unit 201 may detect spoofing before the display control of the gaze point X is completed (in other words, the following determination unit 110 may perform determination before the display control of the gaze point X is completed). For example, the following determination unit 110 may determine that following is not being performed without waiting for the movement of the gaze point X to be completed when it is determined that the value of one calculated deviation amount is a value corresponding to spoofing, and at that timing the spoofing detection unit 201 may detect that spoofing is being performed.

[0099] (Technical Effects) Next, an example of the technical effect obtained by the gaze estimation system 10 according to the fourth embodiment will be described.

[0100] 13 and 14, the gaze estimation system 10 according to the fourth embodiment can determine and detect whether or not spoofing is taking place based on the deviation calculated by the deviation amount calculation unit 106. That is, by utilizing the difference between the actual movement of a human gaze position and the mechanical movement of a gaze position, it is possible to appropriately determine and detect spoofing. The gaze estimation system 10 according to the fourth embodiment exerts beneficial effects when, for example, an authentication process using the gaze position is performed.

[0101] In the gaze estimation system 10 according to the fourth embodiment, it is preferable not to display the trajectory of the gaze point. Alternatively, as shown in Fig. 10, it is preferable to display the movement trajectory of the object that has already moved, but not to display the movement trajectory of the object that will move in the future.

[0102] <Additional Notes> The above-described embodiment may be further described as follows, but is not limited to the following.

[0103] (Appendix 1) The gaze position estimation system described in Appendix 1 is a gaze estimation system characterized by comprising: a display control means for displaying a gaze point at which a subject is gazing so as to move in a predetermined movement pattern; a detection means for detecting eye movement of the subject from an image of the subject; and a tracking determination means for determining whether the subject's eyes are tracking the gaze point based on the relationship between the movement of the gaze point and the movement of the eyes.

[0104] (Appendix 2) The gaze estimation system described in Appendix 2 is the gaze estimation system described in Appendix 1, characterized in that the display control means displays the gaze point in at least one of a first mode in which the gaze point is moved so that a first period in which the gaze point moves at a predetermined acceleration and a second period in which the gaze point is stopped for a predetermined period are alternately repeated; a second mode in which the gaze point is highlighted at regular intervals; a third mode in which the gaze point is alternately displayed and hidden at regular intervals; and a fourth mode in which the size of the gaze point is changed while it is moving.

[0105] (Appendix 3) The gaze estimation system described in Appendix 3 is the gaze estimation system described in Appendix 1 or 2, characterized in that the display control means displays, among the movement trajectories of the moving point of the gaze point, a trajectory through which the gaze point has passed and a trajectory through which the gaze point has not passed in different display modes, or displays, among the movement trajectories of the moving point of the gaze point, at least one of a trajectory through which the gaze point has passed and a trajectory through which the gaze point has not passed.

[0106] (Appendix 4) The gaze estimation system described in Supplementary Note 4 is the gaze estimation system described in any one of Supplementary Notes 1 to 3, characterized in that it includes a correction means for correcting parameters related to the detected eye movement of the subject depending on whether the subject's eyes are tracking the gaze point.

[0107] (Appendix 5) The gaze estimation system described in Supplementary Note 5 is the gaze estimation system described in Supplementary Note 4, characterized in that the correction means calculates a parallax, which is the difference between the position of the point of gaze and the gaze position of the subject, from the detected eye movement of the subject, calculates a time delay of the parallax from the parallax calculated while the gaze point is moving, calculates a deviation amount from the true value of the estimated gaze position based on the parallax and the time delay of the parallax, and determines a correction value for correcting the gaze position of the subject based on the deviation amount.

[0108] (Appendix 6) In the gaze estimation system described in Supplementary Note 6, the correction means calculates the amount of deviation at a plurality of locations in an area where the gaze point moves, and corrects the gaze position by integrating a plurality of correction values ​​determined based on the amount of deviation calculated at the plurality of locations. 6. The gaze estimation system according to claim 5,

[0109] (Appendix 7) The gaze estimation system described in Appendix 7 is the gaze estimation system described in any one of claims 1 to 6, characterized in that it further comprises an impersonation detection means for detecting impersonation of the subject when the eyes of the subject are not tracking the gaze point.

[0110] (Appendix 8) The gaze estimation system described in Appendix 8 is the gaze estimation system described in Appendix 7, characterized in that the impersonation detection means calculates a parallax, which is the difference between the position of the point of gaze and the gaze position of the subject, from the detected eye movement of the subject, calculates a time delay of the parallax from the parallax calculated while the gaze point is moving, calculates a deviation amount from the true value of the estimated gaze position based on the parallax and the time delay of the parallax, and detects impersonation of the subject based on the deviation amount.

[0111] (Appendix 9) The gaze estimation device described in Appendix 8 is the gaze estimation system described in Appendix 7 or 8, characterized in that the display control means does not display the movement trajectory of the gaze point when the impersonation detection means detects impersonation.

[0112] (Appendix 10) The gaze estimation method described in Appendix 10 is a gaze estimation method characterized by displaying a gaze point at which a subject gazes so as to move in a predetermined movement pattern, detecting eye movement of the subject from an image of the subject, and determining whether or not the subject's eyes are following the gaze point based on the relationship between the movement of the gaze point and the eye movement.

[0113] (Appendix 11) The computer program described in Appendix 11 is a computer program characterized by causing a computer to display a point of gaze at which a subject is gazing so as to move in a predetermined movement pattern, detect the movement of the subject's eyes from an image of the subject, and determine whether the subject's eyes are following the point of gaze based on the relationship between the movement of the point of gaze and the movement of the eyes.

[0114] This disclosure may be modified as appropriate within the scope that does not contradict the gist or idea of ​​the invention that can be read from the claims and the entire specification, and gaze estimation systems, gaze estimation methods, and computer programs that involve such modifications are also included in the technical idea of ​​this disclosure. [Explanation of symbols]

[0115] 10 Gaze estimation system 20 Display section 30 Imaging unit 101 Display control unit 102 Motion detection unit 103 Parallax calculation unit 104 Statistical Processing Unit 105 Time delay calculation unit 106 Deviation calculation unit 107 Correction value integration unit 108 Viewpoint correction output unit 110 Following judgment unit 201 Spoofing detection unit 202 Judgment result output unit

Claims

1. a display control means for displaying a gaze point at which a subject gazes so as to move repeatedly between a first period during which the gaze point moves at a predetermined speed and a second period during which the gaze point stops for a predetermined period; a detection means for detecting eye movements of the subject from an image of the subject; calculating a parallax, which is a difference between the position of the gaze point and the viewpoint position of the subject, from the detected eye movement of the subject; Calculating a time delay of the parallax from the parallax calculated while the gaze point is moving; The viewpoint position is corrected based on the parallax and a time delay of the parallax. Correction means; a tracking determination means for determining whether the subject's eyes are tracking the gaze point each time the gaze point stops based on the eye movement; an impersonation detection means for detecting impersonation of the target person based on the determination result of the following determination means; Equipped with the display control means displays, among the movement trajectories along which the gaze point moves, a trajectory through which the gaze point has passed and a trajectory through which the gaze point has not passed in different display modes; A gaze estimation system characterized by:

2. The gaze estimation system of claim 1, characterized in that the display control means displays the gaze point in at least one of the following display modes: a first mode in which the gaze point is displayed so that it moves repeatedly between a first period in which the gaze point moves at a predetermined speed and a second period in which the gaze point is stopped for a predetermined period; a second mode in which the gaze point is highlighted at regular intervals; a third mode in which the gaze point is displayed so that it is alternately displayed and hidden at regular intervals; and a fourth mode in which the size of the gaze point is changed while it is moving.

3. The correction means Calculating a deviation amount of the estimated viewpoint position from a true value at a plurality of locations in an area where the gaze point moves based on the parallax and a time delay of the parallax; determining a correction value for correcting the gaze point position of the subject based on the deviation amount; The plurality of correction values ​​determined based on the deviation amounts calculated at the plurality of locations are integrated to correct the viewpoint position. The gaze estimation system according to claim 2 .

4. The gaze estimation system according to any one of claims 1 to 3, characterized in that the impersonation detection means detects the impersonation of the subject when the subject's eyes do not follow the gaze point.

5. The spoofing detection means calculating a parallax, which is a difference between the position of the gaze point and the viewpoint position of the subject, from the detected eye movement of the subject; Calculating a time delay of the parallax from the parallax calculated while the gaze point is moving; calculating a deviation amount of the estimated viewpoint position from a true value based on the parallax and a time delay of the parallax; Detecting impersonation of the target person based on the amount of deviation The gaze estimation system according to claim 4 .

6. At least one computer a first period during which the gaze point that the subject gazes at moves at a predetermined speed and a second period during which the gaze point is stopped for a predetermined period are alternately displayed; Among the movement trajectories along which the gaze point moves, a trajectory through which the gaze point has passed and a trajectory through which the gaze point has not passed are displayed in different display modes; detecting eye movements of the subject from an image of the subject; calculating a parallax, which is a difference between the position of the gaze point and the viewpoint position of the subject, from the detected eye movement of the subject; Calculating a time delay of the parallax from the parallax calculated while the gaze point is moving; correcting the viewpoint position based on the parallax and a time delay of the parallax; determining whether the subject's eyes are tracking the gaze point each time the gaze point stops based on the eye movement; Detecting impersonation of the target person based on a determination result of whether or not the target person's eyes are tracking the gaze point. A gaze estimation method characterized by:

7. a first period during which the gaze point that the subject gazes at moves at a predetermined speed and a second period during which the gaze point is stopped for a predetermined period are alternately displayed; Among the movement trajectories along which the gaze point moves, a trajectory through which the gaze point has passed and a trajectory through which the gaze point has not passed are displayed in different display modes; detecting eye movements of the subject from an image of the subject; calculating a parallax, which is a difference between the position of the gaze point and the viewpoint position of the subject, from the detected eye movement of the subject; Calculating a time delay of the parallax from the parallax calculated while the gaze point is moving; correcting the viewpoint position based on the parallax and a time delay of the parallax; determining whether the subject's eyes are tracking the gaze point each time the gaze point stops based on the eye movement; Detecting impersonation of the target person based on a determination result of whether or not the target person's eyes are tracking the gaze point. A computer program that causes a computer to operate in such a manner.

Images