Visual line estimation system, visual line estimation method, and computer program

Abstract

To provide a visual line estimation system, a visual line estimation method, and a computer program which properly detect movement of eyes of a target person.SOLUTION: A visual line estimation system 10 according to the present invention has a display control unit 101 for display a fixation point gazed by a target person so that a first period in which the fixation point moves at a predetermined speed and a second period in which the fixation point stops at a predetermined period of time are repeatedly switched, movement detection means for detecting movement of eyes of the target person from an image of the target person, a pursuit determination unit 110 for determining, based on the movement of the eyes, whether or not the eyes of the target person follow the fixation point every time the fixation point stops, and an impersonation detection unit 201 for detecting impersonation as the target person based on the determination result of the pursuit determination unit 110.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] The present disclosure relates to the technical fields of a gaze estimation system, a gaze estimation method, and a computer program for estimating the gaze of a subject. [Background technology]

[0002] As this type of system, a system that detects the gaze of a subject is known. For example, Patent Document 1 discloses a technology in which predetermined visual stimulation information is displayed on a display and the gaze position is estimated from the visual stimulation information and the subject's eye movement. Patent Document 2 discloses a technology in which a detection process for detecting the gaze, such as a corneal reflex method, is performed using each of images captured at a predetermined frame rate. Patent Document 3 discloses a technology in which a gaze detection process is performed using an image of a user received from an imaging device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-024608 A [Patent Document 2] JP 2017-107546 A [Patent Document 3] JP 2018-015218 A Summary of the Invention [Problem to be solved by the invention]

[0004] The technique of asking the subject to gaze at a fixed gaze point not only places a large burden on the subject, but also induces unconscious gaze wandering. The above-mentioned patent documents do not adequately address gaze wandering, and there is room for improvement.

[0005] An object of the present disclosure is to provide a gaze estimation system, a gaze estimation method, and a computer program for solving the above-mentioned problems. [Means for solving the problem]

[0006] One aspect of the gaze estimation system of this disclosure comprises a display control means for displaying a gaze point at which a subject is gazing so that it moves repeatedly between a first period during which the gaze point moves at a predetermined speed and a second period during which the gaze point is stopped for a predetermined period; a detection means for detecting eye movement of the subject from an image of the subject; a tracking determination means for determining whether the eyes of the subject are tracking the gaze point each time the gaze point stops based on the eye movement; and an impersonation detection means for detecting impersonation of the subject based on a determination result of the tracking determination means.

[0007] One aspect of the gaze estimation method of this disclosure displays a point of gaze at which a subject is gazing, repeatedly moving between a first period during which the point of gaze moves at a predetermined speed and a second period during which the point of gaze is stopped for a predetermined period, detects eye movement of the subject from an image of the subject, determines based on the eye movement whether the subject's eyes are following the point of gaze each time the point of gaze stops, and detects impersonation of the subject based on the determination result of whether the subject's eyes are following the point of gaze.

[0008] One aspect of the computer program of this disclosure operates a computer to display a point of gaze at which a subject is gazing, repeatedly moving between a first period during which the point of gaze moves at a predetermined speed and a second period during which the point of gaze is stopped for a predetermined period, detect eye movement of the subject from an image of the subject, determine based on the eye movement whether the subject's eyes are following the point of gaze each time the point of gaze stops, and detect impersonation of the subject based on the determination result of whether the subject's eyes are following the point of gaze. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an overall configuration of a gaze estimation system according to a first embodiment. [Diagram 2] 1 is a block diagram showing a hardware configuration of a gaze estimation system according to a first embodiment. [Diagram 3] 4 is a flowchart showing the flow of operations of the gaze estimation system according to the first embodiment. [Figure 4] 13 is a conceptual diagram (part 1) showing a display mode of a gaze point by the gaze estimation system according to the second embodiment. FIG. [Diagram 5] FIG. 13 is a conceptual diagram (part 2) illustrating a display mode of a gaze point by the gaze estimation system according to the second embodiment. [Figure 6] FIG. 11 is a conceptual diagram (part 3) illustrating a display mode of a gaze point by the gaze estimation system according to the second embodiment. [Figure 7] FIG. 11 is a conceptual diagram (part 4) illustrating a display mode of a gaze point by the gaze estimation system according to the second embodiment. [Figure 8] 13 is a conceptual diagram (part 1) showing a display mode of a trajectory of a gaze point by the gaze estimation system according to the second embodiment. FIG. [Figure 9] FIG. 13 is a conceptual diagram (part 2) illustrating a display mode of a trajectory of a gaze point by the gaze estimation system according to the second embodiment. [Figure 10] FIG. 13 is a conceptual diagram (part 3) illustrating a display mode of a trajectory of a gaze point by the gaze estimation system according to the second embodiment. [Figure 11] FIG. 13 is a block diagram showing the overall configuration of a gaze estimation system according to a third embodiment. [Figure 12] 13 is a flowchart showing the flow of operations of the gaze estimation system according to the third embodiment. [Figure 13] FIG. 13 is a block diagram showing the overall configuration of a gaze estimation system according to a fourth embodiment. [Figure 14] 13 is a flowchart showing the flow of operations of the gaze estimation system according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of a gaze estimation system, a gaze estimation method, and a computer program will be described with reference to the drawings.

[0011] First Embodiment The gaze estimation system according to the first embodiment will be described with reference to FIGS. 1 to 3. FIG.

[0012] (System Configuration) First, the overall configuration of the gaze estimation system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the overall configuration of the gaze estimation system according to the first embodiment.

[0013] As shown in FIG. 1, the gaze estimation system 10 according to the first embodiment includes a display control unit 101, a motion detection unit 102, and a following determination unit 110 as functional blocks for realizing the system's functions.

[0014] The display control unit 101 is configured to be able to display a gaze point on a display device having, for example, a display. The display control unit 101 controls the display so that the gaze point moves along a predetermined movement path. More specifically, the display control unit 101 controls each parameter related to the behavior and display mode of the gaze point, such as the movement path, movement speed, size, color, etc. of the gaze point. A specific display example of the gaze point will be described in detail later.

[0015] The motion detection unit 102 estimates the eye movement of the subject from an image of the subject (i.e., a person gazing at the gaze point). The motion detection unit 102 may acquire an image of the subject from, for example, a camera or the like installed around the display device on which the gaze point is displayed. The motion detection unit 102 may detect, for example, a face area of ​​the subject from the image of the subject and detect eye movement from the image of the face area. The motion detection unit 102 may also estimate the subject's line of sight (for example, which position on the display unit 20 the subject is gazing at) from the eye movement in the image of the subject. Note that a more specific method of detecting eye movement can be appropriately adopted from existing technology, and therefore a detailed description thereof will be omitted here.

[0016] The following determination unit 110 is configured to be able to determine whether or not the subject's eyes are following the gaze point based on the relationship between the movement of the gaze point controlled by the display control unit 101 and the movement of the eyes detected by the movement detection unit 102. The following determination unit 110 may determine that the subject's eyes are following the gaze point when, for example, the subject's eyes are moving so as to follow the movement of the gaze point. However, there is a certain time delay (a deviation due to a delayed reaction) between the movement of the gaze point and the movement of the subject. Therefore, the following determination unit 110 may determine the following by taking such a time delay into consideration. Also, when there is no deviation at all between the movement of the gaze point and the movement of the subject's eyes, it may be determined that the subject's eyes are not following the gaze point (for example, it may be determined that some kind of fraud is being performed).

[0017] (Hardware configuration) Next, a hardware configuration of the viewpoint position estimation device 10 according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the hardware configuration of the viewpoint position estimation device according to the first embodiment.

[0018] As shown in Fig. 2, the gaze estimation system 10 according to the first embodiment includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, and a storage device 14. The gaze estimation system 10 may further include an input device 15 and an output device 16. The CPU 11, the RAM 12, the ROM 13, the storage device 14, the input device 15, and the output device 16 are connected via a data bus 17. Note that the gaze estimation system 10 may include a plurality of each of the CPU 11, the RAM 12, the ROM 13, the storage device 14, the input device 15, and the output device 16.

[0019] The CPU 11 reads a computer program. For example, the CPU 11 is configured to read a computer program stored in at least one of the RAM 12, the ROM 13, and the storage device 14. Alternatively, the CPU 11 may read a computer program stored in a computer-readable storage medium using a storage medium reading device (not shown). The CPU 11 may acquire (i.e., read) a computer program from a device (not shown) disposed outside the gaze estimation system 10 via a network interface. The CPU 11 controls the RAM 12, the storage device 14, the input device 15, and the output device 16 by executing the computer program that the CPU 11 reads. In particular, in this embodiment, when the CPU 11 executes the computer program that it reads, functional blocks for controlling the display of a gaze point, estimating the movement of the subject's eyes, and determining tracking are realized in the CPU 11 (see FIG. 1).

[0020] The RAM 12 temporarily stores a computer program executed by the CPU 11. The RAM 12 temporarily stores data that is temporarily used by the CPU 11 when the CPU 11 is executing the computer program. The RAM 12 may be, for example, a D-RAM (Dynamic RAM).

[0021] The ROM 13 stores computer programs executed by the CPU 11. The ROM 13 may also store other fixed data. The ROM 13 may be, for example, a programmable ROM (P-ROM).

[0022] The storage device 14 stores data that is to be stored long-term by the gaze estimation system 10. The storage device 14 may operate as a temporary storage device for the CPU 11. The storage device 14 may include, for example, at least one of a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device.

[0023] The input device 15 is a device that receives an input instruction from a user of the gaze estimation system 10. The input device 15 may include, for example, at least one of a keyboard, a mouse, and a touch panel.

[0024] The output device 16 is a device that outputs information related to the gaze estimation system 10 to the outside. For example, the output device 16 may be a display device (for example, a display) that can display information related to the gaze estimation system 10.

[0025] (Operation flow) Next, the flow of operations of the viewpoint position estimation device 10 according to the first embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the flow of operations of the viewpoint position estimation device according to the first embodiment.

[0026] 3, during operation of the viewpoint position estimation device 10 according to the first embodiment, the display control unit 101 first starts display control of the gaze point (step S11). Note that the display control of the gaze point may be started, for example, by a terminal operation by the subject, or may be started automatically upon detection of the presence of the subject in the vicinity of a display device or the like on which the gaze point is displayed.

[0027] When the display control of the gaze point is started, the movement detection unit 102 acquires an image of the subject (step S12). Then, the movement detection unit 102 detects the movement of the subject's eyes from the acquired image of the subject (step S13).

[0028] Next, the tracking determination unit 110 determines whether the subject's eyes are tracking the gaze point based on the relationship between the movement of the gaze point controlled by the display control unit 101 and the eye movement detected by the movement detection unit 102 (step S14).

[0029] (Technical effect) Next, an example of the technical effect obtained by the viewpoint position estimation device 10 according to the first embodiment will be described.

[0030] 1 to 3, the gaze point position estimation device 10 according to the first embodiment can detect the movement of the subject's eyes by having the subject gaze at the gaze point. In particular, since the gaze point is controlled to move, it is possible to suppress the subject's wandering gaze compared to, for example, a case in which the subject gazes at a stationary gaze point.

[0031] In this embodiment, whether or not the subject is following the gaze point is determined from the detected eye movement. Therefore, it is possible to determine whether or not the subject is normally gazing at the moving gaze point. This determination result can be used not only to determine whether or not the eye movement can be appropriately detected, but also to correct the subject's gaze position or to detect impersonation, as in the second and third embodiments described later.

[0032] <Second embodiment> The gaze estimation system according to the second embodiment will be described with reference to Fig. 4 to Fig. 10. The second embodiment differs from the first embodiment described above only in some operations (mainly operations related to displaying the stop point), and is otherwise generally similar. Therefore, the following will describe in detail the parts that differ from the first embodiment, and will omit appropriate descriptions of other overlapping parts.

[0033] (System Configuration) The configuration of the gaze estimation system according to the second embodiment may be the same as that of the gaze estimation system according to the first embodiment (see FIG. 1), and therefore a description thereof will be omitted. Also, the hardware configuration of the gaze estimation system according to the second embodiment may be the same as that of the gaze position estimation system 10 according to the first embodiment (see FIG. 2), and therefore a description thereof will be omitted.

[0034] (Operation flow) The operation flow of the gaze estimation system 10 according to the second embodiment may be the same as the operation flow of the gaze estimation system 10 according to the first embodiment (see FIG. 3), and therefore the description thereof will be omitted. However, in the gaze estimation system 10 according to the second embodiment, the display control unit 101 controls the display mode of the gaze point or the display mode of the gaze point trajectory as follows.

[0035] (Display mode of gaze point) The display mode of the gaze point in the gaze estimation system 10 according to the second embodiment will be described with reference to Figs. 4 to 7. Fig. 4 is a conceptual diagram (part 1) showing the display mode of the gaze point by the gaze estimation system according to the second embodiment. Fig. 5 is a conceptual diagram (part 2) showing the display mode of the gaze point by the gaze estimation system according to the second embodiment. Fig. 6 is a conceptual diagram (part 3) showing the display mode of the gaze point by the gaze estimation system according to the second embodiment. Fig. 7 is a conceptual diagram (part 4) showing the display mode of the gaze point by the gaze estimation system according to the second embodiment.

[0036] As shown in FIG. 4, the gaze point X moves on the display surface of the display unit 20 so as to alternately move and stop along a predetermined trajectory. More specifically, the gaze point X moves on a predetermined straight line, stops at the end point of the straight line for a predetermined period (e.g., several seconds), moves again on another straight line from there, and stops at the end point of the straight line for a predetermined period. In this way, the subject can be easily made to gaze at the gaze point X. It is preferable that the gaze point X moves so as to pass through all the points on the display surface of the display unit 20. Here, the shape of the gaze point X is, for example, an ellipse, and the position of the gaze point X is, for example, the center position of the gaze point X. Note that the shape and position of the gaze point X are not limited to this.

[0037] The subject is asked to follow such a movement of the gaze point X with his / her eyes. In order to have the subject follow the gaze point X, a specific instruction may be output to the subject before display control of the gaze point X is started. For example, an instruction may be displayed on the display unit 20 in the form of text or the like to follow the gaze point X. Alternatively, display control may be performed so that the subject naturally follows the gaze point X with his / her eyes. For example, the gaze point X may be displayed as a character, object, or the like that attracts the subject's interest.

[0038] The moving speed and size of the gaze point X are set in advance. However, the moving speed of the gaze point X may be changed as appropriate depending on the reaction of the subject. For example, for a subject in which the delay in the movement of the gaze point relative to the movement of the gaze point is relatively large, the movement of the gaze point X may be slowed down. Alternatively, the size of the gaze point X may be changed as appropriate depending on the reaction of the subject. For example, for a subject in which the variation in the viewpoint position is relatively large, the size of the gaze point X may be increased. In addition, both the moving speed and the size of the gaze point X may be changed as appropriate depending on the reaction of the subject. The control of changing the moving speed of the gaze point X or the control of changing the size of the gaze point X may be executed using the result immediately after the display control of the gaze point X is started (for example, the result measured when moving on the first straight line).

[0039] As shown in FIG. 5, the fixation point X may be controlled to flash (in other words, to be highlighted) at a predetermined interval while moving. In this way, the subject can easily gaze at the fixation point X. Furthermore, by having the fixation point X flash at a constant interval, the subject can easily predict the movement of the fixation point X. This makes it easy to reduce the time delay of the parallax (i.e., the delay of the movement of the line of sight relative to the movement of the fixation point) to a certain value. Note that, even in the display mode shown in FIG. 5, the fixation point X may alternate between moving and stopping, as described in FIG. 4.

[0040] As shown in FIG. 6, the fixation point X may be controlled so as to alternate between a state in which it is visible while moving (X1 in the figure) and a state in which it is invisible (X2 in the figure). Even in this case, it is possible to easily get the subject to gaze at the fixation point X. Furthermore, by periodically repeating a state in which the fixation point X is visible and a state in which it is invisible, the subject can easily predict the movement of the fixation point X. This makes it easy to reduce the time delay of the parallax to a certain value. Note that, even in the display mode shown in FIG. 6, the fixation point X may alternate between a moving state and a stationary state, as described in FIG. 4.

[0041] As shown in FIG. 7, the fixation point X may be controlled to alternate between a state in which it is displayed large while moving (X3 in the figure) and a state in which it is displayed small (X4 in the figure). Even in this case, the subject can easily be made to fixate on the fixation point X. Furthermore, the periodic change in size of the fixation point X makes it easier for the subject to predict the movement of the fixation point X. This makes it easy to reduce the time delay of the parallax to a certain value. Note that, even in the display mode shown in FIG. 7, the fixation point X may alternate between a moving state and a stationary state, as described in FIG. 4.

[0042] Also, in addition to the gaze point X, a number for counting may be displayed. For example, this number may be counted up from a preset initial value to an upper limit value, or may be counted down from a preset initial value to a lower limit value. The number may be counted over time (for example, every second). Also, the number may be counted during the period from when the gaze point X starts to move until it stops, or may be counted every time it flashes, or may be counted when the position of the gaze point X and the position of the eyes are continuously within a predetermined distance.

[0043] (Technical effect of the display mode of the gaze point) Next, an example of a technical effect obtained by the viewpoint position estimation device 10 according to the display mode of the point of interest of the second embodiment will be described.

[0044] According to the viewpoint position estimation device 10 according to the second embodiment, the display mode of the gaze point is controlled as shown in Fig. 4 to Fig. 7. This makes it easier to have the subject gaze at the gaze point X. The display modes shown in Fig. 4 to Fig. 7 may be appropriately combined. For example, the gaze point X, which moves so as to alternately move and stop along a predetermined trajectory as shown in Fig. 4, may be controlled to flash at predetermined intervals while moving as shown in Fig. 5.

[0045] (Display mode of the trajectory of the gaze point) Next, the display mode of the trajectory of the gaze point X will be described with reference to Fig. 8 to Fig. 10. Fig. 8 is a conceptual diagram (part 1) showing a display mode of the trajectory of the gaze point by the gaze estimation system according to the second embodiment. Fig. 9 is a conceptual diagram (part 2) showing a display mode of the trajectory of the gaze point by the gaze estimation system according to the second embodiment. Fig. 10 is a conceptual diagram (part 3) showing a display mode of the trajectory of the gaze point by the gaze estimation system according to the second embodiment.

[0046] As shown in Fig. 8, the movement trajectory of the gaze point X may be displayed. In this way, the subject can know how the gaze point X has moved so far or how the gaze point X will move in the future. As a result, the subject can easily predict the movement of the gaze point X.

[0047] As shown in Fig. 9 and Fig. 10, the movement trajectory of the fixation point X may be displayed in different manners for the movement trajectory that has already moved and the movement trajectory that will move from there. Specifically, as shown in Fig. 9, the movement trajectory that has already moved may be displayed in a solid line, while the movement trajectory that will move from there may be displayed in a dotted line. Alternatively, as shown in Fig. 10, the movement trajectory that has already moved may be displayed normally, while the movement trajectory that will move from there may not be displayed (i.e., only the movement trajectory that has already moved may be displayed).

[0048] (Technical effect of the display mode of the trajectory of the gaze point) Next, an example of the technical effect obtained by the viewpoint position estimation device 10 according to the display mode of the trajectory of the point of interest in the second embodiment will be described.

[0049] According to the viewpoint position estimation device 10 according to the second embodiment, the display mode of the trajectory of the gaze point is controlled as shown in Fig. 8 to Fig. 10. This makes it easier for the subject to predict the movement of the gaze point X, and makes it easier to reduce the time delay of the parallax to a certain value. An example of the technical effect of reducing the time delay of the parallax to a certain value will be described in detail in the third embodiment described later.

[0050] <Third embodiment> A gaze estimation system according to the third embodiment will be described with reference to Figs. 11 and 12. In the third embodiment, an example will be described in which the gaze estimation system functions as a system for correcting (calibrating) a viewpoint position. Note that the third embodiment differs from the first and second embodiments only in some configurations and operations, and is otherwise generally similar. Therefore, hereinafter, the parts that differ from the first and second embodiments will be described in detail, and the explanation of the other overlapping parts will be omitted as appropriate.

[0051] (System Configuration) First, the overall configuration of the gaze estimation system according to the third embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing the overall configuration of the gaze estimation system according to the third embodiment. Note that the hardware configuration of the gaze estimation system according to the third embodiment may be the same as the hardware configuration of the gaze position estimation system 10 according to the first embodiment (see Fig. 2), and therefore the description thereof will be omitted.

[0052] As shown in FIG. 11, the gaze estimation system 10 according to the third embodiment is connected to a display unit 20 and an imaging unit 30. The display unit 20 is a display arranged at a position where a subject, whose gaze position is to be estimated, can be viewed, and displays a gaze point for estimating the gaze position. Display control of the gaze point on the display unit 20 is performed by the gaze estimation system 10. The imaging unit 30 is a camera installed in the periphery of the display unit 20, and is arranged at a position where an image of the subject (particularly an image around the face) can be captured. The image of the subject captured by the imaging unit 30 is configured to be output to the gaze estimation system 10.

[0053] The gaze estimation system 10 according to the third embodiment includes, in addition to the components of the first embodiment (see FIG. 1), a disparity calculation unit 103, a statistical processing unit 104, a time delay calculation unit 105, a deviation calculation unit 106, a correction value integration unit 107, and a viewpoint correction output unit 108.

[0054] The disparity calculation unit 103 calculates disparity, which is the difference between the position of the gaze point controlled by the display control unit 101 and the viewpoint position of the subject estimated by the motion detection unit 102. Note that the disparity calculated by the disparity calculation unit 103 is calculated as a value including random statistical error and a time delay. The disparity calculation unit 103 may be realized as a functional block having the above-mentioned functions in, for example, the CPU 11 (see FIG. 1).

[0055] The statistical processing unit 104 is configured to be capable of executing statistical processing for removing statistical errors contained in the disparity calculated by the disparity calculation unit 103. Specifically, the statistical processing unit 104 removes statistical errors by time-averaging the disparity calculated by the disparity calculation unit 103 for a certain period of time. Note that the above-mentioned statistical processing is merely an example, and statistical errors may be removed using other statistical processing. The statistical processing unit 104 may be realized as a functional block having the above-mentioned functions in, for example, the CPU 11 (see FIG. 1).

[0056] The time delay calculation unit 105 is configured to be able to calculate a time delay included in the parallax calculated by the parallax calculation unit 103. The time delay calculation unit 105 calculates a time delay of the parallax based on at least the parallax when the gaze point is moving. The time delay calculation unit 105 may be realized as a functional block having the above-mentioned functions in, for example, the CPU 11 (see FIG. 1).

[0057] The deviation amount calculation unit 106 calculates the deviation amount between the viewpoint position estimated by the motion detection unit 102 and the true value of the viewpoint position (i.e., the position where the subject is actually looking) based on the disparity from which the statistical error has been removed by the statistical processing unit 104 and the time delay calculated by the time delay calculation unit 105. The deviation amount calculation unit 106 may be realized as a functional block having the above-mentioned functions in, for example, the CPU 11 (see FIG. 1).

[0058] Correction value integrating unit 107 calculates a correction value (in other words, a correction amount for reducing the deviation of the viewpoint position) for the viewpoint position estimated by motion detecting unit 102, based on the deviation amount calculated by deviation amount calculating unit 106. Correction value integrating unit 107 also executes an integration process on the correction values ​​calculated at multiple points on display unit 20, and generates a correction formula for correcting the viewpoint position. By using this correction formula, it becomes possible to correct the viewpoint position even for points for which no correction value has actually been calculated. Correction value integrating unit 107 may be realized as a functional block having the above-mentioned functions, for example, in CPU 11 (see FIG. 1).

[0059] The viewpoint correction output unit 108 outputs the correction equation generated by the correction value integration unit 107 as information for performing calibration of the viewpoint position. The viewpoint correction output unit 108 may have a function of storing the generated correction equation and correcting and outputting the viewpoint position (i.e., a function of outputting the corrected viewpoint position). The viewpoint correction output unit 108 may be realized as a functional block having the above-mentioned functions in, for example, the CPU 11 (see FIG. 1).

[0060] (Operation flow) Next, the flow of operations of the gaze estimation system 10 according to the third embodiment will be described with reference to Fig. 12. Fig. 12 is a flowchart showing the flow of operations of the gaze estimation system according to the third embodiment.

[0061] 12, during operation of the gaze estimation system 10 according to the third embodiment, first, the display control unit 101 starts display control of the gaze point on the display unit 20 (step S101). Note that the display control of the gaze point may be started, for example, by a terminal operation by the subject, or may be started automatically upon detection of the presence of the subject around the display unit 20.

[0062] When the display control of the gaze point is started, the motion detection unit 102 acquires an image of the subject from the imaging unit 30 (step S102). Then, the motion detection unit 102 estimates the gaze point position of the subject from the acquired image of the subject (step S103).

[0063] Next, the disparity calculation unit 103 calculates the disparity, which is the difference between the position of the gaze point and the estimated viewpoint position (step S104). After the disparity is calculated, the statistical processing unit 104 executes statistical processing to remove statistical errors contained in the disparity (step S105).

[0064] Thereafter, the time delay calculation unit 105 calculates the time delay occurring in the disparity (step S106). Then, the deviation amount calculation unit 106 calculates the amount of deviation between the estimated viewpoint position and the actual viewpoint position based on the disparity from which the statistical error has been removed and the time delay amount of the disparity (step S107).

[0065] Once the deviation amount is calculated, it is determined whether or not the display control of the gaze point by the display control unit 101 has ended (step S108). That is, it is determined whether the gaze point has repeatedly moved and stopped along a predetermined route and has completed its movement to the end point. If it is determined that the display control has not ended (step S108: NO), the process is repeated again from step S102. As a result, the deviation amount is calculated at multiple points along the movement route of the gaze point.

[0066] If it is determined that the display control has ended (step S108: YES), the following determination unit 110 determines whether or not the subject's eyes have been following the gaze point (step S109). Note that the following determination unit 110 may be configured to determine whether or not the subject's eyes are following the gaze point before the display control ends (i.e., while the gaze point is displayed). For example, the following determination unit 110 may be configured to determine whether or not the subject's eyes are following the gaze point every time the gaze point X stops. Note that if it is determined that the subject's eyes are not following the gaze point (step S109: NO), the subsequent processing is omitted and the series of operations ends. This is because if the subject's eyes are not following the gaze point, appropriate correction of the viewpoint position cannot be performed.

[0067] On the other hand, if it is determined that the subject's eyes have been tracking the gaze point (step S109: YES), the correction value integration unit 107 calculates multiple correction values ​​from the deviation amounts calculated at multiple locations, and executes integration processing for the multiple correction values ​​(step S109). That is, a correction formula for the viewpoint position is calculated based on the multiple correction values. Then, the viewpoint correction output unit 108 outputs the correction formula generated by the correction value integration unit 107 as information for executing calibration of the viewpoint position (step S110).

[0068] (Viewpoint position correction) Next, correction of the viewpoint position by the gaze estimation system 10 according to the second embodiment will be specifically described. In the following, it is assumed that the gaze point X is displayed in the display mode as shown in FIG. 6 described in the second embodiment.

[0069] If the gaze position estimated by the motion detection unit 102 of the gaze estimation system 10 according to this embodiment is “Xgaze,est”, the correction value is “ΔXcalib”, and the statistical error is “ε”, then the true value of the subject's gaze position, “Xgaze,true”, can be expressed by the following equation (1).

[0070] TIFF2025013835000002.tif33170

[0071] When the disparity does not include a statistical error, the above formula (1) can also be expressed as the following formula (2). TIFF2025013835000003.tif33170In this case, it is possible to ignore "ε" in the following explanation and perform the calculations.

[0072] In addition, if the position of the fixation point X is "Xc" and the time delay is "ΔXdelay", it can also be expressed as in the following equation (3).

[0073] TIFF2025013835000004.tif29170

[0074] Furthermore, from the above expressions (1) and (3), the disparity "Xgaze,est-Xc" calculated by the disparity calculation unit 103 can be expressed by the following expression (4).

[0075] TIFF2025013835000005.tif28170

[0076] Here, the statistical error ε can be removed by the statistical processing executed by the statistical processing unit 104. As a result, it is possible to calculate the correction value ΔXcalib using Xgaze,est-Xc calculated by the disparity calculation unit 103 and the time delay ΔXdelay calculated by the time delay calculation unit 105.

[0077] The correction value ΔXcalib is calculated using the parallax at least when the gaze point X is moving. When the gaze point X is moving, if a sufficient amount of time has passed since the gaze point X started to move, the time delay ΔXdelay becomes small. Specifically, the gaze point moves at a constant speed, so that the subject can predict the movement of the gaze point, and the time delay becomes small and approaches a constant value. The time delay at this time can be estimated, for example, from the viewpoint position when the gaze point X stops, or the elapsed time until the viewpoint is fixed at the stopped position. Therefore, if the parallax when the gaze point X is moving is used, the correction value ΔXcalib can be calculated more easily and accurately. The parallax when the gaze point is moving may be calculated, for example, at the midpoint on the line along which the gaze point X moves.

[0078] The correction value ΔXcalib is calculated at multiple locations on the display unit 20. Then, the correction value integration unit 107 integrates the correction values ​​ΔXcalib integrated at multiple locations to generate a correction formula for the viewpoint position. The correction formula is generated as an equation such as the following equation (5) including predetermined coefficients A and b, for example.

[0079] TIFF2025013835000006.tif42170

[0080] The correction value ΔXcalib may be calculated by integrating not only a correction value calculated when the gaze point X is moving, but also a correction value calculated when the gaze point X is stationary.

[0081] In addition, the position Xc of gaze point X may be the position on the circumference of gaze point X that is closest to the viewpoint position Xgaze,est (i.e., the intersection of the line segment connecting the viewpoint position Xgaze,est and the center position of gaze point X with the circumference of gaze point X).

[0082] In the above example, a case has been described in which disparity calculation, disparity statistical processing, disparity time delay calculation, and deviation amount calculation (i.e., the processing from steps S104 to S107 in FIG. 5) are performed sequentially, but it is not necessary for all of these processes to be performed.

[0083] For example, the viewpoint position may be corrected by calculating the parallax, which allows correction to reduce the influence of the parallax.

[0084] Alternatively, the viewpoint position may be corrected by performing parallax statistical processing, which enables correction that reduces the influence of statistical errors in parallax.

[0085] Alternatively, the viewpoint position may be corrected by calculating a parallax time delay, which makes it possible to perform a correction that reduces the influence of the time delay of the parallax.

[0086] Alternatively, the viewpoint position may be corrected by calculating the amount of deviation, which makes it possible to perform correction that reduces the influence of the amount of deviation of the viewpoint position.

[0087] Furthermore, at least two of the processes of disparity calculation, disparity statistical processing, disparity time delay calculation, and deviation amount calculation may be combined and executed.

[0088] (Technical effect) Next, an example of a technical effect obtained by the gaze estimation system 10 according to the third embodiment will be described.

[0089] As described in Fig. 11 and Fig. 12, the gaze estimation system 10 according to the third embodiment can calculate a correction value for correcting an estimated gaze position (in other words, a value for calibration) by having a subject gaze at the gaze point X. In particular, in this embodiment, the correction value is calculated using parallax when the gaze point X is moving, so that the effect of the time delay ΔXdelay is reduced and the correction value can be obtained in a wider range than when the gaze point X is stationary. As a result, it becomes possible to accurately estimate where the subject is actually looking.

[0090] <Fourth embodiment> Next, the gaze estimation system 10 according to the fourth embodiment will be described with reference to Figs. 13 and 14. In the fourth embodiment, an example will be described in which the gaze estimation system 10 functions as a system for determining whether a target person is impersonating another person. Note that the third embodiment differs only in part of the configuration and operation from the first to third embodiments already described, and the other parts are generally similar. Therefore, hereinafter, the parts that differ from the first to third embodiments will be described in detail, and the explanation of the other overlapping parts will be omitted as appropriate.

[0091] (System Configuration) First, the overall configuration of the gaze estimation system 10 according to the fourth embodiment will be described with reference to Fig. 13. Fig. 13 is a block diagram showing the overall configuration of the gaze estimation system according to the fourth embodiment. Note that the hardware configuration of the gaze estimation system according to the fourth embodiment may be the same as the hardware configuration of the gaze position estimation system 10 according to the first embodiment (see Fig. 2), and therefore the description thereof will be omitted.

[0092] 13, the gaze estimation system 10 according to the fourth embodiment includes, in addition to the components (see FIG. 1) of the gaze estimation system 10 according to the third embodiment, a spoofing detection unit 201 and a determination result output unit 202. More precisely, the gaze estimation system 10 according to the fourth embodiment includes the spoofing detection unit 201 and the determination result output unit 202 instead of the correction value integrating unit 107 and the viewpoint correction output unit 108 according to the third embodiment.

[0093] The spoofing detection unit 201 is configured to be able to detect that spoofing (i.e., an illegal operation using a video or the like) is being performed based on the determination result of the following determination unit 110. For example, when the subject is actually present in front of the imaging unit 30, the amount of deviation is calculated as a value having a reasonable magnitude due to the time delay. On the other hand, when the subject is not actually present in front of the imaging unit 30 and the subject's captured video or the like is directed toward the imaging unit 30, the time delay specific to humans does not occur, and the amount of deviation is calculated as an extremely small value. Therefore, the following determination unit 110 can determine whether or not the subject is properly following the gaze point X by comparing the calculated amount of deviation with a predetermined threshold. Therefore, the spoofing detection unit can detect that spoofing is being performed based on the determination result of the following determination unit 110. Specifically, when the calculated amount of deviation is greater than a predetermined threshold, the following determination unit 110 determines that the subject is properly following the gaze point X. In this case, the spoofing detection unit 201 detects that spoofing is taking place. On the other hand, if the calculated deviation amount is smaller than a predetermined threshold, the following determination unit 110 determines that the target person is not properly following the gaze point X. In this case, the spoofing detection unit 201 does not detect that spoofing is taking place. The spoofing detection unit 201 may be realized as a functional block having the above-mentioned functions in, for example, the CPU 11 (see FIG. 1).

[0094] The determination result output unit 202 is configured to be able to output the determination result by the spoofing detection unit 201. The determination result output unit 202 may be configured to output only the result of whether or not spoofing is occurring, or may be configured to execute a predetermined operation (for example, an alert operation) when spoofing is detected. The determination result output unit 202 may be realized as a functional block having the above-mentioned functions in, for example, the CPU 11 (see FIG. 1).

[0095] (Operation flow) Next, the flow of operations of the gaze estimation system 10 according to the fourth embodiment will be described with reference to Fig. 14. Fig. 14 is a flowchart showing the flow of operations of the gaze estimation system according to the fourth embodiment.

[0096] 14, during operation of the gaze estimation system 10 according to the fourth embodiment, the processes of steps S101 to S108 are first executed as in the first embodiment. That is, a subject is asked to follow the gaze point X with his / her eyes, and the amount of deviation between the estimated gaze position and the true value is calculated from an image captured at that time.

[0097] Particularly in the fourth embodiment, when it is determined that the display control of the gaze point has ended (step S108: YES), the following determination unit 110 determines whether or not the target person's eyes have followed the gaze point (step S109). Then, when it is determined that the target person's eyes have not followed the gaze point (step S109: NO), the spoofing detection unit 201 detects that spoofing is occurring (step S201). On the other hand, when it is determined that the target person's eyes have followed the gaze point (step S109: YES), the spoofing detection unit 201 does not detect that spoofing is occurring (step S202). Then, the determination result output unit 202 outputs the determination result by the spoofing determination unit 201 (step S202).

[0098] In the fourth embodiment, spoofing may be detected by the spoofing detection unit 201 before the display control of the gaze point X is completed (in other words, the following determination unit 110 may perform a determination before the display control of the gaze point X is completed). For example, the following determination unit 110 may determine that following is not being performed without waiting for the movement of the gaze point X to be completed when it is determined that the calculated value of one deviation amount corresponds to spoofing, and at that timing, the spoofing detection unit 201 may detect that spoofing is being performed.

[0099] (Technical effect) Next, an example of a technical effect obtained by the gaze estimation system 10 according to the fourth embodiment will be described.

[0100] As described in Fig. 13 and Fig. 14, the gaze estimation system 10 according to the fourth embodiment can determine and detect whether or not spoofing is taking place based on the deviation amount calculated by the deviation amount calculation unit 106. That is, by utilizing the difference between the movement of the actual human gaze position and the mechanical movement of the gaze position, spoofing can be appropriately determined and detected. The gaze estimation system 10 according to the fourth embodiment exerts a beneficial effect when, for example, an authentication process using the gaze position is executed.

[0101] In addition, in the gaze estimation system 10 according to the fourth embodiment, it is preferable not to display the trajectory of the gaze point. Alternatively, as shown in Fig. 10, it is preferable to display the movement trajectory that has already moved, but not to display the movement trajectory that will move from now on.

[0102] <Additional Notes> The above-described embodiment may be further described as follows, but is not limited to the following.

[0103] (Appendix 1) The gaze position estimation system described in Appendix 1 is a gaze estimation system characterized by comprising: a display control means for displaying a gaze point at which a subject is gazing so as to move in a predetermined moving manner; a detection means for detecting eye movement of the subject from an image of the subject; and a tracking determination means for determining whether the eyes of the subject are tracking the gaze point based on a relationship between the movement of the gaze point and the movement of the eyes.

[0104] (Appendix 2) The gaze estimation system described in Supplementary Note 2 is the gaze estimation system described in Supplementary Note 1, characterized in that the display control means displays the gaze point in at least one of a first mode in which the gaze point is moved so that a first period in which the gaze point moves at a predetermined acceleration and a second period in which the gaze point is stopped for a predetermined period are alternately repeated, a second mode in which the gaze point is highlighted at constant intervals, a third mode in which the gaze point is alternately displayed and hidden at constant intervals, and a fourth mode in which a size of the gaze point is changed while it is moving.

[0105] (Appendix 3) The gaze estimation system described in Supplementary Note 3 is the gaze estimation system described in Supplementary Note 1 or 2, characterized in that the display control means displays, among the movement trajectories of the moving point of the gaze point, a trajectory through which the gaze point has passed and a trajectory through which the gaze point has not passed in different display modes, or displays, among the movement trajectories of the moving point of the gaze point, at least one of a trajectory through which the gaze point has passed and a trajectory through which the gaze point has not passed.

[0106] (Appendix 4) The gaze estimation system described in Supplementary Note 4 is the gaze estimation system described in any one of Supplementary Notes 1 to 3, characterized in that it includes a correction means for correcting a parameter related to the detected eye movement of the subject depending on whether the subject's eyes are tracking the gaze point.

[0107] (Appendix 5) The gaze estimation system described in Supplementary Note 5 is the gaze estimation system described in Supplementary Note 4, characterized in that the correction means calculates a disparity, which is a difference between the position of the point of gaze and a gaze position of the subject, from the detected eye movement of the subject, calculates a time delay of the disparity from the disparity calculated while the gaze point is moving, calculates a deviation amount from a true value of the estimated gaze position based on the disparity and the time delay of the disparity, and determines a correction value for correcting the gaze position of the subject based on the deviation amount.

[0108] (Appendix 6) In the gaze estimation system described in Supplementary Note 6, the correction means calculates the amount of deviation at a plurality of points in an area where the gaze point moves, and corrects the gaze position by integrating a plurality of correction values ​​determined based on the amount of deviation calculated at the plurality of points. The gaze estimation system according to claim 5,

[0109] (Appendix 7) The gaze estimation system described in Appendix 7 is the gaze estimation system described in any one of claims 1 to 6, characterized in that it further comprises an impersonation detection means for detecting impersonation of the subject when the eyes of the subject are not tracking the gaze point.

[0110] (Appendix 8) The gaze estimation system described in Appendix 8 is the gaze estimation system described in Appendix 7, characterized in that the impersonation detection means calculates a disparity, which is a difference between the position of the point of gaze and a gaze position of the subject, from the detected eye movement of the subject, calculates a time delay of the disparity from the disparity calculated while the gaze point is moving, calculates a deviation amount from the true value of the estimated gaze position based on the disparity and the time delay of the disparity, and detects impersonation of the subject based on the deviation amount.

[0111] (Appendix 9) The gaze estimation device described in Appendix 8 is the gaze estimation system described in Appendix 7 or 8, characterized in that the display control means does not display the movement trajectory of the gaze point when the impersonation detection means detects impersonation.

[0112] (Appendix 10) The gaze estimation method described in Appendix 10 is a gaze estimation method characterized by displaying a gaze point at which a subject is gazing so as to move in a predetermined moving manner, detecting eye movement of the subject from an image of the subject, and determining whether or not the eyes of the subject are following the gaze point based on a relationship between the movement of the gaze point and the eye movement.

[0113] (Appendix 11) The computer program described in Appendix 11 is a computer program characterized by causing a computer to operate in the following manner: displaying a point of gaze at which a subject is gazing, the point of gaze being moved in a predetermined manner; detecting eye movement of the subject from an image of the subject; and determining whether or not the subject's eyes are following the point of gaze, based on a relationship between the movement of the point of gaze and the movement of the eyes.

[0114] This disclosure may be modified as appropriate within the scope that does not contradict the gist or idea of ​​the invention that can be read from the claims and the entire specification, and the gaze estimation system, gaze estimation method, and computer program involving such modifications are also included in the technical idea of ​​this disclosure. [Explanation of symbols]

[0115] 10. Gaze Estimation System 20 Display section 30 Imaging unit 101 Display control unit 102 Motion detection unit 103 Parallax calculation unit 104 Statistical Processing Unit 105 Time Delay Calculation Unit 106 Deviation calculation unit 107 Correction value integration unit 108 Viewpoint correction output unit 110 Following judgment unit 201 Spoofing Detection Unit 202 Judgment result output unit

Claims

1. a display control means for displaying a gaze point gazed at by a subject so as to move in a manner that alternates between a first period during which the gaze point moves at a predetermined speed and a second period during which the gaze point stops for a predetermined period; a detection means for detecting eye movements of the subject from an image of the subject; a tracking determination means for determining whether or not the subject's eyes are tracking the gaze point each time the gaze point stops based on the eye movement; an impersonation detection means for detecting impersonation of the target person based on a determination result of the following determination means; A gaze estimation system comprising:

2. The gaze estimation system according to claim 1, characterized in that the display control means displays the gaze point in at least one of a first mode in which the gaze point moves repeatedly between a first period during which the gaze point moves at a predetermined speed and a second period during which the gaze point is stopped for a predetermined period, a second mode in which the gaze point is highlighted at regular intervals, a third mode in which the gaze point is displayed so that it is alternately displayed and hidden at regular intervals, and a fourth mode in which the size of the gaze point is changed while it is moving.

3. The gaze estimation system according to claim 1 or 2, characterized in that the display control means displays, among the movement trajectories of the moving gaze point, a trajectory through which the gaze point has passed and a trajectory through which the gaze point has not passed in different display modes, or displays, among the movement trajectories of the moving gaze point, at least one of a trajectory through which the gaze point has passed and a trajectory through which the gaze point has not passed.

4. The correction means is Calculating the amount of deviation at a plurality of points in an area where the gaze point moves; The plurality of correction values ​​determined based on the deviation amounts calculated at the plurality of locations are integrated to correct the viewpoint position. The gaze estimation system according to claim 3 .

5. The gaze estimation system according to claim 1 , wherein the spoofing detection means detects spoofing of the subject when the eyes of the subject do not follow the gaze point.

6. The spoofing detection means comprises: Calculating a disparity between the position of the gaze point and the viewpoint position of the subject from the detected eye movement of the subject; Calculating a time delay of the parallax from the parallax calculated while the gaze point is moving; Calculating a deviation amount from a true value of the estimated viewpoint position based on the parallax and a time delay of the parallax; Detecting impersonation of the target person based on the deviation amount. The gaze estimation system according to claim 5 .

7. Displaying a gaze point that a subject gazes at such that a first period during which the gaze point moves at a predetermined speed and a second period during which the gaze point stops for a predetermined period are repeated; detecting eye movements of the subject from an image of the subject; determining whether the subject's eyes are tracking the gaze point each time the gaze point stops based on the eye movement; Detecting impersonation of the subject based on a determination result of whether the subject's eyes are tracking the gaze point. A gaze estimation method comprising:

8. Displaying a gaze point that a subject gazes at such that a first period during which the gaze point moves at a predetermined speed and a second period during which the gaze point stops for a predetermined period are repeated; detecting eye movements of the subject from an image of the subject; determining whether the subject's eyes are tracking the gaze point each time the gaze point stops based on the eye movement; Detecting impersonation of the subject based on a determination result of whether the subject's eyes are tracking the gaze point. A computer program for causing a computer to operate in such a manner.

Images