Eyeball movement measuring device, eyeball movement measuring program, and eyeball movement measuring method

Abstract

To provide an eyeball movement measuring device, an eyeball movement measuring program, and an eyeball movement measuring method capable of accurately measuring an eyeball movement even when there is a movement in a subject.SOLUTION: An eyeball movement measuring device 100 includes: a subject position calculation unit 140 for calculating a subject position change amount of a subject from the face image of the subject; an eye angle change amount calculation unit 120 for calculating the eye angle change amount of the subject from the eye image of the subject; an eyeball angle change amount calculation unit 130 for calculating the eyeball angle change amount of the subject from the eye image and correcting the eyeball angle change amount on the basis of the eye angle change amount; and an eyeball movement determination unit 150 for determining the eyeball movement of the subject on the basis of the subject position change amount and the corrected eyeball angle change amount.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an eye movement measurement device, an eye movement measurement program, and an eye movement measurement method.

Background Art

[0002] In recent years, attempts have been made to non-contact measure the eyes of a subject and determine the subject's medical condition or level of arousal from the eye movements derived from the measurement results. Therefore, a technique for non-contact measurement of eye movements is disclosed in Patent Document 1.

[0003] The eye movement feature amount calculation system described in Patent Document 1 images the eyes of a subject at a high frame rate capable of detecting microsaccades, calculates the feature amount of eye movement from the captured image, and determines the state of the subject's brain based on the feature amount.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the technique described in Patent Document 1 assumes that the head of the subject is fixed, and there is a problem that accurate measurement cannot be performed when the position or orientation of the subject's head changes.

[0006] An object of the present disclosure is to provide an eye movement measurement device, an eye movement measurement program, and an eye movement measurement method capable of accurately measuring eye movements even when the subject moves.

Means for Solving the Problems

[0007] The eye movement measurement device according to the present disclosure includes a subject position calculation unit that calculates a subject position change amount, which is a change amount of the three-dimensional position of the subject, from a face image of the subject, an eye angle change amount calculation unit that calculates an eye angle change amount indicating a change amount of the angle of the eye of the subject from an eye image obtained by imaging the eye of the subject, an eye angle change amount calculation unit that calculates an eye angle change amount, which is a change amount of the angle of the eyeball of the subject, from the eye image and corrects the calculated eye angle change amount based on the eye angle change amount calculated by the eye angle change amount calculation unit, and an eye movement determination unit that determines the eye movement of the subject based on the subject position change amount and the corrected eye angle change amount.

[0008] The eye movement measurement program according to the present disclosure is a program that causes a computer to execute a process of calculating a subject position change amount, which is a change amount of the three-dimensional position of the subject, from a face image of the subject, a process of calculating an eye angle change amount indicating a change amount of the angle of the eye of the subject from an eye image obtained by imaging the eye of the subject, a process of calculating an eye angle change amount, which is a change amount of the angle of the eyeball of the subject, from the eye image and correcting the calculated eye angle change amount based on the eye angle change amount, and a process of determining the eye movement of the subject based on the subject position change amount and the corrected eye angle change amount.

[0009] The eye movement measurement method according to the present disclosure is a method in which a computer calculates a subject position change amount, which is a change amount of the three-dimensional position of the subject, from a face image of the subject, calculates an eye angle change amount indicating a change amount of the angle of the eye of the subject from an eye image obtained by imaging the eye of the subject, calculates an eye angle change amount, which is a change amount of the angle of the eyeball of the subject, from the eye image, corrects the calculated eye angle change amount based on the eye angle change amount, and determines the eye movement of the subject based on the subject position change amount and the corrected eye angle change amount.

Effect of the Invention

[0010] According to the present disclosure, it is possible to provide an eye movement measurement device, an eye movement measurement program, and an eye movement measurement method that can accurately measure eye movements even when the subject is moving.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Modes for Carrying Out the Invention

[0012] For the sake of clarity of explanation, the following description and drawings are appropriately omitted and simplified. Also, each element described in the drawings as a functional block for performing various processes can be composed of a CPU (Central Processing Unit), a memory, and other circuits in terms of hardware, and can be realized by a program loaded into the memory or the like in terms of software. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and are not limited to any one. In each drawing, the same elements are denoted by the same reference numerals, and duplicate explanations are omitted as necessary.

[0013] When the above program is loaded into a computer, it includes a set of instructions (or software code) for causing the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, a computer-readable medium or a tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD), or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disc, or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices. The program may also be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, a transitory computer-readable medium or a communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0014] Embodiment 1 Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 shows a schematic block diagram of an eye movement measurement device 100 according to Embodiment 1. The eye movement measurement device 100 shown in FIG. 1 is realized by an arithmetic unit 110. Specifically, the eye movement measurement device 100 can be realized by performing the processing of each functional block described later by an eye measurement program executed by the arithmetic unit 110. Note that the eye movement measurement device 100 can also be configured as dedicated hardware for the processing performed within the arithmetic unit 110.

[0015] The eye movement measurement device 100 according to Embodiment 1 irradiates the eyes of a subject with observation light, and determines the eye movement of the subject based on the change in the distance between the center point of the reflected light due to the fundus recursive reflection of the pupil part and the center point of the reflected light of the direct reflection of the cornea part. However, when the position and orientation of the subject's face change, an error occurs in the calculation of the center point due to the difference in the intensity of the reflected light levels of the pupil and the cornea, and it becomes impossible to accurately measure the eye movement of the subject. Therefore, the eye movement measurement device 100 according to Embodiment 1 calculates the amount of change in the three-dimensional position of the subject (hereinafter referred to as the "subject position change amount") from the face image of the subject. Further, the eye movement measurement device 100 calculates the amount of change in the angle of the eyeball (hereinafter referred to as the "eyeball angle change amount") from the eye image obtained by imaging the eyes of the subject. Then, the eye movement measurement device 100 determines the eye movement based on the subject position change amount and the eyeball angle change amount. However, since the accuracy of the subject position change amount calculated from the face image is not high, it may be difficult to sufficiently increase the measurement accuracy of the eye movement. Therefore, the eye movement measurement device 100 according to Embodiment 1 further calculates the amount of change in the angle of the entire eye of the subject (hereinafter referred to as the "eye angle change amount") from the eye image, and first corrects the eyeball angle change amount based on the eye angle change amount. Next, the eye movement measurement device 100 determines the eye movement based on the subject position change amount and the corrected eyeball angle change amount.

[0016] Specifically, as shown in FIG. 1, the eye movement measurement device 100 according to Embodiment 1 includes an eye angle change amount calculation unit 120, an eyeball angle change amount calculation unit 130, a subject position calculation unit 140, and an eye movement determination unit 150. The eye angle change amount calculation unit 120 calculates the eye angle change amount of the subject from the eye image. The eyeball angle change amount calculation unit 130 calculates the eyeball angle change amount of the subject from the eye image, and corrects the calculated eyeball angle change amount based on the eye angle change amount calculated by the eye angle change amount calculation unit 120. The subject position calculation unit 140 calculates the subject position change amount of the subject from the face image. The eye movement determination unit 150 determines the eye movement of the subject based on the subject position change amount and the corrected eyeball angle change amount.

[0017] In the eye movement measurement device 100 according to Embodiment 1, the amount of change in the eye angle calculated from the eye image is used to correct the amount of change in the eye ball angle. The accuracy of the amount of change in the eye angle calculated from the eye image is higher than the amount of change in the subject's position calculated from the face image. Therefore, by correcting the amount of change in the eye ball angle based on the amount of change in the eye angle, the measurement accuracy of the eye movement can be improved. In particular, the measurement accuracy of the eye movement can be improved to such an extent that micro-saccades necessary for determining the subject's medical condition can be detected.

[0018] Although the schematic block diagram of the eye movement measurement device 100 was shown in FIG. 1, specific examples of the eye movement measurement device 100 include a camera as a photographing means, a processing block for calculating the amount of change in the eye angle, the amount of change in the eye ball angle, the three-dimensional position change amount, etc., and a processing block for performing various processes for controlling the camera. Therefore, in order to explain FIG. 1 more specifically in FIG. 2, FIG. 2 shows a block diagram of the eye movement measurement device 100A according to Embodiment 1. Hereinafter, the eye movement measurement device 100 will be described in more detail with reference to the eye movement measurement device 100A.

[0019] The eye movement measurement device 100A shown in FIG. 2 includes, in addition to an eye angle change amount calculation unit 120A, an eye ball angle change amount calculation unit 130A, a subject position calculation unit 140A, and a detector 150A as an eye movement determination unit 150, a first camera 160, a second camera 170, and a camera control unit 180. Further, the eye movement measurement device 100A includes an illumination device 172 that irradiates observation light onto the subject's eyes. Note that specific devices are used as the sensor and the illumination device in the eye movement measurement device 100A. In addition, information processing parts such as the eye angle change amount calculation unit 120A, the eye ball angle change amount calculation unit 130A, the subject position calculation unit 140A, and the detector 150A are realized by an eye movement measurement program executed by the arithmetic unit 110.

[0020] The first camera 160 is a wide-angle camera and captures a face image including the face of the subject. Here, the face image captured by the first camera 160 will be described. FIG. 3 shows a diagram for explaining the face image according to the first embodiment. As shown in FIG. 3, the face image is an image including the face of the subject. In the following description, the horizontal direction of the face image is the X-axis direction, the vertical direction is the Y-axis direction, the depth direction of the face image is the Z-axis direction (see FIG. 4), and the face-down angle is the pitch angle θ pt , and the face tilt direction is the roll angle θ r . In other words, the pitch angle θ pt is the angle of the face around the X-axis, and the roll angle θ r is the angle of the face around the Z-axis.

[0021] The second camera 170 is a telephoto camera and captures an eye image including the eyes of the subject. Here, the eye image captured by the second camera 170 will be described. FIG. 4 shows a diagram for explaining the imaging ranges of the first camera 160 and the second camera 170. Also, FIG. 5 shows a diagram for explaining the eye image according to the first embodiment. As shown in FIGS. 4 and 5, while the first camera 160 captures a wide range including the face of the subject, the second camera 170 captures only the vicinity of the eyes of the subject. Also, the illumination device 172 irradiates observation light centered around the vicinity of the subject's eyeballs. Also, the imaging direction of the second camera 170 and the irradiation direction of the illumination device 172 are adjusted by a mirror 171 rotatably installed in the eyeball movement measurement device 100. In the following description, the orientation of the horizontal direction of the subject's face is the yaw angle θ y . In other words, the pitch angle θ pt is the angle of the face around the X-axis, the roll angle θ r is the angle of the face around the Z-axis, and the yaw angle θ y is the angle of the face around the Y-axis.

[0022] As shown in FIG. 2, the eye angle change amount calculation unit 120A includes a head end / tail end extraction unit 121 and a roll angle calculation unit 122. Further, the eyeball angle change amount calculation unit 130A includes a corneal reflection region extraction unit 131, a pupil recursive reflection region extraction unit 132, center point calculation units 133 and 134, and an eyeball angle change amount correction unit 135. The subject position calculation unit 140A includes a face / head end / tail end extraction unit 141, a yaw angle calculation unit 142, a pitch angle calculation unit 143, and a face orientation change amount calculation unit 144.

[0023] The head end / tail end extraction unit 121 extracts the position of the head end and the position of the tail end from the eye image captured by the second camera 170. FIG. 5 shows the position P1 of the head end and the position P2 of the tail end extracted by the head end / tail end extraction unit 121. Note that the head end / tail end extraction unit 121 extracts the position P1 of the head end and the position P2 of the tail end from the eye image using existing image recognition processing.

[0024] Based on the position P1 of the head end and the position P2 of the tail end extracted by the head end / tail end extraction unit 121, the roll angle calculation unit 122 calculates the roll angle θ as the eye angle change amount. r Specifically, as shown in FIG. 5, the roll angle calculation unit 122 detects an eye coordinate system with the straight line connecting the position P1 of the head end and the position P2 of the tail end as the X'-axis and the axis orthogonal to the X'-axis as the Y'-axis. Then, the roll angle calculation unit 122 calculates the inclination angle of the eye coordinate system (X', Y') from the image coordinate system (X, Y) of the eye image captured by the second camera 170 as the roll angle θ. r and calculates it.

[0025] The corneal reflection region extraction unit 131 extracts a corneal reflection region Mc, which is a region of a Purkinje image that is the image of the light irradiated by the illumination device 172 and reflected by the subject's cornea, from the eye image captured by the second camera 170. Note that the corneal reflection region extraction unit 131 extracts the corneal reflection region Mc from the eye image using existing image recognition processing.

[0026] The pupil recursive reflection region extraction unit 132 extracts a pupil recursive reflection region Mp, which is a region of an image of light irradiated by the illumination device 172 and recursively reflected in the pupil of the subject, from the eye image captured by the second camera 170. Note that the pupil recursive reflection region extraction unit 132 extracts the pupil recursive reflection region Mp from the eye image using existing image recognition processing.

[0027] The center point calculation unit 133 calculates the image coordinates Cc(X, Y) of the center point of the corneal reflection region Mc extracted by the corneal reflection region extraction unit 131.

[0028] The center point calculation unit 134 calculates the image coordinates Pc(X, Y) of the center point of the pupil recursive reflection region Mp extracted by the pupil recursive reflection region extraction unit 132.

[0029] The eyeball angle change amount correction unit 135 is the roll angle θ as the eyeball angle change amount r , based on the image coordinates Cc(X, Y) of the center point of the corneal reflection region Mc and the image coordinates Pc(X, Y) of the center point of the pupil recursive reflection region Mp, calculates the eyeball angle change amount θ p . Specifically, the eyeball angle change amount correction unit 135 calculates the eyeball angle change amount θ p ' before correction based on the image coordinates Cc(X, Y) of the center point of the corneal reflection region Mc and the image coordinates Pc(X, Y) of the center point of the pupil recursive reflection region Mp. Next, the eyeball angle change amount correction unit 135 corrects the eyeball angle change amount θ r ' based on the roll angle θ as the eyeball angle change amount. Then, the eyeball angle change amount correction unit 135 inputs the corrected eyeball angle change amount θ p ' to the detector 150A. Here, the calculation of the eyeball angle change amount θ p ' before correction is performed using an existing calculation method, for example, the PCCR (Pupil Centre Corneal Reflection) method. Also, the calculated eyeball angle change amount θ p ' includes a yaw angle component, a roll angle component, and a pitch angle component. And the eyeball angle change amount correction unit 135, based on the roll angle θ p as the eyeball angle change amount, based on the roll angle θ r calculates the eyeball angle change amount θp correct the roll angle component of ‘

[0030] The face / inner corner of eye / outer corner of eye extraction unit 141 extracts the position of the face, the position of the inner corner of the eye, and the position of the outer corner of the eye from the face image captured by the first camera 160. Note that the face / inner corner of eye / outer corner of eye extraction unit 141 extracts the position of the face, the position of the inner corner of the eye, and the position of the outer corner of the eye from the face image using existing image recognition processing.

[0031] Based on the position of the face, the position of the inner corner of the eye, and the position of the outer corner of the eye extracted by the face / inner corner of eye / outer corner of eye extraction unit 141, the yaw angle calculation unit 142 calculates the yaw angle θ which is the lateral orientation of the subject's face. y calculate it.

[0032] Based on the position of the face, the position of the inner corner of the eye, and the position of the outer corner of the eye extracted by the face / inner corner of eye / outer corner of eye extraction unit 141, the pitch angle calculation unit 143 calculates the pitch angle θ which is the downward angle of the subject's face. pt calculate it.

[0033] The face orientation change amount calculation unit 144 calculates the subject position change amount which is the change amount of the three-dimensional position of the subject. Specifically, the face orientation change amount calculation unit 144 calculates, as the subject position change amount, the change amount of the orientation of the subject's face (hereinafter referred to as "face orientation change amount") θ. f calculate it. More specifically, the face orientation change amount calculation unit 144 calculates the face orientation change amount θ based on the yaw angle θ calculated by the yaw angle calculation unit 142. y and the pitch angle θ calculated by the pitch angle calculation unit 143. pt Based on this, the face orientation change amount θ. f calculate it.

[0034] Based on the eye angle change amount θ calculated by the eye angle change amount correction unit 135, the face orientation change amount θ of the subject calculated by the face orientation change amount calculation unit 144, and the orientation θ of the mirror 171 input from the camera control unit 180 (described later), the detector 150A determines the eye movement of the subject. Specifically, the detector 150A determines the time series data of the eye angle change amount θ. p and the face orientation change amount θ of the subject calculated by the face orientation change amount calculation unit 144. f and the orientation θ of the mirror 171 input from the camera control unit 180. m (described later), the detector 150A determines the eye movement of the subject. Specifically, the detector 150A determines the time series data of the eye angle change amount θ. p and the face orientation change amount θ. fTime-series data and the orientation θ of the mirror 171 m Based on the time-series data, the type of the subject's eye movement is classified. A specific implementation method of the detector 150A will be described later. Note that the detector 150A may classify the type of the subject's eye movement without using the time-series data of the orientation θ m of the mirror 171.

[0035] The camera control unit 180 adjusts the imaging direction of the second camera 170 and the irradiation direction of the illumination device 172 by adjusting the orientation θ m of the mirror 171. Further, the camera control unit 180 controls the focus of the second camera 170. Further, the camera control unit 180 inputs the orientation θ m of the mirror 171 to the detector 150A as control information of the second camera 170.

[0036] Here, as a method for realizing the detector 150A, two realization methods, i.e., rule-based processing (the first example) and neural network processing (the second example), are conceivable. Therefore, examples of realizing the detector 150A are shown in FIGS. 6 and 7, and these two realization methods will be described.

[0037] FIG. 6 is a block diagram for explaining a first example of the detector 150A according to the first embodiment. In FIG. 6, the detector 150A as the first example is denoted by the reference numeral 151. The detector 151 shown in FIG. 6 includes a feature amount extraction unit 152, a feature amount classification unit 153, and a result output unit 154. The feature amount extraction unit 152 calculates, as an eye movement, a difference obtained by subtracting the orientation change amount θ p of the face input from the face orientation change amount calculation unit 144 and the orientation θ f of the mirror 171 input from the camera control unit 180 from the eye angle change amount θ m input from the eye angle change amount correction unit 135. The feature amount classification unit 153 classifies the eye movement calculated by the feature amount extraction unit 152 according to a preset classification rule and attaches a label to each eye movement. The result output unit 154 displays, on a display unit (not shown), a result indicating how much each eye movement is included for each label.

[0038] FIG. 7 is a block diagram for explaining a second example of the detector 150A according to the first embodiment. In FIG. 7, the detector 150A as the second example is denoted by the reference numeral 155. The detector 155 shown in FIG. 7 includes a feature quantity classification unit 156 and a result output unit 157. The feature quantity classification unit 156 includes a neural network that has been pre-trained. Then, based on the amount of change in eye angle θ input from the eye angle change amount correction unit 135, the amount of change in the subject's face orientation θ input from the face orientation change amount calculation unit 144, and the orientation θ of the mirror 171 input from the camera control unit 180, and the learning result, the neural network classifies the eye movement. Also, based on the classification result, the neural network attaches a label indicating the type of eye movement. The result output unit 157 displays on a display unit (not shown) the result of how much each type of eye movement is included. p , the amount of change in the subject's face orientation θ input from the face orientation change amount calculation unit 144 f , and the orientation θ of the mirror 171 input from the camera control unit 180 m and the learning result, classifies the eye movement. Also, based on the classification result, the neural network attaches a label indicating the type of eye movement. The result output unit 157 displays on a display unit (not shown) the result of how much each type of eye movement is included.

[0039] As described above, in the eye movement measurement device 100A according to the first embodiment, the eye angle change amount calculation unit 130A corrects the eye angle change amount based on the roll angle θ calculated as the eye angle change amount from the eye image. The accuracy of the roll angle θ calculated from the eye image is higher than the roll angle calculated from the face image. Therefore, by correcting the eye angle change amount based on the roll angle θ, the measurement accuracy of the eye movement can be improved. In particular, the measurement accuracy of the eye movement can be improved to such an extent that microsaccades necessary for determining the subject's medical condition can be detected. r based on, corrects the eye angle change amount. The accuracy of the roll angle θ calculated from the eye image r is higher than the roll angle calculated from the face image. Therefore, by correcting the eye angle change amount based on the roll angle θ r , the measurement accuracy of the eye movement can be improved. In particular, the measurement accuracy of the eye movement can be improved to such an extent that microsaccades necessary for determining the subject's medical condition can be detected.

[0040] Also, in the eye movement measurement device 100A according to the first embodiment, by using the first camera 160 and the second camera 170, the accuracy of specifying the three-dimensional position of the subject can be increased. Also, in the eye movement measurement device 100A, by moving the second camera 170 and the lighting device 172 in conjunction with each other, the relevance between the incident angle of the observation light with respect to the pupil and the eye image captured by the second camera 170 can be increased.

[0041] In addition, in the eye movement measurement device 100A according to Embodiment 1, the detector 150A determines the eye movement of the subject based on the amount of change in the eye angle θ p and the amount of change in the face orientation θ f , and in addition to the orientation θ m of the mirror 171 input from the camera control unit 180. Thereby, the measurement accuracy of the eye movement can be further improved.

[0042] In addition, in the eye movement measurement device 100A according to Embodiment 1, either rule-based processing or neural network processing can be applied as the detector 150A, and the versatility of the system can be enhanced.

[0043] Note that the present invention is not limited to the above-described embodiments, and can be appropriately changed without departing from the spirit thereof.

[0044] Some or all of the above embodiments can be described as follows in the appended claims, but are not limited thereto. (Appended Claim 1) A subject position calculation unit that calculates a subject position change amount, which is a change amount of the three-dimensional position of the subject, from the face image of the subject, An eye angle change amount calculation unit that calculates an eye angle change amount indicating a change amount of the angle of the eye of the subject from an eye image obtained by imaging the eye of the subject, An eye movement measurement device comprising: an eye angle change amount calculation unit that calculates an eye angle change amount, which is a change amount of the angle of the eyeball of the subject, from the eye image, and corrects the calculated eye angle change amount based on the eye angle change amount calculated by the eye angle change amount calculation unit; An eye movement determination unit that determines the eye movement of the subject based on the subject position change amount and the corrected eye angle change amount. (Appended Claim 2) A first camera that captures the face image of the subject, A second camera that captures the eye image of the subject, A camera control unit that controls at least the imaging direction of the second camera, comprising The eye movement determination unit further determines the eye movement of the subject based on the control information of the second camera by the camera control unit. The eye movement measurement device according to Supplementary Note 1. (Supplementary Note 3) The eye movement determination unit classifies the type of eye movement of the subject based on the time series data of the subject position change amount and the time series data of the eye angle change amount. The eye movement measurement device according to Supplementary Note 1. (Supplementary Note 4) The eye movement determination unit neural network processing that takes as input the time series data of the subject position change amount and the time series data of the eye angle change amount, rule-based processing that classifies the eye movement calculated from the difference between the time series data of the eye angle change amount and the time series data of the subject position change amount according to a preset rule, classifies the type of eye movement by either one of the above processes. The eye movement measurement device according to Supplementary Note 3. (Supplementary Note 5) The three-dimensional position of the subject includes at least the roll angle, pitch angle, and yaw angle representing the orientation of the subject's face in the face image. The eye movement measurement device according to Supplementary Note 1. (Supplementary Note 6) On a computer, a process of calculating a subject position change amount, which is a change amount of the three-dimensional position of the subject, from the face image of the subject, a process of calculating an eye angle change amount indicating a change amount of the angle of the eye of the subject from an eye image obtained by imaging the eye of the subject, a process of calculating an eyeball angle change amount, which is a change amount of the angle of the eyeball of the subject, from the eye image and correcting the calculated eyeball angle change amount based on the eye angle change amount, a process of determining the eye movement of the subject based on the subject position change amount and the corrected eyeball angle change amount, An eye movement measurement program for causing the above to be executed. (Supplementary Note 7) Cause the computer to perform at least a process of controlling at least the imaging direction of the camera that images the eye image of the subject, and further cause the computer to in the process of determining the eye movement of the subject, further determine the eye movement of the subject based on the control information of the camera. The eye movement measurement program according to Supplementary Note 6. (Supplementary Note 8) Cause the computer to in the process of determining the eye movement of the subject, classify the type of eye movement of the subject based on the time-series data of the subject position change amount and the time-series data of the eye angle change amount. The eye movement measurement program according to Supplementary Note 6. (Supplementary Note 9) The process of determining the eye movement of the subject is either neural network processing that takes as input the time-series data of the subject position change amount and the time-series data of the eye angle change amount, or rule-based processing that classifies the eye movement calculated from the difference between the time-series data of the eye angle change amount and the time-series data of the subject position change amount according to a preset rule. That is, it is either one of the above processes. The eye movement measurement program according to Supplementary Note 8. (Supplementary Note 10) The three-dimensional position of the subject includes at least the pitch angle and the yaw angle representing the orientation of the face of the subject in the face image. The eye movement measurement program according to Supplementary Note 6. (Supplementary Note 11) The computer calculates the subject position change amount, which is the change amount of the three-dimensional position of the subject, from the face image of the subject, and calculates the eye angle change amount indicating the change amount of the angle of the eye of the subject from the eye image that images the eye of the subject. Calculate the amount of change in the angle of the subject's eyeball, which is the amount of change in the eyeball angle, from the eye image, and correct the calculated amount of change in the eyeball angle based on the amount of change in the eye angle. Determine the eye movement of the subject based on the amount of change in the position of the subject and the corrected amount of change in the eyeball angle. Eye movement measurement method. (Appendix 12) The computer further Controls at least the imaging direction of the camera that images the eye image of the subject. The computer When determining the eye movement of the subject, further determine the eye movement of the subject based on the control information of the camera. The eye movement measurement method according to Appendix 11. (Appendix 13) The computer When determining the eye movement of the subject, classify the type of eye movement of the subject based on the time-series data of the amount of change in the position of the subject and the time-series data of the amount of change in the eyeball angle. The eye movement measurement method according to Appendix 11. (Appendix 14) The computer When determining the eye movement of the subject, Neural network processing that takes as input the time-series data of the amount of change in the position of the subject and the time-series data of the amount of change in the eyeball angle, and Rule-based processing that classifies the eye movement calculated from the difference between the time-series data of the amount of change in the eyeball angle and the time-series data of the amount of change in the position of the subject according to a preset rule. Classify the type of eye movement by either one of the above processes. The eye movement measurement method according to Appendix 13. (Appendix 15) The three-dimensional position of the subject includes at least the pitch angle and the yaw angle representing the orientation of the face of the subject in the face image. The eye movement measurement method according to Appendix 11.

Explanation of symbols

[0045] 100, 100A Ocular Movement Measuring Device 110 Calculation Unit 120, 120A Eye Angle Change Amount Calculation Unit 121 Eyebrow Head and Eyebrow Tail Extraction Unit 122 Roll Angle Calculation Unit 130, 130A Ocular Globe Angle Change Amount Calculation Unit 131 Corneal Reflection Region Extraction Unit 132 Pupil Recursive Reflection Region Extraction Unit 133 Center Point Calculation Unit 134 Center Point Calculation Unit 135 Ocular Globe Angle Change Amount Correction Unit 140, 140A Subject Position Calculation Unit 141 Face, Eyebrow Head and Eyebrow Tail Extraction Unit 142 Yaw Angle Calculation Unit 143 Pitch Angle Calculation Unit 144 Face Orientation Change Amount Calculation Unit 150 Ocular Movement Determination Unit 150A, 151, 155 Detector (Ocular Movement Determination Unit) 152 Feature Amount Extraction Unit 153 Feature Amount Classification Unit 154 Result Output Unit 156 Feature Amount Classification Unit 157 Result Output Unit 160 First Camera 170 Second Camera 171 Mirror 172 Lighting Device 180 Camera Control Unit

Claims

1. A subject position calculation unit that calculates a subject position change amount, which is a change amount of the three-dimensional position of the subject, from a face image of the subject; An eye angle change amount calculation unit that calculates an eye angle change amount indicating a change amount of the angle of the eyes of the subject from an eye image obtained by imaging the eyes of the subject; An eyeball angle change amount calculation unit that calculates an eyeball angle change amount, which is a change amount of the angle of the eyeballs of the subject, from the eye image, and corrects the calculated eyeball angle change amount based on the eye angle change amount calculated by the eye angle change amount calculation unit; An eyeball movement determination unit that determines the eyeball movement of the subject based on the subject position change amount and the corrected eyeball angle change amount; An eyeball movement measurement device comprising:

2. A first camera that captures the face image of the subject; A second camera that captures the eye image of the subject; A camera control unit that controls at least the imaging direction of the second camera; comprising: The eyeball movement determination unit further determines the eyeball movement of the subject based on control information of the second camera by the camera control unit. The eyeball movement measurement device according to claim 1.

3. The eyeball movement determination unit classifies the type of eyeball movement of the subject based on time-series data of the subject position change amount and time-series data of the eyeball angle change amount. The eyeball movement measurement device according to claim 1.

4. The eyeball movement determination unit: Neural network processing that takes as input time-series data of the subject position change amount and time-series data of the eyeball angle change amount; Rule-based processing that classifies eyeball movement calculated from the difference between the time-series data of the eyeball angle change amount and the time-series data of the subject position change amount according to a preset rule; classifying the type of the eye movement by any one of the processes; The eye movement measuring apparatus according to claim 3.

5. The three-dimensional position of the subject includes at least a pitch angle and a yaw angle representing the orientation of the face of the subject in the face image. The eye movement measuring apparatus according to claim 1.

6. causing a computer to calculate a subject position change amount, which is a change amount of the three-dimensional position of the subject, from a face image of the subject; calculate an eye angle change amount indicating a change amount of the angle of the eye of the subject from an eye image obtained by imaging the eye of the subject; calculate a change amount of the angle of the eyeball of the subject, which is a change amount of the angle of the eyeball of the subject, from the eye image, and correct the calculated change amount of the angle of the eyeball based on the change amount of the angle of the eye; determine the eye movement of the subject based on the subject position change amount and the corrected change amount of the angle of the eyeball; An eye movement measurement program for causing the above to be executed.

7. causing the computer to at least control an imaging direction of a camera that images the eye image of the subject; further execute the above, causing the computer to in the process of determining the eye movement of the subject, further determine the eye movement of the subject based on the control information of the camera; The eye movement measurement program according to claim 6.

8. causing the computer to in the process of determining the eye movement of the subject, classify the type of the eye movement of the subject based on time-series data of the subject position change amount and time-series data of the change amount of the angle of the eyeball; The eye movement measurement program according to claim 6.

9. A computer calculates a subject position change amount, which is a change amount of the three-dimensional position of the subject, from a face image of the subject, calculates an eye angle change amount indicating a change amount of the angle of the eyes of the subject from an eye image obtained by imaging the eyes of the subject, calculates a eyeball angle change amount, which is a change amount of the angle of the eyeballs of the subject, from the eye image, and corrects the calculated eyeball angle change amount based on the eye angle change amount, and determines the eye movement of the subject based on the subject position change amount and the corrected eyeball angle change amount. An eye movement measurement method.

10. The computer further controls at least the imaging direction of a camera that images the eye image of the subject, and when determining the eye movement of the subject, further determines the eye movement of the subject based on the control information of the camera. The eye movement measurement method according to claim 9. The eye movement measurement method according to claim 9. The eye movement measurement method according to claim 9.

Images