Eye movement measuring device, eye movement measuring method and program

Abstract

To provide an eyeball movement measurement device which can perform detection in which a position of a focus point on an eyeball is associated with an actual position as an absolute position while maintaining detection accuracy of the little displacement of the focus point on the eyeball in the eyeball movement by a gradient method.SOLUTION: An eyeball movement measurement device for detecting the positional change of a focus point of an eyeball in a moving image of the eye imaged including the eyeball comprises: a displacement detection unit which detects the displacement of the focus point in the eyeball movement; a position detection unit which acquires the position of the focus point of the eyeball as a measurement position with a prescribed feature point in the eyeball for each frame of a moving image; and a position integration unit which integrates the displacement detected by the displacement detection unit with the measurement position detected by the position detection unit to obtain the position of the focus point of the eyeball.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an eye movement measuring device, an eye movement measuring method, and a program for detecting the eye movement of a subject. [Background technology]

[0002] It is generally known that humans observe the object space by constantly moving their eyes. That is, the human eye acquires visual information by repeating fixation, in which the gaze is fixed on the object being observed, and saccades, in which the gaze is moved quickly (about 100 degrees (deg) / 500 degrees / second) for a short period of time (about 20ms (milliseconds) to 60ms).

[0003] However, even in a fixation state, the eyeball actually makes so-called fixational microsaccades (hereinafter referred to as microsaccades). This microsaccade is a type of fixational eye movement, and is a minute jump movement with an amplitude of less than 1 degree in terms of the angle of rotation of the eyeball and a duration of tens of milliseconds or less (see, for example, Non-Patent Documents 1 and 2).

[0004] By detecting microsaccades using a highly accurate measuring device that can measure minute short-term displacements of patterns on the eyeball, the state of visual attention can be estimated. Furthermore, when detecting an object that the gaze is directed at (when performing gaze detection), by accumulating the displacement in the eye movement described above, it is possible to estimate the pupil position with high resolution and track where the gaze is directed. Furthermore, as a device capable of detecting the above-mentioned fixational eye movement, there is a device that measures the movement of the eyeball by capturing an image of the blood vessel pattern in the eyeball (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-99716 [Non-patent literature]

[0006] [Non-Patent Document 1] Masanori Takahashi, Hirohisa Isogai, Judy L. Van Raalte, "An attempt to detect microsaccades from eye movements during anticipatory reactions - Using a tennis serve anticipatory reaction task -", Sports Industry Research, Vol. 28, No. 1 (2018), pp. 13-29 [Non-patent document 2] Kazutaka Suzuki, Haruyoshi Toyoda, Ryohei Hanayama, Masahiro Ishii, "Development of a simultaneous binocular fixation micromovement measurement device using an intelligent vision sensor and evaluation of the laterality of microsaccades," Journal of Biomedical Engineering, 53(5), 247-254, 2015 [Non-patent document 3] Hayato Kawakami, Yuta Sasada, Satoru Igarashi, Junichi Akita, "Development of an eye-gaze measurement camera capable of tracking saccades and the possibility of interaction using it," Transactions of Information Processing Society of Japan 56(4), 1174-1183, 2015 Summary of the Invention [Problem to be solved by the invention]

[0007] As described above, minute displacements in microsaccades can be detected to estimate a person's underlying state of attention. One method for detecting minute displacements over a short period of time is the gradient method, which measures the amount of movement of an object using the brightness gradient of pixels in the same image taken in succession and the amount of change in pixel brightness between two images. The gradient method allows for highly sensitive measurement of minute displacements of the object being imaged, and by accumulating these displacements, minute changes in the pupil position used for gaze detection can be measured with high resolution. However, as the period during which gaze detection is performed by accumulating minute displacements becomes longer, errors (bias) in the detection of displacements accumulate, and the absolute position of the pupil may deviate from the actual position.

[0008] Furthermore, in the case of Non-Patent Document 3, the pupil position is determined for each captured image by image processing of the captured image, and therefore can always be determined as an absolute position. However, the resolution of pupil position detection is determined by the accuracy of detecting the position of the center of gravity of the pupil, and it is not possible to detect minute displacements as compared to motion detection using the gradient method, making it impossible to perform highly accurate gaze detection.

[0009] The present invention has been made in consideration of the above circumstances, and provides an eye movement measurement device, an eye movement measurement method, and a program that can detect the position of a point of interest on the eyeball in correspondence with its actual position as an absolute position, while maintaining the detection accuracy of minute displacements of the point of interest on the eyeball using the gradient method. [Means for solving the problem]

[0010] In order to solve the above-mentioned problems, the eye movement measuring device of the present invention is an eye movement measuring device that detects a change in the position of a point of interest of an eyeball in a moving image of the eye that is captured including the eyeball, and is characterized by comprising: a displacement detection unit that detects the displacement of the point of interest due to eyeball movement; a position detection unit that acquires the position of the point of interest of the eyeball as a measurement position using a predetermined feature point on the eyeball for each frame of the moving image; and a position integration unit that integrates the displacement detected by the displacement detection unit and the measurement position detected by the position detection unit to determine the position of the point of interest of the eyeball.

[0011] The eye movement measuring device of the present invention is characterized in that the displacement detection unit comprises a measurement range setting unit that sets an area of ​​the moving image that satisfies predetermined conditions as a measurement range, a brightness gradient calculation unit that calculates a brightness gradient in the measurement range, a brightness change amount calculation unit that calculates an amount of brightness change between frame images in the measurement range, and a movement distance calculation unit that calculates the position change using a gradient method based on each of the brightness change amount and the brightness gradient.

[0012] The eye movement measuring device of the present invention is characterized in that the position detection unit detects the measurement position of the point of interest of the eyeball, using the geometric feature point of the pupil of the eyeball as the point of interest.

[0013] The eye movement measuring device of the present invention is characterized in that the position integration unit calculates the position of the point of interest of the eyeball in accordance with the result of adding the displacement to the previous position, which is the position of the point of interest of the eyeball calculated immediately before, and the difference between the previous measurement position, which is the measurement position of the point of interest of the eyeball calculated immediately before, and the previous position.

[0014] The eye movement measurement method of the present invention is a method for detecting a change in the position of a point of interest of an eyeball in a moving image of an eye that is captured including the eyeball, and is characterized by including a displacement detection process in which a displacement detection unit detects a displacement of the point of interest due to eyeball movement, a position detection process in which a position detection unit obtains the position of the point of interest of the eyeball as a measurement position using a predetermined feature point on the eyeball for each frame of the moving image, and a position integration process in which a position integration unit integrates the displacement detected by the displacement detection unit and the measurement position detected by the position detection unit to determine the position of the point of interest of the eyeball.

[0015] The program of the present invention is a program that causes a computer to execute the operation of an eye movement measuring device that detects changes in the position of a focus point of an eyeball in a moving image of the eye that is captured including the eyeball, and causes the computer to function as a displacement detection means that detects displacement of the focus point due to eyeball movement, a position detection means that obtains the position of the focus point of the eyeball as a measurement position using a predetermined feature point on the eyeball for each frame of the moving image, and a position integration means that integrates the displacement detected by the displacement detection means with the measurement position detected by the position detection means to determine the position of the focus point of the eyeball. [Effects of the Invention]

[0016] As described above, according to the present invention, it is possible to provide an eye movement measurement device, an eye movement measurement method, and a program that can detect the position of a point of interest on an eyeball so that it corresponds to its actual position as an absolute position, while maintaining the detection accuracy of minute displacements of the point of interest on the eyeball during eye movement using the gradient method. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a measurement system using an eye movement measurement device according to an embodiment of the present invention. [Figure 2] FIG. 10 is a diagram showing a frame image and luminance values ​​of pixels in an imaging area in which each part of the eye in the frame image is imaged. [Figure 3] 5 is a conceptual diagram of an eye for explaining the process of setting a measurement range from an imaging region of a pupil 501 in a frame image. FIG. [Figure 4] 10A and 10B are diagrams illustrating calculation of a luminance gradient in an area where a candidate pixel exists. [Figure 5] FIG. 10 is a diagram illustrating an example of calculating a luminance gradient in an image frame. [Figure 6] 10A and 10B are conceptual diagrams illustrating a process for determining the amount of pupil movement by a gradient method in this embodiment. [Figure 7] 10 is a diagram showing the relationship between pixel positions and pixel luminance values ​​between frame images when the amount of movement Δx is large. FIG. [Figure 8] 10 is a conceptual diagram illustrating the derivation of the movement amount in the hopping movement of the pupil on the imaging surface. FIG. [Figure 9] 10A and 10B are diagrams illustrating the linearity of the luminance gradient in the boundary region between the pupil and the iris of the eyeball in the frame image. [Figure 10] 2(b) is a diagram showing the luminance gradient in the boundary region of each of the pupil 501 and the iris 502 in FIG. [Figure 11] FIG. 7 is a diagram showing a frame image pattern obtained by two-dimensionally expanding the pattern shown in FIG. 6(a). [Figure 12] FIG. 10 is a conceptual diagram illustrating a process of generating simultaneous linear equations from a plurality of measurement pixels. [Figure 13] 10 is a flowchart illustrating an example of the operation of a measurement process by a displacement detection unit 12 in the eye movement measurement device of this embodiment. [Figure 14] 10A and 10B are diagrams illustrating a process for determining the center coordinates of an image area of ​​a pupil in a captured image. [Figure 15] 10 is a diagram showing the luminance values ​​of the intersection points where a line segment 200 intersects with a contour 460 or the contour of an image region of a Purkinje image. [Figure 16] 10 is a diagram showing coordinate values ​​of the center coordinate xcn in the X coordinate at the intersection of each contour 460 of the line segment 200. FIG. [Figure 17] 10 is a flowchart illustrating an example of an operation of detecting the position of the pupil by the position detection unit 13 in the eye movement measurement device of this embodiment. [Figure 18] 10 is a diagram showing coordinate values ​​of the pupil in the X coordinate that changes at predetermined time intervals, detected by the displacement detection unit 12 and the position detection unit 13 of this embodiment. FIG. [Figure 19] 10 is a diagram showing coordinate values ​​of the pupil in the X coordinate that changes every predetermined time, which are calculated by the position integration unit 14 of this embodiment. FIG. [Figure 20] 10 is a flowchart illustrating an example of an operation of detecting the position of the pupil by the position integration unit 14 in the eye movement measurement device of this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. 1 is a conceptual diagram showing an example of the configuration of a measurement system using the eye movement measurement device of this embodiment. In Fig. 1, the measurement system 1 includes an eye movement measurement device 10, an infrared irradiation device 20, an imaging lens 30, an infrared filter 40, and an image sensor 50. The eye movement measurement device 10 detects a change in the position of the pupil as a point of interest in the eye (that is, a point of interest on the eye) from a moving image obtained by capturing input human eye movement.

[0019] The infrared irradiation device 20 includes an LED (Light Emitting Diode) that emits infrared light to illuminate the eye area including the eyeball of the subject. In this embodiment, the LED uses a device that emits invisible infrared light (infrared light) with a wavelength of, for example, 770 nm. If visible light is used for illumination, the subject may react to the light, i.e., reduce the amount of light entering the pupil, causing the iris to close, resulting in a change in pupil size and a change in pupil position to avoid the light. Alternatively, the subject may be drawn to the light, making it impossible to measure eye movement occurring in a natural state corresponding to the object being observed, which is used to estimate the subject's psychological state when observing the object. For this reason, in this embodiment, infrared light with an invisible wavelength is used to illuminate the eye, in order to measure eye movement without being affected by the irradiated light.

[0020] The imaging lens 30 is a small imaging lens for capturing an image of the eyeball from a short distance, and forms an image of reflected light (reflected infrared light) from the subject eyeball on the imaging surface of the image sensor 50, which will be described later.

[0021] The infrared filter 40 is an optical filter that transmits infrared light of invisible wavelengths and blocks light of visible wavelengths. In this embodiment, an optical filter with a cutoff wavelength of 700 nm, for example, is used.

[0022] The image sensor 50 is a device that converts the brightness of light irradiating an imaging surface into an electrical signal and outputs it as image data, and is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The image sensor 50 also outputs frame images, which are images captured at a predetermined frame period (the frame period in this embodiment will be described in detail later), as a continuous moving image to the eye movement measurement device 10. Here, each of the frame images in the moving image is captured in multiple gradations.

[0023] The eye movement measuring device 10 includes a data input / output unit 11, a displacement detection unit 12, a position detection unit 13, a position integration unit 14, a display unit 15, and a storage unit 16. The data input / output unit 11 writes the captured image supplied from the image sensor 50 into the storage unit 16 for storage. In addition, the data input / output unit 11 outputs the pupil movement amount (the movement amount in the rotation of the eyeball, which is the x-axis movement amount Δx and the y-axis movement amount Δy between frames in successive frame periods described later) measured by the eyeball movement measurement device 10 and the position of the eyeball (for example, the pupil in the eyeball) to an external device.

[0024] The displacement detection unit 12 measures the movement amount of the eyeball's focus point using a gradient method that measures the movement amount of the pupil by using the brightness gradient and brightness change amount near the boundary between the iris and pupil in the captured image (frame of a moving image) (details will be described later). The displacement detection unit 12 also includes a measurement range setting unit 121, a brightness gradient calculation unit 122, a brightness change amount calculation unit 123, and a movement distance calculation unit .

[0025] The position detection unit 13 performs image processing to detect the edge of the boundary between the pupil and the iris in the eyeball, and detects the position of the pupil (coordinates (xt, yt) of the center of the pupil) as the measurement position (details will be described later). The position detection unit 13 includes an edge detection unit 131, an X coordinate detection unit 132, a Y coordinate detection unit 133, and a position calculation unit .

[0026] The position integration unit 14 integrates the movement amount of the pupil of the eyeball at a predetermined time detected by the displacement detection unit 12 (i.e., the movement amount of the point of interest, movement amount dt(dxt,dyt) described later) with the measured position of the eyeball detected by the position detection unit 13 ((xt,yt) described later), and outputs the integration result as the position of the pupil (position of the pupil's center coordinates (Ct(Cxt,Cyt)) (details will be described later). The display unit 15 is, for example, a liquid crystal display, and displays the eyeball positions and the like determined by the position integration unit 14. The storage unit 16 stores the captured images described above as well as data used by the displacement detection unit 12, the position detection unit 13, and the position integration unit 14.

[0027] <Detection of pupil movement amount by displacement detection unit 12> The measurement range setting unit 121 sets the measurement range as an image region for extracting measurement pixels used to calculate the movement amount of the pupil in a frame image that is a captured image. In this embodiment, a method is used to measure the movement amount of the pupil at a point of interest using a gradient method that measures the movement amount using a luminance gradient and a luminance change amount. Therefore, in order to improve the accuracy of the measured movement amount by using an area with a large luminance gradient to measure a large amount of luminance change even if the movement amount is small, the boundary area of ​​the pupil and iris, which has a higher luminance gradient than other parts of the eyeball, is used as the measurement target, as will be described later.

[0028] 2A and 2B are diagrams showing frame images and luminance values ​​of pixels in image areas in which various parts of the eye are captured in the frame images. Fig. 2A shows a frame image including image areas of a pupil 501, an iris 502, and a bulbar conjunctiva 503 in an eyeball 500. Furthermore, boundary region 505 indicates an image region near the boundary between pupil 501 and iris 502. Reflection region 506 is an image region where a reflected image of the irradiated infrared light source is captured, and has a higher brightness value than the surrounding image region.

[0029] Figure 2(b) is a graph showing the brightness values ​​of each pixel on the line A-A' in the frame image of Figure 2(a). In Figure 2(b), the horizontal axis shows the x-coordinate of the pixel (pixel position), and the vertical axis shows the brightness value of each pixel. A dotted line B1 indicates the luminance value of the pixel in the imaging region of the bulbar conjunctiva 503. A dotted line B2 indicates the luminance values ​​of pixels in the imaging area of ​​the boundary area 505 between the pupil 501 and the iris 502.

[0030] Each of the dotted lines B3 and B4 indicates the luminance value of a pixel in the imaging area of ​​the reflective area 506. A dotted line B5 indicates the luminance value of the pixel in the imaging region of the boundary between the iris 502 and the bulbar conjunctiva 503. In this embodiment, the boundary region between the pupil and the iris of the eyeball has a larger brightness gradient than other regions of the eyeball, so the gradient method is used to calculate the pupil movement amount. For this reason, the image capture region of the pupil is extracted from the frame image in which the eyeball is captured.

[0031] Here, when detecting the imaging region in which the pupil 501 is imaged, the measurement range setting unit 121 compares the luminance value of each pixel in the frame image with the pupil detection threshold value. Then, the measurement range setting unit 121 extracts a group of pixels having a luminance value less than the pupil detection threshold (dotted line 10 in FIG. 2(b)) as the imaging area in which the pupil 501 is imaged. The imaging area of ​​the pupil 501 has the lowest luminance value in the frame image, and therefore can be easily distinguished from the imaging areas of other parts.

[0032] In this embodiment, the pupil detection threshold value is an intermediate value between the brightness value of the pixel where the pupil is imaged and the brightness value of the pixel where the iris is imaged. Furthermore, the pupil detection threshold may be determined by experimentally capturing images of a plurality of human eyes and detecting whether the pupil brightness value can be distinguished from other surrounding areas. In this embodiment, pixels near the boundary region 505 between the pupil 501 and the iris 502, which is the range with a large brightness gradient (the region indicated by the dotted line B2) shown in Figure 2(a), are used as measurement pixels (described later) used to measure (estimate) the movement amount of the pupil 501 using the gradient method.

[0033] FIG. 3 is a conceptual diagram of an eye for explaining the process of setting the measurement range from the imaging area of ​​the pupil 501 in the frame image. After extracting the imaging area of ​​the pupil 501 described in Figure 2, i.e., the inner part of the boundary area 505, the measurement range setting unit 121 extracts the central pixel (center O) of the area of ​​the pupil 501, and sets the number of pixels from the central pixel to the boundary area 505 as the radius R of the area of ​​the pupil 501. Then, the measurement range setting unit 121 multiplies the radius R by a predetermined coefficient value p (p>1, for example, 1.5 in this embodiment) and sets the range of radius R′ as the measurement range (processing target region) 600.

[0034] That is, the measurement range setting unit 121 determines a measurement range 600 that always includes the boundary area 505 between the pupil 501 and the iris 502 as an area for extracting measurement pixels for measuring the movement amount of the pupil (i.e., the movement amount of the point of interest). This prevents the frame image of the subject's eye from including unnecessary images of parts other than the eyeball, such as glasses or eyebrows, which can cause errors, and determines the measurement range 600 as the area where the boundary area of ​​the pupil and iris is likely to be captured.

[0035] Returning to FIG. 1, the luminance gradient calculation unit 122 determines whether or not each pixel in the frame image is within the measurement range 600 described above. Then, the luminance gradient calculation unit 122 determines whether or not the luminance value of each pixel within the measurement range 600 is less than a preset luminance threshold value (first threshold value).

[0036] This allows unnecessary pixels in the frame image of the subject's eye that have high brightness values ​​due to reflections of light sources in the imaged part of the eyeball, which can cause errors in detecting the amount of movement, to be excluded from the candidate pixels to be extracted as measurement pixels. Then, the luminance gradient calculation unit 122 determines pixels within the measurement range 600 whose luminance values ​​are less than the above luminance value as candidate pixels, which are pixels that are candidates for measurement pixels. Next, the luminance gradient calculation unit 122 calculates the luminance gradient at each of the extracted candidate pixels using each of eight adjacent pixels adjacent to the measurement pixel for which the luminance gradient is to be calculated, as explained below with reference to FIG.

[0037] FIG. 4 is a diagram for explaining calculation of the luminance gradient in an area where a candidate pixel exists. Candidate pixel C i,jThe brightness gradient of the region 700 to which the pixel belongs is calculated by the brightness gradient calculation unit 122 as an x-axis direction brightness gradient sx and a y-axis direction brightness gradient sy. In the following description, pixel C i-1,j-1 From pixel C i+1,j+1 The brightness values ​​of each of the i-1,j-1 to luminance value LC i+1,j+1 where pixel C i,j The x-axis luminance gradient sx and the y-axis luminance gradient sy of the pixel C are calculated by dividing the x-axis luminance gradient sx and the y-axis luminance gradient sy by the eight adjacent pixels C i-1,j-1 , C i-1,j , C i-1,j+1 , C i,j-1 , C i,j+1 , C i+1,j-1 , C i+1,j , C i+1,j+1 (These pixels form a pixel region including the measurement pixel).

[0038] The luminance gradient calculation unit 122 calculates the x-axis direction luminance gradient sx using the following formula. sx=[(LC i-1,j-1 -LC i+1,j+1 ) / 2+(LC i,j-1 -LC i,j+1 ) / 2+(LC i+1,j-1 -LC i+1,j+1 ) / 2] / 3 That is, the x-axis direction luminance gradient sx is i-1,j-1 The brightness value of LC i-1,j-1 From pixel C i+1,j+1 The brightness value of LC i+1,j+1 The brightness gradient obtained by subtracting and dividing by two pixels and pixel C i,j-1 The brightness value of LC i,j-1 From pixel C i,j+1 The brightness value of LC i,j+1 The brightness gradient obtained by subtracting and dividing by two pixels and pixel C i+1,j-1 The brightness value of LC i+1,j-1 From pixel C i+1,j+1 The brightness value of LC i+1,j+1 The average value of the brightness gradient obtained by subtracting the value of the pixel value and dividing it by two pixels is calculated.

[0039] Similarly, the luminance gradient calculation unit 122 calculates the y-axis direction luminance gradient sy using the following formula. sy=[(LC i-1,j-1 -LC i+1,j-1 ) / 2+(LC i-1,j -LC i+1,j ) / 2+(LC i-1,j+1 -LC i+1,j+1 ) / 2] / 3 That is, the y-axis luminance gradient y is i-1,j-1 The brightness value of LC i-1,j-1 From pixel C i+1,j-1 The brightness value of LC i+1,j-1 The brightness gradient obtained by subtracting and dividing by two pixels and pixel C i-1,j The brightness value of LC i-1,j From pixel C i+1,j The brightness value of LC i+1,j The brightness gradient obtained by subtracting and dividing by two pixels and pixel C i-1,j+1 The brightness value of LC i-1,j+1 From pixel C i+1,j+1 The brightness value of LC i+1,j+1 The average value of the brightness gradient obtained by subtracting the value of the pixel value and dividing it by two pixels is calculated.

[0040] 5 is a diagram showing an example of calculation of the luminance gradient in an image frame, in which the horizontal axis indicates the coordinate position (pixel position) of a pixel, and the vertical axis indicates the luminance value of the pixel. Pixel position x n-1 The luminance gradient s n-1 is the pixel position x n-2 From the brightness value of the pixel at x n The brightness value of the pixel, 75, is subtracted and divided by the distance of two pixels to get 25 (=(125-75) / 2=50 / 2). Also, pixel position x n+1 The luminance gradient s n+1 is the pixel position x n From the brightness value of the pixel at x n+2 The brightness value of the pixel, 25, is subtracted and divided by the distance of two pixels to get 25 (=(75-25) / 2=50 / 2).

[0041] Returning to FIG. 1, the luminance gradient calculation unit 122 calculates a value obtained by dividing the absolute value of the luminance gradient S of the candidate pixel (i.e., the resultant vector of vector sx·X and vector sy·Y, where X is a unit vector in the x-axis direction and Y is a unit vector in the y-axis direction) by the square root of the luminance value I of the candidate pixel (=|S| / (I) 1 / 2 ) is determined to be equal to or greater than a preset gradient brightness value ratio threshold (second threshold). 2 +sy 2 ) 1 / 2 and the square root of the luminance value (I) 1 / 2 is the square root of the number of photons (N) 1 / 2 In addition, sx is the x-axis luminance gradient that indicates the amount of pupil movement in the x-axis direction, and sy is the y-axis luminance gradient that indicates the amount of pupil movement in the y-axis direction (described later). Then, the brightness gradient calculation unit 122 calculates |S| / (I) 1 / 2 A candidate pixel whose gradient luminance value ratio is equal to or greater than the threshold value is determined as a measurement pixel.

[0042] At this time, the brightness gradient calculation unit 122 may be configured to display on the display unit 15 a confirmation image in which the measurement pixel is marked in the frame image, indicating at which position the pixel becomes the measurement pixel for each of the consecutive frame images. The user may then observe the confirmation image on the display unit 15, adjust the gradient brightness value ratio threshold so that only pixels in the boundary region between the pupil and the iris are measurement pixels, and redo the extraction of measurement pixels, or may detect pixels extracted as measurement pixels outside the boundary region between the pupil and the iris, and perform processing to remove these pixels from the measurement pixels.

[0043] Fig. 6 is a conceptual diagram illustrating the process of calculating the amount of pupil movement using the gradient method in this embodiment. In Fig. 6, in order to simplify the explanation of the gradient method, calculation of the amount of movement in one dimension (only in the x-axis direction) will be explained. Fig. 6(a) shows the distribution of brightness values ​​in one dimension as a gradation. In Fig. 5(a), pattern 201 shows the n-th frame image, and pattern 202 shows the (n+1)-th frame image. Then, the movement amount Δx (the movement amount Δx in the x-axis direction) is calculated using the same measurement pixel 205 in the pattern 201 and the pattern 202. Here, the measurement pixel 205 is a measurement pixel at the same pixel position in each of the frame images of the pattern 201 and the pattern 202.

[0044] 6(b) is a graph illustrating how the shift amount Δx is calculated using the gradient method from the luminance gradient sx (x-axis luminance gradient sx) of the measurement pixel 205 and the luminance change amount ΔI. In FIG. 6, the horizontal axis represents the pixel coordinate position (pixel position), and the vertical axis represents the pixel luminance value. Curve 203 (solid line) represents the relationship between the pixels and luminance values ​​of pattern 201, and curve 204 (dotted line) represents the relationship between the pixels and luminance values ​​of pattern 202. Curve 204 is phase-shifted in the positive direction of the x-axis from curve 203 by the shift amount Δx. This shift amount Δx represents the amount of pupil movement between frame images in a frame period (the imaging period of the image sensor 50). Here, the brightness change amount ΔI is the brightness value I of the measurement pixel 205 of the pattern 201. n From the above, the luminance value I of the measurement pixel 205 of the pattern 201 is n+1 The difference in brightness (I n -I n+1 )

[0045] As the pupil moves (the eyeball rotates), the position of the pupil imaged at measurement pixel 205 of the frame image of pattern 202 moves by an amount Δx from the position of the pupil imaged at measurement pixel 205 of the frame image of pattern 201. In this way, when the pupil position changes, the difference in luminance value I at the position corresponding to the amount of movement Δx in the pupil region is measured as the amount of luminance change ΔI.

[0046] Therefore, the movement amount Δx can be geometrically calculated from the luminance gradient sx of the measurement pixel 205 and the luminance change amount ΔI of the measurement pixel 205 using the following formula from FIG. 6(b). 6(b), it can be seen that there is a relationship in the relational expression ΔI=-sxΔx. As a result, the movement amount Δx can be calculated by the following first equation, which is a modification of the above relational expression. Δx=-ΔI / sx … 1st equation In this first equation, in the x-axis direction, sx has a positive polarity when the gradient is one where the brightness increases in a positive direction, and a negative polarity when the gradient is one where the brightness increases in a negative direction. Also, the movement amount Δx has a positive polarity when moving in a positive direction, and a negative polarity when moving in a negative direction.

[0047] Furthermore, when the movement amount Δx is calculated from the above-mentioned first equation, it is assumed that the luminance gradient sx of the measurement pixel 205 is constant between the pattern 201 and the pattern 202 (between the frame images). That is, the luminance gradient sx at the measurement pixel 205 of the curve 203 of the pattern 201 and the luminance gradient sx' at the measurement pixel 205 of the curve 204 of the pattern 202 must be considered to be constant within an allowable error. Therefore, when the amount of pupil movement Δx between frame images becomes large, the luminance gradient sx of measurement pixel 205 of pattern 201 and the luminance gradient sx' of measurement pixel 205 of pattern 202 do not each have a constant value, and the above first formula cannot calculate the amount of movement Δx from the amount of luminance change ΔI using the frame images of pattern 201 and pattern 202.

[0048] FIG. 7 is a diagram showing the relationship between pixel positions and pixel luminance values ​​between frame images when the amount of movement Δx is large. Since the amount of movement Δx is large, the shift distance between the curve 203 of the pattern 201 and the curve 204 of the pattern 202 at the measurement pixel 205 is large. Therefore, the luminance gradient sx at the measurement pixel 205 of the curve 203 of the pattern 201 and the luminance gradient sx' at the measurement pixel 205 of the curve 204 of the pattern 202 change along the way and are not constant (non-linear). Therefore, when calculating the amount of movement Δx using the brightness value ΔI according to the first formula, it is necessary to select a frame period that results in the amount of movement Δx that results in the same brightness gradient sx.

[0049] Therefore, the image sensor 50 in this embodiment uses an image sensor with a short frame period and capable of high-speed imaging, and for example, has an imaging surface with 512 x 512 pixels in a grid pattern, and is capable of capturing moving images at 500 frames per second.

[0050] Furthermore, the maximum speed of a saccade during normal pupillary saccades (hereafter referred to as saccade speed) is 500 degrees (°) per second. Therefore, the maximum pupil rotation angle in 1 / 500 seconds is 1 degree. As the amount of pupil movement decreases, the maximum speed during saccades also decreases. On the other hand, the speed of a microsaccade (amplitude << 1 degree) is, for example, 100 degrees / second or less at most. Therefore, the pupil rotation angle in 1 / 500 seconds is a maximum of 0.2 degrees.

[0051] Here, as the movement amount of the image sensor 50 of the pupil, the maximum pixel movement amount on the imaging surface in one frame time (1 / 500 seconds) is found using FIG. 8 below. The calculation conditions for the maximum pixel movement amount in FIG. 8 are that the resolution of the captured image of the image sensor 50 is 128×128 pixels, and the size of the eyeball in the captured image is 50 pixels from the center of the pupil.

[0052] Fig. 8 is a conceptual diagram for explaining the derivation of the movement amount in the hopping movement of the pupil on the imaging plane. Fig. 8(a) shows a frame image in a moving image as a captured image, and has a resolution of 128 x 128 pixels. Fig. 8(a) also shows the imaging pixel range of the eyeball on the imaging plane of image sensor 50. Pupil imaging area EA is the area in captured image 50A where the pupil is imaged. A circular area having a radius r (=50 pixels) centered on the center O of the pupil imaging area EA indicates the eyeball imaging area IA, which is an imaging area in which the eyeball portion is imaged.

[0053] FIG. 8(b) is a diagram for explaining calculation of the pixel movement amount from the rotation angle of the rotation movement of the eyeball in a microsaccade. For example, when the rotation angle is 0.2 degrees, the pupil movement amount dx(Δx) is calculated as follows from the relationship dx / r=sin(0.2 deg)s=0.0035, assuming that the radius r of the pupil imaging area EA is 50 pixels. dx = sin(0.2deg) × 50 (pixels) = 0.17 (pixels) Furthermore, when the rotation angle is 1.0 degree, the movement amount dx(Δx) is calculated as follows from the relationship dx / r=sin(1.0 deg)s=0.017, where r is the radius of the pupil imaging area EA and r is 50 pixels. dx = sin(0.2deg) × 50 (pixels) = 0.87 (pixels)

[0054] FIG. 9 is a diagram illustrating the linearity of the luminance gradient in the boundary region between the pupil and the iris of the eyeball in the frame image. 9(a) shows a range Z (near the boundary region of the pupil and iris) in which the graph in FIG. 2(b) is enlarged. Range Z is a 30-pixel region from pixel position 54 to pixel position 84.

[0055] FIG. 9B is a graph showing an enlarged range Z of 30 pixels from pixel position 54 to pixel position 84 in FIG. 9A. Observing the luminance gradient at each pixel in Figure 9(b) reveals that the luminance gradient is constant within a range of 0.17 pixels where the movement amount Δx calculated in Figure 8 is, and that there is no difference in the luminance gradient within a range of 0.87 pixels. Therefore, it is possible to measure the movement amount of the pupil within a movement range where the luminance gradient falls within a similar pixel area (within a region of 9 adjacent pixels including the measurement pixel).

[0056] Therefore, as shown in Figure 8, capturing images at a frame rate of 500 frames / second results in a pupil range with a constant luminance gradient for measuring the movement amount Δx during microsaccades of 100 degrees / second or less. As a result, if the luminance gradient sx between frame images is equal to calculate the movement amount Δx, the movement amount in the x direction will also be equal, suppressing errors due to nonlinearity and allowing the movement amount Δx to be measured with a predetermined accuracy using Equation 1.

[0057] In this embodiment, as described above, the amount of movement of the pupil in the x-axis direction (and the y-axis direction, which will be described later) is calculated using the gradient method from the amount of change in pixel luminance ΔI and the luminance gradient sx in the pixel region where the boundary region between the pupil and the iris is imaged. However, as already mentioned, the change in pupil position accompanying eye movement in microsaccades is suppressed to a minute Δx, so the decrease in accuracy of detecting the amount of movement due to the influence of measurement noise has a significant effect.

[0058] Generally, shot noise occurs due to statistical fluctuations in the number of photons (actually the amount of charge corresponding to the number of photons) accumulated in each pixel on the imaging surface of an image sensor. The shot noise level is the square root of the number of photons (N), where N is the number of accumulated photons. 1 / 2 is proportional to.

[0059] In addition, the level of the luminance signal is proportional to the number of accumulated photons N, and the level of the shot noise is (N) as mentioned above. 1 / 2 Since it is proportional to, the signal-to-noise ratio (SN ratio) is (N) 1 / 2 (=N / (N) 1 / 2 ) will be proportional to Therefore, in measuring the luminance, as the number of accumulated photons N increases, the signal-to-noise ratio improves, and the effect of shot noise in measuring the luminance is reduced.

[0060] On the other hand, since the brightness gradient and brightness change amount are each proportional to the difference value of the number of accumulated photons N, the accuracy does not improve as the number of accumulated photons N increases, as in the case of the brightness signal. Conversely, as the number of accumulated photons N increases, the shot noise (N) included in the difference value 1 / 2 increases in proportion to Therefore, when the luminance gradient is the same, the accuracy of measuring the luminance gradient and the amount of change in luminance is improved when pixels with smaller luminance values ​​are used. Therefore, as already described, the brightness gradient calculation unit 122 calculates |S| / (I) 1 / 2 Candidate pixels whose gradient intensity value ratio is greater than or equal to the threshold value are used as measurement pixels. Here, the square root of the intensity value (I) 1 / 2 is the square root of the number of photons (N) 1 / 2 is proportional to.

[0061] FIG. 10 is a diagram showing the luminance gradient in the boundary region of each of the pupil 501 and the iris 502 in FIG. 2(b). In FIG. 10, the luminance difference ΔI between frame image N and frame image N+1, which is one frame period after frame image N, when the pupil moves by Δx is calculated. When selecting pixels as measurement pixels, if the pixels have the same luminance gradient, the luminance difference ΔI when moved will be the same for all pixels.

[0062] However, the pixel position 601 has a photon count of 200, and the noise amount is 14.1 (=(200) 1 / 2 ) The pixel position 602 has a photon count of 100 and a noise amount of 10 (=(100) 1 / 2 ) Furthermore, the pixel position 603 has a photon count of 20, and the noise amount is 4.7 (=(20) 1 / 2 ) Therefore, the movement amount Δx calculated at pixel position 601 602 is the movement amount Δx calculated at pixel position 603 603 Compared to the conventional method, the noise level is 3 (= 14.1 / 4.7) times higher, resulting in a decrease in measurement accuracy.

[0063] In this embodiment, measurement pixels capturing an image of the boundary region between the pupil and the iris, which is a location where the luminance gradient is higher than other regions, are used to measure the amount of movement Δx (the same applies to the amount of movement Δy in the y-axis direction, which will be described later). As already mentioned, the boundary region between the pupil and the iris has a large luminance gradient, and even a small amount of movement Δx results in a large amount of luminance change ΔI, making it suitable for detecting the amount of movement Δx.

[0064] In order to reduce the influence of the above-mentioned shot noise, when the same brightness gradient is present, a measurement pixel with a lower brightness value (a smaller number of accumulated photons N) is used. Therefore, in this embodiment, as already described, the luminance gradient calculation unit 122 calculates |S| / (I) of each candidate pixel. 1 / 2 is equal to or greater than a preset threshold value for the gradient luminance value ratio, and candidate pixels that are equal to or greater than the threshold value for the gradient luminance value ratio are extracted as measurement pixels.

[0065] In this embodiment, in order to obtain the movement amount of the pupil in the eyeball in two dimensions, the movement amount Δx parallel to the x-axis is defined as the movement amount Δx in the x-axis direction, the movement amount Δy parallel to the y-axis is defined as the movement amount in the y-axis direction, the luminance gradient parallel to the x-axis direction is defined as the luminance gradient in the x-axis direction, and the luminance gradient parallel to the y-axis direction is defined as the luminance gradient in the y-axis direction, and the first one-dimensional equation already explained is expanded to the second linear equation shown below. ΔI=(-s x )·Δx+(-s y )·Δy … Second formula

[0066] In this second equation, the x-axis luminance gradient s x The polarity of is positive when the gradient of brightness increases in the positive direction along the x-axis, and negative when the gradient of brightness increases in the negative direction. Also, the polarity of the x-axis direction movement amount Δx is positive when moving in the positive direction along the x-axis, and negative when moving in the negative direction.

[0067] And in the second equation, the luminance gradient s yis positive when the luminance increases in the positive direction along the y-axis, and the luminance gradient s y The polarity of is negative when the gradient is one where the brightness increases in the negative direction. The polarity of the y-axis direction shift amount Δy is positive when moving in the positive direction along the y-axis, and negative when moving in the negative direction.

[0068] In the second equation, the values ​​obtained by measurement are the luminance gradients sx and sy and the luminance change amount ΔI, so the x-axis direction movement amount Δx and the y-axis direction movement amount Δy are both unknowns. Therefore, the x-axis direction movement amount Δx and the y-axis direction movement amount Δy are calculated by solving the following simultaneous linear equations generated by the above-mentioned second equation. ΔI1=(-s x1 )·Δx+(-s y1 ) Δy ΔI2=(-s x2 )·Δx+(-s y2 ) Δy

[0069] FIG. 11 is a diagram showing a frame image pattern obtained by two-dimensionally expanding the pattern shown in FIG. 6(a). In Figure 11, a predetermined area 801 of the eyeball is imaged, a frame image (801) of the area 801 is obtained, and one frame period later, an area 802 of the eyeball that has moved from the above area 801 by an amount of movement Δx in the x-axis direction and an amount of movement Δy in the y-axis direction is imaged, and a frame image (802) of the area 802 is obtained.

[0070] In the measurement pixel 811, the x-axis direction luminance gradient sx is sx1, the y-axis direction luminance gradient sy is sy1, and the luminance change amount between the luminance value of the region 801 and the luminance value of the region 802 is ΔI1. In addition, in the measurement pixel 812, the x-axis direction luminance gradient sx is sx2, the y-axis direction luminance gradient sy is sy2, and the luminance change amount between the luminance value of the region 801 and the luminance value of the region 802 is ΔI2.

[0071] Furthermore, when extracting each of the measurement pixels 811 and 812 from the candidate pixels, the luminance gradient calculation unit 122 calculates |S1| / (I1) 1 / 2 =(sx1 2 +sy1 2 ) 1 / 2 / (I1) 1 / 2 and |S1| / (I2) 1 / 2 =(sx2 2 +sy2 2 ) 1 / 2 / (I2) 1 / 2 and are determined to be equal to or greater than the gradient brightness value ratio threshold. Then, the movement distance calculation unit 124 generates the simultaneous linear equations described above using the measurement values ​​of the measurement pixel 811 and the measurement pixel 812 (described later).

[0072] The brightness change amount calculation unit 123 calculates the difference in brightness value of a specified measurement pixel between frame images by subtracting the brightness value of the specified measurement pixel in a frame image one period later from the brightness value of the specified measurement pixel in the frame image. The movement distance calculation unit 124 calculates the x-axis direction movement amount Δx and the y-axis direction movement amount Δy by solving the simultaneous linear equations using the x-axis direction brightness gradient sx, the y-axis direction brightness gradient sy, and the brightness change amount ΔI, which are the measurement values ​​of the measurement pixel 811 and the measurement pixel 812.

[0073] Furthermore, as described above, the simultaneous linear equations may be generated and solved using two measurement pixels. However, in order to further improve accuracy, in this embodiment, the simultaneous linear equations are generated from multiple measurement pixels as shown below. Fig. 12 is a conceptual diagram illustrating the process of generating simultaneous linear equations from a plurality of measurement pixels. Fig. 12 shows a case where six measurement pixels p1 to p6 (1≦n≦6) are selected as measurement pixels for calculating a shift vector as an example of n measurement pixels.

[0074] Here, the brightness gradient calculation unit 122 calculates the brightness gradient sx in the x-axis direction. n Absolute value of |sx n|(first absolute value) and the absolute value of the y-axis direction brightness gradient |sy n Compare with |(second absolute value). Then, the brightness gradient calculation unit 122 calculates |sx n |>|sy n The measured pixels (measured pixels p2, p4, and p6 in FIG. 12) of | are grouped as a group of advantageous points (first group), and each n1 ,sy n1 Let's say. Also, each sx of the measurement pixels in the x dominant point group n1 ,sy n1 Add up each of them (also called accumulation) and get Σsx n1 ,Σsy n1 Ask for. Here, in the x-dominant point group, if the gradient in the x-axis direction is negative, the sign is inverted to align the sign of the brightness gradient when accumulating. In Figure 12, measurement pixel p4 in the x-dominant point group is the measurement pixel for which polarity is inverted. ΔIn1=(-)(-sx n1 )Δx+(-)(-sy n1 )Δy =sx n1 Δx+sy n1 Δy

[0075] Then, the luminance variation calculation unit 123 calculates ΣΔIn1 by accumulating the luminance variation ΔIn1 of each measurement pixel in the x dominant point group. As a result, the movement distance calculation unit 124 generates the following first linear equation as one of the linear equations based on the x dominant point group. ΣΔIn1=Σ(-sx n1 )Δx + Σ(-sy n1 )Δy

[0076] On the other hand, the brightness gradient calculation unit 122 calculates |sx n |≦|sy n The measured pixels (measured pixels p1, p3, and p5 in FIG. 12) are grouped as a y dominant point group (second group), and n2 ,sy n2 Let's say. Here, if the gradient in the y-axis direction in the y-dominant point group is negative, the sign is inverted to align the sign of the brightness gradient when performing integration. In Figure 12, measurement pixel p1 in the y-dominant point group is the measurement pixel for which polarity is inverted. ΔIn2=(-)(-sx n2 )Δx+(-)(-sy n2 )Δy =sx n2 Δx+sy n2 Δy

[0077] Furthermore, the luminance variation calculation unit 123 calculates ΣΔIn2 by accumulating the luminance variation ΔIn2 of each measurement pixel in the y dominant point group. As a result, the following second linear equation is generated as the other linear equation based on the y dominant point group. ΣΔIn2=Σ(-sx n2 )Δx + Σ(-sy n2 )Δy

[0078] Then, the movement distance calculation unit 124 solves the simultaneous linear equations consisting of the following first and second linear equations to obtain the amount of movement Δx in the x-axis direction and the amount of movement Δy in the y-axis direction. ΣΔIn1=Σ(-sx n1 )Δx + Σ(-sy n1 )Δy ΣΔIn2=Σ(-sx n2 )Δx + Σ(-sy n2 )Δy Furthermore, the data input / output unit 11 outputs the x-axis direction movement amount Δx and the y-axis direction movement amount Δy calculated by the movement distance calculation unit 124 to an external device. Furthermore, the data input / output unit 11 sequentially displays the x-axis direction movement amount Δx and the y-axis direction movement amount Δy calculated by the movement distance calculation unit 124 on the display unit 15, and writes and stores them in the memory unit 16.

[0079] According to the configuration of the displacement detection unit 12 described above, the eye of the subject is captured as a moving image, and the amount of pupil movement is calculated using the gradient method based on the amount of change in luminance ΔI of the measurement pixel between consecutive frame images of the moving image, and each of the x-axis luminance gradient and the y-axis luminance gradient. Therefore, it is not necessary to determine the absolute position of the pupil with high precision; it is sufficient to be able to detect eye movement indicating a microsaccade. Therefore, the device can be made smaller than conventional devices, and since there is no need to fix the subject's head, it can be made into a device that can be worn on the subject's head. Microsaccades in the subject's everyday state, that is, when the subject is moving and moving their head in everyday life, can be measured with high precision by determining the amount of pupil movement.

[0080] Furthermore, according to the configuration of the displacement detection unit 12, in order to detect the movement of the eyeball, the boundary area between the pupil and the iris in the frame image is used as the area with a large brightness gradient, so that even with a small movement, a large amount of brightness change can be detected, and the amount of pupil movement can be obtained with high accuracy.

[0081] Furthermore, according to the configuration of the displacement detection unit 12, the luminance gradient (S) and the square root of the luminance value (I 1 / 2 ) is equal to or greater than a preset gradient brightness value ratio threshold, the pixels used for measurement are those for which the ratio is equal to or greater than a preset gradient brightness value ratio threshold. This allows the amount of pupil movement to be calculated using measurement pixels (i.e., pixels with low brightness values ​​for the same brightness gradient, or so-called dark pixels) that are less affected by shot noise, which increases in proportion to the square root of the number of photons (brightness value), thereby improving measurement accuracy.

[0082] Furthermore, since the configuration of the displacement detection unit 12 measures the amount of pupil movement using a gradient method that uses a luminance gradient and a luminance change amount, as long as the image capture device has a sufficiently high frame rate, it is possible to obtain high accuracy in measuring the amount of pupil movement of the detected eyeball even if the resolution itself is not that high. Furthermore, even if there is no clear feature, as long as there is a detectable luminance gradient, it is possible to measure the amount of pupil movement of the eyeball.

[0083] FIG. 13 is a flowchart illustrating an example of the operation of the measurement process by the displacement detection unit 12 in the eye movement measurement device of this embodiment. Step S101: When a frame image is supplied from the image sensor 50, which is an external device, the data input / output unit 11 writes the frame image data into the storage unit 16 for storage. The measurement range setting unit 121 erases (clears data) the integrated values ​​Σsx1 and Σsx2 of the x-axis direction luminance gradient sx, the integrated values ​​Σsy1 and Σsy2 of the y-axis direction luminance gradient sy, and the integrated values ​​ΣΔI1 and ΣΔI2 of the luminance change amount ΔI, all of which are calculated from the immediately previous frame image stored in the memory unit 16.

[0084] Step S102: The measurement range setting unit 121 reads the luminance value data of each pixel of the frame image from the storage unit 16 .

[0085] Step S103: Then, the measurement range setting unit 121 determines whether the luminance value of each pixel in the read frame image is less than the pupil detection threshold value. At this time, the measurement range setting unit 121 determines pixel values ​​whose luminance values ​​are less than the pupil detection threshold as pixels in the pupil region where the pupil is captured, while the measurement range setting unit 121 determines pixel values ​​whose luminance values ​​are equal to or greater than the pupil detection threshold as pixels in the region where a part other than the pupil is captured.

[0086] The measurement range setting unit 121 extracts an image area of ​​the pixels in the pupil area and a collection of pixels surrounded by the pixels in the pupil area, and sets this image area as the area where the pupil is imaged (pupil imaged area). Then, the measurement range setting unit 121 sets the distance from the center O of the measurement range 600 to the pixel on the periphery of the measurement range 600 as the radius R. The measurement range setting unit 121 calculates a radius R′ by multiplying the radius R by a predetermined coefficient value p, and sets the range from the center O to the radius R′ as the measurement range 600.

[0087] Step S104: The luminance gradient calculation unit 122 refers to the storage unit 16, selects a pixel in the frame image, and reads the pixel position of that pixel. Here, the luminance gradient calculation unit 122 selects pixels in the order of columns, for example, starting from the top row in the reference frame image, while sequentially switching the selected row.

[0088] Step S105: The luminance gradient calculation unit 122 determines whether the pixel position of the selected pixel is included in the measurement range 600 having a radius R' from the center O. At this time, if the pixel position of the selected pixel is included in the measurement range 600 having a radius R' from the center O, the luminance gradient calculation unit 122 advances the process to step S106. On the other hand, if the pixel position of the selected pixel is not included in the measurement range 600 having the radius R' from the center O, the luminance gradient calculation unit 122 advances the process to step S112.

[0089] Step S106: The luminance gradient calculation unit 122 refers to the storage unit 16 and reads the luminance value of the pixel selected in the frame image. Then, the luminance gradient calculation unit 122 determines whether or not the pixel value of the read pixel is less than the luminance threshold value (first threshold value).

[0090] At this time, if the pixel value of the read pixel is less than the brightness threshold value (first threshold value), the brightness gradient calculation unit 122 extracts this pixel as a candidate pixel, and proceeds to step S107. On the other hand, if the pixel value of the read pixel is not less than the luminance threshold value (first threshold value), the luminance gradient calculation unit 122 advances the process to step S112.

[0091] Step S107: The luminance gradient calculation unit 122 calculates each of the x-axis direction luminance gradient sx and the y-axis direction luminance gradient sy of the candidate pixel from the eight adjacent pixels by the process described with reference to FIG. Then, the brightness gradient calculation unit 122 calculates (sx2 +sy 2 ) 1 / 2 The brightness gradient S is calculated by the following calculation.

[0092] Step S108: The luminance gradient calculation unit 122 calculates |S| / (I) from the calculated luminance gradient S and luminance value I. 1 / 2 Calculate the calculated |S| / (I) 1 / 2 is equal to or greater than the gradient luminance ratio threshold. At this time, the luminance gradient calculation unit 122 calculates |S| / (I) 1 / 2 If is equal to or greater than the gradient luminance ratio threshold, the candidate pixel is extracted as a measurement pixel, and the process proceeds to step S109. On the other hand, the luminance gradient calculation unit 122 calculates |S| / (I) 1 / 2 If is less than the gradient luminance ratio threshold, the process proceeds to step S112.

[0093] Step S109: The luminance gradient calculation unit 122 calculates the luminance gradient sx in the x-axis direction of each measurement pixel. n Absolute value of |sx n | and the y-axis luminance gradient sy n Absolute value of |sy n Compare with |. Then, the brightness gradient calculation unit 122 calculates |sx n |>|sy n The measured pixels of | are grouped as the x dominant point group (first group), and |sx n |≦|sy n The measured pixels of | are grouped as a y dominant point group (second group). Then, when the measurement pixel is a measurement pixel of the x-significant point group, the luminance gradient calculation unit 122 calculates the x-axis direction luminance gradient sx of the measurement pixel of the x-significant point group. n1 and y-axis luminance gradient sy n1 The integrated value Σsx is calculated by integrating each of the above. n1 ,Σsy n1 is calculated and written and stored in the storage unit 16. On the other hand, when the measurement pixel is a measurement pixel of the y-dominant point group, the luminance gradient calculation unit 122 calculates the x-axis luminance gradient sx of the measurement pixel of the y-dominant point group.n2 and y-axis luminance gradient sy n2 The integrated value Σsx is calculated by integrating each of the above. n2 ,Σsy n2 is calculated and written and stored in the storage unit 16.

[0094] Step S110: The luminance change amount calculation unit 123 refers to the frame image data in the storage unit 16, and subtracts the luminance value of the corresponding measurement pixel in the frame image from the luminance value of the measurement pixel in the frame image immediately before that measurement pixel to calculate the luminance change amount ΔI.

[0095] Step S111: When the measurement pixel is a measurement pixel of the x dominant point group, the brightness change amount calculation unit 123 accumulates the brightness change amount ΔI1 (=ΔI) of the measurement pixel of the x dominant point group to calculate the accumulated value ΣΔI1 and writes and stores it in the memory unit 16. On the other hand, when the measurement pixel is a measurement pixel of the y dominant point group, the brightness change amount calculation unit 123 accumulates the brightness change amount ΔI2 (=ΔI) of the measurement pixel of the y dominant point group to calculate the accumulated value ΣΔI2 and writes and stores it in the memory unit 16.

[0096] Step S112: The luminance gradient calculation unit 122 determines whether or not processing has been completed for all pixels in the frame image. Here, the luminance gradient calculation unit 122 makes the determination based on, for example, whether or not the selected pixel is the pixel position of the end point in the frame image. At this time, if the processing for all pixels in the frame image has been completed, the luminance gradient calculation unit 122 advances the processing to step S113. On the other hand, if the processing has not been completed for all pixels in the frame image, the luminance gradient calculation unit 122 advances the processing to step S104.

[0097] Step S113: The movement distance calculation unit 124 calculates the integrated value Σsx n1 , integrated value Σsy n1 and the integrated value ΣΔI1 and the integrated value Σsyn2 , integrated value Σsx n2 The integrated value ΣΔI2 is read out and the following simultaneous linear equations are created. ΣΔIn1=Σ(-sx n1 )Δx + Σ(-sy n1 )Δy ΣΔIn2=Σ(-sx n2 )Δx + Σ(-sy n2 )Δy Then, the movement distance calculation unit 124 solves the simultaneous linear equations, calculates the x-axis direction movement amount Δx and the y-axis direction movement amount Δy as the movement amount of the pupil, and ends the process. At this time, the movement distance calculation unit 124 displays the calculated x-axis direction movement amount Δx and y-axis direction movement amount Δy on the display screen of the display unit 15. Furthermore, the data input / output unit 11 may output each of the calculated x-axis direction movement amount Δx and y-axis direction movement amount Δy.

[0098] <Detection of measurement position of pupil of eyeball by position detection unit 13> The edge detection unit 131 performs predetermined image processing (general edge detection) on the captured image to detect the boundary between the pupil and the iris (the outline of the pupil, which is a characteristic point of the eyeball) as the edge of the pupil image area. Here, the edge detection unit 131 performs, for example, edge detection filter (differential filter or the like) processing on the captured image to detect the edge of the image region of the pupil.

[0099] The X coordinate detection unit 132 detects the X coordinate of a predetermined position on the edge of the image area of ​​the pupil detected by the edge detection unit 131 . The Y coordinate detection unit 133 detects the Y coordinate of a predetermined position on the edge of the pupil image area detected by the edge detection unit 131 . Fig. 14 is a diagram illustrating the process of determining the center coordinates of the image area of ​​the pupil in a captured image. Fig. 14(a) is an example of a captured image of an eyeball. Fig. 14(b) shows the edge of the image area of ​​the pupil obtained by edge processing the captured image of Fig. 14(a). In Fig. 14(b), 400 indicates the image area of ​​the pupil, 600 indicates the image area of ​​the iris, and contour 460 indicates the contour of the image area of ​​the pupil.

[0100] The X-coordinate detection unit 132 arranges a plurality of line segments parallel to the X-axis at predetermined intervals in the Y-axis direction, for example, line segments 200_1, 200_2, 200_3, 200_4, 200_5, 200_6, .... When the line segments 200_1, 200_2, 200_3, 200_4, 200_5, 200_6, ... are collectively referred to as line segment 200. Then, the X-coordinate detection unit 132 detects the X-coordinate at which each of the line segments 200_1, 200_2, 200_3, 200_4, 200_5, 200_6, . . . intersects with the contour 460.

[0101] For example, the line segment 200_4 intersects with the contour 460 at each of the intersections of the coordinate point P0 (X coordinate x0) and the coordinate point P1 (X coordinate x1). The X-coordinate detection unit 132 writes and stores pairs of X-coordinates of the intersections of the line segments 200_1, 200_2, 200_3, 200_4, 200_5, 200_6, ... with the contour 460 in the storage unit 16. For example, the pair of X-coordinates of the intersections of the line segment 200_4 and the contour 460 is (x0, x1).

[0102] Similarly, the Y coordinate detection unit 133 arranges a plurality of line segments parallel to the Y axis, for example, line segments 300_1, 300_2, 300_3, 300_4, 300_5, 300_6, . . . at predetermined intervals in the X axis direction. Then, the Y coordinate detection unit 133 detects the Y coordinate at which each of the line segments 300_1, 300_2, 300_3, 300_4, 300_5, 300_6, . . . intersects with the contour 460.

[0103] For example, the line segment 300_6 intersects with the contour 460 at each of the intersections of the coordinate point P4 (X coordinate y0) and the coordinate point P5 (Y coordinate y1). The Y coordinate detection unit 133 writes and stores pairs of Y coordinates of the intersections of the line segments 300_1, 300_2, 300_3, 300_4, 300_5, 300_6, ... with the contour 460 in the storage unit 16. For example, the pair of Y coordinates of the intersections of the line segment 300_6 and the contour 460 is (y0, y1).

[0104] 14(b), there are image regions 701 and 702 of the Purkinje image in which a light source is reflected on the eyeball. The image regions 701 and 702 of the Purkinje image have higher luminance values ​​than the image region 400 of the pupil. Therefore, not only the contour 460 as the boundary between the image region 400 of the pupil and the image region 470 of the iris, but also the contours as the boundary between the image region 400 of the pupil and the image regions 701 and 702 of the Purkinje image are detected. For example, the line segment 200_6 intersects not only with the contour 460 but also with the contour of the image region 702 of the Purkinje image. Here, the intersection point of the line segment 200_6 and the contour 460 is P2(x2), and the intersection point of the line segment 200_6 and the contour of the image region 702 of the Purkinje image is P3(x3).

[0105] FIG. 15 is a diagram showing the luminance values ​​of the intersection points where the line segment 200 intersects with the contour 460 or the contour of the image region of the Purkinje image. Fig. 15(a) is a graph showing the luminance values ​​at the X coordinates of the intersections P0(x0) and P1(x1) where the line segment 200_4 intersects with the contour 460. In Fig. 15(a), the horizontal axis represents the X coordinate (pixel position), and the vertical axis represents the luminance value of the pixel. In FIG. 15(a), coordinates x0 and x1 on the X coordinate are the coordinates of the intersection between the outline 460 of the pupil image area 400 and the line segment 200_4, and therefore it can be seen that the brightness value has changed compared to the iris image area 470.

[0106] 14(b), the midpoint of the coordinates in the pair of X coordinates (xn, xm) of the intersection between line segment 200 and contour 460 becomes coordinate xc in the central coordinates (sometimes referred to as measurement coordinates) (xc, yc) of pupil image region 400. For example, position calculation unit 134 calculates xc1=(x0+x1) / 2 from the pair of X coordinates (x0, x1) of the intersection between line segment 200_4 and contour 460, and determines this as the central coordinate between the intersections of line segment 200_4, i.e., central coordinate xc1 of the X coordinates. Then, in this embodiment, the position calculation unit 134 averages each of the center coordinates xcn of the X coordinates obtained from each pair (xn, xm) corresponding to each of the line segments 200 stored in the storage unit 16, and sets the average as the final center coordinate xc.

[0107] Fig. 15(b) is a graph showing the luminance values ​​at the X coordinates of the intersections P2 (x2) and P3 (x3) where the line segment 200_6 intersects with the contour 460. In Fig. 15(b), similar to Fig. 15(a), the horizontal axis represents the X coordinate (pixel position) and the vertical axis represents the luminance value of the pixel. In FIG. 15(b), coordinate x2 on the X coordinate is the coordinate of point P2 where contour 460 of image region 400 of the pupil intersects with line segment 200_6, and coordinate x3 is the coordinate of point P3 where contour 460 of image region 702 of the Purkinje image intersects with line segment 200_6. Therefore, it can be seen that the luminance values ​​are changed compared to image region 470 of the iris and image region 702 of the Purkinje image.

[0108] However, in the case of the pair of X coordinates (x2, x3) of the line segment 200_6, the coordinate x3 is the coordinate of the intersection P3 between the contour of the image region 702 of the Purkinje image and the line segment 200_6. Therefore, the center coordinate xcn calculated by the position calculation unit 134 by calculating xc2=(x2+x3) / 2 is not the center coordinate of the X coordinates of the center coordinates (xc, yc) of the image region 400 of the pupil, but is the midpoint of the distance between the contour 460 of the line segment 200_6 and the contour of the image region 702 of the Purkinje image. Therefore, the distance between the coordinates x2 and x3 in FIG. 15(b) is shorter than the distance between the coordinates x0 and x1 in FIG. 14(a), and the center coordinate xc2 is obtained as a smaller value than the center coordinate xc0.

[0109] FIG. 16 is a diagram showing coordinate values ​​of the center coordinates xcn in the X coordinate system at the intersection of each contour 460 of the line segment 200. In FIG. 16, the horizontal axis indicates the number of each line segment 200 (for example, "5" for line segment 200_5), and the vertical axis indicates the coordinate value of the center coordinate xcn obtained from the coordinate pair (xn, xm) of line segment 200 (for example, the number of pixels from the left end in the X coordinate of the captured image). The center coordinates xcn of the line segment 200 on the horizontal axis from numbers "12" to "19" are calculated to be smaller than those of the other numbers. Each of the line segments 200 numbered "12" to "19" is estimated to be an intersection where one of the X coordinates xn intersects with the contour of the image area 702 of the Purkinje image.

[0110] Therefore, the center coordinate xc in the X coordinates calculated using the X coordinate pairs (xn, xm) obtained from each of the line segments 200 numbered "12" to "19" is not the center coordinate xcn in the image region 500 of the pupil. Therefore, when calculating the central coordinate xc of the X coordinates, the position calculation unit 134 needs to exclude each of the pairs of X coordinates (xn, xm) corresponding to the line segments 200 numbered "12" to "19" when calculating the central coordinate xc of the X coordinates at the central coordinates (xc, yc).

[0111] When calculating the central coordinate xc of the X coordinate, the position calculation unit 134 excludes the data of the central coordinate xcn included in the top 1 / 4 group and the data of the central coordinate xcn included in the bottom 1 / 4 group in the data group of the central coordinate xn of the pair of X coordinates (xn, xm) corresponding to each line segment 200. Then, the position calculation unit 134 obtains the average value of each of the center coordinates xcn in the remaining half of the data group as the center coordinate of the X coordinate in the center coordinates (xc, yc) of the image region 400 of the pupil.

[0112] In addition, the position calculation unit 134 also obtains the Y-coordinate center coordinate yc of the center coordinate (xc, yc) of the pupil image area 400, using the pairs of intersections (yn, ym) between each of the line segments 300 and the contour 460, in the same manner as in the process of obtaining the X-coordinate center coordinate xc described above. As a result, the position calculation unit 134 calculates the center coordinates (xc, yc) of the image area 400 of the pupil based on the pupil outline 460, which is a characteristic point of the eyeball, from the captured image.

[0113] 17 is a flowchart illustrating an example of the operation of detecting the position of the pupil by the position detection unit 13 in the eye movement measurement device of this embodiment. The position detection unit 13 performs the following process for each frame of a moving image to detect the center coordinates of the image area of ​​the pupil in the frame. Step S201: The edge detection unit 131 extracts the boundary between the image area 400 of the iris and the image area 500 of the pupil in the frame of the moving image (captured image) by performing a predetermined edge detection process.

[0114] Step S202: The X-coordinate detection unit 132 arranges line segments 200 parallel to the X-axis at predetermined intervals in the Y-axis direction as a plurality of scanning lines. Then, the X coordinate detection unit 132 detects the intersection points where each of the line segments 200 intersects with the contour 460, and obtains the X coordinates xn and xm.

[0115] Step S203: For each line segment 200, the X-coordinate detection unit 132 writes and stores the two X-coordinates xn and xm of the intersection where the line segment 200 intersects with the contour 460 as a pair of intersections (xn, xm) in the memory unit 16 in correspondence with the number of the line segment 200.

[0116] Step S204: The Y coordinate detection unit 133 arranges line segments 300 parallel to the Y axis at predetermined intervals in the X axis direction as a plurality of scanning lines. Then, the Y coordinate detection unit 133 detects the intersection points where each of the line segments 300 intersects with the contour 460, and obtains the Y coordinates yn and ym.

[0117] Step S205: For each line segment 300, the Y coordinate detection unit 133 writes and stores the two Y coordinates yn and ym of the intersection point where the line segment 300 intersects with the contour 460 as a pair of intersection points (yn, ym) in the memory unit 16, corresponding to the number of the line segment 300.

[0118] Step S206: The position calculation unit 134 reads each pair of intersections (xn, xm) from the storage unit 16, and obtains the center coordinate xcn from each pair by calculating xcn=(xn+xm) / 2. Then, the position calculation unit 134 arranges (sorts) the acquired data group of center coordinates xcn in ascending order, excludes the data of center coordinates xcn that fall in the top 1 / 4 group and the data of center coordinates xcn that fall in the bottom 1 / 4 group, calculates the average value of the data of center coordinates xcn in the middle of the above data group, and sets the calculation result as the center coordinate xc in the X coordinate of the pupil image area 400.

[0119] Similarly, the position calculation unit 134 reads each pair of intersections (yn, ym) from the storage unit 16, and obtains the center coordinate ycn from each pair by calculating ycn=(yn+ym) / 2. Then, the position calculation unit 134 arranges the acquired data group of center coordinates ycn in ascending order, excludes the data of center coordinates ycn that fall in the top 1 / 4 group and the data of center coordinates ycn that fall in the bottom 1 / 4 group, calculates the average value of the data of the center coordinate ycn in the middle of the above data group, and sets the calculation result as the center coordinate yc in the Y coordinate of the pupil image area 400. The position calculation unit 134 sets the center coordinates xc and yc as the measurement position xt of the image area of ​​the pupil in the two-dimensional coordinate system.

[0120] Furthermore, in this embodiment, the movement of the eyeball is detected by extracting the outline of the pupil, but the movement of the eyeball may be detected by extracting the outline of the iris. Alternatively, a configuration may be adopted in which feature points of the eyeball (feature points within the pupil of the eyeball) are detected from the captured image by a SIFT (Scale-Invariant Feature Transform) method, and the positions of the feature points are detected as measurement positions.

[0121] In this embodiment, the center coordinates (xc, yc) of the pupil image area 400 are calculated from the intersections of the line segments 200 and 300 and the contour 460. However, the method of detecting the pupil position may not be the above-mentioned method of obtaining the pupil position from the intersection of the outline of the pupil image area and the scanning line, but may be a pupil center detection method that detects the pupil center position using a histogram, a method of detecting the center of gravity of a pupil image, a method of detecting the pupil outline using a Hough transform, or a method of tracking eye movement using a particle filter (see Kodama Hiroshi, Bulletin of Chiba Polytechnic Junior College, No. 15 (October 2010), "Development of an Eye-Gaze ​​Input Control System"), in which feature points of the eye (such as the outline of the pupil or the outline of the iris) are extracted, and the position of the point of interest of the eye indicated by the feature points is used as the measurement position (xt, yt) to detect the movement position Ct.

[0122] <Detection of pupil position by position integration unit 14> The position integration unit 14 calculates the position coordinates of the pupil by integrating the amount of movement (displacement) of the pupil calculated at each predetermined time by the displacement detection unit 12 and the measured position of the pupil (the center position of the pupil in this embodiment) calculated at each predetermined time by the position detection unit 13. Here, the predetermined time is, for example, every time between frames of a moving image in this embodiment. That is, the position integration unit 14 calculates the position coordinate of the pupil in the X coordinate at a predetermined time using the following third equation.

[0123] Cxt+1=Cxt+dxt+(xt-Cxt)×f…3rd formula In the third equation, Cxt is the X-coordinate position coordinate of the pupil (center of the pupil) at time t, and Cxt+1 is the X-coordinate position coordinate of the pupil at time t+1. dxt is the amount of movement (displacement) in the x-axis direction of the pupil in the X-coordinate determined by the displacement detection unit 12 from time t to time t+1. xt is the measured position of the X-coordinate determined by the position detection unit 13 at time t. f is a predetermined coefficient (0≦f≦1).

[0124] When calculating the position coordinates Cxt+1 of the pupil at time t+1 (position coordinates of the center of the pupil), the position integration unit 14 calculates the movement amount dxt as the displacement of the pupil from the position coordinates Cxt between time t and time t+1, relative to the position coordinates Cxt calculated at the immediately preceding time t. The position integration unit 14 then adds the movement amount dxt to the position coordinate Cxt to obtain the position coordinate C'xt+1 (=Cxt+dxt) as the integral result of the displacement. If only this calculation is performed, an error in the movement amount dxt will accumulate, and the position coordinate C'xt+1 will deviate from the actual pupil position.

[0125] For this reason, a correction term of "(xt-Cxt)×f" is added to Equation 3. "(xt-Cxt)" in this correction term is the difference α (=xt-Cxt) indicating the deviation of the position coordinate Cxt at the time t immediately preceding time t+1 from the measured position xt. The coefficient f indicates the degree (proportion) to which the difference α, which is an element of positional displacement, is reflected in the position coordinate C′xt+1 calculated in accordance with the pupil displacement (movement amount dxt).

[0126] Here, the closer (smaller) the coefficient f is to "0", the more the movement amount dxt is reflected, and the closer the position coordinate Cxt is to the position coordinate C'xt+1 obtained by integrating the movement amount dxt. On the other hand, as the coefficient f approaches "1" (is larger), the difference α (=xt-Cxt) from the measurement position xt is reflected more, and the position coordinate Cxt approaches the measurement position xt.

[0127] FIG. 18 is a diagram showing coordinate values ​​of the pupil in the X coordinate that changes at predetermined time intervals, detected by the displacement detection unit 12 and the position detection unit 13 of this embodiment. FIG. 18(a) shows the position coordinates Cxt of the pupil for each unit time (that is, frame of a moving image) that are measured using a predetermined measurement method and then corrected. In FIG. 18(a), the horizontal axis indicates time, and the vertical axis indicates coordinate values ​​in the X coordinate (coordinate values ​​in a coordinate system in which the center in the X coordinate direction of the captured image is set to coordinate value "0").

[0128] FIG. 18(b) shows the position coordinates Cxt obtained by integrating the pupil movement amount dxt obtained by the displacement detection unit 12 using the gradient method. In FIG. 18(b), the horizontal axis indicates time, and the vertical axis indicates coordinate values ​​in the X coordinate (coordinate values ​​in a coordinate system in which the center of the captured image in the X coordinate direction is set to coordinate value "0"). The curve of the position coordinate in FIG. 18(b) has the same shape as the curve of the position coordinate in FIG. 18(a), but it can be seen that as time passes, there is a bias of accumulated error in the + direction of the X coordinate.

[0129] FIG. 18(c) shows the measurement position xt of the pupil obtained by tracking the edge extracted by the displacement detection unit 12. In FIG. 18(c), the horizontal axis indicates time, and the vertical axis indicates coordinate values ​​in the X coordinate (coordinate values ​​in a coordinate system in which the center in the X coordinate direction of the captured image is set to coordinate value "0"). The curve of the position coordinates in FIG. 18(c) has the same shape as the curve of the position coordinates in FIG. 18(a), but it can be seen that noise based on the accuracy of detecting the position of the pupil feature points is superimposed on the true value.

[0130] FIG. 19 is a diagram showing the coordinate values ​​of the pupil in the X coordinate that changes every predetermined time, calculated by the position integration unit 14 of this embodiment. Fig. 19(a) shows the position coordinate Cxt calculated by equation 3 with coefficient f set to "0.1." In Fig. 19(a), the horizontal axis represents time, and the vertical axis represents coordinate values ​​in the X coordinate (coordinate values ​​in a coordinate system in which the center in the X coordinate direction of the captured image is set to coordinate value "0"). The position coordinate curve in Figure 19(a) has a similar shape to the position coordinate curve in Figure 18(a), and it can be seen that the value of the coefficient f well reflects the difference α in the position coordinate C'xt+1 in the third equation to an extent that there is no bias.

[0131] Fig. 19(b) shows the position coordinate Cxt calculated by equation 3 with coefficient f set to "0.5." In Fig. 19(b), the horizontal axis represents time, and the vertical axis represents coordinate values ​​in the X coordinate (coordinate values ​​in a coordinate system in which the center in the X coordinate direction of the captured image is set to coordinate value "0"). The position coordinate curve in Figure 19(b) has a similar shape to the position coordinate curve in Figure 18(a), but since the value of the coefficient f reflects the difference α to a high degree, it can be seen that noise in the curve of the measurement position xt is superimposed on the position coordinate Cxt, as in the case of Figure 18(c).

[0132] Furthermore, although the acquisition of the position coordinate Cxt of the center of the pupil image area 400 in the X coordinate has been described, the position coordinate Cyt of the center of the pupil image area 400 in the Y coordinate can also be acquired by similar processing. To obtain the position coordinate Cyt of the center of the image area 400 in the Y coordinate, the following fourth equation, which is obtained by modifying the third equation so as to correspond to the Y coordinate, is used.

[0133] Cyt+1=Cyt+dyt+(yt-Cyt)×f …4th formula In the above equation (3), Cyt is the Y-coordinate position coordinate of the pupil (center of the pupil) at time t, and Cyt+1 is the Y-coordinate position coordinate of the pupil at time t+1. dyt is the y-axis movement amount Δy of the movement amount (displacement) of the pupil in the Y-coordinate from time t to time t+1, determined by the displacement detection unit 12. yt is the measured position of the Y-coordinate determined by the position detection unit 13 at time t. As in equation (3), f is a predetermined coefficient (0≦f≦1).

[0134] As described above, the position integration unit 14 integrates the pupil movement amounts dxt and dyt and the measurement positions xt and yt using the coefficient f to obtain the position Ct(Cxt, Cyt) as the center coordinates of the pupil. That is, when the position integration unit 14 calculates the position coordinate Ct+1 by adding the amount of movement dt (dxt, dyt) from time t to time t+1 calculated by the displacement detection unit 12 to the position coordinate Ct calculated at the immediately preceding time t, the position integration unit 14 corrects the difference α between the center coordinate (xt, yt) measured by the position detection unit 13 at the immediately preceding time t and the position coordinate Ct at the immediately preceding time t by multiplying it by a coefficient f.

[0135] FIG. 20 is a flowchart illustrating an example of the operation of detecting the position of the pupil by the position integration unit 14 in the eye movement measurement device of this embodiment. Step S301: The position integration unit 14 initializes the position coordinate Ct, that is, initializes each of the position coordinates Cxt and Cyt. At this time, the position integration unit 14 reads from the storage unit 16 the position coordinates xt, yt of the pupil detected by the position detection unit 13 in the captured image at time t=0, for example, the first frame (first frame) in a moving image. Then, the position integration unit 14 sets the read center coordinates xt and yt as the initial values ​​of the position coordinates Cxt and Cyt, respectively.

[0136] Step S302: The position integrating unit 14 initializes the current time t to "0". Furthermore, the position integration unit 14 reads out the coefficient f that is written in advance in the storage unit 16, and sets it as the coefficient to be multiplied by the difference α. The position integration unit 14 writes and stores the initialized pupil position coordinates Ct in the storage unit 16 in association with the time t.

[0137] Step S303: The position integration unit 14 reads from the storage unit 16 the pupil position coordinates Ct obtained at the immediately preceding time t.

[0138] Step S304: The position integration unit 14 reads from the storage unit 16 the pupil movement amount dt(dxt, dyt) between frames from time t to time t+1, which is determined by the displacement detection unit 12.

[0139] Step S305: The position integration unit 14 calculates the position coordinate Cxt+1 using the position coordinate Cxt, the movement amount dxt, the measurement position xt, and the coefficient α using the third equation. That is, the position integration unit 14 adds the movement amount dxt to the position coordinate Cxt, and then adds the result of subtracting the position coordinate Cxt from the measurement position xt and multiplying the result by the coefficient α to the addition result to obtain the position coordinate Cxt+1.

[0140] Step S306: The position integration unit 14 calculates the position coordinate Cyt+1 using the position coordinate Cyt, the movement amount dyt, the measurement position yt, and the coefficient α using the fourth equation. That is, the position integration unit 14 adds the movement amount dyt to the position coordinate Cyt, and then adds the result of subtracting the position coordinate Cyt from the measurement position yt and multiplying the result by the coefficient α to the addition result to obtain the position coordinate Cyt+1.

[0141] Step S307: Then, the position integration unit 14 writes and stores the obtained position coordinates Ct+1 (Cxt+1, Cyt+1) in the storage unit 16 in association with time t+1 (that is, frame number t+1). Furthermore, the position integration unit 14 sets time t+1 as time t and proceeds to step S308. (As a result, in the flow from the next step S302 to step S306, the position integration unit 14 performs a process of calculating position coordinates Ct at time t+2, which is the time next to time t+1.

[0142] Step S308: The position integration unit 14 refers to the storage unit 16 to check whether there is a movement amount dt corresponding to the next frame, that is, determines whether the process of calculating the position coordinates Ct for all frames of the moving image has been completed. At this time, if there is a movement amount dt corresponding to the next frame, the position integration unit 14 advances the process to step S303, whereas if there is no movement amount dt corresponding to the next frame, the position integration unit 14 ends the process.

[0143] As described above, in this embodiment, when the displacement detection unit 12 adds the movement amount dt (dxt, dyt) from time t to time t+1 to calculate the position coordinate Ct calculated at the immediately preceding time t to calculate the position coordinate Ct+1, the difference α between the measurement position (xt, yt) measured by the position detection unit 13 at the immediately preceding time t and the position coordinate Ct at the immediately preceding time t is multiplied by the coefficient f and corrected at a predetermined rate. With this configuration, according to this embodiment, when the amount of movement indicating a highly accurate displacement is integrated to determine the center coordinate of the pupil as the position of the pupil, the highly accurate amount of movement at each time obtained by the gradient method is integrated with the measured position as an absolute position detected by edge detection, and position correction is performed using the difference between the measured position (xt, yt) at the immediately preceding time and the center coordinate at the immediately preceding time. This makes it possible to suppress the accumulation of errors in the amount of movement, and by combining the advantages of the gradient method and edge detection, it is possible to measure the position of the pupil (i.e., the movement of the eyeball) with high accuracy.

[0144] In addition, in this embodiment, the position integration unit 14 uses the third and fourth equations to determine the pupil position coordinates Ct (Cxt, Cyt) by complementarily using the movement amount dt (dxt, dyt) from time t to time t+1 determined by the displacement detection unit 12 and the measurement position (xt, yt) determined by edge detection by the position detection unit 13. Here, since the displacement dt is calculated at regular time intervals, it can also be said that the pupil movement speed is calculated. That is, in this embodiment, the position coordinates Ct (Cxt, Cyt) are calculated as the primary measurement value by combining the moving speed and the measurement position.

[0145] However, it is also possible to apply a Kalman filter algorithm that corrects errors contained in angular velocity by correcting the attitude angle obtained by integrating the angular velocity detected by the gyro sensor using the tilt angle obtained by the acceleration sensor and the absolute direction from the magnetic direction sensor (see, for example, Murakami Tomoya and Yokota Takayoshi, "Research on Sensing 3D Attitude Information Using a Smartphone and a Kalman Filter," Proceedings of the Forum on Information Science and Technology, August 20, 2013, Vol. 12, No. 1, pp. 179-180), thereby performing processing to correct the accumulated error caused by the integration of the movement amount dt (dxt, dyt) in this embodiment using the measurement position (xt, yt).

[0146] A program for implementing the functions of the eye movement measurement device 10 of the present invention may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed to perform the measurement process for detecting the pupil position. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a website provision environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0147] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system. [Explanation of symbols]

[0148] 1. Measurement system 10...Eye movement measuring device 11...Data input / output section 12...Displacement detection unit 13...Position detection unit 14...Position integration section 15...Display section 16...Storage section 20...Infrared irradiation device 30...imaging lens 40...Infrared filter 50...Image sensor 121...Measurement range setting section 122...Brightness gradient calculation unit 123...Luminance change amount calculation unit 124...Movement distance calculation unit 131...Edge detection unit 132...X coordinate detection unit 133...Y coordinate detection unit 134...Position calculation unit 400,470...Image area 460...Contour 500...Eyeball 501...pupil 502...Iris 600...Measurement range

Claims

1. An eye movement measuring device that detects a change in position of a focus point of an eye in a moving image of the eye that includes the eye, a displacement detection unit that detects a displacement of the point of interest due to eye movement; a position detection unit that acquires, for each frame of the moving image, a position of the point of interest of the eyeball as a measurement position using a predetermined feature point of the eyeball; a position integration unit that integrates the displacement detected by the displacement detection unit and the measurement position detected by the position detection unit to determine the position of the point of interest of the eyeball; An eye movement measuring device comprising:

2. The displacement detection unit a measurement range setting unit that sets an area of ​​the moving image that satisfies a predetermined condition as a measurement range; a brightness gradient calculation unit that calculates a brightness gradient in the measurement range; a brightness change amount calculation unit that calculates a brightness change amount between frame images in the measurement range; a movement distance calculation unit that calculates the position change using a gradient method based on the brightness change amount and the brightness gradient; 2. The eye movement measuring device according to claim 1, further comprising:

3. The position detection unit The geometric feature point of the pupil of the eyeball is used as the target point, and the measurement position of the target point of the eyeball is detected.

3. The eye movement measuring device according to claim 1 or 2.

4. The position integration unit The position of the point of interest of the eyeball is calculated based on a result of adding the displacement to a previous position, which is the position of the point of interest of the eyeball calculated immediately before, and a difference between a previous measured position, which is the measurement position of the point of interest of the eyeball calculated immediately before, and the previous position.

4. The eye movement measuring device according to claim 1, wherein the eye movement measuring device is a device for measuring an eye movement.

5. An eye movement measurement method for detecting a change in position of a focus point of an eye in a moving image of an eye that includes the eye, the method comprising: a displacement detection step in which a displacement detection unit detects a displacement of the point of interest due to eye movement; a position detection step in which a position detection unit acquires, for each frame of the moving image, a position of the point of interest of the eyeball as a measurement position using a predetermined feature point of the eyeball; a position integration step in which a position integration unit integrates the displacement detected by the displacement detection unit and the measured position detected by the position detection unit to determine the position of the point of interest of the eyeball; An eye movement measuring method comprising:

6. A program that causes a computer to execute an operation of an eye movement measuring device that detects a change in the position of a focus point of an eye in a moving image of the eye that includes the eye, The computer a displacement detection means for detecting a displacement of the point of interest due to eye movement; a position detection means for acquiring, for each frame of the moving image, the position of the point of interest of the eyeball as a measurement position using a predetermined feature point of the eyeball; a position integration means for integrating the displacement detected by the displacement detection means and the measured position detected by the position detection means to determine the position of the point of interest of the eyeball; A program to function as a

Images