Eye movement measuring device, eye movement measuring method and program

Abstract

To provide an eye movement measurement device wearable on a subject capable of measuring eye movement including microsaccades in the subject's daily condition, that is, in the state of activity or moving the head in daily life.SOLUTION: A disclosed eye movement measurement device is an eye movement measurement device for detecting position changes of focusing point in a captured moving image of the eye including the eyeball. The eye movement measurement device includes: a measurement range setting unit that sets an area that meets certain conditions for moving images as a measurement range; a luminance gradient calculation unit that calculates the brightness gradient in the measurement range; a luminance change amount calculation unit that finds the amount of change in brightness between frame images in the measurement range; and a movement distance calculation unit that finds the position change using the gradient method from the amount of change in brightness and the gradient of brightness.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an eye movement measuring device, an eye movement measuring method, and a program for detecting the eye movement of a subject. [Background technology]

[0002] It is generally known that humans observe the target space by constantly moving their eyes. That is, the human eye obtains visual information by repeating fixation, in which the gaze is fixed on the target, and saccades, in which the gaze is moved quickly (approximately 100 degrees (deg) / ~500 degrees / sec) for a short period of time (approximately 20ms (milliseconds) to 60ms).

[0003] However, even during fixation, the eyeballs actually make so-called fixational eye movements (hereinafter referred to as microsaccades). This microsaccade is a type of fixational eye movement, and is a minute jump movement with an amplitude of less than 1 degree in terms of the angle of rotation of the eyeball and a duration of tens of milliseconds or less.

[0004] There are many research reports suggesting that visual attention is strongly reflected in the control system of microsaccades. Many research reports have shown that the frequency of microsaccades increases with a shift in visual attention, whereas the frequency of microsaccades decreases with the concentration of visual attention.

[0005] Therefore, by analyzing the frequency of microsaccades (see, for example, Non-Patent Documents 1 and 2), it is possible to estimate a person's internal state, such as determining whether they are concentrating their visual attention on the object they are looking at, or whether they are simply looking absentmindedly (without concentrating their attention). [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Masanori Takahashi, Hirohisa Isogai, Judy L. Van Raalte, "An Attempt to Detect Microsaccades from Eye Movements in Anticipatory Reaction Situations - Using an Anticipatory Reaction Task for Tennis Serves -", ​​Sports Industry Research, Vol. 28, No. 1 (2018), pp. 13-29 [Non-Patent Document 2] Kazutaka Suzuki, Haruyoshi Toyoda, Ryohei Hanayama, Masahiro Ishii, "Development of a simultaneous binocular fixation micromovement measurement device using an intelligent vision sensor and evaluation of the left-right difference in microsaccades," Journal of Biomedical Engineering, 53(5), 247-254, 2015 Summary of the Invention [Problem to be solved by the invention]

[0007] As described above, by detecting microsaccades, it is possible to estimate the degree of a person's potential attention to an object to which the gaze is directed. However, when trying to detect microsaccades, it may be necessary to detect eye movements with an eye rotation angle of 0.1 degree or less and a duration of 0.01 seconds or less, and gaze measurement devices designed to detect general saccades are not able to measure such minute and rapid eye movements.

[0008] Therefore, when attempting to detect microsaccades, it is necessary to improve the measurement accuracy and measurement speed of the eye rotation angle. In conventional gaze measurement, the position of the pupil must be determined accurately to estimate the direction of gaze. However, when trying to improve measurement accuracy, the mechanical structure tends to become more complex and the measurement equipment tends to become larger and larger, assuming it is placed in a fixed location, in order to increase the accuracy. Furthermore, in order to improve the measurement accuracy, it is necessary to fix the subject's head so that the eye position does not fluctuate significantly.

[0009] Therefore, the gaze measurement devices in Non-Patent Document 1 and Non-Patent Document 2 measure microsaccades by fixing the subject's head in a large measuring instrument, and therefore cannot measure microsaccades when the subject is walking freely indoors and outdoors in their daily lives. For example, it is not possible to measure microsaccades in a situation where a subject is moving between shelves in a store and looking at products.

[0010] The present invention has been made in consideration of the above-mentioned circumstances, and provides an eye movement measuring device, an eye movement measuring method, and a program that can be worn by a subject and can measure eye movement, including microsaccades, in the subject's everyday state, i.e., when moving or moving the head in daily life. [Means for solving the problem]

[0011] In order to solve the above-mentioned problems, an eye movement measuring device of the present invention is an eye movement measuring device that detects a position change of a focus point of an eye in a moving image of an eye that is captured including the eye, and includes a measurement range setting unit that sets an area of ​​the moving image that satisfies a predetermined condition as a measurement range, a luminance gradient calculation unit that calculates a luminance gradient in the measurement range, a luminance change amount calculation unit that calculates an amount of luminance change between frame images in the measurement range, and a movement distance calculation unit that calculates the position change using a gradient method based on each of the luminance change amount and the luminance gradient. and, Equipped with and the measurement range setting unit selects, within the measurement range, pixels having a luminance value less than a first threshold for reducing an effect of shot noise in the moving image as the measurement range. It is characterized by: The eye movement measuring device of the present invention is characterized in that the measurement range setting unit sets an area including the boundary between the pupil and the iris of the eye as the measurement range.

[0012] The eye movement measuring device of the present invention is characterized in that the moving images are captured at a frame period in which the change in pupil position between the frame images is within a movement range that falls within the measurement range in which the luminance gradient has a constant gradient.

[0013] The eye movement measuring device of the present invention is characterized in that the measurement range setting unit compares the brightness value of each pixel in the frame image with a brightness threshold, extracts a pupil image area which is the image area of ​​the pupil, and sets an area obtained by expanding the pupil image area in an outer circumferential direction from the center point of the pupil image area as the measurement range.

[0015] The eye movement measuring device of the present invention is characterized in that the measurement range setting unit selects, as the measurement range, a group of pixels in an area within the measurement range where the brightness gradient is higher than other areas.

[0016] The eye movement measuring device of the present invention is characterized in that the measurement range setting unit selects pixels having lower luminance values ​​in the luminance gradient as pixels of the measurement range.

[0017] The eye movement measuring device of the present invention is characterized in that the measurement range setting unit selects pixels of the measurement range based on each of the luminance gradient and the luminance value of a pixel within the measurement range.

[0018] The eye movement measuring device of the present invention is characterized in that the measurement range setting unit selects pixels within the measurement range by comparing a numerical value obtained by dividing the brightness gradient by an amount of shot noise estimated from the brightness value of a pixel within the measurement range with a second threshold value.

[0019] The eye movement measuring device of the present invention is characterized in that within the measurement range, pixels whose value obtained by dividing the brightness gradient by the square root of the brightness value of the pixel is greater than a second threshold value are defined as pixels of the measurement range.

[0020] The eye movement measuring device of the present invention is characterized in that the measurement range setting unit obtains an x-axis luminance gradient as a luminance gradient in the x-axis direction and a y-axis luminance gradient as a luminance gradient in the y-axis direction corresponding to the measurement range, and determines an x-axis movement amount that is the movement amount of the pupil in the x-axis direction and a y-axis movement amount that is the movement amount in the y-axis direction by formulating simultaneous linear equations from the luminance change amount, the x-axis luminance gradient, and the y-axis luminance gradient.

[0021] In the eye movement measuring device of the present invention, the measurement range setting unit compares a first absolute value, which is an absolute value of the luminance gradient in the x-axis direction, of each pixel in the measurement range, with a second absolute value, which is an absolute value of the luminance gradient in the y-axis direction, and sets pixels in the measurement range in which the first absolute value is greater than the second absolute value as a first group, and sets pixels in the measurement range in which the second absolute value is greater than the first absolute value as a second group, and sets a numerical value obtained by adding up the luminance gradients in the x-axis direction of each pixel in the first group and multiplying the sum by an amount of movement in the x-axis direction, which is a variable, and a numerical value obtained by adding up the luminance gradients in the y-axis direction of each pixel and multiplying the sum by an amount of movement in the y-axis direction, which is a variable, one equation of the simultaneous linear equations is generated in which a result of adding up the x-axis luminance gradients of each of the pixels in the second group and multiplying the result by an amount of movement in the x-axis direction, which is a variable, and a result of adding up the y-axis luminance gradients of each of the pixels in the second group and multiplying the result by an amount of movement in the y-axis direction, which is a variable, are the integrated value of the amount of change in luminance in each of the measurement ranges, and the x-axis movement amount and the y-axis movement amount are each calculated by calculating the simultaneous linear equations.

[0022] The eye movement measuring device of the present invention is characterized in that it irradiates the eye with infrared light and captures the light reflected from the eye in multiple tones to obtain the moving image.

[0023] The eye movement measuring method of the present invention is a method for detecting a change in position of a focus point of an eye in a moving image of an eye that is captured including the eye, the measurement range setting unit setting an area of ​​the moving image that satisfies a predetermined condition as the measurement range. and selecting, within the measurement range, pixels having a luminance value less than a first threshold for reducing the effect of shot noise in the moving image as the measurement range.The method includes a measurement range setting step, a luminance gradient calculation step in which a luminance gradient calculation unit calculates a luminance gradient in the measurement range, a luminance change amount calculation step in which a luminance change amount calculation unit calculates an amount of luminance change between frame images in the measurement range, and a movement distance calculation step in which a gradient method is used to calculate the position change from each of the luminance change amount and the luminance gradient.

[0024] The program of the present invention is a program for causing a computer to execute the operation of an eye movement measuring device that detects a change in the position of a focus point of the eye in a moving image of the eye that is captured including the eye, and the program causes the computer to set an area of ​​the moving image that satisfies a predetermined condition as a measurement range. and selecting, within the measurement range, pixels having a luminance value less than a first threshold for reducing the effect of shot noise in the moving image as the measurement range. The program functions as a measurement range setting means, a brightness gradient calculation means for calculating a brightness gradient in the measurement range, a brightness change amount calculation means for calculating an amount of brightness change between frame images in the measurement range, and a movement distance calculation means for calculating the position change using a gradient method based on each of the amount of brightness change and the brightness gradient. Effect of the Invention

[0025] As described above, according to the present invention, it is possible to provide an eye movement measuring device, an eye movement measuring method, and a program that can be worn by a subject and can measure microsaccades of the eyeball when the subject is in an everyday state, i.e., when moving and moving their head as in daily life. [Brief description of the drawings]

[0026] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a measurement system using an eye movement measurement device of the present embodiment. [Diagram 2] 10 is a diagram showing a frame image and luminance values ​​of pixels in an imaging area in which each part of the eye is imaged in the frame image. FIG. [Diagram 3] 5 is a conceptual diagram of an eye for explaining the process of setting a measurement range from an imaging region of a pupil 501 in a frame image. FIG. [Figure 4]11A and 11B are diagrams illustrating calculation of a luminance gradient in an area where a candidate pixel exists; [Diagram 5] FIG. 13 is a diagram illustrating an example of calculation of a luminance gradient in an image frame. [Figure 6] 6A and 6B are conceptual diagrams illustrating a process of calculating the amount of pupil movement by a gradient method in the present embodiment. [Figure 7] 13 is a diagram showing the relationship between pixel positions and pixel luminance values ​​between frame images when the amount of movement Δx is large. FIG. [Figure 8] 1 is a conceptual diagram for explaining derivation of a movement amount in hopping movement of a pupil on an imaging surface. [Figure 9] 11 is a diagram illustrating the linearity of the luminance gradient in the boundary region between the pupil and the iris of the eyeball in the frame image. FIG. [Figure 10] 2(b) is a diagram showing the luminance gradient in the boundary region of each of the pupil 501 and the iris 502 in FIG. [Figure 11] FIG. 7 is a diagram showing a frame image pattern obtained by two-dimensionally expanding the pattern shown in FIG. 6(a). [Figure 12] 11 is a conceptual diagram illustrating a process of generating simultaneous linear equations from a plurality of measurement pixels. FIG. [Figure 13] 10 is a flowchart illustrating an example of the operation of a measurement process in the eye movement measurement device according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. 1 is a conceptual diagram showing an example of the configuration of a measurement system using the eye movement measurement device of this embodiment. In Fig. 1, the measurement system 1 includes an eye movement measurement device 10, an infrared irradiation device 20, an imaging lens 30, an infrared filter 40, and an image sensor 50. The eye movement measurement device 10 detects a change in the position of the pupil, which is the point of interest in the eye, from a moving image obtained by capturing input human eye movement.

[0028] The infrared irradiation device 20 includes an LED (Light Emitting Diode) that emits infrared rays to illuminate the eye area including the eyeball of the subject. In this embodiment, the LED is, for example, a device that emits invisible infrared light (infrared rays) with a wavelength of 770 nm. Here, when visible light is used for illumination, the subject may react to the light of the illumination, i.e., the iris may close to reduce the amount of light entering the pupil, causing the size of the pupil to change, and the position of the pupil may change to avoid the light. Or, the subject may be drawn to the light of the illumination, making it impossible to measure eye movement that occurs in a natural state corresponding to the object being observed, which is used to estimate the psychological state of the subject when observing the object. For this reason, in this embodiment, infrared light with an invisible wavelength is used to illuminate the eye in order to measure eye movement without being affected by the irradiated light.

[0029] The imaging lens 30 is a small imaging lens for capturing an image of the eyeball from a close distance, and forms an image of reflected light (reflected infrared light) from the subject eyeball on the imaging surface of the image sensor 50, which will be described later.

[0030] The infrared filter 40 is an optical filter that transmits infrared light of invisible wavelengths and blocks light of visible wavelengths. In this embodiment, for example, an optical filter with a cutoff wavelength of 700 nm is used.

[0031] The image sensor 50 is a device that converts the brightness of light irradiated onto an imaging surface into an electrical signal and outputs the signal as image data, and is, for example, a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor. The image sensor 50 outputs frame images, which are images captured at a predetermined frame period (the frame period in this embodiment will be described in detail later), as a continuous moving image to the eye movement measurement device 10. Here, each of the frame images in the moving image is captured in multiple gradations.

[0032] The eye movement measuring device 10 includes a data input / output unit 11, a measurement range setting unit 12, a luminance gradient calculation unit 13, a luminance change amount calculation unit 14, a movement distance calculation unit 15, a display unit 16, and a memory unit 17. The data input / output unit 11 writes the captured image supplied from the image sensor 50 into the storage unit 17 for storage. In addition, the data input / output unit 11 outputs the pupil movement amount (the movement amount in the rotation of the eyeball, which is the x-axis movement amount Δx and the y-axis movement amount Δy between frames in successive frame periods described later) measured by the eye movement measurement device 10 to an external device.

[0033] The measurement range setting unit 12 sets the measurement range as an image region for extracting measurement pixels used to calculate the movement amount of the pupil in a frame image that is a captured image. In this embodiment, a method is used to measure the movement amount of the eyeball at the point of interest using a gradient method that measures the movement amount using a luminance gradient and a luminance change amount. Therefore, in order to improve the accuracy of the measured amount of movement by using an area with a large luminance gradient to measure a large amount of luminance change even if the amount of movement is small, the boundary area of ​​the pupil and iris, which has a higher luminance gradient than other parts of the eyeball, is used as the measurement target, as will be described later.

[0034] 2A and 2B are diagrams showing a frame image and luminance values ​​of pixels in an image area in which each part of the eye is imaged in the frame image. Fig. 2A shows a frame image including an image area of ​​a pupil 501, an iris 502, and a conjunctiva 503 in an eyeball 500. Moreover, a boundary region 505 indicates an image region near the boundary between the pupil 501 and the iris 502. A reflective region 506 is an image region in which a reflected image of the irradiated infrared light source is captured, and has a higher luminance value than the surrounding image regions.

[0035] Fig. 2(b) is a graph showing the luminance value of each pixel on the line segment A-A' in the frame image of Fig. 2(a). In Fig. 2(b), the horizontal axis shows the x-coordinate of the pixel (pixel position), and the vertical axis shows the luminance value of each pixel. A dotted line B1 indicates the luminance value of a pixel in the imaging region of the sclera 503 of the eye. A dotted line B2 indicates the luminance value of pixels in an imaging region of a boundary region 505 between the pupil 501 and the iris 502.

[0036] Each of the dotted lines B3 and B4 indicates the luminance value of a pixel in the imaging area of ​​the reflective area 506. A dotted line B5 indicates the luminance value of the pixel in the imaging region of the boundary between the iris 502 and the sclera 503 of the eye. In this embodiment, since the boundary area between the pupil and the iris of the eye has a large luminance gradient compared to other areas of the eye, the gradient method is used to calculate the movement amount of the pupil. For this reason, the imaging area of ​​the pupil is extracted from the frame image in which the eye is imaged.

[0037] Here, when detecting the imaging region in which the pupil 501 is imaged, the measurement range setting section 12 compares the luminance value of each pixel of the frame image with the pupil detection threshold value. Then, the measurement range setting unit 12 extracts a collection of pixels having a luminance value less than the pupil detection threshold (dotted line 10 in FIG. 2(b)) as an imaging area in which the pupil 501 is imaged. The imaging area of ​​the pupil 501 has the lowest luminance value in the frame image, and therefore can be easily distinguished from the imaging areas of other parts.

[0038] In this embodiment, the pupil detection threshold value is an intermediate value between the luminance value of the pixel where the pupil is imaged and the luminance value of the pixel where the iris is imaged. Furthermore, the pupil detection threshold may be determined by experimentally capturing images of a number of human eyes and detecting whether the pupil luminance value is distinguishable from other surrounding areas. In this embodiment, pixels near a boundary region 505 between the pupil 501 and the iris 502, which is the range with a large brightness gradient (the region indicated by the dotted line B2) shown in Figure 2(a), are used as measurement pixels (described later) used to measure (estimate) the movement amount of the pupil 501 by the gradient method.

[0039] FIG. 3 is a conceptual diagram of an eye for explaining the process of setting the measurement range from the imaging region of the pupil 501 in the frame image. After extracting the imaging area of ​​the pupil 501 described in FIG. 2, i.e., the inner portion of the boundary area 505, the measurement range setting unit 12 extracts the central pixel (center O) of the pupil 501 area, and sets the number of pixels from the central pixel to the boundary area 505 as the radius R of the pupil 501 area. Then, the measurement range setting unit 12 multiplies the radius R by a predetermined coefficient value p (p>1, for example, 1.5 in this embodiment) and sets the range of radius R′ as the measurement range (processing target area) 600.

[0040] That is, the measurement range setting unit 12 determines a measurement range 600 that necessarily includes a boundary region 505 between the pupil 501 and the iris 502 as a region for extracting measurement pixels for measuring the amount of movement of the pupil (that is, the amount of movement of the eyeball). This prevents the frame image of the subject's eye from including unnecessary images of parts that could cause errors, such as glasses or eyebrows, other than the eyeball, and determines the measurement range 600 as the area that is most likely to contain the boundary area of ​​the pupil and iris.

[0041] Returning to FIG. 1, the luminance gradient calculation unit 13 judges whether or not each pixel in the frame image is within the measurement range 600 described above. Then, the luminance gradient calculation unit 13 determines whether or not the luminance value of each pixel within the measurement range 600 is less than a preset luminance threshold value (first threshold value).

[0042] As a result, in a frame image capturing an image of the subject's eye, unnecessary pixels that have large luminance values ​​due to reflections of light sources in the captured portion of the eyeball and that may cause errors in detecting the amount of movement are excluded from candidate pixels to be extracted as measurement pixels. Then, the luminance gradient calculation unit 13 determines pixels within the measurement range 600 whose luminance values ​​are less than the above luminance value as candidate pixels, which are candidates for measurement pixels. Next, the luminance gradient calculation unit 13 calculates the luminance gradient at each of the extracted candidate pixels using each of eight adjacent pixels adjacent to the measurement pixel for which the luminance gradient is to be calculated, as explained below with reference to FIG.

[0043] FIG. 4 is a diagram for explaining calculation of a luminance gradient in an area where a candidate pixel exists. Candidate pixel C i,j The luminance gradient of the region 700 to which the pixel belongs is calculated by the luminance gradient calculation unit 13 as an x-axis direction luminance gradient sx and a y-axis direction luminance gradient sy. In the following description, pixel C i-1,j-1 From pixel C i+1,j+1 The luminance values ​​of each of the i-1,j-1 to brightness value C i+1,j+1 Here, pixel C i,j The x-axis luminance gradient sx and the y-axis luminance gradient sy of each of the eight adjacent pixels C i-1,j-1 , C i-1,j , C i-1,j+1 , C i,j-1 , C i,j+1 , C i+1,j-1 , C i+1,j , C i+1,j+1 (These pixels form a pixel region including the measurement pixel).

[0044] The luminance gradient calculation unit 13 calculates the x-axis direction luminance gradient sx by the following formula. sx=[(LC i-1,j-1 -LC i+1,j+1 ) / 2+(LC i,j-1 -LC i,j+1 ) / 2+(LC i+1,j-1 -LC i+1,j+1 ) / 2] / 3 That is, the x-axis luminance gradient sx is i-1,j-1 The brightness value of LC i-1,j-1 From pixel C i+1,j+1 The brightness value of LC i+1,j+1 The brightness gradient obtained by subtracting and dividing by two pixels, and pixel C i,j-1 The brightness value of LC i,j-1 From pixel C i,j+1 The brightness value of LC i,j+1 The brightness gradient obtained by subtracting and dividing by two pixels, and pixel C i+1,j-1 The brightness value of LC i+1,j-1 From pixel C i+1,j+1 The brightness value of LC i+1,j+1 It is calculated by subtracting the luminance gradient and dividing it by two pixels, and averaging it.

[0045] Similarly, the luminance gradient calculation unit 13 calculates the y-axis direction luminance gradient sy by the following formula. sy=[(LC i-1,j-1 -LC i+1,j-1 ) / 2+(LC i-1,j -LC i+1,j ) / 2+(LC i-1,j+1 -LC i+1,j+1 ) / 2] / 3 That is, the y-axis luminance gradient y is i-1,j-1 The brightness value of LC i-1,j-1 From pixel C i+1,j-1 The brightness value of LC i+1,j-1 The brightness gradient obtained by subtracting and dividing by two pixels, and pixel C i-1,j The brightness value of LC i-1,j From pixel C i+1,j The brightness value of LC i+1,j The brightness gradient obtained by subtracting and dividing by two pixels, and pixel C i-1,j+1 The brightness value of LC i-1,j+1 From pixel C i+1,j+1 The brightness value of LC i+1,j+1 It is calculated by subtracting the luminance gradient and dividing it by two pixels, and averaging it.

[0046] Fig. 5 is a diagram showing an example of calculation of the luminance gradient in an image frame, in which the horizontal axis indicates the coordinate position (pixel position) of a pixel, and the vertical axis indicates the luminance value of the pixel. Pixel position x n-1 The luminance gradient of s n-1 is the pixel position xn-2 From the brightness value of the pixel at x n The brightness value of the pixel, 75, is subtracted and divided by the distance of two pixels to get 25 (=(125-75) / 2=50 / 2). Also, pixel position x n+1 The luminance gradient of s n+1 is the pixel position x n From the brightness value of the pixel at x n+2 The brightness value of the pixel, 25, is subtracted and divided by the distance of two pixels to get 25 (=(75-25) / 2=50 / 2).

[0047] Returning to FIG. 1, the luminance gradient calculation unit 13 calculates a value obtained by dividing the absolute value of the luminance gradient S of the candidate pixel (i.e., a composite vector of vector sx·X and vector sy·Y, where X is a unit vector in the x-axis direction and Y is a unit vector in the y-axis direction) by the square root of the luminance value I of the candidate pixel (=|S| / (I) 1 / 2 ) is equal to or greater than a preset gradient brightness value ratio threshold (second threshold). Here, |S|=(sx 2 +sy 2 ) 1 / 2 and the square root of the luminance value (I) 1 / 2 is the square root of the number of photons (N) 1 / 2 In addition, sx is the x-axis luminance gradient indicating the amount of pupil movement in the x-axis direction, and sy is the y-axis luminance gradient indicating the amount of pupil movement in the y-axis direction (described later). Then, the luminance gradient calculation unit 13 calculates |S| / (I) 1 / 2 A candidate pixel whose gradient luminance value ratio is equal to or greater than the threshold value is determined as a measurement pixel.

[0048] At this time, the brightness gradient calculation unit 13 may be configured to display on the display unit 16 a confirmation image in which the measurement pixel is marked in the frame image, indicating at which position the pixel in each of the consecutive frame images is the measurement pixel. The user may then observe the confirmation image on display unit 16 and adjust the gradient luminance value ratio threshold so that only pixels in the boundary region between the pupil and the iris are measurement pixels, thereby re-extracting the measurement pixels, or may detect pixels extracted as measurement pixels outside the boundary region between the pupil and the iris, and remove these pixels from the measurement pixels.

[0049] Fig. 6 is a conceptual diagram for explaining a process for obtaining the amount of movement of the pupil by the gradient method in this embodiment. In Fig. 6, in order to simplify the explanation of the gradient method, calculation of the amount of movement in one dimension (only in the x-axis direction) will be explained. Fig. 6(a) shows the distribution of luminance values ​​in one dimension as a gradation. In Fig. 6(a), pattern 201 shows the n-th frame image, and pattern 202 shows the (n+1)-th frame image. Then, the movement amount Δx (the movement amount Δx in the x-axis direction) is calculated using the same measurement pixel 205 in the pattern 201 and the pattern 202. Here, the measurement pixel 205 is a measurement pixel at the same pixel position in each of the frame images of the pattern 201 and the pattern 202.

[0050] Fig. 6(b) is a graph for explaining how the movement amount Δx is calculated by the gradient method from the luminance gradient sx (luminance gradient sx in the x-axis direction) of the measurement pixel 205 and the luminance change amount ΔI. In Fig. 6, the horizontal axis indicates the coordinate position (pixel position) of the pixel, and the vertical axis indicates the luminance value of the pixel. A curve 203 (solid line) indicates the relationship between the pixel and the luminance value of the pattern 201, and a curve 204 (dotted line) indicates the relationship between the pixel and the luminance value of the pattern 202. The phase of the curve 204 is shifted in the + direction of the x-axis from the curve 203 by the movement amount Δx. This movement amount Δx indicates the movement amount of the pupil in the frame period (the imaging period of the image sensor 50) between frame images. Here, the luminance change amount ΔI is the luminance value I n From the above, the luminance value I n+1 The luminance difference (I n -I n+1 ).

[0051] As the pupil moves (the eyeball rotates and moves), the position of the pupil imaged at measurement pixel 205 of the frame image of pattern 202 moves by an amount Δx from the position of the pupil imaged at measurement pixel 205 of the frame image of pattern 201. In this way, as the pupil position changes, the difference in luminance value I at the position corresponding to the amount of movement Δx in the pupil region is measured as the amount of luminance change ΔI.

[0052] Therefore, the movement amount Δx can be geometrically calculated from the luminance gradient sx of the measurement pixel 205 and the luminance change amount ΔI of the measurement pixel 205 in FIG. 6B by the following formula. 6B, it can be seen that there is a relationship in the relational expression ΔI=-sxΔx. As a result, the movement amount Δx can be calculated by the following first formula, which is a modification of the above relational expression. Δx=-ΔI / sx … 1st equation In this first formula, in the x-axis direction, sx has a positive polarity when the gradient is one where the brightness increases in the positive direction, and a negative polarity when the gradient is one where the brightness increases in the negative direction. Also, the amount of movement Δx has a positive polarity when moving in the positive direction, and a negative polarity when moving in the negative direction.

[0053] Furthermore, when calculating the amount of movement Δx from the above formula 1, it is assumed that the luminance gradient sx of the measurement pixel 205 is constant between the pattern 201 and the pattern 202 (between the frame images). That is, the luminance gradient sx at the measurement pixel 205 of the curve 203 of the pattern 201 and the luminance gradient sx' at the measurement pixel 205 of the curve 204 of the pattern 202 must be considered to be constant within an allowable error. Therefore, when the amount of pupil movement Δx between frame images becomes large, the luminance gradient sx of measurement pixel 205 of pattern 201 and the luminance gradient sx' of measurement pixel 205 of pattern 202 will not each be a constant value, and the above first equation cannot calculate the amount of movement Δx from the amount of luminance change ΔI using the frame images of pattern 201 and pattern 202.

[0054] FIG. 7 is a diagram showing the relationship between pixel positions and pixel luminance values ​​between frame images when the amount of movement Δx is large. Since the amount of movement Δx is large, the shift distance between the curve 203 of the pattern 201 and the curve 204 of the pattern 202 at the measurement pixel 205 becomes large. For this reason, the luminance gradient sx at the measurement pixel 205 of the curve 203 of the pattern 201 and the luminance gradient sx' at the measurement pixel 205 of the curve 204 of the pattern 202 change midway and are not constant (non-linear). Therefore, when calculating the amount of movement Δx using the luminance value ΔI according to the first formula, it is necessary to select a frame period that results in the amount of movement Δx that results in the same luminance gradient sx.

[0055] Therefore, the image sensor 50 in this embodiment uses an image sensor with a short frame period and capable of high-speed imaging, for example, with an imaging surface having 512 x 512 pixels in a grid pattern, and capable of capturing moving images at 500 frames per second.

[0056] In addition, the speed of saccades in normal pupil saccadic movements (hereinafter referred to as saccadic speed) is a maximum of 500 degrees (°) per second. Therefore, the rotation angle of the pupil in 1 / 500 seconds is a maximum of 1 degree. In addition, as the amount of movement of the pupil becomes smaller, the maximum speed in saccadic movements also decreases. On the other hand, the speed of a microsaccade (amplitude << 1 degree) is, for example, up to 100 degrees / second, so the pupil rotation angle in 1 / 500 seconds is a maximum of 0.2 degrees.

[0057] Here, as the movement amount of the image sensor 50 of the pupil, the maximum pixel movement amount on the imaging surface in one frame time (1 / 500 seconds) is obtained using FIG. The conditions for calculating the maximum pixel movement amount in FIG. 8 are that the resolution of the captured image of the image sensor 50 is 128×128 pixels, and the size of the eyeball in the captured image is 50 pixels from the center of the pupil.

[0058] Fig. 8 is a conceptual diagram for explaining the derivation of the movement amount in the hopping movement of the pupil on the imaging plane. Fig. 8(a) shows a frame image in a moving image as a captured image, and has a resolution of 128 x 128 pixels. Fig. 8(a) also shows the imaging pixel range of the eyeball on the imaging plane of the image sensor 50. The pupil imaging area EA is the area in the captured image 50A where the pupil is imaged. A circular area having a radius r (=50 pixels) centered on the center O of the pupil imaging area EA indicates an eyeball imaging area IA, which is an imaging area in which a portion of the eyeball is imaged.

[0059] FIG. 8(b) is a diagram for explaining calculation of the pixel movement amount from the rotation angle of the rotation movement of the eyeball during a microsaccade. For example, when the rotation angle is 0.2 degrees, the eyeball movement amount dx (Δx) is calculated as follows from the relationship dx / r=sin(0.2 deg)s=0.0035, assuming that the radius r of the pupil imaging area EA is 50 pixels. dx = sin(0.2deg) x 50 (pixels) = 0.17 (pixels) Furthermore, when the rotation angle is 1.0 degree, the amount of movement dx (Δx) is calculated as follows from the relationship dx / r=sin(1.0 deg)s=0.017, assuming that the radius r of the pupil imaging area EA is 50 pixels. dx = sin(0.2deg) x 50 (pixels) = 0.87 (pixels)

[0060] FIG. 9 is a diagram for explaining the linearity of the luminance gradient in the boundary region between the pupil and the iris of the eyeball in the frame image. 9A shows a region Z (near the boundary region of the pupil and the iris) in which the graph in FIG. 2B is enlarged. The region Z is a region of 30 pixels from pixel position 54 to pixel position 84.

[0061] FIG. 9B is a graph showing an enlarged range Z of 30 pixels from pixel position 54 to pixel position 84 in FIG. Observing the luminance gradient at each pixel in Figure 9(b) shows that the luminance gradient is constant when the movement amount Δx calculated in Figure 8 is in the range of 0.17 pixels, and that there is no difference in the luminance gradient even in the range of 0.87 pixels. Therefore, it is possible to measure the movement amount of the pupil within a movement range where the luminance gradient falls within a similar pixel region (within an area of ​​9 adjacent pixels including the measurement pixel).

[0062] Therefore, as shown in Figure 8, by capturing images at a frame rate of 500 frames / second, the measurement of the movement amount Δx in microsaccades of 100 degrees / second or less is within the range of the pupil with a certain luminance gradient. As a result, if the luminance gradient sx between frame images is equal to calculate the movement amount Δx, the movement amount in the x direction will also be equal, and errors due to nonlinearity can be suppressed, and the movement amount Δx can be measured with a specified accuracy using Equation 1.

[0063] In this embodiment, as described above, the amount of movement of the pupil in the x-axis direction (and the y-axis direction, which will be described later) is calculated using the gradient method from the amount of change in luminance ΔI and the luminance gradient sx of pixels in a pixel region in which an image of the boundary region between the pupil and the iris is captured. However, since the change in pupil position accompanying eye movement during a microsaccade is suppressed to a minute amount Δx as already mentioned, the decrease in accuracy of detecting the amount of movement due to the influence of measurement noise has a large effect.

[0064] Generally, shot noise occurs due to statistical fluctuations in the number of photons (actually, the amount of charge corresponding to the number of photons) accumulated in each pixel on the imaging surface of an image sensor. And, this shot noise level is the square root of the number of accumulated photons (N) 1 / 2 is proportional to.

[0065] In addition, the level of the luminance signal is proportional to the number of accumulated photons N, and the level of the shot noise is proportional to (N) as described above. 1 / 2 Since the signal-to-noise ratio is proportional to (N) 1 / 2 (=N / (N) 1 / 2 ) is proportional to Therefore, in luminance measurement, as the number of accumulated photons N increases, the signal-to-noise ratio improves and the effect of shot noise in luminance measurement is reduced.

[0066] On the other hand, since the brightness gradient and the brightness change amount are each proportional to the difference value of the number of accumulated photons N, the accuracy does not improve as the number of accumulated photons N increases, as in the case of the brightness signal. Conversely, as the number of accumulated photons N increases, the shot noise (N) included in the difference value 1 / 2 increases in proportion to For this reason, when the luminance gradient is the same, the measurement accuracy of the luminance gradient and the amount of change in luminance is improved by using pixels with smaller luminance values. For this reason, as already described, the luminance gradient calculation unit 13 calculates |S| / (I) 1 / 2 Candidate pixels whose gradient intensity value ratio is equal to or greater than the threshold value are used as measurement pixels. Here, the square root of the intensity value (I) 1 / 2 is the square root of the number of photons (N) 1 / 2 is proportional to.

[0067] FIG. 10 is a diagram showing the luminance gradient in the boundary region of each of the pupil 501 and the iris 502 in FIG. In FIG. 10, a luminance difference ΔI between a frame image N and a frame image N+1 one frame period after this frame image N when the pupil moves by Δx is obtained. When pixels are selected as measurement pixels, if the pixels have the same luminance gradient, the luminance difference ΔI when moved will be the same for all pixels.

[0068] However, the pixel position 601 has a photon count of 200, and the noise amount is 14.1 (=(200) 1 / 2 ) at pixel position 602, the number of photons is 100, and the noise amount is 10 (=(100) 1 / 2 ) In addition, the number of photons at pixel position 603 is 20, and the amount of noise is 4.7 (=(20) 1 / 2 ). Therefore, the movement amount Δx calculated at the pixel position 601 602 is the amount of movement Δx calculated at the pixel position 603603 Compared to , the noise level is 3 (= 14.1 / 4.7) times higher, and the measurement accuracy is reduced.

[0069] In this embodiment, the measurement pixels capturing the boundary region between the pupil and the iris, which is a place where the luminance gradient is higher than other regions, are used to measure the amount of movement Δx (the same applies to the amount of movement Δy in the y-axis direction, which will be described later). As already mentioned, the boundary region between the pupil and the iris has a large luminance gradient, and even a small amount of movement Δx results in a large amount of luminance change ΔI, making it suitable for detecting the amount of movement Δx.

[0070] In order to reduce the effect of the above-mentioned shot noise, when the same luminance gradient is present, a measurement pixel having a lower luminance value (a smaller number N of accumulated photons) is used. Therefore, in this embodiment, as already described, the luminance gradient calculation unit 13 calculates |S| / (I) of each candidate pixel. 1 / 2 is determined to be equal to or greater than a preset gradient luminance value ratio threshold, and candidate pixels that are equal to or greater than the gradient luminance value ratio threshold are extracted as measurement pixels.

[0071] In this embodiment, in order to obtain the movement amount of the pupil in the eyeball in two dimensions, the movement amount Δx parallel to the x-axis is defined as the movement amount Δx in the x-axis direction, the movement amount Δy parallel to the y-axis is defined as the movement amount in the y-axis direction, the luminance gradient parallel to the x-axis direction is defined as the luminance gradient in the x-axis direction, and the luminance gradient parallel to the y-axis direction is defined as the luminance gradient in the y-axis direction, and the first equation, which is a one-dimensional equation already described, is expanded to the second equation, which is a linear equation shown below. ΔI=(-s x )·Δx+(-s y )·Δy … Second formula

[0072] In this second formula, the x-axis luminance gradient s x The polarity of is + when the gradient of brightness increases in the positive direction along the x-axis, and is - when the gradient of brightness increases in the negative direction along the x-axis. Also, the polarity of the x-axis direction movement amount Δx is + when moving in the positive direction along the x-axis, and is - when moving in the negative direction along the x-axis.

[0073] And in the second equation, the luminance gradient s y is positive when the luminance increases in the positive direction along the y-axis, and the luminance gradient along the y-axis s y has a negative polarity when the gradient is one where the brightness increases in the negative direction. Also, the y-axis direction movement amount Δy has a positive polarity when moving in the positive direction along the y-axis, and a negative polarity when moving in the negative direction.

[0074] In the second equation, the values ​​obtained by measurement are the luminance gradients sx and sy and the luminance change amount ΔI, so the x-axis direction movement amount Δx and the y-axis direction movement amount Δy are both unknowns. Therefore, by solving the following simultaneous linear equations generated by the above-mentioned second formula, the amount of movement in the x-axis direction Δx and the amount of movement in the y-axis direction Δy are calculated. ΔI1=(-s x1 )·Δx+(-s y1 ) Δy ΔI2=(-s x2 )·Δx+(-s y2 ) Δy

[0075] FIG. 11 is a diagram showing a frame image pattern obtained by two-dimensionally expanding the pattern shown in FIG. 6(a). In FIG. 11, a predetermined area 801 of the eyeball is imaged, and a frame image (801) of the area 801 is obtained. One frame period later, an area 802 of the eyeball that has moved from the above area 801 in the x-axis direction by a movement amount Δx and a y-axis direction by a movement amount Δy is imaged, and a frame image (802) of the area 802 is obtained.

[0076] In the measurement pixel 611, the x-axis direction luminance gradient sx is sx1, the y-axis direction luminance gradient sy is sy1, and the luminance change amount between the luminance value of the region 801 and the luminance value of the region 802 is ΔI1. In addition, in the measurement pixel 812, the x-axis direction luminance gradient sx is sx2, the y-axis direction luminance gradient sy is sy2, and the luminance change amount between the luminance values ​​of the region 801 and the region 802 is ΔI2.

[0077] Furthermore, when extracting each of the measurement pixels 811 and 812 from the candidate pixels, the luminance gradient calculation unit 13 calculates |S1| / (I1) 1 / 2 =(sx1 2 +sy1 2 ) 1 / 2 / (I1) 1 / 2 and |S1| / (I2) 1 / 2 =(sx2 2 +sy2 2 ) 1 / 2 / (I2) 1 / 2 and are determined to be equal to or greater than the gradient luminance value ratio threshold. Then, the movement distance calculation unit 15 generates the simultaneous linear equations described above using the measurement values ​​of the measurement pixels 811 and 812 (described later).

[0078] The brightness change calculation unit 14 calculates the difference in brightness value of a specified measurement pixel between frame images by subtracting the brightness value of the specified measurement pixel in a frame image from the brightness value of the same measurement pixel in a frame image one period later. The movement distance calculation unit 15 calculates the x-axis direction movement amount Δx and the y-axis direction movement amount Δy by solving the above simultaneous linear equations using the x-axis direction luminance gradient sx, the y-axis direction luminance gradient sy, and the luminance change amount ΔI, which are the measurement values ​​of the measurement pixel 811 and the measurement pixel 812.

[0079] As described above, the simultaneous linear equations may be generated and solved using two measurement pixels. However, in order to further improve accuracy, in this embodiment, the simultaneous linear equations are generated from a plurality of measurement pixels as shown below. Fig. 12 is a conceptual diagram for explaining a process of generating simultaneous linear equations from a plurality of measurement pixels. Fig. 12 shows a case where six measurement pixels p1 to p6 (1≦n≦6) are selected as measurement pixels for calculating a shift vector as an example of n measurement pixels.

[0080] Here, the luminance gradient calculation unit 13 calculates the x-axis luminance gradient sx n Absolute value of |sx n|(first absolute value) and the absolute value of the y-axis luminance gradient |sy n Compare with |(second absolute value). Then, the luminance gradient calculation unit 13 calculates |sx n |>|sy n The measurement pixels of | (measurement pixels p2, p4, and p6 in FIG. 12) are grouped as an advantageous point group (first group), and n1 , sy n1 Let us assume that. In addition, each sx of the measurement pixels in the x-dominant point group n1 , sy n1 Add up each of them and get Σsx n1 , Σsy n1 Request. Here, in the x-dominant point group, when the x-axis direction gradient is negative, the sign is inverted in order to make the sign of the brightness gradient consistent when accumulating. In Fig. 12, the measurement pixel p4 in the x-dominant point group is the measurement pixel for which polarity inversion is performed. ΔIn1=(-)(-sx n1 )Δx+(-)(-sy n1 )Δy =sx n1 Δx+sy n1 Δy

[0081] Then, the luminance variation calculation unit 14 accumulates the luminance variation ΔIn1 of each of the measurement pixels in the x dominant point group to obtain ΣΔIn1. As a result, the movement distance calculation unit 15 generates the following first linear equation as one of the linear equations based on the x advantageous point group. ΣΔIn1=Σ(-sx n1 )Δx + Σ(-sy n1 )Δy

[0082] On the other hand, the luminance gradient calculation unit 13 calculates |sx n |≦|sy n The measurement pixels of | (measurement pixels p1, p3, and p5 in FIG. 12) are grouped as a y-dominant point group (second group), and n2 , sy n2 Let us assume that. Here, in the y-dominant point group, when the y-axis direction gradient is negative, the sign is inverted in order to make the sign of the brightness gradient consistent when accumulating. In FIG. 12, the measurement pixel p1 in the y-dominant point group is the measurement pixel for which polarity inversion is performed. ΔIn2=(-)(-sx n2 )Δx+(-)(-sy n2 )Δy =sx n2 Δx+sy n2 Δy

[0083] Furthermore, the luminance variation calculation unit 14 accumulates the luminance variation ΔIn2 of each of the measurement pixels in the y dominant point group to obtain ΣΔIn2. This generates the following second linear equation as the other linear equation based on the y dominant point group. ΣΔIn2=Σ(-sx n2 )Δx + Σ(-sy n2 )Δy

[0084] Then, the movement distance calculation unit 15 solves the simultaneous linear equations consisting of the following first and second linear equations to obtain the amount of movement Δx in the x-axis direction and the amount of movement Δy in the y-axis direction. ΣΔIn1=Σ(-sx n1 )Δx + Σ(-sy n1 )Δy ΣΔIn2=Σ(-sx n2 )Δx + Σ(-sy n2 )Δy Furthermore, the data input / output unit 11 outputs the amount of movement Δx in the x-axis direction and the amount of movement Δy in the y-axis direction calculated by the movement distance calculation unit 15 to an external device. In addition, the data input / output unit 11 may sequentially display the x-axis direction movement amount Δx and the y-axis direction movement amount Δy calculated by the movement distance calculation unit 15 on the display unit 16 and write and store them in the memory unit 17.

[0085] As described above, according to this embodiment, the subject's eye is captured as a moving image, and the amount of pupil movement is calculated using the gradient method based on the amount of change in luminance ΔI of the measurement pixel between successive frame images of the moving image, and each of the x-axis luminance gradient and the y-axis luminance gradient. Therefore, there is no need to determine the absolute position of the pupil with high precision; it is sufficient to detect eye movement indicating a microsaccade, so that the device can be made smaller than conventional devices, and since there is no need to fix the subject's head, it is possible to create a device that can be worn on the subject's head. Microsaccades in the subject's everyday state, that is, when moving and moving the head in daily life, can be accurately measured by determining the amount of pupil movement. This makes it possible to measure microsaccades under conditions that are similar to those in real life, i.e., to measure and analyze human responses to objects under a variety of conditions using a wearable measuring device.

[0086] Furthermore, according to this embodiment, in order to detect eye movement, the boundary area between the pupil and the iris in the frame image is used as the area with a large brightness gradient, so that even with minute movements, a large amount of change in brightness can be detected, and the amount of pupil movement can be obtained with high accuracy.

[0087] In addition, according to the present embodiment, the luminance gradient (S) and the square root of the luminance value (I 1 / 2 ) is equal to or greater than a preset gradient brightness value ratio threshold, the pixels used as measurement pixels are those that are less affected by shot noise (so-called dark pixels with low brightness values ​​for the same brightness gradient), which increases in proportion to the square root of the number of photons (brightness value). This improves the measurement accuracy by calculating the pupil movement amount using measurement pixels that are less affected by shot noise (so-called dark pixels with low brightness values ​​for the same brightness gradient), which increases in proportion to the square root of the number of photons (brightness value).

[0088] Furthermore, according to the present invention, the amount of pupil movement is measured by a gradient method using a luminance gradient and a luminance change amount, so that if an image capture device with a sufficiently high frame rate is used, it is possible to obtain a high measurement accuracy of the amount of pupil movement of the detected eyeball even if the resolution itself is not so high. Furthermore, if there is a detectable luminance gradient even if there is no clear feature, it is possible to measure the amount of eyeball movement.

[0089] FIG. 13 is a flowchart for explaining an example of the operation of the measurement process in the eye movement measurement device according to this embodiment. Step S101: When a frame image is supplied from an image sensor 50, which is an external device, the data input / output unit 11 writes the data of the frame image in the storage unit 17 for storage. The measurement range setting unit 12 erases each of the integrated values ​​Σsx1 and Σsx2 of the x-axis direction luminance gradient sx, the integrated values ​​Σsy1 and Σsy2 of the y-axis direction luminance gradient sy, and the integrated values ​​ΣΔI1 and ΣΔI2 of the luminance change amount ΔI obtained from the immediately previous frame image stored in the memory unit 17 (clearing the data).

[0090] Step S102: The measurement range setting unit 12 reads the luminance value data of each pixel of the frame image from the storage unit 17 .

[0091] Step S103: Then, the measurement range setting unit 12 determines whether or not the luminance value of each pixel in the read frame image is less than the pupil detection threshold value. At this time, the measurement range setting unit 12 determines pixel values ​​whose luminance values ​​are less than the pupil detection threshold as pixels in the pupil region where the pupil is captured, while the measurement range setting unit 12 determines pixel values ​​whose luminance values ​​are equal to or greater than the pupil detection threshold as pixels in the region where a part other than the pupil is captured.

[0092] The measurement range setting unit 12 extracts an image area that is a collection of pixels in the pupil area and pixels surrounded by the pixels in the pupil area, and sets this image area as an area in which the pupil is imaged (pupil imaged area). Then, the measurement range setting unit 12 sets the distance from the center O of the measurement range 600 to a pixel on the outer periphery of the measurement range 600 as the radius R. The measurement range setting unit 12 calculates a radius R′ by multiplying the radius R by a predetermined coefficient value p, and sets the range from the center O to the radius R′ as the measurement range 600.

[0093] Step S104: The luminance gradient calculation unit 13 refers to the storage unit 107, selects a pixel in the frame image, and reads the pixel position of that pixel. Here, the luminance gradient calculation unit 13 selects pixels in the order of columns, for example, starting from the top row in the reference frame image, while sequentially switching the row to be selected.

[0094] Step S105: The luminance gradient calculation unit 13 determines whether or not the pixel position of the selected pixel is included in the measurement range 600 having a radius R' from the center O. At this time, if the pixel position of the selected pixel is included in the measurement range 600 having a radius R' from the center O, the luminance gradient calculation unit 13 advances the process to step S106. On the other hand, if the pixel position of the selected pixel is not included in the measurement range 600 of radius R' from the center O, the luminance gradient calculation unit 13 advances the process to step S112.

[0095] Step S106: The luminance gradient calculation unit 13 refers to the storage unit 107 and reads the luminance value of the pixel selected in the frame image. Then, the luminance gradient calculation unit 13 determines whether or not the pixel value of the read pixel is less than a luminance threshold value (first threshold value).

[0096] At this time, if the pixel value of the read pixel is less than the luminance threshold value (first threshold value), the luminance gradient calculation unit 13 extracts this pixel as a candidate pixel and advances the process to step S107. On the other hand, if the pixel value of the read pixel is not less than the luminance threshold value (first threshold value), the luminance gradient calculation unit 13 advances the process to step S112.

[0097] Step S107: The luminance gradient calculation unit 13 calculates each of the x-axis direction luminance gradient sx and the y-axis direction luminance gradient sy of the candidate pixel from the eight adjacent pixels by the process described with reference to FIG. Then, the luminance gradient calculation unit 13 calculates (sx 2 +sy 2 ) 1 / 2 The brightness gradient S is calculated by the above calculation.

[0098] Step S108: The luminance gradient calculation unit 13 calculates |S| / (I) from the calculated luminance gradient S and luminance value I. 1 / 2 Calculate the calculated |S| / (I) 1 / 2 is equal to or greater than the gradient luminance ratio threshold. At this time, the luminance gradient calculation unit 13 calculates |S| / (I) 1 / 2 If is equal to or greater than the gradient luminance ratio threshold, the candidate pixel is extracted as a measurement pixel, and the process proceeds to step S109. On the other hand, the luminance gradient calculation unit 13 calculates |S| / (I) 1 / 2 If is less than the gradient luminance ratio threshold, the process proceeds to step S112.

[0099] Step S109: The luminance gradient calculation unit 13 calculates the luminance gradient sx in the x-axis direction of each measurement pixel. n Absolute value of |sx n | and the y-axis luminance gradient sy n Absolute value of |sy n Compare with |. Then, the luminance gradient calculation unit 13 calculates |sx n |>|sy n The measured pixels of | are grouped as the x-dominant point group (first group), and |sx n |≦|sy n The measured pixels of | are grouped as a y-dominant point group (second group). Then, when the measurement pixel is a measurement pixel of the x-advantageous point group, the luminance gradient calculation unit 13 calculates the x-axis luminance gradient sxn1 and y-axis luminance gradient sy n1 The integrated value Σsx is calculated by integrating each of the above. n1 , Σsy n1 is calculated and written and stored in the storage unit 17. On the other hand, when the measurement pixel is a measurement pixel of the y-dominant point group, the luminance gradient calculation unit 13 calculates the x-axis luminance gradient sx n2 and y-axis luminance gradient sy n2 The integrated value Σsx is calculated by integrating each of the above. n2 , Σsy n2 is calculated and written and stored in the storage unit 17.

[0100] Step S110: The luminance change amount calculation unit 14 refers to the frame image data in the storage unit 17, and subtracts the luminance value of the corresponding measurement pixel in the frame image from the luminance value of the measurement pixel in the frame image immediately before to calculate the luminance change amount ΔI.

[0101] Step S111: When the measurement pixel is a measurement pixel of the x advantageous point group, the luminance change amount calculation unit 14 accumulates the luminance change amount ΔI1 (=ΔI) of the measurement pixel of the x advantageous point group to calculate the accumulated value ΣΔI1 and writes and stores it in the memory unit 17. On the other hand, when the measurement pixel is a measurement pixel of the y advantageous point group, the brightness change amount calculation unit 14 accumulates the brightness change amount ΔI2 (=ΔI) of the measurement pixel of the y advantageous point group to calculate the accumulated value ΣΔI2 and writes and stores it in the memory unit 17.

[0102] Step S112: The luminance gradient calculation unit 13 determines whether or not the processing has been completed for all pixels in the frame image. Here, the luminance gradient calculation unit 13 makes a determination based on, for example, whether or not the selected pixel is the pixel position of the end point in the frame image. At this time, if the processing for all pixels in the frame image has been completed, the luminance gradient calculation unit 13 advances the process to step S113. On the other hand, if the process has not been completed for all pixels in the frame image, the luminance gradient calculation section 13 advances the process to step S104.

[0103] Step S113: The moving distance calculation unit 15 calculates the integrated value Σsx n1 , integrated value Σsy n1 And the integrated value ΣΔI1 and the integrated value Σsy n2 , integrated value Σsx n2 And the integrated value ΣΔI2 is read out, and the following simultaneous linear equations are created. ΣΔIn1=Σ(-sx n1 )Δx + Σ(-sy n1 )Δy ΣΔIn2=Σ(-sx n2 )Δx + Σ(-sy n2 )Δy Then, the movement distance calculation unit 15 solves the simultaneous linear equations, calculates the amount of movement Δx in the x-axis direction and the amount of movement Δy in the y-axis direction as the amount of movement of the pupil, and ends the process. At this time, the movement distance calculation unit 15 displays on the display screen of the display unit 16 each of the calculated movement amount Δx in the x-axis direction and the movement amount Δy in the y-axis direction. Furthermore, the data input / output unit 11 may output each of the calculated x-axis direction movement amount Δx and y-axis direction movement amount Δy.

[0104] It should be noted that the program for realizing the functions of the eye movement measuring device 10 of the present invention may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to measure the amount of pupil movement. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Additionally, "computer system" also includes WWW systems equipped with a homepage providing environment (or display environment). Additionally, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" also includes devices that hold programs for a certain period of time, such as volatile memory (RAM) within computer systems that become servers or clients when a program is transmitted via a network such as the Internet or a communication circuit such as a telephone circuit.

[0105] The above program may be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication circuit (communication line) such as a telephone circuit. The above program may also be for realizing part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system. [Explanation of symbols]

[0106] 1. Measurement system 10. Eye movement measuring device 11...Data input / output section 12…Measurement range setting section 13...Brightness gradient calculation unit 14...Luminance change amount calculation unit 15...Travel distance calculation section 16…Display section 17...Storage section 20...Infrared irradiation device 30…Imaging lens 40…Infrared filter 50…Image sensor 300...Eyeball 301...pupil 302...Iris 600…Measuring range

Claims

1. An eye movement measuring device for detecting a change in a position of a focus point of an eye in a moving image of an eye that includes the eye, a measurement range setting unit that sets an area of ​​the moving image that satisfies a predetermined condition as a measurement range; a luminance gradient calculation unit that calculates a luminance gradient in the measurement range; a luminance change amount calculation unit for calculating a luminance change amount between frame images in the measurement range; a moving distance calculation unit that calculates the position change by using a gradient method based on each of the luminance change amount and the luminance gradient; Equipped with The measurement range setting unit selects, within the measurement range, pixels having a luminance value less than a first threshold for reducing an effect of shot noise in the moving image as the measurement range. An eye movement measuring device comprising:

2. The measurement range setting unit sets an area including a boundary between the pupil and the iris of the eyeball as the measurement range.

2. The eye movement measuring device according to claim 1 .

3. The moving image is The position change of the pupil between the frame images is captured at a frame cycle in which the luminance gradient is within a movement range that falls within the measurement range having a constant gradient.

3. The eye movement measuring device according to claim 1 or 2.

4. The measurement range setting unit compares the brightness value of each pixel in the frame image with a brightness threshold value, extracts a pupil image area that is an image area of ​​the pupil, and sets an area obtained by enlarging the pupil image area in a peripheral direction from a center point of the pupil image area as a measurement range.

4. The eye movement measuring device according to claim 2 or 3.

5. The measurement range setting unit selects, as the measurement range, a group of pixels in an area within the measurement range in which the luminance gradient is higher than other areas.

5. The eye movement measuring device according to claim 1, wherein the eye movement measuring device is a movable body.

6. The measurement range setting unit selects pixels having lower luminance values ​​in the luminance gradient as pixels in the measurement range.

6. The eye movement measuring device according to claim 1,

7. The measurement range setting unit selects pixels in the measurement range based on the luminance gradient and the luminance value of a pixel within the measurement range.

7. The eye movement measuring device according to claim 1, wherein the eye movement measuring device is a device for measuring an eye movement based on the eye movement.

8. The measurement range setting unit compares a value obtained by dividing the luminance gradient by an amount of shot noise estimated from the luminance value of a pixel within the measurement range with a second threshold value to select pixels within the measurement range.

8. The eye movement measuring device according to claim 1, wherein the eye movement measuring device is a device for measuring an eye movement.

9. The measurement range setting unit sets pixels within the measurement range, the pixels being those for which a value obtained by dividing the luminance gradient by a square root of the luminance value of the pixel is greater than a second threshold value.

9. The eye movement measuring device according to claim 1,

10. The measurement range setting unit, An x-axis luminance gradient is obtained as a luminance gradient in the x-axis direction corresponding to the measurement range, and a y-axis luminance gradient is obtained as a luminance gradient in the y-axis direction. A simultaneous linear equation is established from the luminance change amount, the x-axis luminance gradient, and the y-axis luminance gradient to obtain an x-axis movement amount that is the movement amount of the pupil in the x-axis direction and a y-axis movement amount that is the movement amount in the y-axis direction.

10. The eye movement measuring device according to claim 1,

11. The measurement range setting unit, comparing a first absolute value, which is an absolute value of the luminance gradient in the x-axis direction, with a second absolute value, which is an absolute value of the luminance gradient in the y-axis direction, in each of the measurement ranges; a first group of pixels in a measurement range in which the first absolute value is greater than the second absolute value, and a second group of pixels in a measurement range in which the second absolute value is greater than the first absolute value; generating one of the simultaneous linear equations, in which a result of adding up the x-axis luminance gradients of each of the pixels in the first group and multiplying the result by an amount of movement in the x-axis direction, which is a variable, and adding up the y-axis luminance gradients and multiplying the result by an amount of movement in the y-axis direction, which is a variable, is an integrated value of the amount of change in luminance of each of the pixels; generating the other equation of the simultaneous linear equations, in which a result of adding up the x-axis luminance gradients of each of the pixels in the second group and multiplying the result by an amount of movement in the x-axis direction, which is a variable, and a result of adding up the y-axis luminance gradients and multiplying the result by an amount of movement in the y-axis direction, which is a variable, is an integrated value of the amount of change in luminance in each of the measurement ranges; The simultaneous linear equations are calculated to obtain the x-axis direction movement amount and the y-axis direction movement amount.

11. The eye movement measuring device according to claim 10.

12. The eye is irradiated with infrared light, and the light reflected from the eye is captured in multiple tones to obtain the moving image. The eye movement measuring device according to any one of claims 1 to 11.

13. An eye movement measuring method for detecting a change in position of a focus point of an eye in a moving image of an eye that includes the eye, the method comprising: a measurement range setting step in which a measurement range setting unit sets an area of ​​the moving image that satisfies a predetermined condition as a measurement range, and selects, within the measurement range, pixels having a luminance value less than a first threshold value for reducing an effect of shot noise in the moving image as the measurement range; a luminance gradient calculation step in which a luminance gradient calculation unit calculates a luminance gradient in the measurement range; a luminance change amount calculation step in which a luminance change amount calculation unit calculates a luminance change amount between frame images in the measurement range; a movement distance calculation step in which a movement distance calculation unit calculates the position change by using a gradient method based on each of the luminance change amount and the luminance gradient; 13. An eye movement measuring method comprising:

14. A program for causing a computer to execute an operation of an eye movement measuring device for detecting a change in a position of a focus point of an eye in a moving image of an eye that includes the eye, The computer, a measurement range setting means for setting an area of ​​the moving image that satisfies a predetermined condition as a measurement range, and selecting, within the measurement range, pixels having a luminance value less than a first threshold value for reducing an effect of shot noise in the moving image as the measurement range; a luminance gradient calculation means for calculating a luminance gradient in the measurement range; a luminance change amount calculation means for calculating an amount of luminance change between frame images in the measurement range; a moving distance calculation means for calculating the position change by using a gradient method based on the amount of brightness change and the brightness gradient; A program to function as a

Images