Gaze analysis device and gaze analysis method

Abstract

[Problem] To make possible a gaze analysis device, a program for gaze analysis, and a gaze analysis method capable of determining whether reflected light obtained on an eyeball is a Purkinje image in the cornea region. [Solution] The present invention comprises an eyeball camera 5 capable of photographing the corneal sphere, and three infrared light sources for irradiating the corneal sphere with infrared rays. The first and second light sources are located on a first straight line that passes through the lens center of the eyeball camera 5, and the third light source is located on a second straight line that lies on a first plane containing the first straight line and orthogonal to the optical axis of the lens, said second straight line intersecting the first straight line at the lens center at a prescribed angle. In an imaging plane, which is a second plane orthogonal to the optical axis, three image points that are likely to have been generated by the light rays from the three light sources are detected, and whether or not the three image points are Purkinje images is determined from the shape of a triangle having the three image points as vertices.

Description

Gaze analysis device and gaze analysis method

[0001] The present invention relates to a gaze analysis device and a gaze analysis method for analyzing the direction of a subject's gaze.

[0002] Eye-gaze analysis technology has many applications. For example, there are cameras that adjust their focus based on the direction of the user's gaze as they look through the viewfinder. Another example is a user interface that allows users to control icons on a display with their gaze. In recent years, this technology has also been used to improve the performance of head-mounted displays (HMDs) for mixed reality (MR) and augmented reality (AR).

[0003] Prior art techniques treat the cornea and eyeball as a composite spherical model with two curvatures. Patent Document 1 describes a configuration in which infrared light-emitting diodes 13a and 13b are positioned symmetrically with respect to the optical axis of an eyeball camera that photographs the subject's cornea, irradiating the cornea with infrared light and capturing an image. The light from the two infrared light sources produces two specular reflections on the corneal surface, which are then photographed by the eyeball camera. The direction of the straight line connecting the center of the iris (or the pupil at its center), which is separately detected, and the center of curvature of the cornea is then calculated as the optical axis direction of the eyeball (see Figure 5 of Patent Document 1).

[0004] JP 2000-121917 A JP 2004-129927 A

[0005] "New Fundamentals of Ophthalmic Optics" by Nishinobu Mototsugu, Iwata Koichi, and Uosato Hiroshi, Kanehara Publishing, 2012. "Optics and Glasses, 2nd Edition," edited by Matsumoto Fumiko et al., Igaku Shoin, 2023.

[0006] In the technology described in Patent Document 1, light from two LED light sources is specularly reflected on the corneal surface, forming two images on the imaging plane of the eye camera. These images are sometimes referred to as Purkinje images. The optical axis direction of the eyeball is calculated from these Purkinje images and the pupil center, which is detected separately. However, unless the specular reflection image is generated on a cornea that can be considered spherical, the corneal curvature center (hereinafter sometimes simply referred to as the corneal center) cannot be determined. Therefore, it is necessary to determine whether the detected reflected light image is due to reflection on the cornea before proceeding with further analysis.

[0007] Therefore, an object of the present invention is to provide a gaze analysis device and a gaze analysis method that can proceed with processing after determining whether or not reflected light obtained from the eyeball is a Purkinje image in the corneal region.

[0008] An embodiment of a gaze analysis device based on the concept of the present invention comprises an eyeball camera capable of photographing the corneal surface of the eyeball of a subject, and three infrared light sources that irradiate the corneal surface with infrared light, wherein a first line connecting the installation position of a first infrared light source and the installation position of a second infrared light source among the three infrared light sources is disposed at the center of a lens provided in the eyeball camera and is perpendicular to the optical axis of the lens, and a third infrared light source among the three infrared light sources is disposed on a second line that is in a first plane that includes the first line and is perpendicular to the optical axis of the lens, and that intersects the first line at a predetermined angle with the first line at the center of the lens, The above problem is solved by providing a gaze analysis device and a gaze analysis method that detect three image points that may have been generated by infrared light rays emitted from the first infrared light source, the second infrared light source, and the third infrared light source on the imaging surface of the eye camera, which is provided as a second plane perpendicular to the lens optical axis, and if it is determined that the triangle with the three image points as vertices has a triangular shape based on the arrangement of the first infrared light source, the second infrared light source, and the third infrared light source, the three image points are considered to be a first Purkinje image, a second Purkinje image, and a third Purkinje image caused by the first infrared light source, the second infrared light source, and the third infrared light source, respectively, and calculates the position of the center of curvature of the corneal sphere from the positional relationship of the three image points.

[0009] It is possible to provide a gaze analysis device and a gaze analysis method that can proceed with processing after determining whether or not the reflected light obtained at the eyeball is a Purkinje image in the corneal region.

[0010] FIG. 1 is a conceptual diagram of a human eyeball and a diagram of a gaze analysis terminal and a control box constituting a gaze analysis device according to a first embodiment of the present invention. FIG. 2 is a diagram showing the projection relationship between a three-dimensional coordinate system and a two-dimensional coordinate system. FIG. 3 is a diagram showing the ray paths of a lens. FIG. 4 is a diagram showing an example of the arrangement of three infrared light sources in the gaze analysis terminal according to the first embodiment. FIG. 5 is a diagram showing an example in which an image captured by an eye camera includes three Purkinje images on the corneal surface and an example in which diffusely reflected light outside the corneal surface is included. FIG. 6 is a diagram for confirming the constraints on the light path in specular reflection. FIG. 7 is a diagram showing a method for calculating the projection point of the corneal center onto the imaging plane by drawing. FIG. 8 is a block diagram including processing blocks according to a first embodiment of the gaze analysis device according to the present invention. FIG. 9 is a diagram showing a case in which the corneal center of the subject's right eye is exactly on the optical axis of the lens. FIG. 10 is a diagram showing a typical case in which the corneal center of the subject's right eye is not on the optical axis of the lens. FIG. 11 is a cross-sectional view of an eyeball model in a typical case where the corneal center is offset from the optical axis of the eyeball camera lens. FIG. 12 is a diagram showing a method for calculating angular parameters related to the reference light source, reference specular reflection point, lens center, and corneal center on the cutting plane. FIG. 13(C) is a diagram of the corneal center in three-dimensional coordinates. FIG. 14 is a diagram showing the positional relationship between the corneal center and the pupil center in a three-dimensional coordinate system. FIG. 15 is a diagram showing an example where the line of sight changes up and down as the eyeball rotates. FIG. 16 is a diagram showing a process for obtaining the three-dimensional coordinates of the rotation center and the pupil center. FIG. 17 is a flowchart of a gaze analysis device based on the concept of the present invention. FIG. 18 is a diagram showing an application example of a gaze vector. FIG. 19 is a diagram showing the spatial relationship between a second gaze vector and the imaging system of a field of view camera. FIG. 20 is a diagram showing an arrangement of an infrared light source provided in an eyeball camera in a fourth embodiment of the gaze analysis device based on the concept of the present invention. FIG. 21 is a diagram showing an example of installation of an infrared light source in a gaze analysis device based on the concept of the present invention. Fig. 22 is a diagram showing another example of the arrangement of infrared light sources in a line-of-sight analysis device based on the concept of the present invention, and Fig. 23 is a diagram showing an example in which a cutting plane common to two infrared light sources is not set.

[0011] <Human Eyeball Model> Fig. 1 is a conceptual diagram of a human eyeball and a diagram of a gaze analysis terminal and a control box that constitute a gaze analysis device according to a first embodiment based on the concept of the present invention. A human eyeball model is shown, which is a premise for constructing the first embodiment based on the concept of the present invention. Fig. 1(A) is a conceptual diagram of a human eyeball, and is a schematic diagram of the longitudinal cross-sectional structure when looking at the right eyeball located in front of the nose from a side view of the head, as shown in Fig. 1(B).

[0012] As shown in Figure 1A, the eyeball E is composed of the sclera E1, which surrounds the vitreous body E12 in a spherical shape, and the cornea E2, which has a spherical surface that is approximately 60% of the size of the sclera E1. This is why the cornea and eyeball are treated as a composite spherical model with two curvatures. The eyeball E has a structure in which the spherical surface of the cornea E2 protrudes slightly in front of the sclera E1, which is the sphere that occupies the majority of the eyeball E.

[0013] The cornea E2 is a spherical, transparent membrane, behind which the iris E3 and crystalline lens E4 are located. The iris E3 functions as a diaphragm, attenuating the amount of light entering from the front. The circular window in the iris E3 is the pupil E31.

[0014] Furthermore, the cornea E2 has a spherical shape and a center of curvature E21 can be assumed. However, in Fig. 1A, the position of the center of curvature is displayed for convenience, and no such marker actually exists. Hereinafter, the center of curvature of the cornea E2 will be simply referred to as the corneal center E21, and its three-dimensional coordinates may also be expressed as Cen.

[0015] The iris E3 is the part that determines eye color, the so-called black part of the eye. Most of the cornea E2 overlaps with the iris E3, but the boundary between the area that appears to be the black part and the area that appears to be the white part of the eye does not directly indicate the boundary of the cornea E2.

[0016] 1C is a schematic diagram of the right eyeball when viewed facing the face. The eyeball E is surrounded by the superior rectus muscle E51, inferior rectus muscle E52, lateral rectus muscle E53, and medial rectus muscle E54. The vertical rotation of the eyeball E is mainly due to the coordinated action of the superior rectus muscle E51 and the inferior rectus muscle E52. The horizontal rotation is mainly due to the coordinated action of the lateral rectus muscle E53 and the medial rectus muscle E54.

[0017] There is no physical center of rotation for the eyeball E. However, the center of rotation of the eyeball E, which can rotate smoothly within the limited space of the skull, must be localized in a somewhat limited area. For convenience, this center of rotation is called the center of rotation, and its three-dimensional coordinates are sometimes expressed as Ro.

[0018] In addition, the optical axis of the optical system formed by the eyeball extends from the head toward the front of the face in the human eyeball model. The eyeball is structured to be approximately rotationally symmetrical with respect to the optical axis. Therefore, in this document, Figure 1A may be used as the cross-sectional structure of the eyeball as viewed from the top of the head.

[0019] <First embodiment> <Gaze analysis terminal> Figure 1B is a conceptual diagram of a subject wearing a gaze analysis terminal T1, (D) is the gaze analysis terminal T1, and (E) is a control box 6 that processes the eye image of the subject acquired by the gaze analysis terminal T1 and calculates the gaze vector of the subject.

[0020] As shown in Figures (B) and (D), the gaze analysis terminal T1 has a frame 1 that is hung over the ears of the subject, an arm 2 fixed to the frame 1, a nose pad 3 that supports the frame 1 on the subject's nose, a field of view camera 4 that is positioned above the front of the subject's face and can capture a scene 44 (image capture range) that is substantially the same as what the subject can see, and an eyeball camera 5 that is positioned at the tip of the arm 2 and can capture an image of the cornea of ​​the subject's right eye. In addition, a light source (not shown) that illuminates the cornea with infrared light is provided around the eyeball camera 5.

[0021] The control box 6 stores a calculation unit 61 and a memory unit 62. The memory unit 62 has a program area 621 for storing programs, and an image area 622 for holding images acquired by the eyeball camera 5 and the field of view camera 4. Data is transmitted between the field of view camera 4 and the memory unit 62 via a signal / power cable 43. Conversely, power is also supplied from the control box 6 to the field of view camera 4. Data is also transmitted between the eyeball camera 5 and the memory unit 62 via a signal / power cable 53. Conversely, power is also supplied from the control box 6 to the eyeball camera 5. In addition, the control box 6 may include a power supply 631, a user interface 632, a communication unit 633, and an analysis result output unit 634.

[0022] <Coordinate system for processing eyeball images> The image acquired by the eyeball camera 5 is processed to calculate the gaze vector of the subject's eyeball E. Since the eyeball E to be measured has a three-dimensional structure, a three-dimensional coordinate system is required to define it. The origin GO of this three-dimensional coordinate system is set to the center of the lens 51 of the eyeball camera 5.

[0023] On the other hand, the image acquired by eyeball camera 5 is a two-dimensional image of the three-dimensional structure of eyeball E projected onto imaging surface 52 by the action of lens 51. Therefore, the three-dimensional structure of eyeball E is calculated from the characteristics of the two-dimensional image projected onto imaging surface 52 by ray tracing based on geometric optics.

[0024] FIG. 2A is a diagram showing the projection relationship between a three-dimensional coordinate system and a two-dimensional coordinate system. In the three-dimensional coordinate system, the optical axis of the lens 51 is the GZ axis, and the direction toward the subject's eyeball (in this example, the right eyeball) is defined as the positive direction. An origin GO is set at the center of the lens 51. A plane perpendicular to the GZ axis at the origin GO is referred to as the first plane. On the first plane, the GX axis and the GY axis are set perpendicular to the origin GO. The GX axis, GY axis, and GZ axis form a right-handed three-dimensional coordinate system with the origin at GO. In FIG. 2A, the positive direction of the GX axis is horizontally to the left of the page, and the positive direction of the GY axis is vertically upward. This three-dimensional coordinate system defines the three-dimensional coordinates of each part of the subject's eyeball.

[0025] In a three-dimensional coordinate system, if a plane spanned by the GX-axis and GY-axis can be specified with coordinate values ​​along the GX-axis and the GY-axis, this plane will be called a GX-GY plane. Also, in the same coordinate system, if a plane spanned by the GY-axis and GZ-axis can be specified with coordinate values ​​along the GY-axis and the GZ-axis, this plane will be called a GY-GZ plane. Furthermore, in the same coordinate system, if a plane spanned by the GZ-axis and GX-axis can be specified with coordinate values ​​along the GZ-axis and the GX-axis, this plane will be called a GZ-GX plane.

[0026] Additionally, behind the GX-GY plane (on the negative side of the GZ axis) is a position where an image is formed by the action of lens 51, and an imaging plane 52 is provided at this position as a second plane. This second plane is a plane perpendicular to the GZ axis. A two-dimensional coordinate system is set on imaging plane 52. This two-dimensional coordinate system has its origin o at the intersection of the second plane and the GZ axis. In FIG. 1A, the x-axis is oriented positively to the right of the page, and the y-axis is oriented positively downward.

[0027] In the three-dimensional space defined by the three axes GX, GY, and GZ, the x-axis on the imaging surface 52 is parallel to the GX-axis, and the y-axis is parallel to the GY-axis. In the two-dimensional coordinate system, a plane stretched by the x-axis and y-axis, on which any point can be specified by the coordinate values ​​of the x-axis and the y-axis, will be referred to as the x-y plane.

[0028] <Ray Tracing Based on Geometric Optics> Before discussing the projection of a three-dimensional structure onto a two-dimensional image, we will confirm the lens center and the optical axis of the lens based on geometric optics, which is also described in Non-Patent Document 1. Most lens curved surfaces are spherical. Except when the lens is a perfect sphere, there are separate centers of curvature for the spherical surface facing the object that is the subject and for the spherical surface facing the imaging surface. The straight line passing through these two centers of curvature, which are located at different positions, is the optical axis of the lens.

[0029] A focal point exists for one lens spherical surface. The plane that is the origin of the distance to the focal point, i.e., the focal length, is called the principal plane or main surface. Originally, a principal plane is a plane that is tangent to the lens spherical surface on the optical axis, but since the center of curvature of the spherical surface is on the optical axis, the principal plane is a plane that is perpendicular to the lens optical axis.

[0030] A convex lens has a spherical surface that is convex toward both the object side and the imaging plane side. Depending on the distance between the lens and the subject, compared to the lens thickness, there are thick lenses whose lens thickness must be taken into consideration, and thin lenses whose lens thickness can be ignored. In both cases, the object side and the imaging plane side are assumed to be filled with air with a refractive index of 1, and the refractive index inside the lens is assumed to be n greater than 1. Here, the principal point of a lens refers to the intersection of the principal plane and the optical axis, and as mentioned above, the principal plane is the plane from which the focal length originates. Furthermore, the nodal point is a point that can be considered the center of the lens, and the direction of light rays entering and exiting the lens is the same as that passing through this point.

[0031] Figure 3 shows the ray paths of a lens. Figure 3(A) shows the ray paths of a thick lens, where the lens thickness d cannot be ignored. The lens optical axis is shown horizontally from left to right on the page. An object is located on the left side of the lens, and light rays are emitted from a single object point representing the object, pass through the thick lens, and form a single image point representing the object on the right side. In the case of a thick lens whose object and image sides are filled with air with a refractive index of 1, there are two principal planes, on the object side and the image side, which are the origins of the focal length, the distance to the object point, and the distance to the image point, separated by the thickness d.

[0032] Therefore, the object-side principal plane is referred to as the object-side principal plane, and the intersection of the object-side principal plane and the lens optical axis is the object-side principal point H Ｏ and object-side node N Ｏ The object-side principal plane is the starting point of the object-side focal length of the lens and the distance to the object position. Similarly, there is also a plane that is the starting point of the image-side focal length and the distance to the image-forming position, and this will be called the image-side principal plane. The intersection of the image-side principal plane and the lens optical axis is the image-side principal point H Ｉ and the image-side nodal point N IBoth the object-side principal plane and the image-side principal plane are perpendicular to the optical axis of the thick lens.

[0033] According to the theory of geometric optics, regardless of the direction of incidence, light rays that are incident on the object-side principal plane travel parallel to the lens optical axis and reach the image-side principal plane. Other than that, the situation is the same as in the case of a thin lens. That is, among the outgoing light rays from a point that represents an object (hereinafter referred to as object point light rays), incident light in1 that is parallel to the optical axis of the thick lens is incident on the object-side principal plane, then travels parallel to the optical axis of the thick lens and reaches the image-side principal plane. It then leaves the image-side principal plane as outgoing light out1, passes through the image-side focus, and reaches the image point.

[0034] Furthermore, the object-side nodal point N Ｏ The incident light in2 reaches the image-side nodal point N along the lens optical axis. Ｉ The object point ray reaches the object-side principal plane, travels as outgoing light out2 in the same direction as the incident direction of the incident light in2, and forms an image at the same position as the outgoing light out1. Furthermore, the incident light in3, which passes through the object-side focus, changes direction at the object-side principal plane, travels parallel to the lens optical axis, and reaches the image-side principal plane. It then travels as outgoing light out3 parallel to the optical axis of the thick lens, and forms an image at the same position as the outgoing light out1 and out2.

[0035] Based on the above, a three-dimensional coordinate system for analyzing eyeball images is defined. FIG. 3B is a diagram showing an example in which a three-dimensional coordinate system and a two-dimensional coordinate system are defined when a thick lens 51 is used. First, the lens optical axis is defined as the GZ axis of the three-dimensional coordinate system, and the direction toward the eyeball, which is the object, is defined as the positive direction. The object-side principal plane perpendicular to the GZ axis is referred to as the eyeball-side first plane. The image-side principal plane perpendicular to the GZ axis is referred to as the image-side first plane. Two three-dimensional coordinate systems are defined: an eyeball-side three-dimensional coordinate system based on the position of the eyeball-side first plane, and an image-side three-dimensional coordinate system based on the position of the image-side first plane.

[0036] Regarding the eyeball-side three-dimensional coordinate system, the eyeball-side node N Ｏ That is, the origin GO is located at the intersection of the eyeball-side first plane and the GZ axis. Ｏ and this origin GO is defined in the first plane on the eyeball side. ＯThe GX-axis and GY-axis are set at right angles to each other to form a right-handed three-dimensional coordinate system. Therefore, the GX-GY plane coincides with the eyeball-side first plane. In addition, in the image-side three-dimensional coordinate system, the image-side nodal point N Ｉ That is, the origin GO is located at the intersection of the image-side first plane and the GZ axis. Ｉ and this origin GO is defined in the image-side first plane. Ｉ A right-handed three-dimensional coordinate system is formed by setting the GX axis and the GY axis perpendicular to each other at , and the GX-GY plane coincides with the image-side first plane.

[0037] The GX axis of the eyeball-side three-dimensional coordinate system is parallel to the GX axis of the image-side three-dimensional coordinate system, and the GY axis of the eyeball-side three-dimensional coordinate system is parallel to the GY axis of the image-side three-dimensional coordinate system. In FIG. 1B, the GY axis is positive in the upward direction on the paper, and although not shown, the GX axis is aligned with the origin GO. Ｏ Or GO Ｉ The positive direction is the direction passing through the GX axis and pointing towards the front of the page. A second plane is set that includes the image point on the image side of the lens 51 and is perpendicular to the GZ axis, and an imaging plane 52 is provided on this second plane, and a two-dimensional coordinate system is set here. The origin o is set at the intersection of the second plane and the GZ axis. The two-dimensional coordinate system is also a right-handed x-y system, with the positive direction of the y axis pointing downward on the page, and the positive direction of the x axis passing through the origin o and pointing towards the back of the page. The x axis is parallel to the GX axis, and the y axis is parallel to the GY axis.

[0038] Next, we will show a case where the lens 51 is a thin lens whose thickness can be ignored. Since the thickness d is zero, the object-side principal plane and the image-side principal plane overlap and coincide in Fig. 3(A). Based on this, if a three-dimensional coordinate system is defined for analyzing an eyeball image using the thin lens 51, it becomes the one shown in Fig. 3(C). The eyeball-side first plane and the image-side first plane are aggregated into the first plane. The eyeball-side nodal point N Ｏ and the image-side node N Ｉ are also collected at the node N. Therefore, it is sufficient to provide only one three-dimensional coordinate system with the center of the lens 51 as the origin GO.

[0039] <Projection relationship from object point to image point> Based on Figure 3(C), it is confirmed to which position on the imaging surface on the second plane an object point on the eyeball E is projected. As mentioned above, among the light rays emitted from the object point on the eyeball side, those that are incident on the lens center, i.e., on nodal point N (which is also the origin GO of the three-dimensional coordinate system), maintain their direction and reach the imaging surface 52, where they form an image. In other words, in the case of a thin lens, if a straight line is drawn from the image point through the lens center to the imaging surface 52, this is the projection line that represents the projection destination.

[0040] Even in the case of a thick lens, the object point is Ｏ The ray incident on the image-side nodal point N Ｉ The light is emitted to the image pickup surface 52 and forms an image. Ｏ A straight line parallel to the incident light to the image side nodal point N Ｉ If you draw it from here, this is the projection line that represents the projection destination.

[0041] Now that the projection relationship between object points and image points has been clarified, let us refer back to Figure 2. Figure 2 shows the case of a thin lens 51, but the case of a thick lens will also be explained later as necessary. A second plane is provided parallel to a first plane that includes the center of the lens 51 and is separated by a distance sd from the first plane. An imaging plane 52 is provided on the second plane, and a two-dimensional coordinate system x and y is set thereon.

[0042] Here, the object point OB in FIG. 2A is some feature on the cornea E2, such as the pupil center or other measurement points described later. The object point OB is projected onto the imaging surface 52 by the action of the lens 51. The line segment L is the projection line L that represents this projection direction. The image at the projection point is OBi. The two-dimensional coordinate components at this image point are OBi(ob ｘ , ob ｙ )

[0043] It is necessary to specify the direction of the ray or projection line L from the object point OB to the image point OBi. The projection line L is a line segment passing through three points: the three-dimensional coordinate origin GO, the object point OB, and the image point OBi. The projection line L then exists on a plane uniquely determined by the GY axis and the position of the image point OBi or the object point OB. This plane will be called the projection plane PRy (Figure 2 (B1)). The projection line L also exists on a plane uniquely determined by the GX axis and the position of the image point OBi or the object point OB. This plane will be called the projection plane PRx (Figure 2 (B2)). The projection line L is the intersection of these two projection planes PRx and PRy.

[0044] Therefore, in order to define the direction of the projection line L, the angle ∠oθ formed by the projection plane PRy and the GY-GZ plane is ＧＹ and the angle ∠oφ formed by the projection plane PRx and the GZ-GX plane ＧＸ Then, these two angles (oθ ＧＹ , oφ ＧＸ ) defines the direction of the projection line L. Note that the axis of rotation, for example, GY or GX, may be written as a subscript at the bottom right of a variable that represents an angle or rotation angle. Also, although it is not in accordance with the rules of right-handed coordinates, ∠oφ ＧＸ As shown in FIG. 10B2, the clockwise direction is defined as the positive direction when the GX axis is viewed from the positive region.

[0045] As shown in FIG. 2B1, the projection plane PRy and the GY-GZ plane are spaced apart by ∠oθ ＧＹ The planes intersect at an angle of sd. This angle is determined by the distance sd between the first and second planes and the distance of the image point OBi from the y-axis (the axis perpendicular to the paper). The distance from the y-axis is a constant multiple of the x-coordinate. This constant is the size of one pixel of the image sensor in the x-axis direction.

[0046] As shown in FIG. 2B2, the projection plane PRx and the GZ-GX plane are spaced apart by an angle ∠oφ ＧＸ The planes intersect at an angle of sd. This angle is determined by the distance sd between the first and second planes and the distance of the image point OBi from the x-axis (horizontal axis on the paper). The distance from the x-axis is a constant multiple of the y-coordinate. This constant is the size of one pixel of the image sensor in the y-axis direction.

[0047] Using the distance sd and the two-dimensional coordinates of the image point OBi, ∠oθ is obtained from equation (1-1).ＧＹ can be calculated, and ∠oφ can be calculated from equation (2-1). ＧＸ Conversely, to calculate each component of the two-dimensional coordinates of the image point OBi from the angle, equations (1-2) and (2-2) can be applied. However, here, the size of the image sensor for one pixel is set to 1 in both the x-axis and y-axis directions.

[0048]

[0049] To calculate the gaze vector of the eyeball E, it is necessary to determine the three-dimensional coordinates of the corneal center E21 or the center of rotation E11, which can be the starting point of the vector, and the pupil center Pu, which can be the end point. To determine the three-dimensional coordinates of these object points, it is necessary to identify the direction in which each object point exists and its distance from a reference point or reference line.

[0050] If the entire contour of the cornea E2 or eyeball E as a sphere is projected onto the imaging surface 52, a projection line passing from the center of the projection to the lens center GO can be drawn to determine the direction of the corneal center E21 or the center of rotation E11. However, most of the eyeball E is hidden by the eyelids, and only a portion of the curved surface of the cornea E2 that can be considered a sphere is exposed, making it impossible to capture the entire contour.

[0051] The surface of the sclera E1, which is the white of the eye, lacks smoothness and causes diffuse reflection of light, whereas the cornea E2, which covers the iris E3 and pupil E31 with a spherical surface, is smooth and causes specular reflection of light.

[0052] When a light source is placed at the center GO of the lens 51 and shines onto the cornea E2, light rays traveling in a specific direction are specularly reflected from the surface of the cornea E2, pass through the center of the lens 51 again, and form an image at a certain position on the imaging plane 52. The fact that the specularly reflected light rays return to the same place, the center GO, means that the specular reflection is caused by light rays that are incident in the normal direction to the surface of the cornea E2. The corneal center E21 lies in this normal direction.

[0053] However, placing a light source at the center GO of the lens 51 requires an optical device such as a beam splitter, which impairs compactness and lightness. Therefore, the line-of-sight analysis terminal T1 of the first embodiment is provided with three infrared light sources, each of which is positioned away from the center GO of the lens 51.

[0054] 4A and 4B are diagrams showing an example of the arrangement of three infrared light sources in the gaze analysis terminal of the first embodiment. Fig. 4A shows the arrangement of the eye camera 5 and the three infrared light sources arranged together with the eye camera 5, i.e., the first infrared light source 55R, the second infrared light source 55L, and the third infrared light source 55B, as viewed from the eye of the subject.

[0055] The first infrared light source 55R and the second infrared light source 55L are disposed on either side of the lens 51, and the third infrared light source 55B is disposed below the lens 51. A straight line connecting the center of the first infrared light source 55R and the center of the second infrared light source 55L intersects with the optical axis 56 of the lens 51. This intersection is the center of the lens 51. This straight line may also be referred to as a first straight line LN1.

[0056] The line that passes through the center of the third infrared light source 55B and intersects with the optical axis 56 at the center of the lens may also be referred to as the second line LN2. In this example, the second line is set on the GY axis. Alternatively, the second line may be defined as a line that forms a predetermined angle with the first line LN1.

[0057] 4A, 4B1, and 4B2, the second line LN2 is on the GY axis (not shown) and is perpendicular to the first line LN1. The infrared light sources 55R, 55L, and 55B are provided on the first plane PL1. The optical axis 56 is also the GZ axis of the three-dimensional coordinate system.

[0058] 1B is a diagram showing the configuration of the eyeball camera 5 provided in the gaze analysis terminal T1, as seen from above the head of a subject wearing the gaze analysis terminal T1. An imaging plane 52 is located behind a lens 51, at a distance sd between the lens and the imaging plane. An eyeball camera optical axis 56 is shown, passing from the imaging plane 52 through the center of the lens 51.

[0059] Here, infrared light sources 55R and 55L are arranged on a plane that is perpendicular to eyeball camera optical axis 56 and includes the center of lens 51, i.e., on first plane PL1. Also, imaging plane 52 is provided on second plane PL2 that is behind first plane PL1 and perpendicular to the optical axis of lens 51. Figure 2B2 is a configuration diagram of eyeball camera 5 when viewed from the side of infrared light source 55L.

[0060] 10C1 is a diagram of an eyeball E illuminated by infrared light sources 55R, 55L, and 55B. This eyeball E is imaged as a two-dimensional image on the imaging surface 52 by the imaging action of the lens 51, and this image is converted into electronic data by a large number of imaging elements arranged on the imaging surface 52, and is stored in an image area 622 of the storage unit 62. The calculation unit 61 performs image processing on this electronic data.

[0061] The sclera E1, which is the white of the eye, the iris E3, which is the black part of the eye, and the pupil E31 each have different reflectances to infrared light, so they can be distinguished by the difference in brightness on the imaging plane 52. On the other hand, the boundary between the cornea E2 and the sclera E1 cannot be distinguished by the difference in brightness. Most of the cornea E2 overlaps with the iris E3, and the iris E3 is photographed through the cornea E2. The cornea E2 extends slightly outside the iris E3.

[0062] 1C2 is an enlarged view of the cornea E2, iris E3, and pupil E31 regions in FIG. 1C1. This figure shows an example of three specularly reflected lights generated on the surface of the transparent cornea E2, which externally covers the iris E3 and pupil E31. The image formed on the imaging surface 52 by the lens 51, formed by these reflected lights, is called a Purkinje image.

[0063] Here, the three specular reflections occurring on the corneal surface in three-dimensional space will be referred to as specular reflected light 71R, 71L, and 71B. Then, the images of these reflected light appearing on the imaging plane 52 will be referred to as Purkinje images 71iR, 71iL, and 71iB, respectively. The Purkinje images 71iR, 71iL, and 71iB are formed by the imaging action of the lens 51, where the specular reflected light 71L, 71R, and 71B occurring on the corneal surface have been subjected to a rotationally symmetric transformation about the eye camera optical axis 56, and then to an enlargement or reduction transformation.

[0064] 10D is a diagram showing only the Purkinje images 71iR, 71iL, and 71iB extracted from the imaging plane 52. In addition, 71R, 71L, and 71B represent specularly reflected light, and may also represent the locations on the cornea where specular reflection occurs.

[0065] 5 shows an example in which an image taken by an eyeball camera includes three Purkinje images on the corneal surface and diffusely reflected light outside the corneal surface. Figure 5 (A1) shows an example in which infrared light is specularly reflected and formed as Purkinje images, showing an example of Purkinje image formation in the right eye of a subject wearing a gaze analysis terminal T1, looking down from the top of the head.

[0066] Figure 1 (A2) shows an image of the cornea E2 of the eyeball E photographed from directly in front by the ocular camera 5 under the conditions of (A1). Infrared light from the infrared light source 55R generates specularly reflected light 71R at approximately the center of the cornea E2, and the lens 51 acts to generate a Purkinje image 71iR on the imaging plane 52 (Figure 1 (A3)). However, the image on the imaging plane 52 (Figure 1 (A3)) has been rotated 180° around the GZ axis to match the image in Figure 1 (A2). Two other specularly reflected lights 71L and 71B are also generated on the cornea E2 (Figure 1 (A2)), and Purkinje images 71iL and 71iB are also formed on the imaging plane 52 (Figure 1 (A3)).

[0067] 10B1 shows the state of photography when the subject rotates his / her eyeball to the left. Since the infrared light from the infrared light source 55R does not have an optical path that passes through the center of the lens 51 after specular reflection on the cornea E2, no Purkinje image is formed on the imaging plane 52. Instead, some of the diffusely reflected light 72 generated on the surface of the sclera E1 may form a diffusely reflected light image 72i on the imaging plane 52.

[0068] These diffusely reflected light images are not formed via specular reflection on the cornea, and therefore do not contain information about the surface shape of the cornea E2. Therefore, when analyzing the images formed on the imaging plane 52, it is necessary to reliably remove the diffusely reflected light images.

[0069] 10B2 shows an example in which the eye camera 5 cannot capture the specular reflection occurring within the cornea E2 under the condition of (B1). The infrared light from the infrared light source 55R does not meet the conditions for the specular reflection light occurring on the cornea E2 to form an image by the lens 51, and no Purkinje image is generated. In this example, instead, part of the diffusely reflected light 72 occurring on the surface of the sclera E1 forms an image of diffusely reflected light 72i on the imaging plane 52 [FIG. 10B3].

[0070] Due to the difference in surface smoothness, diffuse reflection of infrared light is likely to occur on the surface of the sclera E1, while diffuse reflection is less likely to occur on the surface of the cornea E2. In addition, the cornea E2 is transparent, and diffuse reflection from the iris E3 behind it forms an image on the imaging plane 52, which contributes to the detection of the pupil E31.

[0071] The Purkinje image formed by specular reflection on the surface of the cornea E2 and the image formed by diffuse reflection on the iris E3 have different brightness levels, so they can be distinguished by the difference in brightness. On the other hand, strong diffuse reflection occurs on the surface of the sclera E1. While specular reflection can occur, diffuse reflection also exists around it.

[0072] In order to accurately measure the surface curvature of the cornea E2 and calculate the position of the corneal center E21, it is necessary to select only the Purkinje image produced by specular reflection from the surface of the cornea E2 and eliminate the image produced by reflection from the sclera E1. For this purpose, three or more infrared light sources are arranged so that three or more Purkinje images are detected in a specific arrangement on the imaging plane 52. This arrangement must be useful for calculating the position of the corneal center E21.

[0073] Therefore, the first Purkinje image 71iR, the second Purkinje image 71iL, and the third Purkinje image 71iB, which are arranged in a specific order, are detected when analyzing the image on the imaging plane 52. Then, only when it is confirmed that the three Purkinje images are arranged in a specific order, is the position of the corneal center E21 calculated based on these two-dimensional coordinate systems.

[0074] Even if a Purkinje image of a specific arrangement is obtained, if there is an image due to diffused reflection in the vicinity thereof, it may not have been obtained on the cornea E2, and therefore it may be possible to exclude this image.

[0075] As mentioned above, only a portion of the spherical surface of the cornea E2 is exposed from the sclera E1, and the direction of the corneal center E21 cannot be determined from its contour. Therefore, the analysis is performed using the Purkinje image as shown below.

[0076] <Constraints on the Optical Path in Specular Reflection> First, the constraints on the optical path in specular reflection will be confirmed based on Fig. 6. Fig. 6(A) shows the optical path of an infrared ray emitted from an infrared light source 55B on the GY axis until it generates a Purkinje image 71iB on the imaging plane 52. The infrared ray passes through the infrared light source 55B, is specularly reflected at point 71B on the surface of the cornea E2, and passes through the lens center GO, generating the Purkinje image 71iB on the imaging plane 52. Here, the specularly reflected infrared ray is incident on and reflected at an angle equiangular to the normal vector N of the surface of the cornea E2.

[0077] The reflected light vector, which represents the direction and orientation of the reflected light, is on the plane formed by the incident light vector and the normal vector at the reflecting surface point 71B. In other words, the incident light vector, reflected light vector, and normal vector are on the same plane. Furthermore, since the surface of the cornea E2 is considered to be spherical, extending the normal vector N backward from the starting point will reach the corneal center E21. Based on this fact, hereinafter, the starting point of the normal vector N may be referred to as the corneal center E21, and the end point may be referred to as being on the corneal surface.

[0078] The infrared light ray travels in the direction of the reflected light vector, passes through the origin GO, which is the center of the lens 51, and reaches the Purkinje image 71iB on the imaging surface 52. On the other hand, if it returns in the opposite direction to the incident light vector, it reaches the infrared light source 55B on the GY axis.

[0079] What is important here is that the infrared light source 55B, the specular reflection point 71B, the origin GO of the three-dimensional coordinate system, the Purkinje image 71iB, and the corneal center E21 are all on the same plane. Furthermore, since the infrared light source 55B is on the GY axis, the same plane includes the GY axis. In other words, the same plane can be said to be a plane uniquely defined by the positions of the GY axis and the corneal center E21.

[0080] The above plane is uniquely defined by the second straight line LN2 connecting the third infrared light source 55B and the origin GO and the position of the corneal center E21, and hereinafter this plane may be referred to as the cutting plane CPy. Here, the second straight line LN2 is on the GY axis. In this case, the projection plane PRy, which defines the projection direction of the object point on the cutting plane CPy onto the imaging plane 52, coincides with the cutting plane CPy. Therefore, equations (1-1) and (1-2) can be applied to the projection directions of the specular reflection point 71B and the Purkinje image 71iB on the same plane.

[0081] 6A shows a cross-sectional view taken along the cutting plane CPy, in which the optical path of the infrared light from the infrared light source 55B to the Purkinje image 71iB and the corneal center E21 are shown on a single plane (on the surface of the paper). When viewed from a viewpoint within this plane, the plane appears to be a straight line, while FIG. 6B shows a view from a viewpoint within the cutting plane CPy looking down from the positive region of the GY axis toward the origin GO. This is also a view of a subject wearing the gaze analysis terminal T1 looking down from roughly the top of their head. The cutting plane CPy is represented by a straight line (dashed line) passing through the origin GO.

[0082] As with the third infrared light source 55B, a light beam emitted from the first infrared light source 55R installed on the GX axis is specularly reflected at point 71R on the surface of the cornea E2 and passes through the lens center GO, generating a Purkinje image 71iR (not shown) on the imaging plane 52. The infrared light source 55R, the specular reflection point 71R, the origin GO of the three-dimensional coordinate system, the Purkinje image 71iR, and the corneal center E21 are all on the same plane.

[0083] This plane is uniquely defined by the first straight line LN1 connecting the first infrared light source 55R, the origin GO, and the second infrared light source 55L, and the position of the corneal center E21, and hereinafter this plane may be referred to as the cutting plane CPx. Here, the first straight line LN1 is on the GX axis. In this case, the projection plane PRx, which defines the projection direction of the object point on the cutting plane CPx onto the imaging plane 52, coincides with the cutting plane CPx. Therefore, equations (2-1) and (2-2) can be applied to the projection directions of the specular reflection point 71R and the Purkinje image 71iR on the same plane.

[0084] Furthermore, the second infrared light source 55L and the Purkinje image 71iL projected onto the imaging surface 52 are also included in the cutting plane CPx, and the formulas (2-1) and (2-2) can also be applied to the projection directions of these.

[0085] 6C is a projected image of the eyeball E on the imaging unit 52. Specular reflections 71R, 71L, and 71B occurring on the surface of the cornea E2 are projected onto the imaging unit 52 as Purkinje images 71iR, 71iL, and 71iB, respectively.

[0086] For the purpose of explanation, Figure 1D is a diagram in which only the Purkinje image is extracted from the image projected onto the imaging unit 52 and is drawn. This diagram is an internal process performed by a computer, and such an image does not actually occur. The intersection line when the cutting plane CPy, which includes the GY axis, the Purkinje image 71iB, and the corneal center E21, intersects with the imaging surface 52 is represented by CPiy. Since this is a projection image of the cutting plane CPy onto the imaging surface 52, it will be referred to as the cutting plane projection image CPiy. Since the cutting plane CPy includes the GY axis, its projection image, the cutting plane projection image CPiy, is parallel to the GY axis and the y axis.

[0087] The line of intersection of the cutting plane CPx, which includes the GX axis, the Purkinje image 71iR, the Purkinje image 71iL, and the corneal center E21, with the imaging surface 52 is represented by CPix. This is a projection image of the cutting plane CPx onto the imaging surface 52, and is therefore referred to as the cutting plane projection image CPix. Because the cutting plane CPx includes the GX axis, the cutting plane projection image CPix, which is its projection image, is parallel to the GX axis and the x axis.

[0088] 7A and 7B are diagrams showing a method for calculating the projection point of the corneal center E21 onto the imaging plane 52 by drawing a diagram. (A) of FIG. 7A shows an example of a state in which the image acquired by the ocular camera 5 is processed to select images that are likely to be Purkinje images and remove others. It is confirmed that there are three images that are likely to be Purkinje images, and that these form an upwardly convex triangle when rotated 180 degrees by the imaging function. After confirmation, the three points that form this triangle are considered to be Purkinje images 71iR, 71iL, and 71iB, and processing continues.

[0089] In FIG. 1B, the two-dimensional coordinates of the Purkinje images 71iR, 71iL, and 71iB are calculated as (x coordinate value, y coordinate value). The x coordinate value is calculated according to the distance from the y axis, and the y coordinate value is calculated according to the distance from the x axis. As a result, 71iR (R ｘ , R ｙ ), 71iL (L ｘ , L ｙ ) and 71iB (B ｘ , B ｙ ) is expressed as

[0090] The infrared light sources 55R and 55L that cause the Purkinje images 71iR and 71iL are arranged on the GX axis. Therefore, on the imaging surface 52, both Purkinje images 71iR and 71iL are projected onto the cutting plane projection image CPix of the cutting plane CPx. Furthermore, the infrared light source 55B that causes the Purkinje image 71iB is arranged on the GY axis. Therefore, on the imaging surface 52, the Purkinje image 71iB is projected onto the cutting plane projection image CPiy of the cutting plane CPy. However, these cutting plane projection images are calculated and do not actually appear on the imaging surface 52 (see FIG. 1B).

[0091] A straight line is drawn between the first Purkinje image 71iL and the second Purkinje image 71iR, and this is set as the first projection line HL. Since the first projection line HL is a line segment of the cutting plane projection image CPix, it must be parallel to the x-axis. If the first projection line HL is not parallel to the x-axis, there is a possibility that an image related to diffuse reflection, not a Purkinje image, has been detected, so the Purkinje image may be detected again.

[0092] Normally, since the first projection line HL is parallel to the x-axis, the y coordinate of the first Purkinje image 71iR and the y coordinate of the second Purkinje image 71iL should be equal. ｙ If the difference between the two y coordinates is within an acceptable range, then the average value of these is ｙ The first projection line HL connecting the Purkinje images 71iR and 71iL forms the base of an upwardly convex triangle (FIG. 1C).

[0093] As described above, the Purkinje image 71iB is generated on the cutting plane projection image CPiy of the cutting plane CPy. As described above, the cutting plane projection image CPiy is parallel to the y-axis. Therefore, a straight line passing through the third Purkinje image 71iB and parallel to the y-axis is drawn and set as the second projection line VL. Alternatively, a straight line intersecting the first projection line HL at 90° may be drawn and set as the second projection line VL.

[0094] In the group of three Purkinje images forming the triangle, a second projection line VL is drawn from the Purkinje image 71iB that forms the apex angle to the first projection line HL. Then, the intersection of the first projection line HL and the second projection line VL is calculated. This is also the intersection of the cutting plane projection images CPix and CPiy. This intersection is designated as E21i (Fig. 1D).

[0095] As described above, the corneal center E21 exists somewhere on the cutting plane CPx, and the corneal center E21 exists somewhere on the cutting plane CPy. In this case, the corneal center E21 exists on the projection line of the intersection of the cutting plane CPx and the cutting plane CPy. Therefore, the intersection will be referred to as the corneal center projection point E21i.

[0096] Through the above construction, the two-dimensional coordinate is E21i (B ｘ , r ｙ ) Here again, C ｘ ≡B ｘ , C ｙ ≡r ｙ Toki, E21i (C ｘ , C ｙ The two-dimensional coordinates of the corneal center E21 are treated as known values. An example of calculating the two-dimensional coordinates of the corneal center E21 has been shown above.

[0097] <Processing Block Diagram> Next, it is necessary to calculate the distance from the reference position to the corneal center E21. Before that, a series of processing flows of a gaze analysis device according to a first embodiment based on the concept of the present invention will be shown. FIG. 8 is a block diagram including processing blocks according to a first embodiment of a gaze analysis device based on the concept of the present invention. The gaze analysis device calculates the subject's gaze direction through processing including blocks A1 to A8. The processing included in these blocks may be executed by dedicated hardware, or may be processed by a calculation unit 61 according to program instructions. In the diagram, arrows between processing blocks indicate the flow of information. The gaze analysis device according to the first embodiment does not perform processing at A1-ad and A8-ad, so these processing blocks are bypassed.

[0098] Block A1 is an eyeball image acquisition unit. This is where an image of the eyeball, which is a three-dimensional object in three-dimensional space, is projected as a two-dimensional image onto imaging surface 52 by the imaging action of lens 51, and after being digitized by imaging elements arranged there, is captured as electronic data in image area 622. This electronic data is then processed in blocks A2 and A5. Alternatively, the electronic image data may be directly processed as described below, separately from being captured in image area 622.

[0099] Block A5 is a two-dimensional processing block, also referred to as the first eyeball image analysis unit. Here, image data captured via the imaging surface 52 is processed according to a two-dimensional coordinate system x and y. This block detects the pupil from the eyeball image. As mentioned above, the eyeball is photographed while illuminated by an infrared light source arranged around the eyeball camera 5, which has sensitivity in the infrared region. This makes it possible to obtain an image that is less affected by ambient light.

[0100] In an eyeball image, the sclera E1, iris E3, and pupil E31 can be relatively easily distinguished because they each have different reflectances of infrared light. In this way, the pupil can be detected by utilizing the difference in reflectance of irradiated light between the pupil and other areas. The pupil and its surroundings are the cornea E2 area. The reflected light used in this analysis is dominated by diffused light from the iris that passes through the cornea E2 and reaches the lens 51 and imaging surface 52. In this block, the range of the pupil area in the eyeball image is determined as two-dimensional coordinates.

[0101] Block A6 is a two-dimensional processing block that detects the center of the pupil region detected in Block A5. Since the pupil can be approximated by a circle or an ellipse, the two-dimensional center coordinates may be obtained by fitting, for example, a circle detection Hough transform or an ellipse detection function from other open source software, such as OpenCV. Block A6 is also called a two-dimensional pupil center detection unit.

[0102] Block A2 is a second eye image analysis unit, a two-dimensional processing block. Here, infrared light from an infrared light source arranged around the eye camera 5 is detected as an image specularly reflected on the cornea. Specular reflection occurs on the cornea. However, in this case, it means obtaining a specular reflection image as a two-dimensional image captured by the imaging action of the lens of the eye camera 5. This specular reflection image is the Purkinje image mentioned above. Because the Purkinje image has high brightness, it is easy to distinguish it from an image caused by diffuse reflection light on the cornea. On the other hand, diffuse reflection light from the sclera can also form an image with high brightness, making it difficult to distinguish it from a Purkinje image.

[0103] Therefore, based on the concept of the present invention, three or more infrared light sources are arranged around the optical axis of the eye camera 5 so that the arrangement of the Purkinje images forms a specific polygon. However, this shape must be useful for identifying the two-dimensional position of the corneal center E21. Here, the two-dimensional position of the corneal center E21 refers to the position where the corneal center E21 is projected onto the imaging surface 52. Once the projected position is known, it is possible to calculate the projection direction starting from the origin GO. This is the direction in which the corneal center E21 exists.

[0104] In the gaze analysis terminal T1 of the first embodiment, three infrared light sources are arranged together with the eyeball camera 5 so that the arrangement of the Purkinje images forms a specific triangle. Three Purkinje images that roughly match this shape are then detected as the first Purkinje image, the second Purkinje image, and the third Purkinje image. Details of this process have already been given.

[0105] Block A3 is a two-dimensional corneal center detection block, which performs two-dimensional processing. This block calculates two-dimensional coordinates indicating the direction of the center of curvature (corneal center E21) when the corneal surface is considered to be a sphere. This is calculated as the foot of a perpendicular line drawn from the apex angle of the triangle formed by the first, second, and third Purkinje images to the base (Figure 7(D)).

[0106] Furthermore, the angle at which the line segment from the apex angle of the triangle intersects with the base is not limited to 90°, and the installation position of the third infrared light source 55B may be designed so that the line segment intersects with the base at a specified angle, as will be described later.

[0107] Block A4 is a corneal center depth calculation unit. This unit calculates the distance between the principal plane of the camera's lens and the corneal center by solving an equation that is formulated by adding constraints on the optical path in specular reflection and the value of the average human corneal radius to the relationship between the position of the infrared light source arranged together with the eye camera 5, the center position of the camera's lens, and the position at which specular reflection occurs in the Purkinje image, and then calculates corneal center depth information for obtaining the three-dimensional coordinates of the corneal center. Details will be described later.

[0108] Block A7 is a pupil center / corneal center calculation block, which is a three-dimensional processing block, and converts the two-dimensional pupil center coordinates obtained by block A6 and the two-dimensional corneal center coordinates obtained by block A3 into three-dimensional coordinates based on the corneal center depth information obtained by block A4.

[0109] Block A8 is a first gaze vector calculation unit, which is a block for three-dimensional processing. Here, a vector starting from the corneal center E21 and ending at the pupil center Pu is calculated, and this vector is set as the first gaze vector. The three-dimensional coordinates of the corneal center E21 (sometimes written as Cen) and pupil center Pu used here are those provided by block A7. The above is an overview of the gaze analysis processing according to the first embodiment.

[0110] Next, a specific example of processing in the corneal center depth calculation unit (block A4) is shown. FIG. 9 is a diagram showing the corneal center E21 of the subject's right eye located exactly on the optical axis (i.e., the GZ axis) of the lens 51. FIG. 9(A) is a GY-GZ cross-sectional view of the right eyeball E and the eyeball camera 5 in three-dimensional space. Because this cross-sectional view includes the GY axis and the corneal center E21, it is also the cutting plane CPy set by the infrared light source 55B installed on the GY axis. This is an example in which the GY-GZ plane and the cutting plane CPy coincide. The nose, shown by the dotted line, is shown to be oriented in this direction, but it does not necessarily exist on the GY-GZ plane or the cutting plane CPy.

[0111] Figure 1B is a GZ-GX cross-sectional view of the right eyeball E and the eyeball camera 5 in three-dimensional space. This cross-sectional view includes the GX axis and the corneal center E21. It is also the cutting plane CPx set by the infrared light source 55R installed on the GX axis. Furthermore, this is also the cutting plane CPx set by the infrared light source 55L installed on the GX axis. This is an example in which the GZ-GX plane and the cutting plane CPx coincide. The nose, shown by the dotted line, is shown to be in this orientation, but it does not exist on the GZ-GX plane or the cutting plane CPx.

[0112] FIG. 1C1 shows the subject's eyeball and Purkinje image projected onto the imaging surface 52. FIG. 1C2 shows the Purkinje images 71iR, 71iL, and 71iB and x-y coordinates on the imaging surface 52. As mentioned above, the origin o of this coordinate system is the point where the GZ axis intersects perpendicularly with the second plane PL2. The x- and y-axes of the vertical and horizontal directions shown in FIG. 1C1 and FIG. 1C2 are shown for the sake of convenience and do not necessarily have to be displayed on the imaging surface 52.

[0113] In Figure 1C2, the straight line CPiy in the vertical direction of the paper is the intersection line between the cutting plane CPy and the imaging plane 52, and is the projection image CPiy of the cutting plane CPy onto the second plane PL2. In this case, the y-axis and the cutting plane projection image CPiy coincide with each other because the cutting plane CPy coincides with the GY-GZ plane.

[0114] The straight line CPix in the left-right direction of the paper is the intersection line between the cutting plane CPx and the imaging plane 52, and is the projection image CPix of the cutting plane CPx onto the second plane PL2. In this case, the x-axis and the cutting plane projection image CPix coincide with each other because the cutting plane CPx and the GZ-GX plane coincide with each other.

[0115] 10A and 10B are diagrams showing a typical case in which the corneal center E21 of the subject's right eye is not located on the optical axis (i.e., the GZ axis) of the lens 51. Fig. 10A shows a diagram in which the GY axis in three-dimensional space and the corneal center E21 of the right eyeball are shown on the same plane, and shows the cutting plane CPy set by the infrared light source 55B installed on the GY axis. The GY-GZ plane and the cutting plane CPy do not coincide.

[0116] In this case, the cutting plane CPy does not include the GZ axis. Instead, the intersection line Intersection-GZ·GX-Plane between the GZ·GX plane and the cutting plane CPy is shown by a dashed line.

[0117] The GY-GX plane and the cutting plane CPy intersect at a predetermined angle with the GY axis as the intersection line, and this angle is reflected in the position of the cutting plane projection image CPiy.

[0118] 1B shows the GX axis in three-dimensional space and the corneal center E21 of the right eyeball on the same plane, and also shows the cutting planes CPx set by the infrared light sources 55R and 55L installed on the GX axis. The GZ-GX plane and the cutting plane CPx do not coincide with each other.

[0119] In this case, the GZ axis is not included in the cutting plane CPx. Instead, the intersection line Intersection-GY·GZ-Plane between the GY·GZ plane and the cutting plane CPx is shown by a dashed line.

[0120] The GZ-GX plane and the cutting plane CPx intersect at a predetermined angle with the GX axis as the intersection line, and this angle is reflected in the position of the cutting plane projection image CPix.

[0121] FIG. 1C1 shows the subject's eyeball and Purkinje image projected onto the imaging plane 52. FIG. 1C2 shows the Purkinje images 71iR, 71iL, and 71iB and x-y coordinates on the imaging plane 52. The cutting plane projection image CPiy is a straight line that passes through the Purkinje image 71iB and is parallel to the y-axis. The cutting plane projection image CPix is ​​a straight line that passes through the Purkinje images 71iR and 71iL and is parallel to the x-axis.

[0122] According to the drawing shown in FIG. 7, the intersection of the cutting plane projection images CPiy and CPix is ​​the corneal center projection point E21i. ｘ , C ｙ ), and these coordinate values ​​are known values.

[0123] The projection direction from the corneal center E21 through the origin GO to the corneal center projection point E21i is the angle ∠Cθ with the GY-GZ plane. ＧＹ and the angle ∠Cφ with the GZ-CX plane ＧＸ is calculated as equations (3-1) and (3-2) by applying equations (1-1) and (1-2) and equations (2-1) and (2-2).

[0124]

[0125] Figure 11 shows a cross-sectional view of the eyeball model in the general situation shown in Figure 10, where the corneal center E21 is off-axis from the optical axis of the lens 51 of the eyeball camera 5. The purpose of the analysis here is to determine the distance from the reference position to the corneal center. The eyeball model is a composite sphere in which the spherical surface of the cornea E2 is partially exposed from the anterior part of the sclera E1.

[0126] The figure shows GO, which is the origin of the three-dimensional coordinate system and the center of the lens 51, the GX axis, and the infrared light sources 55R and 55L on the same axis. The corneal center E21 is also shown. The GX axis, which is the first straight line LN1, and the corneal center E21 are shown on the same plane, which is the cutting plane CPx set by the infrared light sources 55R and 55L installed on the GX axis.

[0127] Generally, the GZ axis is not included in the cutting plane CPx set by the infrared light source on the GX axis. As mentioned above, the GZ-GX plane including the GZ axis intersects with the cutting plane CPx at a predetermined angle with the GX axis as the intersection line. Also, although not shown, there is a GY axis that intersects at right angles with the GX axis at the origin GO, and its positive direction is toward the front of the paper.

[0128] The intersection of the cutting plane CPx and the GY-GZ plane is represented by a dashed-dotted line CPZi passing through the origin GO. For ease of analysis, the CPZ axis, which starts at the origin GO, is set on the cutting plane CPx along the dashed-dotted line CPZi.

[0129] The CPZ axis is introduced to set a two-dimensional coordinate system GX-CPZ on the cutting plane CPx. The origin is GO. As mentioned above, the angle ∠Cφ of the cutting plane CPx with respect to the GZ-GX plane is ＧＸ is obtained by equation (3-2).

[0130] In FIG. 11, the infrared light source 55R is located on the GX axis, and the distance from the center of the lens 51 on the origin GO is ms Ｒ This distance is a designed fixed value. A part of the infrared light emitted from the infrared light source 55R becomes specularly reflected light 71R on the surface of the cornea E2 and enters the center GO of the lens 51.

[0131] As described above, the infrared light source 55R, the center GO of the lens 51, and the specular reflection point 71R are on the cutting plane CPx, and the corneal center E21 is also on the same plane. The vector from the corneal center E21 to the specular reflection point 71R is the normal vector N Ｒ is.

[0132] The infrared light is reflected at the specular reflection point 71R with a normal vector N Ｒ The light is incident on and reflected from the eye at an equal angle relative to the lens 51. The distance between the GX axis and the corneal center E21 on this cutting plane CPx is defined as Len. Len is also the CPZ axis component of the distance between the origin GO and the corneal center E21. Len is a value that can change depending on the relative positions of the eyeball E and the lens 51.

[0133] The angle between the direction from the origin GO, which is the center of the lens, to the corneal center E21 and the CPZ axis is ∠γ. The angle between the direction from the origin GO to the specular reflection point 71R and the CPZ axis is ∠α. Ｒ Furthermore, the normal vector N Ｒ The angle between the CPZ axis and Ｒ Let ∠γ and ∠α Ｒ , ∠β Ｒ The value of can change depending on the relative positions of the eyeball E and the lens 51.

[0134] Furthermore, ∠γ and ∠α Ｒ Regarding , the positive rotation direction is counterclockwise around the origin GO on the paper. Ｒ The clockwise direction is defined as the positive rotation direction around the corneal center E21 on the paper. The distance from the CPZ axis to the specular reflection point 71R is defined as ms Ｒ１ , the distance to the infrared light source 55R is ms Ｒ The difference is ms Ｒ２ Therefore, ms Ｒ = ms Ｒ１ +ms Ｒ２ is.

[0135] A similar analysis may be performed using a specular reflection point 71L generated by a light beam from infrared light source 55L instead of specular reflection point 71R generated by a light beam from infrared light source 55R. Either one of the two specular reflection points 71R or 71L, which are considered equivalent, may be selected for analysis and referred to as the reference specular reflection point, and the infrared light source that emitted the light beam may be referred to as the reference light source to distinguish them. In this case, the Purkinje image generated on the imaging plane 52 via the reference specular reflection point will be referred to as the reference Purkinje image.

[0136] The GX axis here is also the intersection line between the cutting plane CPx and the first plane. Therefore, this intersection line will also be referred to as the analytical reference line. As mentioned above, the cutting plane CPx and the GZ-GX plane intersect at an angle ∠Cφ ＧＸ They intersect forming a line.

[0137] Intersection angle ∠Cφ with GZ-GX plane ＧＸ On the cutting plane CPx, which intersects with Ｒis the angle between the ray of light traveling from the specular reflection point 71R to the origin GO, which is the center of the lens 51, and the CPZ axis. The CPZ axis is an axis located on the intersection of the cutting plane CPx and the GY-GZ plane. Ｒ For this, refer to Fig. 12. Fig. 12 is a diagram showing a method for calculating angle parameters relating to the reference light source, the reference specular reflection point, the lens center, and the corneal center E21 on the cutting plane CPx.

[0138] First, the projection direction from the specular reflection point 71R to the Purkinje image 71iR is determined. As shown in FIG. 7, the coordinates of the Purkinje image are 71iR (R ｘ , C ｙ ) This is because the y coordinates of the Purkinje image 71iR and the corneal central projection point are equal. Therefore, by applying equations (1-1), (1-2), (2-1), and (2-2) to these coordinates, ∠Rθ is obtained as equations (4-1) and (4-2). ＧＹ and ∠Cφ ＧＸ get.

[0139]

[0140] In FIGS. 12A1 and 12A2, the coordinates of the specular reflection point 71R in the three-dimensional space are 71R(R ＧＸ , R ＧＹ , R ＧＺ ) where 71R is (GX-axis component, GY-axis component, GZ-axis component). As described above, the coordinates of the projected image onto the imaging surface 52, i.e., the Purkinje image 71iR, are (R ｘ , C ｙ ) (not shown). (A1) and (A2) in the same figure also show the positional relationship between the specular reflection point 71R and its surroundings in a three-dimensional coordinate system. However, the planes and their angles that are the focus of attention are different in each case.

[0141] In FIG. 12(A1), 71R, p １ , GO and p ２ The rectangle shown by is a part of a plane uniquely defined by the object point 71R and the GX axis. This plane is the projection plane PRx of the specular reflection point 71R, and the Purkinje image 71iR (not shown) is also on this plane. From equation (4-2), the projection plane PRx is located at an angle of ∠Cφ from the GZ-GX plane. ＧＸAs mentioned above, this projection plane PRx is also the cutting plane CPx shown in FIG. 11. Then, in FIG. 12(A1), GO and p ２ The line connecting these is the CPZ axis in Figure 11. Ｒ is the angle formed by the CPZ axis and the line connecting the origin GO and the specular reflection point 71R.

[0142] In FIG. 12(A2), 71R, p ５ , GO and p ３ The rectangle shown by is a part of a plane uniquely defined by the object point 71R and the GY axis, and is the projection plane PRy of the specular reflection point 71R. The Purkinje image 71iR (not shown) is also on this projection plane PRy. From equation (4-1), the projection plane PRy is angled by ∠Rθ between the GY and GZ planes. ＧＹ Furthermore, the triangle GO·p ４ ・p ３ Focusing on this, equation (5-1) is derived using trigonometric functions.

[0143]

[0144] Next, the triangle GO·p in FIG. ４ ・p ２ Focus on the line segment GO・p ２ Using trigonometric functions, the length of the triangle GO·p is expressed as follows: ２ 71R and the triangle GO·p in (A2) of the same figure ４ ・p ３ By using trigonometric functions, we can derive equations (5-3) and (5-4). ＧＹ and ∠Cφ ＧＸ Since it is known, ∠α Ｒ also becomes known.

[0145]

[0146] 12(B1) and (B2) are diagrams showing the relationship between the corneal center E21 and a three-dimensional coordinate system. The coordinates of the corneal center E21 in the three-dimensional space are E21(C ＧＸ , C ＧＹ , C ＧＺ) where E21 is (GX axis component, GY axis component, GZ axis component). As described above, the coordinates of the projected image onto the imaging surface 52, i.e., the corneal center projected point E21i, are (C ｘ , C ｙ ) (not shown).

[0147] In FIG. 1B, E21, q １ , GO and q ２ The rectangle shown by is a part of the projection plane PRx that is uniquely defined by the object point E21 and the GX axis. The corneal center projection point E21i (not shown) is also on this plane. And since this plane includes the corneal center E21 and the GX axis, it is also the cutting plane CPx that is set by the infrared light source on the X axis. Then, in the same figure (B1), GO and q ２ The line connecting these points is the CPZ axis as mentioned above. The angle γ to be calculated is the angle between the CPZ axis and the line connecting the origin GO and the corneal center E21. The angle between the cutting plane CPx and the GZ-GX plane is the angle ∠Cφ calculated by equation (4-2). ＧＸ is.

[0148] Also, in FIG. 2B, the quadrangle E21·q ５ ・GO・q ３ is a part of the projection plane PRy that is uniquely defined by the object point E21 and the GY axis. The angle formed by the projection plane PRy and the GY-GZ plane is ∠Cθ calculated by equation (4-1). ＧＹ Here, the triangle GO·q ４ ・q ３ Focusing on this, equation (6-1) is derived using trigonometric functions.

[0149]

[0150] As mentioned above, a two-dimensional coordinate system GX-CPZ is set on the cutting plane CPx. As mentioned above, Len is the distance from the corneal center to the GX axis, also called the analytical reference line. Therefore, in FIG. 1B1, E21 and q １ The length of the line segment connecting GO and q on the CPZ axis is Len. ２ The length of the line segment connecting these is also Len.

[0151] Next, the triangle GO·q in the same figure (B1)４ ・q ２ Focus on the line segment GO・q ２ Since the length of is Len, we can derive the formula (6-2) by using trigonometric functions. ２ E21 and triangle GO·q in (B2) of the same figure ４ ・q ３ By using trigonometric functions, we can derive equations (6-3) and (6-4). ＧＹ and ∠Cφ ＧＸ Since is known, ∠γ is also known.

[0152]

[0153] As described above, by analyzing the two-dimensional coordinates of the three Purkinje images detected on the imaging plane 52, the angle ∠Cφ formed by the cutting plane CPx and the GZ-GZ plane can be calculated. ＧＸ and the angle ∠Cθ formed by the cutting plane CPy and the GY-GZ plane. ＧＹ Furthermore, in the GX-CPZ coordinate system, which is a two-dimensional coordinate system on the cutting plane CPx, the angle ∠α Ｒ The angle ∠γ formed by the line segment from the origin GO toward the corneal center E21 and the CPZ axis can be calculated. Hereinafter, these will be treated as known values. Meanwhile, Len is an unknown quantity at this stage.

[0154] Referring again to Figure 11, in the two-dimensional GX-CPZ coordinate system set on the cutting plane CPx shown in the figure, the coordinates of the corneal center E21 can be expressed by equation (7-1). However, the coordinates are expressed as (GX component, CPZ component). The coordinates of the specular reflection point 71R can be expressed by equation (7-2). The coordinates of the infrared light source 55R can be expressed by equation (7-3).

[0155]

[0156] Normal vector N Ｒ is a vector from the corneal center E21 to the specular reflection point 71R, and can be expressed by equation (8-1). Ｒ Since the magnitude of is equal to the corneal radius R, the squares of the two are also equal, and therefore equation (8-2) holds.

[0157]

[0158] <First Condition> In addition, in the GX coordinate of the specular reflection point 71R, the distance ms from the origin GO to the infrared light source 55R Ｒ ms Ｒ１ and ms Ｒ２ Divide it into two and make it into equation (9-1). Then, ms Ｒ１ is expressed as equation (9-2). Here, the vectors related to (9-2) are shown in FIG. 13(A). The left side of (9-2) Dis Ｒ ・tan α Ｒ is an equation relating to the amount of the GX-axis component of a vector (hereinafter sometimes referred to as vector GO·71R) that starts from the origin GO and ends at the specular reflection point 71R. Also, the first term on the right side of the equation, R·sin β Ｒ is a term relating to the amount of the GX component of vector E21·71R, i.e., a vector starting from the corneal center E21 and ending at the specular reflection point 71R. Also, the second term Len·tan γ on the right-hand side of the same equation is a term relating to the amount of the GX component of vector GO·E21.

[0159]

[0160] Equation (9-2) indicates that vector GO·71R is equal to the vector sum of vector GO·E21 and vector E21·71R. This will be referred to as the first condition and will be used hereafter. In the example of FIG. 13(A), the left side of (9-2) is a positive value. The first term on the right side of the equation, R·sin β Ｒ is a positive value, and the second term Len·tan γ on the right side is a negative value.

[0161] Also, ms Ｒ１ is Equation (9-3), ms Ｒ２ is expressed as equation (9-4). Furthermore, the distance Dis from the GX axis to the specular reflection point 71R Ｒ can be expressed as equation (9-5). Substituting equation (9-5) into the left side of equation (9-2), Dis Ｒ By eliminating the equation (10-1), we obtain the equation (10-1). Then, by solving the equation (10-1) for Len, we obtain the equation (10-2). Here, the equations (10-1) and (10-2) are equations that formulate the first condition, and the unknowns are Len, R, and β. Ｒ There are three types:

[0162]

[0163] On the other hand, if equations (9-3) and (9-4) are substituted into equation (9-1) and then enclosed with Dis, equation (10-3) is obtained, and if equation (9-5) is substituted into this and Dis is eliminated, equation (10-4) is obtained. The second term on the right side of equation (10-4) contains (α Ｒ +2β Ｒ This is an equation that expresses the physical phenomenon that the angle of incidence and the angle of reflection are equiangular in specular reflection.

[0164]

[0165] <Second Condition> Therefore, the condition defined by formula (10-4) will be referred to as the second condition, and will be used together with the first condition. The second condition is expressed as shown in FIG. 13(B). The left side of formula (10-4) represents the quantity related to the GX axis component of vector GO·infrared light source 55R. The first term on the right side of the formula represents the quantity related to the GX component of vector GO·specular reflection point 71R. The second term on the right side of the formula represents the quantity related to the GX component of vector 71R·55R. In other words, condition 2 defined by formula (10-4) is the condition that "vector GO·55R is equal to the vector sum of vector GO·71R and vector 71R·55R." Here, at specular reflection point 71R, the infrared light ray is irradiated by the normal vector N Ｒ The condition for the beam to be reflected at an equal angle to the beam is included. Here, equation (10-4) is a function of the unknowns Len, R, and β. Ｒ Includes the following three.

[0166] Here, equation (10-2) relating to the first condition and equation (10-4) relating to the second condition are simultaneously solved to eliminate Len, and equation (10-5) is obtained. Ｒ Here are two solutions. One is to substitute the average value for the corneal radius R and find β Ｒ The other is a method of deriving the equations relating to the first condition and the equations relating to the second condition based on the second infrared light source 55L and the specular reflection point 71L caused by the second infrared light source 55L.

[0167] <Average human corneal radius Rａｖｅ The average corneal radius of an adult human is said to be 7.5 mm. For example, if this value is taken as the known value R ａｖｅ Substituting this into equation (10-5), the unknown in this equation is ∠β Ｒ Therefore, we solve the equation to find ∠β Ｒ It can be calculated analytically, but β Ｒ The right side of equation (10-5) is calculated successively for the range of values ​​that can be taken, and β is calculated when the difference with the left side is less than the allowable value. Ｒ The value is β Ｒ It may be determined as the value of ∠β. Ｒ Once this is determined, in equation (10-2) R = R ａｖｅ Since R is also known, the only unknown in (10-2) is Len. Therefore, Len also becomes a known value from (10-2), and the processing in the corneal center depth calculation unit (block A4) is completed.

[0168] <Utilization of the Positional Relationship Between the Second Infrared Light Source and the Second Specular Reflection Point Resulting Therefrom> The second method will be described below. Now, when the second infrared light source 55L and the specular reflection point 71L are formulated in the same manner as the first infrared light source 55R and the specular reflection point 71R, the parameter α Ｌ is a known value, the common parameter γ is a known value, the common parameters Len and R are unknown values, and the parameter β Ｌ is an unknown value. Then, equation (10-2)L corresponding to equation (10-2) is formulated. By eliminating Len from these two equations, equation (33-1) is obtained. Furthermore, the corneal radii that can be matched for the infrared light source 55L and the specular reflection point 71L should both be equal to R, so by reducing these and expressing the tangent in cosine and sine, equation (33-2) is obtained. (33-2) is an equation that formulates the first condition, and the unknown is β Ｒ and β Ｌ There are two types:

[0169]

[0170] Simplifying equation (33-2) using the addition theorem gives equation (33-3), which then passes through (33-4) to obtain β Ｒ Solving for this gives equation (33-5).

[0171]

[0172] On the other hand, solving equation (10-5) including the second condition for the corneal radius R gives equation (34-1).

[0173] Regarding the relationship between the second infrared light source 55L and the second specular reflection point 71L, an equation corresponding to equation (10-5) can be derived, and when this is solved for R, equation (34-2) is obtained. Equation (34-1) and (34-2) are connected with an equal sign to obtain equation (34-3). In equations (34-3) and (33-5), ∠α Ｒ , ∠α Ｌ As already explained with reference to FIG. 12, ms and ∠γ can be calculated by analyzing the Purkinje image of the imaging surface 52. Ｒ and ms Ｌ is the distance from the center of the lens 51 to the infrared light source 55R and the distance from the center of the lens 51 to the infrared light source 55L, and is a known value in design. Ｒ and β Ｌ is an unknown number. Ｒ and ∠α Ｌ Regarding , the positive rotation direction is counterclockwise around the origin GO on the paper. Ｒ and ∠β Ｌ On the paper, the clockwise direction is defined as the positive direction of rotation about the corneal center Cen (E21).

[0174]

[0175] The equation (33-5) derived from only the first condition is the unknown β Ｒ and β Ｌ On the other hand, the equation (34-3) derived by including the second condition also contains the unknown β Ｒ and β Ｌ Therefore, these are simultaneously solved to find the unknown β Ｒ and β Ｌ By solving the above simultaneous equations, β Ｒ and β Ｌ is a known value.

[0176] Then, the known β ＲThe corneal radius R can be calculated by substituting the known β into equation (34-2). Ｌ The corneal radius R can also be calculated by substituting

[0177] Furthermore, the corneal radius R and β calculated in equation (10-2) Ｒ Alternatively, the corneal radius R calculated in equation (10-2) and β Ｌ By applying the above formula, depth information Len can be obtained. In this way, the value of the corneal radius R can be calculated. In this way, being able to measure the corneal radius of a subject will be useful for analyzing the growth of a child's eyeball and other phenomena.

[0178] The key points of the manipulation of the above formulas can be summarized as follows: Either the first Purkinje image 71iR or the second Purkinje image 71iL is selected as the reference Purkinje image, the infrared light source that causes the reference Purkinje image is referred to as the reference infrared light source, and the specular reflection point serving as the object point of the reference Purkinje image is referred to as the reference specular reflection point. The intersection line between the first plane PL1 and the cutting plane CPx coincides with the first straight line LN1, and this first straight line LN1 includes the center of the lens 51 and coincides with the GX axis.

[0179] On the cutting plane CPx, a straight line is provided as the CPZ axis perpendicular to the GX axis at the center GO of the lens 51, thereby setting a two-dimensional coordinate system GX-CPZ with GO as the origin on the cutting plane CPx.

[0180] The radius of curvature R of the cornea E2, Len, and ∠β Ｒ cannot be calculated directly from the group of Purkinje images, and it is necessary to solve an equation derived based on the constraints of specularly reflected infrared light. Therefore, the first and second conditional expressions are derived based on the conditions under which a physical phenomenon occurs in which a light ray emitted from a reference infrared light source is specularly reflected at a reference specular reflection point and then incident on the origin GO, which is the center of the lens 51.

[0181] Parameter ms Ｒ , ∠α Ｒ , Len, R and ∠β ＲApplying trigonometric functions to each component of the G and X axes, Len, R, and β are used as unknowns based on the fact that "vector (origin GO) (reference specular reflection point) is equal to the vector sum of vector (origin GO) (corneal center) and vector (corneal center) (reference reflection point)." Ｒ Formula (10-1) is derived as a formulation of the first condition including the following. This formula compares only the GX axis component, but ∠α Ｒ , ∠γ, ∠β Ｒ By including trigonometric functions with parameters, the positional relationship within the GX-CPZ plane, including not only the GX axis direction but also the CPZ axis direction, is explained.

[0182] Furthermore, the parameter ms Ｒ , ∠γ, Len, R and β Ｒ Applying trigonometric functions to each component of the G and X axes, Len, R, and β are used as unknowns based on the fact that "vector (origin GO) (reference infrared light source) is equal to the vector sum of vector (origin GO) (reference specular reflection point) and vector (reference specular reflection point) (reference infrared light source)." Ｒ The formula (10-4) is derived as a formulation of the second condition including the following: In (10-4), the infrared ray has a normal vector N Ｒ isometric angle (∠α Ｒ +∠β Ｒ ) reflects the physical phenomenon of incidence and reflection, and as a result, the equation (∠α Ｒ +2・∠β Ｒ ) appears.

[0183] Next, the unknown Len is eliminated from the equation (10-2) relating to the first condition and the equation (10-4) relating to the second condition, and R and β are left as unknowns. Ｒ From here on, there are two ways to go.

[0184] The first method involves multiplying the unknown R by the average human corneal radius R. ａｖｅ By applying the above formula (10-5), the unknowns in the formula (10-5) are calculated as β Ｒ This is the only way to solve this.

[0185] The main points of the second method are as follows: The unknowns Len and R are eliminated from the equation [(10-2)] relating to the first condition for the first infrared light source and the equation [(10-2)L] relating to the first condition for the second infrared light source, and the unknown β Ｒ and β Ｌ An equation including these two is derived [Equation (33-5)].

[0186] On the other hand, the unknown R is eliminated from the equation (10-5) or (34-1) containing the second condition for the first infrared light source and the equation (34-2) containing the second condition for the second infrared light source, and the unknown β Ｒ and β Ｌ An equation including these two is derived [Equation (34-3)].

[0187] Therefore, both have two unknowns β Ｒ and β Ｌ Equations (33-5) and (34-3) containing the above equations are simultaneously solved to obtain β Ｒ and β Ｌ and are set as known values. Then, the corneal radius R can be set as a known value from equation (34-1) or (34-2). Using the known R, Len can be calculated from equation (10-2) or (10-2)L. This is the gist. As described above, the specific processing of the corneal center depth calculation unit (block A4) is completed by determining the value of Len.

[0188] Next, a specific example of the pupil center / corneal center calculation unit (block A7) will be shown. Here, the three-dimensional coordinates (C ＧＸ , C ＧＹ , C ＧＺ ) is calculated. However, the three-dimensional coordinates are expressed as (GX component, GY component, GZ component). Len is known from block A4. Len is the distance from the GX axis (which is also the analytical reference line) to the corneal center E21, and is the distance from the GX axis to the corneal center E21 on the cutting plane CPx shown in Figures 11, 12 (B1), (B2), and 13 (C).

[0189] FIG. 13C shows a three-dimensional coordinate system and the coordinates of the corneal center E21, E21(C ＧＸ , C ＧＹ , C ＧＺ In the figure, a quadrangle GO·q２ ・E21・q １ is a plane included in the cutting plane CPx, and the angle Cφ formed by this plane and the GZ-GX plane ＧＸ is known from equation (3-2).

[0190] Also, quadrilateral GO·q ３ ・E21・q ５ is a plane included in the cutting plane CPy, and the angle Cθ formed by this plane and the GY-GZ plane ＧＹ is known from equation (3-1). Furthermore, the length Len between the GX axis and the corneal center E21 measured along the cutting plane CPx is known from the analysis in block A4 above.

[0191] Therefore, trigonometric functions are applied with reference to FIG. ４ ・q ２ Focusing on this, (11-2) gives C ＧＹ is known, and C ＧＺ Furthermore, the triangle GO·q ４ ・q ３ Focusing on this, C ＧＸ becomes known.

[0192]

[0193] Next, referring to FIG. 14, the three-dimensional coordinates Pu(P ＧＸ , P ＧＹ , P ＧＺ ) is shown in Fig. 14. Fig. 14 shows the positional relationship between the corneal center and the pupil center in a three-dimensional coordinate system. As shown in Fig. 14(A), a rectangular parallelepiped can be imagined with the pupil center Pu and the origin GO as vertices. In the figure, in addition to Pu and GO, １ , a ２ , a ３ and a ５ The size of the rectangle is P x 1. ＧＸ and P ＧＺ And the height is P ＧＹ is.

[0194] Light rays diffusely reflected by the iris E3 form an image on the imaging plane 52. Equations (1-1) and (2-1) are applied to the positions of the images of the iris E3 and pupil E31 projected onto the imaging plane 52 to determine the angle formed by the projection plane PRy and the GY-GZ plane. Furthermore, the angle formed by the projection plane PRx and the GZ-GX plane is determined.

[0195] In FIG. 1A, the quadrangle Pu·a １ ・GO・a ２ is a plane included in the projection plane PRx on the GX axis relating to the pupil center Pu as an object point. ｘ , p ｙ ) (not shown), then, from equation (2-1) to equation (12-2), ∠Pφ ＧＸ is calculated. ３ ・GO・a ５ is a plane included in the projection plane PRy of the pupil center Pu as an object point with respect to the GY axis. ＧＹ is calculated.

[0196]

[0197] The known angle Pθ ＧＹ and ∠Pφ ＧＸ Using this, the three-dimensional coordinates of the pupil center Pu (P ＧＸ , P ＧＹ , P ＧＺ ) can be expressed as equations (12-3), (12-4) and (12-5). However, P in equation (12-5) ＧＺ In =t, t is an unknown.

[0198]

[0199] Here, the radius R of the cornea E21 is the measurement value in the processing block A4, or ａｖｅ Then, the relationship between the corneal center E21 and the pupil center Pu can be expressed as equation (13-1). By substituting equations (12-3), (12-4), and (12-5) into this equation, equation (13-2) is obtained. By arranging equation (13-2) in descending order with respect to t, quadratic equation (13-3) for the variable t is obtained. However, coefficients K1, K2, and K3 are as shown below and are known values.

[0200]

[0201] Solving the quadratic equation (13-3) with the solution formula gives equation (13-4), which contains a complex sign in the numerator. See Figure 14(B).

[0202]

[0203] 14B is a cross-sectional view showing the positional relationship between the lens 51 of the eyeball camera 5, the cornea E2, and the pupil center Pu. ０ Through Pu １ The dashed line indicates a straight line extending from the pupil center Pu, which is an object point, to the pupil center image Pui on the image pickup surface 52. As described above, this line segment has an angle ∠Pθ with respect to the GY-GZ plane. ＧＹ The projection plane PRy and the angle ∠Pφ with respect to the GZ-GX plane ＧＸ is the line of intersection with the projection plane PRx.

[0204] The projection line shown by the dashed line and the sphere of the cornea E2 are ０ and Pu １ These intersections are both located at a distance R from the corneal center E21, so they are the solutions to the quadratic equation (13-3). The pupil center Pu to be found here is Pu, which is closer to the origin GO. ０ Therefore, the desired t is expressed by equation (13-5). Now, t is a known value, and through equations (12-3), (12-4), and (12-5), the pupil center Pu (P ＧＸ , P ＧＹ , P ＧＺ ) also become known.

[0205] Next, the first gaze vector Gaze1 (g1 ＧＸ , g1 ＧＹ , g1 ＧＺ ) is calculated. However, the notation of the three-dimensional coordinates is (GX component, GY component, GZ component). This is obtained by equation (14). Since the values ​​on the right side are all known, the first gaze vector Gaze1 can be calculated from this equation.

[0206]

[0207] 14(C) is a diagram showing a first gaze vector Gaze1 on the eyeball E, with the starting point being the corneal center E21 (Cen) and the ending point being the pupil center Pu. The gaze vector calculation unit (block A8) has been described above. The gaze analysis device according to the first embodiment can accurately calculate the gaze vector even if the center of the cornea of ​​the subject is positioned to the left, right, up, or down from the optical axis of the eyeball camera 5.

[0208] <Second embodiment> <Processing block diagram> Next, a second embodiment of a gaze analysis device based on the concept of the present invention will be shown with reference to Fig. 8. In this embodiment, the gaze analysis terminal T1 included in the first embodiment is used as is, so a description of the gaze analysis terminal will be omitted. The processing block diagram of the second embodiment shows the parts that differ from the first embodiment.

[0209] 8, blocks A1 to A8 indicated by double solid or dashed lines are similar to the processing in the first embodiment, and therefore will not be described. Blocks A63, A9, A10, and A11 are processing blocks that are added in the first embodiment.

[0210] The blocks indicated by dashed lines indicate that they may not function in the gaze vector calculation cycle. As mentioned above, this occurs when the Purkinje images are not properly detected in block A2. The dashed arrows indicate that, as mentioned above, a specific block may not function and therefore information may not be transmitted to downstream blocks.

[0211] In a cycle in which the blocks indicated by dashed lines do not function, calculation of the first gaze vector in block A8 is impossible. Therefore, the second embodiment is configured so that calculation of the second gaze vector is possible using block A11 even when this series of blocks temporarily does not function.

[0212] A63 is a two-dimensional rotation center calculation block, which is a two-dimensional processing block. Here, a two-dimensional gaze vector is calculated from the two-dimensional coordinates of the corneal center detected in block A3 and the two-dimensional coordinates of the pupil center detected in block A6. Although the gaze vector is originally a vector in three-dimensional space, it is calculated as a two-dimensional vector projected onto the imaging surface 52 of the camera 5.

[0213] As the eyeball rotates, the gaze vector also changes, but if a straight line is drawn extending from the starting point of the two vectors before and after the change, the two lines will intersect at a single point. This intersection is the point where the center of rotation E11 of the eyeball E is projected onto the image capture surface 52, and will be referred to as the rotation center projection point E11i. In this way, a drawing process is performed to determine the intersection of the two lines that match the directions of the gaze vector before and after the direction change. Because this process is a numerical calculation in memory, there is no problem even if the line extends beyond the image capture surface 52.

[0214] The coordinate values ​​of the rotation center projection point E11i calculated here are sent to the next block A9. As mentioned above, if block A2 does not function, the two-dimensional coordinates of the corneal center cannot be obtained from block A3. In that case, a two-dimensional line of sight vector cannot be created, and a straight line related to this vector cannot be drawn. Therefore, the rotation center projection point E11i of the rotation center E11 of the eyeball E cannot be calculated.

[0215] Block A9 is a three-dimensional center of rotation / radius of rotation calculation unit, which is a three-dimensional processing block. If possible, it obtains the two-dimensional coordinate of the center of rotation projection point E11i by block A63, and the three-dimensional coordinate of the corneal center and the three-dimensional coordinate of the pupil center by block A7. If necessary, it obtains the two-dimensional coordinate of the pupil center from block A6.

[0216] Here, the three-dimensional coordinate of the center of rotation is calculated based on information obtained from blocks A63 and A7. However, in cycles in which block A2 does not function, blocks A63 and A7 also do not function, and the three-dimensional coordinate of the center of rotation cannot be calculated. To prepare for such cases, block A9 stores the three-dimensional coordinate of the center of rotation calculated in the previous cycle and updates it each time a new calculation is made. Unlike the position of the corneal center, the position of the center of rotation does not change much with the rotation of the eyeball E, so this is handled in this way.

[0217] Furthermore, if the three-dimensional coordinates of the corneal center and pupil center cannot be obtained from block A7, the three-dimensional coordinate of the pupil center is calculated using the stored three-dimensional coordinate of the center of gyration and radius of gyration, and the two-dimensional coordinate of the pupil center derived from block A6. This completes the information processing in block A9.

[0218] Block A10 is second gaze vector calculation unit 1, a three-dimensional processing block. Here, a second gaze vector is calculated, starting from the rotation center coordinates calculated by block A9 and ending at the pupil center coordinates calculated by block A7. However, if block A7 does not function, the pupil center coordinates calculated by block A7 cannot be obtained. In such a case, the second gaze vector is calculated by second gaze vector calculation unit 2 in block A11.

[0219] Block A11 is the second gaze vector calculation unit 2, a three-dimensional processing block. Here, a second gaze vector is calculated, starting from the center of rotation coordinates stored in block A9 and ending at the pupil center coordinates. The pupil center is found by solving a quadratic equation using the two-dimensional pupil center coordinates derived from block A6 and the radius of rotation stored in block A9. This process allows the second gaze vector to be calculated even if block A7 is not functioning.

[0220] The line of sight vectors of blocks A10 and A11 are referred to as "second line of sight vectors" to distinguish them from the first line of sight vector of block A8. In principle, the second line of sight vector and the first line of sight vector have the same direction and orientation, and the second line of sight vector replaces the first line of sight vector.

[0221] Next, a specific example of block A63 will be shown. Fig. 15 is a diagram showing an example in which the eyeball E rotates and the gaze changes up and down. As shown in Fig. 15(A), gaze vector Gaze1 is expressed as a vector in three-dimensional space with the center of the cornea of ​​eyeball E as the starting point and the center of the pupil as the ending point. As the eyeball E rotates, the direction of gaze vector Gaze1 also changes.

[0222] 1B is a conceptual diagram showing the overlapping display of changes in the line of sight. The starting points of multiple line of sight vectors Gaze1 do not coincide. This indicates that the corneal center E21, which is the starting point of Gaze1, does not coincide with the center of rotation E11 of the eyeball E.

[0223] To grasp the movement of gaze vector Gaze1 on imaging plane 52, a two-dimensional gaze vector is plotted with the two-dimensional coordinates of the cornea center obtained in block A3 as the start point and the two-dimensional coordinates of the pupil center obtained in block A6 as the end point. This two-dimensional gaze vector is referred to as Gaze1_i (Fig. 1C).

[0224] This Gaze1_i corresponds to the gaze vector Gaze1 projected onto the imaging surface 52. In FIG. 2C, the image is rotated 180 degrees to match FIGS. 2A and 2B. That is, the upward direction of the paper in FIG. 2C is the positive direction of the y-axis. Images other than the projected image E31i of the pupil E31 are omitted.

[0225] Here, a straight line Gaze1_i_ex is drawn that is in the same direction as the projected gaze vector Gaze1_i and passes through the corneal center projection point E21i, which is the starting point of the vector. Hereinafter, this straight line will also be referred to as the two-dimensional extended straight line Gaze1_i_ex (FIG. 1C).

[0226] When the eyeball E turns and the direction of the gaze vector Gaze1 changes, the directions of the two-dimensional gaze vector Gaze1_i and the two-dimensional extended straight line Gaze1_i_ex also change. The two-dimensional extended straight lines Gaze1_i_ex before and after the turn intersect at approximately one point.

[0227] This point of intersection is the point where the center of rotation E11 of the eyeball E is projected onto the imaging surface 52, and is referred to as the center of rotation projection point E11i. However, this center of rotation projection point E11i is not an image formed by projecting the corresponding part of the eyeball in three-dimensional space, but is a calculated point, so it may extend beyond the imaging surface 52.

[0228] Whenever the eyeball E turns and the gaze vector Gaze1 changes, a two-dimensional extension line Gaze1_i_ex is drawn based on the two-dimensional gaze vector Gaze1_i, and the intersection of the two-dimensional extension lines Gaze1_i_ex calculated in the preprocessing cycle can be calculated. The two-dimensional coordinates of the turning center projection point E11i are updated with the coordinates of this intersection.

[0229] However, when the subject's eyeball E hardly rotates, for example, because the subject is gazing at a single point, two-dimensional gaze vectors Gaze1_i in almost the same direction may be calculated one after another. In this way, the position of the intersection between two-dimensional extension lines Gaze1_i_ex that are nearly parallel to each other tends to fluctuate and become unstable. Therefore, if the angle of the newly obtained two-dimensional gaze vector Gaze1_i differs from that obtained in the previous cycle by, for example, 5° or more, the turning center projection point E11i can be updated with the coordinates of the intersection between the two two-dimensional extension lines Gaze1_i_ex. The above is a specific example of the processing of block A63.

[0230] Next, a specific example of the three-dimensional processing in block A9 is shown. The coordinates of the turning center projection point E11i are already known by the preceding block A63. As shown with reference to FIG. 15C, if the angle between the two most recently obtained two-dimensional extension lines Gaze1_i_ex is equal to or greater than a predetermined angle, the intersection of these two lines is set as the turning center projection point E11i.

[0231] Next, a method for calculating the three-dimensional coordinates of the center of rotation will be described. FIG. 16 is a diagram showing the process for obtaining the three-dimensional coordinates of the center of rotation Ro (E11) and the pupil center Pu. Through the above process, the three-dimensional coordinates of the pupil center Pu (P ＧＸ , P ＧＹ , P ＧＺ ) and the three-dimensional coordinates of the corneal center Cen (C ＧＸ , C ＧＹ, C ＧＺ ) are known. Accordingly, the pupil center position vector Pu, the corneal center position vector Cen, and the rotation center position vector Ro, which start from the origin GO, can be expressed as equations (15-1), (15-2), and (15-3), respectively. Furthermore, the first gaze vector Gaze1 (g1 ＧＸ , g1 ＧＹ , g1 ＧＺ ) Equation (15-4) holds for the turning center position vector Ro(r ＧＸ , r ＧＹ , r ＧＺ ) is unknown.

[0232]

[0233] Pupil center projection coordinates Pui(p ｘ , p ｙ ) and the corneal central projection point E21i (C ｘ , C ｙ As mentioned above, the rotation center projection coordinates Roi (roi ｘ , roi ｙ ) is also known from computational construction.

[0234] Here, the rotation center projection coordinate Roi (roi ｘ , roi ｙ The angle between the projection plane PRy and the GY-GZ plane is defined as ∠Roθ. ＧＹ The angle between the projection plane PRx and the GZ-GX plane is ∠Roφ ＧＸ Applying equations (1-1) and (2-1) to these, we get equations (16-1) and (16-2), respectively. The angle ∠Roθ derived here is ＧＹ and ∠Roφ ＧＸ is an index that represents the direction of projection from the origin GO of the three-dimensional coordinate system onto the imaging surface 52. At the same time, it is also an index that represents the direction of projection from the origin GO to the three-dimensional coordinate Ro(r ＧＸ , r ＧＹ , r ＧＺ ) is also an indicator of the direction of

[0235]

[0236] And, ∠Roθ ＧＹ and ∠Roφ ＧＸ and the three-dimensional coordinates Ro(r ＧＸ , r ＧＹ , r ＧＺ ), equations (16-3), (16-4), and (16-5) hold. These are equations for a line that passes through the origin GO and the center of rotation Ro in the GX, GY, GZ coordinate space. However, m is an unknown quantity.

[0237]

[0238] Now, let us refer to Figure 16 (A) and the partially enlarged view (A-α). Since the vector starting at Cen and ending at Ro and the vector Gaze1 are in the same direction but opposite directions, this can be formulated as equation (17-1) using the unknown n. This is the equation of a line passing through the corneal center Cen, the center of rotation Ro, and the pupil center Pu in the GX-GY-GZ coordinate space. By expressing this in each component as equation (17-2), the unknowns m and n can be solved to become known quantities as in (18-1) and (18-2).

[0239]

[0240]

[0241] Applying the known number m to equations (16-3), (16-4), and (16-5), the three-dimensional coordinate of the turning center Ro(r ＧＸ , r ＧＹ , r ＧＺ ) is known. Next, the three-dimensional coordinate Ro(r ＧＸ , r ＧＹ , r ＧＺ ) and the distance Rot_R between the pupil center Pu and the center of rotation E11 are calculated and saved. This is a measure to prepare for the case where the acquisition of a Purkinje image is not working properly and blocks A2, A3, A4, A7, and A8 do not function. However, this processing is limited to cycles in which blocks A2, A3, A4, and A7 are functioning.

[0242] If the above block does not function, the three-dimensional coordinate Ro(rＧＸ , r ＧＹ , r ＧＺ ) and the distance Rot_R between the pupil center Pu and the center of rotation E11, and then the line of sight vector is calculated. The square of Rot_R is obtained by equation (19). The distance Rot_R is the radius of rotation as mentioned above, and is a fixed value. A specific example of processing block A9 has been shown above.

[0243]

[0244] The previously shown Gaze1 is a gaze vector that starts at the coordinate Cen of the corneal center E21 and ends at the pupil center Pu. In contrast, the gaze vector shown below starts at the coordinate Ro of the center of rotation E11 and ends at the pupil center Pu. Hereafter, this gaze vector will be referred to as Gaze2.

[0245] In block A10, the three-dimensional coordinate Ro (r ＧＸ , r ＧＹ , r ＧＺ ) as the starting point, and the three-dimensional coordinates Pu(P ＧＸ , P ＧＹ , P ＧＺ ) as the end point of the gaze vector Gaze2 (g2 ＧＸ , g2 ＧＹ , g2 ＧＺ ) is calculated. FIG. 16B is a diagram showing an example of the second gaze vector Gaze2 defined between the center of rotation Ro (E11) and the pupil center Pu. The pupil center three-dimensional coordinate Pu (P ＧＸ , P ＧＹ , P ＧＺ ) and the three-dimensional coordinate Ro(r ＧＸ , r ＧＹ , r ＧＺ ) to the second gaze vector Gaze2 (g2 ＧＸ , g2 ＧＹ , g2 ＧＺ ) to be determined.

[0246] Second gaze vector Gaze2 (g2 ＧＸ , g2 ＧＹ , g2 ＧＺ) are calculated as equations (20-1), (20-2), and (20-5). Here, the three-dimensional coordinates of the pupil center and the center of rotation are expressed by equations (20-3) and (20-4), respectively.

[0247]

[0248]

[0249] In block A11, a vector having the three-dimensional coordinates of the center of rotation E11 as a starting point and the three-dimensional coordinates of the pupil center Pu as an end point is defined as gaze vector Gaze2 (g2 ＧＸ , g2 ＧＹ , g2 ＧＺ ) which is also calculated using equation (20-1). Block A11 needs to function when block A7 does not function, and in this case the three-dimensional coordinates of the pupil center Pu cannot be obtained. Therefore, block A11 uses the three-dimensional coordinates Ro of the turning center calculated and saved in the previous cycle and the turning radius Rot_R which is the distance from the turning center E11 to the pupil center Pu.

[0250] Furthermore, the two-dimensional coordinates P ui (p ｘ , p ｙ ) is used to calculate the direction of the projection line from the origin GO of the three-dimensional coordinate system to the pupil center Pu. When applying the equations (1-1) and (2-1) in the calculation, the above-mentioned equations (12-1) and (12-2) are used to obtain ∠Pθ ＧＹ and ∠Pφ ＧＸ is calculated.

[0251] As already mentioned, using these angles, the three-dimensional coordinates of the pupil center Pu can be expressed by equations (12-3), (12-4), and (12-5), where t is an unknown quantity.

[0252] Now, let us refer to FIG. 16(C). This figure is a cross-sectional view showing the positional relationship between the lens 51 of the eyeball camera 5, the pupil center Pu, and the rotation center Ro. With reference to this figure, we create a quadratic equation (21-1) for the square of the rotation radius Rot_R. Then, we calculate the coordinates of the pupil center Pu (P ＧＸ , P ＧＹ , P ＧＺ), we obtain quadratic equation (21-2) for unknown t. Rearranging this in descending order with respect to t gives quadratic equation (21-3). Here, Q1, Q2, and Q3 are expressed by equation (21-4), and since the right-hand sides of these are known values, we obtain equation (21-5) using the solution formula. However, this solution contains a complex sign.

[0253]

[0254]

[0255] There are two solutions to the quadratic equation (21-3), one of which is Pu １ This is the solution corresponding to (Fig. 16(C)). Therefore, the Pu with the smaller GZ coordinate ０ Using the solution and known value of t, the three-dimensional coordinates of the pupil center Pu ０ (P ＧＸ , P ＧＹ , P ＧＺ ) to obtain equation (21-7).

[0256]

[0257] Thereafter, as in block A10, the second gaze vector Gaze2 (g2 ＧＸ , g2 ＧＹ , g2 ＧＺ ) is obtained. The second gaze vector calculation unit 2 (block A11) has been described above, and the description of the processing block diagram according to the second embodiment is completed.

[0258] Next, a series of operations of the gaze analysis device of the first and second embodiments based on the concept of the present invention is shown in a flowchart. FIG. 17 is a flowchart of the gaze analysis device based on the concept of the present invention. The gaze analysis device begins operation from "Start" and sets initial values ​​(stp1). Here, various parameters necessary for the operation of the gaze analysis device are set. One of these is the R_F flag, which is composed of at least one bit of memory and switches between two states, set and reset, and is referenced during the series of processes that follow.

[0259] The R_F flag is set when a valid three-dimensional coordinate Ro of the center of rotation has been saved or calculated, but is reset when it is determined that the position indicated by the three-dimensional coordinate has moved significantly and the positional relationship between the eye camera 5 and the eye E has changed.

[0260] As described above, unlike the corneal center Cen, the position of the center of rotation Ro does not change much with the rotation of the eyeball E. The position changes when the subject moves the gaze analysis terminal T, for example, and the positional relationship between the eyeball E and the eye camera 5 changes. In such a case, the R_F flag is reset. Thereafter, when the valid three-dimensional coordinate Ro of the center of rotation is calculated and updated, the R_F flag is set.

[0261] The validity of the three-dimensional rotation center coordinate Ro is determined, for example, as follows: If the coordinate of the rotation center Ro is calculated with a difference within the allowable error despite the rotation of the eyeball E, it is determined to be valid and the R_F flag is set. If the calculated coordinate exceeds the error range, the R_F flag is reset. The rotation of the eyeball E is determined by the movement of the pupil center. In other words, if the three-dimensional rotation center coordinate Ro shows approximately the same coordinate value despite the movement of the pupil center, it can be determined to be a valid three-dimensional rotation center coordinate. However, since the three-dimensional rotation center coordinate Ro is not used in the first embodiment, the R_F flag is not necessary.

[0262] Next, an eyeball image is acquired by the eyeball camera 5 and a field of view image is acquired by the field of view camera 4 (stp2). This is the image acquisition step. However, the eyeball image and the field of view image do not necessarily have to be acquired at the same time. For example, the timing may be such that the eyeball image is captured several times and then one field of view image is acquired.

[0263] The field of view image is an image captured by the field of view camera 4, and captures a portion of the scene unfolding in front of the subject's eyes. Both the eyeball image and the field of view image are stored in the image area 622 of the storage unit 62, and are processed by the calculation unit 61 as necessary. However, the eyeball image may be processed by the calculation unit 61 directly after acquisition.

[0264] Next, the acquired eyeball image is processed to detect the pupil and its center point Pu. This is the pupil center detection step in two-dimensional processing. If the pupil cannot be detected, the result is 'false' and the process returns to step 2 to acquire another eyeball image. On the other hand, if the pupil can be detected, the result is 'true' and the process proceeds to step 4 (step 3).

[0265] In stp4, the eyeball image is processed to detect Purkinje images. A Purkinje image is a collection of Purkinje images as a specific polygon formed by three or more infrared light sources. This is the Purkinje image detection step. If the Purkinje image cannot be detected, the result is 'false' and the state of the R_F flag is referenced (stp7). On the other hand, if the Purkinje image can be detected, the result is 'true' and the process proceeds to stp5.

[0266] In stp7, if R_F is set, it is determined as 'true' and the second line of sight vector is calculated (stp15), and the process proceeds to stp 16. On the other hand, if R_F is reset, it is determined as 'false' and the process returns to st2 to acquire an eyeball image again.

[0267] In step 5, the two-dimensional coordinates of the corneal center are calculated from the arrangement of the detected Purkinje images. This is the step of calculating the two-dimensional coordinates of the corneal center projection point.

[0268] Next, when solving an equation created based on the positional relationship between the first infrared light source placed on a first line that passes through the center of the eye camera lens and is perpendicular to the optical axis of the lens, and the first specular reflection point and the corneal center caused by this, the average human corneal radius is applied to the unknown corneal radius R to calculate the distance Len from the first line to the corneal center (stp6).

[0269] Alternatively, the corneal radius R is determined by simultaneously solving the equation relating to the positional relationship with the first infrared light source and a second equation created based on the positional relationship between the second infrared light source installed on the first line and the second specular reflection point and the corneal center caused by the second infrared light source, and then the distance Len from the first line to the corneal center is calculated (step 6). The Len is corneal center depth information for obtaining the three-dimensional coordinates of the corneal center.

[0270] The two-dimensional coordinates of the pupil center obtained in step 3 and the two-dimensional coordinates of the cornea center obtained in step 5 are converted into three-dimensional coordinates based on the distance Len calculated in step 6, and three-dimensional coordinate values ​​are calculated (step 8). This is the pupil center / cornea center three-dimensional coordinate calculation step.

[0271] This is a step of calculating a first gaze vector (stp9) that starts from the corneal center calculated in stp8 and ends at the pupil center Pu calculated in stp8. In the first embodiment, steps 7, 15, 10, 11, 12, 13, 14, and 16 are unnecessary. The position of the viewpoint gazed by the subject is updated using the first gaze vector calculated in stp9 (stp17).

[0272] On the other hand, in the second embodiment, the three-dimensional coordinates of the turning center Ro and the turning radius are calculated by the method already described, and these values ​​are updated and stored (stp10).

[0273] The updated and stored three-dimensional coordinates of the turning center Ro are compared with the previously described turning radius, and if they are within an allowable error range, the three-dimensional coordinates of the turning center are determined to be valid (stp11). In this case, this is set to 'true', R_F is set (stp12), and a second line-of-sight vector is calculated based on these values ​​(stp14), and the process proceeds to stp16.

[0274] On the other hand, if the error is outside the allowable range, it is judged as 'false', R_F is reset (stp13), and the process proceeds to stp16.

[0275] In stp 16, the line-of-sight vector to be output is selected from the first line-of-sight vector or the second line-of-sight vector, and the process proceeds to stp 17. As a method for selecting the line-of-sight vector, the second line-of-sight vector may always be selected, or the second line-of-sight vector may be selected only when the R_F flag is set. Alternatively, the first line-of-sight vector may always be selected when the conditional branch in stp 4 is 'true'. In this way, the selection conditions may be set in advance.

[0276] In step 17, the subject's gaze point is updated and displayed in the image from the field of view camera 4 in accordance with the selected line-of-sight vector.

[0277] In stp18, if there is a stop or power-off interrupt, the result is 'true' and the series of gaze analysis operations is terminated. On the other hand, if there is no stop or power-off interrupt, the result is 'false' and the process proceeds to stp2, after which stp2 to stp17 are repeated until stp18 becomes true. Hereinafter, the series of operations from stp2 to stp17 may be referred to as a gaze vector calculation cycle.

[0278] It should be noted that the completion can be more reliably determined by providing an end determination routine such as stp18 between stp2 and stp3. The above is the flow of the operation of the line-of-sight analysis device based on the concept of the present invention.

[0279] Next, an application example of the gaze vector analysis device will be described. Fig. 18 is a diagram showing an application example of the gaze vector. Referring to this figure, the relationship between the first gaze vector Gaze1 or the second gaze vector Gaze2 and the field of view camera 4 is shown. In the first or second embodiment, the subject's gaze point GP is superimposed on the image recorded by the field of view camera 4.

[0280] FIG. 1A shows subject A placing a gaze point GP on a product in a vending machine. However, in reality, the gaze point GP is only superimposed on image 45 (enlarged view η) captured by field of view camera 4. At this time, field of view camera 4 is capturing and recording the scene in front of the subject. The subject's gaze point GP is superimposed on recorded image 45. The gaze point GP may be superimposed directly on the image recorded by the field of view camera, or, when the image is played back, a gaze point mark may be superimposed on the relevant location on the frame of the recorded image at the relevant time.

[0281] <Third Embodiment> Next, a third embodiment of a gaze analysis device based on the concept of the present invention will be described. Fig. 18(B) is a diagram showing the configuration of the third embodiment of a gaze analysis device based on the concept of the present invention. As shown in the figure, the third embodiment includes a gaze analysis terminal T3 equipped with one or more eyeball cameras 5Q and an information display board 47, and a control box 6Q that supplies power to the gaze analysis terminal T3, displays information such as announcements, advertisements, and news on the information display board 47, and processes images of the viewer's eyes acquired via the eyeball camera 5Q.

[0282] The hardware configuration of the control box 6Q is almost the same as that of the control box 6 according to the first embodiment, except that a program for extracting the viewer's face area from an eyeball image or an image acquired by a separately provided face detection camera, a program for cutting out the eyeball image from the image area, and a program or hardware for controlling the information display board 47 are added.

[0283] The figure also shows an enlarged view of the eyeball camera 5Q as seen from the viewer. A lens 51 has an optical axis that intersects at a right angle with a line segment connecting the centers of the infrared light sources 55R and 55L provided on both the left and right sides, and a third infrared light source 55B is provided below or above the lens 51.

[0284] Eye camera 5Q uses three infrared light sources 55R, 55L, and 55B to acquire eye images along with three Purkinje images. These images are processed by a calculation unit in control box 6Q. However, only eye images in which all three Purkinje images appear are processed to determine the corneal center and calculate a first gaze vector Gaze1 with this as the start point and the pupil center as the end point. Alternatively, the center of rotation of the eye is further determined and a second gaze vector Gaze2 is calculated with this as the start point and the pupil center as the end point.

[0285] From the viewpoint position and direction of the calculated first gaze vector or second gaze vector, the gaze direction is extended to the surface of the information display board 47, and the intersection of the gaze extension line and the surface of the information display board 47 is set as the gaze point, thereby making it possible to create a heat map in the form of a map that indicates the viewer's gaze frequency and total gaze time using shades of gray. As shown in Figure 1B, by providing multiple eyeball cameras 5Q above and below the information display board 47, it is possible to detect the gaze points of multiple viewers and to acquire and analyze eyeball images of tall and short people. The configuration and effects of the third embodiment have been described above.

[0286] <Superimposition of gaze point> Next, the relationship between the gaze vector representing the gaze direction and the angle of view of the field of view camera 4 will be shown. Fig. 19 is a diagram showing the spatial relationship between the second gaze vector Gaze2 and the imaging system of the field of view camera 4. Fig. 19(A) shows the second gaze vector Gaze2 starting from the center of rotation E11 (Ro), a three-dimensional coordinate system GX, GY, GZ (the GX axis is not visible at this angle of view and is therefore not shown) that defines the direction of Gaze2, and a three-dimensional coordinate system SX, SY, SZ (the SX axis is not visible at this angle of view and is therefore not shown) that defines the imaging system of the field of view camera 4.

[0287] The field of view camera 4 has an imaging lens 41 and an imaging surface 42 (enlarged view λ). A three-dimensional coordinate system SX-SY-SZ is set with the center of the lens 41 as the origin SO and the SZ axis as the lens optical axis. The field of view camera 4 has a wide-angle shooting range 44, which covers almost the entire range that the subject can gaze at by rotating their eyes.

[0288] Here, we consider a virtual screen 46 that is perpendicular to the SZ axis. The distance from the origin SO to the virtual screen 46 is VSL. VSL can be varied depending on the application; for example, it may be set to about 1 meter when the subject is gazing at a nearby object, 5 meters in a large room such as an exhibition hall, or infinity outdoors.

[0289] Although not shown, a distance sensor may be provided near the field of view camera 4, and the distance sensor may be adjusted to a direction substantially parallel to the lens optical axis of the field of view camera 4. In such a configuration, if the distance sensor can measure the distance to a wall or object in the gaze direction, the distance VSL can be automatically set based on that distance. A gaze analysis device equipped with such a mechanism can automatically set the distance to the gaze target.

[0290] First, the second gaze vector Gaze2 is extended from its end point while maintaining its direction, and the point of intersection with the virtual screen 46 is found. If the subject is gazing at a distance VSL, this intersection is the gaze point GP. Then, the projection point of the gaze point GP onto the imaging surface 42 of the field of view camera 4 is found, and this projection point becomes the position where the gaze point should be superimposed. A gaze point display can be written directly at that position, or a database can be created that records that a gaze point display should be superimposed at the appropriate position in the appropriate frame.

[0291] Up until now, the second gaze vector Gaze2 and the three-dimensional coordinates of its starting point, the center of rotation E11 (Ro), have been expressed in the GX-GY-GZ coordinate system with origin GO at the center of lens 51 of eyeball camera 5. These must be converted into the SX-SY-SZ coordinate system with origin SO at the center of lens 41 of field of view camera 4.

[0292] Here, it is assumed that the second line of sight vector Gaze2 and the three-dimensional coordinates of the turning center E11 (Ro), which is the starting point of the second line of sight vector Gaze2, are already expressed in the SX-SY-SZ coordinate system. ＳＸ , g2 ＳＹ , g2 ＳＺ ) [Equation (22-1)], the center of rotation is Ro (r ＳＸ , r ＳＹ , r ＳＺ ) [Equation (22-2)].

[0293]

[0294] Furthermore, the equation of the line passing through the turning center E11 (Ro) and extending in the direction of the second gaze vector Gaze2 is given by equation (22-3). ｓ is an unknown, (v ＳＸ , v ＳＹ , v ＳＺ ) is a variable that represents the position on the line. The point where this line intersects with the virtual screen 46 is the gaze point GP, and the virtual screen 46 is v ＳＺ =VSL plane. Therefore, v ＳＺ Substituting the value of VSL into equation (22-3) gives k ｓ The value of k is obtained as equation (22-4). ｓis a known value.

[0295]

[0296] Then, the three components SX, SY, and SZ as the coordinates of the gaze point GP can be obtained by the formula (22-5). ＳＸ , gp ＳＹ , gp ＳＺ ), the coordinates of the GP can be expressed by equation (22-6).

[0297]

[0298] In order to superimpose a gaze mark at the position corresponding to GP on the image captured by the field of view camera 4, it is necessary to calculate where the gaze point GP in the three-dimensional space is projected on the image capturing surface 42. The projection line onto the image capturing surface is a straight line that passes through GP and the origin SO and reaches the image capturing surface 42, and is given by equation (23-1). However, p ｓ is an unknown, (v2 ＳＸ , v2 ＳＹ , v2 ＳＺ ) is a variable. However, the image plane 42 is v2 ＳＺ =-SD Ｓ Since this is a plane, substituting this into equation (23-1), p ｓ is obtained as equation (23-2) and is a known value.

[0299]

[0300] p ｓ Once these are known, the three-dimensional coordinates of the projection point of the gaze point GP onto the imaging surface 42 can be obtained by equation (23-3). Since the SX and SY components form a two-dimensional coordinate system within the imaging surface 42, the two-dimensional coordinates of the projection point GPi of the gaze point are given by equation (23-4). Therefore, it is sufficient to superimpose the gaze point mark at this position within the imaging surface 42. However, since equation (23-4) is expressed in the SX-SY-SZ coordinate system, the superimposition position may be managed after converting it into an expression of a two-dimensional coordinate system provided within the imaging surface 42.

[0301]

[0302] Here, when the subject is gazing at infinity, the difference between the position of the rotation center E11 (Ro), which is the starting point of the second line of sight vector, and the lens center SO of the field of view camera 4 can be ignored. Therefore, in equation (23-4), r ＳＸ =r ＳＹ = 0, and k as in equation (23-5) ｓ is also simplified to Equation (23-7) via Equation (23-6).

[0303]

[0304]

[0305] Equation (23-7) means that the origin GO of the eyeball camera coordinate system coincides with the field of view camera coordinate system SO, and the direction of the straight line from the gaze point at infinity through SO to the image pickup surface 42 is the same as the direction of the second gaze vector Gaze2. To draw this in Figure 19(A), the straight line of Gaze2 starting from Ro is translated to the origin SO of the lens 41. The intersection of this line and the image pickup surface 42 is the position where the gaze mark should be superimposed.

[0306] Next, an example of conversion from the GX-GY-GZ coordinate system to the SX-SY-SZ coordinate system is shown. This is a process for converting the attitude of the imaging systems of the field of view camera 4 and the eyeball camera 5. Figure 1B shows the center of rotation Ro, which is the starting point of the line of sight vector, as well as the GX-GY-GZ coordinate system and the SX-SY-SZ coordinate system. Both of these imaging systems are built into the line of sight analysis terminal and are fixed at the time of design and manufacturing. Therefore, it is sufficient to build a conversion processing program, for example, into the terminal in advance based on the design values.

[0307] First, a translational transformation is performed to align the origins of both coordinate systems. Figure 1(D) shows the GX(1), GY(1), GZ(1) coordinate system and the SX, SY, SZ coordinate system after the translational movement. The notation (1) is added to the coordinate system after the movement to distinguish between the coordinate systems before and after the translational movement. The origin GO(1) after the movement coincides with the origin SO of the SX, SY, SZ coordinate system. After that, a rotational transformation is performed to align the GZ axis with the SZ axis, and the GX axis with the SX axis. The remaining GY and SY axes naturally coincide.

[0308] The expression of the turning center Ro in GX, GY, and GZ coordinates is Ro(r ＧＸ , r ＧＹ , r ＧＺ ), and the expression in the SX, SY, and SZ coordinate system is Ro(r ＳＸ , r ＳＹ , r ＳＺ ) and SO (SO ＧＸ , SO ＧＹ , SO ＧＺ ) represents the position of the origin SO of the SX-SY-SZ coordinate system in the GX-GY-GZ coordinate system.

[0309] FIG. 1C shows the positional relationship between the turning center Ro, the GX, GY, and GZ coordinate axes with GO as the origin, and the SX, SY, and SZ coordinate axes with SO as the origin. In the figure, the vector Ro(r ＧＸ , r ＧＹ , r ＧＺ ), a vector SO(SO ＧＸ , SO ＧＹ , SO ＧＺ ), the vector Ro(1)(r ＧＸ（１） , r ＧＹ（１） , r ＧＺ（１） ) is assumed.

[0310] Then, the vector Ro going directly from GO to Ro and the vector sum route going from GO to Ro via SO are connected with an equal sign to give equation (24-1). Solving this for vector Ro(1) gives equation (24-2). From equation (24-2), vector Ro(1)(r ＧＸ（１） , r ＧＹ（１） , r ＧＺ（１） ) is calculated using equation (24-3).

[0311]

[0312] The coordinate system GX(1), CY(1), GZ(1) derived by the formula (24-3) has the same axis direction as the GX, GY, GZ coordinate system, but its origin GO(1) coincides with the origin SO of the SX, SY, SZ coordinate system. After the origins of both coordinate systems are matched, an appropriate rotation transformation can be performed to match the axes of the GX(1), CY(1), GZ(1) coordinate system with those of the SX, SY, SZ coordinate system.

[0313] The above shows an example of superimposing a gaze point onto a field of view image. In addition to this example, during the initial adjustment of the gaze analysis terminal, it is also possible to have the subject gaze at several specified locations, link the direction and orientation of the gaze vector at that time to the specified locations, and interpolate the intermediate areas.

[0314] Incidentally, Non-Patent Document 2 states that there is a slight angular difference between the direction of the line connecting the center of corneal curvature and the center of the pupil, i.e., the optical axis direction of the eyeball's optical machinery, and the direction in which the subject is gazing, i.e., the gaze direction. This difference is called the κ angle, and its magnitude varies from person to person. This is because the fovea, a tissue on the retina with the highest visual resolution, is located slightly off the optical axis. Therefore, in order to determine the direction in which the subject is gazing, i.e., the gaze direction (also called the visual axis direction), the optical axis direction calculated from the corneal shape can be corrected by the κ angle as an individual parameter.

[0315] To obtain these personal parameters, the subject can be made to perform a calibration operation. The difference in the kappa angle from the optical line of sight becomes significant at close range. Therefore, the gaze analysis device based on the concept of the present invention is provided with a method for performing simple calibration in the vicinity to obtain the above-mentioned personal parameters.

[0316] Returning to FIG. 18 (C), this figure shows an example of a subject wearing a gaze analysis terminal performing a one-point calibration operation, with the subject gazing at the AR marker 8. This is the scene seen by the subject's eyes, and is an image that can also be recorded by a field of view camera. However, the subject is not looking at any information display board or the like; all they can see is their own fingertip and the AR marker 8. In other words, the gaze point marker GP indicating the gaze point and the upward and rightward arrows on the AR marker are merely shown for the convenience of explanation.

[0317] However, on the calculation unit 61 associated with the gaze analysis device, the calculated gaze point GP is plotted on the virtual screen 46, the center of the AR marker 8 at which the subject is supposed to be gazing is identified, and the amount of positional deviation between this center and the gaze point GP is calculated. The gaze point GP is based on the coordinates calculated by the above-mentioned formula (23-4). The amount of correction at this time may be stored in the storage unit 62 as a personal parameter for the subject, and may be applied to correction during gaze analysis. The above is an example of one-point calibration.

[0318] <Fourth Embodiment> Next, a fourth embodiment of a gaze analysis device based on the concept of the present invention will be described. The hardware of the fourth embodiment is composed of a gaze analysis terminal T4 and a control box 6 (not shown). The gaze analysis terminal T4 has a feature that differs from the first to third embodiments in the arrangement of the infrared light source provided in the eye camera. Figures 20 (B1), (B2), (D1) to (E2) are diagrams showing the arrangement of the infrared light source provided in the eye camera 5 in the fourth embodiment of the gaze analysis device based on the concept of the present invention. Figures (A1) and (A2) show the positions of the infrared light rays already described.

[0319] Figure (A1) shows the arrangement of the infrared light sources 55R, 55L, and 55B of the gaze analysis terminals T1 and T3, along with the three-dimensional coordinate system GX, GY, and GZ. As mentioned above, the origin GO of the three-dimensional coordinate system is located at the center of the lens 51, where the three axes GX, GY, and GZ intersect (the GZ axis is not shown). The GZ axis, which is the optical axis of the lens 51, passes through the intersection of the GX and GY axes and points toward the back of the page. Although not shown, the eyeball, which is the subject of the image, is located at the back of the page and faces toward the front of the page.

[0320] As mentioned above, infrared light sources 55R and 55L are arranged on the same straight line passing through GO. This straight line is called the first straight line, and in the example mentioned above, it coincides with the GX axis. The foot of a perpendicular line drawn from infrared light source 55B to this first straight line coincides with origin GO, which is the center of lens 51.

[0321] 1A2 is a diagram of the cornea irradiated with infrared light from the three infrared light sources arranged as described above, producing three specular reflections. The illustration shows the iris E3 and pupil E31, with the cornea E2 in front of them, and it is the surface of this cornea E2 that produces the specular reflections. The positions of these specular reflections are arranged at the three vertices of an approximately isosceles triangle, reflecting the arrangement of the infrared light sources.

[0322] On the other hand, (B1) of the same figure shows the arrangement of the lens 51 and the infrared light sources 55R, 55L, and 55B in the line-of-sight analysis terminal T4 according to the fourth embodiment, together with the three-dimensional coordinate system GX, GY, and GZ. Here too, the infrared light sources 55R and 55L are arranged on the same straight line that passes through GO, i.e., on the first straight line.

[0323] However, in this case, the first straight line connecting the positions of the infrared light sources 55R and 55L forms a known angle ∠GR with the GX axis. However, even in this case, the foot of a perpendicular line drawn from the infrared light source 55B to the line segment connecting the positions of the infrared light sources 55R and 55L coincides with the origin GO, i.e., the center of the lens 51.

[0324] 1B2 shows three specular reflections on the corneal surface caused by infrared light from the infrared light source arranged as described above. The arrangement of the three light sources is rotated by the known angle ∠GR around the GZ axis, which is the lens optical axis, and this affects the locations of specular reflections 71R, 71L, and 71B on the cornea. In other words, the approximately isosceles triangle with the positions where the three specular reflections occur as vertices is also rotated by ∠GR.

[0325] 10C1 shows the state in which the specularly reflected light is captured as Purkinje images 71iR, 71iL, and 71iB on the imaging surface 52. Images other than the Purkinje images are not shown. When the three Purkinje images are connected by line segments, an approximately isosceles triangle rotated clockwise by angle GR is observed.

[0326] The straight line CP1i that hits the base of the approximately isosceles triangle is a plane on which the optical path of infrared light from the infrared light sources 55R and 55L passes through the lens center GO after being specularly reflected and reaches the imaging surface 52, projected onto the imaging surface 52. This straight line is also the cutting plane projection image CPix.

[0327] Then, a straight line is drawn that passes through the Purkinje image located at the apex angle of the approximately isosceles triangle and intersects the line CP1i at right angles, and this line will be referred to as CP2i. The line CP2i is a line projected onto the image pickup plane 52 by a plane along which an infrared ray emitted from the infrared light source 55B is specularly reflected by the corneal surface 71B, passes through the lens center GO, and reaches the image pickup plane 52. This line is also the cut-plane projection image CPiy. In this way, even if the approximately isosceles triangle formed by the Purkinje images is rotated by an arbitrary angle, the two-dimensional coordinate of the corneal center can be found, as in the first embodiment.

[0328] Then, by rotating the GX and GY axes of the three-dimensional coordinate system and the x and y axes of the two-dimensional coordinate system on the imaging surface 52 by an angle GR around the GZ axis, which is also the lens optical axis, as follows, the first gaze vector Gaze1 and the second gaze vector Gaze2 can be calculated using the same analysis as in the first embodiment.

[0329] 10C2 shows the result of rotating the two-dimensional coordinate system of the imaging surface by angle GR around the GZ axis, i.e., around the origin o of the two-dimensional coordinate system. Here, the axes obtained after rotating the GX axis, GY axis, x axis, and y axis by angle GR are defined as the GX' axis (not shown), GY' axis (not shown), x' axis, and y' axis, respectively. As in the first embodiment, the straight line CP1i, which is a projection of the cutting plane, is parallel to the x' axis, and the straight line CP2i, which is a projection of the other cutting plane, is parallel to the y' axis.

[0330] In other words, even if the infrared light sources 55R, 55L, and 55B are rotated by an arbitrary angle about the GZ axis, the gaze vector can be calculated by the means shown in the first embodiment by rotating the three-dimensional coordinate axes by the same angle. These coincide with the conditions shown in the first embodiment, and the first gaze vector Gaze1 and the second gaze vector Gaze2 can be calculated by the analysis means shown in the same embodiment. However, since the calculated gaze vector Gaze' is a vector under the rotated coordinate system GX'-GY'-GZ, it must be converted back to an expression under the GX-GY-GZ coordinate system.

[0331] Furthermore, even if the rotation angle ∠GR is not known at the time of designing the line-of-sight analysis terminal T4, as shown in FIG. 20 (C1), ∠GR can be known as the angle formed by the line passing through the first Purkinje image 71iR and the second Purkinje image 71iL and the x-axis on the imaging surface 52.

[0332] Furthermore, the rotation transformation matrix acts on object points or image points. It should be noted that, for example, when a coordinate system is rotated in a positive direction, the object points or image points rotate relatively in a negative direction. Therefore, when the GX-GY-GZ coordinate system is rotated by ∠GR in the positive direction to form the GX'-GY'-GZ coordinate system, the rotation transformation of the object points is expressed by equation (25-1), and when returning to the GX-GY-GZ coordinate system, the transformation equation for the object points is expressed by equation (25-2).

[0333]

[0334] Equation (25-3) is an equation obtained by modifying equation (25-1) for a three-dimensional coordinate system for a two-dimensional coordinate system. In the process of calculating the gaze vector, the gaze analysis terminal T4 acquires an image in the two-dimensional x-y coordinate system projected onto the imaging surface 52 via the eyeball camera. In other words, a -∠GR transformation is performed using two-dimensional transformation equation (25-3) to calculate a three-dimensional gaze vector. Then, an ∠GR transformation is performed on the three-dimensional gaze vector using three-dimensional inverse transformation equation (25-2) to express the gaze vector in the GX-GY-GZ coordinate system, which is a normal coordinate system.

[0335] With the above in mind, let us refer again to the processing block diagram in FIG. 8. Processing blocks A1-ad and A8-ad in FIG. 8 are processing blocks when the first straight line and the GX axis do not coincide with each other within the first plane but intersect at a predetermined angle. In this case, immediately after processing block A1 is performed, a two-dimensional rotational transformation process A1-ad is added to the image on the imaging plane 52. The A1-ad transformation is performed according to equation (25-3). This process performs a two-dimensional rotational transformation on the image acquired in processing block A1.

[0336] Furthermore, immediately after processing block A8 is performed, a three-dimensional inverse rotation process A8-ad is added to the gaze vector Gaze1. The A8-ad transformation is performed according to equation (25-2). A three-dimensional inverse rotation transformation is performed on the first gaze vector calculated in processing block A8. Furthermore, if a second gaze vector is calculated in processing block A10 or A11, a three-dimensional inverse rotation transformation is performed on the second gaze vector calculated in processing block A8.

[0337] Returning to FIG. 20 , two further variations in the arrangement of the infrared light sources are shown. (D1) of the same figure shows another arrangement of three infrared light sources along with GX, GY, and GZ coordinates. The GZ axis, not shown, is a right-handed system that passes through GO and points into the page. The infrared light sources 55R and 55L, located on the GX axis, are not equidistant from the origin GO. Reflecting this, the triangle with vertices at specular reflection positions 71R, 71L, and 71B in (D2) of the same figure is not an isosceles triangle. However, by considering the arrangement of the three infrared light sources and determining that it is a group of specular reflection images (Purkinje images) that form a corresponding triangle, it is possible to determine whether or not it is a specular reflection on the cornea.

[0338] In block A3 of the processing block diagram (FIG. 8), as in the first embodiment, the first projection line HL connects the Purkinje images formed by projecting specularly reflected light 71L and 71R, which form the base of the triangle, onto the imaging plane 52. The foot of a perpendicular line drawn from the third Purkinje image, which forms the apex angle, is the corneal center projection point E21i. This perpendicular line is the second projection line VL. Also in this case, the optical path from infrared light source 55R to imaging plane 52 and the optical path from infrared light source 55L to imaging plane 52 are on the same cutting plane CPx that includes the GX axis, so there is no problem with calculating the depth information Len in block A4. Therefore, even with this arrangement of three infrared light sources, it is possible to calculate the line-of-sight vector.

[0339] Figure 20 (E1) shows another example of the arrangement of three infrared light sources along with GX, GY, and GZ coordinates. Infrared light sources 55R and 55L are arranged on the GX axis. A straight line can be drawn between infrared light source 55B on the GX-GY plane and origin GO, but this line forms a known angle SR with the GY axis. Figure 20 (E2) shows specular reflections 71R, 71L, and 71B on the cornea under these illuminations. If the arrangement of the three infrared light sources is known in advance, it can also be determined that the triangle with these vertices is a specular reflection occurring on the cornea.

[0340] However, in this case, for the triangle formed by the three specular reflection points, a straight line that is not perpendicular to the base formed by the other two specular reflection points, but forms a known angle ∠SR with the base, must be drawn as the second projection line VL. The intersection of the first projection line HL and the second projection line VL is the corneal center projection point E21i. While the specular reflection points on the corneal surface have been illustrated in Figures 20 (D1) to (E2), the actual subject of image processing and drawing is the Purkinje image on the imaging surface 52. As described above, the gaze analysis device according to the fourth embodiment relaxes the installation conditions for the infrared light source while maintaining gaze analysis accuracy.

[0341] Here, we will discuss the constraints on the placement of the infrared light sources. The simplest infrared light source placement is to place the first infrared light source on the GX axis and the third infrared light source on the GY axis in the first plane. However, in some cases, diffused reflection light from the sclera E1 is imaged on the imaging plane 52, making it impossible to determine whether the two Purkinje images are genuine Purkinje images resulting from specular reflection from the cornea E2. Therefore, the gaze analysis device based on the concept of the present invention does not adopt the above configuration.

[0342] On the other hand, if there are three or more Purkinje images and they form a specific polygon, and if a figure that can be judged to be that polygon is found on the imaging plane, the image points at those vertices can be considered to be Purkinje images. The simplest shape of a polygon is a triangle.

[0343] If the first and second infrared light sources, among the three or more infrared light sources, are arranged on a first line passing through the center GO of the lens 51, the first Purkinje image caused by the first infrared light source and the second Purkinje image caused by the second infrared light source will exist on the same cutting plane. This is because the cutting plane on which the first Purkinje image exists includes the first line and the corneal center E21, and the cutting plane on which the second Purkinje image exists also includes the first line and the corneal center E21. In other words, the two cutting planes are the same plane.

[0344] The first and second Purkinje images are on the same cutting plane. Therefore, the intersection of the cutting plane and the imaging plane 52, i.e., the cutting plane projection image CPix, is drawn as a straight line connecting the first and second Purkinje images.

[0345] Furthermore, on the first plane, a third infrared light source is provided on a second line that intersects with the first line at the origin GO. The angle between the first line and the second line is set to a known angle in design. Then, on the imaging plane 52, if a line that passes through the third Purkinje image caused by the third infrared light source and forms the known angle in design with the cutting plane projection image CPix is ​​drawn, this line is the cutting plane projection image CPiy, which is the second projection line VL.

[0346] Next, the setting of the coordinate system of the gaze analysis device based on the concept of the present invention will be explained using the first straight line as a reference. First, a lens 51 is installed as the imaging system of the eyeball camera 5. The center of the lens 51 then becomes the origin GO of the three-dimensional coordinate system, and the optical axis of the lens 51 becomes the GZ axis. Then, an imaging surface 52 is installed on a second plane perpendicular to the GZ axis, and the intersection with the GZ axis becomes the origin o of the two-dimensional coordinate system.

[0347] Next, a first line passing through the origin GO is defined, and the first and second infrared light sources must be installed on the first line. With this configuration, the first and second Purkinje images generated on the imaging surface 52 exist on the same cutting plane. For these Purkinje images to exist on the same cutting plane, the first line does not need to be perpendicular to the GZ axis. However, for convenience of subsequent analysis, it is preferable that the first line be perpendicular to the GZ axis.

[0348] When the first straight line is perpendicular to the GZ axis, if a straight line passing through the origin o and parallel to the first straight line is set as the x-axis on the imaging plane 52, the cutting plane CPix passing through the first Purkinje image and the second Purkinje image will be parallel to the x-axis. Furthermore, a straight line perpendicular to the x-axis at the origin o is set as the y-axis on the imaging plane 52. With the above, a two-dimensional x-y coordinate system is set.

[0349] Next, a straight line parallel to the x-axis and intersecting with the GZ-axis at the origin GO is set as the GX-axis. As a result, the first straight line coincides with the GX-axis. Furthermore, a straight line parallel to the y-axis and intersecting with the GZ-axis and GX-axis at the origin GO is set as the GY-axis. With the above steps, the three-dimensional coordinate system GX, GY, GZ is set.

[0350] When the x-axis is set on the imaging surface 52 so as to be at a skew position relative to the first straight line, the cutting plane CPix intersects with the x-axis at a predetermined angle. In this case, the x-y coordinate system and the GX-GY-GZ coordinate system are rotated and corrected around the GZ axis to align the first straight line with the GX axis, and then the line of sight vector is calculated. Then, as described above, reverse rotation correction is performed.

[0351] Figure 21 shows an example of infrared light source placement based on the concept of the present invention. Figure 21(A) is a three-dimensional diagram of a lens 51, a three-dimensional coordinate system GX, GY, GZ, and a corneal sphere E2. The sclera E1 is not shown. In this figure, the infrared light source 55R is located in the positive region on the GX axis, and the infrared light source 55L is located in the negative region. These two infrared light sources 55R and 55L define a common cutting plane CPx.

[0352] 1B shows the configuration of the lens 51 and the cornea E2 as viewed from the negative region of the GX axis toward the origin GO. The cutting plane CPx including the GX axis and the corneal center E21 is represented as a dashed line passing through the origin GO and the corneal center E21.

[0353] The third infrared light source 55B, not shown, is provided on a second line that intersects with the first line at a known angle. However, in FIG. 21A, the GX axis is the first line, and the first line and the second line intersect at 90°. In this example, the second line coincides with the GY axis. The infrared light source 55B will not be described.

[0354] 1C1 shows Purkinje images 71iR, 71iL, and 71iB that form the Purkinje image group on the imaging plane 52 in the above-described infrared light source arrangement. The states of the Purkinje image group detected in Case 1, Case 2, and Case 3 are summarized on one page.

[0355] Case 2 shows an example of a group of Purkinje images when the corneal center E21 is located in the GZ-GX plane. (C2) in the same figure shows the x-axis and y-axis on the imaging plane 52. Purkinje images 71iR and 71iL are projected onto the x-axis.

[0356] On the other hand, Case 1 is a diagram of a group of Purkinje images when the corneal center E21 is located below the GZ-GX plane, and Case 3 is a diagram of a group of Purkinje images when the corneal center E21 is located above the GZ-GX plane.

[0357] FIG. 1C2 shows the drawing performed by the calculation unit 61. The position of the corneal center differs for each case. Regardless of which way the corneal center moves, if a straight line passing through the first Purkinje image 71iR and the second Purkinje image 71iL is drawn, this becomes the cutting plane projection image CPix. Then, if a straight line passing through the third Purkinje image 71iB and intersecting the cutting plane projection image CPix at a known angle is drawn, this becomes the cutting plane projection image CPiy. The intersection of these lines is the corneal center projection point E21i.

[0358] Next, Figure 22 is also a diagram showing an example of the infrared light source arrangement of a line-of-sight analysis device based on the concept of the present invention. In Figure 22(A), infrared light sources 55R and 55L are provided in the negative region on the GX axis. In this case, the GX axis also corresponds to the first line. The cutting plane CPx related to these infrared light sources is common. In this example, both infrared light sources are located on one side of the lens 51, and do not necessarily need to be axially symmetrical with respect to the optical axis of the lens 51. This is a difference from paragraph

[0007] of Patent Document 2.

[0359] 1B shows the configuration of the lens 51 and cornea E2 as viewed from the negative region of the GX axis toward the origin GO. A cutting plane CPx including the GX axis and the corneal center E21 is represented as a dashed line passing through the origin GO and the corneal center E21. In this example, a third infrared light source 55B (not shown) is also provided on a second line perpendicular to the first line.

[0360] 10C1 shows the state of the Purkinje images detected in Case 1, Case 2, and Case 3 all collected on one page, and FIG. 10C2 shows the x- and y-axes and the construction lines. In this example, regardless of which way the corneal center moves, if a straight line is drawn connecting the first and second Purkinje images, and a straight line perpendicular to the first line and passing through the third Purkinje image is drawn, the intersection of these two lines is the corneal center projection point E21i.

[0361] Next, an example will be shown in which a common cutting plane is not set between the infrared light source 55R and the infrared light source 55L. The infrared light sources 55R and 55L shown in Fig. 23 do not set a common cutting plane. This is an installation mode of the infrared light source that is not within the concept of the present invention.

[0362] In FIG. 1A, infrared light sources 55R and 55L are both disposed behind lens 51. However, they are symmetrical with respect to the GZ axis, which is the optical axis. The line segment connecting the installation positions of the two infrared light sources does not pass through origin GO, which is the center of the lens. In this case, the cutting plane set by infrared light source 55R, origin GO, and corneal center E21 and the cutting plane set by infrared light source 55L, origin GO, and corneal center E21 are two different cutting planes and are not common. However, these two cutting planes are common only when corneal center E21 is within the GZ-GX plane.

[0363] Figure 1B is a view looking toward the origin GO from the negative region of the GX axis. In this figure, a light ray emitted from infrared light source 55L located on the front side of the page is specularly reflected 71L on the corneal surface, then travels a lower optical path to reach the lens center GO. On the other hand, a light ray emitted from infrared light source 55R located on the back side of the page is similarly specularly reflected 71R on the corneal surface, then travels a lower optical path to reach the lens center GO. In this way, the light paths that take high optical paths in the GY axis direction on both sides and then take a low optical path in the center after reflection do not all fit within a single plane.

[0364] 10C1 shows the Purkinje images detected in Case 1, Case 2, and Case 3 all collected on one page, and FIG. 10C2 shows the x- and y-axes and construction lines. Case 2 shows the Purkinje images when the corneal center E21 is located in the GZ-GX plane. In this case only, the straight line connecting the first Purkinje image 71iR and the second Purkinje image 71iL is the common cutting plane projection image CPix.

[0365] However, when the corneal center E21 is outside the GZ-GX plane, as in Case 1 and Case 3, the first infrared light source 55R and the second infrared light source 55L set different cutting planes, and the cutting plane projection images also have different directions. Moreover, these directions change depending on the position of the corneal center E21. Therefore, it is not possible to construct the cutting plane projection images CPixR and CPixL shown in Figure 1 (C2), and it is not possible to calculate the corneal center projection point E21i as their intersection.

[0366] Even in such a light source arrangement symmetrical with respect to the optical axis, if an infrared light source arrangement that is not within the concept of the present invention is adopted, it is not possible to set a cutting plane common to the first and second Purkinje images. Therefore, if the corneal center E21 deviates from the GZ-GX plane of the eyeball camera 5, it is not possible to obtain the corneal center projection point E21i, which becomes an obstacle to calculating the gaze vector.

[0367] The above describes the installation conditions for the three infrared light sources for setting two cutting planes related to the two cutting plane projected images that intersect on the imaging surface 52. If some measurement error is allowed, the restrictions on the installation of the light sources can be relaxed, but it is preferable that the three infrared light sources are installed on the first plane, i.e., the plane that is perpendicular to the lens optical axis at the lens center.

[0368] Furthermore, it is preferable that the first and second infrared light sources of the three infrared light sources are installed on a first straight line, i.e., a line that is included in the first plane and passes through the center of the lens, and it is also preferable that the third infrared light source of the three infrared light sources is installed on a second straight line, i.e., a line that is included in the first plane and passes through the center of the lens, and that intersects with the first straight line at a known angle.

[0369] <Selection of Three-Dimensional Coordinate System> Next, a method for selecting a three-dimensional coordinate system when a thin lens or a thick lens is used for the lens 51 will be described. When a thin lens is used as the lens 51, the nodal point, which is the intersection of only one of the principal planes and the lens optical axis, is the lens center. The plane that includes this lens center and is perpendicular to the lens center is the first plane. The GX-axis and GY-axis of the three-dimensional coordinate system are set on this first plane.

[0370] On the other hand, when a thick lens is used as the lens 51, there are two points that can be the lens center: the object-side nodal point (also called the eyeball-side nodal point) and the image-side nodal point.

[0371] When discussing the relationship between the lens and the three-dimensional coordinate system and each part of the eyeball, which is the subject, such as the eyeball center, the pupil center, the positions of the first, second, and third light sources, and the first, second, and third specular reflection positions caused by the light rays from these light sources, the object-side nodal point is regarded as the lens center, and the object-side three-dimensional coordinate system (also referred to as the eyeball-side three-dimensional coordinate system) is discussed as a three-dimensional coordinate system. In this case, the first plane means the eyeball-side first plane.

[0372] On the other hand, when discussing the relationship between the lens and the three-dimensional coordinate system and the images formed on the imaging plane 52, such as the first, second, and third Purkinje images, or the positions of each part of the eyeball projected onto the imaging plane 52, the image-side nodal point is regarded as the lens center, and the image-side three-dimensional coordinate system is discussed as a three-dimensional coordinate system. In this case, the first plane means the image-side first plane.

[0373] CPvx, CPvy...intersection line between cutting plane and virtual screen, E...eyeball, E1...sclera, E11...center of rotation, E11i...projection point of center of rotation, E12...vitreous body, E2...cornea, E21...corneal center (center of corneal curvature), E21i...projection point of corneal center, E3...iris, E31...pupil, E4...lens, E51...superior rectus muscle, E52...inferior rectus muscle, E53...lateral rectus muscle, E54...medial rectus muscle, Gaze1_Line...extension line of gaze vector, Len...distance from GX axis to corneal center, LN1...first line, LN2...second line, Gaze1, Gaze2...gaze vector, Gaze1_i...2D gaze vector, Gaze1_i_ex...2-dimensional extended line, N...normal vector, PL1...first plane, PL2...second plane, Pu...pupil center, GP...gaze point, GPi...gaze projection point, VSL...virtual screen distance, PRx, PRy...projection plane, R ａｖｅ...average value of corneal radius, Rot_R...distance between pupil center and center of rotation, sd...distance between lens and imaging surface, T1, T3, T4...gaze analysis terminal, 1...frame, 2...arm, 3...nose pad, 4...field of view camera, 41...lens, 42...imaging surface, 43...signal and power cable, 44...imaging range, 45...image recorded by field of view camera, 46...virtual screen, 47...information display board, 5, 5Q...eyeball camera, 51...lens, 52...imaging surface, 52v...virtual screen, 53...signal and power cable, 55R, 55L, 55B...infrared light source, 56...eyeball camera optical axis, 6, 6Q...control box, 61...arithmetic unit, 62...storage unit, 621...program area, 622...image area, 631...power supply, 632...user interface, 633: Communication unit, 634: Analysis result output unit, 7: Reflected light, 71, 71L, 71R, 71B: Specular reflected light, 71iL, 71iR, 71iB: Purkinje image, 72: Diffuse reflected light, 8: AR marker.

Claims

1. A gaze analysis device having an eye camera capable of photographing the corneal surface of a subject's eye, and three infrared light sources that irradiate the corneal surface with infrared light, wherein a first line connecting the installation position of a first infrared light source and the installation position of a second infrared light source among the three infrared light sources is positioned at the center of a lens equipped to the eye camera and perpendicular to the optical axis of the lens, and a second line that includes the first line and is in a first plane perpendicular to the optical axis of the lens, and which intersects the first line at the center of the lens at a predetermined angle, and wherein a third infrared light source among the three infrared light sources is positioned on the second line.

2. The gaze analysis device of claim 1, wherein three image points that may have been generated as a result of infrared light emitted from the first infrared light source, the second infrared light source, and the third infrared light source being specularly reflected on the corneal spherical surface are detected on the imaging surface of the eye camera, which is provided as a second plane perpendicular to the optical axis of the lens, and when it is determined that a triangle with the three image points as vertices has a triangular shape based on the arrangement of the first infrared light source, the second infrared light source, and the third infrared light source, the three image points are considered to be a first Purkinje image, a second Purkinje image, and a third Purkinje image caused by the first infrared light source, the second infrared light source, and the third infrared light source, respectively, and the position of the center of curvature of the corneal spherical surface is calculated from the positional relationship of the three image points.

3. The gaze analysis device of claim 2, wherein an x-y coordinate system is set as a two-dimensional coordinate system defining a position within the imaging surface by setting the intersection of the second plane and the optical axis of the lens as the origin o, an x-axis passing through the origin o and parallel to the first straight line, and a y-axis orthogonal to the x-axis at the origin o; and further wherein a GX-GY-GZ coordinate system is set as a three-dimensional coordinate system defining the position of the center of curvature of the corneal spherical surface by setting the optical axis of the lens as the GZ-axis, the intersection of the GZ-axis and the first plane, i.e., the center of the lens, as the origin GO, a first axis located within the first plane, passing through the origin GO, and parallel to the x-axis as the GX-axis, and a second axis located within the first plane, orthogonal to the GX-axis at the origin GO, and parallel to the y-axis as the GY-axis.

4. The gaze analysis device according to claim 3, wherein a straight line is drawn as a first projection line between the first Purkinje image and the second Purkinje image on the imaging surface, a straight line is drawn as a second projection line that passes through the third Purkinje image and intersects with the first projection line at the predetermined angle, and the intersection of the second projection line and the first projection line is defined as a corneal central projection point where the center of curvature of the corneal sphere is projected onto the imaging surface, and the two-dimensional coordinates of the corneal central projection point are calculated.

5. A gaze analysis device as described in claim 4, comprising a memory unit, and processing blocks including an eyeball image acquisition unit that inputs an eyeball image including the corneal spherical surface projected onto the imaging surface and imaged into the memory unit so that it can be processed arithmetically; a first eyeball image analysis unit that detects a pupil area from the acquired eyeball image; a two-dimensional pupil center detection unit that fits a circle to the pupil area and calculates the coordinates in the two-dimensional coordinate system of the pupil center that is the center of the pupil area; a second eyeball image analysis unit that detects a group of Purkinje images consisting of the first Purkinje image, the second Purkinje image, and a third Purkinje image from the acquired eyeball image; and a two-dimensional corneal center detection unit that draws the first projection line and the second projection line for the group of Purkinje images and calculates the coordinates of the corneal center projection point.

6. A configuration comprising a memory unit, and processing blocks including an eyeball image acquisition unit that captures an eyeball image including the corneal spherical surface projected onto the imaging surface and imaged thereon into the memory unit so that it can be processed; a first eyeball image analysis unit that detects a pupil area from the acquired eyeball image; a two-dimensional pupil center detection unit that fits a circle to the pupil area and calculates the coordinate value in the two-dimensional coordinate system of the pupil center, which is the center of the pupil area; a second eyeball image analysis unit that detects a group of Purkinje images consisting of the first Purkinje image, the second Purkinje image, and a third Purkinje image from the acquired eyeball image; and a two-dimensional corneal center detection unit that draws the first projection line and the second projection line for the group of Purkinje images and calculates the coordinates of the corneal center projection point, 5. The gaze analysis device according to claim 4, further comprising: an A1-ad processing unit that calculates an angle GR formed by the x-axis and the first projection line when the x-axis and the first projection line are not parallel, and after the angle GR is calculated, applies a rotation correction of an appropriate angle to the image captured by the eyeball image acquisition unit around an origin o as an axis, thereby correcting the x-axis and the first projection line to be parallel; and the image corrected and output by the A1-ad processing unit is processed as the eyeball image in a subsequent processing block that includes the first eyeball image analysis unit, the two-dimensional pupil center detection unit, the second eyeball image analysis unit, and the two-dimensional cornea center detection unit.

7. The center of curvature of the corneal sphere is sometimes referred to as the corneal center, and as a processing block, in order to analyze the cross-sectional structure of the eyeball when it is assumed to be cut by a plane determined by the first projection line and the corneal center, hereinafter referred to as the cutting plane, calculate an intersection angle φ formed by a GZ-GX plane, which is spanned by the GZ axis and the GX axis, and the cutting plane, from the distance between the first projection line and the origin o in the two-dimensional coordinate system and the distance sd between the center of the lens and the imaging surface, select either the first Purkinje image or the second Purkinje image as a reference Purkinje image, refer to an infrared light source that causes the reference Purkinje image as a reference infrared light source, and refer to a specular reflection point as an object point of the reference Purkinje image as a reference specular reflection point, the intersection line between the first plane and the cutting plane coincides with the first straight line, and the first straight line includes the center of the lens and coincides with the GX axis, On the cutting plane, a straight line is provided as a CPZ axis perpendicular to the GX axis at the center GO of the lens, thereby setting a two-dimensional GX-CPZ coordinate system with the center GO of the lens as the origin GO on the cutting plane, and an angle α formed between the direction from the origin GO toward the reference mirror reflection point and the CPZ axis. Ｒ Regarding the angle α, the angle α is calculated based on the two-dimensional coordinate value of the reference Purkinje image, the projection direction calculated from the distance sd, and the angle φ. Ｒ is set as a known value, and the angle γ formed by the direction from the origin GO toward the corneal center and the CPZ axis is set as a known value based on the projection direction calculated from the two-dimensional coordinate value of the corneal center projection point and the distance sd and the angle φ, and the distance along the GX axis from the origin GO to the installation position of the reference light source is set as a known design value ms Ｒ The magnitude of the CPZ component from the origin GO to the corneal center, i.e., the distance from the GX axis to the corneal center, is defined as unknown Len, the radius of curvature of the corneal sphere is defined as unknown R, and the angle formed by the direction of the normal vector of the corneal sphere at the reference mirror reflection point and the CPZ axis is defined as unknown β Ｒ Then, the first condition is that the vector from the origin GO to the reference specular reflection point is equal to the vector sum of the vector from the origin GO to the corneal center and the vector from the corneal center to the reference specular reflection point. Ｒ and a second condition is that the vector from the origin GO to the reference infrared light source is equal to the vector sum of the vector from the origin GO to the reference specular reflection point and the vector from the reference specular reflection point to the reference infrared light source, and Len, R, and β are calculated. Ｒ and deriving a second equation including the unknowns, and eliminating Len from the first equation and the second equation to obtain R and β Ｒ and then apply the average value of the radii of curvature of the corneal sphere to R to make R a known number, and then solve the third equation, the first equation, and the second equation in order to calculate Len; or, use one of the first Purkinje image or the second Purkinje image that is not used as the reference Purkinje image as a quasi-reference Purkinje image, derive quasi-first equations and quasi-second equations that correspond to the first equation and the second equation based on the quasi-Purkinje image and its associated infrared light source and specular reflection point, and then eliminate the unknown Len from the quasi-first equation and the quasi-second equation to calculate Len. Ｒ β corresponds to Ｌ and the R are unknowns, and then the unknown R is eliminated from the third equation and the semi-third equation, and the β Ｒ and the above β Ｌ and deriving a fourth equation including the unknowns Len and R from the first equation and the semi-first equation, Ｒ and the above β Ｌ and as unknowns, and Ｒ and the above β Ｌ The fourth equation and the fifth equation, which contain two unknowns, are simultaneously solved to obtain the β Ｒ and the β Ｌ 7. The gaze analysis device according to claim 6, further comprising a corneal center depth calculation unit that calculates Len by solving the third equation or the quasi-third equation to make R a known quantity, and then solving either the first equation, the second equation, the quasi-first equation, or the quasi-second equation.

8. As a processing block, since a first projection plane which is a projection plane from the corneal center and includes the GX axis coincides with the cutting plane, the intersection angle formed by the first projection plane and the GZ-GX plane is the known angle φ, and an intersection angle θ formed by a second projection plane which is a projection plane from the corneal center and includes the GY axis and a GY-GZ plane spanned by the GY axis and the GZ axis is calculated from the two-dimensional coordinate value of the corneal center projection point, and assuming a rectangular parallelepiped in the GX-GY-GZ coordinate system with the corneal center as one vertex and one side each on the GX axis, the GY axis, and the GZ axis, Since the side that passes through the corneal center and is parallel to the GX axis is on the cutting plane, Len, which is the distance between the GX axis and the corneal center and is known by the processing of the corneal center depth calculation unit, and the angle φ formed by the cutting plane and the GZ-GX plane are applied to a sine function to calculate the coordinate value of the GY axis component of the corneal center, Len and the angle φ are applied to a cosine function to calculate the coordinate value of the GZ axis component of the corneal center, and the known coordinate value of the GZ axis component and the intersection angle θ are applied to a tangent function to calculate the coordinate value of the GX axis component of the corneal center, thereby calculating the three-dimensional coordinate of the corneal center, and an intersection angle Pφ formed by a third projection plane that is a projection plane from the pupil center and includes the GX axis and the GZ-GX plane is calculated from the two-dimensional coordinate value of the corneal center projection point, the intersection angle Pθ formed by the GY-GZ plane and a fourth projection plane that is a projection plane from the pupil center and includes the GY axis is calculated from two-dimensional coordinate values ​​of the corneal center projection point, and the gaze analysis device according to claim 7 further comprises a pupil center / corneal center calculation unit that calculates the three-dimensional coordinates of the pupil center by solving an equation related to the pupil center that includes the known three-dimensional position of the corneal center and the known corneal radius of curvature R, and that is determined by the projection direction from the pupil center to the origin GO defined by the intersection angle Pφ and the intersection angle Pθ, and a position separated from the corneal center position by the corneal radius of curvature R.

9. A gaze analysis device as described in claim 8, having as a processing block a first gaze vector calculation unit that calculates a first gaze vector whose starting point is a position defined by the three-dimensional coordinates of the corneal center calculated by the pupil center / corneal center calculation unit, and whose ending point is a position defined by the three-dimensional coordinates of the pupil center.

10. A processing block includes a two-dimensional rotation center calculation unit that performs processing in a two-dimensional coordinate system, drawing a line as a two-dimensional extension line between the pupil center detected by the two-dimensional pupil center detection unit and the corneal center projection point detected by the two-dimensional corneal center detection unit, and constructing the two-dimensional extension line each time the calculation results are updated from the two-dimensional pupil center detection unit and the two-dimensional corneal center detection unit, and sets the coordinates of the intersection of the latest two-dimensional extension line and the previously created two-dimensional extension line as the rotation center projection point; a two-dimensional rotation center calculation unit that calculates the three-dimensional coordinates of the intersection of a projection line from the three-dimensional coordinates of the rotation center as an object point to the rotation center projection point on the imaging unit and a line from the three-dimensional coordinates of the pupil center to the three-dimensional coordinates of the corneal center, and updates and saves them as the three-dimensional coordinates of the rotation center; and further calculates and updates and saves a radius of rotation from the three-dimensional coordinates of the rotation center and the three-dimensional coordinates of the pupil center, while 9. The gaze analysis device according to claim 8, further comprising: a three-dimensional center of gyration and radius of gyration calculation unit that calculates three-dimensional pupil center coordinates from the pupil center coordinates obtained by the two-dimensional pupil center detection unit and the updated and saved three-dimensional center of gyration coordinates and the radius of gyration, in a processing cycle in which no calculation results are obtained downstream of the second eyeball image analysis unit due to a failure of the second eyeball image analysis unit to detect the Purkinje image group; and a second gaze vector calculation unit that calculates a second gaze vector that has as its start point the three-dimensional center of gyration coordinates obtained by the three-dimensional center of gyration and radius of gyration calculation unit and as its end point the three-dimensional pupil center coordinates obtained by the pupil center and cornea center calculation unit or the three-dimensional pupil center coordinates obtained by the three-dimensional center of gyration and radius of gyration calculation unit.

11. The gaze analysis device of claim 8, which has as its processing blocks: a first gaze vector calculation unit that calculates a first gaze vector whose starting point is a position defined by the three-dimensional coordinates of the corneal center calculated by the pupil center / corneal center calculation unit, and whose ending point is a position defined by the three-dimensional coordinates of the pupil center; and an A8-ad processing unit that performs an inverse rotation transformation around the GZ axis by the appropriate angle on a three-dimensional vector including the calculated first gaze vector, wherein when an image converted by the A1-ad processing unit is processed as the eyeball image, the A8-ad processing unit outputs the first gaze vector to which the inverse transformation processing has been applied.

12. A processing block includes a two-dimensional rotation center calculation unit that performs processing in a two-dimensional coordinate system, drawing a line as a two-dimensional extension line between the pupil center determined by the two-dimensional pupil center detection unit and the corneal center projection point determined by the two-dimensional corneal center detection unit, and constructing the two-dimensional extension line each time the calculation results are updated from the two-dimensional pupil center detection unit and the two-dimensional corneal center detection unit, and sets the coordinates of the intersection of the latest two-dimensional extension line and the previously created two-dimensional extension line as the rotation center projection point; a two-dimensional rotation center calculation unit that calculates the three-dimensional coordinates of the intersection of a projection line from the three-dimensional coordinates of the rotation center as an object point to the rotation center projection point on the imaging unit and a line from the three-dimensional coordinates of the pupil center to the three-dimensional coordinates of the corneal center, and updates and saves them as the three-dimensional coordinates of the rotation center; and further calculates and updates and saves a radius of rotation from the three-dimensional coordinates of the rotation center and the three-dimensional coordinates of the pupil center, while In a processing cycle in which no calculation results related to downstream of the second eyeball image analysis unit are obtained due to a failure in detecting the Purkinje images by the second eyeball image analysis unit, the device comprises: a three-dimensional center of gyration and radius of gyration calculation unit that calculates three-dimensional pupil center coordinates from the pupil center coordinates obtained by the two-dimensional pupil center detection unit and the updated and saved three-dimensional center of gyration coordinates and the radius of gyration; a second gaze vector calculation unit that calculates a second gaze vector that has as its start point the three-dimensional center of gyration coordinates obtained by the three-dimensional center of gyration and radius of gyration calculation unit and as its end point the three-dimensional pupil center coordinates obtained by the pupil center and cornea center calculation unit or the three-dimensional pupil center coordinates obtained by the three-dimensional center of gyration and radius of gyration calculation unit; and an A8-ad processing unit that performs an inverse rotation transformation by the appropriate angle around the GZ axis on a three-dimensional vector including the calculated second gaze vector, The gaze analysis device according to claim 8, wherein when the A1-ad processing unit processes an image converted as the eyeball image, the A8-ad processing unit outputs the second gaze vector to which the inverse transformation processing has been applied.

13. The gaze analysis device according to claim 12, comprising: a frame to be hung on both ears of the subject; arms fixed to the frame and having tips extending in front of the face of the subject; a nose pad for supporting the frame on the nose of the subject; a field of view camera positioned at the upper front of the face of the subject and capable of capturing a scene in the same direction as the direction visible to the subject; a gaze analysis terminal having the eye camera attached to the tip of the arm and capable of capturing images of the cornea of ​​the subject's eye; and a control box containing the calculation unit, memory unit and power supply for performing a series of processes from the eyeball image acquisition unit to the first gaze vector calculation unit or the second gaze vector calculation unit, and capable of reproducing an image recorded by the field of view camera with a gaze mark superimposed on the gaze point calculated based on the first gaze vector or the second gaze vector.

14. A gaze analysis device as described in claim 13, which is provided with a distance sensor near the field of view camera that can measure distance in a direction that can be seen by the subject, imagines a virtual screen at a position away from the field of view camera the distance output by the distance sensor as the distance to an object in front of the subject, sets the gaze point on the virtual screen based on the first gaze vector or the second gaze vector, and superimposes the gaze mark on the image recorded by the field of view camera at a position corresponding to the gaze point.

15. A gaze analysis device as described in claim 12, comprising one or more eye cameras arranged above, below or to the left or right of an information display board so as to be able to photograph the cornea of ​​one or more subjects, and a control box containing the calculation unit, the memory unit and a power supply for performing a series of processes from the eye image acquisition unit to the first gaze vector calculation unit or the second gaze vector calculation unit, and recording time-series changes in the position of the gaze point on the information display board calculated based on the first gaze vector or the second gaze vector.

16. In a photographing environment having an eyeball camera capable of photographing the corneal spherical surface of a subject's eyeball, and three infrared light sources that irradiate the corneal spherical surface with infrared light rays, a first line connecting the installation position of a first infrared light source and the installation position of a second infrared light source among the three infrared light sources is disposed at the center of a lens equipped in the eyeball camera and is perpendicular to the optical axis of the lens, a second line is located in a first plane that includes the first line and is perpendicular to the optical axis of the lens, and the second line intersects with the first line at the center of the lens at a predetermined angle, and an imaging surface onto which an image is projected by the action of the lens is provided on the second plane that is perpendicular to the optical axis of the lens, a gaze analysis method for detecting three image points that may have been produced on the imaging plane as a result of infrared light rays emitted from the first infrared light source, the second infrared light source, and the third infrared light source being specularly reflected on the corneal spherical surface; and, if it is determined that a triangle having the three image points as vertices has a triangular shape based on the arrangement of the first infrared light source, the second infrared light source, and the third infrared light source, the method considers the three image points to be a first Purkinje image, a second Purkinje image, and a third Purkinje image caused by the first infrared light source, the second infrared light source, and the third infrared light source, respectively, and calculates the position of the center of curvature of the corneal spherical surface from the positional relationship of the three image points.

17. A gaze analysis method as described in claim 16, in which an x-y coordinate system is set as a two-dimensional coordinate system defining a position within the imaging surface, with the intersection of the second plane and the optical axis of the lens as the origin o, an x-axis passing through the origin o and parallel to the first line, and a y-axis perpendicular to the x-axis at the origin o, the method comprising: drawing a straight line as a first projection line between the first Purkinje image and the second Purkinje image on the imaging surface; drawing a straight line as a second projection line passing through the third Purkinje image and intersecting with the first projection line at the specified angle; and calculating the two-dimensional coordinates of the intersection of the second projection line and the first projection line as a corneal central projection point where the center of curvature of the corneal spherical surface is projected onto the imaging surface.

18. The center of curvature of the corneal sphere is sometimes called the corneal center, and in a computing environment in which a GX-GY-GZ coordinate system is set as a three-dimensional coordinate system defining the position of the corneal center, the optical axis of the lens is the GZ axis, the intersection of the GZ axis and the first plane, i.e., the center of the lens, is the origin GO, a first axis that is within the first plane, passes through the origin GO, and is parallel to the x-axis is the GX axis, and a second axis that is within the first plane, is perpendicular to the GX axis at the origin GO, and is parallel to the y-axis is the GY axis, the computing environment includes: an image acquisition step 2 in which an eyeball image including the corneal sphere projected on the imaging surface is input into a computing unit in a manner that allows analysis and processing; and a pupil detection step 3 in which a pupil area is detected from the difference in brightness of the corneal image of the eyeball, and two-dimensional coordinates of the pupil center are calculated by applying circular fitting to the pupil. the method comprises a Purkinje image group detection step 4 for determining that a Purkinje image group consisting of the first Purkinje image, the second Purkinje image, and the third Purkinje image has been detected based on the shape of the triangle formed by the three image points, and proceeding to the next step 5; a corneal center projection point calculation step 5 for setting the intersection of the first projection line and the second projection line drawn on the detected Purkinje image group as the corneal center projection point; in the pupil detection step 3, only when the pupil region is detected is the result determined to be 'true' and proceeding to the Purkinje image group detection step 4; otherwise, returning to the image acquisition step 2 to capture a corneal image of the eyeball; and a corneal center Len calculation step 6 following the corneal center projection point calculation step 5 for calculating depth information as the distance Len from an analysis reference line coinciding with the first straight line to the corneal center. and a pupil center / cornea center three-dimensional coordinate calculation step 8 for calculating three-dimensional coordinates of the pupil center and the cornea center by applying the depth information to the two-dimensional coordinates of the pupil center and the cornea center projection point.

19. A gaze analysis method according to claim 18, further comprising a first gaze vector calculation step 9 for calculating a first gaze vector having the three-dimensional coordinates of the corneal center as a starting point and the three-dimensional coordinates of the pupil center as an end point.

20. A series of processes from the image acquisition step 2 to the gaze vector calculation step is referred to as a processing cycle, and a line is drawn as a two-dimensional extension line on the imaging plane between the pupil center obtained in the pupil detection step 3 and the corneal center projection point obtained in the corneal center projection point calculation step 5, and the coordinates of the intersection of the two-dimensional extension lines before and after the rotation of the eyeball are set as a rotation center projection point, and the three-dimensional coordinates of the intersection of the projection line from the three-dimensional position of the rotation center as the object point to the rotation center projection point on the imaging unit and the line from the three-dimensional position of the pupil center to the three-dimensional coordinates of the corneal center are obtained and updated and saved as the rotation center three-dimensional coordinates, and a rotation center 3D coordinate update step 10 is further provided in which a rotation radius is calculated, updated and saved from the rotation center three-dimensional coordinates and the three-dimensional coordinates of the pupil center.

19. The gaze analysis method according to claim 18, further comprising: a turning center 3D coordinate determination step 11 for determining that the updated turning center three-dimensional coordinate is valid if there is no significant change from the turning center three-dimensional coordinate updated in a previous processing cycle; a second gaze vector calculation step 14 for calculating a second gaze vector having the turning center three-dimensional coordinate determined to be valid as a start point and the pupil center three-dimensional coordinate as an end point; a second second gaze vector calculation step 15 for calculating the second gaze vector having the turning center three-dimensional coordinate updated and saved in the turning center 3D coordinate update step 10 in a previous processing cycle as a start point when the Purkinje image group is not detected in the Purkinje image group detection step 4; a gaze vector selection step 16 for selecting the second gaze vector calculated in the second second gaze vector calculation step 15 as an output gaze vector when it is determined that the Purkinje image group is not detected in the Purkinje image group detection step 4; and a display viewpoint position update step 17 for updating a gaze position based on the selected output gaze vector.

Images