Line-of-sight analysis apparatus, line-of-sight analysis program, and line-of-sight analysis method

Abstract

To provide a line-of-sight analysis device, a program for line-of-sight analysis, and a line-of-sight analysis method capable of determining whether or not reflected light obtained in an eyeball is a Purkinje image in a cornea region.SOLUTION: An eyeball camera 5 capable of photographing a cornea spherical surface and three infrared light sources for irradiating the cornea spherical surface with infrared rays are provided, first and second light sources exist on a first straight line passing through the lens center of the eyeball camera 5, and a third light source is arranged on a second straight line existing in a first plane orthogonal to a lens optical axis including the first straight line and crossing the first straight line at a prescribed angle at the lens center. Three image points which may be generated by the light beams by the three light sources are detected on an imaging surface which is a second plane orthogonal to the optical axis, and it is determined whether the three image points are Purkinje images from a shape of a triangle having the three image points as vertexes.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a gaze analysis device, a gaze analysis program, and a gaze analysis method for analyzing the direction of a subject's gaze. [Background technology]

[0002] Eye-gaze analysis technology has many applications. For example, there are cameras that adjust their focus based on the direction of the user's gaze as they look through the viewfinder. Another example is a user interface that allows users to operate icons on a display with their gaze. In recent years, this technology has also been used to improve the performance of head-mounted displays (HMDs) for mixed reality (MR) and augmented reality (AR).

[0003] Prior art techniques treat the cornea and eyeball as a composite spherical model with two curvatures. Patent Document 1 describes a configuration in which infrared light-emitting diodes 13a and 13b are positioned symmetrically with respect to the optical axis of an eyeball camera that photographs the subject's cornea, irradiating the cornea with infrared light and capturing an image. The light from the two infrared light sources produces two specular reflections on the corneal surface, which are then photographed by the eyeball camera. The direction of the straight line connecting the center of the iris (or the pupil at the center of the iris), which is separately detected, and the center of curvature of the cornea is calculated as the optical axis direction of the eyeball (see Figure 5 of Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2000-121917 [Patent Document 2] Japanese Patent Application Laid-Open No. 2004-129927 [Non-patent literature]

[0005] [Non-Patent Document 1] "New Fundamentals of Ophthalmic Optics" by Mototsugu Nishinobu, Koichi Iwata, and Hiroshi Uosato, Kanehara Publishing, 2012. [Non-patent document 2] Edited by Fumiko Matsumoto et al., "Optics and Eyeglasses, 2nd Edition," Igaku Shoin, 2023. Summary of the Invention [Problem to be solved by the invention]

[0006] In the technology described in Patent Document 1, light from two LED light sources is specularly reflected on the corneal surface, forming two images on the imaging surface of the eye camera. Hereinafter, these images are also referred to as Purkinje images. The optical axis direction of the eyeball is calculated from this Purkinje image and the pupil center, which is detected separately. However, unless the specular reflection image is generated on a cornea that can be considered spherical, the center of corneal curvature (hereinafter, sometimes simply referred to as the corneal center) cannot be determined. Therefore, it is necessary to determine whether or not the detected reflected light image is on the cornea before proceeding with further analysis.

[0007] Therefore, an object of the present invention is to provide a gaze analysis device, a gaze analysis program, and a gaze analysis method that can proceed with processing after determining whether or not reflected light obtained from the eyeball is a Purkinje image in the corneal region. [Means for solving the problem]

[0008] In an embodiment of a gaze analysis device based on the concept of the present invention, there is provided an eyeball camera capable of photographing a corneal spherical surface of an eyeball of a subject, and three infrared light sources that irradiate the corneal spherical surface with infrared light, wherein a first line connecting an installation position of a first infrared light source and an installation position of a second infrared light source among the three infrared light sources passes through a center of a lens provided in the eyeball camera, a third infrared light source among the three infrared light sources is disposed on a second straight line that is in a first plane that includes the first straight line and is perpendicular to the optical axis of the lens, and that intersects with the first straight line at a center of the lens at a predetermined angle; The above problem is solved by a gaze analysis device that detects three image points that may have been generated by infrared light rays emitted from the first infrared light source, the second infrared light source, and the third infrared light source on the imaging surface of the eye camera, which is provided as a second plane perpendicular to the lens optical axis, and if it is determined that the triangle with the three image points as vertices has a triangular shape based on the arrangement of the first infrared light source, the second infrared light source, and the third infrared light source, considers the three image points to be the first Purkinje image, the second Purkinje image, and the third Purkinje image caused by the first infrared light source, the second infrared light source, and the third infrared light source, respectively, and calculates the position of the center of curvature of the corneal sphere from the positional relationship of the three image points. [Effects of the Invention]

[0009] It is possible to provide a gaze analysis device, a gaze analysis program, and a gaze analysis method that can proceed with processing after determining whether reflected light obtained at the eyeball is a Purkinje image in the corneal region. [Brief explanation of the drawings]

[0010] [Figure 1] Figure 1 is a conceptual diagram of the human eyeball. [Figure 2] FIG. 2 is a diagram showing a gaze analysis terminal and a control box that constitute a gaze analysis device according to a first embodiment based on the concept of the present invention. [Figure 3] FIG. 3 is a diagram showing the projection relationship between a three-dimensional coordinate system and a two-dimensional coordinate system. [Figure 4] FIG. 4 is a diagram showing the ray paths of a lens. [Figure 5] FIG. 5 is a diagram showing an example of the arrangement of three infrared light sources of the line-of-sight analysis terminal of the first embodiment. [Figure 6] FIG. 6 shows an example in which an image captured by an eyeball camera contains three Purkinje images on the corneal surface and diffusely reflected light outside the corneal surface. [Figure 7] FIG. 7 is a diagram showing the optical path of infrared light emitted from an infrared light source on the GY axis until it forms a Purkinje image on the imaging surface. [Figure 8] FIG. 8 is a diagram showing a method for calculating the projection point of the center of the cornea onto the imaging plane by drawing a diagram. [Figure 9] FIG. 9 is a processing block diagram of a first embodiment of a gaze analysis device based on the concept of the present invention. [Figure 10] FIG. 10 shows a case where the center of the cornea of ​​the subject's right eye is located exactly on the optical axis of the lens. [Figure 11] FIG. 11 is a diagram showing a typical case in which the center of the cornea of ​​the subject's right eye is not located on the optical axis of the lens. [Figure 12] FIG. 12 is a cross-sectional view of an eyeball model in a typical case where the center of the cornea is off-axis from the optical axis of the eyeball camera lens. [Figure 13] FIG. 13 is a diagram of the corneal center on a three-dimensional coordinate system. [Figure 14] FIG. 14 is a diagram showing the positional relationship between the center of the cornea and the center of the pupil in a three-dimensional coordinate system. [Figure 15] FIG. 15 is a processing block diagram of a second embodiment of a gaze analysis device based on the concept of the present invention. [Figure 16] FIG. 16 is a diagram showing an example in which the eyeball rotates and the line of sight changes up and down. [Figure 17] FIG. 17 is a diagram showing a process for obtaining the three-dimensional coordinates of the center of rotation and the center of the pupil. [Figure 18] FIG. 18 is a flowchart of a gaze analysis device based on the concept of the present invention. [Figure 19] FIG. 19 is a diagram showing an application example of the line-of-sight vector. [Figure 20] FIG. 20 is a diagram showing the spatial relationship between the second line of sight vector and the imaging system of the field of view camera. [Figure 21] FIG. 21 is a diagram showing an arrangement of infrared light sources provided in an eye camera in a fourth embodiment of a gaze analysis device based on the concept of the present invention. [Figure 22] FIG. 22 is a diagram showing the positional relationship between the eyeball, three infrared light sources, a lens and a three-dimensional coordinate system, and an imaging surface and a two-dimensional coordinate system in a fourth embodiment of a gaze analysis device based on the concept of the present invention. [Figure 23] FIG. 23 is a diagram of processing blocks in a fourth embodiment of a line-of-sight analysis device based on the concept of the present invention. [Figure 24] FIG. 24 is a diagram showing an example of installation of an infrared light source in a line-of-sight analysis device based on the concept of the present invention. [Figure 25] FIG. 25 is a diagram showing an example of an infrared light source arrangement in a line-of-sight analysis device based on the concept of the present invention. [Figure 26] FIG. 26 is a diagram showing an example in which a common cutting plane is not set for the two infrared light sources. [Figure 27] FIG. 27 is a diagram showing another example of an installation mode of an infrared light source that is not included in the concept of the present invention. [Figure 28] FIG. 28 is a diagram showing an arrangement of infrared light sources in a line-of-sight analysis device according to the concept of the present invention, in which a first infrared light source and a second infrared light source are installed outside a first plane. [Figure 29] FIG. 29 is a diagram showing a method for analyzing the cross-sectional structure of the eyeball E when an infrared light source is provided on the extended first plane. [Figure 30] FIG. 30 is a diagram showing the relationship between two-dimensional and three-dimensional coordinate systems used in a line-of-sight analysis device according to the concept of the present invention. [Figure 31] FIG. 31 is a cross-sectional cutaway view for depth information analysis in a sixth embodiment based on the concept of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] <Human eye model> 1 shows a human eyeball model that is a premise for constructing a first embodiment based on the concept of the present invention. Figure 1 is a conceptual diagram of a human eyeball, in which (A) is a schematic diagram of the vertical cross-sectional structure when looking at the right eyeball located beyond the nose in a profile of the head, (B) is a schematic diagram of the horizontal cross-sectional structure when looking at the right eyeball from the top of the head, and (C) is a schematic diagram of the right eyeball when viewed from directly facing the face.

[0012] In the human eyeball model, the optical axis of the optical system formed by the eyeball extends from the head toward the front of the face. The structure is approximately rotationally symmetrical with respect to the optical axis. Therefore, (A) and (B) in Figure 1 are shown as having the same structure.

[0013] As shown in Figure 1(A), the eyeball E is composed of the sclera E1, which surrounds the vitreous body E12 in a spherical shape, and the cornea E2, which has a spherical surface that is approximately 60% of the size of the sclera E1. This is why the cornea and eyeball are treated as a composite spherical model with two curvatures. The eyeball E has a structure in which the spherical surface of the cornea E2 protrudes slightly in front of the sclera E1, which is the sphere that occupies the majority of the eyeball E.

[0014] The cornea E2 is a spherical, transparent membrane, behind which the iris E3 and lens E4 are located. The iris E3 acts as a diaphragm, attenuating the amount of light entering from the front. The circular window in the iris E3 is the pupil E31.

[0015] Furthermore, the cornea E2 has a spherical shape and a center of curvature E21 can be assumed. However, in Figures 1(A) and (B), the position of the center of curvature is shown for convenience, and no such marker actually exists. Hereinafter, the center of curvature of the cornea E2 will be simply referred to as the corneal center E21, and its three-dimensional coordinates will sometimes be expressed as Cen.

[0016] The iris E3 is the part that determines eye color. The so-called iris is this iris E3. Most of the cornea E2 overlaps with the iris E3, but the boundary between the area that appears as the iris and the area that appears as the white of the eye does not necessarily indicate the boundary of the cornea E2.

[0017] In FIG. 1C, the superior rectus muscle E51, inferior rectus muscle E52, lateral rectus muscle E53, and medial rectus muscle E54 are provided around the eyeball E. The vertical rotation of the eyeball E is mainly due to the coordinated action of the superior rectus muscle E51 and the inferior rectus muscle E52. The horizontal rotation is mainly due to the coordinated action of the lateral rectus muscle E53 and the medial rectus muscle E54.

[0018] There is no physical center of rotation for eyeball E. However, the center of rotation of eyeball E, which allows for smooth rotation within the limited space in the skull, must be localized in a somewhat limited area. For convenience, this center of rotation is called the center of rotation, and its three-dimensional coordinates are sometimes expressed as Ro.

[0019] FIG. 1D is a diagram showing the positional relationship between eyeball camera 5, one element constituting the first embodiment, and the subject's eyeball E. Three light sources (not shown) are provided around eyeball camera 5 to illuminate the cornea with infrared light. Eyeball camera 5 uses a camera that is sensitive to infrared light, which is on the long wavelength side. The above are the prerequisites for a gaze analysis device based on the concept of the present invention.

[0020] <First embodiment> <Gaze analysis terminal> 2 is a diagram of the gaze analysis terminal and control box that constitute the gaze analysis device according to the first embodiment of the present invention. Figure 2(A) is a conceptual diagram of a subject wearing the gaze analysis terminal T1, and Figure 2(B) shows the gaze analysis terminal T1 and the control box 6 that processes the eyeball image acquired by the terminal T1 and calculates the gaze vector of the subject.

[0021] 2(A) and (B), the gaze analysis terminal T1 has a frame 1 that is hung over the ears of the subject, an arm 2 fixed to the frame 1, a nose pad 3 that supports the frame 1 on the subject's nose, a field of view camera 4 that is positioned above the front of the subject's face and can capture a scene 44 (image capture range) that is substantially the same as what the subject can see, and an eyeball camera 5 that is positioned at the tip of the arm 2 and can capture the cornea of ​​the subject's right eye. In addition, a light source (not shown) that illuminates the cornea with infrared light is provided around the eyeball camera 5.

[0022] The control box 6 stores a calculation unit 61 and a memory unit 62. The memory unit 62 has a program area 621 for storing programs, and an image area 622 for holding images acquired by the eyeball camera 5 and the field of view camera 4. Data transmission between the field of view camera 4 and the memory unit 62 is carried out via a signal / power cable 43. Conversely, power is also supplied from the control box 6 to the field of view camera 4. Data transmission between the eyeball camera 5 and the memory unit 62 is carried out via a signal / power cable 53. Conversely, power is also supplied from the control box 6 to the eyeball camera 5.

[0023] <Coordinate system for processing eyeball images> The image acquired by eyeball camera 5 is processed to calculate the gaze vector of the subject's eyeball E. Since the eyeball E to be measured has a three-dimensional structure, a three-dimensional coordinate system is required to define it. The origin GO of this three-dimensional coordinate system is set at the center of lens 51 of eyeball camera 5.

[0024] On the other hand, the image acquired by eyeball camera 5 is the three-dimensional structure of eyeball E projected as a two-dimensional image onto imaging plane 52 by the action of lens 51. Therefore, the three-dimensional structure of eyeball E is calculated from the features of the two-dimensional image projected onto imaging plane 52 by ray tracing based on geometric optics.

[0025] FIG. 3 is a diagram showing the projection relationship between a three-dimensional coordinate system and a two-dimensional coordinate system. In the three-dimensional coordinate system, the optical axis of the lens 51 is defined as the GZ axis, and the direction toward the subject's eyeball (in this example, the right eyeball) is defined as the positive direction. The origin GO is set at the center of the lens 51. A plane perpendicular to the GZ axis at the origin GO is referred to as the first plane. The GX axis and GY axis, which are perpendicular to the origin GO, are set on the first plane. The GX axis, GY axis, and GZ axis form a right-handed three-dimensional coordinate system with the origin at GO. In FIG. 3, the positive direction of the GX axis is horizontally to the left of the page, and the positive direction of the GY axis is vertically upward of the page. This three-dimensional coordinate system defines the three-dimensional coordinates of each part of the subject's eyeball.

[0026] In a three-dimensional coordinate system, if a plane spanned by the GX and GY axes can be specified with coordinate values ​​along the GX and GY axes, this plane is called a GX·GY plane. Also, in the same coordinate system, if a plane spanned by the GY and GZ axes can be specified with coordinate values ​​along the GY and GZ axes, this plane is called a GY·GZ plane. Furthermore, in the same coordinate system, if a plane spanned by the GZ and GX axes can be specified with coordinate values ​​along the GZ and GX axes, this plane is called a GZ·GX plane.

[0027] Additionally, behind the GX-GY plane (on the negative side of the GZ axis) there is a position where an image is formed by the action of lens 51, and imaging plane 52 is provided at this position as a second plane. This second plane is a plane that is perpendicular to the GZ axis. A two-dimensional coordinate system is set on imaging plane 52. This two-dimensional coordinate system has its origin o at the intersection of this second plane and the GZ axis. In FIG. 3, the positive direction of the x-axis is toward the right of the page, and the positive direction of the y-axis is downward.

[0028] In the three-dimensional space defined by the three axes GX, GY, and GZ, the x-axis is parallel to the GX-axis and the y-axis is parallel to the GY-axis on the imaging surface 52. In the two-dimensional coordinate system, the plane spanned by the x-axis and y-axis, on which any point can be specified by the coordinate values ​​of the x-axis and the y-axis, will be referred to as the x-y plane.

[0029] <Ray tracing based on geometric optics> Before discussing the projection of a 3D structure onto a 2D image, we will first confirm the lens center and the optical axis of the lens based on geometric optics, which is also described in Non-Patent Document 1. Most lens surfaces are spherical. Except when the lens is a perfect sphere, there are separate centers of curvature for the spherical surface facing the subject and the spherical surface facing the imaging surface. The straight line passing through these two centers of curvature, which are located at different positions, is the optical axis of the lens.

[0030] A focal point exists for one lens spherical surface. The plane from which the distance to the focal point, i.e., the focal length, begins is called the principal plane or main surface. Originally, a principal plane is a plane that is tangent to the lens spherical surface on the optical axis, but since the center of curvature of that spherical surface is on the optical axis, the principal plane is a plane that is perpendicular to the lens optical axis.

[0031] A convex lens has a spherical surface that is convex toward both the object side and the imaging plane side. Depending on the distance between the lens and the subject, compared to the lens thickness, there are thick lenses whose lens thickness must be taken into consideration, and thin lenses whose lens thickness can be ignored. In both cases, the object side and imaging plane side are assumed to be filled with air with a refractive index of 1, and the lens interior has a refractive index n greater than 1. Here, the principal point of a lens refers to the intersection of the principal plane and the optical axis, and as mentioned above, the principal plane is the plane from which the focal length originates. Furthermore, the nodal point is a point that can be considered the center of the lens, and the direction of light rays entering and exiting the lens is the same as that passing through this point.

[0032] Figure 4 shows the ray paths of a lens. Figure 4(A) shows the ray paths of a thick lens, where the lens thickness d cannot be ignored. The lens optical axis is shown horizontally from left to right on the page. An object is located on the left side of the lens, and light rays are emitted from an object point representing the object, pass through the thick lens, and form an image point representing the object on the right side. In the case of a thick lens whose object and image sides are filled with air with a refractive index of 1, there are two principal planes, on the object side and the image side, which are the origins of the focal length, the distance to the object point, and the distance to the image point, separated by the thickness d.

[0033] Therefore, the object-side principal plane is referred to as the object-side principal plane, and the intersection of the object-side principal plane and the lens optical axis is the object-side principal point H O and object-side node N O The object-side principal plane is the starting point for the object-side focal length of the lens and the distance to the object position. Similarly, there is also a plane that is the starting point for the image-side focal length and the distance to the image-forming position, and this will be called the image-side principal plane. The intersection of the image-side principal plane and the lens optical axis is the image-side principal point H I and image-side node NI Both the object-side principal plane and the image-side principal plane are perpendicular to the optical axis of the thick lens.

[0034] According to the theory of geometric optics, regardless of the direction of incidence, light rays that enter the object-side principal plane travel parallel to the lens optical axis and reach the image-side principal plane. Other than that, the process is the same as with a thin lens. That is, among the outgoing light rays from a point that represents the object (hereinafter referred to as object point light rays), incident light in1 that is parallel to the optical axis of the thick lens enters the object-side principal plane, then travels parallel to the optical axis of the thick lens and reaches the image-side principal plane. It then leaves the image-side principal plane as outgoing light out1, passes through the image-side focus, and reaches the image point.

[0035] Furthermore, the object-side nodal point N O The incident light in2 reaches the image-side nodal point N along the lens optical axis. I and travels as outgoing light out2 in the same direction as the incident direction of incident light in2, forming an image at the same position as outgoing light out1. Furthermore, of the object point light rays, incident light in3 that passes through the object-side focus changes direction at the object-side principal plane, travels parallel to the lens optical axis, and reaches the image-side principal plane. It then travels as outgoing light out3 parallel to the optical axis of the thick lens, and forms an image at the same position as outgoing light out1 and out2.

[0036] Based on the above, a three-dimensional coordinate system for analyzing eyeball images is defined. FIG. 4B is a diagram showing an example of defining a three-dimensional coordinate system and a two-dimensional coordinate system when a thick lens 51 is used. First, the lens optical axis is defined as the GZ axis of the three-dimensional coordinate system, and the direction toward the eyeball, which is the object, is defined as the positive direction. The object-side principal plane perpendicular to the GZ axis is referred to as the eyeball-side first plane. The image-side principal plane perpendicular to the GZ axis is referred to as the image-side first plane. Two three-dimensional coordinate systems are defined: an eyeball-side three-dimensional coordinate system based on the position of the eyeball-side first plane, and an image-side three-dimensional coordinate system based on the position of the image-side first plane.

[0037] Regarding the eyeball-side three-dimensional coordinate system, the eyeball-side node N O That is, the origin GO is at the intersection of the eyeball side first plane and the GZ axis. O and this origin GO is defined in the first plane on the eyeball side.O The GX-axis and GY-axis are set at right angles to each other to form a right-handed three-dimensional coordinate system. Therefore, the GX-GY plane coincides with the first plane on the eyeball side. In addition, in the image-side three-dimensional coordinate system, the image-side nodal point N I That is, the origin GO is located at the intersection of the image-side first plane and the GZ axis. I and this origin GO is defined in the image-side first plane. I The GX and GY axes are set at right angles to each other to form a right-handed three-dimensional coordinate system. Therefore, the GX-GY plane coincides with the first image-side plane.

[0038] The GX axis of the eyeball-side three-dimensional coordinate system is parallel to the GX axis of the image-side three-dimensional coordinate system, and the GY axis of the eyeball-side three-dimensional coordinate system is parallel to the GY axis of the image-side three-dimensional coordinate system. In FIG. 1B, the GY axis is positive upward on the paper, and although not shown, the GX axis is aligned with the origin GO O or GO I The positive direction is the direction passing through the origin o and pointing towards the front of the page. A second plane is set that includes the image point on the image side of lens 51 and is perpendicular to the GZ axis. An imaging plane 52 is placed on this second plane, and a two-dimensional coordinate system is set here. The origin o is set at the intersection of the second plane and the GZ axis. The two-dimensional coordinate system is also a right-handed x·y system, with the positive direction of the y axis pointing downward on the page, and the positive direction of the x axis passing through the origin o and pointing into the page. The x axis is parallel to the GX axis, and the y axis is parallel to the GY axis.

[0039] Next, we will show the case where the lens 51 is a thin lens whose thickness can be ignored. Since the thickness d is zero, the object-side principal plane and the image-side principal plane overlap and coincide in Fig. 4(A). Based on this, if we define a three-dimensional coordinate system for analyzing eyeball images using the thin lens 51, it becomes the one shown in Fig. 4(C). The eyeball-side first plane and the image-side first plane are aggregated into the first plane. Eyeball-side nodal point N O and the image-side node N I are also collected at the node N. Therefore, it is sufficient to provide only one three-dimensional coordinate system with the center of the lens 51 as the origin GO.

[0040] <Projection relationship from object point to image point> Based on Figure 4(C), it is confirmed at which position on the imaging surface on the second plane an object point on the eyeball E is projected. As mentioned above, among the light rays emitted from the object point on the eyeball side, the light rays that are incident on the lens center, i.e., on nodal point N (which is also the origin GO of the three-dimensional coordinate system), maintain their direction and reach the imaging surface 52, where they form an image. In other words, in the case of a thin lens, if a straight line is drawn from the image point through the lens center to the imaging surface 52, this is the projection line that represents the projection destination.

[0041] Even in the case of a thick lens, the distance from the object point to the nodal point N on the eyeball side O The ray incident on the image-side nodal point N I The light is emitted from the eyeball side nodal point N O A line parallel to the incident light at the image side nodal point N I If you draw it from here, this is the projection line that represents the projection destination.

[0042] Now that the projection relationship between object points and image points has been clarified, let us refer back to Figure 3. Figure 3 shows the case of a thin lens 51, but the case of a thick lens will also be explained later as necessary. A second plane is provided parallel to a first plane that includes the center of lens 51 and is separated by a distance sd from said first plane. An imaging plane 52 is provided on said second plane, and a two-dimensional coordinate system x·y is set thereon.

[0043] Here, the object point OB in FIG. 3(A) is some feature on the cornea E2, such as the pupil center or other measurement points described later. The object point OB is projected onto the imaging plane 52 by the action of the lens 51. The line segment L is the projection line L that represents this projection direction. The image at the projection destination is OBi. The components of the two-dimensional coordinates at this image point are OBi(ob x ,ob y )

[0044] It is necessary to specify the direction of the ray or projection line L from object point OB to image point OBi. Projection line L is a line segment that passes through three points: the three-dimensional coordinate origin GO, object point OB, and image point OBi. Projection line L then exists on a plane uniquely determined by the GY axis and the position of image point OBi or object point OB. We will call this plane the projection plane PRy [Figure 3(B1)]. Projection line L also exists on a plane uniquely determined by the GX axis and the position of image point OBi or object point OB. We will call this plane the projection plane PRx [Figure 3(B2)]. Projection line L is the intersection of these two projection planes PRx and PRy.

[0045] Therefore, to define the direction of the projection line L, the angle ∠oθ formed by the projection plane PRy and the GY·GZ plane is GY and the projection plane PRx and the GZ·GX plane form ∠oφ GX Then, these two angles (oθ GY ,oφ GX ) defines the direction of the projection line L. Note that the axis of rotation, such as GY or GX, may be written as the subscript at the bottom right of a variable that represents an angle or rotation angle. Also, although it is not in accordance with the rules of right-handed coordinates, ∠oφ GX As shown in (B2) of the same figure, the clockwise direction is defined as the positive direction when the GX axis is viewed from the positive region.

[0046] As shown in FIG. 3(B1), the projection plane PRy and the GY·GZ plane are spaced apart by ∠oθ GY They intersect at this angle. This angle is determined by the distance sd between the first and second planes and the distance of the image point OBi from the y-axis (the axis perpendicular to the paper). The distance from the y-axis is a constant multiple of the x-coordinate. This constant is the size of one pixel of the image sensor in the x-axis direction.

[0047] As shown in FIG. 3(B2), the projection plane PRx and the GZ·GX plane are spaced apart by ∠oφ GX They intersect at this angle. This angle is determined by the distance sd between the first and second planes and the distance of the image point OBi from the x-axis (horizontal axis on the paper). The distance from the x-axis is a constant multiple of the y-coordinate. This constant is the size of one pixel of the image sensor in the y-axis direction.

[0048] Using the distance sd and the two-dimensional coordinates of the image point OBi, ∠oθ is obtained from equation (1-1). GY can be calculated, and ∠oφ can be calculated from equation (2-1). GX Conversely, to calculate each component of the two-dimensional coordinates of the image point OBi from the angle, we can apply equations (1-2) and (2-2). However, here the size of the image sensor for one pixel is set to 1 in both the x-axis and y-axis directions.

[0049]

number

[0050] To calculate the gaze vector of the eyeball E, it is necessary to determine the three-dimensional coordinates of the corneal center E21 or the center of rotation E11, which can be the starting point of the vector, and the pupil center Pu, which can be the end point. To determine the three-dimensional coordinates of these object points, it is necessary to identify the direction in which each object point exists and its distance from a reference point or reference line.

[0051] If the entire contour of the cornea E2 or eyeball E as a sphere is projected onto the imaging plane 52, a projection line passing from the center of the projection line through the lens center GO can be drawn to determine the direction of the corneal center E21 or the center of rotation E11. However, most of the eyeball E is hidden by the eyelids, and only a portion of the curved surface of the cornea E2 that can be considered a sphere is exposed, making it impossible to capture the entire contour.

[0052] The surface of the sclera E1, which is the white of the eye, lacks smoothness, causing light to be scattered, whereas the cornea E2, which covers the iris E3 and pupil E31 with a spherical surface, is smooth, causing light to be specularly reflected.

[0053] When a light source is placed at the center GO of the lens 51 and shines onto the cornea E2, light rays traveling in a specific direction are specularly reflected from the surface of the cornea E2, pass through the center of the lens 51 again, and form an image at a certain position on the imaging plane 52. The fact that the specularly reflected light rays return to the same place, the center GO, means that the specular reflection is caused by light rays that are incident in the normal direction to the surface of the cornea E2. The corneal center E21 lies in this normal direction.

[0054] However, placing a light source at the center GO of the lens 51 requires an optical device such as a beam splitter, which impairs compactness and lightness. Therefore, the line-of-sight analysis terminal T1 of the first embodiment is provided with three infrared light sources, avoiding the center GO of the lens 51.

[0055] 5A and 5B are diagrams showing an example of the arrangement of three infrared light sources in the gaze analysis terminal of the first embodiment. Fig. 5A shows the arrangement of eye camera 5 and the three infrared light sources arranged together with the eye camera, namely, first infrared light source 55R, second infrared light source 55L, and third infrared light source 55B, as viewed from the eyeball side of the subject.

[0056] The first infrared light source 55R and the second infrared light source 55L are disposed on either side of the lens 51, and the third infrared light source 55B is disposed below the lens 51. A straight line connecting the center of the first infrared light source 55R and the center of the second infrared light source 55L intersects with the optical axis 56 of the lens 51. This intersection is the center of the lens 51. This straight line may also be referred to as the first straight line.

[0057] The line that passes through the center of the third infrared light source 55B and intersects with the optical axis 56 at the center of the lens may also be referred to as the second line. In this example, the second line is set on the GY axis. Alternatively, the second line may be defined as a line that forms a predetermined angle with the first line.

[0058] 5(A), (B1), and (B2), the second line is on the GY axis (not shown) and is perpendicular to the first line. The infrared light sources 55R, 55L, and 55B are located on a first plane. The optical axis 56 is also the GZ axis of the three-dimensional coordinate system.

[0059] 5(B1) is a diagram showing the configuration of the eyeball camera 5 provided in the gaze analysis terminal T1, as seen from above the head of a subject wearing the gaze analysis terminal T1. An imaging plane 52 is located behind the lens 51, at a distance sd between the lens and the imaging plane. An eyeball camera optical axis 56 is shown, passing from the imaging plane 52 through the center of the lens 51.

[0060] Here, infrared light sources 55R and 55L are arranged on a plane that is perpendicular to eyeball camera optical axis 56 and includes the center of lens 51, i.e., on a first plane. Also, imaging surface 52 is provided behind the first plane on a second plane that is perpendicular to the optical axis of lens 51. Figure 2(B2) is a configuration diagram of eyeball camera 5 when viewed from the side of infrared light source 55L.

[0061] 10(C1) is a diagram of eyeball E illuminated by infrared light sources 55R, 55L, and 55B. This eyeball E is imaged as a two-dimensional image on imaging surface 52 by the imaging action of lens 51, and this is converted into electronic data by a large number of imaging elements arranged on imaging surface 52, and is stored in image area 622 of storage unit 62. This electronic data is subjected to image processing by calculation unit 61.

[0062] The sclera E1, which is the white of the eye, the iris E3, which is the black part of the eye, and the pupil E31 each have different reflectances to infrared light, so they can be distinguished by the difference in brightness on the imaging plane 52. On the other hand, the boundary between the cornea E2 and the sclera E1 cannot be distinguished by the difference in brightness. Most of the cornea E2 overlaps with the iris E3, and the iris E3 is photographed through the cornea E2. However, the cornea E2 extends slightly outside the iris E3.

[0063] Figure 5(C2) is an enlarged view of the cornea E2, iris E3, and pupil E31 regions in Figure 5(C1). This figure shows an example of three specularly reflected lights generated on the surface of the transparent cornea E2, which covers the iris E3 and pupil E31 from the outside. The image formed by these reflected lights on the imaging plane 52 by the imaging action of the lens 51 is called a Purkinje image.

[0064] Here, the three specular reflections occurring on the corneal surface in three-dimensional space will be referred to as specular reflected light 71R, 71L, and 71B. Then, the images of these appearing on the imaging plane 52 will be referred to as Purkinje images 71iR, 71iL, and 71iB, respectively. The Purkinje images 71iR, 71iL, and 71iB are formed by the imaging action of the lens 51, where the specular reflected light 71L, 71R, and 71B occurring on the corneal surface have been subjected to rotational symmetric transformation about the eye camera optical axis 56, and then to enlargement or reduction transformation.

[0065] 10D is a diagram showing only Purkinje images 71iR, 71iL, and 71iB extracted from the imaging plane 52. In addition, 71R, 71L, and 71B represent specularly reflected light, and may also represent the locations on the cornea where specular reflection occurs.

[0066] Figure 6 shows an example in which an image taken by an eyeball camera contains three Purkinje images on the corneal surface and diffusely reflected light outside the corneal surface. Figure 6 (A1) shows an example in which infrared light is specularly reflected and formed as a Purkinje image, and is an example of Purkinje image formation in the right eye of a subject wearing the gaze analysis terminal T1, looking down from the top of the head.

[0067] Figure 1 (A2) is a diagram of the cornea E2 of eyeball E photographed from directly in front by ocular camera 5 under the conditions of (A1). Infrared light from infrared light source 55R generates specularly reflected light 71R at approximately the center of cornea E2, and Purkinje image 71iR is generated on imaging plane 52 by the action of lens 51 [Figure 1 (A3)]. However, the image on imaging plane 52 [Figure 1 (A3)] has been rotated 180° around the GZ axis to match Figure 1 (A2). Two other specularly reflected lights 71L and 71B are also generated on cornea E2 [Figure 1 (A2)], and Purkinje images 71iL and 71iB are also formed on imaging plane 52 [Figure 1 (A3)].

[0068] 10B1 shows the state of photography when the subject rotates his / her eyeball to the left. Since the infrared light from the infrared light source 55R does not have an optical path that passes through the center of the lens 51 after specular reflection on the cornea E2, no Purkinje image is formed on the imaging plane 52. Instead, some of the diffusely reflected light 72 generated on the surface of the sclera E1 may form a diffusely reflected light image 72i on the imaging plane 52.

[0069] These diffusely reflected light images are not formed via specular reflection on the cornea, and therefore do not contain information about the surface shape of the cornea E2. Therefore, when analyzing the images formed on the imaging plane 52, it is necessary to reliably remove the diffusely reflected light images.

[0070] Figure 1 (B2) shows an example in which the ocular camera 5 cannot capture the specular reflection occurring within the cornea E2 under the circumstances of (B1). The infrared light from the infrared light source 55R does not meet the conditions for the specular reflection light occurring on the cornea E2 to form an image by the lens 51, and no Purkinje image is generated. Instead, in this example, part of the diffusely reflected light 72 occurring on the surface of the sclera E1 forms an image as a diffusely reflected light image 72i on the imaging plane 52 (Figure 1 (B3)).

[0071] Due to the difference in surface smoothness, the surface of the sclera E1 is prone to diffuse reflection of infrared light, while the surface of the cornea E2 is less prone to diffuse reflection. In addition, the cornea E2 is transparent, and diffuse reflection from the iris E3 behind it forms an image on the imaging plane 52, which contributes to the detection of the pupil E31.

[0072] The Purkinje image, which is a specular reflection from the surface of the cornea E2, and the image, which is a diffuse reflection from the iris E3, have a difference in brightness, so they can be distinguished by the difference in brightness. On the other hand, strong diffuse reflection occurs on the surface of the sclera E1. While specular reflection can occur, diffuse reflection also exists around it.

[0073] In order to accurately measure the surface curvature of the cornea E2 and calculate the position of the corneal center E21, it is necessary to select only the Purkinje image produced by specular reflection from the surface of the cornea E2 and eliminate the image produced by reflection from the sclera E1. For this purpose, three or more infrared light sources are arranged so that three or more Purkinje images are detected in a specific arrangement on the imaging plane 52. This arrangement must be useful for calculating the position of the corneal center E21.

[0074] Therefore, the first Purkinje image 71iR, the second Purkinje image 71iL, and the third Purkinje image 71iB, which are arranged in a specific order, are detected when analyzing the image on the imaging plane 52. Then, only when it is confirmed that the three Purkinje images are arranged in a specific order, is the position of the corneal center E21 calculated based on these two-dimensional coordinate systems.

[0075] Even if a Purkinje image of a specific arrangement is obtained, if there is an image due to diffused reflection in the vicinity thereof, it may not have been obtained on the cornea E2, and so this may be excluded.

[0076] As mentioned above, only a portion of the spherical surface of the cornea E2 is exposed from the sclera E1, and the direction of the corneal center E21 cannot be determined from its contour. Therefore, an analysis is performed using the Purkinje image as shown below.

[0077] <Constraints on the light path in specular reflection> First, let us examine the constraints on the optical path in specular reflection. Fig. 7(A) shows the optical path of an infrared ray emitted from infrared light source 55B on the GY axis until it forms Purkinje image 71iB on imaging plane 52. The infrared ray forms Purkinje image 71iB on imaging plane 52 after passing through infrared light source 55B, specular reflection at point 71B on the surface of cornea E2, and lens center GO. Here, the specularly reflected infrared ray is incident on and reflected at an angle equiangular to the normal vector N of the surface of cornea E2.

[0078] The reflected light vector, which represents the direction and orientation of the reflected light, is on the plane formed by the incident light vector and the normal vector at the reflecting surface point 71B. In other words, the incident light vector, reflected light vector, and normal vector are on the same plane. Furthermore, since the surface of the cornea E is considered to be spherical, extending the normal vector N backward from the starting point will reach the corneal center E21. Based on this fact, hereinafter, the starting point of the normal vector N may be referred to as the corneal center E21, and the end point may be referred to as being on the corneal surface.

[0079] The infrared light ray travels in the direction of the reflected light vector, passes through origin GO, which is the center of lens 51, and reaches Purkinje image 71iB on imaging plane 52. On the other hand, if it returns in the opposite direction to the incident light vector, it reaches infrared light source 55B on the GY axis.

[0080] What is important here is that the infrared light source 55B, the specular reflection point 71B, the origin GO of the three-dimensional coordinate system, the Purkinje image 71iB, and the corneal center E21 are all on the same plane. Furthermore, since the infrared light source 55B is on the GY axis, the same plane includes the GY axis. In other words, the same plane can be said to be a plane uniquely defined by the positions of the GY axis and the corneal center E21.

[0081] The plane is uniquely defined by the second line connecting the third infrared light source 55B and the origin GO and the position of the corneal center E21, and hereinafter may be referred to as the cutting plane CPy. Here, the second line is on the GY axis. In this case, the projection plane PRy, which defines the projection direction of the object point on the cutting plane CPy onto the imaging plane 52, coincides with the cutting plane CPy. Therefore, equations (1-1) and (1-2) can be applied to the projection direction of the specular reflection point 71B and the Purkinje image 71iB on the same plane.

[0082] Figure 7(A) shows a cross-sectional view taken along the cutting plane CPy, in which the optical path of the infrared light from the infrared light source 55B to the Purkinje image 71iB and the corneal center E21 are shown on a single plane (on the page). When viewed from a viewpoint within this plane, the plane appears to be a straight line, and Figure 7(B) is a view looking down from a viewpoint within the cutting plane CPy, from the positive region of the GY axis toward the origin GO. This is also a view looking down on the subject wearing the gaze analysis terminal T1, roughly from the top of the head. The cutting plane CPy is represented by a straight line (dashed line) passing through the origin GO.

[0083] As with the third infrared light source 55B, a light beam emitted from the first infrared light source 55R installed on the GX axis is specularly reflected at point 71R on the surface of the cornea E2 and passes through the lens center GO, generating a Purkinje image 71iR (not shown) on the imaging plane 52. The infrared light source 55R, specular reflection point 71R, the origin GO of the three-dimensional coordinate system, the Purkinje image 71iR, and the corneal center E21 are all on the same plane.

[0084] The above plane is uniquely defined by the first straight line connecting the first infrared light source 55R, the origin GO, and the second infrared light source 55L, and the position of the corneal center E21, and hereinafter this plane may be referred to as the cutting plane CPx. Here, the first straight line is on the GX axis. In this case, the projection plane PRx, which defines the projection direction of the object point on the cutting plane CPx onto the imaging plane 52, coincides with the cutting plane CPx. Therefore, equations (2-1) and (2-2) can be applied to the projection direction of the specular reflection point 71R and the Purkinje image 71iR on the same plane.

[0085] Furthermore, the second infrared light source 55L and the Purkinje image 71iL projected onto the imaging plane 52 are also included in the cutting plane CPx, and the formulas (2-1) and (2-2) can also be applied to the projection directions of these.

[0086] 7(C) is a projected image of the eyeball E on the imaging unit 52. Specular reflections 71R, 71L, and 71B occurring on the surface of the cornea E2 are projected onto the imaging unit 52 as Purkinje images 71iR, 71iL, and 71iB, respectively.

[0087] For the purpose of explanation, Figure 1(D) is a diagram in which only the Purkinje image is extracted from the image projected onto the imaging unit 52 and drawn. This diagram is an internal process performed by a computer, and such an image does not actually occur. The intersection line when the cutting plane CPy, which includes the GY axis, the Purkinje image 71iB, and the corneal center E21, intersects with the imaging surface 52 is represented by CPiy. Since this is a projection image of the cutting plane CPy onto the imaging surface 52, it will be referred to as the cutting plane projection image CPiy. Because the cutting plane CPy includes the GY axis, its projection image, the cutting plane projection image CPiy, is parallel to the GY axis and the y axis.

[0088] The line of intersection of the cutting plane CPx, which includes the GX axis, the Purkinje image 71iR, the Purkinje image 71iL, and the corneal center E21, and the imaging plane 52 is represented by CPix. This is a projection image of the cutting plane CPx onto the imaging plane 52, and is therefore referred to as the cutting plane projection image CPix. Because the cutting plane CPx includes the GX axis, the cutting plane projection image CPix, which is its projection image, is parallel to the GX axis and the x axis.

[0089] 8 shows a method for calculating the projection point of the corneal center E21 onto the imaging plane 52 by drawing a diagram. (A) of FIG. 8 shows an example of a state in which images acquired by the ocular camera 5 are processed to select images that may be Purkinje images and remove others. It is confirmed that there are three images that may be Purkinje images, and that these form an upwardly convex triangle when rotated 180 degrees by the imaging function. After confirmation, the three points that form this triangle are considered to be Purkinje images 71iR, 71iL, and 71iB, and processing continues.

[0090] In FIG. 1B, the two-dimensional coordinates of the Purkinje images 71iR, 71iL, and 71iB are calculated as (x coordinate value, y coordinate value). The x coordinate value is calculated according to the distance from the y axis, and the y coordinate value is calculated according to the distance from the x axis. As a result, 71iR(R x ,R y ), 71iL(L x ,L y ) and 71iB(B x ,By )

[0091] The infrared light sources 55R and 55L that cause the Purkinje images 71iR and 71iL are located on the GX axis. Therefore, on the imaging plane 52, both Purkinje images 71iR and 71iL are projected onto the cutting plane projection image CPix of the cutting plane CPx. The infrared light source 55B that causes the Purkinje image 71iB is located on the GY axis. Therefore, on the imaging plane 52, the Purkinje image 71iB is projected onto the cutting plane projection image CPiy of the cutting plane CPy. However, these cutting plane projection images are calculated and do not actually appear on the imaging plane 52 (see FIG. 1B).

[0092] A straight line is drawn between the first Purkinje image 71iL and the second Purkinje image 71iR, and this is designated as the first projection line HL. Since this first projection line is a line segment of the cutting plane projection image CPix, it must be parallel to the x-axis. If it is not parallel, there is a possibility that an image related to diffuse reflection, not a Purkinje image, has been detected, so the Purkinje image may be detected again.

[0093] Normally, since the first projection line HL is parallel to the x-axis, the y coordinate of the first Purkinje image 71iR and the y coordinate of the second Purkinje image 71iL should be equal, and this is defined as ry. If the difference between the y coordinates of the two is within an allowable range, the average value of these is defined as r. y The first projection line HL connecting the Purkinje images 71iR and 71iL forms the base of an upwardly convex triangle [Fig.

[0094] As described above, the Purkinje image 71iB is generated on the cutting plane projection image CPiy of the cutting plane CPy. As described above, the cutting plane projection image CPiy is parallel to the y-axis. Therefore, a straight line passing through the third Purkinje image 71iB and parallel to the y-axis is drawn and set as the second projection line VL. Alternatively, a straight line intersecting the first projection line HL at 90° may be drawn and set as the second projection line VL.

[0095] In the group of three Purkinje images forming the triangle, a second projection line VL is drawn from the Purkinje image 71iB that forms the vertex angle to the first projection line HL. Then, the intersection of the first projection line HL and the second projection line VL is calculated. This is also the intersection of the cutting plane projection images CPix and CPiy. This intersection is designated as E21i (Fig. 1(D)).

[0096] As mentioned above, the corneal center E21 exists somewhere on the cutting plane CPx, and the corneal center E21 exists somewhere on the cutting plane CPy. In this case, the corneal center E21 exists on the projection line of the intersection point between the cutting plane CPx and the cutting plane CPy. Therefore, the intersection point will be referred to as the corneal center projection point E21i.

[0097] Through the above drawing, the two-dimensional coordinate is E21i(B x ,r y ) Here again, C x ≡B x ,C y ≡r y Toki, E21i(C x ,C y The two-dimensional coordinates of the corneal center E21 are treated as known values.

[0098] <Processing block diagram> Next, it is necessary to calculate the distance from the reference position to the corneal center E21. Before that, the flow of a series of processes in a gaze analysis device according to a first embodiment based on the concept of the present invention is shown. FIG. 9 is a processing block diagram of a gaze analysis device according to a first embodiment based on the concept of the present invention. The gaze analysis device calculates the gaze direction of the subject through processing including blocks A1 to A8. The processing included in these blocks may be executed by dedicated hardware, or may be processed by a calculation unit 61 according to program instructions. In the figure, arrows between processing blocks indicate the flow of information.

[0099] Block A1 is an eyeball image acquisition unit. This involves projecting an image of the eyeball, which is a three-dimensional object in three-dimensional space, as a two-dimensional image onto imaging surface 52 through the imaging action of lens 51, digitizing it using imaging elements arranged there, and then capturing it as electronic data in image area 622. This electronic data is then processed in blocks A2 and A5. Alternatively, the electronic image data may be directly processed as described below, separate from being captured in image area 622.

[0100] Block A5 is a two-dimensional processing block, also called the first eyeball image analysis unit. Here, image data captured via imaging surface 52 is processed according to a two-dimensional coordinate system x·y. This block detects the pupil from the eyeball image. As mentioned above, the eyeball is photographed under illumination by an infrared light source placed around eyeball camera 5, which has sensitivity in the infrared region. This allows for the capture of an image that is less affected by ambient light.

[0101] In an eyeball image, the sclera E1, iris E3, and pupil E31 can be relatively easily distinguished because they each have different reflectances of infrared light. In this way, the pupil can be detected by utilizing the difference in reflectance of irradiated light between the pupil and other areas. The pupil and its surroundings are the cornea E2 area. The reflected light used in this analysis is dominated by diffused light from the iris that passes through the cornea E2 and reaches the lens 51 and imaging plane 52. In this block, the range of the pupil area in the eyeball image is determined as two-dimensional coordinates.

[0102] Block A6 is a two-dimensional processing block that detects the center of the pupil region detected in Block A5. Since the pupil can be approximated by a circle or ellipse, the two-dimensional center coordinates can be obtained by fitting, for example, using a circle detection Hough transform or other open source functions such as the ellipse detection function in OpenCV. This block A6 is also called the two-dimensional pupil center detection unit.

[0103] Block A2 is a second eye image analysis unit, a two-dimensional processing block. Here, infrared light from an infrared light source arranged around the eye camera 5 is detected as an image specularly reflected on the cornea. Specular reflection occurs on the cornea. However, in this case, it means obtaining a specular reflection image as a two-dimensional image captured by the imaging action of the lens of the eye camera 5. This specular reflection image is the Purkinje image mentioned above. Since the Purkinje image has high brightness, it is easy to distinguish it from an image caused by diffuse reflection light on the cornea. On the other hand, diffuse reflection light from the sclera can also form an image with high brightness, which makes it difficult to distinguish it from a Purkinje image.

[0104] Therefore, based on the concept of the present invention, three or more infrared light sources are arranged around the optical axis of the eye camera 5 so that the arrangement of the Purkinje images forms a specific polygon. However, this shape must be useful for identifying the two-dimensional position of the corneal center E21, as the arrangement of the Purkinje images. Here, the two-dimensional position of the corneal center E21 refers to the position where the corneal center E21 is projected onto the imaging plane 52. Once the projected position is known, it is possible to calculate the direction starting from the origin GO. This is the direction in which the corneal center E21 exists.

[0105] In the gaze analysis terminal T1 of the first embodiment, three infrared light sources are arranged together with the eyeball camera 5 so that the arrangement of the Purkinje images forms a specific triangle. Then, three Purkinje images that roughly match this shape are detected as the first Purkinje image, the second Purkinje image, and the third Purkinje image. Details of this process have already been given.

[0106] Block A3 is a two-dimensional corneal center detection block, which performs two-dimensional processing. This block calculates two-dimensional coordinates indicating the direction of the center of curvature (corneal center E21) when the corneal surface is considered to be a sphere. This is calculated as the foot of a perpendicular line drawn from the apex angle of the triangle formed by the first, second, and third Purkinje images to the base (Figure 8(D)).

[0107] Furthermore, the angle at which the line segment from the apex angle of the triangle intersects with the base is not limited to 90°, and the installation position of the third infrared light source 55B may be designed so that the line segment intersects with the base at a specified angle, as will be described later.

[0108] Block A4 is a corneal center depth calculation unit. This unit calculates the distance between the principal plane of the camera's lens and the corneal center by solving an equation that is formulated by adding constraints on the optical path in specular reflection and the average human corneal radius to the relationship between the position of the infrared light source arranged together with the eye camera 5, the center position of the camera's lens, and the position at which specular reflection occurs in the Purkinje image. This will be described in detail later.

[0109] Block A7 is a pupil center and corneal center calculation block, which is a three-dimensional processing block. Here, the two-dimensional coordinates of the pupil center obtained by block A6 and the two-dimensional coordinates of the corneal center obtained by block A3 are converted into three-dimensional coordinates based on the corneal center depth information obtained by block A4.

[0110] Block A8 is a first gaze vector calculation unit, which is a three-dimensional processing block. Here, a vector starting from the corneal center E21 and ending at the pupil center Pu is calculated, and this is set as the first gaze vector. The three-dimensional coordinates of the corneal center E21 (sometimes written as Cen) and pupil center Pu used here are those provided by block A7. The above is an overview of the gaze analysis processing according to the first embodiment.

[0111] Next, a specific example of processing in the corneal center depth calculation unit (block A4) is shown. FIG. 10 is a diagram showing that the corneal center E21 of the subject's right eye is located exactly on the optical axis (i.e., the GZ axis) of the lens 51. FIG. 10(A) is a GY-GZ cross-sectional view of the right eye E and the eye camera 5 in three-dimensional space. Because this cross-sectional view includes the GY axis and the corneal center E21, it is also the cutting plane CPy set by the infrared light source 55B installed on the GY axis. This is an example in which the GY-GZ plane and the cutting plane CPy coincide. The nose, shown by the dotted line, is shown to be oriented in this direction, but it does not exist on the GY-GZ plane or the cutting plane CPy.

[0112] Figure 1(B) is a GZ-GX cross-sectional view of the right eyeball E and eyeball camera 5 in three-dimensional space. This cross-sectional view includes the GX axis and the corneal center E21. It is also the cutting plane CPx set by the infrared light source 55R installed on the GX axis. Furthermore, this is also the cutting plane CPx set by the infrared light source 55L installed on the GX axis. This is an example where the GZ-GX plane and the cutting plane CPx coincide. The nose, shown by the dotted line, is shown to be in this orientation, but it does not exist on the GZ-GX plane or the cutting plane CPx.

[0113] FIG. 1C1 shows the subject's eyeball and Purkinje image projected onto the imaging plane 52. FIG. 1C2 shows Purkinje images 71iR, 71iL, and 71iB and x-y coordinates on the imaging plane 52. As mentioned above, the origin o of this coordinate system is the point where the GZ axis intersects perpendicularly with the second plane. The x- and y-axes of the vertical and horizontal directions shown in FIG. 1C1 and FIG. 1C2 are shown for the sake of convenience and do not necessarily have to be displayed on the imaging plane 52.

[0114] In Figure 2 (C2), the straight line CPiy in the vertical direction of the paper is the intersection line between the cutting plane CPy and the imaging plane 52, and is the projection image CPiy of the cutting plane CPy onto the second plane. In this case, the y-axis and the cutting plane projection image CPiy coincide with each other. This is because the cutting plane CPy coincides with the GY-GZ plane.

[0115] The straight line CPix in the left-right direction of the paper is the intersection line between the cutting plane CPx and the imaging plane 52, and is the projection image CPix of the cutting plane CPx onto the second plane. In this case, the x-axis and the cutting plane projection image CPix coincide with each other because the cutting plane CPx and the GZ-GX plane coincide with each other.

[0116] 11 is a diagram showing a typical case where the corneal center E21 of the subject's right eye is not located on the optical axis (i.e., the GZ axis) of the lens 51. Figure 11(A) shows the GY axis in three-dimensional space and the corneal center E21 of the right eyeball on the same plane, and also shows the cutting plane CPy set by the infrared light source 55B installed on the GY axis. The GY-GZ plane and the cutting plane CPy do not coincide.

[0117] In this case, the cutting plane CPy does not include the GZ axis. Instead, the intersection line between the GZ·GX plane and the cutting plane CPy, Intersection‐GZ·GX‐Plane, is shown by a dashed line.

[0118] The GY-GX plane and the cutting plane CPy intersect at a predetermined angle with the GY axis as the intersection line, and this angle is reflected in the position of the cutting plane projection image CPiy.

[0119] 1B shows the GX axis in three-dimensional space and the corneal center E21 of the right eyeball on the same plane, and also shows the cutting planes CPx set by the infrared light sources 55R and 55L installed on the GX axis. The GZ-GX plane and the cutting plane CPx do not coincide.

[0120] In this case, the cutting plane CPx does not include the GZ axis. Instead, the intersection line between the GY·GZ plane and the cutting plane CPx, Intersection‐GY·GZ‐Plane, is shown by a dashed line.

[0121] The GZ-GX plane and the cutting plane CPx intersect at a predetermined angle with the GX axis as the intersection line, and this angle is reflected in the position of the cutting plane projection image CPix.

[0122] Figure (C1) shows the subject's eyeball and Purkinje image projected onto the imaging plane 52. Figure (C2) shows the Purkinje images 71iR, 71iL, and 71iB and the x and y coordinates on the imaging plane 52. The cutting plane projection image CPiy is a line that passes through the Purkinje image 71iB and is parallel to the y axis. The cutting plane projection image CPix is ​​a line that passes through the Purkinje images 71iR and 71iL and is parallel to the x axis.

[0123] According to the drawing shown in FIG. 8, the intersection of the cutting plane projection images CPiy and CPix is ​​the corneal center projection point E21i. As mentioned above, E21i(C x ,C y ), and these coordinate values ​​are known.

[0124] The projection direction from the corneal center E21 through the origin GO to the corneal center projection point E21i is the inclination ∠Cθ from the GY·GZ plane. GY and the inclination ∠Cφ from the GZ·CX plane GX is calculated as equations (3-1) and (3-2) by applying equations (1-1) and (1-2) and equations (2-1) and (2-2).

[0125]

number

[0126] Figure 12 shows a cross-sectional view of the eyeball model in the general situation shown in Figure 11, where the corneal center E21 is off-axis from the optical axis of the lens 51 of the eyeball camera 5. The purpose of the analysis here is to determine the distance from the reference position to the corneal center. The eyeball model is a composite sphere in which the spherical surface of the cornea E2 is partially exposed from the anterior part of the sclera E1.

[0127] 1A shows GO, which is the origin of the three-dimensional coordinate system and the center of lens 51, the GX axis, and infrared light sources 55R and 55L on the same axis. Corneal center E21 is also shown. The GX axis, which is the first straight line, and corneal center E21 are shown on the same plane, which is the cutting plane CPx set by infrared light sources 55R and 55L installed on the GX axis.

[0128] Generally, the GZ axis is not included in the cutting plane CPx set by the infrared light source on the GX axis. As mentioned above, the GZ-GX plane, which includes the GZ axis, and the cutting plane CPx are tilted at a certain angle with the GX axis as the axis. Also, although not shown, there is a GY axis that is perpendicular to the GX axis at the origin GO, and its positive direction is toward the front of the paper.

[0129] The dashed-dotted line CPZi, which passes through the origin GO, is shown as the intersection of the cutting plane CPx and the GY-GZ plane. For ease of analysis, the CPZ axis, which starts at the origin GO and runs along the dashed-dotted line CPZi, is set on the cutting plane CPx.

[0130] The CPZ axis is introduced to set a two-dimensional coordinate system GX·CPZ on the cutting plane CPx. The origin is GO. As mentioned above, the inclination angle ∠Cφ of the cutting plane CPx with respect to the GZ·GX plane is GX is obtained by equation (3-2).

[0131] 12(A), infrared light source 55R is located on the GX axis, and its distance from the center of lens 51 on origin GO is ms. This distance is a fixed, designed value. A portion of the infrared light emitted from infrared light source 55R becomes specularly reflected light 71R on the surface of cornea E2 and enters center GO of lens 51.

[0132] As described above, the infrared light source 55R, the center GO of the lens 51, and the specular reflection point 71R are all on the cutting plane CPx, and the corneal center E21 is also on the same plane. The vector pointing from the corneal center E21 to the specular reflection point 71R is the normal vector N of the corneal surface E21 at the specular reflection point 71R.

[0133] The infrared light is incident on and reflected from the specular reflection point 71R at an angle equiangular to the normal vector N. On this cutting plane CPx, the distance between the GX axis and the corneal center E21 is defined as Len. Len is a value that can change depending on the relative positions of the eyeball E and the lens 51.

[0134] Furthermore, the angle between the CPZ axis and the direction from the origin GO, which is the center of the lens, toward the corneal center E21 is defined as ∠γ. Furthermore, the angle between the CPZ axis and the direction from the origin GO toward the specular reflection point 71R is defined as ∠α. Furthermore, the angle between the normal vector N and the CPZ axis is defined as ∠β. The values ​​of ∠γ, ∠α, and ∠β can change depending on the relative positions of the eyeball E and the lens 51.

[0135] For ∠γ and ∠α, the positive rotation direction is counterclockwise around the origin GO on the paper. For ∠β, the positive rotation direction is clockwise around the corneal center E21 on the paper. Also, the distance from the CPZ axis to the specular reflection point 71R is ms1, the distance to the infrared light source 55R is ms, and the difference between them is ms2. Therefore, ms = ms1 + ms2.

[0136] The specific value of the corneal radius R is the average corneal radius R ave As mentioned above, the value of ms is fixed, while the values ​​of ms1 and ms2 can change depending on the movement of the specular reflection position 71R, because the position of the specular reflection 71R changes depending on the orientation of the cornea E2.

[0137] A similar analysis may be performed using specular reflection point 71L generated by the light beam from infrared light source 55L instead of specular reflection point 71R generated by the light beam from infrared light source 55R. Either one of the two specular reflection points 71R or 71L, which are considered equivalent, may be selected for analysis and referred to as the reference specular reflection point, and the infrared light source that emitted the light beam may be referred to as the reference light source to distinguish them. In this case, the Purkinje image generated on imaging plane 52 via the reference specular reflection point will be referred to as the reference Purkinje image.

[0138] The GX axis here is also the line of intersection between the cutting plane CPx and the first plane. Therefore, this line of intersection is also called the analytical reference line. As mentioned above, the cutting plane CPx is angled by ∠Cφ with respect to the GZ·GX plane. GX It is just tilted.

[0139] The angle α shown in Figure 12(A) is angle Cφ with respect to the GZ·GX plane. GXThis angle α is the angle between the CPZ axis and the ray of light traveling from the specular reflection point 71R toward GO, the center of the lens 51, on the inclined cutting plane CPx. The CPZ axis is the axis located on the intersection of the cutting plane CPx and the GY-GZ plane. ∠α is calculated as follows, with reference to Figures 1(B) and 1(C).

[0140] First, the projection direction from the specular reflection point 71R to the Purkinje image 71iR is determined. As shown in FIG. 8, the coordinates of the Purkinje image are 71iR(R x ,C y ) This is because the y coordinates of the Purkinje image 71iR and the corneal central projection point are equal. Therefore, by applying equations (1-1), (1-2), (2-1), and (2-2) to these coordinates, we obtain ∠Rθ as equations (4-1) and (4-2). GY and ∠Cφ GX get.

[0141]

number

[0142] In Figures 12(B) and (C), the coordinates of the specular reflection point 71R in the three-dimensional space are 71R(R GX ,R GY ,R GZ ) where 71R is (GX-axis component, GY-axis component, GZ-axis component). As described above, the coordinates of the projected image onto the imaging surface 52, i.e., the Purkinje image 71iR, are (R x ,C y ) (not shown). (B) and (C) of the same figure also show the positional relationship between specular reflection point 71R and its surroundings in a three-dimensional coordinate system. However, the plane of interest and its tilt angle are different in each case.

[0143] In Figure 12(B), the rectangle indicated by 71R, p1, GO, and p2 is part of a plane uniquely defined by the object point 71R and the GX axis. This plane is the projection plane PRx of the specular reflection point 71R, and the Purkinje image 71iR (not shown) also lies on this plane. From equation (4-2), the projection plane PRx is angled by ∠Cφ from the GZ·GX plane. GXIt is an inclined plane. And as mentioned above, this projection plane PRx is also the cutting plane CPx shown in Figure 1(A). Then, in Figure 1(B), the line connecting GO and p2 is the CPZ axis in Figure 1(A). And the angle α to be found is the angle between the CPZ axis and the line connecting the origin GO and the specular reflection point 71R.

[0144] In Figure 12(C), the rectangle indicated by 71R, p5, GO, and p3 is part of the projection plane PRy, which is uniquely defined by the object point 71R and the GY axis, and is the projection plane PRy of the specular reflection point 71R. The Purkinje image 71iR (not shown) is also on this projection plane PRy. From equation (4-1), the projection plane PRy is angled by ∠Rθ from the GY·GZ plane. GY It is an inclined plane. Furthermore, by focusing on the triangle GO·p4·p3, equation (5-1) can be derived using trigonometric functions.

[0145]

number

[0146] Next, we focus on the triangle GO·p4·p2 in Figure 12(B). Using trigonometric functions, with the length of the line segment GO·p2 as u, we can derive equation (5-2). Furthermore, by focusing on the triangle GO·p2·71R and the triangle GO·p4·p3 in Figure 12(C), we can derive equations (5-3) and (5-4) by using trigonometric functions. The angle Rθ on the right-hand side of equation (5-4) GY and ∠Cφ GX Since is known, ∠α is also known.

[0147]

number

[0148] FIG. 13 is a diagram of the corneal center E21 on a three-dimensional coordinate system. In (A) and (B) of the same figure, the coordinate of the corneal center E21 in the three-dimensional space is E21(C GX ,C GY ,C GZ) where E21 is (GX axis component, GY axis component, GZ axis component). As described above, the coordinates of the projected image onto the imaging surface 52, that is, the corneal center projected point E21i, are (C x ,C y ) (not shown). (A) and (B) in the same figure show the position of the corneal center E21 in the same three-dimensional coordinate system, but the plane of interest and its inclination angle are different.

[0149] In Figure 1(A), the rectangle indicated by E21, q1, GO, and q2 is part of the projection plane PRx, which is uniquely defined by the object point E21 and the GX axis. The corneal center projection point E21i (not shown) is also on this plane. Since this plane includes the corneal center E21 and the GX axis, it is also the cutting plane CPx set by the infrared light source on the X axis. In this case, the line connecting GO and q2 in Figure 1(A) is the CPZ axis mentioned above. The angle ∠γ to be found is the angle between the CPZ axis and the line connecting the origin GO and the corneal center E21. The inclination angle of the cutting plane CPx with respect to the GZ·GX plane is ∠Cφ, calculated using equation (4-2). GX is.

[0150] In addition, in FIG. 1B, the rectangle E21·q5·GO·q3 is a part of the projection plane PRy that is uniquely defined by the object point E21 and the GY axis. The tilt angle of the projection plane PRy with respect to the GY-GZ plane is calculated by ∠Cθ GY Here, focusing on the triangle GO·q4·q3, equation (6-1) can be derived using trigonometric functions.

[0151]

number

[0152] As mentioned above, the two-dimensional coordinate system GX·CPZ is set on the cutting plane CPx. Also as mentioned above, Len is the distance from the corneal center to the GX axis, also known as the analytical reference line. Therefore, in Figure 13(A), the length of the line segment connecting E21 and q1 is Len. The length of the line segment connecting GO and q2 on the CPZ axis is also Len.

[0153] Next, we focus on the triangle GO·q4·q2 in Figure (A). Since the length of the line segment GO·q2 is Len, we can derive equation (6-2) using trigonometric functions. Furthermore, we can derive equations (6-3) and (6-4) by focusing on the triangle GO·q2·E21 and the triangle GO·q4·q3 in Figure (B) and using trigonometric functions. ∠Cθ on the right-hand side of equation (6-4) GY and ∠Cφ GX Since is known, ∠γ is also known.

[0154]

number

[0155] As described above, by analyzing the two-dimensional coordinates of the three Purkinje images detected on the imaging plane 52, the angle ∠Cφ formed by the cutting plane CPx and the GZ·GZ plane can be calculated. GX and the angle ∠Cθ between the cutting plane CPy and the GY·GZ plane GY Furthermore, in the GX·CPZ coordinate system, which is a two-dimensional coordinate system on the cutting plane CPx, it is possible to calculate the angle ∠α formed by the line segment from the origin GO toward the reference specular reflection point and the CPZ axis, and the angle ∠γ formed by the line segment from the origin GO toward the corneal center E21 and the CPZ axis. Hereinafter, these will be treated as known values. On the other hand, Len is not a known value at this stage.

[0156] Referring again to Figure 12, in the two-dimensional GX-CPZ coordinate system established on the cutting plane CPx shown in Figure 12(A), the coordinates of the corneal center E21 can be expressed by equation (7-1). However, the coordinates are expressed as (GX component, CPZ component). The coordinates of the specular reflection point 71R can be expressed by equation (7-2). The coordinates of the infrared light source 55R can be expressed by equation (7-3).

[0157]

number

[0158] The normal vector N is a vector pointing from the corneal center E21 to the specular reflection point 71R, so it can be expressed by equation (8-1). Also, since the magnitude of the normal vector N is equal to the corneal radius R, the squares of these are also equal, so equation (8-2) holds.

[0159]

number

[0160] Furthermore, in the GX coordinates of the specular reflection point 71R, the distance ms from the origin GO to the infrared light source 55R is divided into ms1 and ms2, which is given by equation (9-1). Then, ms1 becomes equation (9-2). And ms2 becomes equation (9-3). Furthermore, the distance Dis from the GX axis to the specular reflection point 71R can be expressed by equation (9-4).

[0161]

number

[0162] Substituting equation (9-4) into equation (9-2) and eliminating Dis gives equation (10-1). Then, solving equation (10-1) for Len gives equation (10-2). Here, the corneal radius R is set to the average value R ave In equation (10-2), Len cannot be determined because ∠β is also an unknown in addition to Len. On the other hand, if equations (9-2) and (9-3) are substituted into equation (9-1) and enclosed with Dis, equation (10-3) is obtained, and if equation (9-4) is substituted into this and Dis is eliminated, equation (10-4) is obtained. Then, if equation (10-2) is substituted into equation (10-4) and Len is eliminated, equation (10-5) is obtained.

[0163]

number

[0164] Focus on equation (10-5). The only unknown here is ∠β. Therefore, ∠β is calculated from this equation. This can be found analytically, but it is also possible to calculate the right-hand side of equation (10-5) successively using the range of values ​​that β can take, and determine the β value when the difference with the left-hand side is within an acceptable value. After ∠β is determined, Len is determined using equation (10-2). As shown above, determining the value of Len completes the specific processing of the corneal central depth calculation unit (block A4).

[0165] The key points of the manipulation of the above formulas can be summarized as follows: Either the first Purkinje image 71iR or the second Purkinje image 71iL is selected as the reference Purkinje image, the infrared light source that causes the reference Purkinje image is referred to as the reference infrared light source, and the specular reflection point serving as the object point of the reference Purkinje image is referred to as the reference specular reflection point. The intersection line between the first plane and the cutting plane CPx coincides with the first straight line, which includes the center of the lens 51 and coincides with the GX axis.

[0166] On the cutting plane CPx, a straight line is provided as the CPZ axis perpendicular to the GX axis at the center GO of the lens 51, thereby setting a two-dimensional coordinate system GX·CPZ with GO as the origin on the cutting plane CPx.

[0167] The radius of curvature R of the cornea E2, Len, and ∠β cannot be calculated directly from the Purkinje images, and it is necessary to solve equations derived based on the constraints of specularly reflected infrared light. Therefore, the first and second conditional expressions are derived based on the conditions under which a physical phenomenon occurs in which a light ray emitted from a reference infrared light source is specularly reflected at a reference specular reflection point and incident on the origin GO, which is the center of the lens 51.

[0168] By applying trigonometric functions to the parameters ms, ∠α, Len, R, and ∠β, a first conditional equation (equation (10-4)) is derived, which includes Len, R, and β as unknowns, based on the fact that, for each component of the GX axis and the CPZ axis, the vector from the reference infrared light source to GO is equal to the vector sum of the vector from the reference infrared light source to the reference specular reflection point and the vector from the reference specular reflection point to GO.

[0169] Furthermore, by applying trigonometric functions to the parameters ms, ∠γ, Len, R, and β, a second conditional equation (equation (10-1)) is derived, which includes Len, R, and β as unknowns, based on the fact that for each component of the GX axis and the CPZ axis, the vector from GO to the reference specular reflection point is equal to the vector sum of the vector from GO to the center of the cornea and the vector from the center of the cornea to the reference specular reflection point.

[0170] Next, the unknown Len is eliminated from the first and second conditional expressions, and the third conditional expression (equation (10-5)) containing R and β as unknowns is derived.

[0171] Then, as the mean value substitution process, the unknown quantity R included in the third conditional expression is assigned the average human corneal radius R. ave Then, solve the third conditional expression to make β a known quantity. Furthermore, the corneal radius R ave By substituting β, which has become a known quantity, into the first or second conditional expression, the only unknown quantity is Len, and then this is solved to make Len a known quantity. This is the gist of the mathematical operations in the corneal central depth calculation unit.

[0172] Next, a specific example of the pupil center and corneal center calculation unit (block A7) is shown. Here, the three-dimensional coordinates (C GX ,C GY ,C GZ) is calculated. However, the three-dimensional coordinates are expressed as (GX component, GY component, GZ component). Len is known from block A4. Len is the distance from the GX axis (which is also the analytical reference line) to the corneal center E21, and is the distance from the GX axis to the corneal center E21 on the cutting plane CPx shown in Figure 12(A) and Figures 13(A), (B), and (C).

[0173] FIG. 13(C) shows the three-dimensional coordinate system and the coordinate of the corneal center E21, E21(C GX ,C GY ,C GZ In the figure, the quadrangle GO·q2·E21·q1 is a plane included in the cutting plane CPx, and the angle Cφ formed by this plane and the GZ·GX plane is GX is known from equation (3-2).

[0174] In addition, the quadrilateral GO·q3·E21·q5 is a plane included in the cutting plane CPy, and the angle Cθ formed by this plane and the GY·GZ plane GY is known from equation (3-1). And, from the analysis in block A4 above, the length Len between the GX axis and the corneal center E21 measured along the cutting plane CPx is known.

[0175] Therefore, we apply trigonometric functions by referring to Figure 13(C). Focusing on the triangle GO·q4·q2, we can see that C GY is known, and C GZ Furthermore, when we focus on the triangle GO·q4·q3, we can obtain C GX becomes known.

[0176]

number

[0177] Next, referring to FIG. 14, the three-dimensional coordinates of the pupil center Pu(P GX ,P GY ,P GZ) is shown below. Figure 14 is a diagram showing the positional relationship between the corneal center and the pupil center in a three-dimensional coordinate system. As shown in Figure 14(A), a rectangular parallelepiped can be imagined with the pupil center Pu and the origin GO as vertices. In the figure, in addition to Pu and GO, a1, a2, a3, and a5 are shown as vertices with their coordinates. The size of the rectangle is P GX and P GZ And the height is P GY is.

[0178] Light rays diffusely reflected by the iris E3 form an image on the imaging plane 52. Equations (1-1) and (2-1) are applied to the positions of the images of the iris E3 and pupil E31 projected onto the imaging plane 52 to find the angle between the projection plane PRy and the GY·GZ plane. Furthermore, the angle between the projection plane PRx and the GZ·GX plane is found.

[0179] In FIG. 1A, the rectangle Pu·a1·GO·a2 is a plane included in the projection plane PRx about the GX axis related to the pupil center Pu as an object point. x ,p y ) (not shown), then, from equation (2-1) to equation (12-2), ∠Pφ GX is calculated. In addition, the rectangle Pu·a3·GO·a5 is a plane included in the projection plane PRy of the pupil center Pu as an object point with respect to the GY axis. Therefore, from equation (1-1) to equation (12-1), ∠Pθ GY is calculated.

[0180]

number

[0181] ∠Pθ is now known GY and ∠Pφ GX Using the above, the three-dimensional coordinates of the pupil center Pu(P GX ,P GY ,P GZ ) can be expressed as equations (12-3), (12-4) and (12-5). However, P in equation (12-5) GZ In =t, t is an unknown.

[0182]

number

[0183] Here, the radius of the cornea E21 is r=R ave The average value for humans is applied as a known value. Then, the relationship between the corneal center E21 and the pupil center Pu can be expressed as equation (13-1). By substituting equations (12-3), (12-4), and (12-5) into this equation, equation (13-2) is obtained. By arranging equation (13-2) in descending order with respect to t, a quadratic equation (13-3) for the variable t is obtained. However, the coefficients K1, K2, and K3 are as shown below and are known values.

[0184]

number

[0185] Solving the quadratic equation (13-3) with the solution formula gives us equation (13-4), which contains a complex sign in the numerator. See Figure 14(B).

[0186]

number

[0187] 14(B) is a cross-sectional view showing the positional relationship between the lens 51 of the eye camera 5, the cornea E2, and the pupil center Pu. A dashed line indicates a straight line extending from the origin GO of the three-dimensional coordinate system, which is also the center of the lens 51, through the pupil center Pu0 to Pu1. This dashed line is a projection line that projects the pupil center Pu, which is an object point, onto the pupil center image Pui on the imaging plane 52. As mentioned above, this line has a slope of ∠Pθ GY Projection plane PRy and inclination ∠Pφ GX is the line of intersection with the projection plane PRx.

[0188] The projection line shown by the dashed line intersects with the sphere of the cornea E2 at two points, Pu0 and Pu1. Both of these intersections are at a distance R from the corneal center E21.ave Since it is located at , it is the solution of the quadratic equation (13-3). The pupil center Pu to be found here is Pu0, which is closer to the origin GO. Therefore, the t to be found is expressed by equation (13-5). Now that t is a known value, the pupil center Pu (P GX ,P GY ,P GZ ) and the three-dimensional coordinates of

[0189] Next, the first gaze vector Gaze1(g1 GX ,g1 GY ,g1 GZ ) is calculated. However, the notation of the three-dimensional coordinates is (GX component, GY component, GZ component). This is obtained by equation (14). Since the values ​​on the right side are all known, the first gaze vector Gaze1 can be calculated from this equation.

[0190]

number

[0191] 14(C) is a diagram showing the first gaze vector Gaze1 on the eyeball E, with the corneal center E21 (Cen) as the starting point and the pupil center Pu as the end point. The gaze vector calculation unit (block A8) has been described above. The gaze analysis device according to the first embodiment can accurately calculate the gaze vector even if the center of the cornea of ​​the subject is positioned to the left, right, up, or down from the optical axis of the eyeball camera 5.

[0192] <Second embodiment> <Processing block diagram> Next, a second embodiment of a gaze analysis device based on the concept of the present invention will be described. In the second embodiment, the gaze analysis terminal T1 included in the first embodiment is used as is, so a description of the gaze analysis terminal will be omitted. The processing blocks of the second embodiment that differ from the first embodiment will be described.

[0193] 15 is a processing block diagram of a second embodiment of a gaze analysis device based on the concept of the present invention. Blocks A1 to A8 indicated by double solid or dashed lines are similar to the processing in the first embodiment, and therefore their explanation will be omitted. Blocks A63, A9, A10, and A11 are processing blocks added in the second embodiment.

[0194] The blocks indicated by dashed lines indicate that they may not function in the gaze vector calculation cycle. As mentioned above, this occurs when the Purkinje image group is not properly detected in block A2. The dashed arrows indicate that, as mentioned above, a specific block may not function and therefore information may not be transmitted to downstream blocks.

[0195] In a cycle in which the blocks indicated by dashed lines do not function, calculation of the first gaze vector in block A8 is impossible. Therefore, the second embodiment is configured so that calculation of the second gaze vector is possible using block A11 even when this series of blocks temporarily does not function.

[0196] A63 is a two-dimensional rotation center calculation unit, which is a two-dimensional processing block. Here, a two-dimensional gaze vector is calculated from the two-dimensional coordinates of the corneal center detected in block A3 and the two-dimensional coordinates of the pupil center detected in block A6. Although the gaze vector is originally a vector in three-dimensional space, it is calculated as a two-dimensional vector projected onto the imaging surface 52 of camera 5.

[0197] As the eyeball rotates, the gaze vector also changes, but if a line is drawn extending from the starting point of the two vectors before and after the change, the two lines will intersect at a single point. This intersection is the point where the center of rotation E11 of the eyeball E is projected onto the imaging surface 52, and will be referred to as the rotation center projection point E11i. In this way, a drawing process is performed to determine the intersection of the two lines that match the directions of the gaze vector before and after the direction change. Because this process is a numerical calculation in memory, it does not matter if the line extends beyond the imaging surface 52.

[0198] The coordinate values ​​of the rotation center projection point E11i calculated here are sent to the next block A9. As mentioned above, if block A2 does not function, the two-dimensional coordinates of the corneal center cannot be obtained from block A3. In that case, a two-dimensional line of sight vector cannot be created, and a straight line related to this vector cannot be drawn. Therefore, the rotation center projection point E11i of the rotation center E11 of the eyeball E cannot be calculated.

[0199] Block A9 is a three-dimensional center of rotation and radius of rotation calculation unit, which is a three-dimensional processing block. If possible, it obtains the two-dimensional coordinate of the center of rotation projection point E11i from block A63, and the three-dimensional coordinate of the corneal center and the three-dimensional coordinate of the pupil center from block A7. If necessary, it obtains the two-dimensional coordinate of the pupil center from block A6.

[0200] Here, the three-dimensional coordinate of the center of rotation is calculated based on information obtained from blocks A63 and A7. However, in cycles in which block A2 does not function, blocks A63 and A7 also do not function, and the three-dimensional coordinate of the center of rotation cannot be calculated. To prepare for such a situation, block A9 saves the three-dimensional coordinate of the center of rotation calculated in the previous cycle and updates it each time a new calculation is made. Unlike the position of the corneal center, the position of the center of rotation does not change much with the rotation of the eyeball E, so this is handled in this way.

[0201] Furthermore, if the three-dimensional coordinates of the corneal center and pupil center cannot be obtained from block A7, the three-dimensional coordinate of the pupil center is calculated using the stored three-dimensional coordinate of the center of rotation and radius of rotation, and the two-dimensional coordinate of the pupil center derived from block A6. This completes the information processing in block A9.

[0202] Block A10 is second gaze vector calculation unit 1, a three-dimensional processing block. Here, a second gaze vector is calculated, starting from the rotation center coordinates obtained by block A9 and ending at the pupil center coordinates obtained by block A7. However, if block A7 does not function, the pupil center coordinates obtained by block A7 cannot be obtained. In such a case, the second gaze vector is calculated by second gaze vector calculation unit 2 in block A11.

[0203] Block A11 is the second gaze vector calculation unit 2, a three-dimensional processing block. Here, a second gaze vector is calculated, starting from the center of rotation coordinates stored in block A9 and ending at the pupil center coordinates. The pupil center is found by solving a quadratic equation using the two-dimensional pupil center coordinates derived from block A6 and the radius of rotation stored in block A9. This process allows the second gaze vector to be calculated even if block A7 is not functioning.

[0204] The line of sight vectors of blocks A10 and A11 are referred to as "second line of sight vectors" to distinguish them from the first line of sight vector of block A8. In principle, the second line of sight vector and the first line of sight vector have the same direction and orientation, and the second line of sight vector replaces the first line of sight vector.

[0205] Next, a specific example of block A63 will be shown. Fig. 16 is a diagram showing an example in which the eyeball E rotates and the gaze changes up and down. As shown in Fig. 16(A), the gaze vector Gaze1 is expressed as a vector in three-dimensional space with the center of the cornea of ​​the eyeball E as the starting point and the center of the pupil as the ending point. As the eyeball E rotates, the direction of the gaze vector Gaze1 also changes.

[0206] Figure 1B is a conceptual diagram showing the overlapping display of changes in the gaze. The starting points of multiple gaze vectors Gaze1 do not coincide. This indicates that the corneal center E21, which is the starting point of Gaze1, does not coincide with the center of rotation E11 of the eyeball E.

[0207] To grasp the movement of gaze vector Gaze1 on the imaging plane 52, a two-dimensional gaze vector is drawn with the two-dimensional coordinates of the cornea center obtained in block A3 as the start point and the two-dimensional coordinates of the pupil center obtained in block A6 as the end point. This two-dimensional gaze vector is called Gaze1_i (Fig. 1C).

[0208] This Gaze1_i corresponds to the line of sight vector Gaze1 projected onto the imaging surface 52. In FIG. 1C, the image is rotated 180° to match FIG. 1A and FIG. 1B. Images other than the projected image E31i of the pupil E31 are omitted.

[0209] Here, a straight line Gaze1_i_ex is drawn that runs in the same direction as the projected gaze vector Gaze1_i and passes through the corneal center projection point E21i, which is the starting point of the vector. Hereinafter, this straight line will also be referred to as the two-dimensional extension straight line Gaze1_i_ex [Figure 1(C)].

[0210] When the eyeball E turns and the direction of the gaze vector Gaze1 changes, the directions of the two-dimensional gaze vector Gaze1_i and the two-dimensional extended straight line Gaze1_i_ex also change. The two-dimensional extended straight lines Gaze1_i_ex before and after the turn intersect at approximately one point.

[0211] This point of intersection is the point where the center of rotation E11 of the eyeball E is projected onto the imaging plane 52, and is referred to as the center of rotation projection point E11i. However, since this center of rotation projection point E11i is a calculated point and not an image formed by projecting a corresponding part of the eyeball in three-dimensional space, it may extend beyond the imaging plane 52.

[0212] Each time the eyeball E rotates and the gaze vector Gaze1 changes, a two-dimensional extension line Gaze1_i_ex is drawn based on the two-dimensional gaze vector Gaze1_i, and the intersection of the two-dimensional extension lines Gaze1_i_ex calculated in the preprocessing cycle can be calculated. The two-dimensional coordinates of the rotation center projection point E11i are updated with the coordinates of this intersection.

[0213] However, when the subject's eyeball E hardly rotates, for example, because the subject is gazing at a single point, two-dimensional gaze vectors Gaze1_i in almost the same direction may be calculated one after another. In this way, the position of the intersection between two-dimensional extension lines Gaze1_i_ex that are nearly parallel to each other tends to fluctuate and become unstable. Therefore, if the angle of the newly obtained two-dimensional gaze vector Gaze1_i differs from that obtained in the previous cycle by, for example, 5° or more, the rotation center projection point E11i can be updated with the coordinates of the intersection between the two two-dimensional extension lines Gaze1_i_ex. The above is a specific example of the processing of block A63.

[0214] Next, a specific example of the three-dimensional processing in block A9 is shown. The coordinates of the turning center projection point E11i are known by the preceding block A63. As shown with reference to FIG. 16(C), if the angle between the two most recently obtained two-dimensional extended lines Gaze1_i_ex is equal to or greater than a predetermined angle, the intersection of these two lines is set as the turning center projection point E11i.

[0215] Next, we will explain how to calculate the three-dimensional coordinates of the center of rotation. Figure 17 is a diagram showing the process for obtaining the three-dimensional coordinates of the center of rotation Ro (E11) and the pupil center Pu. Through the process described above, the three-dimensional coordinates of the pupil center Pu (P GX ,P GY ,P GZ ) and the three-dimensional coordinates of the corneal center Cen(C GX ,C GY ,C GZ ) are known. Accordingly, the pupil center position vector Pu, the corneal center position vector Cen, and the rotation center position vector Ro, which start from the origin GO, can be expressed as equations (15-1), (15-2), and (15-3), respectively. Furthermore, the first gaze vector Gaze1 (g1 GX ,g1 GY ,g1 GZ ) Equation (15-4) holds true for the turning center position vector Ro(r GX ,r GY ,r GZ ) is unknown.

[0216]

number

[0217] Pupil center projection coordinates Pui(p x ,p y ) and the corneal central projection point E21i(C x ,C y As mentioned above, the rotation center projection coordinate Roi (roi x ,roi y ) is also known from computational construction.

[0218] Here, the rotation center projection coordinate Roi(roi x ,roi y The angle between the projection plane PRy and the GY·GZ plane is defined as ∠Roθ GY The angle between the projection plane PRx and the GZ·GX plane is ∠Roφ GX Applying equations (1-1) and (2-1) to these, we get equations (16-1) and (16-2), respectively. The angle ∠Roθ derived here is GY and ∠Roφ GX is an index that represents the direction of projection from the origin GO of the three-dimensional coordinate system onto the imaging surface 52. At the same time, it is also an index that represents the direction of projection from the origin GO to the three-dimensional coordinate Ro(r GX ,r GY ,r GZ ) is also an indicator of the direction of

[0219]

number

[0220] And, ∠Roθ GY and ∠Roφ GX and the three-dimensional coordinates Ro(r GX ,r GY ,r GZ ), equations (16-3), (16-4) and (16-5) hold, where m is an unknown.

[0221]

number

[0222] Let us now refer to Figure 17(A) and the enlarged partial view (A-α). The vector with Cen as its starting point and Ro as its end point is opposite to the vector Gaze1, but their directions are the same, so this can be formulated as equation (17-1) using the unknown n. This can be expressed as equation (17-2) for each component, and the unknowns m and n can be solved to become known quantities as in (18-1) and (18-2).

[0223]

number

[0224]

number

[0225] Applying the known number m to equations (16-3), (16-4) and (16-5), the three-dimensional coordinate of the turning center Ro(r GX ,r GY ,r GZ ) is known. Next, the three-dimensional coordinate Ro(r GX ,r GY ,r GZ ) and the distance Rot_R between the pupil center Pu and the center of rotation E11 are calculated and saved. This is a measure to prepare for the case where the acquisition of Purkinje images is not working properly and blocks A2, A3, A4, A7, and A8 do not function. However, this processing is only performed in cycles where blocks A2, A3, A4, and A7 are functioning.

[0226] If the above block does not function, the three-dimensional coordinate Ro(r GX ,r GY ,r GZ) and the distance Rot_R between the pupil center Pu and the center of rotation E11, and then the line of sight vector is calculated. The square of Rot_R is obtained by equation (19). The distance Rot_R is the radius of rotation as mentioned above, and is a fixed value. A specific example of processing block A9 has been shown above.

[0227]

number

[0228] The previously shown Gaze1 is a gaze vector that starts at the coordinate Cen of the corneal center E21 and ends at the pupil center Pu. In contrast, the gaze vector shown below starts at the coordinate Ro of the center of rotation E11 and ends at the pupil center Pu. Hereafter, this gaze vector will be referred to as Gaze2.

[0229] In block A10, the three-dimensional coordinate Ro(r GX ,r GY ,r GZ ) as the starting point, and the three-dimensional coordinate of the pupil center Pu(P GX ,P GY ,P GZ ) as the gaze vector Gaze2(g2 GX ,g2 GY ,g2 GZ ) is calculated. FIG. 17(B) is a diagram showing an example of the second gaze vector Gaze2 defined between the rotation center Ro (E11) and the pupil center Pu. The pupil center three-dimensional coordinate Pu (P GX ,P GY ,P GZ ) and the three-dimensional coordinate Ro(r GX ,r GY ,r GZ ) to the second gaze vector Gaze2(g2 GX ,g2 GY ,g2 GZ ) is determined.

[0230] Second gaze vector Gaze2(g2 GX ,g2 GY ,g2 GZ) are calculated as equations (20-1), (20-2), and (20-5). Here, the three-dimensional coordinates of the pupil center and the center of rotation are expressed by equations (20-3) and (20-4), respectively.

[0231]

number

[0232] In block A11, the vector starting from the three-dimensional coordinate of the center of rotation E11 and ending at the three-dimensional coordinate of the pupil center Pu is called the gaze vector Gaze2 (g2 GX ,g2 GY ,g2 GZ ) which is also calculated using equation (20-1). Block A11 needs to function when block A7 does not function, and in this case the three-dimensional coordinates of the pupil center Pu cannot be obtained. Therefore, block A11 uses the three-dimensional coordinates Ro of the turning center calculated and saved in the previous cycle, and the turning radius Rot_R, which is the distance from the turning center E11 to the pupil center Pu.

[0233] Furthermore, the two-dimensional coordinates Pui(p x ,p y ) to calculate the direction of the projection line from the origin GO of the three-dimensional coordinate system to the pupil center Pu. When applying equations (1-1) and (2-1) to the calculation, the above-mentioned equations (12-1) and (12-2) are used to obtain ∠Pθ GY and ∠Pφ GX is calculated.

[0234] As mentioned above, using these angles, the three-dimensional coordinates of the pupil center Pu can be expressed by equations (12-3), (12-4), and (12-5), where t is an unknown quantity.

[0235] Now, let us refer to FIG. 17(C). This figure is a cross-sectional view showing the positional relationship between the lens 51 of the eyeball camera 5, the pupil center Pu, and the rotation center Ro. With reference to this figure, we create a quadratic equation (21-1) for the square of the rotation radius Rot_R. Then, the coordinates of the pupil center Pu(P GX ,P GY ,P GZ ), we obtain quadratic equation (21-2) for unknown t. Rearranging this in descending order with respect to t gives quadratic equation (21-3). Here, Q1, Q2, and Q3 are expressed by equation (21-4), and since the right-hand sides of these are known values, we obtain equation (21-5) using the solution formula. However, this solution contains a complex sign.

[0236]

number

[0237]

number

[0238] There are two solutions to the quadratic equation (21-3), one of which corresponds to Pu1 (Figure 17(C)). Therefore, equation (21-6) is obtained as the solution for Pu0, which has the smaller GZ coordinate. Using the known t obtained as the solution, the three-dimensional coordinate of the pupil center Pu0 (P GX ,P GY ,P GZ ) to obtain equation (21-7).

[0239]

number

[0240] Thereafter, the second gaze vector Gaze2 (g2 GX ,g2 GY ,g2 GZ) is obtained. The second line of sight vector calculation unit 2 (block A11) has been described above, and the explanation of the processing block diagram according to the second embodiment is completed.

[0241] Next, a series of operations of the gaze analysis device of the first and second embodiments based on the concept of the present invention is shown in a flowchart. Figure 18 is a flowchart of the gaze analysis device based on the concept of the present invention. The gaze analysis device begins operation from "Start" and sets initial values ​​(stp1). Here, various parameters necessary for the operation of the gaze analysis device are set. One of these is the R_F flag, which is composed of at least one bit of memory and switches between two states, set and reset, and is referenced during the series of processes that follow.

[0242] The R_F flag is set when a valid three-dimensional coordinate Ro of the center of rotation has been saved or calculated, but is reset when it is determined that the position indicated by the three-dimensional coordinate has moved significantly and the positional relationship between the eye camera 5 and the eye E has changed.

[0243] As mentioned above, unlike the corneal center Cen, the position of the center of rotation Ro does not change much with the rotation of the eyeball E. The position changes when the subject moves the gaze analysis terminal T, for example, and the positional relationship between the eyeball E and the eye camera 5 changes. In such a case, the R_F flag is reset. After that, when the valid three-dimensional coordinate Ro of the center of rotation is calculated and updated, the R_F flag is set.

[0244] The validity of the three-dimensional rotation center coordinate Ro is determined, for example, as follows: If the coordinate of the rotation center Ro is calculated with a difference within the allowable error despite the rotation of the eyeball E, it is determined to be valid and the R_F flag is set. If the calculated coordinate exceeds the error range, the R_F flag is reset. The rotation of the eyeball E is determined by the movement of the pupil center. In other words, if the three-dimensional rotation center coordinate Ro shows approximately the same coordinate value despite the movement of the pupil center, it can be determined to be a valid three-dimensional rotation center coordinate.

[0245] Next, an eyeball image is acquired by the eyeball camera 5 and a field of view image is acquired by the field of view camera 4 (stp2). This is the image acquisition step. However, the eyeball image and the field of view image do not necessarily have to be acquired at the same time; for example, the field of view image may be acquired once after several shots of the eyeball image.

[0246] The field of view image is an image captured by the field of view camera 4, and captures a portion of the scene in front of the subject's eyes. Both the eyeball image and the field of view image are stored in the image area 622 of the storage unit 62, and are processed by the calculation unit 61 as necessary. However, the eyeball image may be processed by the calculation unit 61 directly after acquisition.

[0247] Next, the acquired eyeball image is processed to detect the pupil and its center point Pu. This is the pupil center detection step in 2D processing. If it cannot be detected, the result is 'false' and the process returns to stp2 to acquire another eyeball image. On the other hand, if the pupil is detected, the result is 'true' and the process proceeds to stp4 (stp3).

[0248] In stp4, the eye image is processed to detect Purkinje images. A Purkinje image is a collection of Purkinje images as a specific polygon formed by three or more infrared light sources. This is the Purkinje image detection step. If the Purkinje image cannot be detected, the result is 'false' and the state of the R_F flag is checked (stp7). On the other hand, if the Purkinje image can be detected, the result is 'true' and the process proceeds to stp5.

[0249] In stp7, if R_F is set, it is determined as 'true' and the second gaze vector is calculated (stp15), and the process proceeds to stp16. On the other hand, if R_F is reset, it is determined as 'false' and the process returns to stp2 to acquire an eyeball image again.

[0250] In stp5, the two-dimensional coordinates of the corneal center are calculated from the arrangement of the detected Purkinje images. This is the step for calculating the two-dimensional coordinates of the corneal center projection point.

[0251] Next, an equation is formulated by adding constraints on the optical path in specular reflection and the average human corneal radius to the relationship between the positions of the infrared light sources arranged around the eye camera, the center position of the camera lens, and the position at which specular reflection occurs in the Purkinje image, and then solving the equation, the distance Len from the analytical reference line, which is the intersection between the first plane, which is the principal plane of the camera lens, and the cutting plane on which the optical path in specular reflection falls, to the corneal center is calculated (stp6). This distance is corneal center depth information for obtaining the three-dimensional coordinates of the corneal center.

[0252] The two-dimensional coordinate of the pupil center obtained in stp3 and the two-dimensional coordinate of the cornea center obtained in stp5 are used to calculate three-dimensional coordinates based on the distance Len calculated in stp6, and three-dimensional coordinate values ​​are calculated (stp8). This is the pupil center / cornea center three-dimensional coordinate calculation step.

[0253] This is a step of calculating a first gaze vector whose starting point is the corneal center calculated in stp8 and whose ending point is the pupil center Pu calculated in stp8 (stp9).

[0254] Next, the three-dimensional coordinates of the turning center Ro and the turning radius are calculated using the method described above, and these values ​​are updated and stored (stp10).

[0255] The updated and stored three-dimensional coordinates of the turning center Ro are compared with the previously described turning radius, and if they are within the allowable error range, the three-dimensional coordinates of the turning center are determined to be valid (stp11). In this case, this is set to 'true', R_F is set (stp12), and the second line-of-sight vector is calculated based on these values ​​(stp14), and the process proceeds to stp16.

[0256] On the other hand, if the error is outside the allowable range, it is judged as 'false', R_F is reset (stp13) and the process proceeds to stp16.

[0257] In stp16, the line of sight vector to be output is selected from the first line of sight vector or the second line of sight vector, and the process proceeds to stp17. As a method for selecting the line of sight vector, the second line of sight vector may always be selected, or the second line of sight vector may be selected only when the R_F flag is set. Alternatively, if the conditional branch in stp4 is 'true', the first line of sight vector may always be selected. In this way, the selection conditions may be set in advance.

[0258] In stp17, the subject's gaze point is updated and displayed in the image of the view camera 4 according to the selected line-of-sight vector.

[0259] In stp18, if there is a stop or power-off interrupt, the result is 'true' and the series of gaze analysis operations ends. On the other hand, if there is no stop or power-off interrupt, the result is 'false' and the process proceeds to stp2, after which stp2 to stp17 are repeated until stp18 becomes true. Hereinafter, the series of operations from stp2 to stp17 may be referred to as the gaze vector calculation cycle.

[0260] It should be noted that if an end determination routine such as stp8 is provided between stp2 and stp3, the end determination can be made more reliably. The above is the flow of the operation of the line-of-sight analysis device based on the concept of the present invention.

[0261] Next, an application example of the gaze vector analysis device will be shown. Fig. 19 is a diagram showing an application example of gaze vectors. Referring to this figure, the relationship between the first gaze vector Gaze1 or the second gaze vector Gaze2 and the field of view camera 4 is shown. In the first or second embodiment, the subject's gaze point GP is superimposed on the image recorded by the field of view camera 4.

[0262] FIG. 1A shows subject A placing a gaze point GP on a product in a vending machine. However, in reality, the gaze point GP is only superimposed on image 45 (enlarged view η) captured by field of view camera 4. At this time, field of view camera 4 is capturing and recording the scene in front of the subject. The subject's gaze point GP is superimposed on recorded image 45. This may be done by superimposing the gaze point GP directly on the image recorded by the field of view camera, or by superimposing a gaze point mark on the relevant location on the frame of the recorded image at the relevant time when the image is played back. As described above, the gaze analysis device according to the second embodiment can continue gaze calculation even if the Purkinje image is not detected properly.

[0263] Third Embodiment Next, a third embodiment of a gaze analysis device based on the concept of the present invention will be described. Fig. 19(B) is a diagram showing the configuration of the third embodiment of a gaze analysis device based on the concept of the present invention. As shown in the figure, the third embodiment includes a gaze analysis terminal T3 equipped with one or more eyeball cameras 5Q and an information display board 47, and a control box 6Q that supplies power to the gaze analysis terminal T3, displays information such as announcements, advertisements, and news on the information display board 47, and processes images of the viewer's eyes acquired via the eyeball cameras 5Q.

[0264] The hardware configuration of the control box 6Q is almost the same as that of the control box 6 according to the first embodiment, but it has been added with a program for extracting the face area of ​​the viewer from an eyeball image or an image acquired by a separately provided face detection camera, a program for cutting out the eyeball image from the image area, and a program or hardware for controlling the information display board 47.

[0265] The figure also shows an enlarged view of eyeball camera 5Q as seen from the viewer. Lens 51 has an optical axis that intersects at a right angle with a line segment connecting the centers of infrared light sources 55R and 55L provided on both the left and right sides, and third infrared light source 55B is provided below or above lens 51.

[0266] Eye camera 5Q uses three infrared light sources 55R, 55L, and 55B to acquire eye images along with three Purkinje images. These images are processed by the calculation unit in control box 6Q. However, only eye images in which all three Purkinje images appear are processed to determine the center of the cornea and calculate a first gaze vector Gaze1 with this as the start point and the center of the pupil as the end point. Alternatively, the center of rotation of the eye is further determined and a second gaze vector Gaze2 is calculated with this as the start point and the center of the pupil as the end point.

[0267] From the viewpoint position and direction of the calculated first gaze vector or second gaze vector, the gaze direction is extended to the surface of information display board 47, and the intersection of the gaze extension line and the surface of information display board 47 is set as the gaze point, thereby making it possible to create a heat map in the form of a map that shows the viewer's gaze frequency and total gaze time using shades of gray. As shown in FIG. 1B, by providing multiple eyeball cameras 5Q above and below information display board 47, it is possible to detect the gaze points of multiple viewers and to acquire and analyze eyeball images of tall and short people. The configuration and effects of the third embodiment have been described above.

[0268] <Gaze point superimposition> Next, the relationship between the line-of-sight vector, which indicates the line-of-sight direction, and the angle of view of the field of view camera 4 will be shown. Fig. 20 is a diagram showing the spatial relationship between the second line-of-sight vector Gaze2 and the imaging system of the field of view camera 4. Fig. 20(A) shows the second line-of-sight vector Gaze2, which starts from the center of rotation E11 (Ro), a three-dimensional coordinate system GX·GY·GZ (the GX axis is not visible due to this attitude and is therefore not shown) that defines the direction of Gaze2, and a three-dimensional coordinate system SX·SY·SZ (the SX axis is not visible due to this attitude and is therefore not shown) that defines the imaging system of the field of view camera 4.

[0269] The field of view camera 4 has an imaging lens 41 and an imaging surface 42 (enlarged view λ). A three-dimensional coordinate system SX·SY·SZ is set with the center of the lens 41 as the origin SO and the SZ axis as the lens optical axis. The field of view camera 4 has a wide-angle shooting range 44, which covers almost the entire range that the subject can gaze at by rotating their eyes.

[0270] Here, we consider a virtual screen 46 that is perpendicular to the SZ axis. The distance from the origin SO to the virtual screen 46 is VSL. VSL can be varied depending on the application; for example, it may be about 1 meter when the subject is gazing at a nearby object, 5 meters in a large room such as an exhibition hall, or infinity outdoors.

[0271] First, the second gaze vector Gaze2 is extended from its end point, maintaining its direction, to find the point of intersection with the virtual screen 46. If the subject is gazing at a location at distance VSL, this intersection is the gaze point GP. Then, the projection point of the gaze point GP onto the imaging surface 42 of the field of view camera 4 is found, and this projection point becomes the position where the gaze point should be superimposed. A gaze point display can be written directly at that position, or a database can be created that records that a gaze point display should be superimposed at the appropriate position of the appropriate frame.

[0272] Up until now, the second gaze vector Gaze2 and the three-dimensional coordinates of its starting point, the center of rotation E11 (Ro), have been expressed in the GX·GY·GZ coordinate system with the origin GO being the center of the lens 51 of the eye camera 5. These must be converted into the SX·SY·SZ coordinate system with the origin SO being the center of the lens 41 of the field of view camera 4.

[0273] Here, we assume that the second gaze vector Gaze2 and the three-dimensional coordinates of its starting point, the turning center E11 (Ro), are already expressed in the SX·SY·SZ coordinate system. Then, the second gaze vector is Gaze2 (g2 SX ,g2 SY ,g2 SZ ) (Equation (22-1)), the turning center is Ro(r SX ,r SY ,r SZ ) [Equation (22-2)].

[0274]

number

[0275] Furthermore, the equation of the line passing through the turning center E11 (Ro) and extending in the direction of the second gaze vector Gaze2 is given by equation (22-3). s is an unknown, (v SX ,v SY ,v SZ ) is a variable that represents the position on the line. The point where this line intersects with the virtual screen 46 is the gaze point GP, and the virtual screen 46 is v SZ =VSL plane. Therefore, v SZ Substituting the value of VSL into equation (22-3), k s The value of k is obtained as equation (22-4). s is a known value.

[0276]

number

[0277] Then, the three components SX, SY, and SZ as the coordinates of the gaze point GP are obtained by equation (22-5). SX ,gp SY ,gp SZ ), the coordinates of the GP can be expressed by equation (22-6).

[0278]

number

[0279] In order to superimpose a gaze mark at the position corresponding to the GP on the image captured by the field of view camera 4, it is necessary to calculate where the gaze point GP in the three-dimensional space is projected on the image capture surface 42. The projection line onto the image capture surface is a straight line that passes through the GP and the origin SO and reaches the image capture surface 42, and is given by equation (23-1). However, p s is an unknown, (v2 SX ,v2 SY ,v2 SZ ) is a variable. However, the image plane 42 is v2 SZ =-SD S Since this is a plane, substituting this into equation (23-1), p sis obtained as equation (23-2) and is a known value.

[0280]

number

[0281] p s Once these are known, the three-dimensional coordinates of the projection point of the fixation point GP onto the imaging surface 42 can be obtained by equation (23-3). Then, since the SX and SY components form a two-dimensional coordinate system within the imaging surface 42, the two-dimensional coordinates of the projection point GPi of the fixation point are given by equation (23-4). Therefore, it is sufficient to superimpose the fixation point mark at this position within the imaging surface 42. However, since equation (23-4) is expressed in the SX·SY·SZ coordinate system, the superimposition position may be managed after converting it into an expression of a two-dimensional coordinate system provided within the imaging surface 42.

[0282]

number

[0283] Here, when the subject is gazing at infinity, the difference between the position of the rotation center E11 (Ro), which is the starting point of the second line of sight vector, and the lens center SO of the field of view camera 4 can be ignored, so in equation (23-4), r SX =r SY = 0, and k as shown in equation (23-5) s is also simplified to equation (23-7) via equation (23-6).

[0284]

number

[0285]

number

[0286] Equation (23-7) means that the origin GO of the eyeball camera coordinate system coincides with the field of view camera coordinate system SO, and the slope of the line from the gaze point at infinity through SO to the image pickup surface 42 is the slope of the second gaze vector Gaze2 itself. To draw this in Figure 1(A), the line of Gaze2 starting from Ro is translated to the origin SO of the lens 41. The intersection of this line and the image pickup surface 42 is the position where the gaze mark should be superimposed.

[0287] Next, an example of conversion from the GX·GY·GZ coordinate system to the SX·SY·SZ coordinate system is shown. This is a process for converting the attitude of the imaging systems of the field of view camera 4 and eyeball camera 5. Figure 20(B) shows the center of rotation Ro, which is the starting point of the line of sight vector, as well as the GX·GY·GZ coordinate system and the SX·SY·SZ coordinate system. Both of these imaging systems are built into the line of sight analysis terminal and are fixed at the time of design and manufacturing. Therefore, it is sufficient to build a conversion processing program, for example, into the terminal in advance based on the design values.

[0288] First, a translational transformation is performed to align the origins of both coordinate systems. Figure 1(D) shows the coordinate systems after the translational movement, the GX(1)·GY(1)·GZ(1) coordinate system and the SX·SY·SZ coordinate system. To distinguish between the coordinate systems before and after the translational movement, the notation (1) is added to the coordinate system after the movement. The origin GO(1) after the movement coincides with the origin SO of the SX·SY·SZ coordinate system. After that, a rotational transformation is performed to align the GZ axis with the SZ axis, and the GX axis with the SX axis. The remaining GY axis and SY axis will naturally coincide.

[0289] The rotation center Ro is expressed in GX·GY·GZ coordinates as Ro(r GX ,r GY ,r GZ ), and the expression in the SX·SY·SZ coordinate system is Ro(r SX ,r SY ,r SZ ) and SO(SO GX ,SO GY ,SO GZ ) represents the position of the origin SO of the SX·SY·SZ coordinate system in the GX·GY·GZ coordinate system.

[0290] Figure 1(C) shows the positional relationship between the turning center Ro, the GX·GY·GZ coordinate axes with GO as the origin, and the SX·SY·SZ coordinate axes with SO as the origin. In the figure, the vector Ro(r GX ,r GY ,r GZ ), the vector SO(SO GX ,SO GY ,SO GZ ), the vector Ro(1)(r GX(1) ,r GY(1) ,r GZ(1) ) is assumed.

[0291] Then, the vector Ro going directly from GO to Ro and the vector sum route going from GO to Ro via SO are connected with an equal sign to give equation (24-1). Solving this for vector Ro(1) gives equation (24-2). From equation (24-2), vector Ro(1)(r GX(1) ,r GY(1) ,r GZ(1) ) is calculated using equation (24-3).

[0292]

number

[0293] The coordinate system GX(1)·CY(1)·GZ(1) derived from equation (24-3) has the same axis direction as the GX·GY·GZ coordinate system, but its origin GO(1) coincides with the origin SO of the SX·SY·SZ coordinate system. After the origins of both coordinate systems are aligned, an appropriate rotation transformation can be applied to align each axis of the GX(1)·CY(1)·GZ(1) coordinate system with the SX·SY·SZ coordinate system.

[0294] The above shows an example of superimposing a gaze point onto a field of view image. In addition to this example, during the initial adjustment of the gaze analysis terminal, it is also possible to have the subject gaze at several specified locations, link the direction and orientation of the gaze vector at that time to the specified locations, and interpolate the intermediate areas.

[0295] Incidentally, Non-Patent Document 2 states that there is a slight angular difference between the direction of the line connecting the center of corneal curvature and the center of the pupil, i.e., the optical axis direction of the eyeball's optical machinery, and the direction in which the subject is gazing, i.e., the gaze direction. This difference is called the κ angle, and its magnitude varies from person to person. This is because the fovea, a tissue on the retina with the highest visual resolution, is located slightly off the optical axis. Therefore, in order to determine the direction in which the subject is gazing, i.e., the gaze direction (also called the visual axis direction), the optical axis direction calculated from the corneal shape can be corrected by the κ angle, an individual parameter.

[0296] To obtain these personal parameters, the subject can be calibrated. The difference in the kappa angle from the optical line of sight becomes significant at close range. Therefore, the gaze analysis device based on the concept of the present invention is provided with a method for obtaining the above-mentioned personal parameters by performing simple calibration at close range.

[0297] Returning to FIG. 19 (C), this figure shows an example of a subject wearing a gaze analysis terminal performing a one-point calibration operation, with the subject gazing at the AR marker 8. This is the scene seen by the subject's eyes, and is an image that can also be recorded by a field of view camera. However, the subject is not looking at any information display board or the like; all they can see is their fingertip and the AR marker 8. In other words, the gaze point marker GP indicating the gaze point and the upward and rightward arrows on the AR marker are merely shown for the convenience of this explanation.

[0298] However, the calculation unit 61 of the gaze analysis device plots the calculated gaze point GP on the virtual screen 46, identifies the center of the AR marker 8 where the subject is supposed to be gazing, and calculates the positional deviation between this center and the gaze point GP. The gaze point GP is based on the coordinates calculated by the above-mentioned formula (23-4). The correction amount at this time may be stored in the memory unit 62 as a personal parameter of the subject and applied to correction during gaze analysis. The above is an example of one-point calibration.

[0299] <Fourth embodiment> Next, a fourth embodiment of the gaze analysis device based on the concept of the present invention will be described. The hardware of the fourth embodiment is composed of a gaze analysis terminal T4 and a control box 6 (not shown). The gaze analysis terminal T4 has a feature different from the first to third embodiments in the arrangement of the infrared light source provided in the eye camera. Figure 21 is a diagram showing the arrangement of the infrared light source provided in the eye camera 5 in the fourth embodiment of the gaze analysis device based on the concept of the present invention. Figure 21 (A1) shows the arrangement of the infrared light sources 55R, 55L, and 55B of the above-mentioned gaze analysis terminals T1 to T3 together with the three-dimensional coordinate system GX, GY, and GZ.

[0300] As mentioned above, the origin GO of the three-dimensional coordinate system is located at the center of lens 51, where the three axes GX, GY, and GZ intersect (the GZ axis is not shown). The GZ axis, which is the optical axis of lens 51, passes through the intersection of the GX and GY axes and points toward the back of the page. Although not shown, the eyeball, which is the subject of the photograph, is located at the back of the page and faces toward the front of the page.

[0301] As described above, infrared light sources 55R and 55L are arranged on the same straight line passing through GO. In this case, this straight line is the GX axis. The foot of a perpendicular line drawn from infrared light source 55B to a line segment connecting the positions of infrared light sources 55R and 55L coincides with origin GO, i.e., the center of lens 51.

[0302] Figure 1 (A2) is a diagram of the cornea when it is irradiated with infrared light from the three infrared light sources arranged as described above, producing three specular reflections. The illustration shows the iris E3 and pupil E31, with the cornea E2 in front of them, and it is the surface of this cornea E2 that produces the specular reflections. The positions of these specular reflections are arranged at the three vertices of an approximately isosceles triangle, reflecting the arrangement of the infrared light sources.

[0303] On the other hand, (B1) of the same figure shows the arrangement of the lens 51 and the infrared light sources 55R, 55L, and 55B in the line-of-sight analysis terminal T4 according to the fourth embodiment, together with the three-dimensional coordinate system GX·GY·GZ. Here too, the infrared light sources 55R and 55L are arranged on the same straight line passing through GO.

[0304] However, in this case, the line segment connecting the positions of infrared light sources 55R and 55L forms a known angle ∠GR with the GX axis. However, even in this case, the foot of a perpendicular line drawn from infrared light source 55B to the line segment connecting the positions of infrared light sources 55R and 55L coincides with origin GO, i.e., the center of lens 51.

[0305] Figure 1B2 shows three specular reflections on the corneal surface caused by infrared light from the infrared light source arranged as described above. The arrangement of the three light sources is rotated by the known angle ∠GR around the GZ axis, which is the lens optical axis, and this affects the locations of specular reflections 71R, 71L, and 71B on the cornea. In other words, the approximately isosceles triangle with the positions where the three specular reflections occur as vertices is also rotated by ∠GR.

[0306] 10C1 shows the state in which the specularly reflected light is captured as Purkinje images 71iR, 71iL, and 71iB on the imaging plane 52. Images other than the Purkinje images are not shown. If the three Purkinje images are connected by lines, an approximately isosceles triangle rotated clockwise by angle ∠GR is observed.

[0307] The straight line CP1i that corresponds to the base of the approximately isosceles triangle is the plane on which the infrared light rays from the infrared light sources 55R and 55L are mirror-reflected, pass through the lens center GO, and reach the imaging surface 52, and is projected onto the imaging surface 52, i.e., the cutting plane projection image CPix.

[0308] Then, a straight line is drawn that passes through the Purkinje image located at the apex angle of the approximately isosceles triangle and intersects the line CP1i at a right angle, and this line will be referred to as CP2i. The line CP2i is a line projected onto the imaging plane 52 by a plane along which an infrared ray emitted from the infrared light source 55B is specularly reflected by the corneal surface 71B, passes through the lens center GO, and reaches the imaging plane 52. In other words, the line CP2i is a cut-plane projection image CPiy. Even if the approximately isosceles triangle formed by the Purkinje images is rotated by an arbitrary angle, it is possible to determine the two-dimensional coordinates of the corneal center, as in the first embodiment.

[0309] Then, by rotating the GX and GY axes of the three-dimensional coordinate system and the x and y axes of the two-dimensional coordinate system on the imaging surface 52 by an angle ∠GR around the GZ axis, which is also the lens optical axis, as follows, the first gaze vector Gaze1 and the second gaze vector Gaze2 can be calculated using the same analysis as in the first embodiment.

[0310] 22(C2) shows the result of rotating the two-dimensional coordinate system of the imaging surface by angle GR around the GZ axis, i.e., around the origin o of the two-dimensional coordinate system. As in the first embodiment, the straight line CP1i, which is a projection of the cutting plane, is parallel to the x-axis, and the straight line CP2i, which is a projection of the other cutting plane, is parallel to the y-axis. With this in mind, refer to FIG. 22.

[0311] 22 is a diagram showing the positional relationship between eyeball E, three infrared light sources, a lens and a three-dimensional coordinate system, and an imaging surface 52 and a two-dimensional coordinate system in a fourth embodiment of a gaze analysis device based on the concept of the present invention. The R, L, and B symbols in the diagram represent infrared light sources 55R, 55L, and 55B, respectively. In FIG. 22(A), two infrared light sources 55R (R in the diagram) and 55L (L in the diagram) are arranged on the GX-GY plane, with lens 51 between them, on a straight line (dashed line) that passes through origin GO, which is also the center of lens 51, and forms a known angle ∠GR with respect to the GX axis.

[0312] Furthermore, a third infrared light source 55B (B in the figure) is placed on the GX-GY plane, on a line that is perpendicular to the line and passes through the origin. The infrared light rays from these three infrared light sources are specularly reflected on the surface of the cornea E2, forming three Purkinje images at the three vertices of an approximately isosceles triangle in a two-dimensional coordinate system on the imaging plane 52. The Purkinje images on the imaging plane 52 are as shown in Figure 21 (C1).

[0313] On the other hand, in Figure 22(B), the GX·GY·GZ coordinate system is set by rotating the GX·GY·GZ coordinate system clockwise by angle GR around the GZ axis, and a two-dimensional coordinate system x'·y' is set on the virtual imaging surface 52'. The virtual x' and y' axes are parallel to the virtual GX' and GY' axes, respectively. Physically, there is no difference from Figure 22(A), but as shown in Figure 21(C2), the straight lines CP1i and CP2i formed by projecting a plane containing the optical path of the infrared light onto the imaging surface are parallel to the x' and y' axes, respectively, which means that they are also parallel to the GX' and GY' axes, respectively.

[0314] In this case, these conditions match those shown in the first embodiment, and the first gaze vector Gaze1 and the second gaze vector Gaze2 can be calculated using the analysis means shown in the first embodiment.

[0315] As described above, even if the infrared light sources 55R, 55L, and 55B are rotated by an arbitrary angle around the GZ axis, it is clear that the gaze vector can be calculated by the means shown in the first embodiment by rotating the three-dimensional coordinate axes by the same angle. However, since the calculated gaze vector Gaze' is a vector under the coordinate system GX'·GY'·GZ, it must be converted into an expression under the GX·GY·GZ coordinate system.

[0316] Furthermore, even if the rotation angle ∠GR is not known at the time of designing the line-of-sight analysis terminal T4, it is possible to make ∠GR known as the angle formed by the line passing through the first Purkinje image 71iR and the second Purkinje image 71iL and the x-axis on the imaging surface 52, as shown in FIG. 21(C1).

[0317] Furthermore, the rotation transformation matrix acts on object points or image points. It should be noted that, for example, when a coordinate system is rotated in a positive direction, the object points or image points rotate in a negative direction relative to it. Therefore, when the GX·GY·GZ coordinate system is rotated in a positive direction by ∠GR to form the GX'·GY'·GZ coordinate system, the rotation transformation of the object points is expressed by equation (25-1), and when returning to the GX·GY·GZ coordinate system, the transformation equation for the object points is expressed by equation (25-2).

[0318]

number

[0319] Equation (25-3) is an equation obtained by modifying equation (25-1) for a three-dimensional coordinate system for a two-dimensional coordinate system. In the process of calculating the gaze vector, the gaze analysis terminal T4 acquires an image in the two-dimensional x·y coordinate system projected onto the imaging surface 52 via the eyeball camera. In other words, a -∠GR transformation is performed using the two-dimensional transformation equation (25-3) to calculate the three-dimensional gaze vector. Then, an ∠GR transformation is performed on the three-dimensional gaze vector using the three-dimensional inverse transformation equation (25-2) to express the gaze vector in the GX·GY·GZ coordinate system, which is a normal coordinate system.

[0320] FIG. 23 is a processing block diagram of a fourth embodiment of a gaze analysis device based on the concept of the present invention. FIG. 23(A) is a simplified version of the processing block diagram (FIG. 9) of the first embodiment, to which a coordinate rotation transformation has been added, resulting in a processing block diagram of a gaze analysis device according to the fourth embodiment. Immediately after processing block A1 is performed, a two-dimensional rotation transformation process A1-ad is added to the image on the imaging surface 52. The A1-ad transformation is performed according to equation (25-3). A two-dimensional rotation transformation is performed on the image acquired in processing block A1. Furthermore, immediately after processing block A8 is performed, a three-dimensional inverse rotation process A8-ad is added to the gaze vector Gaze1. The A8-ad transformation is performed according to equation (25-2). A three-dimensional inverse rotation transformation is performed on the first gaze vector calculated in processing block A8.

[0321] 15B is a processing block diagram of the gaze analysis device according to the fourth embodiment, which adds coordinate rotation transformation to the processing block diagram (FIG. 15) according to the second embodiment. Immediately after processing block A1 is performed, two-dimensional rotation transformation processing A1-ad is added to the image on imaging plane 52. Furthermore, immediately after processing block A8 is performed, three-dimensional inverse rotation processing A8-ad is added to gaze vector Gaze1, and three-dimensional inverse rotation processing A8-ad is added to gaze vector Gaze2 after processing blocks A10 and A11 are performed, respectively.

[0322] Returning to FIG. 21, two further variations in the arrangement of the infrared light sources are shown. (D1) of the same figure shows another arrangement of three infrared light sources along with the GX, GY, and GZ coordinates. The GZ axis is not shown, but is a right-handed system that passes through GO and points into the page. The infrared light sources 55R and 55L, which are located on the GX axis, are not equidistant from the origin GO. Reflecting this, the triangle with vertices at specular reflection positions 71R, 71L, and 71B in (D2) of the same figure is not an isosceles triangle. However, by considering the arrangement of the three infrared light sources and determining that it is a group of specular reflection images (Purkinje images) that form the corresponding triangle, it is possible to determine whether or not it is a specular reflection on the cornea.

[0323] In block A3 of the processing block diagram (FIGS. 9, 15, and 23), as in the first embodiment, the first projection line HL connects the Purkinje images formed by projecting specularly reflected light 71L and 71R, which form the base of the triangle, onto the imaging plane 52. The foot of a perpendicular line drawn from the third Purkinje image, which forms the apex angle, is the corneal central projection point E21i. This perpendicular line is the second projection line VL. Also in this case, the optical path from infrared light source 55R to imaging plane 52 and the optical path from infrared light source 55L to imaging plane 52 are on the same cutting plane CPx that includes the GX axis, so there is no problem with calculating depth information Len in block A4. Therefore, even with this arrangement of three infrared light sources, the line of sight vector can be calculated.

[0324] Figure 21 (E1) shows another arrangement of three infrared light sources along with the GX-GY-GZ coordinates. Infrared light sources 55R and 55L are arranged on the GX axis. A straight line can be drawn between infrared light source 55B on the GX-GY plane and origin GO, but this line forms a known angle SR with the GY axis. Figure 21 (E2) shows specular reflections 71R, 71L, and 71B on the cornea under these illuminations. If the arrangement of the three infrared light sources is known in advance, it can also be determined that the triangle with these vertices is a specular reflection occurring on the cornea.

[0325] However, in this case, for the triangle formed by the three specular reflection points, a second projection line must be drawn from 71B, which corresponds to the apex angle, to the base formed by the other two specular reflection points, but rather a straight line that forms a known angle ∠SR with the base. The intersection of the first projection line and the second projection line is the corneal center projection point E21i. While the specular reflection points on the corneal surface have been illustrated in Figures 21(D1) to 21(E2) above, the actual subject of image processing and drawing is the Purkinje image on the imaging plane 52. As described above, the gaze analysis device according to the fourth embodiment relaxes the installation conditions for the infrared light source while maintaining gaze analysis accuracy.

[0326] Here, we will discuss the constraints on the placement of infrared light sources. The simplest infrared light source placement is to place the first infrared light source on the GX axis and the third infrared light source on the GY axis on the first plane. However, diffused reflection from the sclera E1 may be imaged on the imaging plane 52, making it impossible to determine whether the two Purkinje images are genuine Purkinje images caused by specular reflection from the cornea E2. Therefore, the gaze analysis device based on the concept of the present invention does not adopt the above configuration.

[0327] On the other hand, if there are three or more Purkinje images and they form a specific polygon, and if a figure that can be judged to be that polygon is found on the 52 lines of the imaging plane, the image points at those vertices can be considered to be Purkinje images. The simplest shape of a polygon is a triangle.

[0328] Furthermore, if the first and second infrared light sources, among the three or more infrared light sources, are arranged on a first line passing through the center GO of the lens 51, the first Purkinje image caused by the first infrared light source and the second Purkinje image caused by the second infrared light source will exist on the same cutting plane. This is because the cutting plane on which the first Purkinje image exists includes the first line and the corneal center E21, and the cutting plane on which the second Purkinje image exists also includes the first line and the corneal center E21. In other words, the two cutting planes are the same plane.

[0329] The first and second Purkinje images are on the same cutting plane. Therefore, the intersection of the cutting plane and the imaging plane 52, i.e., the cutting plane projection image CPix, is drawn as a straight line connecting the first and second Purkinje images.

[0330] Furthermore, on the first plane, a third infrared light source is provided on a second line that intersects with the first line at the origin GO. The angle between the first line and the second line is set to a known angle in design. Then, on the imaging plane 52, if a line that passes through the third Purkinje image caused by the third infrared light source and forms the known angle in design with the cutting plane projection image CPix is ​​drawn, this line is the cutting plane projection image CPiy, which is the second projection line.

[0331] Next, the setting of the coordinate system of the gaze analysis device based on the concept of the present invention will be explained using the first straight line as a reference. First, lens 51 is installed as the imaging system of eyeball camera 5. The center of lens 51 then becomes the origin GO of the three-dimensional coordinate system, and the optical axis of lens 51 becomes the GZ axis. Then, imaging surface 52 is installed on a second plane perpendicular to the GZ axis, and the intersection with the GZ axis becomes the origin o of the two-dimensional coordinate system.

[0332] Next, a first line passing through the origin GO is defined, and the first and second infrared light sources must be installed on this first line. With this configuration, the first and second Purkinje images generated on the imaging surface 52 exist on the same cutting plane. For these Purkinje images to exist on the same cutting plane, the first line does not need to be perpendicular to the GZ axis. However, for convenience of subsequent analysis, it is preferable that the first line be perpendicular to the GZ axis.

[0333] When the first line is perpendicular to the GZ axis, if a line passing through the origin o and parallel to the first line is set as the x-axis on the imaging plane 52, the cutting plane CPix passing through the first Purkinje image and the second Purkinje image will be parallel to the x-axis. Furthermore, a line perpendicular to the x-axis at the origin o is set as the y-axis on the imaging plane 52. With the above steps, a two-dimensional x-y coordinate system is established.

[0334] Next, a straight line parallel to the x-axis and intersecting with the GZ-axis at the origin GO is set as the GX-axis. As a result, the first straight line coincides with the GX-axis. Furthermore, a straight line parallel to the y-axis and intersecting with the GZ-axis and GX-axis at the origin GO is set as the GY-axis. With this, the three-dimensional coordinate system GX·GY·GZ is set.

[0335] When the x-axis is set on the imaging surface 52 so that it is skewed relative to the first line, the cutting plane CPix intersects with the x-axis at a predetermined angle. In this case, the x-y coordinate system and the GX-GY-GZ coordinate system are rotated around the GZ axis to align the first line with the GX axis, and then the line of sight vector is calculated. Then, as described above, reverse rotation correction is performed.

[0336] Figure 24 shows an example of infrared light source placement based on the concept of the present invention. Figure 24(A) is a three-dimensional diagram of a lens 51, a three-dimensional coordinate system GX·GY·GZ, and a corneal sphere E2. The sclera E1 is not shown. In this figure, infrared light source 55R is placed in the positive region on the GX axis, and infrared light source 55L is placed in the negative region. These two infrared light sources 55R and 55L define a common cutting plane CPx.

[0337] 1B shows the configuration of the lens 51 and cornea E2 as viewed from the negative region of the GX axis toward the origin GO. The cutting plane CPx containing the GX axis and the corneal center E21 is represented as a dashed line passing through the origin GO and the corneal center E21.

[0338] The third infrared light source 55B (not shown) is provided on a second line that intersects with the first line at a known angle, except that in Figure 25(A) the GX axis is the first line, and the first line and the second line intersect at an angle of 90°.

[0339] 10C1 shows Purkinje images 71iR, 71iL, and 71iB that form the Purkinje image group on the imaging plane 52 in the above-mentioned infrared light source arrangement. To save space, the appearances of the Purkinje image group detected in three cases, i.e., Cases 1 to 3, are summarized on one page.

[0340] Case 2 is an example of a group of Purkinje images when the corneal center E21 is located in the GZ-GX plane. Figure 2 (C2) shows the x-axis and y-axis on the imaging plane 52. Purkinje images 71iR and 71iL are projected onto the x-axis.

[0341] On the other hand, Case 1 shows a group of Purkinje images when the corneal center E21 is below the GZ-GX plane, and Case 3 shows a group of Purkinje images when the corneal center E21 is above the GZ-GX plane.

[0342] FIG. 2C2 shows the drawing performed by the calculation unit 61. The position of the corneal center differs for each case. Regardless of which way the corneal center moves, if a straight line passing through the first Purkinje image 71iR and the second Purkinje image 71iL is drawn, this will become the cutting plane projection image CPix. If a straight line passing through the third Purkinje image 71iB and intersecting with the cutting plane projection image CPix at a known angle is drawn, this will become the cutting plane projection image CPiy. The intersection of these two is the corneal center projection point E21i.

[0343] Next, Figure 25 is also a diagram showing an example of the infrared light source arrangement of a line-of-sight analysis device based on the concept of the present invention. In Figure 25(A), infrared light sources 55R and 55L are provided in the negative region on the GX axis. In this case, the GX axis also corresponds to the first line. The cutting plane CPx related to these infrared light sources is common. In this example, both infrared light sources are located on one side of the lens 51, and do not necessarily need to be axially symmetrical with respect to the optical axis of the lens 51. This is a difference from paragraph

[0007] of Patent Document 2.

[0344] 1B shows the configuration of the lens 51 and cornea E2 as viewed from the negative region of the GX axis toward the origin GO. A cutting plane CPx including the GX axis and the corneal center E21 is represented as a dashed line passing through the origin GO and the corneal center E21. In this example, a third infrared light source 55B (not shown) is also provided on a second line perpendicular to the first line.

[0345] Figure 2(C1) shows the Purkinje images detected in Cases 1 to 3 all collected on one page, and Figure 2(C2) shows the x- and y-axes and the construction lines. In this example, regardless of which way the corneal center moves, if we connect the first and second Purkinje images with a straight line, draw a line that is perpendicular to the first line and passes through the third Purkinje image, and then construct the intersection of these two lines, this is the corneal central projection point E21i.

[0346] Next, an example will be shown in which a common cutting plane is not set between the infrared light source 55R and the infrared light source 55L. The infrared light sources 55R and 55L shown in Fig. 26 do not set a common cutting plane. This is an installation mode of the infrared light source that is not within the concept of the present invention.

[0347] In FIG. 5A, infrared light sources 55R and 55L are both positioned behind lens 51. The line segment connecting the installation positions of the two infrared light sources does not pass through origin GO, which is the center of the lens. In this case, the cutting plane set by infrared light source 55R, origin GO, and corneal center E21 and the cutting plane set by infrared light source 55L, origin GO, and corneal center E21 are two different cutting planes and are not common. However, these two cutting planes are common only when corneal center E21 is within the GZ-GX plane.

[0348] Figure 1B is a view looking toward the origin GO from the negative region of the GX axis. In this figure, a light ray emitted from infrared light source 55L located on the front side of the page is specularly reflected 71L from the corneal surface, then travels a lower optical path to reach the lens center GO. On the other hand, a light ray emitted from infrared light source 55R located on the back side of the page is similarly specularly reflected 71R from the corneal surface, then travels a lower optical path to reach the lens center GO. In this way, the light paths that take high optical paths in the GY axis direction on both sides and then take a low optical path in the center after reflection do not all fit within a single plane.

[0349] Figure (C1) in the same figure shows the Purkinje images detected in Cases 1 to 3 all collected on one page, and Figure (C2) further shows the x-y axes and construction lines. Case 2 shows the Purkinje images when the corneal center E21 is located in the GZ-GX plane. In this case only, the line connecting the first Purkinje image 71iR and the second Purkinje image 71iL is the common cutting plane projection image CPix.

[0350] However, when the corneal center E21 is outside the GZ-GX plane, as in Case 1 and Case 3, the first infrared light source 55R and the second infrared light source 55L set different cutting planes, and the cutting plane projection images also have different directions. Moreover, these directions change depending on the position of the corneal center E21. Therefore, it is not possible to construct the cutting plane projection images CPixR and CPixL shown in Figure 1 (C2), nor is it possible to calculate the corneal center projection point E21i as their intersection.

[0351] Figure 27 also shows another example of the installation mode of infrared light sources that is not within the concept of the present invention. In Figure 27(A), infrared light sources 55R and 55L are both placed in front of lens 51. The line segment connecting the installation positions of the two infrared light sources does not pass through origin GO, which is the center of the lens. In this case, the cutting plane set by infrared light source 55R, origin GO, and corneal center E21 and the cutting plane set by infrared light source 55L, origin GO, and corneal center E21 are two different cutting planes and are not common. However, these two cutting planes become common only when corneal center E21 is within the GZ-GX plane.

[0352] In this example, in Case 1 and Case 3, the inclination of the cutting plane projection image passing through the first and second Purkinje images is unknown, so it cannot be drawn, and therefore the intersection point of the two lines shown in Figure (C2) cannot actually be found, and therefore the corneal central projection point cannot be calculated.

[0353] Even in this case where the light source arrangement is symmetrical with respect to the optical axis, if an infrared light source arrangement that is not within the concept of the present invention is adopted, it is not possible to set a cutting plane common to the first and second Purkinje images. Therefore, if the corneal center E21 deviates from the GZ-GX plane associated with the eye camera 5, it is not possible to obtain the corneal center projection point E21i, which becomes an obstacle to calculating the gaze vector.

[0354] The above describes the installation conditions for the three infrared light sources for setting two cutting planes for the two cutting plane projection images that intersect on the imaging surface 52. If some measurement error is allowed, the restrictions on the light source installation can be relaxed, but it is preferable that the three infrared light sources are installed on the first plane, that is, the plane that is perpendicular to the lens optical axis at the lens center.

[0355] Furthermore, it is preferable that the first and second infrared light sources of the three infrared light sources are installed on a first straight line, i.e., a line that is included in the first plane and passes through the center of the lens, and it is also preferable that the third infrared light source of the three infrared light sources is installed on a second straight line, i.e., a line that is included in the first plane and passes through the center of the lens, and that intersects with the first straight line at a known angle.

[0356] <Selection of 3D coordinate system> Next, we will explain how to select a three-dimensional coordinate system when a thin lens or a thick lens is used as lens 51. When a thin lens is used as lens 51, the nodal point, which is the intersection of the single principal plane and the lens optical axis, is the lens center. The plane that includes this lens center and is perpendicular to the lens center is the first plane. The GX and GY axes of the three-dimensional coordinate system are set on this first plane.

[0357] On the other hand, when a thick lens is used as the lens 51, there are two points that can be the lens center: the object-side nodal point (also called the eyeball-side nodal point) and the image-side nodal point.

[0358] When discussing the relationship between the lens and the three-dimensional coordinate system and each part of the eyeball (the subject), such as the center of the eyeball, the center of the pupil, the positions of the first, second, and third light sources, and the positions of the first, second, and third specular reflections caused by the light rays from these light sources, the object-side nodal point is regarded as the lens center, and the object-side three-dimensional coordinate system (also called the eyeball-side three-dimensional coordinate system) is discussed as a three-dimensional coordinate system. In this case, the first plane means the eyeball-side first plane.

[0359] On the other hand, when discussing the relationship between the lens and the three-dimensional coordinate system and the images formed on the imaging plane 52, such as the first, second, and third Purkinje images, or the positions of each part of the eyeball projected onto the imaging plane 52, the image-side nodal point is regarded as the lens center, and the image-side three-dimensional coordinate system is discussed as a three-dimensional coordinate system. In this case, the first plane means the image-side first plane.

[0360] Fifth Embodiment: Extended First Plane and Extended Second Plane Next, a gaze analysis device according to a fifth embodiment will be shown, in which the first and second infrared light sources are arranged outside the first plane and still fall within the scope of the present invention. Fig. 28 is a diagram showing an arrangement of infrared light sources in a gaze analysis device according to the present invention, in which the first and second infrared light sources are installed outside the first plane.

[0361] In Figure 1A, the first infrared light source 55R is located behind the GX-GY plane, which forms the first plane, and the second infrared light source 55L is located in front of the GX-GY plane. Even in this case, the line passing through the installation positions of the first infrared light source 55R and the first infrared light source 55L passes through the origin GO, which is the center of the lens 51, and the first and second infrared light sources can be said to be on the first line. In addition, the third infrared light source 55B is provided in the negative region of the GY axis, but is not shown in the figure.

[0362] Figure 1B shows the lens center GO and cornea E2 as viewed from the negative region of the GX axis. Infrared light emitted from second infrared light source 55L on the front side of the page travels along a low optical path, is specularly reflected at cornea E2 surface 71L, and the reflected light takes a higher optical path to reach lens center GO. On the other hand, infrared light emitted from first infrared light source 55R on the back side of the page travels along a high optical path, is specularly reflected at cornea E2 surface 71R, and the reflected light takes a lower optical path to reach lens center GO.

[0363] The optical path of the first infrared light source 55R is included in the cutting plane 1 defined by the first straight line and the corneal center E21. On the other hand, the optical path of the second infrared light source 55L is included in the cutting plane 1 defined by the first straight line and the corneal center E21. In other words, the same cutting plane CPx is set for the first infrared light source 55R and the second infrared light source 55L. In Figure 2(B), the two dashed dotted lines extending from the corneal center E21 are the cutting plane CPx.

[0364] FIG. (C1) is a compilation on one page of the Purkinje images detected in Cases 1 to 4, and FIG. (C2) is a diagram showing the x·y axes and the plotting lines. As shown in FIG. (C2), the first Purkinje image 71iR and the second Purkinje image 71iL can be connected by a straight line, and this can be used as the cut plane projection image CPix. This is because the cut plane related to the first Purkinje image 71iR and the cut plane related to the second Purkinje image 71iL are common.

[0365] However, the direction of the cut plane projection image CPix is not constant. As the corneal center E21 moves away from the GZ·GX plane, the inclination with respect to the x-axis of the imaging surface 52 increases. Therefore, the cut plane projection image CPiy passing through the third Purkinje image 71iB cannot be managed by the angle with the cut plane projection image CPix. When adopting such an infrared light source arrangement mode, the third infrared light source 55B must be provided on the GY axis parallel to the y-axis. In this way, on the imaging surface 52, the cut plane CPiy can be plotted by drawing a straight line passing through the third Purkinje image 71iB and parallel to the y-axis. And the intersection point of the cut plane projection image CPix and the cut plane projection image CPiy is calculated as the corneal center projection point E21i.

[0366] As described above, it has been shown that even in the configuration where the infrared light source is arranged at a position deviating from the first plane, there may be a case where the corneal center projection point E21i can be detected by plotting on the imaging surface 52. This plane that deviates from the first plane and includes the GY axis and the first straight line may be referred to as the extended first plane.

[0367] However, with this configuration, the first straight line serving as the analysis reference line and the GX axis do not match. In the eyeball cross-section analysis method shown in reference to FIG. 12, the GX axis was the analysis reference line and also the first straight line. And the distance Len from the analysis reference line to the corneal center E21 was calculated. In order to apply this calculation method even when the infrared light source arrangement deviates from the first plane, the following rotation transformation is necessary.

[0368] <Rotation transformation around the GY axis> 29 is a diagram showing a method for analyzing the cross-sectional structure of the eyeball E when an infrared light source is provided on the extended first plane. The first line connecting the installation positions of the first infrared light source 55R and the second infrared light source 55L is the center of the lens 51 and passes through the origin GO of the three-dimensional coordinate system GX·GY·GZ. However, the first line is not on the first plane, but on the extended first plane.

[0369] The extended first plane is a plane that intersects with the first plane at the GY axis (not shown) and is rotated by a known angle, for example, ∠δ°, around the GY axis. ∠δ is a design value of the eyeball camera 5 and is a known value. The third infrared light source 55B is installed on the second line, but is not shown in the figure. However, when an infrared light source is installed on the extended first plane, the second line is limited to being installed on the GY axis.

[0370] In the first to fourth embodiments, the depth information calculation process performed in block A4 uses the GX axis, on which the first and second infrared light sources are provided, as the analysis reference line, and calculates the distance Len from this analysis reference line to the corneal center E21. This analysis reference line is also the first straight line. However, in the fifth embodiment, the analysis reference line on which the first and second infrared light sources are provided is in a direction different from the GX axis, so the analysis method used in the first to fourth embodiments cannot be directly applied to ∠α, ∠β, ∠γ, and Len, which are necessary for analysis.

[0371] To solve this problem, a coordinate rotation transformation is performed around the axis GY so that the extended first plane can be regarded as the first plane. By performing a coordinate rotation transformation on the rear part from GO and projecting the second plane PL2 onto the extended second plane PL2ex, a Purkinje image equivalent to that generated by the infrared light source on the first plane can be obtained, and the analysis methods used in the first to fourth embodiments can be applied to ∠α, ∠β, ∠γ, and Len.

[0372] Here, the 3D coordinate system of the first plane and the 2D coordinate system of the second plane are rotated by an angle ∠δ around GY as the axis. After the transformation, they are defined as the extended first plane PL1ex and the extended second plane PL2ex, respectively, and the analysis method described above is applied to calculate the first gaze vector Gaze1 or the second gaze vector Gaze2. After calculating these gaze vectors, they are rotated inversely in 3D space to create gaze vectors in the GX·GY·GZ coordinate system.

[0373] As shown in Figure 29(B1), for an object point Ps on the eyeball, Ps(p GX ,p GY ,p GZ ) Furthermore, the angle between the projection plane PRy containing the projection line from GO to the object point Ps and the GY·GZ plane is ∠θ GY Also, as shown in Figure (C1), the angle between the projection plane PRx, which contains the projection line from GO to the object point Ps, and the GZ·GX plane is ∠φ GX Let's say.

[0374] As shown in Figure (B2), in the coordinate system EX·GY·EZ after rotation around GY, the coordinates of the object point Ps are expressed as Pe(p EX ,p GY ,p EZ ) Furthermore, the angle between the projection plane PRy containing the projection line from GO to the object point Pe and the GY·EZ plane is ∠η GY Also, as shown in Figure (C2), the angle between the projection plane PRx containing the projection line from GO to the object point Pe and the EZ·EX plane is ∠χ EX Let's say.

[0375] In this way, we derive equations that relate the 3D coordinate values ​​of the same object points Ps and Pe before and after the rotation transformation, the angles between the projection plane PRy and the GY·GZ plane and the GY·EZ plane, and the angles between the projection plane PRx and the GZ·GX plane and the EZ·EX plane. These equations enable the rotation transformation.

[0376] ∠δ with GY as the axis GY For the axis rotation transformation of , the object point is -∠δ GYApply the rotation formula (26-1), which can be expanded to give formula (26-2).

[0377]

number

[0378] Conversely, the transformation from the EX·GY·EZ coordinate system to the GX·GY·GZ coordinate system is done using equation (27-1), which can be expanded to give equation (27-2). Note that the GY coordinate value remains unchanged by a rotation transformation around the GY axis. After calculating the first gaze vector Gaze1 or the second gaze vector Gaze2 in the EX·GY·EZ coordinate system, it is necessary to transform it into the GX·GY·GZ coordinate system using equation (27-1).

[0379]

number

[0380] The relationship before and after rotational transformation is necessary for line of sight analysis regarding the direction from the lens center GO to each object point on the eyeball E. The projection lines from each object point to each projected image are straight lines that connect the object point and the image point and pass through the lens center GO. The direction of each projection line is defined by the angle between the two projection planes.

[0381] In the GX·GY·GZ coordinate system, there is a projection plane PRy, which is a plane defined by the object point Ps and the GY axis, and a projection plane PRx, which is a plane defined by the object point Ps and the GX axis. The angle between the projection plane PRy and the GY·GZ plane is ∠θ GY , the angle between the projection plane PRx and the GZ·GX plane is ∠φ GX As described above, the projection direction from the lens center GO to the object point Ps is determined by these two angles.

[0382] In the EX·GY·EZ coordinate system after rotational transformation, there is a projection plane PRy, which is a plane defined by the object point Ps and the GY axis, and a projection plane PRex, which is a plane defined by the object point Ps and the EX axis. The angle between the projection plane PRy and the GY·EZ plane is ∠η GY, the angle between the projection plane PRex and the EZ·EX plane is ∠χ EX These two angles define the direction from the lens center GO to the object point Pe, i.e., the object point Ps. Object point Ps and object point Pe are the same object; the only difference is that the coordinate systems that serve as the index reference for their positions are transformed between the GX·GY·GZ system and the EX·GY·EZ system.

[0383] ∠θ GY and ∠η GY The relationship between these is given by equations (28-1) and (28-2).

[0384]

number

[0385] The relationship between ∠φGX and ∠χEX is given by equations (29-1) and (29-2).

[0386]

number

[0387]

number

[0388] FIG. 30 is a diagram showing the relationship between two-dimensional and three-dimensional coordinate systems used in a line-of-sight analysis device according to the concept of the present invention. The line-of-sight analysis device according to the concept of the present invention has a two-dimensional x·y coordinate system set on the imaging surface 52 of the second plane PL2. This is called coordinate system CO1. Furthermore, there is a three-dimensional GX·GY·GZ coordinate system in which the first plane PL1 is defined as a GX·GY plane and a GZ axis perpendicular to this is added. This is called coordinate system CO2. CO1 and CO2 are defined by an angle θ related to the projection direction. GY and ∠φ GX Share.

[0389] Furthermore, if the first line is not within the first plane, the extended first plane PL1ex is set as the EX·GY plane, and a three-dimensional EX·GY·EZ coordinate system is required by adding the EZ axis perpendicular to this, which is called the coordinate system CO3. The projection destination of the coordinate system CO3 is the imaging plane 52ex of the extended second plane PL2ex, where two-dimensional ex·ey coordinates are set, which is called the coordinate system CO4.

[0390] The visual information used in the gaze analysis process is an image obtained in the coordinate system CO1. The corneal center projection point E21i is obtained from the first, second, and third Purkinje images obtained by optical projection through the lens 51, and the pupil center projection point is obtained from the pupil image. Furthermore, the rotation center projection point is obtained from the movement of the gaze vector projection image before and after the rotation of the eyeball E.

[0391] CO2 and CO3 are both 3D coordinate systems and share the GY axis. They are related by a rotation transformation around the GY axis, and CO2 is expressed as an angle ∠δ GY If you rotate it by -∠δ, it becomes CO3. GY When you rotate it, it turns back into CO2.

[0392] The information derived from CO2 is calculated as the position and projection direction to the origin GO in the coordinates of CO3. The objects are the positions of the first, second, and third infrared light sources, the first, second, and third specular reflection points, the pupil center Pu, the corneal center Cen, and the center of rotation Ro.

[0393] CO4 is the two-dimensional coordinate system of the projection of CO3, and consists of the ex axis parallel to the EX axis and the ey axis parallel to the GY axis. The coordinate components are (p2 ex ,p2 ey ) This is a computational projection, where the conversion of CO3 to CO4 is done by equation (30-1). The conversion of CO4 to CO3 is done by equation (30-2).

[0394]

number

[0395] CO4 is the angle ∠η with respect to the projection direction of the object points grasped by CO3, i.e., the first, second, and third infrared light sources, the first, second, and third specular reflection points, the pupil center Pu, the corneal center Cen, and the center of rotation Ro. GY and ∠χ EX However, the calculation of the projection point is not required.

[0396] Using these conversion equations, the analysis described with reference to FIG. 12 is performed, i.e., ∠α, ∠β, and ∠γ are calculated based on the eyeball image, and then depth information Len is calculated. This depth information, the cornea-centered projection point, and the pupil-centered projection point are converted into three-dimensional coordinates to calculate the first gaze vector Gaze1 or the second gaze vector Gaze2. After that, these gaze vectors are converted into a GX·GY·GZ coordinate system representation by rotational transformation using equation (27-1) or (27-2). This completes the fifth embodiment.

[0397] Sixth Embodiment Next, a sixth embodiment of the gaze analysis device based on the concept of the present invention will be described. The gaze analysis device is characterized by block A4. In block A4 of the previous embodiment, the corneal radius is calculated as an average value R ave The depth information Len is calculated by performing analysis using either the first or second infrared light source as a reference infrared light source. In contrast, in the sixth embodiment, the corneal radius R is calculated by performing analysis using both the first and second infrared light sources as reference infrared light sources.

[0398] 31 is a cross-sectional view of a cutting plane for analyzing depth information in a sixth embodiment based on the concept of the present invention. This is an improved version of the analysis method shown in FIG. 12. The previous method uses the corneal radius R ave Using these average values, the depth information was calculated using either the first or second infrared light source as a reference infrared light source.

[0399] On the other hand, in the sixth embodiment, the corneal radius R is set as a variable, and the corneal radius R is calculated based on the occurrence position of the specular reflection point related to the first infrared light source and the installation position of the light source. Rand the corneal radius R calculated based on the occurrence position of the specular reflection point related to the second infrared light source and the installation position of the light source. L The depth information Len is calculated on the condition that the two match.

[0400] Each parameter is formulated with reference to Figure 31. The distance ms from the infrared light source 55R on the GX axis, i.e., the analytical reference line, to the origin GO R and the distance Dis between the specular reflection point 71R and the analytical reference line due to the same light source. R Formulating based on this, we obtain equations (31-1), (31-2) and (31-3).

[0401]

number

[0402] GX coordinate of specular reflection point 71R, i.e., ms R1 Focusing on this, we obtain equation (31-4), which can be solved for Len to obtain equation (31-5). Substituting this into equation (31-3) and eliminating Len gives equation (31-6).

[0403]

number

[0404] For the infrared light source 55L and the specular reflection point 71L, equation (32-5) is obtained using a similar formulation as above. Since Len in equation (31-5) and Len in equation (32-5) should be the same, equation (33-1) is obtained. Furthermore, the corneal radii that can be matched for the infrared light source 55L and the specular reflection point 71L should both be equal, R, so in equation (33-1) both sides are multiplied by R as the same value. By reducing these and expressing the tangent in cosine and sine, equation (33-2) is obtained.

[0405]

number

[0406] Simplifying equation (33-2) using the addition theorem gives equation (33-3), which then passes through (33-4) to obtain β R Solving for this gives equation (33-5).

[0407]

number

[0408] Distance from origin GO to infrared light source 55R (ms) R Solving equation (31-6) for the corneal radius R gives equation (34-1).

[0409]

number

[0410] Similarly, the distance ms from the origin GO to the infrared light source 55L L Equation (34-2) holds for . Equation (34-1) and (34-2) are connected with an equal sign to obtain equation (34-3). In equation (34-3) and equation (33-5), ∠α R , ∠α L As already explained with reference to FIGS. 13 and 14, and ∠γ can be calculated by analyzing the Purkinje image on the imaging surface 52. R and ms L are the distances from the center of the lens 51 to the infrared light source 55R and the distances from the center of the lens 51 to the infrared light source 55L, and are known values ​​in design. R and β L is an unknown number. R and ∠α L On the paper, the positive rotation direction is counterclockwise around the origin GO. R and ∠β L On the paper, the clockwise direction is defined as the positive direction of rotation around the corneal center Cen (E21).

[0411]

number

[0412] Substituting equation (33-5) into equation (34-3), β R Eliminating this, equation (34-3) becomes L It can be considered as an equation for β L When the values ​​in the range that can be taken are substituted into the equation, the absolute value of the left side is minimized. L and set the value as β L It can also be the solution of

[0413] β L Substituting the solution of into equation (33-5), β R In addition, β can be calculated in equation (34-1). R The corneal radius R can be calculated by substituting the solution of L The corneal radius R can also be calculated by substituting the solution of

[0414] Furthermore, in equation (31-5), the corneal radius R and the calculated β R Alternatively, the depth information Len can be calculated by applying the corneal radius R and the calculated β L By applying the above formula, the depth information Len can be obtained. This allows the value of the corneal radius R to be calculated. In this way, being able to measure the corneal radius of a subject can be useful for analyzing the growth of a child's eyeball and other phenomena.

[0415] The key points of the manipulation of the above formulas will now be summarized. The key points of the manipulation of the formulas in the corneal central depth calculation unit have already been shown, but in this case, a conditional formula is derived and solved from a similar perspective for the Purkinje image that was not selected as the reference Purkinje image. That is, for the Purkinje image that was not selected and the infrared light source and specular reflection point that caused it, the 1L conditional formula, the 2L conditional formula [modified formula (32-5)], and the 3L conditional formula that correspond to the 1st conditional formula [formula (31-3)], the 2nd conditional formula [formula (31-4), modified formula (31-5)], and the 3rd conditional formula [formula (31-6), modified formula (34-1)] are derived and solved.

[0416] The parameters in the three conditional expressions above, for example, ∠α and ∠β, are respectively defined as ∠α L and ∠β L The parameters Len, R, and ∠γ are the same as those in the previous three conditional expressions.

[0417] By the same operation as that used to derive the third conditional expression, R and ∠β are used as unknowns. L However, the mean value substitution process is not performed. Instead, the following operations are performed. First, Len is eliminated from the second and second L conditional expressions, and β and β are used as unknowns. L Next, R is eliminated from the third conditional expression and the third L conditional expression, and β and β are left as unknowns. L The fifth conditional expression (expression (34-3)) including the above is derived.

[0418] The fourth and fifth conditions have two unknowns, β and β. L So, the unknowns β and β L Solving these as simultaneous equations with two unknowns, β and β L is set as a known quantity. Then, the third conditional expression or the third L conditional expression, in which R is the only unknown, is solved to set R as a known quantity. Next, a conditional expression in which Len is the only unknown, such as the first conditional expression or the second conditional expression, or the first L conditional expression or the second L conditional expression, is solved to set Len as a known quantity. By performing the above operations, R and Len can be calculated without substituting an average value for the corneal radius R. The above summarizes the key points of the operations on the formulas.

[0419] Seventh Embodiment Next, a seventh embodiment of a gaze analysis device based on the concept of the present invention will be described. This gaze analysis device is a glasses-type or goggle-type gaze analysis terminal shown in FIG. 2, and is equipped with a field of view camera 4. A distance sensor (not shown) is provided near the field of view camera. The distance sensor is adjusted to a direction approximately parallel to the lens optical axis of the field of view camera 4. The distance sensor improves the function of the seventh embodiment of the gaze analysis device regarding the superimposition of a gaze point on the image of the field of view camera 4 shown in FIG. 20.

[0420] To calculate the three-dimensional coordinates of the gaze point GP, it is necessary to set the distance VSL from the lens center of the field of view camera 4 to the virtual screen 46. If a distance sensor installed near the field of view camera can measure the distance to a wall or object in the gaze direction, the distance VSL can be automatically set based on that distance. The gaze analysis device according to the seventh embodiment, which is equipped with such a mechanism, has the advantage that the distance to the gaze target can be automatically set. [Explanation of symbols]

[0421] CPvx,CPvy: Intersection of the cutting plane and the virtual screen, E···Eyeball, E1···Sclera, E11···Center of rotation E11i...Turning center projection point, E12...Vitreous body, E2...Cornea, E21···Corneal center (corneal curvature center), E21i···Projection point of the corneal center, E3...iris, E31...pupil, E4...lens, E51: Superior rectus muscle, E52: Inferior rectus muscle, E53: Lateral rectus muscle, E54···Medial rectus muscle, Gaze1_Line···Line extension of gaze vector, Len: The distance from the GX axis to the center of the cornea. Gaze1, Gaze2...gaze vector, Gaze1_i···2D gaze vector, Gaze1_i_ex···2D extension line, N···normal vector, Pu···Pupil center, GP···Gaze point, GPi···Gaze projection point, VSL: Virtual screen distance, PRx, PRy: Projection plane, R ave ···mean corneal radius, Rot_R: Distance between the pupil center and the center of rotation, sd...Distance between the lens and the image plane, 1. Frame, 2. Arm, 3. Nose pad, 4. Vision camera, 41: Lens; 42: Imaging surface; 43: Signal and power cables; 44: Imaging range, 45: Recorded image by field of view camera, 46. ​​Virtual screen, 47. Information display board, 5,5Q···Eyeball camera, 51···Lens, 52···Imaging surface, 52v···Virtual screen, 53···Signal and power cables, 55R, 55L, 55B···Infrared light source, 56···Ocular camera optical axis, 6, 6Q··· Control box, 61··· Calculation unit, 62··· Memory unit, 621...Program area, 622...Image area, 7...Reflected light, 71,71L,71R,71B...Specular reflected light, 71iL, 71iR, 71iB···Purkinje statue, 72···Diffuse reflection light, 8···AR marker.

Claims

1. an eye camera capable of photographing the corneal spherical surface of the eyeball of the subject; and three infrared light sources for irradiating the corneal spherical surface with infrared light; a first straight line connecting an installation position of a first infrared light source and an installation position of a second infrared light source among the three infrared light sources passes through a center of a lens provided in the eye camera, a third infrared light source among the three infrared light sources is disposed on a second straight line that is in a first plane that includes the first straight line and is perpendicular to the optical axis of the lens, and that intersects with the first straight line at a center of the lens at a predetermined angle; Alternatively, a configuration is adopted in which an extended second line is assumed to be perpendicular to both the optical axis and the first line at the center of the lens, and a third infrared light source among the three infrared light sources is disposed on the extended second line; A gaze analysis device that detects three image points that may have been generated by infrared light rays emitted from the first infrared light source, the second infrared light source, and the third infrared light source on the imaging surface of the eye camera, which is provided as a second plane perpendicular to the lens optical axis, and if it is determined that the triangle with the three image points as vertices has a triangular shape based on the arrangement of the first infrared light source, the second infrared light source, and the third infrared light source, considers the three image points to be a first Purkinje image, a second Purkinje image, and a third Purkinje image caused by the first infrared light source, the second infrared light source, and the third infrared light source, respectively, and calculates the position of the center of curvature of the corneal spherical surface from the positional relationship of the three image points.

2. The first infrared light source and the second infrared light source are provided on the first straight line that is in the first plane, and the third infrared light source is provided on the second straight line that intersects with the first straight line at the lens center at the predetermined angle, a two-dimensional coordinate system defining a position within the imaging surface, the origin of which is an intersection of the second plane and the lens optical axis, an x-axis passing through the origin o and parallel to the first straight line, and a y-axis perpendicular to the x-axis at the origin o; 2. The gaze analysis device of claim 1, further comprising: a GX-GY-GZ coordinate system, as a three-dimensional coordinate system defining the position of the center of curvature of the corneal spherical surface, wherein the lens optical axis is defined as the GZ axis; the intersection of the GZ axis and the first plane, i.e., the lens center, is defined as the origin GO; a first axis that is within the first plane, passes through the origin GO, and is parallel to the x-axis is defined as the GX axis; and a second axis that is within the first plane, is perpendicular to the GX axis at the origin GO, and is parallel to the y-axis is defined as the GY axis.

3. drawing a straight line as a first projection line between the first Purkinje image and the second Purkinje image on the imaging plane; a straight line is drawn as a second projection line that passes through the third Purkinje image and intersects with the first projection line at the predetermined angle; 3. The gaze analysis device according to claim 2, wherein the two-dimensional coordinates of the intersection of the second projection line and the first projection line are calculated as coordinates of a corneal center projection point where the center of curvature of the corneal spherical surface is projected onto the imaging surface.

4. A storage unit is provided, As a processing block, an eyeball image acquisition unit that captures an eyeball image including the corneal spherical surface of the eyeball of the subject, which is projected and formed on the imaging surface, into the storage unit so that the image can be processed; a first eyeball image analysis unit that detects a pupil region from the acquired eyeball image; a two-dimensional pupil center detection unit that fits a circle to the pupil region and calculates coordinate values ​​in the two-dimensional coordinate system of a pupil center that is the center of the pupil region; a second eyeball image analysis unit that detects a Purkinje image group including the first Purkinje image, the second Purkinje image, and a third Purkinje image from the acquired eyeball image; a two-dimensional corneal center detection unit that draws the first projection line and the second projection line with respect to the group of Purkinje images and calculates the coordinates of the corneal center projection point, an A1-ad processing unit that calculates an angle GR formed by the x-axis and the first projection line when the x-axis and the first projection line are not parallel, and after the angle GR is calculated, performs a rotation correction of an appropriate angle with an origin o as an axis on the image captured by the eyeball image acquisition unit to correct the x-axis and the first projection line so that they are parallel; 4. The gaze analysis device according to claim 3, wherein the image corrected and output by the A1-ad processing unit is used as the eyeball image and processed in a subsequent processing block including the first eyeball image analysis unit, the two-dimensional pupil center detection unit, the second eyeball image analysis unit, and the two-dimensional cornea center detection unit.

5. The center of curvature of the corneal sphere is sometimes referred to as the corneal center, Further processing blocks include: In order to analyze a cross-sectional structure of the eyeball when it is assumed to be cut along a cutting plane determined by the first projection line and the corneal center, calculating an angle φ of intersection between the cutting plane and a GZ-GX plane, which is defined by the GZ axis and the GX axis, from a distance between the first projection line and the origin o in the two-dimensional coordinate system and a projection distance between the lens center and the imaging surface; one of the first Purkinje image and the second Purkinje image is selected as a reference Purkinje image, an infrared light source that causes the reference Purkinje image is referred to as a reference infrared light source, a specular reflection point as an object point of the reference Purkinje image is referred to as a reference specular reflection point, an intersection line between the first plane and the cutting plane coincides with the first straight line, the first straight line includes the lens center and coincides with the GX axis, a straight line is provided on the cutting plane as a CPZ axis perpendicular to the GX axis at the lens center GO, thereby setting a two-dimensional GX-CPZ coordinate system on the cutting plane with the lens center GO as the origin; In the GX-CPZ coordinate system, the angle γ formed by the direction from the lens center GO toward the corneal center and the CPZ axis is set as a known value based on the projection direction calculated based on the two-dimensional coordinate value of the corneal center projection point and the projection distance, and the angle φ, Regarding the angle α formed by the direction from the lens center GO toward the reference mirror surface reflection point and the CPZ axis, the angle α is set to a known value based on the projection direction calculated based on the two-dimensional coordinate value of the reference Purkinje image and the projection distance, and the angle φ, The distance along the GX axis from the lens center GO to the installation position of the reference light source is set to a known design value ms, The radius of curvature, which is the distance from the corneal spherical surface to the corneal center, is defined as an unknown quantity R, and the CPZ axis component of the distance from the lens center GO to the corneal center is defined as an unknown quantity Len. The angle formed by the direction of the reference specular reflection point and the CPZ axis when the corneal center is taken as the starting point is defined as an unknown quantity β. The condition under which a physical phenomenon occurs in which a light ray emitted from the reference light source is specularly reflected at the reference specular reflection point and incident on the lens center GO is defined as: applying trigonometric functions to the ms, the α, the Len, the R, and the β, and deriving a first conditional expression including the Len, the R, and the β as unknowns based on the fact that, for each component of the GX axis and the CPZ axis, a vector from the installation position of the reference infrared light source to the lens center GO is equal to the vector sum of a vector from the installation position of the reference infrared light source to the reference specular reflection point and a vector from the reference specular reflection point to the lens center GO; applying trigonometric functions to the ms, the γ, the Len, the R, and the β, and deriving a second conditional expression including the Len, the R, and the β as unknowns based on the fact that, for each component of the GX axis and the CPZ axis, the vector from the lens center GO to the reference specular reflection point is equal to the vector sum of the vector from the lens center GO to the corneal center and the vector from the corneal center to the reference specular reflection point; Eliminating the unknown Len from the first conditional expression and the second conditional expression to derive a third conditional expression including the unknowns R and β; Below, the average value substitution process is as follows: The unknown quantity R included in the third conditional expression is the average human corneal radius R ａｖｅ Substituting the above formula to make the only unknown variable β, then solving the third conditional expression to make β a known variable, the corneal radius R ａｖｅ a corneal center depth calculation unit that substitutes the β that has become a known number into the first conditional expression or the second conditional expression to make the Len the only unknown number in the first conditional expression or the second conditional expression, and then solves the first conditional expression or the second conditional expression to make the Len a known number; The three-dimensional coordinates of the corneal center are calculated using the known value Len, the γ, and the intersection angle φ, and then the projection direction from the lens center GO to the pupil center and the distance between the corneal center and the pupil center are calculated as the corneal radius R ａｖｅ and a three-dimensional pupil center / corneal center calculation unit that calculates three-dimensional coordinates of the pupil center from the two-dimensional coordinates of the pupil center, which are known quantities, under the condition that:

6. Further processing blocks include: a first gaze vector calculation unit that calculates a first gaze vector having a start point at a position defined by the three-dimensional coordinates of the corneal center calculated by the three-dimensional pupil center / corneal center calculation unit and an end point at a position defined by the three-dimensional coordinates of the pupil center; an A8-ad processing unit that performs an inverse rotation transformation around the GZ axis by the appropriate angle on a three-dimensional vector including the calculated first line of sight vector, The gaze analysis device of claim 5, wherein when the A1-ad processing unit processes an image converted as the eyeball image, the A8-ad processing unit outputs the first gaze vector to which the inverse transformation processing has been applied.

7. Further processing blocks include: a two-dimensional rotation center calculation unit that processes in a two-dimensional coordinate system, draws a line as a two-dimensional extended line between the pupil center detected by the two-dimensional pupil center detection unit and the corneal center projection point detected by the two-dimensional corneal center detection unit, and creates the two-dimensional extended line every time a calculation result is updated from the two-dimensional pupil center detection unit and the two-dimensional corneal center detection unit, and sets the intersection coordinates of the latest two-dimensional extended line and the previously created two-dimensional extended line as a two-dimensional rotation center; a conversion process from a two-dimensional coordinate system to a three-dimensional coordinate system, the conversion process including: calculating, updating, and storing three-dimensional coordinates of a center of rotation from the three-dimensional coordinates of the pupil center and the three-dimensional coordinates of the cornea center calculated by the three-dimensional pupil center / cornea center calculation unit using a proportional relationship between the pupil center coordinates calculated by the two-dimensional pupil center detection unit, the cornea center projection point calculated by the two-dimensional cornea center detection unit, and the two-dimensional center of rotation calculated by the two-dimensional center of rotation calculation unit; and further calculating, updating, and storing a radius of rotation from the three-dimensional coordinates of the center of rotation and the three-dimensional coordinates of the pupil center; a three-dimensional turning center / turning radius calculation unit that calculates the pupil center three-dimensional coordinates from the pupil center coordinates obtained by the two-dimensional pupil center detection unit and the updated and saved three-dimensional turning center coordinates and turning radius in a processing cycle in which no calculation results related to the downstream of the second eyeball image analysis unit are obtained due to a failure in detecting the Purkinje image group by the second eyeball image analysis unit; a second gaze vector calculation unit that calculates a second gaze vector having a start point in the three-dimensional coordinate of the center of rotation calculated by the three-dimensional center of rotation and radius of rotation calculation unit, and an end point in the three-dimensional coordinate of the pupil center calculated by the three-dimensional pupil center and cornea center calculation unit or the three-dimensional coordinate of the pupil center calculated by the three-dimensional center of rotation and radius of rotation calculation unit, The gaze analysis device of claim 5, wherein when the A1-ad processing unit processes an image converted as the eyeball image, the A8-ad processing unit outputs the second gaze vector to which the inverse transformation processing has been applied.

8. 8. The gaze analysis device according to claim 6, wherein the first straight line on which the first infrared light source and the second infrared light source are installed is aligned with the GX axis, and the second straight line on which the third infrared light source is installed is aligned with the GY axis.

9. a frame to be hung on both ears of the subject, arms fixed to the frame and having tips extending up to the front of the face of the subject, a nose pad for supporting the frame on the nose of the subject, a field of view camera located at the top front of the face of the subject and capable of photographing a scene in the same direction as the direction that the subject can see, and a gaze analysis terminal having the eye camera located at the tip of the arm and capable of photographing the cornea of ​​the eye of the subject; 9. The gaze analysis device of claim 8, further comprising: the calculation unit for performing a series of processes from the eyeball image acquisition unit to the first gaze vector calculation unit or the second gaze vector calculation unit; and a control box containing the memory unit and a power supply; and capable of superimposing a gaze mark on the gaze point calculated based on the first gaze vector or the second gaze vector on the image recorded by the field of view camera.

10. 10. The gaze analysis device of claim 9, further comprising: a distance sensor near the field of view camera capable of measuring distance in a direction visible to the subject; a virtual screen positioned a distance from the field of view camera equal to the distance output by the distance sensor as the distance to an object in front of the subject; a gaze point set on the virtual screen based on the first gaze vector or the second gaze vector; and a gaze mark superimposed on an image recorded by the field of view camera at a location corresponding to the gaze point.

11. one or more eye cameras provided above, below, or to the left and right of the information display board so as to be able to photograph the corneas of one or more subjects; a control box that houses the calculation unit, the storage unit, and a power source for performing a series of processes from the eyeball image acquisition unit to the first gaze vector calculation unit or the second gaze vector calculation unit, The gaze analysis device according to claim 8 , further comprising: a time-series change in the position of the gaze point on the information display board calculated based on the first gaze vector or the second gaze vector, the time-series change being recorded.

12. As an additional process in the corneal center depth calculation unit, Of the first Purkinje image or the second Purkinje image, a Purkinje image that is not the reference Purkinje image is referred to as a second reference Purkinje image, an infrared light source that causes the second reference Purkinje image is referred to as a second reference infrared light source, and a specular reflection point as an object point of the second reference Purkinje image is referred to as a second reference specular reflection point, The angle α formed by the direction from the lens center GO toward the second reference mirror surface reflection point and the CPZ axis Ｌ Regarding the second reference Purkinje image, the angle α is calculated based on the two-dimensional coordinate value of the second reference Purkinje image and the projection direction calculated based on the projection distance. Ｌ are known quantities, The distance along the GX axis from the lens center GO to the installation position of the second reference light source is a known design value ms Ｌ year, The angle formed by the direction of the second reference specular reflection point and the CPZ axis when the corneal center is taken as the origin is defined as an unknown quantity β L With that in mind, The condition under which a physical phenomenon occurs in which a light ray emitted from the second reference light source is specularly reflected at the second reference specular reflection point and is incident on the lens center GO is defined as: Said ms Ｌ and the above α Ｌ and the Len, the R, and the β Ｌ By applying a trigonometric function to each component of the GX axis and the CPZ axis, the vector from the installation position of the second reference infrared light source to the lens center GO is equal to the vector sum of the vector from the installation position of the second reference infrared light source to the second reference specular reflection point and the vector from the second reference specular reflection point to the lens center GO. Ｌ Derive the first L conditional expression including Said ms Ｌ and the γ, the Len, the R, and the β Ｌ By applying a trigonometric function to each component of the GX axis and the CPZ axis, the vector from the lens center GO to the second reference specular reflection point is equal to the vector sum of the vector from the lens center GO to the corneal center and the vector from the corneal center to the second reference specular reflection point, and the Len, the R, and the β are used as unknowns based on the fact that Ｌ Derive a second L-condition including The Len is eliminated from the first L conditional expression and the second L conditional expression, and the R and the β are left as unknowns. Ｌ Derive a third L-condition including Without performing the mean value substitution process, The Len is eliminated from the second conditional expression and the second L conditional expression, and the β and β are used as unknowns. Ｌ The fourth conditional expression including The R is eliminated from the third conditional expression and the third L conditional expression, and the β and β are used as unknowns. Ｌ Derive a fifth conditional expression including The unknown quantities β and β in the fourth and fifth conditional expressions are Ｌ After constructing a simultaneous equation with two unknowns including Ｌ are known quantities, The conditional expression in which the only unknown quantity is R, i.e., the third conditional expression or the third L conditional expression, is solved to make R a known quantity, The gaze analysis device of claim 5, wherein the conditional equation in which the only unknown quantity is Len, for example, the first conditional equation or the second conditional equation, or the first L conditional equation or the second L conditional equation, is then solved to make Len a known quantity.

13. an eye camera capable of photographing the corneal spherical surface of the eyeball of the subject; and three infrared light sources for irradiating the corneal spherical surface with infrared light; a first straight line connecting an installation position of a first infrared light source and an installation position of a second infrared light source among the three infrared light sources is disposed at a center of a lens of the eyeball camera so as to be perpendicular to an optical axis of the lens; a third infrared light source among the three infrared light sources is disposed on a second straight line that is in a first plane that includes the first straight line and is perpendicular to the lens optical axis, and that intersects with the first straight line at the center of the lens at a predetermined angle; Further, in a calculation processing environment having a storage unit and a calculation unit, the corneal image of the eyeball projected onto an imaging surface on a second plane perpendicular to the optical axis by the imaging action of the lens is stored in the storage unit and then processed by the calculation unit, or the corneal image of the eyeball projected onto the imaging unit is directly processed by the calculation unit, an image acquisition step 2 of acquiring a corneal image of the eyeball projected on the imaging surface in a manner that allows for analysis processing, or acquiring the image in a manner that allows for analysis processing via the storage unit; a Purkinje image group detection step 4 in which image points that may have been generated by infrared light rays emitted from the first infrared light source, the second infrared light source, and the third infrared light source are detected from the corneal image of the eyeball, and if it is determined that a triangle having the three image points as vertices has a triangular shape based on the arrangement of the first infrared light source, the second infrared light source, and the third infrared light source, the three image points are considered to have been detected as a Purkinje image group consisting of a first Purkinje image, a second Purkinje image, or a third Purkinje image caused by the first infrared light source, the second infrared light source, and the third infrared light source, respectively, and the process proceeds to the next step; a corneal central projection point calculation step 5 for drawing a straight line as a first projection line between the first Purkinje image and the second Purkinje image for the detected group of Purkinje images, drawing a straight line as a second projection line that passes through the third Purkinje image and intersects with the first projection line at the predetermined angle, and calculating two-dimensional coordinates of an intersection of the second projection line and the first projection line as a point where the center of curvature of the corneal spherical surface is projected onto the imaging plane, and treating the calculated coordinates as a corneal central projection point.

14. The center of curvature of the corneal sphere is sometimes called the corneal center. a pupil detection step 3 for detecting a pupil region from a difference in brightness of the corneal image of the eyeball and calculating two-dimensional coordinates of the pupil center through circular fitting is provided between the image acquisition step 2 and the Purkinje image group detection step 4, and only when the pupil region is detected in the pupil detection step 3, the result is determined to be "true" and the process proceeds to the Purkinje image group detection step 4, and otherwise the process returns to the image acquisition step 2 and captures the corneal image of the eyeball; After the corneal center projection point calculation step 5, a corneal center Len calculation step 6 is performed to calculate depth information as a distance Len from an analysis reference line that coincides with the first straight line to the corneal center. and a pupil center / cornea center three-dimensional coordinate calculation step 8 for calculating the pupil center three-dimensional coordinate and the cornea center three-dimensional coordinate by applying the depth information to the two-dimensional coordinate of the pupil center and the cornea center projection point.

15. 15. The gaze analysis program according to claim 14, wherein the calculation unit is caused to perform a process including a first gaze vector calculation step 9 of calculating a first gaze vector having the three-dimensional coordinates of the cornea center as a start point and the three-dimensional coordinates of the pupil center as an end point.

16. A series of processes from the image acquisition step 2 to the gaze vector calculation step is referred to as a processing cycle. A process in a two-dimensional coordinate system performed after the pupil center / cornea center three-dimensional coordinate calculation step 8, a straight line is drawn as a two-dimensional extended line between the pupil center obtained by the two-dimensional pupil center detection step 8 and the corneal center projection point obtained by the corneal center projection point calculation step 5, and the two-dimensional extended line is drawn every time the calculation results are updated by the two-dimensional pupil center detection step 8 and the corneal center projection point calculation step 5, and the intersection of the latest two-dimensional extended line and the two-dimensional extended line drawn in the previous processing cycle is set as a two-dimensional rotation center; a rotation center 3D coordinate updating step 10 for calculating, updating, and maintaining a rotation center 3D coordinate based on the proportional relationship between the two-dimensional rotation center, the pupil center coordinate, and the corneal center projection point, as well as the pupil center 3D coordinate and the corneal center 3D coordinate; a turning center 3D coordinate determination step 11 for determining that the updated turning center 3D coordinate is valid if there is no significant change from the turning center 3D coordinate updated in the previous processing cycle; and a second gaze vector calculation step 14 of calculating a second gaze vector having the three-dimensional coordinate of the center of rotation determined to be valid as a start point and the three-dimensional coordinate of the center of the pupil as an end point.

17. A series of processes from the image acquisition step 2 to the gaze vector calculation step is referred to as a processing cycle. a rotation center 3D coordinate update step 10, after the first line of sight vector calculation step 9, for calculating and updating a rotation center 3D coordinate based on a point where lines extending in the direction of the first line of sight vector behind a starting point of the first line of sight vector before and after the rotation of the eyeball intersect with each other; a turning center 3D coordinate determination step 11 for determining that the updated turning center 3D coordinate is valid if there is no significant change from the turning center 3D coordinate updated in the previous processing cycle; a second gaze vector calculation step 14 of calculating a second gaze vector having the three-dimensional coordinates of the rotation center determined to be valid as a start point and the three-dimensional coordinates of the pupil center as an end point; a second line-of-sight vector calculation step 15 for calculating the second line-of-sight vector having the turning center three-dimensional coordinate updated and saved in the turning center three-dimensional coordinate update step 10 as a starting point in a preprocessing cycle when the Purkinje image group is not detected in the Purkinje image group detection step 4; and if it is determined in said Purkinje image group detection step 4 that the Purkinje image group is not detected, selecting the second gaze vector calculated in said second gaze vector calculation step 15 as an output gaze vector, and updating an attention position based on the output gaze vector in a display viewpoint position update step 17.

18. a two-dimensional coordinate system defining a position within the imaging surface, the origin of which is an intersection of the second plane and the lens optical axis, an x-axis passing through the origin o and parallel to the first straight line, and a y-axis perpendicular to the x-axis at the origin o; Furthermore, in a computing environment for processing the image acquired in the image acquisition step 2 under an imaging configuration in which a GX-GY-GZ coordinate system is set as a three-dimensional coordinate system defining the position of the center of curvature of the corneal spherical surface, the lens optical axis is the GZ axis, the intersection of the GZ axis and the first plane, i.e., the lens center, is the origin GO, a first axis that is in the first plane, passes through the origin GO, and is parallel to the x-axis is the GX axis, and a second axis that is in the first plane, is perpendicular to the GX axis at the origin GO, and is parallel to the y-axis is the GY axis, In order to analyze a cross-sectional structure of the eyeball when it is assumed to be cut along a cutting plane determined by the first projection line and the corneal center, In the corneal center Len calculation step 6, calculating an angle φ of intersection between the cutting plane and a GZ-GX plane, which is defined by the GZ axis and the GX axis, from a distance between the first projection line and the origin o in the two-dimensional coordinate system and a projection distance between the lens center and the imaging surface; one of the first Purkinje image and the second Purkinje image is selected as a reference Purkinje image, an infrared light source that causes the reference Purkinje image is referred to as a reference infrared light source, a specular reflection point as an object point of the reference Purkinje image is referred to as a reference specular reflection point, an intersection line between the first plane and the cutting plane coincides with the first straight line, the first straight line includes the lens center and coincides with the GX axis, a straight line is provided on the cutting plane as a CPZ axis perpendicular to the GX axis at the lens center GO, thereby setting a two-dimensional GX-CPZ coordinate system on the cutting plane with the lens center GO as the origin; In the GX-CPZ coordinate system, the angle γ formed by the direction from the lens center GO toward the corneal center and the CPZ axis is set as a known value based on the projection direction calculated based on the two-dimensional coordinate value of the corneal center projection point and the projection distance, and the angle φ, Regarding the angle α formed by the direction from the lens center GO toward the reference mirror surface reflection point and the CPZ axis, the angle α is set to a known value based on the projection direction calculated based on the two-dimensional coordinate value of the reference Purkinje image and the projection distance, and the angle φ, The distance along the GX axis from the lens center GO to the installation position of the reference light source is set to a known design value ms, The radius of curvature, which is the distance from the corneal spherical surface to the corneal center, is defined as an unknown quantity R, and the CPZ axis component of the distance from the lens center GO to the corneal center is defined as an unknown quantity Len. The angle formed by the direction of the reference specular reflection point and the CPZ axis when the corneal center is taken as the starting point is defined as an unknown quantity β. The condition under which a physical phenomenon occurs in which a light ray emitted from the reference light source is specularly reflected at the reference specular reflection point and incident on the lens center GO is defined as: applying trigonometric functions to the ms, the α, the Len, the R, and the β, and deriving a first conditional expression including the Len, the R, and the β as unknowns based on the fact that, for each component of the GX axis and the CPZ axis, a vector from the installation position of the reference infrared light source to the lens center GO is equal to the vector sum of a vector from the installation position of the reference infrared light source to the reference specular reflection point and a vector from the reference specular reflection point to the lens center GO; applying trigonometric functions to the ms, the γ, the Len, the R, and the β, and deriving a second conditional expression including the Len, the R, and the β as unknowns based on the fact that, for each component of the GX axis and the CPZ axis, the vector from the lens center GO to the reference specular reflection point is equal to the vector sum of the vector from the lens center GO to the corneal center and the vector from the corneal center to the reference specular reflection point; Eliminating the unknown Len from the first conditional expression and the second conditional expression to derive a third conditional expression including the unknowns R and β; Below, the average value substitution process is as follows: The unknown quantity R included in the third conditional expression is the average human corneal radius R ａｖｅ Substituting the above formula to make the only unknown variable β, then solving the third conditional expression to make β a known variable, the corneal radius R ａｖｅ 15. The gaze analysis program according to claim 14, wherein the calculation unit is caused to perform a process of solving the first conditional expression or the second conditional expression to make Len a known quantity by substituting the β that has become a known quantity into Len, so that the only unknown quantity in the first conditional expression or the second conditional expression is Len.

19. As an additional process in the corneal center Len calculation step 6, Of the first Purkinje image or the second Purkinje image, a Purkinje image that is not the reference Purkinje image is referred to as a second reference Purkinje image, an infrared light source that causes the second reference Purkinje image is referred to as a second reference infrared light source, and a specular reflection point as an object point of the second reference Purkinje image is referred to as a second reference specular reflection point, The angle α formed by the direction from the lens center GO toward the second reference mirror surface reflection point and the CPZ axis Ｌ Regarding the second reference Purkinje image, the angle α is calculated based on the two-dimensional coordinate value of the second reference Purkinje image and the projection direction calculated based on the projection distance. Ｌ are known quantities, The distance along the GX axis from the lens center GO to the installation position of the second reference light source is a known design value ms Ｌ year, The angle formed by the direction of the second reference specular reflection point and the CPZ axis when the corneal center is taken as the origin is defined as an unknown quantity β L With that in mind, The condition under which a physical phenomenon occurs in which a light ray emitted from the second reference light source is specularly reflected at the second reference specular reflection point and is incident on the lens center GO is defined as: Said ms Ｌ and the above α Ｌ and the Len, the R, and the β Ｌ By applying a trigonometric function to each component of the GX axis and the CPZ axis, the vector from the installation position of the second reference infrared light source to the lens center GO is equal to the vector sum of the vector from the installation position of the second reference infrared light source to the second reference specular reflection point and the vector from the second reference specular reflection point to the lens center GO. Ｌ Derive the first L conditional expression including Said ms Ｌ and the γ, the Len, the R, and the β Ｌ By applying a trigonometric function to each component of the GX axis and the CPZ axis, the vector from the lens center GO to the second reference specular reflection point is equal to the vector sum of the vector from the lens center GO to the corneal center and the vector from the corneal center to the second reference specular reflection point, and the Len, the R, and the β are used as unknowns based on the fact that Ｌ Derive a second L-condition including The Len is eliminated from the first L conditional expression and the second L conditional expression, and the R and the β are left as unknowns. Ｌ Derive a third L-condition including Without performing the mean value substitution process, The Len is eliminated from the second conditional expression and the second L conditional expression, and the β and β are used as unknowns. Ｌ The fourth conditional expression including The R is eliminated from the third conditional expression and the third L conditional expression, and the β and β are used as unknowns. Ｌ Derive a fifth conditional expression including The unknown quantities β and β in the fourth and fifth conditional expressions are Ｌ After constructing a simultaneous equation with two unknowns including Ｌ are known quantities, The conditional expression in which the only unknown quantity is R, i.e., the third conditional expression or the third L conditional expression, is solved to make R a known quantity, The gaze analysis program of claim 18 then causes the calculation unit to perform a process of solving a conditional equation in which the only unknown quantity is Len, for example, the first conditional equation or the second conditional equation, or the first L conditional equation or the second L conditional equation, to make Len a known quantity.

20. an eye camera capable of photographing the corneal spherical surface of the eyeball of the subject; and three infrared light sources for irradiating the corneal spherical surface with infrared light; a first straight line connecting an installation position of a first infrared light source and an installation position of a second infrared light source among the three infrared light sources is disposed at a center of a lens of the eyeball camera so as to be perpendicular to an optical axis of the lens; an image acquisition step 2 in which a third infrared light source among the three infrared light sources is disposed on a second straight line that is in a first plane that includes the first straight line and is perpendicular to the lens optical axis, and that intersects with the first straight line at the center of the lens at a predetermined angle, and the corneal image of the eyeball projected on the imaging surface is input into a calculation unit in a manner that allows analysis and processing; a Purkinje image group detection step 4 in which image points that may have been generated by infrared light rays emitted from the first infrared light source, the second infrared light source, and the third infrared light source are detected from the corneal image of the eyeball, and when it is determined that a triangle having the three image points as vertices has a triangular shape based on the arrangement of the first infrared light source, the second infrared light source, and the third infrared light source, it is determined that a Purkinje image group consisting of the first Purkinje image, the second Purkinje image, or the third Purkinje image caused by the first infrared light source, the second infrared light source, and the third infrared light source, respectively, has been detected, and the process proceeds to the next step; and a corneal central projection point calculation step 5 for drawing a straight line as a first projection line between the first Purkinje image and the second Purkinje image for the detected group of Purkinje images, drawing a straight line as a second projection line that passes through the third Purkinje image and intersects with the first projection line at the predetermined angle, and calculating two-dimensional coordinates of an intersection of the second projection line and the first projection line as a point where the center of curvature of the corneal spherical surface is projected onto the imaging plane, and treating the calculated two-dimensional coordinates as a corneal central projection point.

21. The center of curvature of the corneal sphere is sometimes called the corneal center. a pupil detection step 3 for detecting a pupil region from a difference in brightness of the corneal image of the eyeball and calculating two-dimensional coordinates of the pupil center through circular fitting is provided between the image acquisition step 2 and the Purkinje image group detection step 4, and only when the pupil region is detected in the pupil detection step 3, the result is determined to be "true" and the process proceeds to the Purkinje image group detection step 4, and otherwise the process returns to the image acquisition step 2 and captures the corneal image of the eyeball; Following the corneal center projection point calculation step 5, a corneal center Len calculation step 6 is performed to calculate depth information as a distance Len from an analysis reference line that coincides with the first straight line to the corneal center.

21. The gaze analysis method according to claim 20, further comprising a pupil center / cornea center three-dimensional coordinate calculation step 8 for calculating the pupil center three-dimensional coordinate and the cornea center three-dimensional coordinate by applying the depth information to the two-dimensional coordinate of the pupil center and the cornea center projection point.

22. 22. The gaze analysis method according to claim 21, further comprising a first gaze vector calculation step 9 of calculating a first gaze vector having the three-dimensional coordinates of the cornea center as a start point and the three-dimensional coordinates of the pupil center as an end point.

23. A series of processes from the image acquisition step 2 to the gaze vector calculation step is referred to as a processing cycle. A process in a two-dimensional coordinate system performed after the pupil center / cornea center three-dimensional coordinate calculation step 8, a straight line is drawn as a two-dimensional extended line between the pupil center obtained by the two-dimensional pupil center detection step 8 and the corneal center projection point obtained by the corneal center projection point calculation step 5, and the two-dimensional extended line is created every time the calculation results are updated by the two-dimensional pupil center detection step 8 and the corneal center projection point calculation step 5, and the intersection of the latest two-dimensional extended line and the two-dimensional extended line drawn in the previous processing cycle is set as a two-dimensional rotation center; a rotation center 3D coordinate updating step 10 for calculating, updating, and maintaining a rotation center 3D coordinate based on the proportional relationship between the two-dimensional rotation center, the pupil center coordinate, and the corneal center projection point, as well as the pupil center 3D coordinate and the corneal center 3D coordinate; a turning center 3D coordinate determination step 11 for determining that the updated turning center 3D coordinate is valid if there is no significant change from the turning center 3D coordinate updated in the previous processing cycle; The gaze analysis method according to claim 21, further comprising: a second gaze vector calculation step 14 of calculating a second gaze vector having the three-dimensional coordinate of the center of rotation determined to be valid as a start point and the three-dimensional coordinate of the center of the pupil as an end point.

24. A series of processes from the image acquisition step 2 to the gaze vector calculation step is referred to as a processing cycle. a rotation center 3D coordinate update step 10, after the first line of sight vector calculation step 9, for calculating and updating a rotation center 3D coordinate based on a point where lines extending in the direction of the first line of sight vector behind a starting point of the first line of sight vector before and after the rotation of the eyeball intersect with each other; a turning center 3D coordinate determination step 11 for determining that the updated turning center 3D coordinate is valid if there is no significant change from the turning center 3D coordinate updated in the previous processing cycle; a second gaze vector calculation step 14 of calculating a second gaze vector having the three-dimensional coordinates of the rotation center determined to be valid as a start point and the three-dimensional coordinates of the pupil center as an end point; a second line-of-sight vector calculation step 15 for calculating the second line-of-sight vector having the turning center three-dimensional coordinate updated and saved in the turning center three-dimensional coordinate update step 10 as a starting point in a preprocessing cycle when the Purkinje image group is not detected in the Purkinje image group detection step 4; and a display viewpoint position updating step (17) of selecting the second gaze vector calculated in the second gaze vector calculation step (15) as an output gaze vector, and updating an attention position based on the output gaze vector, when it is determined in the Purkinje image group detection step (4) that the Purkinje image group has not been detected.

25. a two-dimensional coordinate system defining a position within the imaging surface, the origin of which is an intersection of the second plane and the lens optical axis, an x-axis passing through the origin o and parallel to the first straight line, and a y-axis perpendicular to the x-axis at the origin o; Furthermore, in an aspect of processing the image acquired in the image acquisition step 2 under an imaging configuration in which a GX-GY-GZ coordinate system is set as a three-dimensional coordinate system defining the position of the center of curvature of the corneal spherical surface, the lens optical axis is the GZ axis, the intersection of the GZ axis and the first plane, i.e., the lens center, is the origin GO, a first axis that is in the first plane, passes through the origin GO, and is parallel to the x-axis is the GX axis, and a second axis that is in the first plane, is perpendicular to the GX axis at the origin GO, and is parallel to the y-axis is the GY axis, In order to analyze a cross-sectional structure of the eyeball when it is assumed to be cut along a cutting plane determined by the first projection line and the corneal center, In the corneal center Len calculation step 6, calculating an angle φ of intersection between the cutting plane and a GZ-GX plane, which is defined by the GZ axis and the GX axis, from a distance between the first projection line and the origin o in the two-dimensional coordinate system and a projection distance between the lens center and the imaging surface; one of the first Purkinje image and the second Purkinje image is selected as a reference Purkinje image, an infrared light source that causes the reference Purkinje image is referred to as a reference infrared light source, a specular reflection point as an object point of the reference Purkinje image is referred to as a reference specular reflection point, an intersection line between the first plane and the cutting plane coincides with the first straight line, the first straight line includes the lens center and coincides with the GX axis, a straight line is provided on the cutting plane as a CPZ axis perpendicular to the GX axis at the lens center GO, thereby setting a two-dimensional GX-CPZ coordinate system on the cutting plane with the lens center GO as the origin; In the GX-CPZ coordinate system, the angle γ formed by the direction from the lens center GO toward the corneal center and the CPZ axis is set as a known value based on the projection direction calculated based on the two-dimensional coordinate value of the corneal center projection point and the projection distance, and the angle φ, Regarding the angle α formed by the direction from the lens center GO toward the reference mirror surface reflection point and the CPZ axis, the angle α is set to a known value based on the projection direction calculated based on the two-dimensional coordinate value of the reference Purkinje image and the projection distance, and the angle φ, The distance along the GX axis from the lens center GO to the installation position of the reference light source is set to a known design value ms, The radius of curvature, which is the distance from the corneal spherical surface to the corneal center, is defined as an unknown quantity R, and the CPZ axis component of the distance from the lens center GO to the corneal center is defined as an unknown quantity Len. The angle formed by the direction of the reference specular reflection point and the CPZ axis when the corneal center is taken as the starting point is defined as an unknown quantity β. The condition under which a physical phenomenon occurs in which a light ray emitted from the reference light source is specularly reflected at the reference specular reflection point and incident on the lens center GO is defined as: applying trigonometric functions to the ms, the α, the Len, the R, and the β, and deriving a first conditional expression including the Len, the R, and the β as unknowns based on the fact that, for each component of the GX axis and the CPZ axis, a vector from the installation position of the reference infrared light source to the lens center GO is equal to the vector sum of a vector from the installation position of the reference infrared light source to the reference specular reflection point and a vector from the reference specular reflection point to the lens center GO; applying trigonometric functions to the ms, the γ, the Len, the R, and the β, and deriving a second conditional expression including the Len, the R, and the β as unknowns based on the fact that, for each component of the GX axis and the CPZ axis, the vector from the lens center GO to the reference specular reflection point is equal to the vector sum of the vector from the lens center GO to the corneal center and the vector from the corneal center to the reference specular reflection point; Eliminating the unknown Len from the first conditional expression and the second conditional expression to derive a third conditional expression including the unknowns R and β; Below, the average value substitution process is as follows: The unknown quantity R included in the third conditional expression is the average human corneal radius R ａｖｅ Substituting the above formula to make the only unknown variable β, then solving the third conditional expression to make β a known variable, the corneal radius R ａｖｅ 22. The gaze analysis method according to claim 21, further comprising the step of substituting the β that has become a known quantity into Len, so that the only unknown quantity in the first conditional expression or the second conditional expression is Len, and then solving the first conditional expression or the second conditional expression to make Len a known quantity.

26. As an additional process in the corneal center Len calculation step 6, Of the first Purkinje image or the second Purkinje image, a Purkinje image that is not the reference Purkinje image is referred to as a second reference Purkinje image, an infrared light source that causes the second reference Purkinje image is referred to as a second reference infrared light source, and a specular reflection point as an object point of the second reference Purkinje image is referred to as a second reference specular reflection point, The angle α formed by the direction from the lens center GO toward the second reference mirror surface reflection point and the CPZ axis Ｌ Regarding the second reference Purkinje image, the angle α is calculated based on the two-dimensional coordinate value of the second reference Purkinje image, the projection direction calculated based on the projection distance, and the angle φ. Ｌ are known quantities, The distance along the GX axis from the lens center GO to the installation position of the second reference light source is a known design value ms Ｌ year, The angle formed by the direction of the second reference specular reflection point and the CPZ axis when the corneal center is taken as the origin is defined as an unknown quantity β Ｌ With that in mind, The condition under which a physical phenomenon occurs in which a light ray emitted from the second reference light source is specularly reflected at the second reference specular reflection point and is incident on the lens center GO is defined as: Said ms Ｌ and the above α Ｌ and the Len, the R, and the β Ｌ By applying a trigonometric function to each component of the GX axis and the CPZ axis, the vector from the installation position of the second reference infrared light source to the lens center GO is equal to the vector sum of the vector from the installation position of the second reference infrared light source to the second reference specular reflection point and the vector from the second reference specular reflection point to the lens center GO. Ｌ Derive the first L conditional expression including Said ms Ｌ and the γ, the Len, the R, and the β Ｌ By applying a trigonometric function to each component of the GX axis and the CPZ axis, the vector from the lens center GO to the second reference specular reflection point is equal to the vector sum of the vector from the lens center GO to the corneal center and the vector from the corneal center to the second reference specular reflection point, and the Len, the R, and the β are used as unknowns based on the fact that Ｌ Derive a second L-condition including The Len is eliminated from the first L conditional expression and the second L conditional expression, and the R and the β are left as unknowns. Ｌ Derive a third L-condition including Without performing the mean value substitution process, The Len is eliminated from the second conditional expression and the second L conditional expression, and the β and β are used as unknowns. Ｌ The fourth conditional expression including The R is eliminated from the third conditional expression and the third L conditional expression, and the β and β are used as unknowns. Ｌ Derive a fifth conditional expression including The unknown quantities β and β in the fourth and fifth conditional expressions are Ｌ After constructing a simultaneous equation with two unknowns including Ｌ are known quantities, The conditional expression in which the unknown quantity is only R, i.e., the third conditional expression or the third L conditional expression, is solved to make R a known quantity, The gaze analysis method of claim 25, further comprising the step of solving a conditional equation in which the only unknown quantity is Len, for example, the first conditional equation or the second conditional equation, or the first L conditional equation or the second L conditional equation, to make Len a known quantity.

Images