Eye-line direction detection device and eye-line direction detection method

Abstract

To reliably determine the direction of the subject's gaze. [Solution] The gaze direction detection device 1 comprises a camera 10 that acquires a face image by capturing the face of subject A, and an image processing device 20 that processes the face image acquired by the camera 10 to determine the gaze direction of subject A. The image processing device 20 detects the three-dimensional positions of the right pupil, left pupil, and nose based on the face image, identifies the three-dimensional position of the intersection of a second straight line that passes through the three-dimensional position of the nose and intersects perpendicularly with a first straight line connecting the three-dimensional positions of the right pupil and the three-dimensional positions of the left pupil, and the first straight line, and determines the gaze direction of subject A using the first distance between the three-dimensional position of the intersection and the three-dimensional position of the right pupil, the second distance between the three-dimensional position of the intersection and the three-dimensional position of the left pupil, and the third distance between the three-dimensional position of the intersection and the three-dimensional position of the nose.

Description

[Technical Field]

[0001] The present invention relates to a gaze direction detection device and a gaze direction detection method for detecting the direction of gaze from a person's image. [Background technology]

[0002] In recent years, devices have become widespread that use an optical system including a near-infrared light source and a video camera to obtain an image of a subject, detect the pupil center and the corneal reflection image center, and then detect the direction of gaze based on their relative positions and the three-dimensional position of the pupil (see Patent Documents 1, 2, and 3 below). These devices acquire an image (bright pupil image) by irradiating the subject's face with light that tends to make the pupil relatively brighter, acquire an image (dark pupil image) by irradiating the subject's face with light that tends to make the pupil relatively darker, and then calculate a difference image using these images to detect the pupil center of the subject and detect the direction of gaze based on the detection result. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Patent No. 5858433 [Patent Document 2] Patent No. 5915981 [Patent Document 3] Patent No. 4517049 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] In the devices described above, detecting the corneal reflection image is necessary to determine the direction of the subject's gaze. For example, if the deviation of the subject's gaze direction from the optical system becomes large, the corneal reflection image on the image may become distorted, not appear on the image, or overlap with the white of the eye. This can lead to a decrease in the accuracy of corneal reflection image detection or even failure to detect the corneal reflection image at all. As a result, it can become difficult to detect the direction of the subject's gaze.

[0005] This invention has been made in view of the above problems, and aims to provide a gaze direction detection device and a gaze direction detection method that can reliably determine the gaze direction of a subject. [Means for solving the problem]

[0006] To solve the above problems, a gaze direction detection device according to one embodiment of the present invention comprises an optical system that acquires a face image by imaging the face of a subject, and a processing unit that processes the face image acquired by the optical system to determine the gaze direction of the subject. The processing unit detects the three-dimensional positions of the right pupil, the left pupil, and the nose based on the face image, identifies the three-dimensional position of the intersection of a second straight line that passes through the three-dimensional position of the nose and intersects perpendicularly with a first straight line connecting the three-dimensional positions of the right pupil and the three-dimensional positions of the left pupil, and the first straight line, and determines the gaze direction of the subject using a first distance between the three-dimensional position of the intersection and the three-dimensional position of the right pupil, a second distance between the three-dimensional position of the intersection and the three-dimensional position of the left pupil, and a third distance between the three-dimensional position of the intersection and the three-dimensional position of the nose.

[0007] Alternatively, a gaze direction detection device according to another embodiment of the present invention comprises an optical system that acquires a face image by imaging the face of a subject, and a processing unit that processes the face image acquired by the optical system to determine the gaze direction of the subject. The processing unit detects the three-dimensional positions of the right pupil, the left pupil, and a reference part set on the face based on the face image, identifies the three-dimensional position of the intersection of a second straight line that passes through the three-dimensional position of the reference part and intersects perpendicularly with a first straight line connecting the three-dimensional position of the right pupil and the three-dimensional position of the left pupil, and the first straight line, and determines the gaze direction of the subject using a first distance between the three-dimensional position of the intersection and the three-dimensional position of the right pupil, a second distance between the three-dimensional position of the intersection and the three-dimensional position of the left pupil, and a third distance between the three-dimensional position of the intersection and the three-dimensional position of the reference part.

[0008] Alternatively, another embodiment of the present invention provides a method for detecting the direction of gaze, which includes an optical system that acquires a facial image by imaging the eyes of a subject, and a processing unit that processes the facial image acquired by the optical system to determine the direction of the subject's gaze. Based on the facial image, the method detects the three-dimensional positions of the right pupil, the left pupil, and the nose, identifies the three-dimensional position of the intersection of a second straight line that passes through the three-dimensional position of the nose and intersects perpendicularly with a first straight line connecting the three-dimensional positions of the right pupil and the three-dimensional positions of the left pupil, and determines the direction of the subject's gaze using a first distance between the three-dimensional position of the intersection and the three-dimensional position of the right pupil, a second distance between the three-dimensional position of the intersection and the three-dimensional position of the left pupil, and a third distance between the three-dimensional position of the intersection and the three-dimensional position of the nose.

[0009] According to the line-of-sight direction detection device or method of the above-described embodiment, based on the face image of the subject acquired by the optical system, the three-dimensional positions of the right pupil, left pupil, and nose part (or reference part) are detected. Based on these, the three-dimensional position of the intersection point of the first straight line connecting the left and right pupils and the second straight line perpendicularly intersecting the first straight line from the nose part (or reference part) is specified. The line-of-sight direction of the subject is determined using the first distance between the intersection point and the right pupil, the second distance between the intersection point and the left pupil, and the third distance between the intersection point and the nose part (or reference part). Thus, since the line-of-sight direction of the subject can be determined based on the detection positions of the left and right pupils and the nose part (or reference part) on the face image, the line-of-sight direction can be stably determined regardless of the magnitude of the deviation of the line-of-sight direction with respect to the optical system as seen from the subject.

[0010] Here, the processing unit may determine the horizontal component of the line-of-sight direction of the subject from the relationship between the first distance and the second distance, and determine the vertical component of the line-of-sight direction of the subject from the third distance. When the line-of-sight direction of the subject changes horizontally, the relationship between the first distance and the second distance changes, and when the line-of-sight direction of the subject changes vertically, the third distance changes. By utilizing this property, the horizontal component of the line-of-sight direction of the subject and the vertical component of the line-of-sight direction of the subject can be stably determined.

[0011] Also, the processing unit may determine the horizontal component of the line-of-sight direction of the subject based on the ratio of the first distance to the second distance, and determine the vertical component of the line-of-sight direction of the subject based on the magnitude of the third distance. When the line-of-sight direction of the subject changes horizontally, the ratio of the first distance to the second distance changes, and when the line-of-sight direction of the subject changes vertically, the value of the third distance changes. By utilizing this property, the horizontal component of the line-of-sight direction of the subject and the vertical component of the line-of-sight direction of the subject can be stably determined.

[0012] Further, the processing unit may obtain candidate information regarding the line-of-sight direction based on the shape information of the right pupil or the left pupil in the face image, and determine the line-of-sight direction by selecting from the candidate information based on the first distance, the second distance, and the third distance. In this case, the true line-of-sight direction can be selected based on the line-of-sight direction determined using the first to third distances from the candidate information obtained by utilizing the property that the pupil shape in the face image changes according to the line-of-sight direction. As a result, the line-of-sight direction of the subject can be accurately determined.

[0013] Further, the processing unit may use, as the shape information, the ellipticity and the information regarding the direction of the major axis or the minor axis. By doing so, candidate information that accurately reflects the line-of-sight direction of the subject can be obtained. As a result, the line-of-sight direction of the subject can be determined more accurately.

[0014] Further, the processing unit may detect the head pose of the subject based on the three-dimensional position of the right pupil, the three-dimensional position of the left pupil, and the three-dimensional position of the nose, and determine the line-of-sight direction based on the head pose. In this case, even when the subject's head moves, the line-of-sight direction of the subject in the stationary coordinate system can be stably detected.

[0015] Further, the processing unit further detects the three-dimensional position of the forehead based on the face image, specifies a third line that perpendicularly intersects the first line after passing through the three-dimensional position of the forehead, and then specifies a fourth line that is parallel to the third line and passes through the intersection point, and determines the component of the line-of-sight direction in the vertical direction of the subject based on the relative position of the intersection point on the plane including the second line and the fourth line. By doing so, the vertical head pose of the subject can be specified with a reduced amount of calculation, and the component of the line-of-sight direction in the vertical direction based on the head pose can be stably detected.

[0016] The line-of-sight direction detection device according to the embodiment includes: [1] an optical system that acquires a face image by imaging the face of a subject, The system includes a processing unit that processes the facial image acquired by the optical system to determine the direction of the subject's gaze, The processing unit detects the three-dimensional positions of the right pupil, the left pupil, and the nose based on the facial image. The three-dimensional position of the intersection of a second straight line that passes through the three-dimensional position of the nose and intersects perpendicularly with a first straight line connecting the three-dimensional position of the right pupil and the three-dimensional position of the left pupil, and the first straight line, is determined. The direction of the subject's gaze is determined using the following: a first distance between the three-dimensional position of the intersection and the three-dimensional position of the right pupil, a second distance between the three-dimensional position of the intersection and the three-dimensional position of the left pupil, and a third distance between the three-dimensional position of the intersection and the three-dimensional position of the nose. It is a "gaze direction detection device".

[0017] The gaze direction detection device of the embodiment [2] "The processing unit determines the horizontal gaze direction component of the subject from the relationship between the first distance and the second distance, and determines the vertical gaze direction component of the subject from the third distance, The eye-line direction detection device described in [1] above may also be the same.

[0018] The gaze direction detection device of the embodiment [3] "The processing unit determines the horizontal gaze direction component of the subject based on the ratio of the first distance and the second distance, and determines the vertical gaze direction component of the subject based on the magnitude of the third distance, This may be the "line of sight detection device described in [2] above".

[0019] The gaze direction detection device of the embodiment states: [4] "The processing unit acquires candidate information regarding the gaze direction based on the shape information of the right pupil or the left pupil in the face image, and determines the gaze direction by selecting from the candidate information based on the first distance, the second distance, and the third distance." A gaze direction detection device as described in any of the above [1] to [3].

[0020] The gaze direction detection device of the embodiment [5] "The processing unit uses information regarding ellipticity and the direction of the major axis or minor axis as shape information, This may be the "line of sight detection device described in [4] above".

[0021] The gaze direction detection device of the embodiment [6] "The processing unit detects the head posture of the subject based on the three-dimensional position of the right pupil, the three-dimensional position of the left pupil, and the three-dimensional position of the nose, and determines the gaze direction based on the head posture, This may be any of the gaze direction detection devices described in [1] to [5] above.

[0022] The gaze direction detection device of the embodiment may be [7] "a gaze direction detection device according to any of [1] to [6] above, wherein the processing unit further detects the three-dimensional position of the forehead based on the face image, identifies a third straight line that passes through the three-dimensional position of the forehead and intersects the first straight line perpendicularly, identifies a fourth straight line that is parallel to the third straight line and passes through the intersection, and determines the vertical gaze direction component of the subject based on the relative position of the intersection on a plane including the second straight line and the fourth straight line."

[0023] The gaze direction detection device of the embodiment includes [8] an optical system that acquires a face image by capturing the face of a subject, The system includes a processing unit that processes the facial image acquired by the optical system to determine the direction of the subject's gaze, The processing unit detects the three-dimensional positions of the right pupil, the left pupil, and the reference portion set on the face based on the face image. The three-dimensional position of the intersection of a second straight line that passes through the three-dimensional position of the reference portion and intersects perpendicularly with a first straight line connecting the three-dimensional position of the right pupil and the three-dimensional position of the left pupil, and the first straight line, is identified. The direction of the subject's gaze is determined using the following: a first distance between the three-dimensional position of the intersection and the three-dimensional position of the right pupil, a second distance between the three-dimensional position of the intersection and the three-dimensional position of the left pupil, and a third distance between the three-dimensional position of the intersection and the three-dimensional position of the reference part. It is a "gaze direction detection device".

[0024] The gaze direction detection device of the embodiment [9] "The reference part is the nose or the forehead, This may be the "eye-line direction detection device described in [8] above". [Effects of the Invention]

[0025] According to this disclosure, the direction of the subject's gaze can be reliably determined. [Brief explanation of the drawing]

[0026] [Figure 1] This is a perspective view showing a gaze direction detection device according to an embodiment. [Figure 2] This is a plan view showing the lens portion of a camera. [Figure 3] This figure shows the hardware configuration of the image processing device according to the embodiment. [Figure 4] This is a block diagram showing the functional configuration of an image processing apparatus according to an embodiment. [Figure 5] Figure 4 is a conceptual diagram illustrating the function of acquiring the line-of-sight vector using the pupil shape method by the calculation unit 23. [Figure 6] Figure 4 is a conceptual diagram illustrating the calculation process in three-dimensional coordinates performed by the calculation unit 23. [Figure 7] Figure 4 is a conceptual diagram illustrating the calculation process in three-dimensional coordinates performed by the calculation unit 23. [Figure 8] This diagram shows the vertical change in subject A's line-of-sight vector PT, viewed from the horizontal direction. [Figure 9] This flowchart shows the procedure for detecting the direction of gaze using the gaze direction detection device 1. [Figure 10]This is a conceptual diagram illustrating the calculation process in three-dimensional coordinates performed by the calculation unit 23 in the modified example. [Modes for carrying out the invention]

[0027] Hereinafter, preferred embodiments of the gaze direction detection device and gaze direction detection method according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are denoted by the same reference numerals, and redundant descriptions are omitted.

[0028] First, the configuration of the gaze direction detection device 1 according to the embodiment of this disclosure will be explained using Figures 1 to 4. The gaze direction detection device 1 is a computer system that detects the gaze direction of subject A by imaging the face of subject A, including the left and right eyes and nose. Subject A is a person whose gaze direction is to be detected, and can also be called a test subject. The gaze direction of the detected subject means the direction of a vector from a starting point to an ending point, and preferably conceptually includes the spatial position of the vector. However, the gaze direction of the detected subject does not necessarily have to be represented by a precise direction using a vector, etc., and may be represented by a rough direction such as a bearing. The purpose of use of the gaze direction detection device 1 is not limited in any way, and for example, the gaze direction detection device 1 can be used for detecting distracted driving, confirming the driver's safety check operation of side mirrors and rearview mirrors, detecting driver drowsiness, investigating the degree of interest in products, data input to computers used in amusement devices, etc., and as a diagnostic device for diagnosing autism in infants and young children.

[0029] As schematically shown in Figure 1, the gaze direction detection device 1 comprises a pair of cameras 10 that function as a stereo camera, which is an optical system, and an image processing device 20. In the following, the pair of cameras 10 will be described as follows, if necessary: ​​the left camera 10 located to the left of subject A. L And, the right camera 10 located to the right of subject A. RA distinction is made between the two. In this embodiment, the gaze direction detection device 1 further includes a display device 30 which is the object that subject A is looking at. However, the purpose of using the gaze direction detection device 1 is not limited as described above, so the object in the line of sight of subject A is not limited to the display device 30, but could be, for example, the windshield of a car. Therefore, the display device 30 is not an essential element of the gaze direction detection device 1. Each camera 10 is connected to the image processing device 20 by wireless or wired connection, and various data or commands are sent and received between the camera 10 and the image processing device 20. Camera calibration is performed in advance for each camera 10.

[0030] Camera 10 is used to capture a facial image of subject A, including both eyes and nose. The pair of cameras 10 are arranged at a predetermined distance along the horizontal direction and are positioned lower than subject A's face to prevent reflections from appearing in the facial image when subject A is wearing glasses. The elevation angle of the cameras 10 relative to the horizontal direction is set to a range of, for example, 20 to 35 degrees, taking into consideration both reliable detection of the pupil and avoiding obstruction of subject A's field of view. Camera calibration is performed in advance for each camera 10. In this embodiment, the pair of cameras 10 are located at the bottom of the display device 30, but the pair of cameras 10 may also be located on the back side of the display device 30. In that case, the display device 30 is made of a transparent material so that the face of subject A can be captured by the cameras 10. The configuration in which a pair of cameras 10 are placed on the back of the display device 30 has the advantage that the line of sight direction toward a wide area of ​​the display device 30 can be detected with high accuracy using a conventional method that uses corneal reflection images.

[0031] In this embodiment, each camera 10 is an imaging device capable of capturing images at predetermined time intervals (for example, at 120 fps). The camera 10 captures the face of subject A in response to a command from the image processing device 20 and outputs the face image data to the image processing device 20.

[0032] Figure 2 schematically shows the lens portion of camera 10. As shown in this figure, in camera 10, the objective lens 11 is housed in a circular opening 12, and a light source 13 is mounted outside the opening 12. The light source 13 is a device for irradiating illumination light towards the eyes of subject A, and consists of a plurality of light-emitting elements 13a (first light source) and a plurality of light-emitting elements 13b (second light source). The light-emitting elements 13a are semiconductor light-emitting elements (LEDs) with a central wavelength of output light of 850 nm, and are arranged in a ring shape at equal intervals along the edge of the opening 12. The light-emitting elements 13b are semiconductor light-emitting elements with a central wavelength of output light of 940 nm, and are arranged vertically in a vertical direction to the left and right outside the light-emitting elements 13a. The distance from the optical axis of camera 10 to the light-emitting elements 13b is greater than the distance from the optical axis to the light-emitting elements 13a. Each light-emitting element 13a, 13b is provided to emit illumination light along the optical axis of camera 10. Note that the arrangement of the light source 13 is not limited to the configuration shown in Figure 2; other arrangements are also acceptable as long as the camera 10 can be considered as a pinhole model. The light-emitting elements 13a and 13b each emit illumination light at different timings and for different durations in response to commands from the image processing device 20. The intensity of the illumination light emitted by the light-emitting elements 13a and 13b is pre-set so that when illuminated on subject A, the background other than the pupil is the same brightness.

[0033] The light-emitting element (first light source) 13a constituting the light source 13 is a light source for illuminating the face of subject A with illumination light (first illumination light) to obtain a bright pupil image. A bright pupil image is an image in which subject A's pupil appears relatively brighter compared to a dark pupil image, which will be described later.

[0034] The light-emitting element (second light source) 13b constituting the light source 13 is a light source for illuminating the face of subject A with illumination light (second illumination light) to obtain a dark pupil image. A dark pupil image is an image in which subject A's pupil appears relatively darker compared to the aforementioned bright pupil image.

[0035] The image processing device 20 is a computer (processing unit) that controls the camera 10 and the light source 13, and determines the pupil position and gaze direction of subject A. The image processing device 20 may be constructed using a stationary or portable personal computer (PC), a workstation, or other types of computers. Alternatively, the image processing device 20 may be constructed using a combination of multiple computers of any type. When multiple computers are used, these computers are connected via a communication network such as the Internet or an intranet.

[0036] Figure 3 shows a typical hardware configuration of the image processing device 20. The image processing device 20 includes a CPU (processor) 101 that runs an operating system and application programs, a main memory unit 102 consisting of ROM and RAM, an auxiliary memory unit 103 consisting of a hard disk or flash memory, a communication control unit 104 consisting of a network card or wireless communication module, an input device 105 such as a keyboard or mouse, and an output device 106 such as a display or printer.

[0037] Each functional element of the image processing device 20, described later, is realized by loading predetermined software onto the CPU 101 or main memory unit 102, operating the communication control unit 104, input device 105, output device 106, etc., under the control of the CPU 101, and reading and writing data to the main memory unit 102 or auxiliary memory unit 103. The data and database necessary for processing are stored in the main memory unit 102 or auxiliary memory unit 103.

[0038] As shown in Figure 4, the image processing device 20 comprises a lighting control unit 21, an image acquisition unit 22, and a calculation unit 23 as functional components. The lighting control unit 21 controls the lighting timing and duration of the light source 13. The image acquisition unit 22 is a functional element that acquires image data (face image data) from the camera 10 by controlling the camera's shooting timing in synchronization with the lighting timing of the light source 13. The calculation unit 23 is a functional element that detects the three-dimensional positions of the pupil and nose of subject A based on the image data and determines the direction of subject A's gaze. A gaze is the line connecting the center of subject A's pupil and the point of fixation of subject A (the point the subject is looking at). The term "gaze" includes the meaning (concept) of a starting point, an ending point, and a direction. A "gaze vector" is a vector representation of the direction of the subject's gaze and is one form of representing the "gaze direction." The output destination of the gaze direction detection result from the image processing device 20 is not limited in any way. For example, the image processing device 20 may display the determination result on a monitor as an image, graphic, or text, store it in a storage device such as memory or a database, or transmit it to another computer system via a communication network.

[0039] The image acquisition unit 22 is a functional element that controls the timing of the camera 10's shooting. Specifically, it controls the camera 10 to repeatedly capture images at a predetermined frame rate (e.g., 120 fps) and a predetermined exposure time, and to alternately acquire bright pupil images and dark pupil images as face images.

[0040] The lighting control unit 21 is a functional element that controls the lighting timing of the light-emitting elements 13a and 13b and the amount of light emitted during the exposure period of the camera 10, in synchronization with the shooting timing of the camera 10. In this embodiment, the lighting control unit 21 controls the amount of light emitted by the light-emitting elements 13a and 13b by setting their respective lighting periods. Specifically, the lighting control unit 21 lights up the light-emitting elements 13a and 13b so as to alternately repeat the lighting of the light-emitting element 13a in synchronization with the timing of capturing a bright pupil image and the lighting of the light-emitting element 13b in synchronization with the timing of capturing a dark pupil image.

[0041] Due to the functions of the image acquisition unit 22 and the lighting control unit 21, a bright pupil image is acquired when the pupil is captured by the camera 10 at the timing when illumination light is emitted from the light-emitting element 13a to the eyeball of subject A, and a dark pupil image is acquired when the pupil is captured by the camera 10 at the timing when illumination light is emitted from the light-emitting element 13b. This is due to the following properties: In other words, when the illumination light to subject A's eyeball is incident from a position relatively far from the optical axis of the camera 10, the illumination light that enters through the pupil of the eyeball, is reflected inside the eyeball and passes through the pupil again does not easily reach the camera 10, so the pupil appears relatively dark. Conversely, when the illumination light to subject A's eyeball is incident from a position relatively close to the optical axis of the camera 10, the illumination light that enters through the pupil of the eyeball, is reflected inside the eyeball and passes through the pupil again easily reaches the camera 10, so the pupil appears relatively bright.

[0042] The following describes the details of the calculation unit 23's function for determining the gaze direction of subject A.

[0043] The calculation unit 23 uses two images, a bright pupil image and a dark pupil image, acquired alternately at predetermined time intervals (for example, time intervals corresponding to 120 fps), to calculate the difference by subtracting the brightness of each pixel in the dark pupil image from the brightness of each pixel in the bright pupil image. Then, the calculation unit 23 detects the position of the right pupil image of subject A on the image (hereinafter also referred to as the two-dimensional position) and the two-dimensional position of the left pupil image of subject A for each of the consecutively acquired difference images. That is, the calculation unit 23 binarizes the difference images based on a pupil threshold, removes isolated points, denoises using morphological processing, and labels them. Then, the calculation unit 23 detects the group of pixels that has the most pupil-like shape as the right pupil image and the left pupil image. At this time, the calculation unit 23 processes the right pupil image and the left pupil image as follows. In other words, even when the pupil is hidden by the eyelid or eyelashes, the boundary between the eyelid or eyelashes and the pupil is excluded as a false pupil contour, and only the true pupil contour is fitted to the ellipse. The position of the true pupil contour on the difference image is detected, and the center position of the pupil image is calculated from the ellipse equation obtained by the ellipse fitting.

[0044] Furthermore, the calculation unit 23 refers to the continuously acquired bright pupil images or dark pupil images to detect the two-dimensional position of the center of the nostrils between the left and right nostrils of subject A as the two-dimensional position of subject A's nose. At this time, the calculation unit 23 uses the detection method described in Japanese Patent No. 4431749 by the present inventor.

[0045] Furthermore, the calculation unit 23 continuously acquires the two-dimensional position of the center of the right pupil image detected based on the difference images acquired at the same or approximately the same time for each pair of cameras 10. Then, the calculation unit 23 uses the two-dimensional positions of the centers of the two right pupil images obtained corresponding to the pair of cameras 10 to calculate (detect) the three-dimensional position of the center of the right pupil of subject A. The three-dimensional position calculated at this time is the position in the world coordinate system, which is a stationary coordinate system, and is calculated by the stereo matching method, which treats the pair of cameras 10 as stereo cameras (the same applies hereafter). Similarly, the calculation unit 23 continuously calculates (detects) the three-dimensional position of the center of the left pupil and the three-dimensional position of the center of the nostrils of subject A, using the two-dimensional positions of the centers of the two left pupil images and the two-dimensional positions of the centers of the nostrils obtained corresponding to the pair of cameras 10.

[0046] In addition, the calculation unit 23 continuously obtains candidate information regarding the line-of-sight direction of the subject A as follows for each of the difference images continuously obtained from either one of the pair of cameras 10. Specifically, the calculation unit 23 uses the "pupil shape method" described in Japanese Patent No. 6963820 by the present inventor based on the shape information on the image as a result of elliptical fitting of the right pupil image in the difference image, and obtains candidate information representing two candidates for the line-of-sight vector of the right eye in the world coordinate system which is a stationary coordinate system. The calculation unit 23 uses, as the shape information, information regarding the ellipticity and the direction of the major axis or minor axis of the ellipse. At this time, the calculation unit 23 may also obtain candidate information of the line-of-sight vector of the left eye based on the shape information of the left pupil image in the difference image, or may obtain candidate information of the line-of-sight vector by averaging the line-of-sight vector of the right eye and the line-of-sight vector of the left eye. Further, when obtaining the line-of-sight vector from the shape information of the pupil image, the calculation unit 23 may perform calibration processing considering the deviation between the optical axis of the eyeball and the line-of-sight vector as described in Japanese Patent No. 6963820.

[0047] FIG. 5 is a conceptual diagram for explaining the function of obtaining the line-of-sight vector PT using the pupil shape method by the calculation unit 23. Here, assuming that the subject A is located at infinity from the right camera 10, the state of the right eyeball EB of the subject when the subject A looks straight at the right camera 10, when the subject A tilts the line of sight 60 degrees to the right in the horizontal direction from the front toward the right camera 10, and when the subject A tilts the line of sight 60 degrees to the left in the horizontal direction from the front toward the right camera 10 is shown as viewed from the vertical direction. In this case, the right pupil image P on the difference image obtained from the right camera 10 R becomes a perfect circle when the subject A looks straight at the right camera 10, and becomes an ellipse when the subject A looks obliquely. Further, when the subject A looks in such a way that the line-of-sight vectors PT are line-symmetric with respect to the line connecting the right camera 10 R and the center C R of the right eyeball (for example, when they are line-symmetric at ±60 degrees), the right pupil image P on the difference image R when the subject A looks straight at the right camera 10, when the subject A tilts the line of sight 60 degrees to the right in the horizontal direction from the front toward the right camera 10, and when the subject A tilts the line of sight 60 degrees to the left in the horizontal direction from the front toward the right camera 10 is shown as viewed from the vertical direction. In this case, the right pupil image P on the difference image obtained from the right camera 10 R is a perfect circle when the subject A looks straight at the right camera 10, and becomes an ellipse when the subject A looks obliquely. Also, when the subject A looks so as to have line-symmetric line-of-sight vectors PT with respect to the line connecting the right camera 10 L when the subject A looks straight at the right camera 10, when the subject A tilts the line of sight 60 degrees to the right in the horizontal direction from the front toward the right camera 10, and when the subject A tilts the line of sight 60 degrees to the left in the horizontal direction from the front toward the right camera 10 is shown as viewed from the vertical direction. In this case, the right pupil image P on the difference image obtained from the right camera 10 R is a perfect circle when the subject A looks straight at the right camera 10, and becomes an ellipse when the subject A looks obliquely. Further, when the subject A looks in such a way that the line-of-sight vectors PT are line-symmetric with respect to the line connecting the right camera 10 R and the center C EB of the right eyeball (for example, when they are line-symmetric at ±60 degrees), the right pupil image P on the difference imageL The ellipticity and the inclination of the major or minor axis of the ellipse are the same. Taking these properties into consideration, the calculation unit 23 uses the right camera 10 as candidate information regarding the gaze direction of subject A. R We obtain information representing two line-of-sight vectors PT that are symmetrical to each other from a given perspective.

[0048] The calculation unit 23 then determines the horizontal gaze direction component and the vertical gaze direction component based on the head posture of subject A, based on the three-dimensional positions of the center of the right pupil, the center of the left pupil, and the center of the nostrils of subject A, which were calculated. Based on the determination results, the calculation unit 23 determines the true gaze direction by selecting information related to the true gaze vector from the acquired candidate gaze vector information.

[0049] As shown in Figure 5, when subject A changes their gaze 60 degrees to the right without moving their head, subject A's right eyeball EB is at the center C of the right eyeball. EB Because it rotates around the right camera 10 R From this perspective, the center of the right pupil C P The subject moves a distance 'a' to the left compared to when looking straight ahead. Also, if subject A changes their gaze 60 degrees to the left without moving their head, the right camera 10 R From this perspective, the center of the right pupil C P The gaze moves a distance 'a' to the right compared to when the gaze is directed straight ahead. Utilizing this property, the calculation unit 23 determines the true direction of the line of sight.

[0050] In detail, as shown in Figures 6 and 7, the calculation unit 23 calculates the three-dimensional position C of the center of the nostril. N Passing through, the three-dimensional position C of the center of the right pupil. PR and the three-dimensional position C of the center of the left pupil PL The calculation unit 23 calculates (identifies) the three-dimensional position of the intersection point B of the line (second line) m, which intersects the line (first line) l perpendicularly with the line l, on a three-dimensional coordinate system. Then, the calculation unit 23 calculates the three-dimensional position of the intersection point B and the three-dimensional position C of the center of the right pupil. PR The first distance between and , the three-dimensional position of intersection point B and the three-dimensional position C of the center of the left pupilPL The second distance between and and a variable t representing the ratio (1-t) / t are calculated, and based on the value of this variable t, the horizontal component of the line of sight of subject A relative to the line l is determined. Furthermore, the calculation unit 23 calculates the three-dimensional position of intersection B and the three-dimensional position C of the center of the nostrils. N The distance (third distance) d is calculated, and based on the value of this distance d, the vertical component of the line of sight of subject A, relative to the straight line m, is determined.

[0051] Figure 8 illustrates how subject A's gaze vector PT changes vertically while keeping their head still. When subject A shifts their gaze upward, subject A's right eyeball EB moves to the center C of the right eyeball. EB Because it rotates around the center of the right pupil C P This is the three-dimensional position C of the center of the nostrils, compared to when looking straight ahead. N Move away from it. Also, if subject A looks downwards, the center of the right pupil C P This is the three-dimensional position C of the center of the nostrils, compared to when looking straight ahead. N It approaches this. Using this property, the calculation unit 23 determines the vertical component of the line of sight.

[0052] For example, with a marker displayed on the display device 30 at a position corresponding to the frontal direction of subject A, and subject A looking at the marker, the calculation unit 23 pre-calculates the initial value t0 of variable t and the initial value d0 of distance d using the bright pupil image and dark pupil image acquired from the camera 10. The calculation unit 23 then calculates variable t and distance d using the bright pupil image and dark pupil image of subject A acquired continuously, detects the change from the initial value t0 of variable t, and determines that if it increases, the gaze direction has a "left" component, and if it decreases, the gaze direction has a "right" component. In addition, the calculation unit 23 detects the change from the initial value d0 of distance d, and determines that if it increases, the gaze direction has an "up" component, and if it decreases, the gaze direction has a "down" component. At this time, the calculation unit 23 may determine the direction representing the gaze direction from the change from the initial value t0 of variable t, or from the change from the initial value d0 of distance d. For example, the calculation unit 23 may determine the line of sight direction as "30 degrees to the right," etc.

[0053] Furthermore, the calculation unit 23 determines the true gaze vector (gaze direction) of subject A by selecting from candidate gaze vector information corresponding to the acquisition time of the image to be judged, based on the horizontal and vertical components of the determined gaze direction. For the determination, candidate gaze vector information for the right eye, candidate gaze vector information for the left eye, or candidate gaze vector information for the averaged gaze vector is used, and the gaze vector that is closest to the determined gaze direction is selected as the true gaze direction from among the candidate information. At this time, the calculation unit 23 converts the horizontal and vertical components of the determined gaze direction from a Cartesian coordinate system based on lines l and m to the gaze direction in the world coordinate system, and determines the true gaze vector by comparing the converted gaze direction with the candidate information. Then, the calculation unit 23 uses the true gaze vector to perform subsequent processing such as detecting distracted driving, and outputs the processing result to the output device 106, etc.

[0054] Next, the procedure for detecting the direction of gaze using the gaze direction detection device 1, that is, the procedure for the gaze direction detection method according to the embodiment, will be described. Figure 9 is a flowchart showing the procedure for detecting the direction of gaze using the gaze direction detection device 1.

[0055] When the gaze direction detection process is started, first, face image data for a specific frame is acquired by a pair of cameras 10 (step S1). Next, the calculation unit 23 of the image processing device 20 uses the face image data to detect the three-dimensional positions of the center of the right pupil, the center of the left pupil, and the center of the nostrils (step S2). Subsequently, the calculation unit 23 uses the difference image obtained from the face image data to acquire candidate gaze vector information (step S3).

[0056] Furthermore, the calculation unit 23 uses the three three-dimensional positions detected in step S2 to determine the horizontal and vertical components of the gaze direction based on the head posture of subject A (step S5). Next, the calculation unit 23 determines the true gaze vector by selecting from candidate gaze vector information based on the determination result in step S5 (step S5). Then, the image acquisition unit 22 of the image processing device 20 determines whether or not to acquire the next frame of the face image data (step S6).

[0057] If it is determined in step S6 to acquire the next frame (step S6; Yes), the processes in steps S1 to S5 are repeated to determine the true gaze vector for the next frame consecutively. If it is determined in step S6 not to acquire the next frame (step S6; No), the calculation unit 23 uses the true gaze vectors of the multiple frames that have been determined consecutively up to that point to perform subsequent processing and output the processing result (step S7).

[0058] The operation and effects of the gaze direction detection device 1 described above will now be explained.

[0059] According to the gaze direction detection device 1 of this embodiment, the three-dimensional positions of the right pupil, left pupil, and nose are detected based on the facial image data of subject A acquired by a pair of cameras 10. Based on these, the three-dimensional position of the intersection point B of a straight line l connecting the left and right pupils and a straight line m perpendicular to line l from the nose is identified. The gaze direction of subject A is then determined using the distance between intersection point B and the right pupil, the distance between intersection point B and the left pupil, and the distance between intersection point B and the nose. In this way, the gaze direction of subject A can be determined based on the detected positions of the left and right pupils and the nose in the facial image data, so the gaze direction can be reliably determined regardless of the magnitude of the deviation in the gaze direction relative to the optical system including the pair of cameras 10 as seen from subject A.

[0060] The image processing device 20 of this embodiment determines the horizontal gaze direction component of subject A based on the ratio of the distance between intersection B and the right pupil to the distance between intersection B and the left pupil, and determines the vertical gaze direction component of subject A based on the magnitude of the distance between intersection B and the nose. When subject A's gaze direction changes horizontally, the ratio of the distance between intersection B and the right pupil to the distance between intersection B and the left pupil changes, and when subject A's gaze direction changes vertically, the distance between intersection B and the nose changes. By utilizing this property, the horizontal gaze direction component and the vertical gaze direction component of subject A can be reliably determined.

[0061] Conventionally, when detecting a subject's gaze direction using the pupil shape method, two gaze vectors PT that are symmetrical to each other from the camera's perspective are obtained. To determine the true gaze vector from these two gaze vectors PT, it is assumed that this can be determined by comparing the difference in ellipticity of the pupil images in two face images obtained by cameras positioned side by side. However, this method makes it difficult to determine the vertical gaze direction of subject A. Furthermore, with this method, if the angle between the two cameras as seen from the subject is small, the difference in ellipticity becomes small, reducing the accuracy of gaze direction determination. In addition, when using two cameras as an optical system, the two cameras are generally integrated by a housing or similar to eliminate the need for recalibration after stereo calibration of the cameras. In this case, if the angle between the two cameras is made small for miniaturization and to prevent the pupil image from being obscured by the nose, etc., and to ensure reliable capture of the pupil image, the accuracy of gaze direction determination decreases. Furthermore, it is conceivable to determine the true gaze vector that is close to the detected face direction from two symmetrical gaze vectors PT as seen from the camera, but if the premise that face direction and gaze direction generally do not differ significantly is broken, it will not be possible to determine them correctly. On the other hand, according to this embodiment, the horizontal gaze direction component and the vertical gaze direction component of subject A can be reliably determined.

[0062] Furthermore, the image processing device 20 detects the head posture of subject A and determines the direction of gaze based on the head posture. In this case, even if subject A's head moves, the direction of subject A's gaze in a stationary coordinate system can be reliably detected.

[0063] The gaze direction detection device 1 of this embodiment can be used, for example, in surgery such as endoscopic surgery. Specifically, it can simultaneously measure the gaze direction and face orientation of the patient A, who is the surgeon performing the surgery, and use the face orientation measurement results to control the movement of surgical instruments by changing the rotation speed of the drill or turning the drill on / off. At the same time, by measuring the gaze direction, safety measures can be implemented, such as enabling control only when patient A is looking at the display screen.

[0064] Furthermore, the gaze direction detection device 1 can also be used for purposes such as saving video data during surgery, including endoscopic surgery. The gaze direction detection device 1 can detect the gaze direction of the subject A (the surgeon) over a wide range of angles and control the saving of video data based on the detection results. For example, it can switch the video data to be saved according to the gaze direction of subject A, saving the video when the gaze direction is directed towards a display screen showing images captured by an endoscope camera, and switching to and saving the video on another display screen when the gaze direction is directed towards that screen. In addition, when the gaze direction is directed towards subject A's hands, it can switch to and save the video acquired by a camera showing the hands. With this usage configuration, video data for learning surgical procedures can be automatically saved in a short amount of time.

[0065] When using the gaze direction detection device 1 during surgery, it is assumed that the subject A's gaze direction will be directed in various directions within the operating room. Therefore, multiple optical systems, including cameras, are placed near multiple display devices. In such a configuration, it may not be possible to detect the subject A's corneal reflection regardless of which optical system is used. In this embodiment, however, the gaze direction can be reliably detected even if the corneal reflection cannot be detected.

[0066] The present invention is not limited to the embodiments described above. The configurations of the above embodiments can be modified in various ways.

[0067] In the gaze direction detection device 1 described above, a pair of cameras 10 are included as an optical system, but an optical system consisting of a single camera may also be used. In that case, as described in Japanese Patent No. 7030317 by the present inventor, the image processing device 20 can detect three three-dimensional positions: the center of the right pupil, the center of the left pupil, and the center of the nostrils, based on face image data acquired from a single camera. Then, similar to the embodiment described above, the image processing device 20 can acquire candidate gaze vector information using the pupil shape method and determine the true gaze vector from the candidate information using the results of detecting the three three-dimensional positions.

[0068] Furthermore, in addition to detecting the position of the center of subject A's nostrils, the image processing device 20 may also operate to detect the position of a marker attached to subject A's nose, such as the tip of the nose, as the position of the nose. Alternatively, the image processing device 20 may detect the position of a marker attached on top of a mask or the like that covers the nose, as long as it can detect a position corresponding to the nose. In this case as well, the image processing device 20 can correctly determine subject A's true line of sight vector. In any case, the image processing device 20 operates to reliably detect the position of the nose using the camera 10, even if subject A's head rotates significantly. Also, when subject A is a surgeon performing surgery, attaching the marker on top of the mask makes it less likely for the marker to move during surgery, so the image processing device 20 can stably detect the position of the nose.

[0069] Furthermore, the image processing device 20 determines the horizontal component of subject A's line of sight based on the ratio of the distance between intersection B and the right pupil to the distance between intersection B and the left pupil, but the horizontal component of the line of sight may also be determined directly based on these distance values. Subject A's eye movements include convergence and divergence. This means that when subject A is looking at something far away, the lines of sight of both eyes become nearly parallel, but when subject A is looking at something close up, the eyes converge and become crossed. When both eyes become crossed, the interpupillary distance shortens. For example, in the case of spinal endoscopic surgery, when subject A is looking at a display screen in front of them that is far away, both eyes are diverged and the interpupillary distance is relatively long, and when looking at something close and downward, both eyes converge and the interpupillary distance is relatively short. Even when it is necessary to consider that large convergence and divergence movements occur under such circumstances, the image processing device 20 can accurately estimate the horizontal component of the line of sight of both eyes of subject A.

[0070] Furthermore, as described in Japanese Patent No. 7030317, the present inventor's image processing device 20 may detect the three-dimensional position of the rotation center of the eyeball instead of the three-dimensional position of the pupil center, and use the detection result to detect the head posture of subject A. The pupil center moves slightly forward and backward relative to the skull in response to the rotation of the eyeball. In contrast, the rotation center of the eyeball does not move relative to the skull even when the eyeball rotates. Therefore, the head posture can be determined by detecting the three-dimensional position of the rotation center of the eyeball, and the direction of subject A's gaze can be determined based on that head posture.

[0071] Alternatively, the calculation unit 23 of the image processing device 20 may use facial image data acquired with two markers attached to the nose and forehead area of ​​subject A to detect the three-dimensional positions of the two markers, and determine the head posture of subject A based on these three-dimensional positions. In this case, the three-dimensional position of the forehead can be detected using markers attached to a surgical cap.

[0072] The process by which the calculation unit 23 identifies the head posture and determines the direction of gaze in this case will be explained below with reference to Figures 10(a) and 10(b). In addition to the same functions as in the embodiment described above, the calculation unit 23 uses the three-dimensional position of the forehead of subject A to identify the head posture and determine the direction of gaze. The calculation unit 23 uses the three-dimensional coordinate C of the forehead of subject A. FH The calculation unit 23 then detects the three-dimensional coordinate C of the forehead, as shown in Figure 10(a). FH The calculation unit 23 calculates (identifies) a line (third line) m1 that passes through and intersects line l perpendicularly at intersection point B1 in three-dimensional coordinates. Then, the calculation unit 23 calculates (identifies) a line (fourth line) m2 that is parallel to line m1 and passes through intersection point B in three-dimensional coordinates. Furthermore, as shown in Figure 10(b), the calculation unit 23 calculates (identifies) a plane PLA containing lines m2 and line m in three-dimensional coordinates, and determines the vertical component of the subject A's line of sight based on the change in the relative position of intersection point B on plane PLA. For example, the calculation unit 23 determines the three-dimensional position C of the forehead. FH and the three-dimensional position C of the nose N The straight line m3 connecting the two points is used as the vertical axis for calculations, and the vertical component of the line of sight is determined based on the relationship between this vertical axis and the intersection point B. This relationship can be derived, for example, by having subject A look at a point with known coordinates displayed on the display device 30 while rotating their head up and down.

[0073] According to the above modification, the vertical head posture of subject A can be identified with reduced computational load, and the vertical gaze direction component based on that head posture can be reliably detected. In the method using only the nose shown in Figure 8, the line m tends to move due to pupil movement caused by the vertical rotation of the eyeball. Therefore, the vertical direction of the head cannot be calculated accurately, and consequently, the gaze direction, which is calculated depending on the direction of the head, may also not be calculated accurately. In contrast, according to the above modification, since the positions of both the forehead and nose are used, even if the eyeball rotates vertically and the pupil moves, the line m3 does not move as a result, so the head direction and gaze direction can be calculated more accurately.

[0074] Furthermore, in the gaze direction detection device 1 described above, instead of the calculation unit 23 calculating the two-dimensional and three-dimensional positions of the nose as the two-dimensional and three-dimensional positions of the reference unit, the calculation unit 23 may calculate the two-dimensional and three-dimensional positions of the forehead of subject A based on a marker or the like set on the head of subject A's face, and use that as the two-dimensional and three-dimensional positions of the reference unit. In this modified example, the calculation unit 23 uses the three-dimensional position of the forehead instead of the three-dimensional position of the nose to determine the horizontal gaze direction component and the vertical gaze direction component of subject A. [Explanation of Symbols]

[0075] 1... Line of sight detection device, 10... Camera (optical system), 13... Light source (optical system), 20... Image processing device (processing unit), 23... Calculation unit, A... Subject, d... Distance, B... Intersection, C N ,C PL ,C PR ...Three-dimensional position, l...straight line (first straight line), m...straight line (second straight line).

Claims

1. An optical system that acquires facial images by capturing images of the subject's face, The system includes a processing unit that processes the facial image acquired by the optical system to determine the direction of the subject's gaze, The processing unit detects the three-dimensional positions of the right pupil, the left pupil, and the nose based on the facial image. The three-dimensional position of the intersection of a second straight line that passes through the three-dimensional position of the nose and intersects perpendicularly with a first straight line connecting the three-dimensional position of the right pupil and the three-dimensional position of the left pupil, and the first straight line, is determined. The direction of the subject's gaze is determined using a first distance between the three-dimensional position of the intersection and the three-dimensional position of the right pupil, a second distance between the three-dimensional position of the intersection and the three-dimensional position of the left pupil, and a third distance between the three-dimensional position of the intersection and the three-dimensional position of the nose. Eye-line direction detection device.

2. The processing unit determines the horizontal line of sight component of the subject from the relationship between the first distance and the second distance, and determines the vertical line of sight component of the subject from the third distance. The line of sight direction detection device according to claim 1.

3. The processing unit determines the horizontal line of sight component of the subject based on the ratio of the first distance and the second distance, and determines the vertical line of sight component of the subject based on the magnitude of the third distance. The line of sight direction detection device according to claim 2.

4. The processing unit acquires candidate information regarding the gaze direction based on the shape information of the right pupil or the left pupil in the face image, and determines the gaze direction by selecting from the candidate information based on the first distance, the second distance, and the third distance. A line-of-sight direction detection device according to any one of claims 1 to 3.

5. The processing unit uses information regarding ellipticity and the direction of the major or minor axis as shape information. The line of sight direction detection device according to claim 4.

6. The processing unit detects the head posture of the subject based on the three-dimensional position of the right pupil, the three-dimensional position of the left pupil, and the three-dimensional position of the nose, and determines the gaze direction based on the head posture. A line-of-sight direction detection device according to any one of claims 1 to 3.

7. The aforementioned processing unit, Based on the aforementioned facial image, the three-dimensional position of the forehead is further detected. A third straight line is identified that passes through the three-dimensional position of the forehead and intersects the first straight line perpendicularly, and then a fourth straight line is identified that is parallel to the third straight line and passes through the intersection point. The vertical component of the subject's line of sight is determined based on the relative position of the intersection point on the plane containing the second and fourth lines. The line of sight direction detection device according to claim 1.

8. An optical system that acquires facial images by capturing images of the subject's face, The system includes a processing unit that processes the facial image acquired by the optical system to determine the direction of the subject's gaze, The processing unit detects the three-dimensional positions of the right pupil, the left pupil, and the reference portion set on the face based on the face image. The three-dimensional position of the intersection point of a second straight line that passes through the three-dimensional position of the reference portion and intersects perpendicularly with a first straight line connecting the three-dimensional position of the right pupil and the three-dimensional position of the left pupil, and the first straight line, is identified. The direction of the subject's gaze is determined using a first distance between the three-dimensional position of the intersection and the three-dimensional position of the right pupil, a second distance between the three-dimensional position of the intersection and the three-dimensional position of the left pupil, and a third distance between the three-dimensional position of the intersection and the three-dimensional position of the reference part. Eye-line direction detection device.

9. The aforementioned reference area is the nose or the forehead. The line of sight direction detection device according to claim 8.

10. A method for detecting a gaze direction, comprising: an optical system that acquires a facial image by imaging the eyes of a subject; and a processing unit that processes the facial image acquired by the optical system to determine the gaze direction of the subject, Based on the aforementioned facial image, the three-dimensional positions of the right pupil, left pupil, and nose are detected. The three-dimensional position of the intersection of a second straight line that passes through the three-dimensional position of the nose and intersects perpendicularly with a first straight line connecting the three-dimensional position of the right pupil and the three-dimensional position of the left pupil, and the first straight line, is determined. The direction of the subject's gaze is determined using a first distance between the three-dimensional position of the intersection and the three-dimensional position of the right pupil, a second distance between the three-dimensional position of the intersection and the three-dimensional position of the left pupil, and a third distance between the three-dimensional position of the intersection and the three-dimensional position of the nose. A method for detecting the direction of gaze.

Images