Sight direction estimation device, sight direction estimation method and program product

Abstract

A sight direction estimation device (10, 10A) comprises: an imaging unit (16) for capturing an image of a head portion including an eyeball of a person (OP); an apparent sight direction derivation unit (102) for deriving an apparent sight direction connecting a center position (E) of the eyeball and an apparent pupil center position (D) on the surface of the eyeball corresponding to a pupil center position (B) based on the image captured by the imaging unit; and a sight direction estimation unit (108) for estimating an actual sight direction of the person based on a predetermined correspondence between the actual sight direction and the apparent sight direction and the apparent sight direction derived by the sight direction derivation unit. A sight direction estimation method and a sight direction estimation program are also provided.

Description

Technical Field

[0001] The present disclosure relates to a sight line direction estimation device, a sight line direction estimation method and a sight line direction estimation program. Background Art

[0002] There is known a technique for estimating a person's line of sight direction using a captured image obtained by capturing an image of a person's head. For example, there is proposed a technique for estimating an iris position and an eyeball position using a captured image obtained by capturing an image of a person's head, and estimating a line of sight direction using the estimated iris position and eyeball position (see, for example, JP 2008-102902A (reference 1)). In this technique, an eyeball center is estimated from a plurality of images captured in a state where the user's facial posture is changed while the user is looking at a camera, and a direction from the eyeball center toward the iris center is derived as a line of sight direction using the estimated eyeball center and the current iris center.

[0003] As another technology for estimating the direction of a person's line of sight, a technology for correcting the error of the line of sight generated by each user in the case where the line of sight is detected from the center of corneal curvature and the center of pupil obtained from the position of the reflection image on the corneal surface is also proposed (see, for example, JP2011-217764A (reference 2)). In this technology, a line of sight detection device that detects the line of sight from the center of corneal curvature toward the center of pupil, an imaging device that captures an image of the user's eyes, and a display device are provided, the line of sight toward the gaze point of a feature area displayed on the display device is estimated as the actual line of sight, and a correction value is determined for each user.

[0004] However, in the technology of JP2008-102902A (reference 1), the user needs to look at the camera, and a complex process needs to be operated. In addition, in the technology of JP2011-217764A (reference 2), a sight line detection device for detecting the sight line and an imaging device for taking an image of the user's eyes are required, and the device structure is complex. In addition, in the case of estimating the sight line direction using an image obtained by taking an image of the user's eyes, it is necessary to correct the position of the feature part related to the sight line of the front part of the eyeball of the user estimated based on the taken image. That is, the sight line direction of the user is the direction from the center of the eyeball toward the center of the front part of the eyeball such as the iris center and the pupil center, but the light refracted by the cornea of ​​the eyeball is imaged on the image taken by the imaging device, and the imaging device is set in a direction away from the sight line direction of the user. Therefore, it is also necessary to consider the influence caused by the refraction caused by the cornea of ​​the eyeball. Therefore, there is room for improvement in estimating the sight line direction of a person with high accuracy.

[0005] Therefore, there is a need for a gaze direction estimating device, a gaze direction estimating method, and a gaze direction estimating program that are capable of estimating a gaze direction with a simple configuration by taking into account the influence caused by refraction caused by the cornea of ​​the eyeball. Summary of the invention

[0006] According to one aspect of the present disclosure, a sight line direction estimating device includes: an imaging unit that captures an image of a head portion including an eyeball of a person; an eyeball center position calculating unit that calculates the center position of the eyeball using the image captured by the imaging unit; a pupil center position calculating unit that calculates an apparent pupil center position on the surface of the eyeball corresponding to the pupil center position of the eyeball using the image captured by the imaging unit; a sight line direction deriving unit that derives an apparent sight line direction in a direction connecting the eyeball center position and the apparent pupil center position; and a sight line direction estimating unit that estimates the sight line direction of the person based on the apparent sight line direction derived by the sight line direction deriving unit, using a predetermined correspondence between the sight line direction connecting the eyeball center position and the pupil center position and the apparent sight line direction.

[0007] According to the sight line direction estimation device of this aspect of the present disclosure, the eyeball center position calculation unit calculates the eyeball center position, and the pupil center position calculation unit calculates the apparent pupil center position on the eyeball surface corresponding to the pupil center position of the eyeball using the image captured by the imaging unit. The sight line direction derivation unit derives the apparent sight line direction in the direction connecting the eyeball center position and the apparent pupil center position. The sight line direction estimation unit estimates the sight line direction of the person corresponding to the apparent sight line direction using a predetermined correspondence relationship. The predetermined correspondence relationship represents the correspondence relationship between the sight line direction of the person connecting the eyeball center position and the pupil center position and the apparent sight line direction. In this way, since the actual sight line direction corresponding to the apparent sight line direction is estimated, the sight line direction can be estimated with a simple structure by suppressing the influence caused by refraction caused by the cornea of ​​the eyeball.

[0008] The eyeball center position calculation unit can calculate the eyeball center position based on the position of the feature points on the face around the eyeball on the captured image. In the captured image obtained by capturing the face including the eye, feature areas such as the outer corner of the eye and the inner corner of the eye that characteristically represent the eye are captured. It is assumed that the space between the outer corner of the eye and the inner corner of the eye corresponds to the size of the eyeball. Therefore, by fitting the feature points on the face around the eyeball such as the outer corner of the eye and the inner corner of the eye to a predetermined eyeball model indicating a standard eyeball, the center position of the eyeball model can be used as the eyeball center position.

[0009] The eyeball center position calculation unit includes an irradiation unit that emits light and can calculate the eyeball center position based on the position of the reflected light from the irradiation unit reflected by the surface of the eyeball. That is, the eyeball center position calculation unit can calculate the eyeball center position by the so-called corneal reflection method. In the corneal reflection method, the reflected light at the cornea is detected, and the eyeball center position is calculated based on the positional relationship between the detected corneal position and the pupil center position included in the captured image.

[0010] The eyeball center position calculation unit may store the eyeball center position calculated based on the position of the feature point on the face, and use the stored eyeball center position. It may be difficult to always perform the calculation of the eyeball center position by the corneal reflection method. Therefore, usually by storing the calculated eyeball center position and using the stored eyeball center position, the eyeball center position can be used without calculating by the corneal reflection method, so the calculation load of the device can be suppressed.

[0011] The sight line direction estimation unit can estimate the sight line direction of the person using a mapping diagram in which the sight line direction of the person connecting the center position of the eyeball and the center position of the pupil corresponds to the apparent sight line direction. In this way, the sight line direction of the person can be estimated in a short time using a mapping diagram in which the sight line of the person corresponds to the apparent sight line direction.

[0012] The gaze direction estimation method disclosed in the present invention includes causing a computer to: calculate the center position of a person's eyeball using a captured image in which the head part of the person's eyeball is captured; calculate the apparent pupil center position on the surface of the eyeball corresponding to the pupil center position of the eyeball using the captured image; derive the apparent gaze direction in the direction connecting the eyeball center position and the apparent pupil center position; and estimate the person's gaze direction based on the apparent gaze direction using a predetermined correspondence between the person's gaze direction connecting the eyeball center position and the pupil center position and the apparent gaze direction.

[0013] The gaze direction estimating program of the present disclosure causes a computer to function as a gaze direction estimating device.

[0014] In this way, according to the gaze direction estimation method and the gaze direction estimation program, it is also possible to estimate the gaze direction by suppressing the influence caused by the refraction caused by the cornea of ​​the eyeball.

[0015] According to the present disclosure as described above, it is possible to obtain an effect of estimating a sight line direction by taking into account the influence caused by refraction due to the cornea of ​​the eyeball with a simple configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The foregoing and additional features and characteristics of the present disclosure will become more apparent from the following detailed description considered with reference to the accompanying drawings, in which:

[0017] Figure 1 is a block diagram showing an example of a structure in which the gaze direction estimation device according to the first embodiment is realized by a computer;

[0018] Figure 2 is an image diagram showing an example of the configuration of an electronic device of the gaze direction estimation device according to the first embodiment;

[0019] Figure 3 is an image diagram showing an example of a coordinate system in the gaze direction estimating device according to the first embodiment;

[0020] Figure 4 is a block diagram showing an example of a structure in which a device body of a gaze direction estimating device according to the first embodiment is functionally classified;

[0021] Figure 5A and 5B is an image diagram showing an example of a schematic diagram for explaining a first derivation method of a sight line direction, wherein Figure 5A shows the line of sight in the model's eye, and Figure 5B shows a captured image obtained by capturing an image of an eye;

[0022] Fig. 6A and 6B is an image diagram showing an example of a schematic diagram for explaining a second derivation method of a sight line direction, wherein Fig. 6A shows the line of sight in the model's eye, and Figure 6B shows a captured image obtained by capturing an image of an eye;

[0023] Figure 7 is an image diagram showing an example of a detailed schematic diagram related to the line of sight of a model eyeball;

[0024] Figure 8 is an image diagram showing an example of a mapping of a correspondence relationship between a line of sight error and an angle relative to a direction toward a camera;

[0025] Fig. 9 is a flow chart showing an example of a processing flow of the gaze direction estimation device by a computer according to the first embodiment;

[0026] Fig.10 is a block diagram showing an example of a structure in which a device body of a gaze direction estimating device according to a second embodiment is functionally classified; and

[0027] Fig.11 : is a flowchart showing an example of a processing flow of the gaze direction estimation device by a computer according to the second embodiment. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments related to the disclosed technology are described with reference to the accompanying drawings.

[0029] (First embodiment)

[0030] This embodiment describes an example of estimating a person's line of sight direction using a captured image obtained by capturing an image of the person's head. In this embodiment, as an example of a person's line of sight direction, the line of sight direction of an occupant of a vehicle such as a car as a moving object is estimated by a line of sight direction estimation device.

[0031] Figure 1 An example of the configuration of a gaze direction estimation device 10 that realizes the operation as the gaze direction estimation device of the disclosed technique by a computer is shown.

[0032] like Figure 1 As shown, the computer operating as the sight line direction estimation device 10 includes a device body 12, and the device body 12 includes a central processing unit (CPU) 12A, a random access memory (RAM) 12B, and a read-only memory (ROM) 12C. The ROM 12C includes a sight line direction estimation program 12P for realizing various functions of estimating the sight line direction by suppressing the sight line error ρ. The device body 12 includes an input / output interface (hereinafter, referred to as I / O) 12D, and the CPU 12A, RAM 12B, ROM 12C, and I / O 12D are connected via a bus 12E so that commands and data can be exchanged respectively. An input unit 12F such as a keyboard and a mouse, a display unit 12G such as a display, and a communication unit 12H for communicating with an external device are connected to the I / O 12D. In addition, an illumination unit 14 such as a near-infrared light emitting diode (LED) that irradiates the head of the occupant, a camera 16 that takes an image of the head portion of the occupant, and a distance sensor 18 that measures the distance to the head portion of the occupant are connected to the I / O 12D. Although not shown, a nonvolatile memory capable of storing various data may be connected to the I / O 12D.

[0033] By reading out the sight line direction estimation program 12P from the ROM 12C and expanding the sight line direction estimation program 12P in the RAM 12B, and executing the sight line direction estimation program 12P expanded in the RAM 12B by the CPU 12A, the device body 12 operates as the sight line direction estimation device 10. The sight line direction estimation program 12P includes processing for realizing various functions of estimating the sight line direction by suppressing the sight line error ρ (details are described below).

[0034] In this embodiment, since the actual sight direction can be derived by the first derivation method (details are described below) of deriving the sight direction using information related to the characteristic region characteristically representing the eye, the lighting unit 14 is not an essential configuration and can be omitted.

[0035] Figure 2 An example of the configuration of an electronic device mounted on a vehicle as the gaze direction estimating device 10 is shown.

[0036] like Figure 2 As shown, the vehicle is equipped with a device body 12 of the sight line direction estimation device 10, a lighting unit 14 that illuminates the occupant, a camera 16 that takes an image of the head portion of the occupant OP, and a distance sensor 18. The configuration example of this embodiment shows a case where the lighting unit 14 and the camera 16 are installed above the column 5 holding the steering device 4, and the distance sensor 18 is installed below the column 5. Figure 2 , the occupant's line of sight direction is shown as line of sight 3 .

[0037] Figure 3 An example of a coordinate system in the gaze direction estimating device 10 is shown.

[0038] The coordinate system in the case of specifying a position differs depending on how to handle the object as the center. Examples include a coordinate system centered on a camera that captures an image of a person's face, a coordinate system centered on the captured image, and a coordinate system centered on the person's face. In the following description, a coordinate system centered on a camera is referred to as a camera coordinate system, and a coordinate system centered on a face is referred to as a face model coordinate system. Figure 3 The illustrated example shows an example of the relationship among the camera coordinate system, the face model coordinate system, and the image coordinate system used in the gaze direction estimation device 10 according to the present embodiment.

[0039] In the camera coordinate system, the right side viewed from the camera 16 is the X direction, the lower side is the Y direction, and the front side is the Z direction, and the origin is derived by calibration. The camera coordinate system is determined so that the directions of the x-axis and the y-axis coincide with those in the image coordinate system, and the origin of the image coordinate system is the upper left corner of the image.

[0040] The face model coordinate system is used to determine the position and posture of the face, and express the positions of the areas of the eyes and mouth on the face. For example, facial image processing generally uses a technique in which data is projected onto an image using data called a face model in which the three-dimensional positions of the characteristic areas of the face such as the eyes and mouth are described, and the position and posture of the face are estimated by combining the positions of the eyes and mouth. An example of a coordinate system set in a face model is a face model coordinate system, and the left direction viewed from the face is the Xm direction, the lower direction is the Ym direction, and the rear direction is the Zm direction.

[0041] The relationship between the camera coordinate system, the face model coordinate system and the image coordinate system is predetermined, and coordinate conversion can be performed between the camera coordinate system, the face model coordinate system and the image coordinate system. Figure 2 ) can detect the distance between the head portion of the occupant OP and the camera 16, so the mutual relationship among the camera coordinate system, the face model coordinate system, and the image coordinate system can be specified using camera properties such as the focal length of the camera 16 and the distance between the head portion and the camera 16. In the following description, unless distinguished, the case where the camera coordinate system is applied will be described.

[0042] Figure 4 An example of a block diagram configuration in which the device body 12 of the gaze direction estimation device 10 according to the present embodiment is classified in functional configuration is shown.

[0043] like Figure 4 As shown, the gaze direction estimation device 10 includes an imaging unit 101 such as a camera, an apparent gaze direction derivation unit 102 , a correction amount derivation unit 106 connected to a map 107 , a gaze direction derivation unit 108 , and an output unit 109 .

[0044] The imaging unit 101 is a functional unit that acquires a captured image and outputs the captured image to the apparent sight line direction deriving unit 102. In this embodiment, the camera 16 as an example of an imaging device is used as an example of the imaging unit 101. The camera 16 captures an image of a head portion including a part of an eyeball of the occupant OP of the vehicle and outputs the captured image.

[0045] The apparent sight line direction deriving unit 102 is a functional unit for deriving the apparent sight line direction, which derives the apparent sight line direction of a person using the captured image from the imaging unit 101. Here, the apparent sight line direction is derived from the three-dimensional eyeball center position and the three-dimensional pupil center position estimated from the two-dimensional captured image. That is, when specifying the sight line direction of a person, it is difficult to physically detect the eyeball center position and the pupil center position. Therefore, based on the two-dimensional captured image obtained by capturing at least the eyeball surface, the three-dimensional eyeball center position and the three-dimensional pupil center position are estimated, and the direction from the estimated eyeball center position to the pupil center position is derived as the apparent sight line direction.

[0046] The apparent sight line direction deriving unit 102 includes an eyeball center calculating unit 103, an apparent pupil center calculating unit 104, and an apparent sight line direction calculating unit 105, so as to derive the apparent sight line direction. The pupil of the eyeball included in the captured image is captured at the eyeball surface, that is, at the position where light passes through the corneal surface. The pupil center position observed on the corneal surface is described as the apparent pupil center position.

[0047] The eyeball center calculation unit 103 is a functional unit that calculates the eyeball center position based on the captured image, and uses the captured image from the imaging unit 101 to perform calculations to estimate the three-dimensional eyeball center position. The apparent pupil center calculation unit 104 is a functional unit that calculates the apparent pupil center position based on the captured image, and uses the captured image from the imaging unit 101 to perform calculations to estimate the three-dimensional apparent pupil center position. The apparent sight line direction calculation unit 105 is a functional unit that calculates the apparent sight line direction using the apparent eyeball center position and the apparent pupil center position. The apparent sight line direction calculation unit 105 calculates the apparent sight line direction using the eyeball center position calculated by the eyeball center calculation unit 103 and the apparent pupil center position calculated by the apparent pupil center calculation unit 104.

[0048] The correction amount deriving unit 106 is a functional unit that determines the correction amount from the apparent sight line direction to the actual sight line direction, and derives the correction amount using the correspondence relationship between the apparent sight line direction and the correction amount to the actual sight line direction stored in the map 107. In the map 107, the error of the actual sight line direction of a person with respect to the apparent sight line direction is predetermined as the correction amount, and a map of the apparent sight line direction corresponding to the correction amount is stored (details are described below).

[0049] The sight line direction deriving unit 108 is a functional unit for deriving an actual sight line direction, and performs processing for deriving the sight line direction of a person by correcting the apparent sight line direction calculated by the apparent sight line direction deriving unit 102 using the correction amount derived by the correction amount deriving unit 106. The output unit 109 outputs information indicating the sight line direction of the person calculated by the sight line direction deriving unit 108.

[0050] Here, the derivation of the apparent sight line direction includes a first derivation method for deriving the sight line direction using information related to a characteristic region characteristically representing the eye, and a second derivation method for deriving the sight line direction by a so-called corneal reflection method using information related to the cornea. In the first derivation method, the two-dimensional position (position on the image) of the characteristic region characteristically representing the eye is detected, and the eyeball 2 modeled with a three-dimensional model is fitted based on the characteristic region, thereby deriving the eyeball center position in three dimensions, and deriving the three-dimensional sight line direction. In the second derivation method, the three-dimensional sight line direction is derived using the position of the reflected light of the corneal surface of the light emitted toward the eyeball and the two-dimensional position (position on the image) of the pupil center position.

[0051] Figure 5A and 5B An example of a schematic diagram for explaining the first derivation method is shown. Figure 5A shows the line of sight in the model eye 2, Figure 5B An example of an image obtained by photographing eyes on a person's face is shown.

[0052] like Figure 5B As shown in FIG. 1 , an image obtained by photographing an eye includes a pupil 22 and a characteristic region 24 that characteristically represents the outer corner and the inner corner of the eye. In the first derivation method, the apparent pupil center position D observed on the corneal surface is calculated using the photographed image. Then, the position of each characteristic region 24 as a characteristic point is calculated, and a three-dimensional eyeball model with a predetermined standard is fitted to the characteristic points of the outer corner and the inner corner. Figure 5A As shown, the center position of the fitted three-dimensional eyeball model is estimated as the center position E of the eyeball 2, and the vector connecting the center position E of the eyeball 2 and the apparent pupil center position D is regarded as the sight line vector g eb , that is, the apparent line of sight direction.

[0053] FIG. 6 shows an example of a schematic diagram for explaining the second derivation method. Fig. 6A shows the line of sight in the model eye 2, and Figure 6B An example of an image obtained by photographing eyes on a person's face is shown.

[0054] like Fig. 6A As shown, the camera 16 captures the reflected light of the near-infrared light radiated from the lighting unit 14, such as an LED arranged near the camera 16 that captures the human eye. In the camera 16, the reflected image in the cornea is captured at the position p of the intersection of the cornea and the straight line connecting the position A of the center of curvature of the cornea and the lighting unit 14. That is, as Figure 6B As shown, the reflected image in the cornea is formed around the pupil 22 in the captured image ( Figure 6B In the second derivation method, based on the relationship between the cornea reflection image reflected on the cornea as the position p and the center position D of the pupil 22, the vector from the position A of the center of curvature of the cornea to the center position B of the pupil determined from the apparent pupil center position D is estimated as the sight line vector g by a known method. cr .

[0055] Since light is refracted at the cornea of ​​the eyeball, the refraction of light at the cornea affects the derivation of the apparent sight line direction. For example, the apparent sight line direction derived by the first derivation method described above and the apparent sight line direction derived by the second derivation method may have an angle difference (hereinafter referred to as sight line error). Describe the sight line error in the apparent sight line direction.

[0056] Figure 7 An example of a detailed schematic diagram related to the line of sight of the model eye 2 is shown.

[0057] exist Figure 7In FIG. 1 , the center position A of the curvature of the cornea, the center position B of the pupil, and the center position E of the eyeball are shown as the actual positions of the model eyeball 2. The position P of the reflected image by the illumination on the corneal surface and the apparent pupil center position D observed on the corneal surface are shown as the apparent positions. Position C is the position of the imaging device of the camera. Figure 7 , the positions are shown by the distance s between position A and position B, the distance u between position A and position E, and the distance r between position A and position p. In addition, the positions are shown by the angle θ formed between the vector from position A to position D and the vector from position A to position p, the angle (line of sight error) ρ formed between the vector from position E to position B and the vector from position E to position D, and the angle ω formed between the vector from position E to position D and the vector from position E to position C.

[0058] In the second derivation method, since the position A of the center of curvature of the cornea can be derived using the corneal reflection method, information about the position p of the corneal reflection image, the position A of the center of curvature of the cornea, and the apparent pupil center position D observed on the corneal surface can be obtained as information related to the cornea. The center position E of the eyeball 2 can also be derived using information related to the cornea. Therefore, considering the refraction of light at the cornea, calculation can be performed, and the sight line vector g from the center position E of the eyeball 2 to the center position B of the pupil can be derived. cr .

[0059] On the other hand, in the first derivation method, since the observed position is fitted to the model eyeball 2, the information related to the cornea is only the apparent pupil center position D, and the refraction of light at the cornea can be ignored. That is, in the first derivation method, the vector connecting the apparent pupil center position and the eyeball center position is calculated as the sight line vector. Therefore, the sight line vector g calculated by the second derivation method is cr It may be related to the sight vector g calculated by the first derivation method eb Inconsistent. Line of sight vector g cr and the sight vector g eb The angle ρ of the angle difference between the two, which is the line of sight error, is not constant but varies according to the position and orientation of the eyeball 2. Therefore, relative to the line of sight vector g calculated by the second derivation method, cr , the deterioration of the estimation accuracy of the line of sight direction derived by the first derivation method increases as the line of sight error ρ increases.

[0060] Here, it is assumed that the sight vector g cr Close to the actual sight line vector, when the actual sight line vector is toward the position C of the imaging device of the camera, the actual sight line vector is substantially the same as the sight line vector g ebOn the other hand, the line of sight error ρ increases as the angle of the position C of the imaging device from the actual line of sight vector toward the camera increases. This means that the line of sight error ρ increases as the line of sight direction deviates from the direction toward the camera, and the line of sight error ρ has a similar value to the line of sight vector g. eb and the correlation of the angle ω formed between the vector from the center position E of the eyeball 2 to the position C of the imaging device of the camera.

[0061] Therefore, if the correlation between the line of sight error ρ and the angle ω is known, the apparent line of sight vector g can be obtained from eb Export the actual sight vector.

[0062] Figure 8 An example of the correlation between the line of sight error ρ and the angle ω is shown.

[0063] Figure 8 It is derived from experiments on the correlation between the angle ω and the sight error ρ that occurs when the person's sight direction is actually changed. Figure 8 As shown, when the person's gaze direction is toward the camera, the gaze error ρ and angle ω are zero, because the actual gaze vector is basically the same as the gaze vector g eb In addition, the line of sight error ρ (absolute value) increases as the line of sight deviates from the direction towards the camera.

[0064] Therefore, the gaze direction estimation device 10 according to this embodiment estimates the gaze direction by suppressing the gaze error ρ. That is, in this embodiment, the gaze direction estimation device 10 estimates the gaze direction by suppressing the gaze error ρ. Figure 8 The correlation shown is used as the correction amount, that is, in the gaze direction estimation device 10, the error of the actual gaze direction of the person relative to the apparent gaze direction is predetermined as the correction amount, and the error is stored in the map 107 by making the apparent gaze direction and the correction amount correspond to each other. Therefore, the correction amount deriving unit 106 derives the correction amount stored in the map 107 as the error of the actual gaze direction of the person relative to the apparent gaze direction using the apparent gaze direction derived by the apparent gaze direction deriving unit 102. Then, the gaze direction deriving unit 108 can derive the actual gaze direction (gaze vector) using the correction amount derived by the correction amount deriving unit 106.

[0065] Next, the operation of the gaze direction estimation device 10 for estimating the gaze direction by suppressing the gaze error p is described. In this embodiment, the gaze direction estimation device 10 is operated by the device main body 12 of a computer.

[0066] Fig. 9An example of the processing flow of the gaze direction estimation program 12P in the gaze direction estimation device 10 implemented by a computer is shown. In the device body 12, the gaze direction estimation program 12P is read out from the ROM 12C and expanded in the RAM 12B, and the gaze direction estimation program 12P expanded in the RAM 12B is executed by the CPU 12A.

[0067] First, in step S100, a process of acquiring an image captured by the camera 16 is performed. The process of step S100 is to acquire an image captured by the camera 16. Figure 4 1 is an example of an operation of outputting a captured image by the imaging unit 101 shown.

[0068] In the next step S102, the three-dimensional center position E of the eyeball 2 in the camera coordinate system is estimated using the feature points of the eye such as the inner and outer corners of the eye on the captured image. The processing of step S102 is included in Figure 4 1 is an example of the operation of the eyeball center calculation unit 103 in the apparent gaze direction derivation unit 102 shown. In step S102, the center position of the eyeball 2 is derived using the above-mentioned first derivation method to estimate the three-dimensional center position of the eyeball 2. First, the captured image is processed, and the facial feature positions of the areas representing the eyes and mouth in the facial model coordinate system are specified to estimate the position and posture of the face. Next, the feature points of the eyes such as the inner and outer corners of the eyes are specified, and the eyeball 2 modeled by the three-dimensional model is fitted to the estimated position and posture of the face and the eye feature points so that the center position of the eyeball 2 is derived in three dimensions. The coordinates e of the center position E of the eyeball 2 are determined by converting the center position of the eyeball 2 of the derived facial model coordinate system into the center position of the eyeball 2 of the camera coordinate system = (e x ,e y ,e z ) t .

[0069] Next, in step S104, the captured image is processed to detect the apparent pupil center position D on the captured image. Here, the center coordinates of the two-dimensional position of the image coordinate system detected in step S104 on the captured image are D=(D x ,D y ).

[0070] In the next step S106, the three-dimensional apparent pupil center position D in the camera coordinate system is estimated based on the apparent pupil center position D in the image coordinate system. First, the Z coordinate of the apparent pupil center position D in the camera coordinate system is calculated using the distance measurement result detected by the distance sensor 18. z Here, when the focal length f of the camera 16 is expressed in pixel units, the three-dimensional coordinates d of the apparent pupil center position D in the camera coordinate system = (Dx ,D y ,D z ) can be expressed by the following formula (1). The three-dimensional apparent pupil center position D is estimated using formula (1).

[0071] d = ((D x , d z ) / f, (D y , d z ) / f, d z ) (1)

[0072] The processing of step S104 and step S106 is included in Figure 4 An example of the operation of the apparent pupil center calculation unit 104 in the apparent sight line direction derivation unit 102 is shown.

[0073] Next, in step S108, the apparent sight line vector is calculated using the center position E of the eyeball 2 estimated in step S102 and the apparent pupil center position D estimated in step S106. The apparent sight line vector is a vector connecting the center position E of the eyeball 2 and the apparent pupil center position D. When the vector is the apparent sight line vector g eb =(g eb_x ,g eb_y ,g eb_z ), the apparent sight vector g eb It can be expressed by the following formula (2).

[0074]

[0075] The processing of step S108 is included in Figure 4 An example of the operation of the apparent gaze direction calculation unit 105 in the apparent gaze direction derivation unit 102 is shown.

[0076] In the next step S110, the apparent sight line vector g is derived. eb The angle ω formed between the vector from the center position E of the eyeball 2 to the position C of the imaging device of the camera.

[0077] That is, the sight line vector connecting the center position E of the eyeball 2 estimated in step S102 and the apparent pupil center position D estimated in step S106 is the apparent sight line vector g eb =(g eb_x ,g eb_y ,g eb_z Since position C is the origin of the camera coordinate system, the vector connecting position E and position C is vector EC(=(-e)=(-e x ,-e y ,-e z )t ), and the angle ω of the inner product of the vectors can be expressed by the following formula (3).

[0078]

[0079] When formula (2) is substituted into formula (3), the result is expressed by the following formula (4), and the angle ω can be determined.

[0080]

[0081] In this case, since the sign (positive / negative) of the angle ω cannot be specified in formula (4), if -((Dydz) / f)ez+dze is the x component of the cross product geb×(-e) shown in the following formula (5): y is positive, the sign / negativity of the angle ω is set to positive, if -((Dydz) / f)ez+dze y If ω is negative, the angle ω is negative.

[0082]

[0083] Next, in step S112, the sight error ρ corresponding to the angle ω derived in step S110 is extracted, and the angle correction amount is derived with reference to the map 107. The processing of step S110 and step S112 is Figure 4 An example of the operation of the correction amount deriving unit 106 is shown.

[0084] Then, in the next step S114, the angle formed between the corrected sight line vector and the vector toward the camera is calculated. Next, in step S116, the actual sight line direction (sight line vector) in the camera coordinate system is derived using the apparent sight line vector calculated in step S108, the angle calculated in step S110, and the angle calculated in step S114. Then, in the next step S118, the actual sight line direction (sight line vector) in the camera coordinate system derived in step S116 is output, and the processing routine ends.

[0085] The processing of step S114 and step S116 is Figure 4 The example of the operation of the line of sight direction deriving unit 108 is shown in FIG. Figure 4 An example of the operation of the output unit 109 is shown.

[0086] An example of the process of deriving the actual line-of-sight direction (line-of-sight vector) in the camera coordinate system in the above-mentioned step S116 is described.

[0087] First, the actual angle estimated by the processing of step S110 and step S112 is ω+ρ. The actual sight line vector is derived using the actual angle (ω+ρ) estimated by the processing of step S110 and step S112 and the angle of the apparent sight line vector calculated in step S108. Here, the actual sight line vector is represented by g' eb =(g' eb_x ,g' eb_y ,g' eb_z ).g' eb Determined by the following calculation.

[0088] In this embodiment, a case where a formula is derived using the Rodriguez rotation formula is described as an example.

[0089] In the Rodriguez rotation formula, when the vector obtained by rotating the three-dimensional vector v by an angle θ around the three-dimensional unit vector k as the rotation axis is v rot When the vector v rot It is given by the following equation (6). Here, the angle θ is determined by the right-hand system.

[0090] v rot =vcosθ+(k×v)sinθ+k(k·v)(1-cosθ)…(6)

[0091] Since by making the vector g eb Including vector g eb and the plane of vector EC to obtain the actual line of sight vector g' eb , so in formula (6), v = g eb , v rot =g' eb , θ = ρ, and the rotation axis k is a vector including g eb and the normal vector of the plane of vector EC. k is determined by the cross product of the following equation (7) (k is a unit vector).

[0092]

[0093] By placing these v, v rot , θ and k are substituted into equation (6) and transformed to obtain the following equation (8).

[0094]

[0095] In this way, we can get from the vector g eb , angle ω and angle ρ determine the actual sight vector g' eb .

[0096] As described above, according to the gaze direction estimation device of the present embodiment, the apparent gaze direction derived without considering the light refraction at the cornea of ​​the eyeball is corrected in the actual gaze direction. That is, the gaze direction is derived so as to suppress the gaze error between the derived apparent gaze direction and the actual gaze direction. Therefore, even if the gaze direction is derived without considering the refraction of light at the cornea of ​​the eyeball, the gaze direction can be derived with high accuracy in the actual gaze direction.

[0097] Although an example in which the correspondence between the angle ω and the sight error ρ is stored in the map 107 and the actual sight vector is derived with reference to the map 107 is described in this embodiment, the disclosed technology does not limit the correspondence stored in the map 107 to the angle ω and the sight error ρ. For example, instead of the angle ω, a vector related to the angle ω may be stored as a parameter, and the sight error ρ may correspond to the stored parameter and be stored in the map 107. Examples of the vector related to the angle ω include the apparent sight vector g eb And a vector EC from the center position of the eyeball 2 to the camera 16.

[0098] (Second embodiment)

[0099] Next, the second embodiment will be described. With regard to the derivation of the sight line direction, the second embodiment applies the technology disclosed in the following sight line direction estimation device, which is capable of switching between deriving the sight line direction by the first derivation method and deriving the sight line direction by the second derivation method. Since the second embodiment has basically the same configuration as the first embodiment, the same components are denoted by the same reference numerals, and their detailed description is omitted.

[0100] In this embodiment, since the three-dimensional sight line direction is derived using the position of reflected light on the corneal surface of light emitted toward the eyeball (second derivation method), the illumination unit 14 is an essential component.

[0101] In the derivation of the sight line direction, the three-dimensional sight line direction can be derived with high accuracy using the position of the reflected light on the corneal surface. However, it is difficult to always detect the position of the reflected light on the corneal surface, that is, it may be difficult to always detect the position of the reflected light by the corneal reflection method. Therefore, in this embodiment, when it is difficult to derive the sight line direction by the second derivation method, the sight line direction is derived by the first derivation method.

[0102] Fig.10 An example of a block diagram configuration in which the device main body 12 of the gaze direction estimation device 10A according to this embodiment is functionally classified is shown.

[0103] like Fig.10 As shown, the sight line direction estimation device 10A also includes Figure 4The deriving unit 102A of the corneal reflection method in the sight line direction estimation device 10 shown in FIG. Figure 4 The apparent sight line direction deriving unit 102 shown in FIG. 1 is referred to as a first deriving unit 102, and the deriving unit 102A by the cornea reflection method is referred to as a second deriving unit 102A.

[0104] The second derivation unit 102A included in the gaze direction estimation device 10A of the present embodiment includes an eyeball center calculation unit 103A, a pupil center calculation unit 104A, and a gaze direction calculation unit 105A. The gaze direction derivation unit 102A is a functional unit that derives the gaze direction by the corneal reflection method, and uses the captured image from the imaging unit 101 to derive the gaze direction of a person in consideration of the refraction of light at the corneal surface.

[0105] The eyeball center calculation unit 103A is a functional unit that calculates the eyeball center position based on the captured image, and performs calculations to estimate the three-dimensional eyeball center position using the captured image from the imaging unit 101. The pupil center calculation unit 104A is a functional unit that calculates the pupil center position of the pupil based on the captured image, and performs calculations to estimate the three-dimensional apparent pupil center position using the captured image from the imaging unit 101. The sight line direction calculation unit 105A is a functional unit that calculates the sight line direction using the eyeball center position and the apparent pupil center position. The apparent sight line direction calculation unit 105A calculates the sight line direction of a person in consideration of the refraction of light at the corneal surface using the eyeball center position calculated by the eyeball center calculation unit 103A and the pupil center position calculated by the pupil center calculation unit 104A.

[0106] Since the person's line of sight direction taken into account and derived by the second derivation unit 102A is basically consistent with the actual line of sight direction, the person's line of sight is output to the output unit 109, and information representing the person's line of sight direction is output from the output unit 109.

[0107] In this embodiment, each of the eyeball center calculation unit 103 of the first derivation unit 102 and the eyeball center calculation unit 103A of the second derivation unit 102A includes a memory (in Fig.10 Indicated by the symbol M), it stores information indicating the center position of the eyeball 2.

[0108] Next, an operation of estimating a gaze direction by suppressing a gaze error p in the gaze direction estimation device 10A according to this embodiment will be described. In this embodiment, the gaze direction estimation device 10A is operated by the device body 12 of a computer.

[0109] Fig.11An example of the flow of processing of the gaze direction estimation program 12P in the gaze direction estimation device 10A realized by a computer is shown.

[0110] In the device body 12 , the gaze direction estimation program 12P is read out from the ROM 12C and expanded in the RAM 12B, and the gaze direction estimation program 12P expanded in the RAM 12B is executed by the CPU 12A. Fig.11 The processing flow shown in the figure executes the processing of steps S101, S103, S105, S107 and S109 without executing Fig. 9 The processing of step S102 shown in FIG. Execute the processing of step S108A instead of executing Fig. 9 The process of step S108 is shown.

[0111] In this embodiment, when an image captured by the camera 16 is acquired in step S100, it is determined whether corneal reflection occurs by determining in step S101 whether a reflected image of light emitted from the illumination unit 14 is included in the captured image. The process of step S101 is an example of a process for determining whether a derivation process of the line of sight direction can be performed by the second derivation unit.

[0112] That is, when an affirmative determination is made in step S101, the process proceeds to step S103 so that the apparent sight line direction is derived by the first deriving unit 102. On the other hand, when a negative determination is made in step S101, the process proceeds to step S105 so that the sight line direction is derived by the second deriving unit 102A.

[0113] In step S105, the three-dimensional center position E of the eyeball 2 in the camera coordinate system is estimated using the second derivation method by the corneal reflection method. The processing of step S103 is included in Fig.10 An example of the operation of the eyeball center calculation unit 103A in the gaze direction derivation unit 102A is shown.

[0114] like Figure 7 As shown, when the coordinate of the center position A of the curvature of the cornea in the camera center coordinate system is a=(a x , a y , a z ) t , the sight vector determined by the second derivation method is g cr =(g cr x ,g cr_y ,g cr_z ) t , and the distance between the center position A of the curvature of the cornea and the center position E of the eyeball 2 is u, the coordinate e of the center position E of the eyeball 2 in the camera center coordinate system is (e x ,ey ,e z ) t It can be expressed by the following formula (9). Vector a and sight vector g cr It can be determined by corneal reflection method.

[0115]

[0116] In step S105, the coordinates e of the center position E of the eyeball 2 in the derived camera center coordinate system are stored in the memory. In step S105, the coordinates e of the center position E of the eyeball 2 are stored together in the memory of each of the eyeball center calculation unit 103 of the first derivation unit 102 and the eyeball center calculation unit 103A of the second derivation unit 102A.

[0117] In the next step S107, the three-dimensional pupil center position B in the camera coordinate system is estimated by a known method through the second derivation method of the corneal reflection method. In the next step S109, the sight line vector g is derived using the center position E of the eyeball 2 estimated in step S105 and the center position B of the pupil estimated in step S107. cr That is, the sight vector g cr is a vector connecting the center position E of the eyeball 2 and the center position B of the pupil. Then, the process proceeds to step S118, and the sight line vector g is output. cr .

[0118] The processing of step S105 is Fig.10 The process shown in step S107 is an example of the operation of the pupil center calculation unit 104A, and the process of step S109 is an example of the operation of the sight line direction calculation unit 105A.

[0119] On the other hand, when an affirmative determination is made in step S101, the process proceeds to step S103 so that the apparent sight line direction is derived by the first deriving unit 102. In step S103, the coordinates e of the center position E of the eyeball 2 in the camera center coordinate system stored in the memory are acquired. x ,e y ,e z ) t When the coordinate e of the center position E of the eyeball 2 is not stored in step S105, it can be Fig. 9 The center position E of the eyeball 2 is derived in the same manner as in step S102 shown in FIG.

[0120] Next, as described above, in step S104, the apparent pupil center position D (two-dimensional position) is detected, and in the next step S106, the three-dimensional apparent pupil center position D in the camera coordinate system is estimated.

[0121] Next, in step S108A, the apparent sight line vector is calculated. When the coordinates e of the center position E of the eyeball 2 are not stored in step S105, the apparent sight line vector g is calculated using the center position E of the eyeball 2 estimated in the same manner as in step S102 and the apparent pupil center position D estimated in step S106. eb (See formula (2) above).

[0122] On the other hand, when the coordinate e of the center position E of the eyeball 2 is stored in step S105, in step S108A, the apparent sight line vector g is calculated using the following formula (10) instead of the above formula (2): eb .

[0123]

[0124] Then, as described above, in step S110, the angle ω is derived, and in step S112, the sight error ρ corresponding to the angle ω derived in step S110 is extracted, and the angle correction amount is derived with reference to the map 107. Next, in step S114, the angle formed between the corrected sight vector and the vector toward the camera is calculated, and in step S116, the actual sight direction (sight vector) in the camera coordinate system is derived. Then, in step S118, the actual sight direction (sight vector) in the camera coordinate system is output, and the processing routine is terminated.

[0125] As described above, according to the gaze direction estimation device of the present embodiment, even when the derivation of the gaze direction taking into account the refraction of light at the cornea is switched to the derivation of the gaze direction without taking into account the refraction of light at the cornea, the gaze direction is derived so as to suppress the gaze error. Therefore, even when the gaze direction is derived without taking into account the refraction of light at the cornea of ​​the eyeball, the gaze direction can be derived with high accuracy in the actual gaze direction.

[0126] In addition, since the center position of the eyeball 2 is stored and used when deriving the line of sight direction taking into account the refraction of light at the cornea, the actual line of sight direction can be derived with high accuracy compared to the case where the center position of the eyeball is derived using only the feature points of the eye on the captured image.

[0127] The device body 12 included in the gaze direction estimation device according to each of the above-described embodiments may configure each constituent element by hardware such as an electronic circuit having the above-described functions, and may configure at least a part of the constituent elements so as to realize the functions by a computer.

[0128] In addition, in each of the above-mentioned embodiments, in order to simplify the description, the case where the sight line direction is derived for one eyeball 2 is described, but the sight line direction may be derived for each of a plurality of eyeballs 2. In this case, preferably, one direction observed by a pair of left and right eyeballs 2 is specified. At least one eyeball 2 in the pair of left and right eyeballs 2 may be an estimation target in the sight line direction. In this case, in the pair of left and right eyeballs 2, the eyeball 2 whose sight line direction can be derived by the second derivation method may be selected as the target, or by comparing images obtained by photographing the pair of left and right eyeballs 2, the eyeball 2 on the high-definition photographed image may be selected as the target.

[0129] In addition, in each of the above-mentioned embodiments, in order to simplify the description, an example is described in which the correspondence between the angle ω and the line of sight error ρ is pre-stored in the mapping map 107, but the disclosed technology is not limited to pre-storing the correspondence in the mapping map 107. For example, the correspondence between the angle ω and the line of sight error ρ can be learned. In this case, a visual target capable of specifying the angle ω can be provided in advance, the correspondence between the angle ω and the line of sight error ρ can be derived by observing the visual target, and the derived correspondence can be pre-stored in the mapping map 107.

[0130] In addition, in each of the above-mentioned embodiments, an example of a sight line direction estimation device including the distance sensor 18 that measures the distance to the head portion of the occupant has been described, but the sight line direction estimation device is not limited to including the distance sensor 18. For example, when the distance between the head portion of the occupant and the camera 16 is known and the change in the distance is within a predetermined distance range that is predicted to be less affected by the estimation of the sight line direction, the predetermined distance between the head portion of the occupant and the camera 16 may be stored in the sight line direction estimation device, and the stored distance may be read out and used. In this case, it is preferable that a plurality of different distances are stored step by step and selectively, and a distance selected from the plurality of stored distances is used.

[0131] Although the present disclosure is described using embodiments, the technical scope of the present disclosure is not limited to the scope described in the above embodiments. Various changes or improvements can be applied to the above embodiments without departing from the spirit of the present disclosure, and the changed or improved embodiments are also included in the technical scope of the present disclosure.

[0132] The principles of the present invention, preferred embodiments and modes of operation have been described in the foregoing description. However, the present invention, which is intended to be protected, should not be construed as being limited to the specific embodiments disclosed. In addition, the embodiments described herein should be considered illustrative rather than restrictive. Changes and modifications may be made by others and equivalents used without departing from the spirit of the present invention. Therefore, it is expressly intended to cover all such changes, modifications and equivalents that fall within the spirit and scope of the present invention as defined in the claims.

Claims

1. A sight direction estimation device, characterized in that: include: An imaging unit, wherein the imaging unit photographs a head portion including an eyeball of a person to obtain a photographed image; an apparent sight line direction deriving unit configured to derive, based on the captured image, an apparent sight line direction connecting the center position of the eyeball and an apparent pupil center position on the surface of the eyeball corresponding to the pupil center position; and a gaze direction estimating unit, wherein the gaze direction estimating unit estimates the actual gaze direction of the person based on a predetermined correspondence between the actual gaze direction and the apparent gaze direction and the apparent gaze direction derived by the apparent gaze direction deriving unit, The predetermined correspondence means that the angular difference between the actual line of sight direction and the apparent line of sight direction increases as the actual line of sight direction deviates from the direction toward the imaging unit, and the angular difference has a correlation with the angle formed between the apparent line of sight direction and the vector from the center position of the eyeball to the position of the imaging unit.

2. The sight line direction estimation device according to claim 1, characterized in that: in, The center position of the eyeball is derived based on the positions of feature points on the face around the eyeball on the captured image.

3. The sight line direction estimation device according to claim 1, characterized in that: in The center position of the eyeball is derived based on the position of reflected light emitted from a light irradiation unit and reflected by a surface of the eyeball.

4. The sight line direction estimation device according to claim 3, characterized in that: in, Based on the position of the reflected light, the center position of the eyeball is derived using the center position of the curvature of the cornea of ​​the eyeball.

5. The sight line direction estimation device according to any one of claims 1 to 4, characterized in that: in, The apparent gaze direction deriving unit derives the apparent gaze direction using the eyeball center position derived in advance and stored in the memory unit as the eyeball center position.

6. The sight line direction estimation device according to claim 1, characterized in that: in, The apparent sight direction deriving unit comprises: an eyeball center position deriving unit that derives the center position of the eyeball based on the captured image; and An apparent pupil center position deriving unit that derives the apparent pupil center position on the eyeball surface corresponding to the pupil center position of the eyeball based on the captured image.

7. A method for estimating a line of sight direction, characterized in that: This includes enabling the computer to: deriving, based on a captured image obtained by capturing an image of a head portion including an eyeball of a person by an imaging unit, an apparent sight line direction connecting an eyeball center position and an apparent pupil center position corresponding to the pupil center position on a surface of the eyeball; and estimating the actual sight line direction of the person based on a predetermined correspondence between the actual sight line direction of the person and the apparent sight line direction and the derived apparent sight line direction, The predetermined correspondence means that the angular difference between the actual line of sight direction and the apparent line of sight direction increases as the actual line of sight direction deviates from the direction toward the imaging unit, and the angular difference has a correlation with the angle formed between the apparent line of sight direction and the vector from the center position of the eyeball to the position of the imaging unit.

8. A program product comprising a gaze direction estimation program, characterized in that: The gaze direction estimating program causes a computer to function as the gaze direction estimating device according to any one of claims 1 to 6.

Images