Gaze Estimation Device, Computer Program for Gaze Estimation, and Gaze Estimation Method

Abstract

To provide a line-of-sight estimation device capable of accurately estimating a line-of-sight direction of a driver on the basis of an image in which a face of a driver is photographed even when the face of the driver is directed to a left or right direction to a front of the vehicle.SOLUTION: A line-of-sight estimation device detects a face feature point in a face image, acquires reliability of the detected face feature point, estimates a first line-of-sight direction of a driver by using the face feature point, and acquires reliability of the first line-of-sight direction on the basis of the reliability of the face feature point. When the reliability of the first line-of-sight direction is not higher than a prescribed threshold, whether the driver is in a gazing state is determined on the basis of the first line-of-sight direction and other plurality of line-of-sight directions in a prescribed period. When the driver is in the gazing state, a gazed object estimated to be gazed by the driver is detected from among objects photographed in a front image obtained by photographing an environment in front of the driver on the basis of the first line-of-sight direction . When the reliability of the face feature point is not higher than the prescribed threshold, and the driver is in the gazing state, the first line-of-sight direction is corrected so as to be directed to a gazed object.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a line-of-sight estimation device, a computer program for line-of-sight estimation, and a line-of-sight estimation method.

Background Art

[0002] Conventionally, the line-of-sight direction of a driver driving a vehicle has been estimated. Based on the line-of-sight direction of the driver, the state of the driver has been monitored.

[0003] The line-of-sight direction of the driver is estimated based on face feature points such as the outer and inner corners of the eyes and the pupil center detected from an image of the driver's face. The driver's face is photographed, for example, using a monitoring camera arranged on the steering column. When the driver's face is facing forward in front of the vehicle, the driver's face is photographed from the front using the monitoring camera, so that the face feature points can be accurately detected based on the photographed image.

[0004] For example, a line-of-sight direction estimation device has been proposed that estimates the line-of-sight direction of a driver based on a face image of the driver while driving a vehicle, determines whether the driver is looking in the forward direction, estimates a reference direction based on the estimated line-of-sight direction during the period when it is determined that the driver is looking in the forward direction, and corrects the line-of-sight direction based on this reference direction (see Patent Document 1). According to this line-of-sight direction estimation device, the line-of-sight direction is corrected based on the reference direction estimated when it is determined that the vehicle is going straight.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] During vehicle operation, the driver may turn their face left and right to check the situation around the vehicle. At this time, the monitoring camera captures the driver's face in a direction tilted from the front.

[0007] When the driver's face is turned left or right with respect to the vehicle's front, the monitoring camera captures the driver's face obliquely. As a result, it may not be possible to accurately detect facial feature points based on the captured image. If the positions of the facial feature points are incorrect, there is a risk that the driver's line-of-sight direction cannot be accurately estimated either.

[0008] Therefore, an object of the present disclosure is to provide a line-of-sight estimation device that can accurately estimate the driver's line-of-sight direction based on an image of the driver's face even when the driver's face is turned left or right with respect to the vehicle's front.

Means for Solving the Problem

[0009] According to one embodiment, a line-of-sight estimation device is provided. This line-of-sight estimation device includes a feature point detection unit that detects facial feature points in a face image including the driver's face and obtains the reliability of the detected facial feature points, a first line-of-sight estimation unit that estimates the driver's first line-of-sight direction using the facial feature points and obtains the reliability of the first line-of-sight direction based on the reliability of the facial feature points, a determination unit that determines whether the driver is in a fixation state based on the first line-of-sight direction and a plurality of other line-of-sight directions of the driver within a predetermined period before the time when the first line-of-sight direction was estimated when the reliability of the first line-of-sight direction is less than or equal to a predetermined threshold, an object detection unit that detects a fixation object, which is estimated to be the object that the driver is fixating on, among the objects reflected in a front image representing the environment in front of the driver based on the first line-of-sight direction when the driver is in a fixation state, and a correction unit that corrects the first line-of-sight direction to face the fixation object when the reliability of the facial feature points is less than or equal to a predetermined threshold and the driver is in a fixation state.

[0010] In addition, in this line-of-sight estimation device, further, a face orientation direction estimation unit that estimates the face orientation direction in which the driver's face is facing using the face feature points, and a second line-of-sight estimation unit that estimates the second line-of-sight direction of the driver based on the position of the face feature points and the position of the gazed object in the front image are provided. When the reliability of the first line-of-sight direction is equal to or less than a predetermined threshold value and the driver is in the gazing state, the correction unit obtains a correction amount based on the product of the angle formed by the face orientation direction and the second line-of-sight direction and the reciprocal of the reliability of the first line-of-sight direction, and preferably corrects the first line-of-sight direction by moving the first line-of-sight direction by an angle represented by the correction amount in the direction from the second line-of-sight direction toward the face orientation direction.

[0011] In addition, in this line-of-sight estimation device, it is preferable that the second line-of-sight estimation unit estimates the position of the driver's glabella based on the positions of the face feature points representing the driver's left eye and the face feature points representing the driver's right eye, and estimates the direction from the position of the glabella to the gazed object as the second line-of-sight direction.

[0012] Furthermore, in this line-of-sight estimation device, it is preferable that the determination unit determines that the driver is in the gazing state when the variance of the first line-of-sight direction and a plurality of other line-of-sight directions is within a predetermined reference value.

[0013] According to another embodiment, a computer program for gaze estimation is provided. This computer program for gaze estimation detects facial feature points in a face image including the driver's face and determines the reliability of the detected facial feature points, estimates the driver's first gaze direction using the facial feature points, and determines the reliability of the first gaze direction based on the reliability of the facial feature points. When the reliability of the first gaze direction is equal to or less than a predetermined threshold, it determines whether the driver is in a gazing state based on the first gaze direction and a plurality of other gaze directions of the driver within a predetermined period before the time when the first gaze direction was estimated. When the driver is in a gazing state, it detects a gazing object estimated to be gazed at by the driver among the objects shown in the front image representing the environment in front of the driver based on the first gaze direction, and when the reliability of the facial feature points is equal to or less than a predetermined threshold and the driver is in a gazing state, it corrects the first gaze direction so as to face the gazing object, and is characterized in that the processor is caused to execute this.

[0014] According to still another embodiment, a gaze estimation method is provided. This gaze estimation method detects facial feature points in a face image including the driver's face and determines the reliability of the detected facial feature points, estimates the driver's first gaze direction using the facial feature points, and determines the reliability of the first gaze direction based on the reliability of the facial feature points. When the reliability of the first gaze direction is equal to or less than a predetermined threshold, it determines whether the driver is in a gazing state based on the first gaze direction and a plurality of other gaze directions of the driver within a predetermined period before the time when the first gaze direction was estimated. When the driver is in a gazing state, it detects a gazing object estimated to be gazed at by the driver among the objects shown in the front image representing the environment in front of the driver based on the first gaze direction, and when the reliability of the facial feature points is equal to or less than a predetermined threshold and the driver is in a gazing state, it corrects the first gaze direction so as to face the gazing object, and is characterized in that a gaze estimation device executes this.

Advantages of the Invention

[0015] The gaze estimation device according to the present disclosure can accurately estimate the gaze direction of the driver based on the captured image of the driver's face even when the driver's face is turned to the left or right with respect to the front of the vehicle.

Brief Description of the Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0017] FIGS. 1(A) to 1(D) are views for explaining the outline of the operation of the monitoring system including the gaze estimation device 11 of the present embodiment. FIG. 1(A) is a view showing the vehicle, FIG. 1(B) is a view showing that a face image including the driver's face is captured by the monitoring camera, FIG. 1(C) is a view showing the face image with the driver facing forward, and FIG. 1(D) is a view showing the face image with the driver facing left.

[0018] As shown in FIG. 1(A), the vehicle 10 includes a monitoring system 1. The monitoring system 1 includes a line-of-sight estimation device 11 that estimates a line-of-sight direction v1, and a monitoring device 12 that monitors a state related to the driving of the driver 30 based on the first line-of-sight direction v1 estimated by the line-of-sight estimation device 11, etc.

[0019] As shown in FIG. 1(B), the line-of-sight estimation device 11 uses a monitoring camera 2 disposed on the steering column 31 in the vehicle interior to capture a face image including the face of the driver 30 driving the vehicle 10, and estimates the first line-of-sight direction v1 based on the face image.

[0020] When the vehicle 10 is going straight, it is considered that the face of the driver 30 faces forward of the vehicle front. As shown in FIG. 1(C), when the face of the driver 30 faces forward of the vehicle front, based on the captured face image 40, face feature points such as the outer corners f1, f6 of the left and right eyes, the inner corners f3, f4, and the pupil centers f2, f5 are accurately detected. Therefore, it is considered that the reliability of the line-of-sight direction v1 estimated based on these face feature points is also high. The monitoring device 12 monitors the state related to the driving of the driver based on the first line-of-sight direction v1 when the reliability of the first line-of-sight direction v1 is higher than a predetermined reliability threshold.

[0021] On the other hand, when the vehicle 10 makes a left turn, the driver 30 turns the face to the left with respect to the vehicle front in order to check the situation in the left turn direction. As shown in FIG. 1(D), when the face of the driver 30 faces left with respect to the vehicle front, there may be a case where face feature points cannot be accurately detected based on the captured face image. Therefore, there is a possibility that the accuracy of the line-of-sight direction estimated based on this face feature point is not high.

[0022] Therefore, when the reliability of the estimated line-of-sight direction is low in the line-of-sight estimation device 11, it is determined whether the driver 30 is in a gazing state based on the first line-of-sight direction v1 and a plurality of other line-of-sight directions of the driver 30 within a predetermined period. Next, when the driver 30 is in a gazing state, the line-of-sight estimation device 11 detects a gazing object that is estimated to be gazed at by the driver 30. Next, the line-of-sight estimation device 11 corrects the first line-of-sight direction v1 using the gazing object, and outputs the corrected first line-of-sight direction v4 (described later) as an estimated value. The monitoring device 12 monitors the state related to the driving of the driver based on the corrected first line-of-sight direction v4.

[0023] Therefore, even when the driver's face is facing left or right with respect to the front of the vehicle, the monitoring system 1 can accurately estimate the driver's line-of-sight direction based on the image of the driver's face being photographed, so that the state of the driver can be correctly monitored.

[0024] FIG. 2 is a schematic configuration diagram of a vehicle 10 in which the line-of-sight estimation device 11 of the present embodiment is mounted. The monitoring system 1 includes a monitoring camera 2, a front camera 3, a user interface (UI) 4, a line-of-sight estimation device 11, a monitoring device 12, and the like.

[0025] The monitoring camera 2, the front camera 3, the UI 4, the line-of-sight estimation device 11, and the monitoring device 12 are communicably connected via an in-vehicle network 13 compliant with a standard such as a controller area network.

[0026] The monitoring camera 2 is disposed in the vehicle interior so as to be able to photograph a face image including the face of the driver driving the vehicle 10. The monitoring camera 2 is an example of a photographing device that photographs a face image including the face of the driver. As shown in FIG. 1(B), the monitoring camera 2 is disposed, for example, on the steering column 31. Note that the monitoring camera 2 may be disposed on a steering wheel, a rearview mirror, a meter panel, a meter hood, or the like in the vehicle interior.

[0027] The monitoring camera 2 captures a face image including the driver's face, for example, at a predetermined period. The monitoring camera 2 is an example of an imaging device that captures a face image including the driver's face. The monitoring camera 2 includes a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to infrared rays, such as a CCD or a C-MOS, and an imaging optical system that forms an image of an area to be imaged on the two-dimensional detector. The predetermined period can be, for example, 0.1 to 0.5 seconds. Preferably, the monitoring camera 2 has a projector together with the two-dimensional detector. The projector is an LED (light-emitting diode), for example, two near-infrared LEDs arranged on both sides of the imaging optical system. By irradiating the driver with near-infrared light, it is possible to capture the driver's face without giving discomfort to the driver even in low illumination conditions such as at night. Also, a band-pass filter for removing light having wavelength components other than near-infrared may be provided inside the imaging optical system, and a visible light cut filter for removing light other than near-infrared irradiated from the near-infrared LED may be provided in front of the projector.

[0028] The front camera 3 is mounted, for example, in the passenger compartment of the vehicle 10 so as to face the front of the vehicle 10. The front camera 3 is an example of an imaging device that captures a front image representing the environment in front of the driver. The front camera 3 captures a front image representing a predetermined area in front of the vehicle 10 and the driver at the same front image capturing time as the face image capturing time at the same period as the monitoring camera 2. The front image shows pedestrians around the vehicle 10, other vehicles, or road features such as crosswalks on the road surface included in a predetermined area in front of the vehicle 10. The front camera 3 includes a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light, such as a CCD or a C-MOS, and an imaging optical system that forms an image of an area to be imaged on the two-dimensional detector. Preferably, the front camera 3 has a wide field of view so as to include the driver's field of view. For example, the front camera 3 preferably has a field of view close to 180 degrees.

[0029] UI4 is an example of a notification unit. UI4 is controlled by the monitoring device 12 to notify the driver of attention requests or the like that require the driver to pay attention to the environment around the vehicle. UI4 has a display device 4a such as a liquid crystal display or a touch panel to display attention requests or the like. Further, UI4 may have an acoustic output device (not shown) for notifying the driver of attention requests or the like. Also, UI4 has, as an input device for inputting operation information from the driver to the vehicle 10, for example, a touch panel or operation buttons. Examples of the operation information include information indicating that the driver has approved an attention request. UI4 outputs the input operation information to the monitoring device 12 via the in-vehicle network 13.

[0030] The gaze estimation device 11 estimates the driver's gaze direction based on the face image and outputs the estimated gaze direction to the monitoring device 12. The gaze estimation device 11 has a communication interface (IF) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 has an interface circuit for connecting the gaze estimation device 11 to the in-vehicle network 13.

[0031] The memory 22 is an example of a storage unit and has, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. Then, the memory 22 stores a computer program of an application used in information processing executed by the processor 23 and various data. In the memory 22, the face image input from the monitoring camera 2 is stored in association with the face image capture time. Also, in the memory 22, the front image input from the front camera 3 is stored in association with the front image capture time.

[0032] All or part of the functions of the line-of-sight estimation device 11 are functional modules realized by, for example, a computer program operating on the processor 23. The processor 23 includes a feature point detection unit 231, a line-of-sight estimation unit 232, a determination unit 233, an object detection unit 234, and a correction unit 235. Alternatively, the functional modules of the processor 23 may be dedicated arithmetic circuits provided in the processor 23. The processor 23 includes one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 23 may further include other arithmetic circuits such as a logical arithmetic unit, a numerical arithmetic unit, or a graphic processing unit. Details of the operation of the line-of-sight estimation device 11 will be described later.

[0033] The monitoring device 12 detects an object that is estimated to be being gazed at by the driver among the objects shown in the front image based on the front image and the driver's line-of-sight direction. The monitoring device 12 determines whether the driver is gazing at a predetermined object when the vehicle 10 is traveling through an intersection. The monitoring device 12 determines that the degree of the driver's involvement in driving is low if the driver is not gazing at a predetermined object. For example, when the vehicle 10 is turning left or right at an intersection, the monitoring device 12 determines that the degree of the driver's involvement in driving is low if the driver 30 is not gazing at a pedestrian even though there is a pedestrian in the direction of the crosswalk. Also, when the vehicle 10 is traveling through an intersection, the monitoring device 12 determines that the degree of the driver's involvement in driving is low if the driver is not gazing at a traffic signal even though there is a traffic signal. When the monitoring system 1 determines that the degree of the driver's involvement in driving is low, it notifies the driver via the UI4 of a request to pay attention to the environment around the vehicle.

[0034] In FIG. 2, the line-of-sight estimation device 11 and the monitoring device 12 are described as separate devices, but all or part of these devices may be configured as one device. The line-of-sight estimation device 11 and the monitoring device 12 are, for example, an electronic control unit (ECU).

[0035] Figure 3 is an example of an operation flowchart regarding the gaze estimation process of the gaze estimation device 11 of the present embodiment. Next, the gaze estimation process of the monitoring system 1 will be described below with reference to Figure 3. Each time a face image is input from the monitoring camera 2, the gaze estimation device 11 executes a gaze estimation process according to the operation flowchart shown in Figure 3.

[0036] First, the feature point detection unit 231 inputs a face image and a front image from the monitoring camera 2 via the in-vehicle network 13 (step S101). Each time the monitoring camera 2 captures a face image, the monitoring camera 2 outputs the face image and the face image capture time when the face image was captured to the gaze estimation device 11 via the in-vehicle network 13. The feature point detection unit 231 stores the face image in the memory 22 in association with the face image capture time.

[0037] Also, the feature point detection unit 231 inputs a front image and a front image capture time from the front camera 3 via the in-vehicle network 13. Each time the front camera 3 generates a front image, the front camera 3 outputs the front image and the front image capture time when the front image was captured to the gaze estimation device 11 and the monitoring device 12 via the in-vehicle network 13. The feature point detection unit 231 stores the front image in the memory 22 in association with the front image capture time.

[0038] Next, the feature point detection unit 231 detects face feature points in the face image and determines the reliability of the detected face feature points (step S102). The feature point detection unit 231 includes a discriminator that is trained to input a face image and detect predetermined parts such as the outer corner of the eye, the inner corner of the eye, and the pupil center as face feature points. This discriminator inputs a face image, detects the types and positions of the face feature points included in the face image, and outputs the reliability of the detected face feature points. The reliability is represented by a numerical value between 0.0 and 1.0, for example. When the reliability when face feature points representing the driver's left eye or / and right eye are detected is lower than a predetermined reference value, the feature point detection unit 231 extracts the region including the left eye or / and right eye from the face image and generates a partial image. If necessary, the resolution of the extracted region may be increased. The feature point detection unit 231 may input this partial image to the discriminator, detect the types and positions of the face feature points, and determine the reliability of the detected face feature points.

[0039] The discriminator is, for example, a convolutional neural network (CNN) having a plurality of layers connected in series from the input side to the output side. By inputting a face image including face feature points as teacher data into the CNN in advance and performing learning, the CNN operates as a discriminator that detects the types and positions of face feature points. The discriminator preferably outputs the reliability of the face feature points such that the reliability of the estimated feature points follows a normal distribution centered on the reliability of the true feature points. Also, as the discriminator, a machine learning model such as a support vector machine or a random forest may be used.

[0040] The positions of the face feature points are represented using the monitoring camera coordinate system. In the monitoring camera coordinate system, with the center of the imaging plane as the origin, the Za axis is set in the optical axis direction of the monitoring camera 2, the Xa axis is set in a direction orthogonal to the Za axis and parallel to the ground, the Ya axis is set in a direction orthogonal to the Za axis and the Xa axis, and the origin is at the height at which the monitoring camera 2 is installed from the ground.

[0041] The feature point detection unit 231 converts the position of the facial feature points represented in the monitoring camera coordinate system into the position represented in the vehicle coordinate system, and notifies the gaze estimation unit 232 together with the feature point detection time (face image capture time) indicating the time when the facial feature points are detected. The vehicle coordinate system has its origin at the center of the rear axle connecting the two rear wheels of the vehicle 10, with the Zb axis set in the traveling direction of the vehicle 10, the Xb axis set in a direction orthogonal to the Zb axis and parallel to the ground, and the Yb axis set in the vertical direction. The conversion formula for converting the position of the facial feature points from the monitoring camera coordinate system to the vehicle coordinate system is represented by a combination of a rotation matrix representing the rotation between the coordinate systems and a translation vector representing the translation between the coordinate systems.

[0042] Next, the gaze estimation unit 232 estimates the first gaze direction v1 of the driver using the facial feature points, and obtains the reliability of the first gaze direction v1 based on the reliability of the facial feature points (step S103). The gaze estimation unit 232 obtains the position of the center of the eyeball based on the positions of the outer corner and inner corner of the eye. For example, the gaze estimation unit 232 sets the position obtained by advancing a predetermined distance in a predetermined direction (for example, the negative direction of the Zb axis) from the midpoint of the line segment connecting the position of the outer corner and the position of the inner corner of the eye as the position of the center of the eyeball. The gaze estimation unit 232 estimates the direction connecting the position of the center of the eyeball and the position of the center of the pupil as the first gaze direction v1. The gaze estimation unit 232 may estimate the first gaze direction v1 based on either the left or right eye of the driver, or may estimate the gaze directions of each of the left and right eyes and use the averaged direction as the first gaze direction v1. Note that the gaze estimation unit 232 may estimate the driver's gaze direction based on the positions of the center of the pupil and the Purkinje image, which are facial feature points.

[0043] In addition, the gaze estimation unit 232 obtains the reliability of the first gaze direction v1 based on the reliability of a plurality of facial feature points (the outer corners of the eyes, the inner corners of the eyes, and the pupil centers) used to estimate the first gaze direction v1 of the driver. For example, the gaze estimation unit 232 sets the average value of the reliabilities of the plurality of facial feature points (the outer corners of the eyes, the inner corners of the eyes, and the pupil centers) as the reliability of the first gaze direction v1. Alternatively, the gaze estimation unit 232 may set the minimum value among the reliabilities of the plurality of facial feature points (the outer corners of the eyes, the inner corners of the eyes, and the pupil centers) as the reliability of the first gaze direction v1. The gaze estimation unit 232 stores the first gaze direction v1 and the reliability in the memory 22 in association with the first gaze estimation time (feature point detection time) at which the first gaze direction was estimated.

[0044] Next, the gaze estimation unit 232 determines whether the reliability of the first gaze direction v1 is less than or equal to a reliability threshold (step S104). As the reliability threshold, for example, it can be set to 0.7 to 0.9. The gaze estimation unit 232 may end a series of processes when the reliability is less than or equal to a predetermined lower reliability limit value (for example, 0.4). This is because when the reliability of the first gaze direction v1 is particularly low, there is a possibility that a meaningful gaze direction cannot be obtained even if the first gaze direction v1 is corrected.

[0045] When it is determined that the reliability of the first gaze direction v1 is less than or equal to the reliability threshold (step S104 - Yes), the determination unit 233 determines whether the driver is in a gazing state based on the first gaze direction v1 and a plurality of other gaze directions of the driver 30 within a predetermined period before the time at which the first gaze direction v1 was estimated (step S105). The predetermined period can be set to be between 1 and 2 seconds before the first gaze estimation time at which the reliability of the first gaze direction v1 was obtained. The determination unit 233 reads out the gaze directions of the driver estimated within the predetermined period before the time at which the first gaze direction v1 was estimated from the memory 22 as the plurality of other gaze directions.

[0046] The determination unit 233 represents each of the first line-of-sight direction v1 and a plurality of other line-of-sight directions by a line segment having a starting point at the origin of the vehicle coordinate system and a predetermined length r, and represents the position of the end point of the line segment in polar coordinates (r, θ, φ). θ is the angle from the Xb axis of the projection component obtained by projecting the line segment onto the XbYb plane, and φ is the angle from the Zb axis to the line segment. The determination unit 233 obtains the variance of θ and the variance of φ for the line-of-sight directions including the first line-of-sight direction v1 and the plurality of other line-of-sight directions, and obtains the sum (or the average value of the variances) of the two variances as the variance of the line-of-sight direction.

[0047] When the variance of the line-of-sight direction is equal to or less than a predetermined variance reference value, if the first line-of-sight direction v1 is estimated, the determination unit 233 determines that the driver is in a gazing state (step S105 - Yes).

[0048] Note that the determination unit 233 may have a discriminator that is trained to input the polar coordinates (r, θ, φ) representing the positions of the end points for a plurality of first line-of-sight directions v1 and identify whether the driver is in a gazing state. This discriminator inputs the polar coordinates (r, θ, φ) representing the positions of the end points for a plurality of first line-of-sight directions v1 and outputs discriminant information indicating whether the driver is in a gazing state. The determination unit 233 may determine whether the driver is in a gazing state based on the discriminant information.

[0049] In addition, the line-of-sight estimation unit 232 may have a discriminator that is trained to input the reliabilities of the outer corners and inner corners of the eyes, which are the face feature points used to estimate the reliability of the first line-of-sight direction v1, and detect the reliability of the position of the pupil center. This discriminator inputs the reliabilities of the outer corners and inner corners of the eyes, which are the face feature points used to estimate the reliability of the first line-of-sight direction v1, and outputs the reliability of the position of the pupil center. The line-of-sight estimation unit 232 may determine whether the driver is in a gazing state based on the reliability of the position of the pupil center. When the reliability of the position of the pupil center is high, it is estimated that the driver is in a gazing state.

[0050] Next, when it is determined that the driver is in a gazing state, the object detection unit 234 detects a gazing object estimated to be gazed at by the driver among the objects shown in the front image representing the environment in front of the driver based on the first line-of-sight direction v1 (step S106). Details of the process of detecting the gazing object will be described later.

[0051] Next, the correction unit 235 corrects the first line-of-sight direction v1 so as to face the gazing object shown in the front image (step S107). Details of the process of correcting the first line-of-sight direction v1 will be described later.

[0052] Next, the correction unit 235 notifies the monitoring device 12 of the corrected first line-of-sight direction and ends the series of processes (step S108).

[0053] Also, when it is determined that the reliability of the facial feature points is greater than the reliability threshold (step S104 - No), the correction unit 235 notifies the monitoring device 12 of the first line-of-sight direction v1 and ends the series of processes (step S108).

[0054] When it is determined that the driver is not in a gazing state (step S105 - No), since the first line-of-sight direction v1 cannot be corrected, the line-of-sight direction is not estimated and the series of processes ends.

[0055] FIG. 4 is an example of an operation flowchart regarding the gazing object detection process in step S106 described above. First, the object detection unit 234 reads out from the memory 22 a front image taken at the front image shooting time that coincides with the first line-of-sight direction estimation time when the first line-of-sight direction v1 was estimated. The object detection unit 234 detects the type and position of the objects shown in the front image (step S201). The object detection unit 234 has a discriminator that has been trained to detect the objects shown in the image by inputting the front image. This discriminator inputs the front image and detects the type and position of the objects shown in the front image. Objects include people such as pedestrians considered to be located on the road and around the road, traffic signals, and road features such as crosswalks.

[0056] The identifier is, for example, a convolutional neural network (CNN) having a plurality of layers connected in series from an input side to an output side. By inputting an image including road features such as a person such as a pedestrian, a traffic signal, and a crosswalk as teacher data into the CNN and performing learning in advance, the CNN operates as an identifier that detects the type and position of an object. Also, as the identifier, a machine learning model such as a support vector machine or a random forest may be used.

[0057] FIG. 5 is a diagram showing an example of a front image. On the front image 500, an object area 501 representing a pedestrian, an object area 502 representing another pedestrian, and an object area 503 representing a crosswalk are shown.

[0058] The front image is an image in which an object is projected onto an imaging plane that is orthogonal to the Zc axis of the front camera coordinate system and whose center intersects the Zc axis. In the front camera coordinate system, with the center of the imaging plane as the origin, the Zc axis is set in the optical axis direction of the front camera 3, the Xc axis is set in a direction orthogonal to the Zc axis and parallel to the ground, the Yc axis is set in a direction orthogonal to the Zc axis and the Xc axis, and the origin is at the height at which the front camera 3 is installed from the ground.

[0059] Next, the object detection unit 234 detects a fixation object that is at the position closest to the intersection of the first line-of-sight direction v1 for which reliability has been obtained and the front image among the objects shown in the front image (step S202).

[0060] The object detection unit 234 performs viewpoint conversion on the front image and converts it into an image with the center of the driver's eyeball as the viewpoint. As such viewpoint conversion of the image, a known method can be used.

[0061] The object detection unit 234 places the starting point of the vector having the direction of the first line-of-sight direction v1 at the center of the driver's eyeball, extends this vector, and determines the distance between the position 504 of the intersection point orthogonal to the front image after viewpoint conversion and each of the object regions 501 to 503. As the distance between the intersection point 504 and the object region, for example, the distance between the intersection point 504 and the centroid of the object region can be used. The object detection unit 234 detects the object region 501 representing a pedestrian at the position closest to the intersection point 504 as the object of fixation, and a series of processes ends. Note that the object detection unit 234 may compare the change in the direction of the first line-of-sight direction v1 at a plurality of times with the change in the positions of the object regions 501, 502, and 503, and detect, as the object of fixation, the object region showing the change in position most similar to the change in the direction of the first line-of-sight direction v1.

[0062] The object detection unit 234 may output the type and position of the object represented in the front image to the monitoring device 12. Based on the type and position of the object represented in the front image input from the object detection unit 234 and the driver's line-of-sight direction, the monitoring device 12 may detect an object estimated to be being stared at by the driver among the objects shown in the front image and determine the degree of involvement in the driver's driving. Further, the monitoring device 12 may execute the same object detection process as the object detection unit 234 and detect an object estimated to be being stared at by the driver among the objects shown in the front image.

[0063] FIG. 6 is an example of an operation flowchart regarding the correction process in step S107 described above. First, the line-of-sight estimation unit 232 estimates the driver's second line-of-sight direction based on the position of the object of fixation in the front image (step S301).

[0064] FIG. 7 is a diagram for explaining an example of the second line-of-sight direction detection process. For the face image 42 used for estimating the first line-of-sight direction v1, the line-of-sight estimation unit 232 estimates the position of the bridge of the driver's nose p1 based on the position of the inner corner of the eye f3, which is a face feature point representing the driver's left eye in the vehicle coordinate system, and the position of the inner corner of the eye f4, which is a face feature point representing the driver's right eye. The line-of-sight estimation unit 232 estimates the position of the bridge of the nose p1 as the position at a predetermined distance above the center position of the line segment connecting the inner corner of the eye f3 and the inner corner of the eye f4 (for example, in the positive direction of the Yb axis).

[0065] The object detection unit 234 performs viewpoint conversion on the front image and converts it into an image with the position of the bridge of the driver's nose p1 as the viewpoint. As such viewpoint conversion of the image, a known method can be used.

[0066] The correction unit 235 estimates the direction from the position of the bridge of the nose p1 to the object region 501, which is the object being gazed at, as the second line-of-sight direction v2. The line-of-sight estimation unit 232 obtains the direction from the position of the bridge of the nose p1 to the centroid of the object region 501 on the front image 500 shown in FIG. 5 as the second line-of-sight direction v2.

[0067] Next, the line-of-sight estimation unit 232 estimates the face orientation direction in which the driver's face is facing using the face feature points (step S302). The line-of-sight estimation unit 232 is an example of a face orientation direction estimation unit. FIG. 8 is a diagram for explaining an example of the face orientation direction estimation process. The feature point detection unit 231 inputs the face image 42 used for estimating the first line-of-sight direction v1 into the discriminator, detects the positions of the outer corners of the eyes f1, f6, the inner corners of the eyes f3, f4, the tip of the nose point f7, and the corner of the mouth points f8, f9, and notifies the line-of-sight estimation unit 232. The line-of-sight estimation unit 232 fits the detected face feature points to a three-dimensional face model representing the three-dimensional shape of the face. The line-of-sight estimation unit 232 estimates the orientation of the face of the three-dimensional face model when each face feature point fits best to the three-dimensional face model as the face orientation direction v3 of the driver. The line-of-sight estimation unit 232 converts the second line-of-sight direction v2 represented in the monitoring camera coordinate system into the vehicle coordinate system.

[0068] Note that the line-of-sight estimation unit 232 may estimate the face orientation direction of the driver as the direction that is orthogonal to the line segment connecting the positions of the outer corners and the inner corners of the eyes, which are the face feature points of the driver, starts from the center position of the line segment, and extends outward in a direction parallel to the ground. The line-of-sight estimation unit 232 may estimate the face orientation direction based on either the left or right eye of the driver, or may estimate the face orientation direction of each of the left and right eyes and use the average of these directions as the face orientation direction.

[0069] Next, the correction unit 235 corrects the first line-of-sight direction v1 based on the second line-of-sight direction v2 and the face orientation direction v3 (step S303). FIG. 9 is a diagram for explaining an example of the correction process.

[0070] First, as shown in FIG. 9, the correction unit 235 obtains the angle α formed by the face orientation direction v3 and the second line-of-sight direction v2. Further, the correction unit 235 obtains the correction amount β as the product of the angle α, the reciprocal c of the reliability of the first line-of-sight direction v1, and the coefficient k. The correction amount β increases as the reliability of the first line-of-sight direction v1 decreases and decreases as the reliability of the first line-of-sight direction v1 increases. Here, when the reliability is equal to or lower than a predetermined reliability reference value (for example, when the reliability is 0.4 or lower), the correction unit 235 may set the reliability of the face feature points to 0.4 and obtain the reciprocal c of the reliability of the face feature points. Thereby, it is possible to prevent the correction amount β from diverging when the reliability of the first line-of-sight direction v1 is low.

[0071] Then, as shown in FIG. 9, the correction unit 235 moves the first line-of-sight direction v1 by an angle represented by the correction amount β in the direction from the second line-of-sight direction v2 toward the face orientation direction v3 to correct the first line-of-sight direction v1 and obtain the corrected first line-of-sight direction v4.

[0072] As described above, this position estimation device can accurately estimate the line-of-sight direction of the driver based on the image of the driver's face even when the driver's face is facing the front of the vehicle and also when the driver's face is facing left or right with respect to the front of the vehicle. Therefore, the monitoring system can correctly monitor the state of the driver based on the line-of-sight direction of the driver.

[0073] In the present disclosure, the line-of-sight estimation device, the computer program for line-of-sight estimation, and the line-of-sight estimation method of the above-described embodiments can be appropriately modified without departing from the gist of the present disclosure. Further, the technical scope of the present disclosure is not limited to those embodiments, but extends to the invention described in the claims and its equivalents.

[0074] For example, in the above-described embodiment, based on one first line-of-sight direction, among the objects shown in the front image, the fixation object estimated to be being fixated on by the driver has been detected. The object detection unit obtains the average position of the intersections of the first line-of-sight direction and each of a plurality of other line-of-sight directions of the driver within a predetermined period before the time when the first line-of-sight direction was estimated, with the front image, and among the objects shown in the front image, the object at the position closest to the average position may be detected as the fixation object. Further, the object detection unit may have a discriminator that inputs the first line-of-sight direction and a plurality of other line-of-sight directions of the driver within a predetermined period before the time when the first line-of-sight direction was estimated, and is learned to detect the fixation object in the image. This discriminator inputs the first line-of-sight direction and the plurality of other line-of-sight directions, and detects the fixation object from the front image. The discriminator is, for example, a convolutional neural network (CNN) having a plurality of layers connected in series from the input side to the output side. By inputting an image including a plurality of line-of-sight directions and fixation objects in advance as teacher data to the CNN and performing learning, the CNN operates as a discriminator that detects the fixation object.

[0075] Further, when the determination unit detects the fixation object in the front image, if the first line-of-sight direction of the driver is outside the field of view of the front camera, the first line-of-sight direction of the driver may not be within the front image. Therefore, the determination unit may use the front images captured at a plurality of past times to track the movement of the object regions shown in these front images, estimate the position of the object region shown in the current front image, and detect the fixation object based on the relationship between the estimated object region and the first line-of-sight direction.

[0076] In the above-described embodiment, the vehicle has one front camera that captures the environment in front of the driver. However, the vehicle may have a plurality of front cameras that capture the environment in front of the driver. It is preferable that the plurality of front cameras have different optical axis directions on the left and right so that a part of their fields of view overlaps. The determination unit can detect a fixation object that is estimated to be being fixated on by the driver using a plurality of front images captured by the plurality of front cameras. This makes it possible to detect the fixation object even when the driver's line of sight is directed in a direction that cannot be captured by a single front camera.

[0077] Further, when the reliability of the first line-of-sight direction is equal to or less than a predetermined threshold value and the driver is in a fixation state, the correction unit may obtain a correction amount based on the product of the angle formed by the face orientation direction and the first line-of-sight direction and the reciprocal of the reliability of the first line-of-sight direction, and correct the first line-of-sight direction by moving the first line-of-sight direction by an angle represented by the correction amount in the direction from the first line-of-sight direction toward the face orientation direction.

[0078] Furthermore, the correction unit may obtain the correction amount using an identifier that has been learned to input the face image, the first line-of-sight direction, and the second line-of-sight direction and output the correction amount.

Explanation of Reference Numerals

[0079] 1 Monitoring system 2 Monitoring camera 3 Front camera 4 User interface 4a Display device 10 Vehicle 11 Line-of-sight estimation device 12 Monitoring device 13 In-vehicle network 21 Communication interface 22 Memory 23 Processor 231 Feature point detection unit 232 Line-of-sight estimation unit 233 Determination unit 234 Object detection unit 235 Correction unit

Claims

1. A feature point detection unit that detects a plurality of facial feature points in a facial image including the face of the driver and obtains the reliability of each of the detected plurality of facial feature points; A first line-of-sight estimation unit that estimates the first line-of-sight direction of the driver using the plurality of facial feature points, and obtains the average value or the minimum value of the reliabilities of the plurality of facial feature points as the reliability of the first line-of-sight direction; When the reliability of the first line-of-sight direction is equal to or less than a predetermined threshold value, a determination unit that determines whether or not the driver is in a gazing state based on the first line-of-sight direction and a plurality of other line-of-sight directions of the driver within a predetermined period before the time when the first line-of-sight direction is estimated; When the driver is in the gazing state, an object detection unit that detects a gazing object that is estimated to be gazed at by the driver among the objects reflected in a front image representing the environment in front of the driver based on the first line-of-sight direction; A correction unit that corrects the first line-of-sight direction so as to face the gazing object when the reliability of the first line-of-sight direction is equal to or less than a predetermined threshold value and the driver is in the gazing state; A line-of-sight estimation device having the above.

2. Furthermore, A face orientation direction estimation unit that estimates the face orientation direction in which the face of the driver is facing using the plurality of facial feature points; Based on the positions of the facial feature points representing the left eye of the driver and the positions of the facial feature points representing the right eye of the driver, the position of the bridge of the driver's nose is estimated, and the direction from the position of the bridge of the nose to the gazing object is estimated as the second line-of-sight direction of the driver; Having The correction unit, when the reliability of the first line-of-sight direction is equal to or less than a predetermined threshold value and the driver is in the gazing state, Obtains a correction amount based on the product of the angle formed by the face orientation direction and the second line-of-sight direction and the reciprocal of the reliability of the first line-of-sight direction, The line-of-sight estimation device according to claim 1, wherein the first line-of-sight direction is corrected by moving the first line-of-sight direction by an angle represented by the correction amount in a direction from the second line-of-sight direction toward the face orientation direction.

3. The determination unit determines that the driver is in the gazing state when the variance of the first line-of-sight direction and the plurality of other line-of-sight directions is within a predetermined reference value. The line-of-sight estimation device according to claim 1 or 2.

4. Detect a plurality of facial feature points in a facial image including the face of the driver and obtain the reliability of each of the detected plurality of facial feature points, Estimate the first line-of-sight direction of the driver using the plurality of face feature points, and obtain the average value or the minimum value of the reliability of the plurality of face feature points as the reliability of the first line-of-sight direction. When the reliability of the first line-of-sight direction is equal to or lower than a predetermined threshold, determine whether the driver is in a gazing state based on the first line-of-sight direction and a plurality of other line-of-sight directions of the driver within a predetermined period before the time when the first line-of-sight direction is estimated. When the driver is in the gazing state, detect a gazing object estimated to be gazed at by the driver among the objects shown in the front image representing the environment in front of the driver based on the first line-of-sight direction. When the reliability of the first line-of-sight direction is equal to or lower than a predetermined threshold and the driver is in the gazing state, correct the first line-of-sight direction so as to face the gazing object. A computer program for line-of-sight estimation, which causes a processor to execute the above.

5. Detect a plurality of face feature points in a face image including the face of the driver and obtain the reliability of each of the detected plurality of face feature points. Estimate the first line-of-sight direction of the driver using the plurality of face feature points, and obtain the average value or the minimum value of the reliability of the plurality of face feature points as the reliability of the first line-of-sight direction. When the reliability of the first line-of-sight direction is equal to or lower than a predetermined threshold, determine whether the driver is in a gazing state based on the first line-of-sight direction and a plurality of other line-of-sight directions of the driver within a predetermined period before the time when the first line-of-sight direction is estimated. When the driver is in the gazing state, detect a gazing object estimated to be gazed at by the driver among the objects shown in the front image representing the environment in front of the driver based on the first line-of-sight direction. When the reliability of the first line-of-sight direction is equal to or lower than a predetermined threshold and the driver is in the gazing state, correct the first line-of-sight direction so as to face the gazing object. A line-of-sight estimation method, characterized in that the above is executed by a line-of-sight estimation device.

Images