Sight line detection device

Abstract

To provide a sight line detection device capable of improving recognition accuracy of a sight line direction of a driver.SOLUTION: A sight line detection device 1 detects a sight line of a driver D of a vehicle 2. The sight line detection device 1 includes: an FOE sensor 3 used for detecting an infinite far point ahead of the vehicle 2; a face recognition camera 4 that images a face of the driver D; an infinite far point detection unit 11 that detects an infinite far point on the basis of detection data from the FOE sensor 3; a zero point learning unit 12 that learns a position of a zero point P in sight line detection of the driver D on the basis of the infinite far point detected by the infinite far point detection unit 11; and a sight line direction recognition unit 13 that recognizes a sight line direction of the driver D relative to the zero point P on the basis of image data from the face recognition camera 4 and the position of the zero point P learned by the zero point learning unit 12.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a line-of-sight detection device.

Background Art

[0002] For example, in Patent Document 1, based on a driver's face image captured by an imaging device, the face orientation angle of the driver is recognized, and when zero-point setting conditions including that the vehicle speed of the vehicle is equal to or higher than a predetermined speed are satisfied, based on the face orientation angle of the driver at that time, a zero point of the face orientation angle of the driver is set, and based on the current face orientation angle with respect to the zero point, a technique for diagnosing the state of the driver is disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above prior art, after setting the zero point of the driver's face orientation angle, if the posture of the vehicle changes, the line-of-sight direction of the driver with respect to the zero point may shift and be misrecognized.

[0005] An object of the present invention is to provide a line-of-sight detection device capable of improving the recognition accuracy of the driver's line-of-sight direction.

Means for Solving the Problems

[0006] One aspect of the present invention is a line-of-sight detection device that detects the line of sight of a driver of a vehicle, including a sensor used for detecting an infinite point in front of the vehicle, an imaging unit that captures an image of the driver's face, an infinite point detection unit that detects the infinite point based on the detection data of the sensor, a zero-point learning unit that learns the position of the zero point in the detection of the driver's line of sight based on the infinite point detected by the infinite point detection unit, and a line-of-sight direction recognition unit that recognizes the line-of-sight direction of the driver with respect to the zero point based on the image data of the imaging unit and the position of the zero point learned by the zero-point learning unit.

Effect of the Invention

[0007] According to the present invention, the recognition accuracy of the driver's line-of-sight direction can be improved.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0010] FIG. 1 is a schematic configuration diagram showing a line-of-sight detection device according to an embodiment of the present invention. In FIG. 1, the line-of-sight detection device 1 of the present embodiment is mounted on a vehicle 2 (see FIG. 3) such as an automobile. The line-of-sight detection device 1 is a device that detects the line of sight of the driver D of the vehicle 2.

[0011] The line-of-sight detection device 1 includes an FOE sensor 3, a face recognition camera 4, an ECU (Electronic Control Unit) 5, and a warning device 6.

[0012] The FOE sensor 3 is a sensor used to detect the focus of expansion (FOE) at an infinite distance in front of the vehicle 2. That is, the FOE sensor 3 is a sensor capable of detecting the focus of expansion at an infinite distance in front of the vehicle 2. As the FOE sensor 3, a stereo camera, a depth camera, LiDAR (Light Detection And Ranging), or the like is used. The FOE sensor 3 is arranged so as to face the front of the vehicle 2 (see Fig. 3).

[0013] The face recognition camera 4 is an imaging unit that images the face of the driver D of the vehicle 2. The face recognition camera 4 images the face of the driver D and acquires an imaged image. Although not shown, the face recognition camera 4 is arranged at a position facing the driver D seated in the driver's seat of the vehicle 2.

[0014] The ECU 5 is composed of a CPU, a RAM, a ROM, an input / output interface, and the like. The ECU 5, for example, loads a program recorded in the ROM into the RAM and executes the program loaded into the RAM with the CPU.

[0015] When the ignition switch 7 is turned ON, the ECU 5 recognizes the face orientation angle of the driver D and diagnoses the state of the driver D based on the output data of the FOE sensor 3 and the face recognition camera 4, and controls the warning device 6 according to the recognition / diagnosis result. The warning device 6 issues a warning by voice or screen display or the like.

[0016] The ECU 5 includes an infinite point detection unit 11, a zero point learning unit 12, a line-of-sight direction recognition unit 13, a side glance determination unit 14, and a warning control unit 15.

[0017] The infinite point detection unit 11 detects the infinite point in front of the vehicle 2 based on the detection data (output data) of the FOE sensor 3. The infinite point is, for example, a point (converging point) where a distant road appears to converge toward the center of the driver D's field of view when the vehicle 2 is in operation.

[0018] The zero - point learning unit 12 learns the position of the zero - point P (see FIG. 3) in the detection of the driver D's line of sight based on the infinite - point detected by the infinite - point detection unit 11. The zero - point learning unit 12 outputs the position information of the zero - point P obtained by learning to the line - of - sight direction recognition unit 13.

[0019] The zero - point P in the detection of the driver D's line of sight indicates the direction of the line of sight when the driver D is correctly looking at the infinite - point ahead during the running of the vehicle 2. When the attitude (pitch angle, yaw angle, roll angle) of the vehicle 2 changes, the infinite - point ahead of the vehicle 2 changes, so the position of the zero - point P in the detection of the driver D's line of sight also changes.

[0020] The line - of - sight direction recognition unit 13 recognizes the line - of - sight direction of the driver D with respect to the zero - point P based on the image data (output data) of the face - recognition camera 4 and the position of the zero - point P learned by the zero - point learning unit 12. When the position of the zero - point P is corrected, the line - of - sight direction recognition unit 13 recognizes the line - of - sight direction of the driver D with respect to the corrected zero - point P.

[0021] The side - glance determination unit 14 determines whether the driver D of the vehicle 2 is making a side - glance based on the line - of - sight direction of the driver D with respect to the zero - point P recognized by the line - of - sight direction recognition unit 13.

[0022] When the side - glance determination unit 14 determines that the driver D is making a side - glance, the warning control unit 15 controls the warning device 6 to give a warning by the warning device 6.

[0023] FIG. 2 is a flowchart showing the procedure of the line - of - sight detection process executed by the ECU 5. In FIG. 2, the ECU 5 first determines whether the ignition switch 7 is in the ON operation state (procedure S101).

[0024] When the ECU 5 determines that the ignition switch 7 is in the ON operation state, it acquires the detection data of the FOE sensor 3 (procedure S102). Then, the ECU 5 calculates the infinite - point ahead of the vehicle 2 based on the detection data of the FOE sensor (procedure S103).

[0025] Subsequently, the ECU 5 learns the position of the zero point P in the detection of the driver D's line of sight based on the infinite far point in front of the vehicle 2 (step S104). For example, the ECU 5 uses a machine learning algorithm to learn the position of the zero point P in the detection of the driver D's line of sight from the infinite far point in front of the vehicle 2.

[0026] Subsequently, the ECU 5 executes the above-described step S101 again. Therefore, while the ignition switch 7 is in the ON state, the above-described steps S101 to S104 are repeatedly executed.

[0027] Also, the ECU 5 acquires the image data of the face recognition camera 4 (step S105). Then, the ECU 5 recognizes the line-of-sight direction of the driver D with respect to the zero point P based on the image data of the face recognition camera 4 and the position of the zero point P in the detection of the driver D's line of sight obtained in step S104 (step S106).

[0028] Subsequently, the ECU 5 determines whether the driver D is looking away based on the line-of-sight direction of the driver D with respect to the zero point P (step S107). When the line-of-sight direction of the driver D with respect to the zero point P deviates from the direction of the zero point P by a specified angle or more, it is determined that the driver D is looking away.

[0029] When the ECU 5 determines that the driver D is looking away, the ECU 5 controls the warning device 6 to issue a warning (step S108). As a result, the driver D becomes aware of looking away due to the warning and self-evokes attention.

[0030] After the ECU 5 executes step S108, or when it is determined in step S107 that the driver D is not looking away, the ECU 5 determines whether the ignition switch 7 is in the OFF state (step S109).

[0031] When the ECU5 determines that the ignition switch 7 has been turned ON, it executes the procedure S105 again. Therefore, while the ignition switch 7 remains in the ON state, the above procedures S105 to S109 are repeatedly executed.

[0032] When the ECU5 determines in procedure S101 or S109 that the ignition switch 7 is in the OFF state, it terminates this process.

[0033] Here, the infinite point detection unit 11 executes procedures S102 and S103. The zero point learning unit 12 executes procedure S104. The line-of-sight direction recognition unit 13 executes procedures S105 and S106. The side glance determination unit 14 executes procedure S107. The warning control unit 15 executes procedure S108.

[0034] In the line-of-sight detection device 1 as described above, when the vehicle 2 is running, the face of the driver D of the vehicle 2 is imaged by the face recognition camera 4, the line-of-sight direction of the driver D is recognized based on the image data of the face recognition camera 4, and it is determined whether the driver D is making a side glance from the line-of-sight direction of the driver D.

[0035] At this time, as shown in Fig. 3(a), based on the detection data of the FOE sensor 3, the infinite point in front of the vehicle 2 is detected. Then, based on the infinite point in front of the vehicle 2, the position of the zero point P in the detection of the line of sight of the driver D is learned. And based on the image data of the face recognition camera 4 and the position of the zero point P, the line-of-sight direction of the driver D with respect to the zero point P is recognized.

[0036] When the ignition switch 7 is in the ON state, the position of the zero point P in the detection of the line of sight of the driver D is learned at any time. Therefore, as shown in Fig. 3(b), even if the posture of the vehicle 2 changes due to the movement of passengers or luggage during the running of the vehicle 2, the zero point P in the detection of the line of sight of the driver D is updated according to the posture of the vehicle 2. And the line-of-sight direction of the driver D with respect to the updated zero point P is recognized.

[0037] Incidentally, the attitude of the vehicle 2 varies depending on the type of the power train of the vehicle 2, options, etc., even among vehicles of the same model. Also, the attitude of the vehicle 2 changes depending on the loading conditions such as the passengers and luggage in the vehicle 2. When the attitude of the vehicle 2 changes, as shown in FIG. 3, the line-of-sight position of the driver D with respect to the vehicle 2 changes. Therefore, in order to ensure the recognition accuracy of the line-of-sight direction of the driver D of the vehicle 2, it is necessary to correct the position of the zero point P in the detection of the line of sight of the driver D.

[0038] Therefore, in the present embodiment, when the vehicle 2 is running, the infinite far point in front of the vehicle 2 is detected based on the detection data of the FOE sensor 3 used for detecting the infinite far point in front of the vehicle 2. Then, based on the infinite far point in front of the vehicle 2, the position of the zero point P in the detection of the line of sight of the driver D of the vehicle 2 is learned. Also, when the vehicle 2 is running, the face of the driver D is imaged by the face recognition camera 4. Then, based on the image data of the face recognition camera 4 and the learned position of the zero point P, the line-of-sight direction of the driver D with respect to the zero point P is recognized. In this way, by using the FOE sensor 3, the zero point P in the detection of the line of sight of the driver D is learned at any time. For this reason, even if the attitude of the vehicle 2 changes during the running of the vehicle 2, the position of the zero point P is newly learned, and the line-of-sight direction of the driver D with respect to the updated zero point P is recognized. As a result, the misrecognition accuracy of the line-of-sight direction of the driver D is improved. As a result, in the determination of whether the driver D is looking away during the running of the vehicle 2, the determination delay and the misjudgment can be reduced.

[0039] Also, in the present embodiment, when learning the zero point P in the detection of the line of sight of the driver D, it is not necessary to recognize the face of the driver D by the face recognition camera 4 with the running speed of the vehicle 2 increased to the required specified speed. Therefore, the learning of the zero point P can be realized in a short time and easily.

[0040] Note that the present invention is not limited to the above-described embodiments. For example, in the above-described embodiments, when the ignition switch 7 is turned on, the zero point P in the detection of the driver D's line of sight is learned as needed until the ignition switch 7 is turned off. However, it is not particularly limited to such a form. For example, the zero point P in the detection of the driver D's line of sight may be learned as needed only while the vehicle 2 is running.

Explanation of Signs

[0041] 1... Line-of-sight detection device, 2... Vehicle, 3... FOE sensor (sensor), 4... Face recognition camera (imaging unit), 11... Infinity point detection unit, 12... Zero point learning unit, 13... Line-of-sight direction recognition unit, D... Driver, P... Zero point.

Claims

Claim 1 A line-of-sight detection device for detecting the line of sight of a driver of a vehicle, comprising: a sensor used for detecting an infinite point in front of the vehicle; an imaging unit for imaging the face of the driver; an infinite point detection unit for detecting the infinite point based on the detection data of the sensor; a zero-point learning unit for learning the position of a zero point in the detection of the driver's line of sight based on the infinite point detected by the infinite point detection unit; a line-of-sight direction recognition unit for recognizing the line-of-sight direction of the driver with respect to the zero point based on the image data of the imaging unit and the position of the zero point learned by the zero-point learning unit.

Images