Gaze direction detection device

Abstract

Calibration is performed based on the driver's posture while driving. [Solution] The gaze direction detection device detects the gaze direction of a driver based on a facial image captured by an imaging device 11 that captures a facial image of the driver of a vehicle 100. The gaze direction detection device includes a gaze direction detection unit 332 that detects the gaze direction of the driver based on the facial image captured by the imaging device, and a calibration unit 333 that calibrates the gaze direction of the driver based on the gaze direction of the driver detected by the gaze direction detection unit when a warning sound is output by a sound generation device 13 of the vehicle and a display accompanying the warning is displayed on display devices 14, 15 of the vehicle while the vehicle is traveling.

Description

[Technical Field]

[0001] The present disclosure relates to a gaze direction detection device. [Background technology]

[0002] Conventionally, gaze direction detection devices have been known that detect the gaze direction of a driver based on an image of the driver's face and calibrate the gaze direction of the driver based on the gaze direction of the driver when a predetermined condition is met (Patent Documents 1 to 3).

[0003] In particular, the device described in Patent Document 1 displays a target for the driver to gaze at on a virtual screen in front of the driver using a head-up display, and performs calibration processing for gaze direction detection based on the gaze direction detected from a captured image acquired by a camera in this state. Such calibration processing is performed while the vehicle is stopped. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-183473 [Patent Document 2] Japanese Patent Publication No. 2022-152715 [Patent Document 3] Japanese Patent Application Laid-Open No. 2010-030361 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the driver's posture while driving the vehicle may differ from that while the vehicle is stopped. For this reason, the device described in Patent Document 1 may not necessarily perform calibration based on the driver's posture while driving the vehicle.

[0006] In view of the above problems, an object of the present disclosure is to perform calibration based on the driver's posture while driving. [Means for solving the problem]

[0007] The gist of the present disclosure is as follows.

[0008] (1) A gaze direction detection device that detects the gaze direction of a driver of a vehicle based on a facial image captured by an imaging device that captures a facial image of the driver, a gaze direction detection unit that detects the gaze direction of the driver based on the face image captured by the imaging device; and a calibration unit that calibrates the driver's gaze direction based on the driver's gaze direction detected by the gaze direction detection unit when a sound representing a warning is output by a sound generating device of the vehicle and a display accompanying the warning is displayed on a display device of the vehicle while the vehicle is traveling. [Effects of the Invention]

[0009] According to the present disclosure, calibration can be performed based on the driver's posture while driving. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic block diagram showing the configuration of a gaze direction detection system. [Figure 2] FIG. 2 is a schematic side view partially showing the interior of a vehicle equipped with the gaze direction detection system. [Figure 3] FIG. 3 is a flowchart showing the flow of the calibration process. [Figure 4] FIG. 4 is a flowchart showing the flow of the calibration process. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, the embodiments will be described in detail with reference to the drawings. In the following description, like components are designated by like reference numerals.

[0012] First, the configuration of a gaze direction detection system 1 including a gaze direction detection device according to one embodiment will be described with reference to Fig. 1 and Fig. 2. The gaze direction detection device detects the gaze direction of a driver based on a facial image captured by a driver monitor camera 11 that captures a facial image of the driver. Fig. 1 is a schematic block diagram showing the configuration of the gaze direction detection system 1. Fig. 2 is a schematic side view partially showing the interior of a vehicle 100 equipped with the gaze direction detection system 1.

[0013] The gaze direction detection system 1 is mounted on the vehicle 100 and detects the gaze direction of the driver of the vehicle 100. The gaze direction of the driver detected by the gaze direction detection system 1 is used, for example, to determine whether or not the driver is looking away while driving.

[0014] 1, in this embodiment, the gaze direction detection system 1 includes a driver monitor camera 11, a speed sensor 12, a speaker 13, a display 14, a mirror indicator 15, and an electronic control unit (hereinafter referred to as "ECU") 30. These elements are communicatively connected via, for example, an in-vehicle network 21.

[0015] The driver monitor camera 11 is an example of an imaging device that captures an image of the driver (especially a facial image including the driver's face). In this embodiment, as shown in FIG. 2, the driver monitor camera 11 is provided on the top of the steering column 101 and is positioned facing the driver. Note that the driver monitor camera 11 may be provided in a position other than the top of the steering column 101 as long as it can capture an image of the driver of the vehicle 100. For example, the driver monitor camera 11 may be provided in the rearview mirror, meter panel, meter hood, etc. of the vehicle 100. The driver monitor camera 11 outputs the captured images to the ECU 30 at predetermined intervals via the in-vehicle network 21.

[0016] The speed sensor 12 detects the speed of the vehicle 100. The speed of the vehicle 100 detected by the speed sensor 12 is input to the ECU 30 via the in-vehicle network 21.

[0017] The speaker 13 is an example of a sound generating device that outputs a sound representing a warning. The speaker 13 receives a sound signal from the ECU 30 via the in-vehicle network 21 and outputs a sound in accordance with the sound signal. Note that the gaze direction detection system 1 may include another sound generating device, such as a buzzer, as long as it can output a sound representing a warning.

[0018] The display 14 is an example of a display device that displays a warning. The display 14 is a display device of any type, such as a liquid crystal display or an organic EL display. In this embodiment, the display 14 is disposed on the meter panel of the vehicle 100 as shown in FIG. 2 . However, the display 14 may be disposed anywhere so long as the driver can check the screen. The display 14 receives an image signal from the ECU 30 via the in-vehicle network 21 and displays an image in accordance with this image signal.

[0019] The mirror indicator 15 is an example of a display device that displays a warning. The mirror indicator 15 is provided, for example, in a rearview mirror or a side mirror. The mirror indicator 15 is embedded in the rearview mirror or the side mirror and is configured to light up, for example, when a warning to the driver is required. The mirror indicator 15 receives a signal from the ECU 30 via the in-vehicle network 21 and lights up in accordance with this signal.

[0020] In addition, the gaze direction detection system 1 may have other display devices that display warnings other than the display 14 and the mirror indicator 15, such as a head-up display that displays a virtual image on the front window of the vehicle 100, as long as the warning can be displayed.

[0021] The ECU 30 functions as a gaze direction detection device that detects the gaze direction of the driver. As shown in Fig. 1, the ECU 30 has a communication interface 31, a storage unit 32, and a processor 33. The communication interface 31 is a circuit for connecting the ECU 30 to the in-vehicle network 21.

[0022] The storage unit 32 stores data. The storage unit 32 includes, for example, at least one of a volatile semiconductor memory, a non-volatile semiconductor memory, a hard disk drive (HDD), and a solid state drive (SSD). The storage unit 32 stores computer programs executed by the processor 33 of the ECU 30. The storage unit 32 also stores data used in the computer programs executed by the processor 33, such as data transmitted from the driver monitor camera 11, etc.

[0023] The processor 33 has one or more central processing units (CPUs) and their peripheral circuits. The processor 33 may further have other arithmetic circuits such as a logic operation unit or a numerical operation unit. The processor 33 executes a computer program stored in the storage unit 32.

[0024] 1, the processor 33 includes a warning unit 331, a gaze direction detection unit 332, and a calibration unit 333. Each of these units included in the processor 33 is a functional module realized by a computer program running on the processor 33, for example.

[0025] The warning unit 331 issues a warning to the driver when it is necessary. For example, the warning unit 331 issues a warning to the driver to grip the steering wheel 102 when the driver is not gripping the steering wheel 102. The warning unit 331 also issues a warning when the vehicle 100 approaches a lane marking. Alternatively, the warning unit 331 issues a warning when another vehicle is traveling nearby in the lane to which the driver is about to change lanes. When it is necessary to issue a warning to the driver, the warning unit 331 causes the speaker 13 to output a sound representing the warning. The warning unit 331 also causes the display 14 and the mirror indicator 15 to display a message accompanying the warning. Specifically, the warning unit 331, for example, displays an image representing the warning on the display 14 or turns on the mirror indicator 15.

[0026] The gaze direction detection unit 332 detects the gaze direction of the driver based on the image captured by the driver monitor camera 11. In this embodiment, the gaze direction is represented by the horizontal or vertical angle between the longitudinal direction of the vehicle 100 and the line of sight of the driver. The gaze direction detection unit 332 inputs information about the gaze direction of the driver detected by the gaze direction detection unit 332 to the calibration unit 333. The gaze direction detection unit 332 also inputs the information about the gaze direction of the driver to, for example, an inattentiveness determination unit (not shown) that determines whether or not the driver is looking away.

[0027] The gaze direction detection unit 332 inputs, for example, an image acquired from the driver monitor camera 11 into a classifier that has been trained in advance to identify the positions of the pupil and the corneal reflection of the light source. The gaze direction detection unit 332 then detects the gaze direction based on the identified positional relationship between the pupil and the corneal reflection. As such a classifier, for example, a convolutional neural network (CNN) that receives a face image as input and outputs the positions of the pupil and the corneal reflection is used. Note that the gaze direction detection unit 332 may detect the driver's gaze direction using a method different from the above-mentioned method.

[0028] The calibration unit 333 calibrates the driver's gaze direction. Calibration is a process of matching the driver's gaze direction detected by the gaze direction detection unit 332 with the driver's actual gaze direction. Here, the gaze direction of the driver detected by the gaze direction detection unit 332 is detected differently for each driver and each driver's posture, and may not match the driver's actual gaze direction. Even if the detected gaze direction deviates from the actual gaze direction due to changes in the driver and the driver's posture in this way, calibration can be performed to match the deviated gaze direction.

[0029] In this embodiment, when the warning unit 331 outputs a sound indicating a warning from the speaker 13 and displays a warning on the display 14 or the mirror indicator 15 while the vehicle 100 is traveling (hereinafter simply referred to as "when a warning is issued by the warning unit 331"), the calibration unit 333 calibrates the driver's gaze direction based on the driver's gaze direction detected by the gaze direction detection unit 332.

[0030] In this embodiment, the calibration unit 333 determines whether the vehicle 100 is moving based on the output of the speed sensor 12, and determines that the vehicle 100 is moving, for example, when the speed of the vehicle 100 detected by the speed sensor 12 is not zero. However, the calibration unit 333 may determine whether the vehicle 100 is moving using other methods. For example, the calibration unit 333 may determine that the vehicle 100 is moving when the driving range of the vehicle 100 is in the drive range (D range) rather than the parking range (P range) or the like.

[0031] For example, when an image representing a warning is displayed on the display 14 by the warning unit 331 while the vehicle 100 is traveling, the calibration unit 333 performs calibration in relation to the position where the image is displayed. Specifically, when a warning is issued by the warning unit 331, the calibration unit 333 corrects the driver's gaze direction detected by the gaze direction detection unit 332 so that the driver's gaze direction detected by the gaze direction detection unit 332 matches the direction of the position where the image representing the warning is displayed relative to the driver's eyes. Since it is considered that the driver is looking at the image representing the warning when it is displayed on the display 14, performing such calibration makes it possible to match the driver's gaze direction detected by the gaze direction detection unit 332 with the actual gaze direction. In particular, in this embodiment, calibration can be performed while the vehicle 100 is traveling, not while the vehicle 100 is stopped. Therefore, even if the driver's posture changes between when the vehicle 100 is stopped and when the vehicle 100 is traveling, the driver's gaze direction can be calibrated based on the driver's posture while the vehicle 100 is traveling.

[0032] The posture detection unit 334 detects the posture of the driver based on the image captured by the driver monitor camera 11. For example, the posture detection unit 334 detects the posture of the driver by inputting the image acquired from the driver monitor camera 11 to a classifier that has been trained in advance to identify the posture of the driver. As such a classifier, for example, a convolutional neural network (CNN) that receives an image of the driver as input and outputs the posture of the driver is used.

[0033] 3 is a flowchart showing the flow of a calibration process for calibrating the driver's line of sight. The illustrated calibration process is executed by the processor 33 of the ECU 30. In particular, the example shown in FIG. 3 shows a case where the warning unit 331 causes the display 14 to display a warning when it is necessary to warn the driver.

[0034] When the calibration process is started, first, the calibration unit 333 determines whether or not the vehicle 100 is moving based on the output of the speed sensor 12 (step S11). If it is determined in step S11 that the vehicle 100 is not moving, the calibration process is terminated.

[0035] On the other hand, if it is determined in step S11 that the vehicle 100 is traveling, the calibration unit 333 determines whether or not the warning unit 331 has caused the speaker 13 to output a sound representing a warning and caused the display 14 to display a display accompanying the warning (hereinafter also referred to as a "warning display"), that is, whether or not a warning has been issued by the warning unit 331. If it is determined in step S12 that the warning unit 331 has not issued a warning, the calibration process is terminated.

[0036] On the other hand, if it is determined in step S12 that the warning unit 331 has issued a warning, the calibration unit 333 determines whether the gaze direction detected by the gaze direction detection unit 332 is staying around the direction of the warning display on the display 14 (step S13). Here, even if there is an error in the driver's gaze direction detected by the gaze direction detection unit 332 compared to the actual gaze direction, the general direction is the same. Therefore, if the driver is gazing at the warning display on the display 14, the gaze direction of the driver detected by the gaze direction detection unit 332 will be around the direction of the warning display on the display 14. Therefore, by determining whether the gaze direction detected by the gaze direction detection unit 332 is staying around the direction of the warning display on the display 14, it is possible to determine whether the driver is generally looking at the warning display. If it is determined in step S13 that the detected gaze direction is not staying around the direction of the warning display, the calibration process is terminated.

[0037] On the other hand, if it is determined in step S13 that the detected gaze direction is staying around the direction of the warning display, the calibration unit 333 calibrates the driver's gaze direction based on the gaze direction detected by the gaze direction detection unit 332 (step S14).

[0038] Fig. 4 is a flowchart showing the flow of calibration processing according to one modified example. In the example shown in Fig. 4, the driver's posture is detected by posture detection unit 334 before and after a warning is issued by warning unit 331, and calibration is performed only if there is no change in the driver's posture. This makes it possible to more reliably perform calibration based on the driver's line of sight while driving vehicle 100. Note that steps S21, S23, S24, and S27 in Fig. 4 are the same as steps S11 to S14 in Fig. 3, respectively, and therefore description thereof will be omitted.

[0039] If it is determined in step S21 that the vehicle 100 is traveling, the posture detection unit 334 detects the posture of the driver (step S22). If it is determined in step S24 that the detected gaze direction is staying around the direction of the warning display, the posture detection unit 334 detects the posture of the driver again (step S25). Then, the calibration unit 333 determines whether the driving posture of the driver detected in step S25 has changed from the driving posture of the driver detected in step S22 (step S26). If it is determined in step S26 that the driving posture has changed, the calibration process is terminated. On the other hand, if it is determined in step S26 that the driving posture has not changed, the calibration unit 333 calibrates the direction of the driver's gaze (step S27).

[0040] Although preferred embodiments according to the present disclosure have been described above, the present disclosure is not limited to these embodiments, and various modifications and changes can be made within the scope of the claims. [Explanation of symbols]

[0041] 1...Gaze direction detection system, 11...Driver monitor camera, 13...Speaker, 14...Display, 15...Mirror indicator, 21...In-vehicle network, 30...ECU

Claims

[Claim 1] A gaze direction detection device that detects the gaze direction of a driver of a vehicle based on a facial image captured by an imaging device that captures a facial image of the driver, a gaze direction detection unit that detects the gaze direction of the driver based on the face image captured by the imaging device; and a calibration unit that calibrates the driver's gaze direction based on the driver's gaze direction detected by the gaze direction detection unit when a sound representing a warning is output by a sound generating device of the vehicle and a display accompanying the warning is displayed on a display device of the vehicle while the vehicle is traveling.

Images