Inattentive driving determination device, inattentive driving determination system, and inattentive driving determination method

Abstract

To enable detection of inattentive driving by a driver due to gazing at a portable terminal without connecting a vehicle with the portable terminal.SOLUTION: An inattentive driving determination device that determines whether or not a driver of a vehicle 100 is performing inattentive driving, comprises: a line-of-sight direction detection unit 331 that detects a direction of the driver's line of sight; a terminal position detection unit 332 that detects a position of a portable terminal present in a cabin of the vehicle on the basis of an image acquired from a cabin camera 12 for imaging the interior of the cabin of the vehicle; an inattentive driving determination unit 334 that determines whether or not the driver is in an inattentive driving state on the basis of whether or not the line-of-sight direction of the driver is within an inattentive driving determination region; and a region setting unit 333 that, when the portable terminal is present in the cabin of the vehicle, sets a region where the portable terminal is positioned as an inattentive driving determination region.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to a peeking determination device, a peeking determination system, and a peeking determination method.

Background Art

[0002] Conventionally, there has been known a driver monitoring system that issues a warning to a driver when the driver operates a mobile terminal during vehicle driving (such as Patent Document 1). In particular, the system described in Patent Document 1 acquires an image captured by a camera of a mobile terminal via communication between the mobile terminal and the vehicle, and issues a warning when it is determined based on such an image that the driver is looking at the mobile terminal.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When determining whether a driver is looking at a mobile terminal using an image captured by a camera of the mobile terminal, it is necessary to connect the mobile terminal owned by the driver and the vehicle so that they can communicate. In this case, it is necessary to connect the mobile terminal and the vehicle every time the driver gets in and out or changes, which is troublesome. Also, when the driver does not perform the connection operation or when the mobile terminal does not want to support the connection to the vehicle, it is impossible to determine whether the driver is looking at the mobile terminal.

[0005] In view of the above problems, an object of the present disclosure is to be able to detect peeking by a driver who is staring at a mobile terminal without connecting the vehicle and the mobile terminal.

Means for Solving the Problems

[0006] The gist of the present disclosure is as follows.

[0007] (1) A glance determination device that determines whether or not a driver of a vehicle is glancing aside, a line-of-sight direction detection unit that detects the line-of-sight direction of the driver, a terminal position detection unit that detects the position of a mobile terminal in the vehicle interior based on an image obtained from an in-vehicle camera that images the vehicle interior, a glance determination unit that determines whether or not the driver is in a glancing state based on whether or not the line-of-sight direction of the driver is within a glance determination region, a region setting unit that sets, as a glance determination region, a region where the mobile terminal is located when there is a mobile terminal in the vehicle interior, A glance determination device comprising the above. (2) The glance determination device according to (1) above, wherein the region setting unit sets, as a glance determination region, a region preset regardless of the position of the mobile terminal in addition to the region where the mobile terminal is located. (3) A glance determination system comprising the glance determination device according to (1) or (2) above, a driver monitor camera that images an image of the driver's face, an in-vehicle camera that images the vehicle interior, and the line-of-sight direction detection unit detects the line-of-sight direction of the driver based on an image imaged by the driver monitor camera, the in-vehicle camera is arranged to image a region in front of the driver. A glance determination system. (4) The glance determination system according to (3) above, wherein the in-vehicle camera is arranged to image downward and forward from the ceiling of the vehicle. (5) A glance determination method for determining whether or not a driver of a vehicle is glancing aside, detecting the line-of-sight direction of the driver, detecting the position of a mobile terminal in the vehicle interior based on an image obtained from an in-vehicle camera that images the vehicle interior, Determining whether the driver is in a side-glancing state based on whether the line-of-sight direction of the driver is within a side-glancing determination area; When there is a mobile terminal in the vehicle interior, setting an area where the mobile terminal is located as a side-glancing determination area; A side-glancing determination method including the above.

Advantages of the Invention

[0008] According to the present disclosure, it becomes possible to detect side-glancing of a driver by gazing at a mobile terminal without connecting the vehicle and the mobile terminal.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments will be described in detail with reference to the drawings. In the following description, the same reference numerals are assigned to similar components.

[0011] Referring to FIGS. 1 to 3, the configuration of a sidelong glance determination system 1 including a sidelong glance determination device according to an embodiment will be described. FIG. 1 is a schematic block diagram showing the configuration of the sidelong glance determination system 1. FIG. 2 is a schematic side view partially showing the inside of a vehicle 100 equipped with the sidelong glance determination system 1.

[0012] The sidelong glance determination system 1 is mounted on the vehicle 100 and determines whether the driver of the vehicle 100 is making a sidelong glance. As shown in FIG. 2, the vehicle 100 includes a steering wheel 102 attached via a steering column 101, a front window 103 disposed in front of the driver, and a driver's seat 104 on which the driver sits.

[0013] As shown in FIG. 1, in the present embodiment, the sidelong glance determination system 1 includes a driver monitor camera 11, an in-vehicle camera 12, a human machine interface (HMI) 13, and an electronic control unit (hereinafter referred to as "ECU") 30. The driver monitor camera 11, the in-vehicle camera 12, the HMI 13, and the ECU 30 are communicably connected via, for example, an in-vehicle network 21. The in-vehicle network 21 is a network compliant with a standard such as CAN (Controller Area Network), for example.

[0014] The driver monitor camera 11 captures an image of the driver's face. In the present embodiment, the driver monitor camera 11 is provided above the steering column 101 and is disposed facing the driver so as to be able to capture the driver, specifically, a part of the driver's face and upper body. Note that the driver monitor camera 11 may be provided at a position different from above the steering column 101 as long as it can capture the driver of the vehicle 100. For example, the driver monitor camera 11 may be provided on the steering wheel 102, the rearview mirror, the meter panel, the meter hood, etc. of the vehicle 100.

[0015] The driver monitoring camera 11 includes a camera and a projector. For example, the camera is a CMOS (Complementary Metal Oxide Semiconductor) camera or a CCD (Charge Coupled Device) camera, and the projector is an LED (Light Emitting Diode). Also, in order to be able to image the driver's face without causing discomfort to the driver even in low illuminance situations such as at night, the projector is preferably a near-infrared LED, and it is also preferable that the camera can detect near-infrared rays. For example, the projector is two near-infrared LEDs arranged on both sides of the camera. Also, a filter such as a visible light cut filter may be provided in the camera. The driver monitoring camera 11 outputs the images captured at predetermined intervals to the ECU 30 via the in-vehicle network 21.

[0016] The in-vehicle camera 12 images the interior of the vehicle 100. In this embodiment, the in-vehicle camera 12 is provided on the ceiling of the vehicle 100 and is arranged to image downward in the front direction of the vehicle 100. In particular, the in-vehicle camera 12 is arranged above the seat 104 on which the driver is seated or above the rear side of this seat 104. As a result, the in-vehicle camera 12 is arranged to image the area located in front of the driver. For example, the in-vehicle camera 12 is arranged above the rear side of the driver's seat 104 and is arranged to image from behind the driver toward the front side of the vehicle (toward the front window 103). Also, in this embodiment, the in-vehicle camera 12 is arranged such that the space located between the driver's face and the front window 103 of the vehicle 100 is within the angle of view. Note that as long as the in-vehicle camera 12 can image the area located in front of the driver, particularly as long as the space located between the driver's face and the front window 103 is within the angle of view, the in-vehicle camera 12 may be arranged anywhere inside the vehicle 100 and in any orientation.

[0017] The in-vehicle camera 12 also includes a camera and a projector, similar to the driver monitoring camera. In the in-vehicle camera 12 as well, the projector is preferably a near-infrared LED, and the camera preferably can detect near-infrared rays. The in-vehicle camera 12 also outputs, via the in-vehicle network 21, images captured at predetermined intervals to the ECU 30. Also, the in-vehicle camera 12 may be a monocular camera or a stereo camera. When a stereo camera is used as the in-vehicle camera 12, the in-vehicle camera 12 can measure the relative positions of objects inside the vehicle 100.

[0018] The HMI 13 is an interface for inputting and outputting information between the driver or passenger and the sidelong glance determination system 1. The HMI 13 includes an information providing device for providing various information to the driver or passenger and an input device for the driver or passenger to perform input operations.

[0019] Specifically, the HMI 13 includes, as the information providing device, a display 14 for displaying character information or image information. The display 14 is an example of a display device for displaying images. The display 14 is a display device of any type, such as a liquid crystal display or an organic EL display. The display 14 is arranged so that at least the driver can view its screen. Therefore, the display 14 is arranged, for example, on the instrument panel, meter panel, etc. of the vehicle 100. The display 14 receives an image signal from the ECU 30 via the in-vehicle network 21 and displays an image according to this image signal. Note that the vehicle 100 may include, as the information providing device, another type of display device, such as a head-up display, instead of or in addition to the display 14.

[0020] In addition, the HMI 13 has a speaker 15 as an information providing device. The speaker 15 is an example of a device that outputs sound. The speaker 15 receives an audio signal from the ECU 30 via the in-vehicle network 21 and outputs sound according to this audio signal. Note that the HMI 13 may include devices other than the display 14 and the speaker 15 (for example, a vibration device, etc.) that provide various information to the driver or passengers as an information providing device.

[0021] In addition, the HMI 13 has a touch panel 16 as an input device. The touch panel 16 is an example of a device that receives input when touched by the driver or a passenger. When the driver or passenger performs an operation by contact, the touch panel 16 outputs the operation signal to the ECU 30 via the in-vehicle network 21. Note that the HMI 13 may include devices other than the touch panel 16 (for example, buttons, switches, etc.) for the driver or passengers to perform input operations as an input device.

[0022] The ECU 30 functions as a distraction determination device that determines whether the driver of the vehicle 100 is distracted. The ECU 30 also controls the information provided by the information providing device of the HMI 13. Therefore, the ECU 30 controls the image displayed on the display 14 and the sound output from the speaker 15. As shown in FIG. 1, the ECU 30 includes a communication interface 31, a storage unit 32, and a processor 33.

[0023] The communication interface 31 is a circuit for connecting the ECU 30 to the in-vehicle network 21.

[0024] The storage unit 32 stores data. The storage unit 32 has, for example, at least one of a volatile semiconductor memory, a non-volatile semiconductor memory, a hard disk drive (HDD), and a solid state drive (SSD). The storage unit 32 stores a computer program executed by the processor 33 of the ECU 30. Further, the storage unit 32 stores data used in a computer program executed by the processor 33, such as data transmitted from the driver monitor camera 11 and the like.

[0025] The processor 33 has one or more CPUs (Central Processing Unit) and its peripheral circuits. The processor 33 may further have other arithmetic circuits such as a logical arithmetic unit or a numerical arithmetic unit. The processor 33 executes the computer program stored in the storage unit 32.

[0026] FIG. 3 is a functional block diagram of the processor 33 of the ECU 30. As shown in FIG. 3, the processor 33 includes a line-of-sight direction detection unit 331, a terminal position detection unit 332, a region setting unit 333, a side-glance determination unit 334, and a notification unit 335.

[0027] The line-of-sight direction detection unit 331 detects the driver's line-of-sight direction based on the image captured by the driver monitor camera 11. The line-of-sight direction is represented by the horizontal or vertical angle between the longitudinal direction of the vehicle 100 and the driver's line of sight. The line-of-sight direction detection unit 331 inputs the information on the driver's line-of-sight direction detected by the line-of-sight direction detection unit 331 to the side-glance determination unit 334.

[0028] Specifically, the line-of-sight direction detection unit 331 inputs the image acquired from the driver monitor camera 11 to a discriminator that has been pre-trained to identify the positions of the pupil and the corneal reflection of the light source. Then, the line-of-sight direction detection unit 331 detects the line-of-sight direction based on the positional relationship between the identified pupil and corneal reflection. As such a discriminator, for example, a convolutional neural network (CNN) that outputs the positions of the pupil and corneal reflection when a face image is input is used.

[0029] Alternatively, the line-of-sight direction detection unit 331 may input the image acquired from the driver monitoring camera 11 to an identifier that identifies the positions of the inner corners and the irises of the driver's eyes. In this case, the line-of-sight direction detection unit 331 detects the line-of-sight direction based on the positional relationship between the identified inner corners and the irises.

[0030] The terminal position detection unit 332 detects the presence or absence of a mobile terminal in the interior of the vehicle 100 and the position of the mobile terminal in the interior of the vehicle 100 based on the image acquired from the in-vehicle camera 12. In particular, in the present embodiment, when a stereo camera is used as the in-vehicle camera 12, the terminal position detection unit 332 detects the relative position with respect to the position of the driver's eyes. The terminal position detection unit 332 inputs the position information of the mobile terminal detected by the terminal position detection unit 332 to the area setting unit 333.

[0031] Specifically, the terminal position detection unit 332 inputs the image acquired from the in-vehicle camera 12 to an identifier that has been pre-trained to identify the position of the mobile terminal. As such an identifier, for example, a convolutional neural network (CNN) that outputs the presence or absence of a mobile terminal in the image and, if there is a mobile terminal, its position when an image of the interior of the vehicle 100 captured by the in-vehicle camera 12 is input is used.

[0032] In addition, the terminal position detection unit 332 detects, as the mobile terminal, a device that has a screen and can be carried by a user. For example, the mobile terminal includes a smartphone, a tablet, a portable game terminal, and the like.

[0033] The area setting unit 333 sets a side glance determination area. The side glance determination area is an area where the driver is determined to be in a side glance state when the driver's line of sight direction is within the side glance determination area. In the present embodiment, the side glance determination area is set as an area based on the position of the driver's face, particularly the position of the driver's eyes. Therefore, the side glance determination area is represented, for example, as a range of angles in the vertical direction (up-and-down direction) and the horizontal direction with respect to the position of the driver's eyes as a reference. The area setting unit 333 inputs the information on the side glance determination area set by the area setting unit 333 to the side glance determination unit 334.

[0034] FIG. 4 is a diagram showing the side glance determination area set by the area setting unit 333. FIG. 4 shows the area in front of the interior of the vehicle 100. In particular, FIG. 4(A) shows a state where there is no mobile terminal X in the area in front of the interior of the vehicle 100, and FIG. 4(B) shows a state where there is a mobile terminal X in the area in front of the interior of the vehicle 100.

[0035] During the operation of the vehicle 100, the driver needs to look ahead of the vehicle 100. Therefore, the state where the driver is looking ahead of the vehicle 100 is basically not a side glance state. On the other hand, during the operation of the vehicle 100, the driver does not need to look at areas other than the front of the vehicle 100. Therefore, the state where the driver is looking in a direction other than the front of the vehicle 100 is basically a side glance state.

[0036] Therefore, in this embodiment, as shown in FIG. 4(A), when there is no mobile terminal X in the interior of the vehicle 100, the region setting unit 333 basically does not set the region in front of the vehicle 100, particularly the region where the front window 103 is located, as the sidelong glance determination region. In other words, the region setting unit 333 sets the region where the front window 103 is located as a non-sidelong glance determination region (the lightly shaded region A surrounded by a broken line in the figure) that is not the sidelong glance determination region. On the other hand, when there is no mobile terminal X in the interior of the vehicle 100, the region setting unit 333 basically sets the region other than the region in front of the vehicle 100, that is, the region other than the region where the front window 103 is located, as the sidelong glance determination region (the darkly shaded region B surrounded by a solid line in the figure). In particular, in this embodiment, the region other than the region where the front window 103 is located is a region that is set as the sidelong glance determination region in advance regardless of the presence or absence of the mobile terminal. Hereinafter, such a region is also referred to as the basic region B1.

[0037] Note that in this embodiment, the entire region where the front window 103 is located is set as the non-sidelong glance determination region. However, only a part of the front window 103 (for example, the region located in front of the driver) may be set as the non-sidelong glance determination region. Alternatively, the region other than the region where the front window 103 is located may also be set as the non-sidelong glance determination region. Specifically, in addition to the region where the front window 103 is located, for example, the regions where the side mirror, rear mirror, and meter panel are located may also be set as the non-sidelong glance determination region. And the region other than the region set as the non-sidelong glance determination region in this way is set as the basic region B1.

[0038] In addition, in the present embodiment, the basic region B1 is a preset fixed region. However, the basic region B1 may be a region that changes according to the driving situation. For example, when the turn signal is not blinking, the region where the side mirror and the side window located on the side of the driver are located is included in the basic region B1. However, when the turn signal is blinking, the region where the side mirror or the side window on the side where the turn signal is blinking is located may be set not to be included in the basic region B1.

[0039] In addition, in the present embodiment, as shown in FIG. 4(B), when there is a mobile terminal X in the vehicle interior of the vehicle 100, the region setting unit 333 sets the region where the mobile terminal X is located (the darkly shaded region B2 surrounded by the solid line and the dashed-dotted line in FIG. 4(B)) as the side glance determination region. Therefore, the region setting unit 333 sets the region B2 where the mobile terminal X is located and the basic region B1 as the side glance determination region B. On the other hand, when there is a mobile terminal X in the vehicle interior of the vehicle 100, the region setting unit 333 sets the region in front of the vehicle 100, particularly the region where the front window 103 is located, excluding the region where the mobile terminal X is located, as the non-side glance determination region A.

[0040] Therefore, when there is a mobile terminal X in the vehicle interior of the vehicle 100, when the mobile terminal X is at least partially located in a region different from the basic region B1, the side glance determination region B is expanded compared to the normal time when there is no mobile terminal X in the vehicle interior of the vehicle 100. On the other hand, even when there is a mobile terminal X in the vehicle interior of the vehicle 100, when the entire mobile terminal X is located within the basic region B1, the side glance determination region B is set to the same region as the normal time when there is no mobile terminal X in the vehicle interior of the vehicle 100.

[0041] The sidelong glance determination unit 334 determines whether the driver is in a sidelong glance state based on whether the driver's line of sight direction is within the sidelong glance determination area B. Information regarding the driver's line of sight direction is acquired from the line of sight direction detection unit 331. Also, information regarding the sidelong glance determination area B is acquired from the area setting unit 333. In the present embodiment, the sidelong glance determination unit 334 determines that the driver is in a sidelong glance state when the line of sight direction detected by the line of sight direction detection unit 331 is within the sidelong glance determination area B for a preset predetermined reference time T (for example, several seconds). On the other hand, the sidelong glance determination unit 334 determines that the driver is not in a sidelong glance state when the line of sight direction of the driver detected by the line of sight direction detection unit 331 is within the sidelong glance determination area B but the duration thereof is less than the reference time T. Further, the sidelong glance determination unit 334 determines that the driver is not in a sidelong glance state when the line of sight direction of the driver detected by the line of sight direction detection unit 331 is not within the sidelong glance determination area B. The sidelong glance determination unit 334 inputs information regarding the determination result to the notification unit 335.

[0042] Note that in the present embodiment, the determination as to whether the driver is in a sidelong glance state is made based on the same reference time T in all areas within the sidelong glance determination area B. However, the reference time T may be set to different times for each area within the sidelong glance determination area B. For example, the side mirror, rearview mirror, meter panel, etc. may require temporary visual recognition by the driver even during the operation of the vehicle 100. Therefore, the reference time T in the area where the side mirror, rearview mirror, meter panel, etc. are located may be set to a longer time than the reference time T in the other areas within the sidelong glance determination area B (including the area where the mobile terminal X is located).

[0043] The notification unit 335 notifies the driver when the sidelong glance determination unit 334 determines that the driver is in a sidelong glance state. The notification unit 335 causes, for example, a display 14 to display that the driver is in a sidelong glance state. Alternatively, the notification unit 335 causes, for example, a speaker 15 to output a voice indicating that the driver is in a sidelong glance state.

[0044] FIG. 5 is a flowchart showing the flow of the area setting process for setting the side glance determination area. The area setting process shown in FIG. 5 is executed by the processor 33 at regular time intervals.

[0045] In the area setting process, first, the area setting unit 333 determines whether or not it is detected by the terminal position detection unit 332 that there is a mobile terminal in the vehicle 100 interior (step S11).

[0046] If it is determined in step S11 that a mobile terminal is detected in the vehicle 100 interior, the area setting unit 333 sets, in addition to the reference area B1, the area B2 where the mobile terminal is located as the side glance determination area (step S12). Therefore, in the example shown in FIG. 4, as shown in FIG. 4(B), the reference area B1 and the area B2 where the mobile terminal X is located are set as the side glance determination area B. On the other hand, if it is determined in step S11 that a mobile terminal is not detected in the vehicle 100 interior, the area setting unit 333 sets the reference area B1 as the side glance determination area (step S13). Therefore, in the example shown in FIG. 4, as shown in FIG. 4(A), the entire area where the front window 103 is located is set as the side glance determination area B.

[0047] FIG. 6 is a flowchart showing the flow of the side glance determination process for determining whether or not the driver is in a side glance state. The side glance determination process shown in FIG. 6 is executed by the processor 33 at regular time intervals.

[0048] In the side glance determination process, first, the line-of-sight direction detection unit 331 acquires an image that is transmitted from the driver monitor camera 11 to the ECU 30 and stored in the storage unit 32. Then, the line-of-sight direction detection unit 331 detects the line-of-sight direction of the driver based on the acquired image of the driver (step S21).

[0049] Next, the side glance determination unit 334 determines whether or not the driver's line-of-sight direction detected in step S21 is within the side glance determination area (step S22). The side glance determination unit 334 uses, as the side glance determination area, the area set by the area setting process shown in FIG. 5, that is, the area set in step S12 or S13 of FIG. 5.

[0050] If it is determined in step S22 that the driver's line-of-sight direction is not within the side glance determination area, the side glance determination process ends without notifying the driver. On the other hand, if it is determined in step S22 that the driver's line-of-sight direction is within the side glance determination area, the notification unit 335 notifies the driver that he / she is in a side glance state via the display 14 or the speaker 15.

[0051] In the above embodiment, the position of the mobile terminal is detected based on the image captured by the in-vehicle camera 12, and the side glance determination area is set based on the detected position of the mobile terminal. Therefore, according to the above embodiment, it is possible to determine whether or not the driver is in a side glance state of looking at the mobile terminal without connecting the ECU 30 of the vehicle 100 to the mobile terminal.

[0052] As described above, the preferred embodiments according to the present disclosure have been described. However, the present disclosure is not limited to these embodiments, and various modifications and changes can be made within the scope of the claims.

Explanation of Reference Numerals

[0053] 1 Glance Detection System 11 Driver Monitoring Camera 12 In-Vehicle Camera 30 ECU 33 Processor

Claims

1. A sidelong glance determination device for determining whether a driver of a vehicle is making a sidelong glance, comprising: a line-of-sight direction detection unit that detects the line-of-sight direction of the driver; a terminal position detection unit that detects the position of a mobile terminal in the vehicle interior based on an image acquired from an in-vehicle camera that images the vehicle interior; a sidelong glance determination unit that determines whether the driver is in a sidelong glance state based on whether the line-of-sight direction of the driver is within a sidelong glance determination region; a region setting unit that, when there is a mobile terminal in the vehicle interior, sets the region where the mobile terminal is located as the sidelong glance determination region. A sidelong glance determination device.

2. The sidelong glance determination device according to claim 1, wherein the region setting unit sets, in addition to the region where the mobile terminal is located, a region preset regardless of the position of the mobile terminal as the sidelong glance determination region.

3. A sidelong glance determination system comprising the sidelong glance determination device according to claim 1 or 2, comprising: a driver monitor camera that images an image of the driver's face; an in-vehicle camera that images the vehicle interior, wherein the line-of-sight direction detection unit detects the line-of-sight direction of the driver based on an image imaged by the driver monitor camera, and the in-vehicle camera is arranged to image a region located in front of the driver.

4. The sidelong glance determination system according to claim 3, wherein the in-vehicle camera is arranged to image downward and forward from the ceiling of the vehicle.

5. A sidelong glance determination method for determining whether a driver of a vehicle is making a sidelong glance, comprising: detecting the line-of-sight direction of the driver; detecting the position of a mobile terminal in the vehicle interior based on an image acquired from an in-vehicle camera that images the vehicle interior; determining whether the driver is in a sidelong glance state based on whether the line-of-sight direction of the driver is within a sidelong glance determination region; and when there is a mobile terminal in the vehicle interior, setting the region where the mobile terminal is located as the sidelong glance determination region. A sidelong glance determination method.

Images