Driver behavior determination device, driver behavior determination method, and computer program for driver behavior determination

Abstract

To provide a driver behavior determination device capable of determining whether or not a driver is operating a portable device even if the portable device is not appearing in an image of the inside of a cabin.SOLUTION: A driver behavior determination device comprises: a sight line direction detection unit 31 for detecting a sight line direction of a driver of a vehicle 10 based on an image captured by an imaging unit 3 which is provided so as to image the inside of a cabin of the vehicle 10; and an operation determination unit 33 for calculating an angle formed from a normal direction of an operation screen of a portable device 11 represented in information received from the portable device 11 via a communication unit 2, which is communicable with the portable device 11 positioned inside of the vehicle 10, and the sight line direction and, in a case where the angle is included in a predetermined angle range, determining that the driver is operating the portable device 11.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a driver behavior determination device, a driver behavior determination method, and a computer program for driver behavior determination, which determine a predetermined behavior of a driver of a vehicle. [Background technology]

[0002] Technologies are being researched that monitor the driver of a vehicle and detect inappropriate behavior of the driver while driving the vehicle. In particular, since it is very dangerous if the driver operates a mobile device while driving and becomes distracted from the vehicle's surroundings, technologies have been proposed that detect the driver's operation of a mobile device while driving (for example, see Patent Documents 1 to 3).

[0003] In the technology disclosed in Patent Document 1, it is determined whether or not a driver is gazing at a display unit of a portable terminal device based on image data captured by a camera equipped in the portable terminal device. In addition, in the technology disclosed in Patent Document 2, it is determined whether or not a user of the portable device is a driver by analyzing an image captured by a camera installed in the portable device.

[0004] In addition, the technology disclosed in Patent Document 3 performs image analysis on the image captured by the driver's camera to determine whether the driver is engaging in smartphone driving, i.e., driving while operating a smartphone. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2016-6628 A [Patent Document 2] JP 2018-166268 A [Patent Document 3] JP 2019-74944 A Summary of the Invention [Problem to be solved by the invention]

[0006] In the technology disclosed in Patent Document 1 and the technology disclosed in Patent Document 2, the mobile device is required to always capture the driver, which increases the power consumption of the mobile device. In addition, in the technology disclosed in Patent Document 3, if the mobile device is not captured in the image captured by the vehicle-mounted camera, it is not possible to determine whether the driver is operating the mobile device. However, the capture range of the vehicle-mounted camera is limited, or depending on the installation position of the camera, the steering wheel or the like may create a blind spot in part of the capture range. Therefore, in some cases, the mobile device may not be captured in the image generated by the camera.

[0007] Therefore, an object of the present invention is to provide a driver behavior determination device that can determine whether or not a driver is operating a mobile device even if the mobile device is not displayed in an image of the vehicle interior. [Means for solving the problem]

[0008] According to one embodiment, there is provided a driver behavior determination device, which includes a gaze direction detection unit that detects the gaze direction of a driver of the vehicle based on an image obtained by an imaging unit that is provided to capture an image of the interior of the vehicle, and an operation determination unit that calculates an angle between the gaze direction and a normal direction of an operation screen of a portable device represented in information received from the portable device via a communication unit that is capable of communicating with the portable device located in the vehicle, and determines that the driver is operating the portable device if the angle is within a predetermined angle range.

[0009] The driver behavior determination device preferably further includes a hand direction detection unit that detects the extension direction of one of the driver's hands based on the image. In this case, the operation determination unit preferably determines that the driver is operating the portable device only when the position where the extension direction of the hand and the line of sight intersect is included in a space in the vehicle cabin where the portable device can be operated.

[0010] In addition, in this driver behavior determination device, it is preferable that the operation determination unit does not determine that the driver is operating a mobile device when the line of sight direction is within a predetermined frontal range centered on the vehicle's traveling direction.

[0011] According to another embodiment, a driver behavior determination method is provided, which includes receiving information indicating a normal direction of an operation screen of a portable device located in a vehicle via a communication unit capable of communicating with the portable device, detecting a line-of-sight direction of a driver of the vehicle based on an image obtained by an imaging unit provided to capture an image of an interior of the vehicle, calculating an angle between the normal direction of the operation screen of the portable device and the line-of-sight direction, and determining that the driver is operating the portable device if the angle is within a predetermined angle range.

[0012] According to still another embodiment, a computer program for determining driver behavior is provided, the computer program for determining driver behavior includes instructions to cause a processor mounted on the vehicle to execute the following operations: receive information indicating a normal direction of an operation screen of a portable device located in a vehicle via a communication unit capable of communicating with the portable device, detect a line-of-sight direction of a driver of the vehicle based on an image obtained by an imaging unit provided to capture an image of an interior of the vehicle, calculate an angle between the normal direction of the operation screen of the portable device and the line-of-sight direction, and determine that the driver is operating the portable device if the angle is within a predetermined angle range. Effect of the Invention

[0013] The driver behavior determination device according to the present disclosure has an advantage of being able to determine whether or not the driver is operating a mobile device even if the mobile device is not displayed in an image of the vehicle interior. [Brief description of the drawings]

[0014] [Figure 1] 1 is a schematic configuration diagram of a driving assistance system including a driver behavior determination device. [Diagram 2]2 is a hardware configuration diagram of an ECU that is an example of a driver behavior determination device. [Diagram 3] FIG. 2 is a functional block diagram of a processor of an ECU related to a driver behavior determination process. [Figure 4] 1 is a diagram showing an example of the relationship between the driver's line of sight when the driver is operating a mobile device, the normal direction of the operation screen of the mobile device, and the extending direction of the driver's hand. [Diagram 5] 4 is an operational flowchart of a driver behavior determination process executed by a processor of an ECU of a vehicle. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, a driver behavior determination device, a driver behavior determination method executed by the driver behavior determination device, and a computer program for driver behavior determination will be described with reference to the drawings. The inventors have focused on the fact that when a driver operates a mobile device such as a smartphone or a tablet, the driver generally holds the mobile device so that the operation screen faces the driver in order to view the operation screen of the mobile device. Therefore, the driver behavior determination device receives information indicating the normal direction of the operation screen from the mobile device and determines whether or not the angle formed by the normal direction of the operation screen of the mobile device and the line of sight of the driver is included within a predetermined angle range. Then, when the angle is included within the predetermined angle range, the driver behavior determination device determines that the driver is operating the mobile device. This makes it possible to determine whether or not the driver is operating the mobile device even if the mobile device is not shown in an image generated by a camera installed in the vehicle cabin.

[0016] 1 is a schematic diagram of a driving assistance system including a driver behavior determination device. The driving assistance system 1 mounted on a vehicle 10 includes a communication device 2 capable of communicating with a mobile device 11 located in the cabin of the vehicle 10, a driver monitor camera 3, a notification device 4, and an electronic control unit (ECU) 5 which is an example of a driver behavior determination device. The communication device 2, the driver monitor camera 3, the notification device 4, and the ECU 5 are connected to each other so as to be able to communicate with each other via an in-vehicle network conforming to a communication standard such as a controller area network.

[0017] The mobile device 11 is a device that the driver can carry, and is, for example, a mobile phone such as a smartphone or a tablet. The mobile device 11 is provided with an operation screen configured with, for example, a touch panel, and the driver operates the mobile device 11 by touching the operation screen, causing the mobile device 11 to execute a predetermined operation. The mobile device 11 also has a built-in sensor for measuring the normal direction of the operation screen, such as a gyro sensor, and the sensor measures the normal direction of the operation screen at predetermined intervals.

[0018] Furthermore, the mobile device 11 has a communication device for transmitting or receiving a wireless signal conforming to a predetermined short-range wireless communication standard, and is configured to be able to communicate with the communication device 2 via the communication device. The predetermined short-range wireless communication standard can be, for example, Bluetooth (registered trademark). The mobile device 11 generates a wireless signal including information indicating the normal direction of the operation screen at predetermined intervals or each time the normal direction of the operation screen changes by a predetermined angle or more, and transmits the wireless signal to the communication device 2 via the communication device. The mobile device 11 may have a communication interface conforming to a bus standard such as Universal Serial Bus (USB) for connecting to the communication device 2 via a wire. In this case, the mobile device 11 may transmit information indicating the normal direction of the operation screen to the communication device 2 via a wire.

[0019] The communicator 2 is an example of a communication unit, and communicates with the mobile device 11. To that end, the communicator 2 is configured to transmit or receive a wireless signal conforming to the above-mentioned predetermined short-range wireless communication standard, for example. When the communicator 2 and the mobile device 11 are connected by wire, the communicator 2 may have a communication interface and a communication circuit conforming to a bus standard such as Universal Serial Bus (USB). Every time the communicator 2 receives information indicating the normal direction of the operation screen of the mobile device 11 from the mobile device 11, the communicator 2 passes the received information to the ECU 5.

[0020] The driver monitor camera 3 is an example of an imaging unit, and has a two-dimensional detector composed of an array of photoelectric conversion elements, such as CCD or C-MOS, that are sensitive to visible light or infrared light, and an imaging optical system that forms an image of an area to be photographed on the two-dimensional detector. The driver monitor camera 3 may further have a light source, such as an infrared LED, for illuminating the driver. The driver monitor camera 3 is attached, for example, on the instrument panel or in its vicinity, facing the driver so that the head of the driver seated on the driver's seat of the vehicle 10 is included in the area to be photographed, that is, so that the driver's head can be photographed. The driver monitor camera 3 photographs the driver at a predetermined photographing period (for example, 1 / 30 to 1 / 10 seconds) and generates an image in which the driver is photographed (hereinafter, referred to as a driver image). The driver image obtained by the driver monitor camera 3 may be a color image or a gray image. Every time the driver monitor camera 3 generates a driver image, it outputs the generated driver image to the ECU 5 via the in-vehicle network.

[0021] The notification device 4 is provided in the vehicle cabin of the vehicle 10 and provides a predetermined notification to the driver by light, sound, text display, or image display. To achieve this, the notification device 4 has at least one of a speaker, a light source, or a display device, for example. When the notification device 4 receives a notification indicating a warning to the driver from the ECU 5, it notifies the driver of the warning by sound from the speaker, by emitting or blinking the light source, or by displaying a warning message on the display device.

[0022] The ECU 5 determines the behavior of the driver. In this embodiment, the ECU 5 determines whether the driver is operating the mobile device 11, and when it is determined that the driver is operating the mobile device 11, the ECU 5 controls the vehicle 10 to warn the driver or to make an emergency stop of the vehicle 10.

[0023] Fig. 2 is a hardware configuration diagram of the ECU 5. As shown in Fig. 2, the ECU 5 has a communication interface 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 may each be configured as a separate circuit, or may be configured integrally as a single integrated circuit.

[0024] The communication interface 21 has an interface circuit for connecting the ECU 5 to the in-vehicle network. Every time the communication interface 21 receives information indicating the normal direction of the operation screen of the mobile device 11 from the communicator 2, the communication interface 21 passes the information to the processor 23. Every time the communication interface 21 receives a driver image from the driver monitor camera 3, the communication interface 21 passes the received driver image to the processor 23. Furthermore, when the communication interface 21 receives information to be notified to the driver via the notification device 4, such as a notification indicating a warning to the driver, from the processor 23, the communication interface 21 outputs the information to the notification device 4.

[0025] The memory 22 is an example of a storage unit, and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores various algorithms and various data used in the driver behavior determination process executed by the processor 23 of the ECU 5. For example, the memory 22 stores various parameters used for detecting the line of sight direction and determining whether the driver is operating a mobile device. Furthermore, the memory 22 temporarily stores a driver image and various data generated during the driver behavior determination process.

[0026] The processor 23 includes one or more central processing units (CPUs) and their peripheral circuits. The processor 23 may further include other arithmetic circuits such as a logic arithmetic unit, a numerical arithmetic unit, or a graphic processing unit. The processor 23 executes the driver behavior determination process.

[0027] 3 is a functional block diagram of the processor 23 related to the driver behavior determination process. The processor 23 has a gaze direction detection unit 31, a hand direction detection unit 32, an operation determination unit 33, and a warning processing unit 34. Each of these units in the processor 23 is, for example, a functional module realized by a computer program running on the processor 23. Alternatively, each of these units in the processor 23 may be a dedicated arithmetic circuit provided in the processor 23.

[0028] The gaze direction detection unit 31 detects the gaze direction of the driver based on the driver image every time the processor 23 receives the driver image. To this end, the gaze direction detection unit 31 detects an area in the driver image in which the driver's face appears (hereinafter referred to as a face area) by inputting the driver image to a classifier that has been trained in advance to detect the driver's face from the image, for example. The gaze direction detection unit 31 can use, as such a classifier, for example, a deep neural network (DNN) having a convolutional neural network type (CNN) architecture or an AdaBoost classifier.

[0029] The gaze direction detection unit 31 further detects an area (hereinafter referred to as an eye area) in which the driver's eyes are represented from the face area of ​​the driver image. In this case, the gaze direction detection unit 31 detects edge pixels in the face area by applying an edge detection filter such as a Sobel filter. The gaze direction detection unit 31 then detects a line in which edge pixels continue in a substantially horizontal direction, and detects two such lines aligned vertically within a range in which the eye is assumed to be located in the face area for each of the left and right eyes as the upper and lower eyelids of the eye. The gaze direction detection unit 31 may detect the upper and lower eyelids of each of the left and right eyes of the driver from the driver image according to another method for detecting the upper and lower eyelids of the eye from an image. When the upper and lower eyelids of each of the left and right eyes of the driver are detected from the driver image, the gaze direction detection unit 31 detects the area surrounded by the upper eyelid and the lower eyelid for at least one of the left and right eyes of the driver as the eye area.

[0030] When the eye region is detected, the gaze direction detection unit 31 detects the corneal reflection image (also called Purkinje image) of the light source and the center of the pupil (hereinafter simply called the pupil center of gravity) from the eye region. In this case, the gaze direction detection unit 31 detects the Purkinje image by, for example, template matching between a template of the Purkinje image and the eye region. Similarly, the gaze direction detection unit 31 may detect the pupil by template matching between a template of the pupil and the eye region, and may set the center of gravity of the region in which the detected pupil is represented as the pupil center of gravity. Note that the gaze direction detection unit 31 may detect the Purkinje image and the pupil center of gravity according to another method for detecting the Purkinje image and the pupil center of gravity from the eye region. Then, the gaze direction detection unit 31 calculates the distance between the Purkinje image and the pupil center of gravity, and detects the driver's gaze direction by referring to a table showing the relationship between the distance and the driver's gaze direction. Note that such a table may be stored in the memory 22 in advance. Furthermore, when the gaze direction detection unit 31 detects the gaze direction for each of the left and right eyes, it may take the average value of the gaze directions for each of the left and right eyes as the gaze direction of the driver. Furthermore, the gaze direction detection unit 31 estimates the position of the driver's eyes in real space based on the direction from the driver monitor camera 3 to the position of the driver's eyes and the distance from the driver monitor 3 to the driver's eyes. Note that the gaze direction detection unit 31 may obtain the direction from the driver monitor camera 3 to the position of the eyes from the position of the eyes on the driver image. Furthermore, the gaze direction detection unit 31 may estimate the distance from the driver monitor 3 to the driver's eyes from information indicating the position of the driver's seat acquired from the driver's seat.

[0031] Every time the gaze direction detection unit 31 detects the gaze direction of the driver, it notifies the operation determination unit 33 of the detected gaze direction and the position of the driver's eyes.

[0032] Each time the processor 23 receives a driver image, the hand direction detection unit 32 detects the extension direction of the driver's hand based on the driver image. The extension direction of the hand here means the extension direction of the part from the elbow to the forearm. For example, the hand direction detection unit 32 can estimate the extension direction of the hand in real space by detecting the posture of the driver shown in the driver image according to a method such as OpenPose that detects the posture of a person shown in an image. The hand direction detection unit 32 can also obtain the direction from the driver monitor camera 3 to the elbow based on the position of the elbow on the driver image. Furthermore, the hand direction detection unit 32 can estimate the distance from the driver monitor camera 3 to the elbow based on the distance from the driver monitor camera 3 to the driver seat, the estimated value of the length of the driver's upper arm, and the extension direction of the upper arm estimated from the driver's posture. As described above, the distance from the driver monitor camera 3 to the driver seat is estimated based on the position of the driver seat. The estimated value of the length of the driver's upper arm may be stored in advance in the memory 22. The hand direction detection unit 32 can calculate a straight line representing the extension direction of the driver's hand in real space based on the extension direction of the driver's forearm, the estimated distance from the driver monitor camera 3 to the driver's elbow, and the direction from the driver monitor camera 3 to the elbow.

[0033] Every time the hand direction detection unit 32 detects the extending direction of each of the driver's left and right hands, it notifies the operation determination unit 33 of a straight line representing the extending direction of the detected hands. Note that, when the hand direction detection unit 32 can detect only the extending direction of one of the driver's left or right hands because most of one hand and arm are not shown on the driver image, it is sufficient to notify the operation determination unit 33 of a straight line representing the extending direction of the detected hand.

[0034] The operation determination unit 33 determines whether the driver is operating the mobile device 11 based on the line of sight of the driver, the position of the eyes, the normal direction of the operation screen of the mobile device 11, and a straight line representing the extension direction of the driver's hand.

[0035] FIG. 4 is a diagram showing an example of the relationship between the driver's line of sight when the driver is operating the mobile device 11, the normal direction of the operation screen of the mobile device 11, and the extending direction of the driver's hand.

[0036] As described above, generally, when the driver 400 operates the portable device 11, the driver 400 looks at the operation screen of the portable device 11. Therefore, the operation screen of the portable device 11 and the driver 400 face each other directly, so that the angle θ between the line of sight 401 of the driver 400 and the normal direction 402 of the operation screen of the portable device 11 falls within a certain angle range. On the other hand, the portable device 11 may be located outside the shooting range 410 of the driver monitor camera 3.

[0037] Furthermore, when the driver 400 is operating the portable device 11, it is assumed that the portable device 11 is located ahead of the extending direction 403 of the driver 400's hand. Furthermore, it is assumed that the portable device 11 is located in a space 411 (hereinafter, sometimes simply referred to as an operable section) in the vehicle interior where the driver 400 can operate the portable device 11. That is, it is impossible for the portable device 11 to be located outside the vehicle interior, or to be located in a place in the vehicle interior where other devices such as the steering wheel are present.

[0038] From this, the operation determination unit 33 calculates the angle between the driver's line of sight and the normal direction of the operation screen of the mobile device 11, and determines whether the angle is within a predetermined angle range (for example, 5° or less, 10° or less, or 20° or less). Furthermore, the operation determination unit 33 calculates the position where the driver's line of sight and the extension direction of the driver's left and right hands intersect (hereinafter, may be simply called the intersection position) based on the straight lines representing the driver's eye position, line of sight, and the extension direction of the left and right hands. At this time, if the shortest distance between the line of sight and the extension direction of the hands is equal to or less than a predetermined distance threshold, the operation determination unit 33 determines that the line of sight and the extension direction of the hands intersect, and the position on the straight line representing the extension direction of the hands when they are shortest may be set as the intersection position. Furthermore, if the shortest distance between the line of sight and the extension direction of the hands is farther away than the predetermined distance threshold, the operation determination unit 33 does not detect the intersection position. Then, when the angle between the line of sight of the driver and the normal direction of the operation screen of the mobile device 11 is included in a predetermined angle range and the intersection position is included in the operable space, the operation determination unit 33 determines that the driver is operating the mobile device 11. Note that information specifying the operable space may be stored in the memory 22 in advance.

[0039] When the operation determination unit 33 determines that the driver is operating the mobile device 11, it notifies the warning processing unit 34 of the determination result.

[0040] When the warning processing unit 34 receives from the operation determination unit 33 the determination result that the driver is operating the portable device 11, the warning processing unit 34 executes a predetermined warning process. For example, the warning processing unit 34 causes a speaker of the notification device 4 to emit a voice signal or a warning sound requesting the driver to stop operating the portable device 11. Alternatively, the warning processing unit 34 causes a display of the notification device 4 to display a warning message requesting the driver to stop operating the portable device 11.

[0041] The warning processing unit 34 may continue to receive a determination result from the operation determination unit 33 that the driver is operating the mobile device 11 even after a predetermined period has elapsed since the warning processing unit 34 warned the driver to stop operating the mobile device 11 via the notification device 4. In such a case, the warning processing unit 34 may control each unit of the vehicle 10 to make an emergency stop of the vehicle 10.

[0042] 5 is an operational flowchart of the driver behavior determination process executed by the processor 23. The processor 23 may execute the driver behavior determination process in accordance with the following operational flowchart at predetermined intervals.

[0043] The line of sight direction detection unit 31 of the processor 23 detects the line of sight direction of the driver based on the driver image (step S101).

[0044] The operation determination unit 33 of the processor 23 calculates an angle θ between the driver's line of sight and the normal direction of the operation screen of the mobile device 11, which is included in the information from the mobile device 11 received via the communication device 2 (step S102). Then, the operation determination unit 33 determines whether the angle θ is within a predetermined angle range (step S103).

[0045] If the angle θ between the driver's line of sight and the normal direction of the operation screen of the mobile device 11 is within a predetermined angle range (step S103-Yes), the hand direction detection unit 32 of the processor 23 detects the extension direction of the driver's left and right hands (step S104). The operation determination unit 33 calculates the position where the extension direction of the driver's left and right hands intersects with the line of sight of the driver (step S105). Then, the operation determination unit 33 determines whether the position where the extension direction of either the driver's left or right hand intersects with the line of sight of the driver is included within the cabin of the vehicle 10 and in a space where the mobile device 11 can be operated (step S106).

[0046] When the crossing position of either the left or right hand of the driver is included in the operable space (step S106-Yes), the operation determination unit 33 determines that the driver is operating a mobile device (step S107). Then, the warning processing unit 34 of the processor 23 executes a predetermined warning process via the notification device 4 (step S108).

[0047] On the other hand, in step S103, if the angle θ between the driver's line of sight and the normal direction of the operation screen of the mobile device 11 is outside the predetermined angle range (step S103-No), the operation determination unit 33 determines that the driver is not operating the mobile device (step S109). Also, in step S106, if the crossing positions of the driver's left and right hands are outside the operable space (step S106-No), the operation determination unit 33 also determines that the driver is not operating the mobile device (step S109).

[0048] After step S108 or S109, the processor 23 ends the driver behavior determination process.

[0049] As described above, the driver behavior determination device determines whether the driver is operating a portable device based on whether the angle between the normal direction of the operation screen of the portable device and the line of sight of the driver is within a predetermined angle range. Therefore, even if the portable device is not displayed in the image generated by the driver monitor camera, the driver behavior determination device can determine whether the driver is operating a portable device. Furthermore, the driver behavior determination device determines that the driver is operating a portable device only when the intersection position between the line of sight of the driver and the extension direction of either the left or right hand of the driver is included in the space in the vehicle cabin where the driver can operate the portable device. This allows the driver behavior determination device to more accurately determine whether the driver is operating a portable device.

[0050] According to the modified example, the operation determination unit 33 may determine that the driver is operating the portable device 11 when the angle between the driver's line of sight and the normal direction of the operation screen of the portable device 11 is within a predetermined angle range, regardless of the direction in which the driver's hand is extended. In this case, the hand direction detection unit 32 and the processes of steps S104 to S106 in the operation flowchart shown in Fig. 5 may be omitted. Therefore, according to this modified example, the driver behavior determination device can determine whether the driver is operating the portable device with a smaller amount of calculation.

[0051] In the above embodiment or modification, the operation determination unit 33 may not determine that the driver is operating the mobile device 11 when the driver's line of sight direction is included within a predetermined front range centered on the traveling direction of the vehicle 10. Information representing the predetermined front range may be stored in advance in the memory 22. According to this modification, the driver behavior determination device can prevent erroneous determination that the driver is operating the mobile device even though the driver is gazing at the traveling direction of the vehicle 10 because the normal direction of the operation screen of the mobile device is approximately parallel to the traveling direction of the vehicle.

[0052] According to another modification, the gaze direction detection unit 31 may detect the driver's gaze direction as well as the driver's facial direction from the driver image. Then, the operation determination unit 33 may determine that the driver is operating the mobile device when the angle between the driver's gaze direction and the normal direction of the operation screen of the mobile device 11 falls within a predetermined angle range.

[0053] In this case, the gaze direction detection unit 31 detects a plurality of feature points of the driver's face, such as the corners of the eyes, the inner corners of the eyes, the tip of the nose, and the corners of the mouth, from the face area of ​​the driver image. In this case, the gaze direction detection unit 31 detects the feature points of the face by, for example, inputting the face area to a classifier that has been trained in advance to detect the feature points of the face shown in the image. As such a classifier, the gaze direction detection unit 31 may use, for example, a NN with a CNN type architecture, a support vector machine, or an AdaBoost classifier. Alternatively, the gaze direction detection unit 31 may detect the feature points of the driver's face from the face area according to template matching between a template representing the feature points of the face and the face area, or another method for detecting the feature points of the face. The gaze direction detection unit 31 fits the detected feature points of the face to a three-dimensional face model that represents the three-dimensional shape of the face. Then, the gaze direction detection unit 31 may detect the face direction of the three-dimensional face model when each feature point fits best to the three-dimensional face model as the face direction of the driver. The gaze direction detection unit 31 may detect the direction of the driver's face based on the driver image in accordance with another method for determining the direction of the face depicted in the image.

[0054] According to this modified example, the driver behavior judgment device determines whether or not the driver is operating a mobile device by taking into account not only the driver's line of sight but also the direction of the driver's face, thereby making it possible to more accurately determine whether or not the driver is operating a mobile device.

[0055] According to yet another modified example, the warning processing unit 34 may not execute the warning process if the vehicle 10 is stopped even if it receives a determination result that the driver is operating the portable device 11. The warning processing unit 34 may receive a measurement value of the vehicle speed of the vehicle 10 from a vehicle speed sensor (not shown) provided in the vehicle 10, and determine whether the vehicle 10 is stopped based on the measurement value.

[0056] In addition, a computer program for realizing the functions of the processor 23 of the ECU 5 according to the above embodiment or modified example may be provided in a form recorded on a computer-readable portable recording medium, such as a semiconductor memory, a magnetic recording medium, or an optical recording medium.

[0057] As described above, those skilled in the art can make various modifications to the embodiments within the scope of the present invention. [Explanation of symbols]

[0058] 1. Driving assistance system 2 Communication device 3 Driver monitor camera 4 Notification device 5 Electronic control unit (driver behavior judgment device) 10 Vehicles 11 Mobile devices 21 Communication Interface 22 Memory 23 Processors 31 Gaze direction detection unit 32 Hand direction detection unit 33 Operation judgment section 34 Warning Processing Section

Claims

1. a gaze direction detection unit that detects a gaze direction of a driver of the vehicle based on an image obtained by an imaging unit provided to capture an image of the interior of the vehicle; an operation determination unit that calculates an angle between a normal direction of an operation screen of the portable device represented in information received from the portable device via a communication unit capable of communicating with the portable device located within the vehicle and the line of sight, and determines that the driver is operating the portable device when the angle is within a predetermined angle range; A driver behavior determination device having the above configuration.

2. A hand direction detection unit detects an extending direction of a hand of the driver based on the image, 2. The driver behavior determination device according to claim 1, wherein the operation determination unit determines that the driver is operating the portable device only when a position where the extension direction of the hand and the line of sight intersect is included in a space in the passenger compartment of the vehicle in which the portable device can be operated.

3. the operation determination unit does not determine that the driver is operating the mobile device when the line of sight direction is included within a predetermined front range centered on the traveling direction of the vehicle. The driver behavior determination device according to claim 1 or 2.

4. receiving information indicating a normal direction of an operation screen of a portable device located within a vehicle via a communication unit capable of communicating with the portable device; Detecting a line of sight direction of a driver of the vehicle based on an image obtained by an imaging unit provided to capture an image of an interior of the vehicle; Calculating an angle between a normal direction of an operation screen of the portable device and the line of sight direction; If the angle is within a predetermined angle range, it is determined that the driver is operating the portable device. A driver behavior determination method comprising:

5. receiving information indicating a normal direction of an operation screen of a portable device located within a vehicle via a communication unit capable of communicating with the portable device; Detecting a line of sight direction of a driver of the vehicle based on an image obtained by an imaging unit provided to capture an image of an interior of the vehicle; Calculating an angle between a normal direction of an operation screen of the portable device and the line of sight direction; If the angle is within a predetermined angle range, it is determined that the driver is operating the portable device. A computer program for determining driver behavior, which causes a processor mounted on the vehicle to execute the above steps.

Images