Driver status determination device

Abstract

To restrict a time for driver state determination from becoming longer due to a position change of a steering equipped with a driver monitor camera.SOLUTION: A driver state determination device includes: a driver state determination section for determining a state of a driver based on a photographed image of a driver monitor camera; and a position change information acquisition section for acquiring position change information of a steering when a position change of the steering is performed. The driver state determination section is configured to: estimate a position of the face of the driver in the photographed image after the position change based on the position of the face of the driver in the photographed image of the driver monitor camera before the position change of the steering and the position change information of the steering when the position change of the steering is performed; and determine the state of the driver from the estimation result of the position of the face of the driver and the photographed image after the position change.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a driver state determination device.

Background Art

[0002] Conventionally, as a technical document related to a driver state determination device, Japanese Unexamined Patent Application Publication No. 2021-026626 is known. This publication shows that when detecting the posture movement of the driver during the automatic driving mode and not detecting the holding of the steering wheel by the driver, a request to hold the steering wheel for the driver is started.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, a driver monitor camera used to determine the state of the driver is often provided on the steering column. In this case, when the driver changes the position of the steering wheel, the position of the driver monitor camera also changes, so there is a problem that it takes time to determine the state of the driver.

Means for Solving the Problems

[0005] One aspect of the present invention is a driver state determination device for determining the facial state of a vehicle driver based on an image captured by a driver monitor camera mounted on the steering wheel of a vehicle, comprising: a driver state determination unit that determines the driver's state based on an image captured by the driver monitor camera; and a position change information acquisition unit that acquires steering wheel position change information when the steering wheel's position is changed, wherein when the steering wheel's position is changed, the driver state determination unit estimates the position of the driver's face in the image captured by the driver monitor camera before the steering wheel's position is changed, based on the position of the driver's face in the image captured by the driver monitor camera before the steering wheel's position is changed and the steering wheel position change information, and determines the driver's state from the estimated position of the driver's face and the image captured after the position change.

[0006] In a driver state determination device according to one aspect of the present invention, the driver state determination unit determines whether the driver's state is in a driver abnormal state based on steering position change information and images captured by a driver monitor camera, and when the steering position has been changed, it may be less likely to determine that the driver's state is in a driver abnormal state compared to when the steering position has not been changed. [Effects of the Invention]

[0007] According to one aspect of the present invention, it is possible to suppress the time required to determine the driver's state due to changes in the position of the steering wheel on which the driver monitoring camera is installed. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram of a driver state determination device according to one embodiment. [Figure 2] This diagram illustrates how the image capture range of the driver monitoring camera changes due to a change in the steering wheel position. [Figure 3] This flowchart shows an example of a process for determining the driver's state based on a change in the steering wheel position. [Figure 4]This flowchart shows an example of a process for changing the threshold for detecting abnormal conditions due to changes in steering wheel position. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings.

[0010] Figure 1 is a block diagram showing a driver state determination device 100 according to one embodiment. The driver state determination device 100 shown in Figure 1 is mounted on a vehicle and determines the state of the vehicle's driver. The driver state determination device 100 uses, for example, an image of the driver's face captured by a driver monitor camera 1 to determine whether or not the driver is distracted.

[0011] As shown in Figure 1, the driver status determination device 100 is equipped with an ECU (Electronic Control Unit) 10 that comprehensively manages the device. The ECU 10 is an electronic control unit having a CPU (Central Processing Unit) and a memory unit. The memory unit consists of, for example, ROM (Read Only Memory), RAM (Random Access Memory), and EEPROM (Electrically Erasable Programmable Read-Only Memory).

[0012] The ECU10 implements various functions, for example, by executing a program stored in the memory unit using the CPU. The ECU10 may be composed of multiple electronic units. The ECU10 is connected to the driver monitor camera 1, the steering position detection unit 2, and the HMI3 [Human Machine Interface].

[0013] The driver monitor camera 1 is a camera for imaging the driver. The driver monitor camera 1 can be a digital camera having an image sensor such as a CCD (Charge Coupled Device) or CIS (CMOS Image Sensor).

[0014] The driver monitoring camera 1 is mounted on the steering wheel of the vehicle. The installation method of the driver monitoring camera 1 is not limited as long as it is integrated with the steering wheel. For example, the driver monitoring camera 1 is mounted on the cover of the steering column and captures images of the driver's head at a predetermined frame rate. The driver monitoring camera 1 may also be integrated with the steering wheel. The driver monitoring camera 1 transmits the captured images of the driver to the ECU 10.

[0015] The steering position detection unit 2 detects the position of the vehicle's steering wheel. The steering position detection unit 2 includes, for example, a tilt sensor and a telescopic sensor. The tilt sensor is a sensor that detects the vertical position of the steering wheel. The telescopic sensor is a sensor that detects the horizontal position of the steering wheel. The steering position detection unit 2 detects the position of the steering wheel from the detection results of the tilt sensor and the telescopic sensor. The steering position detection unit 2 transmits steering position information regarding the position of the steering wheel to the ECU 10.

[0016] HMI3 is an interface for inputting and outputting information between ECU10 and the driver. HMI3 includes, for example, a display and speakers located inside the vehicle cabin. HMI3 outputs images from the display and audio from the speakers in response to control signals from ECU10. The display may be a MID (Multi-Information Display) or a HUD (Head-Up Display). HMI3 may also be equipped with various indicators.

[0017] Next, the functional configuration of the ECU 10 will be described. As shown in Figure 1, the ECU 10 includes a position change information acquisition unit 11, a driver recognition range determination unit 12, and a driver state determination unit 13.

[0018] When the position of the vehicle's steering is changed, the position change information acquisition unit 11 acquires the position change information of the steering. Based on the steering position information detected by the steering position detection unit 2, the position change information acquisition unit 11 determines whether the position of the vehicle's steering has been changed. For example, when the vertical or front-rear position of the steering is changed by a certain value or more, the position change information acquisition unit 11 determines that the position of the vehicle's steering has been changed.

[0019] When the position change information acquisition unit 11 determines that the position of the vehicle's steering has been changed, based on the steering position information detected by the steering position detection unit 2 before and after the determination, the position change information acquisition unit 11 acquires the position change information of the steering. The position change information includes, for example, the steering position information before the position change and the steering position information after the position change.

[0020] When it is determined by the position change information acquisition unit 11 that the position of the steering has been changed, the driver recognition range determination unit 12 determines whether the position of the steering has deviated from the driver recognition range due to the position change.

[0021] The driver recognition range is a preset range used to determine whether the position (orientation) of the driver monitoring camera 1 has become a position where it cannot image the driver's face due to a change in the position of the steering. In other words, when the position of the steering is within the driver recognition range, it can be inferred that the driver monitoring camera 1 is imaging the driver's face. The driver recognition range is a three-dimensional range including a certain range in the vertical and front-rear directions of the steering. Based on the steering position information detected by the steering position detection unit 2, the driver recognition range determination unit 12 determines whether the steering has deviated from the driver recognition range.

[0022] The driver state determination unit 13 determines the state of the driver of the vehicle based on the captured image of the driver monitoring camera 1. The driver state determination unit 13 determines, for example, whether the driver is in a side-glancing state from the captured image of the driver's face. The driver state determination unit 13 may determine whether the driver is in a distracted state of visually recognizing a mobile terminal or the like, and may also determine whether the driver is in a drowsy state or a state of being under the influence of alcohol.

[0023] When it is determined by the position change information acquisition unit 11 that the position of the steering wheel has been changed, the driver state determination unit 13 estimates the position of the driver's face in the captured image after the position change based on the position of the driver's face in the captured image of the driver monitoring camera 1 before the position change of the steering wheel and the position change information of the steering wheel.

[0024] Here, FIG. 2 is a diagram for explaining the change in the imaging range of the driver monitoring camera 1 due to the position change of the steering wheel. In FIG. 2, the driver 50, the steering wheel 20, the driver monitoring camera 1, and the imaging range PA of the driver monitoring camera 1 are shown. The steering wheel 20 has a position change mechanism that can change the position of the steering wheel 20 with respect to the vehicle. The steering wheel 20, the driver monitoring camera 1, and the imaging range PA after the position change are shown by dashed lines.

[0025] As shown in FIG. 2, when the driver monitoring camera 1 is provided integrally with the steering wheel 20, when the driver 50 changes the position of the steering wheel 20, the imaging range PA of the driver monitoring camera 1 with respect to the driver 50 will change significantly. As a result, in the conventional driver state determination device, the position of the driver 50's face reflected in the captured image of the driver monitoring camera 1 changes significantly before and after the position change of the steering wheel 20, and it may take time to determine the state of the driver 50 using the captured image.

[0026] Therefore, when the driver state determination unit 13 determines that the steering wheel 20 has been repositioned, it estimates the position of the driver's face in the image captured by the driver monitor camera 1 after the reposition change, based on the position of the driver's face in the image captured by the driver monitor camera 1 before the reposition change of the steering wheel 20 and the reposition change information of the steering wheel 20. As shown in Figure 2, the driver state determination unit 13 recognizes that there is a relationship between the reposition change of the steering wheel 20 and the change in the imaging range PA of the driver monitor camera 1. For example, it uses a pre-prepared image processing program to estimate the change in the position of the driver's face in the image captured from the reposition change information of the steering wheel 20, thereby estimating the position of the driver's face in the image captured after the reposition change.

[0027] The driver status determination unit 13 determines the driver's status from the estimated position of the driver's face and the captured image after the position change. Specifically, the driver status determination unit 13 performs face recognition processing on a predetermined range including the estimated position of the driver's face, rather than the entire captured image, thereby recognizing the driver's face status in a shorter time compared to performing face recognition processing on the entire captured image. The driver status determination unit 13 determines the driver's status, such as distracted driving, from the recognized driver's face status.

[0028] Furthermore, if the driver status determination unit 13 determines that the steering wheel has deviated from the driver recognition range determination unit 12, it may maintain the determination result of the driver's status immediately prior to that determination. The steering wheel deviating from the driver recognition range is, for example, when the driver accidentally releases the steering wheel's position lock and the steering wheel drops down. The steering wheel deviating from the driver recognition range may also include cases where the driver accidentally moves the steering wheel to a position that makes steering difficult when adjusting its position.

[0029] The driver state determination unit 13 maintains the determination result from before the steering wheel deviated from the driver recognition range, in order to avoid making an incorrect determination of the driver state, because if the steering wheel deviates from the driver recognition range, there is a high possibility that the driver's face has moved out of the imaging range of the driver monitor camera 1.

[0030] If the driver status determination unit 13 determines that the steering wheel has deviated from the driver recognition range determination unit 12, it outputs a voice notification via the HMI 3 prompting the driver to return the steering wheel to its original position. The position return notification is, for example, a voice notification instructing the driver to return the steering wheel to its normal position.

[0031] The driver status determination unit 13 may also determine whether the driver's condition is abnormal based on the image captured by the driver monitor camera 1. An abnormal driver condition is a state in which it can be inferred that the driver is unable to continue driving normally. For example, the driver status determination unit 13 may determine that the driver is in an abnormal state if the driver's face suddenly turns downwards and the driver's posture drops. The driver status determination unit 13 may also determine that the driver is in an abnormal state if the driver's head suddenly drops backward and hits the headrest, and then the driver's face remains facing upwards for a certain period of time.

[0032] Furthermore, the driver state determination unit 13 may be less likely to determine that the driver state is abnormal when the steering wheel 20 is repositioned compared to when the steering wheel 20 is not repositioned. When the steering wheel 20 is repositioned, the position of the driver monitor camera 1 also changes, which reduces the accuracy of driver state determination. By making it less likely to determine that the driver state is abnormal, misdetermining can be suppressed.

[0033] Specifically, the driver state determination unit 13 may, for example, make it less likely to determine that a driver is in an abnormal state by changing the threshold value for determining the angle of the driver's face used for abnormality determination to a larger value. Alternatively, the driver state determination unit 13 may make it less likely to determine that a driver is in an abnormal state by changing the threshold value for the duration of the driver's face being in an upward position used for abnormality determination to a longer time.

[0034] Next, the control of the driver state determination device 100 will be explained with reference to Figures 3 and 4. Figure 3 is a flowchart showing an example of the driver state determination process based on a change in steering wheel position.

[0035] As shown in Figure 3, the ECU 10 of the driver state determination device 100 determines, in S10, whether or not the position of the vehicle's steering wheel 20 has been changed by the position change information acquisition unit 11. The position change information acquisition unit 11 makes the above determination based on the steering position information detected by the steering position detection unit 2. If the ECU 10 determines that the position of the steering wheel 20 has been changed, it proceeds to S11. If the ECU 10 does not determine that the position of the steering wheel 20 has been changed, it proceeds to S14.

[0036] In S11, the ECU10 acquires position change information of the steering 20 using the position change information acquisition unit 11.

[0037] In S12, the ECU 10 determines whether the position of the steering wheel 20 has deviated from the driver recognition range using the driver recognition range determination unit 12. The driver recognition range determination unit 12 makes the above determination based on the steering wheel position information detected by the steering wheel position detection unit 2. If the ECU 10 determines that the position of the steering wheel 20 has not deviated from the driver recognition range, it proceeds to S13. If the ECU 10 determines that the position of the steering wheel 20 has deviated from the driver recognition range, it proceeds to S15.

[0038] In S13, the ECU 10 estimates the position of the driver's face in the image captured after the steering wheel position change using the driver state determination unit 13. The driver state determination unit 13 estimates the position of the driver's face based on the position of the driver's face in the image captured by the driver monitor camera 1 before the steering wheel position change and the steering wheel position change information.

[0039] In S14, the ECU 10 determines the driver status using the driver status determination unit 13. If the determination in S10 is NO, the driver status determination unit 13 determines the driver status by performing normal driver face recognition processing based on the image captured by the driver monitor camera 1. Normal driver face recognition processing is a process that recognizes the driver's face from the current image based on the position of the driver's face recognized in the image captured for the previous driver status determination.

[0040] Also in S14, if the process goes through S13, the driver state determination unit 13 determines the driver's state from the estimated position of the driver's face in S13 and the captured image after the position change. The driver state determination unit 13 determines the driver's state, such as distracted driving or drowsiness. Based on the determination result, the driver state determination unit 13 may issue a warning to the driver via the HMI 3. After that, the ECU 10 terminates the driver state determination process.

[0041] In S15, the ECU 10 maintains the driver state determination result from the driver state determination unit 13 before the steering 20 deviated from the driver recognition range. However, if this is the first determination after the vehicle ignition is turned on, the driver state determination is not performed.

[0042] In S16, the ECU 10, via the driver status determination unit 13, outputs an audio notification to the driver via the HMI 3 prompting them to return the steering wheel to its original position. After that, the ECU 10 terminates the driver status determination process.

[0043] Figure 4 is a flowchart showing an example of a threshold change process for determining an abnormal state due to a change in the position of the steering wheel 20. As shown in Figure 4, in S20, the ECU 10 determines whether or not the position of the vehicle's steering wheel 20 has been changed by the position change information acquisition unit 11. If the ECU 10 determines that the position of the steering wheel 20 has been changed, it proceeds to S21. If the ECU 10 does not determine that the position of the steering wheel 20 has been changed, it proceeds to S22.

[0044] In S21, the ECU 10 changes the driver abnormality detection threshold to a value that is less likely to detect an abnormality than usual, using the driver state determination unit 13. The driver state determination unit 13 may also change the threshold for the duration of the driver's face being in an upward position, which is used for abnormality detection, to a shorter time. After that, the ECU 10 terminates the threshold change process.

[0045] In S22, if the ECU10 has changed the driver abnormality detection threshold to a value that is more likely to detect abnormalities than usual, it will return the threshold to its normal value. If the threshold is already at its normal value, the ECU10 may omit the process in S22. After that, the ECU10 will terminate the threshold change process.

[0046] As described above, the driver state determination device 100 according to this embodiment estimates the position of the driver's face in the captured image after the position change using the position change information of the steering wheel 20 when the position of the steering wheel 20, which is equipped with the driver monitor camera 1, is changed. Therefore, compared to the case where face image recognition processing is performed on the entire captured image to find the position of the driver's face after the position change, it is possible to suppress the time required for driver state determination.

[0047] Furthermore, if the driver state determination device 100 determines that the steering wheel 20 has deviated from the driver recognition range due to a change in position, it maintains the determination result from before the steering wheel deviated from the driver recognition range. This prevents incorrect driver state determination when the driver's face is outside the imaging range of the driver monitor camera 1. In addition, if the driver state determination device 100 determines that the steering wheel 20 has deviated from the driver recognition range, it outputs an audio notification to the driver instructing them to return the steering wheel 20 to its original position, thereby appropriately informing the driver that the steering wheel 20 is in an inappropriate position.

[0048] Furthermore, the driver state determination device 100 makes it less likely to determine that the driver's state is abnormal when the steering wheel 20's position is changed, compared to when the steering wheel 20's position is not changed. When the steering wheel 20's position is changed, the position of the driver monitor camera 1 also changes, which reduces the accuracy of driver state determination. By making it less likely to determine that the driver is in an abnormal state, misjudgments can be suppressed.

[0049] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above. The present invention can be implemented in various forms, starting with the embodiments described above, by making various changes and improvements based on the knowledge of those skilled in the art.

[0050] For example, the ECU 10 does not necessarily need to have a driver recognition range determination unit 12. Also, audio output for position return notification is not required. [Explanation of symbols]

[0051] 1...Driver monitor camera, 2...Steering position detection unit, 3...HMI, 10...ECU, 11...Position change information acquisition unit, 12...Driver recognition range determination unit, 13...Driver status determination unit, 20...Steering, 50...Driver, 100...Driver status determination device.

Claims

1. A driver state determination device that determines the facial state of the vehicle's driver based on an image captured by a driver monitor camera mounted on the vehicle's steering wheel, A driver status determination unit determines the state of the driver based on the image captured by the driver monitor camera, When the steering wheel position is changed, a position change information acquisition unit acquires information about the steering wheel position change, Equipped with, The driver status determination unit estimates the position of the driver's face in the image captured by the driver monitor camera after the steering wheel position change, based on the position of the driver's face in the image captured by the driver monitor camera before the steering wheel position change and the steering wheel position change information, and determines the driver's status from the estimated position of the driver's face and the image captured after the position change.

2. The system further includes a driver recognition range determination unit that determines whether the steering wheel has deviated from a preset driver recognition range as a result of the steering wheel's position change. The driver state determination device according to claim 1, wherein the driver state determination unit maintains the determination result of the driver state immediately before the determination when the driver recognition range determination unit determines that the steering wheel has deviated from the driver recognition range.

3. The driver state determination device according to claim 2, wherein when the driver recognition range determination unit determines that the steering wheel has deviated from the driver recognition range, it outputs an audible notification prompting the driver to return the steering wheel to its original position.

4. The driver status determination unit determines whether the driver's status is abnormal or not based on the steering wheel position change information and the image captured by the driver monitor camera. A driver state determination device according to any one of claims 1 to 3, wherein when the steering wheel position is changed, it is less likely to determine that the driver's state is in an abnormal state compared to when the steering wheel position is not changed.

Images