Driver state determination device

Abstract

To restrict a time for driver state determination from becoming longer due to a position change of a steering equipped with a driver monitor camera.SOLUTION: A driver state determination device includes: a driver state determination section for determining a state of a driver based on a photographed image of a driver monitor camera; and a position change information acquisition section for acquiring position change information of a steering when a position change of the steering is performed. The driver state determination section is configured to: estimate a position of the face of the driver in the photographed image after the position change based on the position of the face of the driver in the photographed image of the driver monitor camera before the position change of the steering and the position change information of the steering when the position change of the steering is performed; and determine the state of the driver from the estimation result of the position of the face of the driver and the photographed image after the position change.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a driver condition determination device. [Background technology]

[0002] Conventionally, JP 2021-026626 A is known as a technical document related to a driver state determination device. This publication describes that when a driver's posture change is detected during an autonomous driving mode and the driver's holding of the steering wheel is not detected, a request to the driver to hold the steering wheel is started. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-026626 A Summary of the Invention [Problem to be solved by the invention]

[0004] By the way, the driver monitor camera used to judge the driver's condition is often installed on the steering column. In this case, when the driver changes the steering position, the position of the driver monitor camera also changes, so there is a problem that it takes time to judge the driver's condition. [Means for solving the problem]

[0005] One aspect of the present invention is a driver state determination device that determines the state of the face of a vehicle driver based on an image captured by a driver monitor camera provided with respect to the steering wheel of the vehicle, and includes a driver state determination unit that determines the state of the driver based on the image captured by the driver monitor camera, and a position change information acquisition unit that acquires steering wheel position change information when the steering wheel position is changed, and when the steering wheel position is changed, the driver state determination unit estimates the position of the driver's face in the captured image after the position change based on the position of the driver's face in the captured image captured by the driver monitor camera before the steering wheel position change and the steering wheel position change information, and determines the driver's state from the estimated result of the driver's face position and the captured image after the position change.

[0006] In a driver condition determination device according to one embodiment of the present invention, the driver condition determination unit determines whether the driver's condition is an abnormal driver condition based on steering position change information and an image captured by a driver monitor camera, and may make it less likely that the driver's condition is determined to be an abnormal driver condition when the steering position is changed compared to when the steering position is not changed. Effect of the Invention

[0007] According to one aspect of the present invention, it is possible to suppress the time required for determining the driver's state due to a change in the position of a steering wheel on which a driver monitor camera is provided. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a driver condition determination device according to an embodiment; [Diagram 2] 10A and 10B are diagrams for explaining a change in the imaging range of the driver monitor camera due to a change in the steering position. [Diagram 3] 10 is a flowchart showing an example of a driver state determination process based on a change in steering position; [Figure 4]10 is a flowchart showing an example of a threshold change process for determining an abnormal state due to a change in steering position; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] Fig. 1 is a block diagram showing a driver state determination device 100 according to an embodiment. The driver state determination device 100 shown in Fig. 1 is mounted on a vehicle and determines the state of the driver of the vehicle. The driver state determination device 100 determines whether the driver is looking away using, for example, an image of the driver's face captured by a driver monitor camera 1.

[0011] As shown in Fig. 1, the driver state determination device 100 includes an ECU (Electronic Control Unit) 10 that performs overall management of the device. The ECU 10 is an electronic control unit that includes a CPU (Central Processing Unit) and a storage unit. The storage unit is configured, for example, with a ROM (Read Only Memory), a RAM (Random Access Memory), and an EEPROM (Electrically Erasable Programmable Read-Only Memory).

[0012] In the ECU 10, for example, various functions are realized by executing a program stored in a storage unit by a CPU. The ECU 10 may be composed of a plurality of electronic units. The ECU 10 is connected to a driver monitor camera 1, a steering position detection unit 2, and an HMI 3 [Human Machine Interface].

[0013] The driver monitor camera 1 is a camera for capturing an image of the driver. The driver monitor camera 1 can be a digital camera having an image sensor such as a CCD (Charge Coupled Device) or a CIS (CMOS Image Sensor).

[0014] The driver monitor camera 1 is provided on the steering wheel of the vehicle. The installation method of the driver monitor camera 1 is not limited as long as it is provided integrally with the steering wheel. As an example, the driver monitor camera 1 is provided on a cover of a steering column, and captures an image of the driver's head at a predetermined frame rate. The driver monitor camera 1 may be provided integrally with the steering wheel. The driver monitor camera 1 transmits the captured image of the driver to the ECU 10.

[0015] The steering position detection unit 2 detects the position of the steering wheel of the vehicle. The steering position detection unit 2 has, for example, a tilt sensor and a telescopic sensor. The tilt sensor is a sensor that detects the up-down position of the steering wheel. The telescopic sensor is a sensor that detects the front-rear position of the steering wheel. The steering position detection unit 2 detects the position of the steering wheel from the detection results of the tilt sensor and the telescopic sensor. The steering position detection unit 2 transmits steering position information regarding the position of the steering wheel to the ECU 10.

[0016] The HMI 3 is an interface for inputting and outputting information between the ECU 10 and the driver. The HMI 3 includes, for example, a display and a speaker provided in the vehicle cabin. The HMI 3 outputs images from the display and sounds from the speaker in response to a control signal from the ECU 10. The display may be a multi-information display (MID) or a head-up display (HUD). The HMI 3 may include various indicators.

[0017] Next, a description will be given of the functional configuration of the ECU 10. As shown in FIG.

[0018] The position change information acquisition unit 11 acquires steering position change information when the position of the steering wheel of the vehicle is changed. The position change information acquisition unit 11 determines whether or not the position of the steering wheel of the vehicle has been changed based on the steering position information detected by the steering position detection unit 2. The position change information acquisition unit 11 determines that the position of the steering wheel of the vehicle has been changed when, for example, the up-down position or the front-rear position of the steering wheel has been changed by a certain value or more.

[0019] When it is determined that the steering position of the vehicle has been changed, the position change information acquisition unit 11 acquires steering position change information based on steering position information before and after the determination detected by the steering position detection unit 2. The position change information includes, for example, steering position information before the position change and steering position information after the position change.

[0020] When the position change information acquisition unit 11 determines that the steering position has been changed, the driver recognition range determination unit 12 determines whether the position change has caused the steering position to deviate from the driver recognition range.

[0021] The driver recognition range is a preset range used to determine whether a change in steering position has caused the driver monitor camera 1 to be unable to capture an image of the driver's face. In other words, if the steering position is within the driver recognition range, it can be assumed that the driver monitor camera 1 is capturing an image of the driver's face. The driver recognition range is a three-dimensional range that includes a certain range in the up-down and front-back directions of the steering wheel. The driver recognition range determination unit 12 determines whether the steering wheel has deviated from the driver recognition range based on the steering wheel position information detected by the steering wheel position detection unit 2.

[0022] The driver state determination unit 13 determines the state of the driver of the vehicle based on the captured image of the driver monitor camera 1. For example, the driver state determination unit 13 determines whether the driver is looking away from the camera based on the captured image of the driver's face. The driver state determination unit 13 may determine whether the driver is looking away from the camera by viewing a mobile terminal or the like, or may determine whether the driver is drowsy or drunk.

[0023] When the position change information acquisition unit 11 determines that the steering position has been changed, the driver state determination unit 13 estimates the position of the driver's face in the captured image after the position change based on the position of the driver's face in the captured image of the driver monitor camera 1 before the steering position change and the steering position change information.

[0024] Here, Fig. 2 is a diagram for explaining the change in the imaging range of the driver monitor camera 1 due to a change in the position of the steering wheel. Fig. 2 shows the driver 50, the steering wheel 20, the driver monitor camera 1, and the imaging range PA of the driver monitor camera 1. The steering wheel 20 has a position change mechanism that can change the position of the steering wheel 20 relative to the vehicle. The steering wheel 20, the driver monitor camera 1, and the imaging range PA after the position change are shown by dashed lines.

[0025] 2, in the case where the driver monitor camera 1 is provided integrally with the steering wheel 20, when the driver 50 changes the position of the steering wheel 20, the imaging range PA of the driver monitor camera 1 with respect to the driver 50 changes significantly. As a result, in the conventional driver state determination device, the position of the face of the driver 50 reflected in the image captured by the driver monitor camera 1 changes significantly before and after the position of the steering wheel 20, and it may take a long time to determine the state of the driver 50 using the captured image.

[0026] Therefore, when it is determined that the position of the steering wheel 20 has been changed, the driver state determination unit 13 estimates the position of the driver's face in the captured image after the position change based on the position of the driver's face in the captured image of the driver monitor camera 1 before the position change of the steering wheel 20 and the position change information of the steering wheel 20. Since there is a relationship between the position change of the steering wheel 20 and the change in the imaging range PA of the driver monitor camera 1 as shown in Fig. 2, the driver state determination unit 13 estimates the position of the driver's face in the captured image after the position change using, for example, an image processing program prepared in advance to estimate the position change of the driver's face in the captured image from the position change information of the steering wheel 20.

[0027] The driver state determination unit 13 determines the state of the driver from the estimation result of the driver's face position and the captured image after the position change. Specifically, the driver state determination unit 13 performs face recognition processing not on the entire captured image but on a predetermined range including the estimated position of the driver's face, thereby recognizing the state of the driver's face in a shorter time than when face recognition processing is performed on the entire captured image. The driver state determination unit 13 determines the state of the driver, such as a distracted state, from the recognized state of the driver's face.

[0028] In addition, when the driver recognition range determination unit 12 determines that the steering wheel has deviated from the driver recognition range, the driver state determination unit 13 may maintain the determination result of the driver's state immediately before the determination. A case where the steering wheel deviates from the driver recognition range is, for example, a case where the driver mistakenly releases the position lock of the steering wheel and the steering wheel falls down. A case where the steering wheel deviates from the driver recognition range may include a case where the driver mistakenly moves the steering wheel to a position where it is difficult to operate the steering wheel when adjusting the position of the steering wheel.

[0029] When the steering wheel deviates from the driver's recognition range, there is a high possibility that the driver's face is out of the imaging range of the driver monitor camera 1, so the driver state determination unit 13 maintains the determination result before the steering wheel deviates from the driver's recognition range to avoid making an erroneous determination of the driver state.

[0030] When the driver recognition range determination unit 12 determines that the steering wheel has deviated from the driver recognition range, the driver state determination unit 13 outputs a position return notification by voice via the HMI 3 to prompt the driver to return the steering wheel position. The position return notification is, for example, a voice notification to tell the driver to return the steering wheel position to a normal range.

[0031] The driver state determination unit 13 may determine whether the driver is in an abnormal driver state based on the image captured by the driver monitor camera 1. The abnormal driver state is a state in which it is assumed that the driver cannot continue driving normally. The driver state determination unit 13 determines that the driver is in an abnormal driver state, for example, when the driver's face suddenly turns downward and the driver's posture drops. The driver state determination unit 13 may also determine that the driver is in an abnormal driver state when the driver's head suddenly turns backward and hits the headrest, and then the driver's face continues to face upward for a certain period of time.

[0032] In addition, the driver state determination unit 13 may make it more difficult to determine that the driver's state is an abnormal driver state when the position of the steering wheel 20 has been changed, compared to when the position of the steering wheel 20 has not been changed. When the position of the steering wheel 20 has been changed, the position of the driver monitor camera 1 also changes, which reduces the accuracy of determining the driver's state, so by making it more difficult to determine that the driver is in an abnormal driver state, it is possible to suppress erroneous determinations.

[0033] Specifically, the driver state determination unit 13 may make it difficult to determine that the driver is in an abnormal state by, for example, changing the face direction angle determination threshold value for determining that the driver's face is facing down, which is used in the abnormality determination, to a larger value. The driver state determination unit 13 may make it difficult to determine that the driver is in an abnormal state by changing the threshold value for the duration of the driver's face facing up, which is used in the abnormality determination, to a longer time.

[0034] Next, the control of the driver condition determination device 100 will be described with reference to Figures 3 and 4. Figure 3 is a flow chart showing an example of a driver condition determination process in response to a change in steering position.

[0035] As shown in Fig. 3, in S10, the ECU 10 of the driver state determination device 100 determines whether or not the position of the steering wheel 20 of the vehicle has been changed by the position change information acquisition unit 11. The position change information acquisition unit 11 makes the above determination based on the steering position information detected by the steering position detection unit 2. If it is determined that the position of the steering wheel 20 has been changed, the ECU 10 proceeds to S11. If it is not determined that the position of the steering wheel 20 has been changed, the ECU 10 proceeds to S14.

[0036] In S11 , the ECU 10 acquires position change information of the steering wheel 20 from the position change information acquisition unit 11 .

[0037] In S12, the ECU 10 determines whether or not the position of the steering wheel 20 has deviated from the driver recognition range by the driver recognition range determination unit 12. The driver recognition range determination unit 12 makes the above determination based on the steering position information detected by the steering position detection unit 2. If the ECU 10 determines that the position of the steering wheel 20 has not deviated from the driver recognition range, the process proceeds to S13. If the ECU 10 determines that the position of the steering wheel 20 has deviated from the driver recognition range, the process proceeds to S15.

[0038] In S13, the ECU 10 estimates the position of the driver's face in the captured image after the position change by the driver state determination unit 13. The driver state determination unit 13 estimates the position of the driver's face based on the position of the driver's face in the captured image of the driver monitor camera 1 before the steering position is changed and the steering position change information.

[0039] In S14, the ECU 10 judges the driver state by the driver state judgment unit 13. If the judgment in S10 is NO, the driver state judgment unit 13 judges the driver state by performing normal driver face recognition processing based on the captured image of the driver monitor camera 1. The normal driver face recognition processing is, for example, processing for recognizing the driver's face from the current captured image based on the position of the driver's face recognized in the captured image used for the previous driver state judgment.

[0040] Similarly, in S14, if the process has gone through S13, the driver state determination unit 13 determines the driver state from the estimation result of the driver's face position in S13 and the captured image after the position change. The driver state determination unit 13 determines the driver state such as a distracted state or a drowsy state. The driver state determination unit 13 may issue a warning to the driver through the HMI 3 based on the determination result. After that, the ECU 10 ends the driver state determination process.

[0041] In S15, the ECU 10 maintains the result of the determination made by the driver state determination unit 13 before the steering wheel 20 departed from the driver recognition range. Note that if this is the first determination after the vehicle ignition is turned on, the ECU 10 does not perform a driver state determination.

[0042] In S16, the ECU 10 causes the driver state determination unit 13 to output a voice position return notification via the HMI 3 to prompt the driver to return the steering position. After that, the ECU 10 ends the driver state determination process.

[0043] Fig. 4 is a flowchart showing an example of a threshold change process for determining an abnormal state due to a change in the position of the steering wheel 20. As shown in Fig. 4, in S20, the ECU 10 determines whether or not the position of the steering wheel 20 of the vehicle has been changed by the position change information acquisition unit 11. If it is determined that the position of the steering wheel 20 has been changed, the ECU 10 proceeds to S21. If it is not determined that the position of the steering wheel 20 has been changed, the ECU 10 proceeds to S22.

[0044] In S21, the ECU 10 changes the determination threshold for the driver abnormality determination by the driver state determination unit 13 to a value that is less likely to cause an abnormality determination than usual. The driver state determination unit 13 may change the threshold for the duration of the driver's face facing up, which is used for abnormality determination, to a shorter time. After that, the ECU 10 ends the current threshold change process.

[0045] In S22, if the ECU 10 has changed the determination threshold value for the driver abnormality determination to a value that is more likely to cause an abnormality determination than usual, the ECU 10 returns the determination threshold value to the normal value. If the determination threshold value is the normal value, the ECU 10 may omit the process of S22. After that, the ECU 10 ends the current threshold change process.

[0046] According to the driver condition determination device 100 of this embodiment described above, when the position of the steering wheel 20 on which the driver monitor camera 1 is mounted is changed, the position of the driver's face in the captured image after the position change is estimated using position change information of the steering wheel 20, so that the time required for driver condition determination can be reduced compared to when face image recognition processing is performed on the entire captured image to find the position of the driver's face after the position change.

[0047] Moreover, when the driver state determination device 100 determines that the steering wheel 20 has deviated from the driver recognition range due to a position change, it maintains the determination result before the steering wheel deviated from the driver recognition range, thereby making it possible to avoid making an erroneous determination of the driver state in a situation where the driver's face is out of the imaging range of the driver monitor camera 1. In addition, when the driver state determination device 100 determines that the steering wheel 20 has deviated from the driver recognition range, it can appropriately inform the driver that the position of the steering wheel 20 is inappropriate by outputting a voice notification to the driver to return the position of the steering wheel 20.

[0048] Furthermore, in the driver condition determination device 100, when the position of the steering wheel 20 is changed, the driver's condition is less likely to be determined to be an abnormal driver condition compared to when the position of the steering wheel 20 is not changed. When the position of the steering wheel 20 is changed, the position of the driver monitor camera 1 also changes, which reduces the accuracy of determining the driver's condition, so by making it less likely to be determined that the driver is in an abnormal driver condition, it is possible to suppress erroneous determinations.

[0049] Although the embodiment of the present invention has been described above, the present invention is not limited to the above-mentioned embodiment. The present invention can be embodied in various forms including the above-mentioned embodiment and various modifications and improvements based on the knowledge of those skilled in the art.

[0050] For example, the ECU 10 does not necessarily have to include the driver recognition range determination unit 12. Also, the voice output of the position return notification is not essential. [Explanation of symbols]

[0051] 1...driver monitor camera, 2...steering position detection unit, 3...HMI, 10...ECU, 11...position change information acquisition unit, 12...driver recognition range determination unit, 13...driver state determination unit, 20...steering wheel, 50...driver, 100...driver state determination device.

Claims

1. A driver state determination device that determines a facial state of a driver of a vehicle based on an image captured by a driver monitor camera provided on a steering wheel of the vehicle, comprising: A driver state determination unit that determines a state of the driver based on an image captured by the driver monitor camera; a position change information acquisition unit that acquires information about a change in the position of the steering wheel when the position of the steering wheel is changed; Equipped with The driver condition determination unit, when the steering position is changed, estimates the position of the driver's face in the captured image after the position change based on the position of the driver's face in the captured image of the driver monitor camera before the steering position change and steering position change information, and determines the state of the driver from the estimated result of the position of the driver's face and the captured image after the position change.

2. A driver recognition range determination unit is further provided for determining whether or not the steering wheel deviates from a preset driver recognition range due to the position change when the steering wheel position is changed, 2. The driver state determination device according to claim 1, wherein, when the driver recognition range determination unit determines that the steering has deviated from the driver recognition range, the driver state determination unit maintains a determination result of the driver's state immediately before the determination.

3. 3. The driver state determination device according to claim 2, further comprising: a position return notification that prompts the driver to return the steering wheel position to its original position when the driver recognition range determination unit determines that the steering wheel has deviated from the driver recognition range.

4. The driver state determination unit determines whether or not the state of the driver is an abnormal driver state based on the steering position change information and the image captured by the driver monitor camera, The driver state determination device according to any one of claims 1 to 3, wherein when the steering position is changed, the driver state is less likely to be determined to be the driver abnormal state compared to when the steering position is not changed.

Images