Driver imaging device
The driver imaging device estimates vehicle direction and driver orientation using onboard sensors, eliminating the need for precise alignment and reducing calibration costs, thus enhancing accuracy and flexibility in camera placement.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- TOYOTA JIDOSHA KK
- Filing Date
- 2023-08-29
- Publication Date
- 2026-05-26
AI Technical Summary
Existing driver imaging devices require precise alignment and calibration of the driver monitoring camera, which is costly and prone to errors due to vibration, necessitating complex preset settings and frequent recalibration.
A driver imaging device with a camera ECU that estimates vehicle direction using acceleration and angular velocity sensors, determining driver face and gaze orientation without requiring precise alignment or pre-configuration, and updates estimates when the steering position changes.
Reduces calibration costs and eliminates the need for precise alignment, improving accuracy and reducing errors in determining driver orientation and gaze, while allowing flexible camera mounting positions.
Smart Images
Figure 0007865293000001 
Figure 0007865293000002 
Figure 0007865293000003
Abstract
Description
Technical Field
[0001] The present disclosure relates to a driver imaging device.
Background Art
[0002] As a technical document related to a conventional driver imaging device, Japanese Unexamined Patent Application Publication No. 2016-34810 is known. Japanese Unexamined Patent Application Publication No. 2016-34810 shows a device that calculates the degree of concentration on the driver's driving operation from the deviation angle between the traveling direction of the vehicle and the line-of-sight direction of the driver. The traveling direction of the vehicle is calculated based on the vehicle speed and the steering angle.
Prior Art Document
Patent Document
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] For determining the driver's face orientation with respect to the traveling direction of the vehicle and the fixation point, various preset settings based on the interior design information of the vehicle cabin and precise alignment of the driver monitoring camera at the time of installation are required. In addition, if the position of the driver monitoring camera is displaced due to vibration or the like during vehicle travel, a determination error occurs, and calibration costs are incurred to correct it.
[0005] An object of the present disclosure is to provide a driver imaging device that can reduce calibration costs while eliminating the need for preset settings and precise alignment of the driver monitoring camera.
Means for Solving the Problems
[0006] This disclosure relates to a driver imaging device including a driver monitor camera fixed in front of the driver's seat inside the vehicle cabin to image the driver, wherein the driver monitor camera has a camera ECU, which estimates the direction of travel of the vehicle based on acceleration information from the vehicle's acceleration sensor or acceleration information from an in-camera acceleration sensor built into the driver monitor camera, and determines the driver's face orientation relative to the direction of travel and the point of gaze based on the driver's face image captured by the driver monitor camera and the direction of travel. death , If the driver monitoring camera is mounted on the steering column, the estimated direction of travel of the vehicle is updated when the steering position is adjusted by the vehicle's steering position adjustment mechanism. [Effects of the Invention]
[0007] This disclosure makes it possible to provide a driver imaging device that can reduce calibration costs while eliminating the need for pre-configuration and precise alignment of the driver monitor camera. [Brief explanation of the drawing]
[0008] [Figure 1] This figure shows a vehicle equipped with a driver imaging device according to one embodiment. [Figure 2] Figure 1 is a block diagram of the functional configuration of the vehicle shown. [Figure 3] Figure 2 is a flowchart showing the processing performed by the camera ECU. [Modes for carrying out the invention]
[0009] The embodiments of this disclosure will be described below with reference to the drawings.
[0010] Figure 1 is a diagram showing a vehicle equipped with a driver imaging device according to one embodiment. Figure 2 is a block diagram of the functional configuration of the vehicle shown in Figure 1. As shown in Figures 1 and 2, the vehicle V comprises a driver imaging device 1, an acceleration sensor 2, a yaw rate sensor 3, a steering position adjustment mechanism 4, and an HMI (Human Machine Interface) 5.
[0011] The driver imaging device 1 includes a driver monitor camera 10. The driver monitor camera 10 is fixed in front of the driver's seat inside the vehicle V. The driver monitor camera 10 images the driver 9. The driver monitor camera 10 images the driver 9's face. The driver monitor camera 10 has a camera ECU 11, an in-camera acceleration sensor 12, and a gyro sensor 13.
[0012] The camera ECU 11 is built into the driver monitor camera 10. The camera ECU 11 is not a vehicle ECU. The camera ECU 11 is an electronic control unit having a CPU (Central Processing Unit) and a memory unit such as ROM (Read Only Memory) or RAM (Random Access Memory). In the camera ECU 11, for example, various functions are realized by the CPU executing a program stored in the memory unit.
[0013] The camera-integrated acceleration sensor 12 is built into the driver monitor camera 10. The camera-integrated acceleration sensor 12 detects the acceleration of the driver monitor camera 10. For example, the camera-integrated acceleration sensor 12 detects the acceleration of the driver monitor camera 10 in the forward / backward direction and the lateral acceleration of the driver monitor camera 10. The camera-integrated acceleration sensor 12 transmits the acceleration information of the driver monitor camera 10 to the camera ECU 11.
[0014] The gyro sensor 13 is built into the driver monitor camera 10. The gyro sensor 13 measures the yaw rate (rotational angular velocity) of the driver monitor camera 10. The gyro sensor 13 transmits the angular velocity information of the driver monitor camera 10 to the camera ECU 11.
[0015] The acceleration sensor 2 is mounted on the vehicle V. The acceleration sensor 2 detects the acceleration of the vehicle V. For example, the acceleration sensor 2 detects the acceleration of the vehicle V in the longitudinal direction and the lateral acceleration of the vehicle V. For example, the acceleration sensor 2 transmits the acceleration information of the vehicle V to the camera ECU 11.
[0016] The yaw rate sensor 3 detects the yaw rate (rotational angular velocity) of the vehicle V around the vertical axis of its center of gravity. The yaw rate sensor 3 is, for example, a gyro sensor. The yaw rate sensor 3 transmits the yaw rate information of the vehicle V to the camera ECU 11.
[0017] The steering position adjustment mechanism 4 adjusts the steering position of the vehicle V. The steering position adjustment mechanism 4 has an actuator that controls the front-to-back position of the steering wheel by the steering wheel's telescopic mechanism. The steering position adjustment mechanism 4 also has an actuator that controls the up-and-down position of the steering wheel by the steering wheel's tilt mechanism. When the steering position of the vehicle V is adjusted, the steering position adjustment mechanism 4 transmits the steering position information to the camera ECU 11.
[0018] HMI5 is an interface for inputting and outputting information between the vehicle and the occupant in response to control signals from the camera ECU11. HMI5 has a display or speaker, etc. HMI5 provides notifications to the occupant of the vehicle V. For example, HMI5 notifies the driver 9 of the direction of their face and the state of their gaze. HMI5 also provides notifications to encourage increased concentration.
[0019] The camera ECU 11 has the following functional configuration: a direction of travel estimation unit 14, an update processing unit 15, a front-to-back distance estimation unit 16, a camera orientation estimation unit 17, a face orientation determination unit 18, and a gaze point determination unit 19.
[0020] The traveling direction estimation unit 14 estimates the traveling direction D of the vehicle V based on the acceleration information of the acceleration sensor 2 of the vehicle V or the acceleration information of the in-camera acceleration sensor 12. In addition to the above acceleration information, the traveling direction estimation unit 14 estimates the traveling direction D of the vehicle V based on the angular velocity information of the yaw rate sensor 3 of the vehicle V or the angular velocity information of the gyro sensor 13 of the driver monitor camera 10. In addition to the above acceleration information, the traveling direction estimation unit 14 may estimate the traveling direction D of the vehicle V based on, for example, the detection result of the geomagnetic sensor of the vehicle V or the information of GNNS [Global Navigation Satellite System]. The traveling direction estimation unit 14 estimates the traveling direction D of the vehicle V at predetermined intervals. The traveling direction estimation unit 14 estimates the traveling direction D of the vehicle V, for example, every few seconds.
[0021] When a predetermined condition is satisfied, the update processing unit 15 performs an update process of updating the estimation result of the traveling direction D of the vehicle V by the traveling direction estimation unit 14. Specifically, when the driver monitor camera 10 is of the type mounted on the steering column, the update processing unit 15 performs an update process of the estimation result of the traveling direction D of the vehicle V regardless of whether the predetermined period has elapsed when the steering position of the vehicle V is adjusted by the steering position adjustment mechanism 4 of the vehicle V. That is, when the steering position is adjusted, the update processing unit 15 causes the traveling direction estimation unit 14 to re-estimate the traveling direction D of the vehicle V.
[0022] For example, when the change in the front-back position of the steering with respect to the driver 9 becomes a certain value or more, or when the change in the vertical angle of the steering becomes a certain angle or more, the update processing unit 15 causes the traveling direction D of the vehicle V to be re-estimated assuming that the steering position adjustment has been performed. When the driver monitor camera 10 is mounted on the rearview mirror, the update processing unit 15 may perform an update process of the estimation result of the traveling direction D of the vehicle V regardless of whether the predetermined period has elapsed when the position of the rearview mirror is adjusted.
[0023] The front-to-rear distance estimation unit 16 estimates the front-to-rear distance between the driver monitor camera 10 and the driver 9 based on the front-to-rear position information of the driver's seat, if the driver's seat of the vehicle V is an electric seat. For example, the front-to-rear distance estimation unit 16 calculates the difference in the front-to-rear direction between the relative position of the driver monitor camera 10 with respect to a predetermined position of the vehicle V and the relative position of the driver's seat with respect to that predetermined position of the vehicle V as the front-to-rear distance between the driver monitor camera 10 and the driver 9.
[0024] The front-to-rear distance estimation unit 16 may estimate the front-to-rear distance between the driver monitor camera 10 and the driver 9 based on the in-vehicle image captured by the driver monitor camera 10. In this case, the driver's seat does not need to be an electric seat. The front-to-rear distance estimation unit 16 estimates the front-to-rear distance between the driver monitor camera 10 and the driver 9 from the arrangement of the in-vehicle equipment of the vehicle V in the image. The front-to-rear distance estimation unit 16 estimates the front-to-rear distance between the driver monitor camera 10 and the driver 9 based on, for example, the shape or position of the driver's seat, passenger seat, and ceiling lights in the in-vehicle image.
[0025] The camera orientation estimation unit 17 estimates the orientation of the driver monitor camera 10 based on the in-vehicle image captured by the driver monitor camera 10. The camera orientation estimation unit 17 estimates the orientation of the driver monitor camera 10 from the arrangement of the in-vehicle equipment of the vehicle V within the image. For example, the camera orientation estimation unit 17 estimates the orientation of the driver monitor camera 10 based on the shape or position of the driver's seat, passenger seat, and ceiling lights, etc., in the in-vehicle image.
[0026] The face orientation determination unit 18 determines the face orientation of the driver 9 relative to the vehicle V's direction of travel D based on the face image of the driver 9 captured by the driver monitor camera 10, the direction of travel D of the vehicle V estimated by the direction of travel estimation unit 14, the front-to-rear distance between the driver monitor camera 10 and the driver 9 estimated by the front-to-rear distance estimation unit 16, and the orientation of the driver monitor camera 10 estimated by the camera orientation estimation unit 17.
[0027] The face direction determination unit 18 calculates, for example, the relative angle between the driver's face direction and the direction of travel D of the vehicle V. The face direction determination unit 18 determines, for example, whether the relative angle between the driver's face direction and the direction of travel D of the vehicle V is greater than a predetermined threshold. This threshold may be determined, for example, based on the vehicle speed of the vehicle V. For example, the threshold may tend to be smaller as the vehicle speed of the vehicle V increases. The face direction determination unit 18 determines, for example, whether the state in which the relative angle between the driver's face direction and the direction of travel D of the vehicle V is greater than the above threshold has lasted for a predetermined period of time or longer. If the face direction determination unit 18 determines that the state in which the relative angle between the driver's face direction and the direction of travel D of the vehicle V is greater than the above threshold has lasted for a predetermined period of time or longer, it determines that the driver 9 may not be properly checking the direction of travel D of the vehicle V (for example, distracted driving).
[0028] The gaze point determination unit 19 determines the gaze point of the driver 9 relative to the vehicle V's direction of travel D based on the driver 9's face image captured by the driver monitor camera 10, the direction of travel D of the vehicle V estimated by the direction of travel estimation unit 14, the front-to-rear distance between the driver monitor camera 10 and the driver 9 estimated by the front-to-rear distance estimation unit 16, and the orientation of the driver monitor camera 10 estimated by the camera orientation estimation unit 17.
[0029] The gaze point determination unit 19 calculates, for example, the relative angle between the driver's gaze direction and the vehicle V's direction of travel D. The gaze point determination unit 19 determines, for example, whether the relative angle between the driver's gaze direction and the vehicle V's direction of travel D is greater than a predetermined threshold. This threshold may be determined, for example, based on the vehicle speed of the vehicle V. For example, the threshold may tend to be smaller as the vehicle speed of the vehicle V increases. The gaze point determination unit 19 determines, for example, whether the state in which the relative angle between the driver's gaze direction and the vehicle V's direction of travel D is greater than the threshold has lasted for a predetermined period of time or longer. If the state in which the relative angle between the driver's gaze direction and the vehicle V's direction of travel D is greater than the threshold has lasted for a predetermined period of time or longer, the gaze point determination unit 19 determines that the driver 9 may not be properly checking the vehicle V's direction of travel D (for example, distracted driving).
[0030] If the relative angle between the driver's face orientation and the vehicle V's direction of travel D remains greater than the threshold for a predetermined period of time, the camera ECU 11 will alert the driver 9 via the speaker or display of the HMI 5. If the relative angle between the driver's gaze direction and the vehicle V's direction of travel D remains greater than the threshold for a predetermined period of time, the camera ECU 11 will alert the driver 9 via the speaker or display of the HMI 5.
[0031] Next, we will explain the processing performed by the camera ECU 11. Figure 3 is a flowchart showing the processing performed by the camera ECU 11.
[0032] As shown in Figure 3, in step S1, the camera ECU 11 estimates the direction of travel D of the vehicle V. In step S2, the camera ECU 11 determines whether or not the steering position of the vehicle V has been adjusted. If the camera ECU 11 determines that the steering position of the vehicle V has been adjusted (step S2: YES), it proceeds to step S3. If the camera ECU 11 determines that the steering position of the vehicle V has not been adjusted (step S2: NO), it proceeds to step S4.
[0033] In step S3, the camera ECU 11 updates the estimated result of the vehicle V's direction of travel D. In step S4, the camera ECU 11 estimates the longitudinal distance between the driver monitor camera 10 and the driver 9. In step S5, the camera ECU 11 estimates the orientation of the driver monitor camera 10. In step S6, the camera ECU 11 determines the driver 9's face orientation and gaze point relative to the vehicle V's direction of travel D, based on the driver 9's face image, the vehicle V's direction of travel D, the longitudinal distance between the driver monitor camera 10 and the driver 9, and the orientation of the driver monitor camera 10.
[0034] As explained above, the driver imaging device 1 determines the driver's face orientation and gaze point relative to the vehicle V's direction of travel D by estimating the vehicle V's direction of travel D using the camera ECU 11 of the driver monitor camera 10. Compared to conventional devices that require precise positioning of the driver monitor camera 10 during installation, after pre-setting the mounting position of the driver monitor camera 10, the orientation of the driver monitor camera 10, the vehicle V's interior design information, the driver's face orientation, and three-dimensional geometric information of the driver's seat front in the vehicle interior for gaze point determination, the driver imaging device 1 can determine the driver's face orientation and gaze point relative to the vehicle V's direction of travel D without requiring pre-setting or precise positioning. Furthermore, since the vehicle V's direction of travel D is estimated by the camera ECU 11, calibration costs can be reduced.
[0035] The camera ECU 11 estimates the direction of travel D of the vehicle V based on acceleration information from the acceleration sensor 2 or the acceleration information from the camera's internal acceleration sensor 12, as well as angular velocity information from the vehicle V's yaw rate sensor 3 or the angular velocity information from the gyro sensor 13 built into the driver monitor camera 10. By estimating the direction of travel D of the vehicle V while taking into account angular velocity information that changes according to the vehicle V's movement, the accuracy of estimating the direction of travel D of the vehicle V can be improved compared to when angular velocity information is not considered.
[0036] If the driver's seat of the vehicle V is an electric seat, the camera ECU 11 estimates the front-to-rear distance between the driver monitor camera 10 and the driver 9 based on the front-to-rear position information of the driver's seat, and determines the driver 9's face orientation and gaze point based on the front-to-rear distance, the driver 9's face image, and the direction of travel D. By estimating the front-to-rear distance between the driver monitor camera 10 and the driver 9 based on the front-to-rear position information of the electric seat of the driver's seat, and determining the driver 9's face orientation while considering the front-to-rear distance, the accuracy of determining the driver 9's face orientation and gaze point can be improved compared to when the front-to-rear distance between the driver monitor camera 10 and the driver 9 is not considered.
[0037] When the driver monitor camera 10 is mounted on the steering column, the camera ECU 11 updates the estimated direction of travel D of the vehicle V when the steering position is adjusted by the steering position adjustment mechanism 4 of the vehicle V. This is because, when the driver monitor camera 10 is mounted on the steering column, the position and orientation of the driver monitor camera 10 change when the steering position is adjusted by the steering position adjustment mechanism 4 of the vehicle V. By updating the estimated direction of travel D of the vehicle V, the camera ECU 11 can suppress any discrepancies in the direction of travel D of the vehicle V estimated by the camera ECU 11.
[0038] The camera ECU 11 estimates the orientation of the driver monitor camera 10 and the front-to-back distance between the driver monitor camera 10 and the driver 9 based on the in-vehicle image captured by the driver monitor camera 10, from the arrangement of the in-vehicle equipment in the image of the vehicle V. Based on the orientation and front-to-back distance of the driver monitor camera 10, the driver 9's face image, and the direction of travel D, the camera ECU 11 determines the driver 9's face orientation and gaze point. By estimating the orientation of the driver monitor camera 10 and the front-to-back distance between the driver monitor camera 10 and the driver 9 through image processing based on the arrangement of in-vehicle equipment such as the driver's seat, passenger seat, and ceiling lights in the image of the in-vehicle image captured by the driver monitor camera 10, the accuracy of determining the driver 9's face orientation and gaze point can be improved compared to when the orientation and front-to-back distance of the driver monitor camera 10 are not considered.
[0039] According to the driver imaging device 1 of this disclosure, the direction of travel D of the vehicle V can be estimated regardless of the mounting position of the driver monitor camera 10, so it is not necessary to change the design of the mounting position of the driver monitor camera 10 for each vehicle V. In other words, the degree of freedom in designing the mounting position of the driver monitor camera 10 is improved.
[0040] While embodiments of this disclosure have been described above, this disclosure is not limited to the embodiments described above. This disclosure can be implemented in various forms, including the embodiments described above, with various modifications and improvements based on the knowledge of those skilled in the art.
[0041] The direction of travel estimation unit 14 does not necessarily have to rely on the angular velocity information from the vehicle V's yaw rate sensor 3 and the angular velocity information from the driver monitor camera 10's gyro sensor 13 when estimating the direction of travel D of the vehicle V.
[0042] The direction of travel estimation unit 14 may estimate the direction of travel D of the vehicle V based solely on the detection results of the acceleration sensor 2. This allows for cost reduction, for example, through CAN [Controller Area Network] communication.
[0043] The face orientation determination unit 18 does not necessarily have to rely on the front-to-back distance between the driver monitor camera 10 and the driver 9 estimated by the front-to-back distance estimation unit 16, and the orientation of the driver monitor camera 10 estimated by the camera orientation estimation unit 17 when determining the face orientation of the driver 9. The camera ECU 11 does not necessarily have to have an update processing unit 15 and a front-to-back distance estimation unit 16.
[0044] The gaze point determination unit 19 does not necessarily have to rely on the front-to-back distance between the driver monitor camera 10 and the driver 9 estimated by the front-to-back distance estimation unit 16, or the orientation of the driver monitor camera 10 estimated by the camera orientation estimation unit 17, when determining the gaze point of the driver 9.
[0045] The driver imaging device 1 may include multiple driver monitor cameras 10. The driver monitor cameras 10 may be installed, for example, near each of the two front pillars (A-pillars) of the vehicle V. This allows for the combination of facial images of the driver 9 captured from multiple angles (internal generation of three-dimensional information or ensuring redundancy in backlit situations, etc.), thereby improving the performance of determining the driver 9's facial orientation. For example, it is possible to improve the accuracy of determining at least one of the driver 9's direct gaze and distracted gaze states. Furthermore, even if it becomes necessary to change the mounting position of the driver monitor cameras 10 for each vehicle V or each driver 9, the driver imaging device 1 can estimate the direction of travel D of the vehicle V regardless of the mounting position of the driver monitor cameras 10, as described above, thus enabling performance improvements using multiple driver monitor cameras 10 at low cost.
[0046] The direction of travel of the vehicle V estimated by the direction of travel estimation unit 14 may be stored in, for example, non-volatile memory. This improves the efficiency of estimating the direction of travel D of the vehicle V, for example, when the vehicle V is started up. [Explanation of Symbols]
[0047] 1...Driver imaging device, 2...Accelerometer, 3...Yaw rate sensor, 4...Steering position adjustment mechanism, 9...Driver, 10...Driver monitor camera, 11...Camera ECU, 12...In-camera accelerometer, 13...Gyro sensor, V...Vehicle.
Claims
1. A driver imaging device including a driver monitor camera that is fixed in front of the driver's seat inside the vehicle's cabin and images the driver, The aforementioned driver monitor camera has a camera ECU, The camera ECU estimates the direction of travel of the vehicle based on acceleration information from the vehicle's acceleration sensor or acceleration information from an in-camera acceleration sensor built into the driver monitor camera, and determines the driver's face orientation and gaze point relative to the direction of travel of the vehicle based on the driver's face image captured by the driver monitor camera and the direction of travel. If the driver monitor camera is mounted on the steering column, the driver imaging device updates the estimated direction of travel of the vehicle when the steering position is adjusted by the vehicle's steering position adjustment mechanism.
2. The driver imaging device according to claim 1, wherein the camera ECU estimates the direction of travel of the vehicle based on angular velocity information from the vehicle's yaw rate sensor or angular velocity information from a gyro sensor built into the driver monitor camera, in addition to the acceleration information.
3. The camera ECU, when the driver's seat of the vehicle is an electric seat, estimates the front-to-rear distance between the driver monitor camera and the driver based on the front-to-rear position information of the driver's seat. The driver imaging device according to claim 1 or 2, which determines the driver's face orientation and point of gaze based on the front-to-back distance, the driver's face image, and the direction of travel.
4. The camera ECU estimates the orientation of the driver monitor camera and the front-to-rear distance between the driver monitor camera and the driver based on the in-vehicle image captured by the driver monitor camera and the arrangement of the vehicle's in-vehicle equipment within the image. The driver imaging device according to claim 1 or 2, which determines the driver's face orientation and point of gaze based on the orientation of the driver monitor camera, the front-to-back distance, the driver's face image, and the direction of travel.