Driver imaging device

Abstract

To provide a driver imaging device making it possible to reduce a calibration cost and obviate the need for preliminary setting or strict alignment of a driver monitor camera.SOLUTION: A driver imaging device 1 includes a driver monitor camera 10 that is fixed to a position ahead of a driver seat in a component of a vehicle and images a driver. The driver monitor camera 10 includes a camera ECU 11. The camera ECU 11 estimates an advancing direction of the vehicle on the basis of acceleration information of an acceleration sensor 2 of the vehicle or acceleration information of an in-camera acceleration sensor 12 incorporated in the driver monitor camera 10, and determines an orientation of the driver's face and a gaze point with respect to the advancing direction of the vehicle on the basis of the face image of the driver imaged by the driver monitor camera 10 and the advancing direction.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to driver imaging devices. [Background technology]

[0002] Japanese Patent Publication No. 2016-34810 is known as a technical document related to a conventional driver imaging device. Japanese Patent Publication No. 2016-34810 discloses a device that calculates the driver's concentration on driving operations from the deviation angle between the vehicle's traveling direction and the driver's line of sight. The vehicle's traveling direction is calculated based on the vehicle speed and steering angle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-34810 A Summary of the Invention [Problem to be solved by the invention]

[0004] In order to determine the driver's facial orientation and gaze point relative to the vehicle's travel direction, various pre-settings based on the vehicle interior design information and precise alignment of the driver monitor camera when installed are required. In addition, if the driver monitor camera is misaligned due to vibrations caused by vehicle travel, a determination error occurs, and calibration costs are required to correct this.

[0005] The present disclosure aims to provide a driver imaging device that can reduce calibration costs while eliminating the need for pre-settings or strict alignment of a driver monitor camera. [Means for solving the problem]

[0006] The driver imaging device of the present disclosure is a driver imaging device that includes a driver monitor camera that is fixed in front of the driver's seat inside the vehicle cabin to image the driver, and the driver monitor camera has a camera ECU that estimates the vehicle's traveling direction based on acceleration information from the vehicle's acceleration sensor or acceleration information from an in-camera acceleration sensor built into the driver monitor camera, and determines the driver's facial orientation and point of gaze relative to the vehicle's traveling direction based on the driver's facial image captured by the driver monitor camera and the traveling direction. Effect of the Invention

[0007] According to the present disclosure, it is possible to provide a driver imaging device that can reduce calibration costs while eliminating the need for advance settings or precise alignment of the driver monitor camera. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing a vehicle equipped with a driver imaging device according to an embodiment; [Diagram 2] FIG. 2 is a block diagram of a functional configuration of the vehicle shown in FIG. [Diagram 3] 3 is a flowchart showing a process performed by a camera ECU shown in FIG. 2. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0010] Fig. 1 is a diagram showing a vehicle equipped with a driver imaging device according to an embodiment. Fig. 2 is a block diagram showing a functional configuration of the vehicle shown in Fig. 1. As shown in Fig. 1 and Fig. 2, the vehicle V includes a driver imaging device 1, an acceleration sensor 2, a yaw rate sensor 3, a steering position adjustment mechanism 4, and an HMI [Human Machine Interface] 5.

[0011] The driver imaging device 1 includes a driver monitor camera 10. The driver monitor camera 10 is fixed in front of the driver's seat inside the cabin of the vehicle V. The driver monitor camera 10 images a driver 9. The driver monitor camera 10 images the face of the driver 9. The driver monitor camera 10 has a camera ECU 11, an acceleration sensor 12 within the camera, and a gyro sensor 13.

[0012] The camera ECU 11 is built into the driver monitor camera 10. The camera ECU 11 is not a vehicle ECU. The camera ECU 11 is an electronic control unit having a CPU [Central Processing Unit] and a storage unit such as a ROM [Read Only Memory] or a RAM [Random Access Memory]. In the camera ECU 11, for example, various functions are realized by the CPU executing a program stored in the storage unit.

[0013] The in-camera acceleration sensor 12 is built into the driver monitor camera 10. The in-camera acceleration sensor 12 detects the acceleration of the driver monitor camera 10. The in-camera acceleration sensor 12 detects, for example, the acceleration in the forward / rearward direction of the driver monitor camera 10 and the lateral acceleration of the driver monitor camera 10. The in-camera acceleration sensor 12 transmits acceleration information of the driver monitor camera 10 to the camera ECU 11.

[0014] The gyro sensor 13 is built in the driver monitor camera 10. The gyro sensor 13 measures the yaw rate (rotational angular velocity) of the driver monitor camera 10. The gyro sensor 13 transmits angular velocity information of the driver monitor camera 10 to the camera ECU 11.

[0015] The acceleration sensor 2 is mounted on the vehicle V. The acceleration sensor 2 detects the acceleration of the vehicle V. The acceleration sensor 2 detects, for example, the longitudinal acceleration of the vehicle V and the lateral acceleration of the vehicle V. The acceleration sensor 2 transmits acceleration information of the vehicle V to the camera ECU 11, for example.

[0016] The yaw rate sensor 3 detects a yaw rate (rotational angular velocity) around a vertical axis of the center of gravity of the vehicle V. The yaw rate sensor 3 is, for example, a gyro sensor. The yaw rate sensor 3 transmits yaw rate information of the vehicle V to the camera ECU 11.

[0017] The steering position adjustment mechanism 4 adjusts the position of the steering wheel of the vehicle V. The steering position adjustment mechanism 4 has an actuator that controls the front-rear position of the steering wheel by a telescopic mechanism of the steering wheel. The steering position adjustment mechanism 4 has an actuator that controls the up-down position of the steering wheel by a tilt mechanism of the steering wheel. When the steering position of the vehicle V is adjusted, the steering position adjustment mechanism 4 transmits steering position information to the camera ECU 11.

[0018] The HMI 5 is an interface for inputting and outputting information between the vehicle and the occupant in response to a control signal from the camera ECU 11. The HMI 5 has a display, a speaker, etc. The HMI 5 provides notifications to the occupant of the vehicle V. For example, the HMI 5 provides notifications of the face direction and gaze point of the driver 9. For example, the HMI 5 provides notifications to encourage improved concentration.

[0019] The camera ECU 11 has, as its functional configuration, a traveling direction estimation unit 14, an update processing unit 15, a front-rear distance estimation unit 16, a camera direction estimation unit 17, a face direction determination unit 18, and a gaze point determination unit 19.

[0020] The traveling direction estimation unit 14 estimates the traveling direction D of the vehicle V based on the acceleration information of the acceleration sensor 2 of the vehicle V or the acceleration information of the in-camera acceleration sensor 12. The traveling direction estimation unit 14 estimates the traveling direction D of the vehicle V based on the angular velocity information of the yaw rate sensor 3 of the vehicle V or the angular velocity information of the gyro sensor 13 of the driver monitor camera 10 in addition to the above acceleration information. The traveling direction estimation unit 14 may estimate the traveling direction D of the vehicle V based on, for example, the detection result of the geomagnetic sensor of the vehicle V or information of the GNNS [Global Navigation Satellite System] in addition to the above acceleration information. The traveling direction estimation unit 14 estimates the traveling direction D of the vehicle V at predetermined intervals. The traveling direction estimation unit 14 estimates the traveling direction D of the vehicle V, for example, every few seconds.

[0021] When a predetermined condition is satisfied, the update processing unit 15 performs an update process for updating the estimation result of the traveling direction D of the vehicle V by the traveling direction estimation unit 14. Specifically, when the driver monitor camera 10 is a steering column mounted type, the update processing unit 15 performs an update process for the estimation result of the traveling direction D of the vehicle V when the steering position adjustment mechanism 4 of the vehicle V adjusts the steering position, regardless of whether the above-mentioned predetermined period has elapsed. In other words, the update processing unit 15 causes the traveling direction estimation unit 14 to re-estimate the traveling direction D of the vehicle V when the steering position adjustment is performed.

[0022] For example, when the change in the longitudinal position of the steering wheel relative to the driver 9 exceeds a certain value, or when the change in the vertical angle of the steering wheel exceeds a certain angle, the update processing unit 15 determines that the steering position has been adjusted, and re-estimates the traveling direction D of the vehicle V. Note that, when the driver monitor camera 10 is mounted on the rearview mirror, the update processing unit 15 may update the estimation result of the traveling direction D of the vehicle V when the position of the rearview mirror is adjusted, regardless of whether the above-mentioned predetermined period has elapsed.

[0023] When the driver's seat of the vehicle V is an electric seat, the front-rear distance estimation unit 16 estimates the front-rear distance between the driver monitor camera 10 and the driver 9 based on the front-rear position information of the driver's seat. The front-rear distance estimation unit 16 calculates, for example, the difference in the front-rear direction between the relative position of the driver monitor camera 10 with respect to a predetermined position of the vehicle V and the relative position of the driver's seat with respect to the predetermined position of the vehicle V as the front-rear distance between the driver monitor camera 10 and the driver 9.

[0024] The front-rear distance estimation unit 16 may estimate the front-rear distance between the driver monitor camera 10 and the driver 9 based on the vehicle interior image captured by the driver monitor camera 10. In this case, the driver's seat does not need to be an electric seat. The front-rear distance estimation unit 16 estimates the front-rear distance between the driver monitor camera 10 and the driver 9 from the arrangement of the vehicle interior equipment of the vehicle V in the image. The front-rear distance estimation unit 16 estimates the front-rear distance between the driver monitor camera 10 and the driver 9 based on, for example, the shapes or positions of the driver's seat, passenger seat, ceiling light, etc. in the vehicle interior image.

[0025] The camera direction estimation unit 17 estimates the direction of the driver monitor camera 10 based on the vehicle interior image captured by the driver monitor camera 10. The camera direction estimation unit 17 estimates the direction of the driver monitor camera 10 from the arrangement of the vehicle interior equipment of the vehicle V in the image. The camera direction estimation unit 17 estimates the direction of the driver monitor camera 10 based on, for example, the shapes or positions of the driver's seat, passenger seat, ceiling light, etc. in the vehicle interior image.

[0026] The facial orientation determination unit 18 determines the facial orientation of the driver 9 relative to the traveling direction D of the vehicle V based on the facial image of the driver 9 captured by the driver monitor camera 10, the traveling direction D of the vehicle V estimated by the traveling direction estimation unit 14, the front-to-rear distance between the driver monitor camera 10 and the driver 9 estimated by the front-to-rear distance estimation unit 16, and the direction of the driver monitor camera 10 estimated by the camera orientation estimation unit 17.

[0027] The face direction determination unit 18 calculates, for example, a relative angle between the face direction of the driver 9 and the traveling direction D of the vehicle V. For example, the face direction determination unit 18 determines whether or not the relative angle between the face direction of the driver 9 and the traveling direction D of the vehicle V is greater than a predetermined threshold value. The threshold value may be determined based on the vehicle speed of the vehicle V. For example, the threshold value may tend to be smaller as the vehicle speed of the vehicle V increases. The face direction determination unit 18 determines, for example, whether or not a state in which the relative angle between the face direction of the driver 9 and the traveling direction D of the vehicle V is greater than the above threshold value has continued for a predetermined period of time or more. When a state in which the relative angle between the face direction of the driver 9 and the traveling direction D of the vehicle V is greater than the above threshold value has continued for a predetermined period of time or more, the face direction determination unit 18 determines that there is a possibility that the driver 9 is not appropriately checking the traveling direction D of the vehicle V (for example, looking aside, etc.).

[0028] The gaze point determination unit 19 determines the gaze point of the driver 9 relative to the traveling direction D of the vehicle V based on the facial image of the driver 9 captured by the driver monitor camera 10, the traveling direction D of the vehicle V estimated by the traveling direction estimation unit 14, the front-to-rear distance between the driver monitor camera 10 and the driver 9 estimated by the front-to-rear distance estimation unit 16, and the direction of the driver monitor camera 10 estimated by the camera orientation estimation unit 17.

[0029] The gaze point determination unit 19 calculates, for example, a relative angle between the gaze direction of the driver 9 and the traveling direction D of the vehicle V. For example, the gaze point determination unit 19 determines whether or not the relative angle between the gaze direction of the driver 9 and the traveling direction D of the vehicle V is greater than a predetermined threshold value. The threshold value may be determined based on the vehicle speed of the vehicle V. For example, the threshold value may tend to be smaller as the vehicle speed of the vehicle V increases. The gaze point determination unit 19 determines, for example, whether or not a state in which the relative angle between the gaze direction of the driver 9 and the traveling direction D of the vehicle V is greater than the above threshold value has continued for a predetermined period of time or more. When a state in which the relative angle between the gaze direction of the driver 9 and the traveling direction D of the vehicle V is greater than the above threshold value has continued for a predetermined period of time or more, the gaze point determination unit 19 determines that there is a possibility that the driver 9 is not properly checking the traveling direction D of the vehicle V (for example, looking aside, etc.).

[0030] When the state in which the relative angle between the direction of the driver's 9's face and the traveling direction D of the vehicle V is larger than the threshold continues for the above-mentioned predetermined period or longer, the camera ECU 11 issues a warning to the driver 9 via a speaker, display, etc. of the HMI 5. When the state in which the relative angle between the direction of the driver's 9's gaze and the traveling direction D of the vehicle V is larger than the above-mentioned threshold continues for the above-mentioned predetermined period or longer, the camera ECU 11 issues a warning to the driver 9 via a speaker, display, etc. of the HMI 5.

[0031] Next, a description will be given of the processing performed by the camera ECU 11. FIG.

[0032] 3, in step S1, the camera ECU 11 estimates the traveling direction D of the vehicle V. In step S2, the camera ECU 11 determines whether or not the steering position of the vehicle V has been adjusted. If it is determined that the steering position of the vehicle V has been adjusted (step S2: YES), the camera ECU 11 proceeds to step S3. If it is determined that the steering position of the vehicle V has not been adjusted (step S2: NO), the camera ECU 11 proceeds to step S4.

[0033] In step S3, the camera ECU 11 performs an update process of the estimation result of the traveling direction D of the vehicle V. In step S4, the camera ECU 11 estimates the front-rear distance between the driver monitor camera 10 and the driver 9. In step S5, the camera ECU 11 estimates the orientation of the driver monitor camera 10. In step S6, the camera ECU 11 determines the facial orientation and gaze point of the driver 9 with respect to the traveling direction D of the vehicle V based on the facial image of the driver 9, the traveling direction D of the vehicle V, the front-rear distance between the driver monitor camera 10 and the driver 9, and the orientation of the driver monitor camera 10.

[0034] As described above, according to the driver imaging device 1, the traveling direction D of the vehicle V is estimated by the camera ECU 11 of the driver monitor camera 10 to determine the facial orientation and gaze point of the driver 9 relative to the traveling direction D of the vehicle V, so compared to a conventional device in which the mounting position of the driver monitor camera 10, the orientation of the driver monitor camera 10, vehicle interior design information of the vehicle V, the facial orientation of the driver 9, and three-dimensional geometric information in front of the driver's seat in the vehicle interior for judging the gaze point are set in advance and the driver monitor camera 10 is strictly positioned when mounted, it is possible to determine the facial orientation and gaze point of the driver 9 relative to the traveling direction D of the vehicle V without the need for advance setting or strict positioning. In addition, since the traveling direction D of the vehicle V is estimated by the camera ECU 11, the calibration cost can be reduced.

[0035] The camera ECU 11 estimates the traveling direction D of the vehicle V based on the angular velocity information of the yaw rate sensor 3 of the vehicle V or the angular velocity information of the gyro sensor 13 built into the driver monitor camera 10, in addition to the acceleration information of the acceleration sensor 2 or the acceleration information of the in-camera acceleration sensor 12. By estimating the traveling direction D of the vehicle V based on the angular velocity information that changes according to the traveling of the vehicle V, the estimation accuracy of the traveling direction D of the vehicle V can be improved compared to the case where the angular velocity information is not taken into consideration.

[0036] When the driver's seat of the vehicle V is an electric seat, the camera ECU 11 estimates the front-rear distance between the driver monitor camera 10 and the driver 9 based on the front-rear position information of the driver's seat, and determines the facial direction and gaze point of the driver 9 based on the front-rear distance, the facial image of the driver 9, and the traveling direction D. As a result, by estimating the front-rear distance between the driver monitor camera 10 and the driver 9 based on the front-rear position information of the driver's seat, which is an electric seat, and determining the facial direction of the driver 9 taking the front-rear distance into consideration, it is possible to improve the accuracy of determining the facial direction and gaze point of the driver 9 compared to when the front-rear distance between the driver monitor camera 10 and the driver 9 is not taken into consideration.

[0037] When the driver monitor camera 10 is a steering column mounted type, the camera ECU 11 performs an update process of the estimation result of the traveling direction D of the vehicle V when the steering position adjustment mechanism 4 of the vehicle V adjusts the steering position. As a result, when the driver monitor camera 10 is a steering column mounted type, the position and orientation of the driver monitor camera 10 are changed when the steering position adjustment mechanism 4 of the vehicle V adjusts the steering position. Therefore, by performing the update process of the estimation result of the traveling direction D of the vehicle V, it is possible to suppress deviation from occurring in the traveling direction D of the vehicle V estimated by the camera ECU 11.

[0038] The camera ECU 11 estimates the direction of the driver monitor camera 10 and the front-rear distance between the driver monitor camera 10 and the driver 9 from the arrangement of the vehicle interior equipment in the image based on the vehicle interior image captured by the driver monitor camera 10, and determines the facial direction and gaze point of the driver 9 based on the direction and front-rear distance of the driver monitor camera 10, the facial image of the driver 9, and the traveling direction D. As a result, by estimating the direction of the driver monitor camera 10 and the front-rear distance between the driver monitor camera 10 and the driver 9 by image processing based on the arrangement of the vehicle interior equipment such as the driver's seat, passenger seat, and ceiling light in the image based on the vehicle interior image captured by the driver monitor camera 10, the accuracy of determining the facial direction and gaze point of the driver 9 can be improved compared to a case where the direction of the driver monitor camera 10 and the front-rear distance are not taken into consideration.

[0039] According to the driver imaging device 1 of the present disclosure, the traveling direction D of the vehicle V can be estimated regardless of the mounting position of the driver monitor camera 10, so there is no need to change the design of the mounting position of the driver monitor camera 10 for each vehicle V. In other words, the degree of freedom in designing the mounting position of the driver monitor camera 10 is improved.

[0040] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments. The present disclosure can be implemented in various forms including the above-described embodiments and various modifications and improvements based on the knowledge of those skilled in the art.

[0041] The traveling direction estimating unit 14 does not have to estimate the traveling direction D of the vehicle V based on the angular velocity information of the yaw rate sensor 3 of the vehicle V and the angular velocity information of the gyro sensor 13 of the driver monitor camera 10.

[0042] The traveling direction estimating unit 14 may estimate the traveling direction D of the vehicle V based only on the detection result of the acceleration sensor 2. This makes it possible to achieve low costs, for example, through CAN [Controller Area Network] communication.

[0043] The face direction determination unit 18 does not necessarily have to determine the face direction of the driver 9 based on the front-rear distance between the driver monitor camera 10 and the driver 9 estimated by the front-rear distance estimation unit 16 and the direction of the driver monitor camera 10 estimated by the camera direction estimation unit 17. The camera ECU 11 does not necessarily have to include the update processing unit 15 and the front-rear distance estimation unit 16.

[0044] When determining the gaze point of the driver 9, the gaze point determination unit 19 does not have to base the determination on the front-to-rear distance between the driver monitor camera 10 and the driver 9 estimated by the front-to-rear distance estimation unit 16 and the direction of the driver monitor camera 10 estimated by the camera orientation estimation unit 17.

[0045] The driver imaging device 1 may include a plurality of driver monitor cameras 10. The driver monitor cameras 10 may be provided, for example, near each of the two front pillars (A pillars) of the vehicle V. This allows for improved performance in determining the face direction of the driver 9 by combining face images of the driver 9 captured from a plurality of angles (internal generation of three-dimensional information or ensuring redundancy in the case of backlighting, etc.). For example, it is possible to improve the accuracy of determining at least one of the driver 9's normal viewing state and inattentive state. In addition, even if it becomes necessary to change the mounting position of the driver monitor camera 10 for each vehicle V or each driver 9, the driver imaging device 1 can estimate the traveling direction D of the vehicle V regardless of the mounting position of the driver monitor camera 10 as described above, so that the performance improvement by the plurality of driver monitor cameras 10 can be realized at low cost.

[0046] The traveling direction of the vehicle V estimated by the traveling direction estimation unit 14 may be stored in, for example, a non-volatile memory. This improves the efficiency of estimating the traveling direction D of the vehicle V, for example, when the vehicle V is started. [Explanation of symbols]

[0047] 1...driver imaging device, 2...acceleration sensor, 3...yaw rate sensor, 4...steering position adjustment mechanism, 9...driver, 10...driver monitor camera, 11...camera ECU, 12...acceleration sensor in camera, 13...gyro sensor, V...vehicle.

Claims

1. A driver imaging device including a driver monitor camera fixed in front of a driver's seat in a vehicle interior to image a driver, The driver monitor camera has a camera ECU, The camera ECU estimates the vehicle's direction of travel based on acceleration information from an acceleration sensor of the vehicle or acceleration information from an in-camera acceleration sensor built into the driver monitor camera, and determines the driver's facial orientation and point of gaze relative to the vehicle's direction of travel based on the driver's facial image captured by the driver monitor camera and the vehicle's direction of travel.

2. 2. The driver imaging device according to claim 1, wherein the camera ECU estimates a traveling direction of the vehicle based on, in addition to the acceleration information, angular velocity information of a yaw rate sensor of the vehicle or angular velocity information of a gyro sensor built into the driver monitor camera.

3. the camera ECU estimates a front-rear distance between the driver monitor camera and the driver based on front-rear position information of the driver's seat when the driver's seat of the vehicle is an electric seat; 3. The driver imaging device according to claim 1, further comprising: a facial direction and a gaze point of the driver are determined based on the front-rear distance, a facial image of the driver, and the traveling direction.

4. 3. The driver imaging device according to claim 1, wherein the camera ECU updates the estimation result of the vehicle's traveling direction when a steering position adjustment mechanism of the vehicle adjusts a steering position of the driver monitor camera mounted on a steering column.

5. the camera ECU estimates a direction of the driver monitor camera and a front-rear distance between the driver monitor camera and the driver from an arrangement of interior equipment of the vehicle in the image based on an interior image captured by the driver monitor camera; 3. The driver imaging device according to claim 1, further comprising: a facial direction and a gaze point of the driver are determined based on a direction of the driver monitor camera, the front-rear distance, a facial image of the driver, and the traveling direction.

Images