Driver monitoring device

Abstract

The driver's face is captured from the front. [Solution] In a driver monitoring device having a function to capture the face of the driver (6) of a vehicle (1), the driver monitoring camera (7) captures an image of the driver's (6) face projected onto the windshield (3) of the vehicle (1), and an image capture screen of the driver's (6) face is obtained based on the captured image of the driver's (6) face.

Description

Technical Field

[0001] The present invention relates to a driver monitoring device.

Background Art

[0002] There is a known driver monitoring device that captures an image of the face of a vehicle driver using a driver monitoring camera installed on a steering column and monitors the expression and face orientation of the driver based on the captured face image of the driver (see, for example, Patent Document 1). In this driver monitoring device, the driver monitoring camera is configured to directly capture the driver's face obliquely from the side.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when the driver's face is configured to be directly captured obliquely from the side by the driver monitoring camera in this way, the driver's face to be captured may be blocked by the driver's arm or hand, and when the driver's face is configured to be captured obliquely from above by the driver monitoring camera, the driver's face to be captured may be blocked by a hat. As a result, there arises a problem that the driver's face cannot be surely captured by the driver monitoring camera. On the other hand, if the driver monitoring camera is arranged so that the driver's face can be captured from the front, such a problem can be avoided, but in this case, there arises a problem that the driver's field of view is obstructed by the driver monitoring camera.

Means for Solving the Problems

[0005] Therefore, according to the present invention, a driver monitoring device having a function to capture the face of a vehicle driver is provided, in which the driver monitor camera captures an image of the driver's face projected onto the vehicle's windshield, and the driver monitor device acquires an image of the driver's face based on the captured image of the driver's face. [Effects of the Invention]

[0006] The driver's face can be reliably captured from the image screen. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is a diagram illustrating the view of the windshield from inside the vehicle. [Figure 2] Figure 2 is a side cross-sectional view of Figure 1. [Figure 3] Figure 3 is a diagram illustrating the view of the windshield from inside the vehicle. [Figure 4] Figure 4 is a side cross-sectional view of Figure 3. [Figure 5] Figure 5 is an enlarged side cross-sectional view of a portion of the windshield. [Figure 6] Figure 6 shows the functional configuration of the driver monitoring device. [Figure 7] Figure 7 is a flowchart for executing the driver monitoring process. [Modes for carrying out the invention]

[0008] Figures 1 and 3 illustrate the view from the driver's perspective, Figure 2 is a side cross-sectional view of Figure 1, and Figure 4 is a side cross-sectional view of Figure 3. Referring to Figures 1 through 4, 1 represents the vehicle, 2 represents the passenger compartment, 3 represents the transparent windshield, 4 represents the steering wheel, 5 represents the dashboard, and 6 represents the driver. Vehicle 1 is also equipped with a driver monitoring device that has the function of capturing the face of the driver 6 of vehicle 1. Based on the captured image of the driver 6's face, this driver monitoring device determines the state of the driver 6 while driving vehicle 1, for example, whether the driver 6 is drowsy, dozing off, or distracted, and has the function of issuing a warning to the driver 6 when it is determined that the driver 6 is drowsy, dozing off, or distracted.

[0009] For this driver monitoring device to function properly, the most important issue is that it is possible to reliably acquire an image of the driver's face while the vehicle 1 is in motion. In Figures 1 and 2, the symbol X indicates the direction of the driver's gaze when the driver 6 is looking ahead in the direction of travel of the vehicle 1 while the vehicle 1 is in motion. This direction of the driver's gaze X roughly coincides with the direction in front of the driver's face. While the vehicle 1 is in motion, this direction of the driver's gaze X, i.e., the direction in front of the driver's face X, is usually not obstructed by the driver's arms, hands, or hat. Therefore, if a driver monitoring camera for capturing the driver's face is installed in the direction of the driver's gaze X and the driver's face is captured from directly in front by this driver monitoring camera, it is possible to reliably acquire an image of the driver's face without it being obstructed by the driver's arms, hands, or hat. However, in this case, the driver's view is obstructed by the driver monitoring camera, which presents a problem.

[0010] Therefore, in this invention, the driver monitor camera 7 captures an image of the driver's face projected onto the windshield 3 of the vehicle 1. In this case, in the embodiments shown in Figures 3 and 4, the driver monitor camera 7 captures an image of the driver's face projected onto the inner surface of the windshield 3 of the vehicle 1, and the driver monitor camera 7 is installed on the dashboard 5 in front of the driver 6 so as not to obstruct the driver's view. In Figure 3, the image of the driver's face projected onto the inner surface of the windshield 3 as seen from the driver monitor camera 7 is graphically shown by a dashed line F, and in Figure 4, the area of ​​the image of the driver's face projected onto the inner surface of the windshield 3 as seen from the driver monitor camera 7 is shown by an arrow F. Also in Figure 4, the shooting range of the driver monitor camera 7 for capturing the image of the driver's face projected onto the inner surface of the windshield 3 is shown by Θ.

[0011] In the embodiments shown in Figures 3 and 4, the central part of the windshield 3 is formed in a slightly concave shape that bulges outward. Therefore, the dimensions of the image of the driver's face projected onto the concave inner surface of the windshield 3, as seen by the driver monitor camera 7, are larger than when the inner surface of the windshield 3 is flat. Consequently, in the embodiments shown in Figures 3 and 4, the accuracy of the image of the driver's face captured by the driver monitor camera 7 is improved. Of course, the present invention can also be applied when the inner surface of the windshield 3 is flat.

[0012] Furthermore, as described above with reference to Figures 1 and 2, when the driver 6 is visually checking the area in front of the vehicle 1 while driving, the driver 6's line of sight X roughly coincides with the direction in front of the driver 6's face. Therefore, when the driver 6's face is projected onto the inner surface of the windshield 3, which is in the driver 6's line of sight X, the image of the driver 6's face projected onto the inner surface of the windshield 3 is the image of the driver 6's face as seen from directly in front of the driver 6. Accordingly, in the embodiments shown in Figures 3 and 4, the shape of the windshield 6 and the installation position of the driver monitor camera 7 are set so that the image of the driver 6's face captured by the driver monitor camera 7 is the image of the driver 6's face as seen from directly in front of the driver 6, along the line of sight of the driver 6 when visually checking the area in front of the vehicle 1.

[0013] Furthermore, in the embodiments shown in Figures 3 and 4, the driver monitor camera 7 is installed to photograph the inner surface of the windshield 3 located above the driver monitor camera 7. With the driver monitor camera 7 installed in this manner, when the driver monitor camera 7 captures an image of the driver's face reflected on the inner surface of the windshield 3, it also simultaneously captures the external scenery above, such as the sky, through the windshield 3. In this case, if the inside of the vehicle compartment 2 is brighter than the outside of the vehicle compartment 2, the inner surface of the windshield 3 acts like a mirror, allowing the driver monitor camera 7 to capture an image of the driver's face reflected on the inner surface of the windshield 3. On the other hand, if the outside of the vehicle compartment 2 is brighter than the inside of the vehicle compartment 2, the light flowing into the driver monitor camera 7 from outside the vehicle compartment 2 becomes stronger, making it almost impossible to see the image of the driver's face reflected on the inner surface of the windshield 3, and thus difficult to capture an image of the driver's face reflected on the inner surface of the windshield 3. Therefore, some countermeasures are necessary.

[0014] Therefore, in one embodiment of the present invention, the portion of the windshield 3 in which at least the image of the driver's face 6 is reflected and captured by the driver monitor camera 7 is provided with a function to suppress the transmission of light from outside the vehicle 1 toward the driver monitor camera 7. An example of a windshield 3 provided with this function to suppress the transmission of light from outside the vehicle 1 toward the driver monitor camera 7 is shown in Figure 5. Figure 5 shows an enlarged side cross-sectional view of the portion of the windshield 3 in which at least the image of the driver's face 6 is reflected and captured by the driver monitor camera 7.

[0015] Referring to Figure 5, in the example shown in Figure 5, the windshield 3 is made of laminated glass consisting of a transparent glass layer 3a, a transparent glass layer 3b, and a transparent resin film layer 3c formed between the transparent glass layers 3a and 3b. In this example, within the windshield 3, the inner surface portion of the windshield (the inner surface portion of the transparent glass layer 3b) in which at least the image of the driver's face is reflected and captured by the driver monitor camera 7 is formed in a cross-sectional waveform that alternately repeats a band-shaped horizontal surface portion P extending in the lateral direction of the vehicle 1 and a band-shaped inclined surface portion Q extending in the lateral direction of the vehicle 1 in the longitudinal direction of the vehicle 1. Furthermore, in this example, a light transmission suppression layer is formed on the band-shaped horizontal surface portion P to suppress the transmission of light from outside the vehicle 1 toward the driver monitor camera 7, and the image of the driver's face reflected on the band-shaped inclined surface portion Q is captured by the driver monitor camera 7.

[0016] In this case, as shown in Figure 5, a light transmission suppression layer is formed across the entire surface of each strip-shaped horizontal portion P to suppress the transmission of light from outside the vehicle 1 toward the driver monitor camera 7. This light transmission suppression layer can be formed, for example, from a light-shielding layer that does not allow light from outside the vehicle 1 toward the driver monitor camera 7 to pass through. Alternatively, this light transmission suppression layer can be formed from an external light reflection layer that reflects light from outside the vehicle 1 toward the driver monitor camera 7. In this case, if the amount of light toward the driver monitor camera 7 is to be reduced by diffusely reflecting the light from outside the vehicle 1 toward the driver monitor camera 7, this external light reflection layer is created by forming a fine uneven shape on the surface of the strip-shaped horizontal portion P of the transparent glass layer 3b.

[0017] On the other hand, this light transmission suppression layer can also be formed from an external light polarization layer that polarizes light directed from outside the vehicle 1 toward the driver monitor camera 7. That is, if the light transmission suppression layer is formed from such an external light polarization layer, then only light vibrating in one direction, such as vertically or horizontally, will pass through the external light polarization layer, thus reducing the amount of light transmitted through the windshield 3, and consequently suppressing the transmission of light directed from outside the vehicle 1 toward the driver monitor camera 7. Thus, in the example shown in Figure 5, the light transmission suppression layer is composed of a light-shielding layer that does not allow light directed from outside the vehicle 1 toward the driver monitor camera 7 to pass through, an external light-reflecting layer that reflects light directed from outside the vehicle 1 toward the driver monitor camera 7, or an external light polarization layer that polarizes light directed from outside the vehicle 1 toward the driver monitor camera 7.

[0018] On the other hand, in the example shown in FIG. 5, each strip-shaped inclined surface portion Q is formed in a concave shape with respect to the interior of the passenger compartment 2, and images of different portions of the face of the driver 6 are reflected on each strip-shaped inclined surface portion Q of the windshield 3 when viewed from the driver monitor camera 7. In this case, the concave shape and the inclination angle of each strip-shaped inclined surface portion Q are set so that the images of the different portions of the face of the driver 6 reflected on each strip-shaped inclined surface portion Q form an image of the face of the driver 6 as a whole when viewed from the driver monitor camera 7. Thus, in the example shown in FIG. 5, the strip-shaped inclined surface portion Q is formed in a concave shape with respect to the interior of the passenger compartment 2, images of different portions of the face of the driver 6 are reflected on each strip-shaped inclined surface portion Q, and the concave shape and the inclination angle of each strip-shaped inclined surface portion Q are set so that an image of the face of the driver 6 is formed as a whole when viewed from the driver monitor camera 7.

[0019] In this case, since the image of the face of the driver 6 captured by the driver monitor camera 7 is an image in a form that combines images of different portions of the face of the driver 6, there may be a case where the facial expression cannot be clearly read from the image. In such a case, it is conceivable to correct the image of the face of the driver 6 captured by the driver monitor camera 7 using an image learning method. In this case, in the embodiment according to the present invention, for example, an image of the face of the driver 6 captured by the driver monitor camera 7 is corrected using a neural network for image recognition.

[0020] That is, in the embodiment according to the present invention, a large number of image data pairs are pre-acquired as training data, where one image in each pair is the image of the face of the driver 6 captured by the driver monitoring camera 7, and the other is the image of the face of the driver 6 captured directly from the front at this time. The weights of the neural network for image recognition are learned so that the image of the face of the driver 6 captured by the driver monitoring camera 7 matches the image of the face of the driver 6 captured directly from the front at that time. When the learning is completed, the learned neural network for image recognition is used to correct the image of the face of the driver 6. That is, the image of the face of the driver captured by the driver monitoring camera 7 is input into this learned neural network for image recognition, and based on the output of this learned neural network for image recognition, the image of the face of the driver 6 when captured directly from the front is obtained. From this image, it is determined whether the driver 6 has drowsiness, is sleeping, or is looking away.

[0021] Next, another embodiment will be described in which the driver monitoring camera 7 is installed so as to photograph the inner surface of the windshield 3 located above, and the driver monitoring camera 7 captures the image of the face of the driver 6 reflected on the inner surface of the windshield 3. In this embodiment, the inner surface portion of the windshield 3 is formed from a smooth surface. Therefore, in this embodiment as well, when the driver monitoring camera 7 captures the image of the face of the driver 6 reflected on the inner surface of the windshield 3, the external scenery above, for example, the sky, is also captured simultaneously through the windshield 3. Therefore, in this embodiment, the image of the face of the driver 6 is extracted from the image captured by the driver monitoring camera 7.

[0022] For example, in this embodiment, using an object detection method from the image captured by the driver monitoring camera 7, the contour of the face image is surrounded by a bounding box. Next, only the portion surrounded by the bounding box is extracted, and thereby the image of the face of the driver 6 is extracted. From this extracted image of the face of the driver 6, it is determined whether the driver 6 has drowsiness, is sleeping, or is looking away.

[0023] Figure 6 shows a functional configuration diagram of the control unit 10 of the driver monitoring device installed in the vehicle 1, where 11 represents the electronic control unit. As shown in Figure 6, the electronic control unit 11 consists of a digital computer and is equipped with a CPU (processor) 13 connected to each other by a bidirectional bus 12, a memory 14 consisting of ROM and RAM, and input / output ports 15. Image data captured by the driver monitoring camera 7 is input to the electronic control unit 11. The control unit 10 of the driver monitoring device also includes a warning device 8 that issues a warning to the driver 6 based on the output signal of the electronic control unit 11, and a vehicle control device 9 that controls the vehicle 1 based on the output signal of the electronic control unit 11.

[0024] Figure 7 shows a routine for performing driver monitoring processing, which is repeatedly executed within the electronic control unit 11 of the control unit 10 of the driver monitoring device. Referring to Figure 7, first, in step 20, the driver monitor camera 7 captures an image of the driver's face projected onto the inner surface of the windshield 3. Next, in step 21, the image of the driver's face captured by the driver monitor camera 7 is adjusted. That is, as described above, either the image of the driver's face captured by the driver monitor camera 7 is corrected, or the image of the driver's face is extracted from the image captured by the driver monitor camera 7. For this purpose, the embodiment according to the present invention includes a face image adjustment unit that corrects the image of the driver's face captured by the driver monitor camera 7, or a face image adjustment unit that extracts the image of the driver's face from the image captured by the driver monitor camera 7. In this case, the electronic control unit 11 constitutes this face image adjustment unit.

[0025] Next, in step 22, a detection process is performed to determine whether the driver 6 is drowsy, dozing off, or distracted, based on the image of the driver 6's face adjusted by the face image adjustment unit. Then, in step 23, it is determined whether the driver 6 is drowsy, dozing off, or distracted, based on the results of this detection process. If it is determined that the driver 6 is drowsy, dozing off, or distracted, the process proceeds to step 24, where the warning device 8 issues a warning to the driver 6 via audio and / or screen display. In addition, if it is determined in step 23 that the driver 6 is dozing off, the vehicle control device 9 can also stop the vehicle 1.

[0026] As an example of application, instead of mounting the rear camera that captures images of the area behind vehicle 1 on the back door or rear spoiler, the rear camera can be positioned to capture an image of the area behind vehicle 1 reflected in the rear window, thereby improving the aesthetics and aerodynamic characteristics.

[0027] Thus, in the embodiment of the present invention, in a driver monitor device having a function to capture the face of the driver 6 of the vehicle 1, the driver monitor camera 7 captures an image of the driver 6's face projected onto the windshield 3 of the vehicle 1, and an image of the driver 6's face is obtained based on the captured image of the driver 6's face. [Explanation of symbols]

[0028] 1 vehicle 3. Windshield 5 Dashboard 6 drivers 7. Driver monitoring camera 10 Control unit of the driver monitoring device

Claims

1. A driver monitoring device having a function to capture the face of a vehicle driver, wherein the driver monitoring camera captures an image of the driver's face projected onto the vehicle's windshield, and the driver monitoring device acquires an image of the driver's face based on the captured image of the driver's face.

2. The driver monitoring device according to claim 1, wherein the shape of the windshield and the installation position of the driver monitoring camera are set such that the image of the driver's face captured is an image of the driver's face as seen from directly in front of the vehicle in the direction of travel, in line with the driver's line of sight when the driver is looking ahead in the direction of travel of the vehicle.

3. The driver monitoring device according to claim 1, wherein the portion of the windshield in which at least the image of the driver's face is reflected and which is captured by the driver monitoring camera is provided with a function to suppress the transmission of light from outside the vehicle toward the driver monitoring camera.

4. The driver monitor device according to claim 3, wherein the inner surface portion of the windshield, in which at least the image of the driver's face is reflected and captured by the driver monitor camera, is formed in a cross-sectional wave shape in which a strip-shaped horizontal surface portion extending in a strip-like manner in the direction of the vehicle and a strip-shaped inclined surface portion extending in a strip-like manner in the direction of the vehicle are alternately repeated in the direction of the vehicle's longitudinal direction, and a light transmission suppression layer is formed on the strip-shaped horizontal surface portion to suppress the transmission of light from outside the vehicle toward the driver monitor camera, and the image of the driver's face reflected on the strip-shaped inclined surface portion is captured by the driver monitor camera.

5. The driver monitor device according to claim 4, wherein the light transmission suppression layer comprises a light-shielding layer that prevents light from passing through from outside the vehicle toward the driver monitor camera, an external light-reflecting layer that reflects light from outside the vehicle toward the driver monitor camera, or an external light-polarizing layer that polarizes light from outside the vehicle toward the driver monitor camera.

6. The driver monitor device according to claim 4, wherein the strip-shaped inclined surface portion is formed in a concave shape that is recessed with respect to the interior of the vehicle, and the concave shape and inclination angle of each strip-shaped inclined surface portion are set such that images of different parts of the driver's face are projected onto each strip-shaped inclined surface portion, and when viewed from the driver monitor camera, the whole image forms the driver's face.

7. The driver monitoring device according to claim 4, further comprising a face image adjustment unit for correcting the image of the driver's face captured by a driver monitoring camera.

8. The driver monitoring device according to claim 1, wherein the inner surface portion of the windshield is formed from a smooth surface, and the device has a face image adjustment unit that extracts an image of the driver's face from an image captured by a driver monitoring camera.

Images