Vehicle display device

Abstract

Provided is a vehicle display device capable of stably detecting the positions of the eyes of a user even when external light is incident. A head-up display device (1) as a vehicle display device according to the present invention projects display light (L1) onto a front windshield (2) of a vehicle, generates and displays a virtual image (V) by the reflected display light (L1), on the front windshield (2), so as to be superimposed on a real scene transparent through the front windshield (2), captures an image of a driver (D) of the vehicle by projecting infrared light (L2) onto the driver (D), and is capable of detecting the positions of the eyes (E) of the driver (D) on the basis of the captured image. A control unit (10) detects the position of a feature point of the face of the driver (D) in the captured image and stores a positional relationship with the detected positions of the eyes (E), and, when it is difficult to detect the positions of the eyes (E) in the captured image, estimates a viewpoint position of the driver (D) on the basis of the position of the feature point detected at this time and the positional relationship stored beforehand.

Description

[Technical field]

[0001] The present invention relates to a display device for a vehicle, such as a head-up display device, that displays a virtual image on a front windshield, combiner, or the like of a vehicle and is capable of detecting the position of a user's eyes. [Background technology]

[0002] A head-up display device generates and displays a virtual image by using display light reflected by a reflective translucent member, which is superimposed on a real scene (the scenery in front of the vehicle) that passes through a reflective translucent member such as the vehicle's front windshield or combiner. By providing the user with the information they desire through a virtual image while minimizing the line of sight movement of the user, such as the driver of the vehicle, it contributes to safe and comfortable vehicle operation.

[0003] In addition, some head-up display devices illuminate the user with infrared light to capture an image of the user and detect the position of the user's eyes (position of the pupils, etc.) based on the captured image, thereby helping to grasp the user's state, such as whether they are looking away or falling asleep.

[0004] For example, the head-up display device described in Patent Document 1 forms a display image by reflecting visible light emitted from a display means toward a user by a combiner member, and is equipped with an infrared irradiating means that irradiates infrared rays toward the user, a mirror member that reflects the visible light emitted from the display means toward the combiner member and transmits the infrared rays reflected by the user and the combiner member, a plurality of imaging means that sense the infrared rays that transmit through the mirror member and capture images of the user from different directions, and an image processing means that calculates the position of the user's eyes based on images captured by the imaging means, making it possible to accurately calculate the position of the user's eyes. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2008-126984 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, when strong external light such as sunlight or street light is incident on a head-up display device, the external light noise is added to the captured image, causing a problem of failure to detect the position of the user's eyes. In the head-up display device described in Patent Document 1, when the infrared component of the external light passes through the mirror member and enters the imaging means, a blown-out noise area due to saturation or the like is formed in the captured image, and even if this noise area does not necessarily overlap with the position of the user's eyes (the position of the eye image) in the captured image, it may cause an obstacle to detecting the position of the eyes.

[0007] The present invention has been made in consideration of the above circumstances, and has an object to provide a display device for a vehicle that can stably detect the position of a user's eyes even when external light is incident. [Means for solving the problem]

[0008] In order to solve the above problems, the present invention provides a display device for a vehicle that projects display light onto a reflective translucent member provided in a vehicle, generates and displays a virtual image by the display light reflected by the reflective translucent member so as to be superimposed on an actual scene transmitted through the reflective translucent member, and projects infrared light onto a user of the vehicle to capture an image of the user, and is capable of detecting a viewpoint position of the user based on the captured image, the display device comprising: eye detection means for detecting a position of the eyes of the user in the captured image; viewpoint position detection means for detecting the viewpoint position based on the position of the eyes detected by the eye detection means; feature point detection means for detecting positions of feature points of the user in the captured image; and storage means for storing a positional relationship between the positions of the eyes detected by the eye detection means and the positions of the feature points detected by the feature point detection means, and when it is difficult for the eye detection means to detect the positions of the eyes, the viewpoint position detection means estimates the viewpoint position based on the positions of the feature points detected by the feature point detection means and the positional relationship stored in the storage means. The eye detection means detects the positions of the left and right eyes of the user in a captured image obtained by periodically or irregularly capturing images while the virtual image is being displayed, and the storage means stores the positional relationship obtained from the captured image when the distance between the left and right eyes becomes maximum after a predetermined reference time.The phrase "when it is difficult for the eye detection means to detect the eye position" refers not only to the case where the eye detection means fails to detect the eye position, but also to the case where a predetermined condition is met that makes it difficult to detect the eye position due to the incidence of external light, etc.

[0009] In addition, the vehicle display device of the present invention may include a face direction detection means for detecting the direction of the user's face, and the viewpoint position detection means may estimate the viewpoint position based on the direction of the face detected by the face direction detection means.

[0011] Alternatively, the vehicle display device according to the present invention may include a gaze information acquisition means for acquiring gaze information of the user, the eye detection means for detecting the position of the eye in an image acquired by regular or irregular imaging during the display of the virtual image, and the storage means for storing the positional relationship obtained from an image captured when the user faces forward based on the gaze information acquired by the gaze information acquisition means. "When the user faces forward" means when the user faces the direction in which the image is captured, and does not only refer to the case where the user's face or gaze is strictly facing the imaging direction, but also includes the case where the face or gaze satisfies a predetermined condition that is considered to be facing the imaging direction. Effect of the Invention

[0012] According to the vehicle display device of the present invention, the position of the user's eyes can be stably detected even when external light is incident thereon. [Brief description of the drawings]

[0013] [Figure 1] 1 is an explanatory diagram showing a head-up display device according to an embodiment of the invention; [Diagram 2] 2 is an explanatory diagram showing an image of a driver captured by the head-up display device of FIG. 1 together with feature points. FIG. [Diagram 3] 2 is an explanatory diagram showing a state in which external light enters the head-up display device of FIG. 1. [Figure 4] FIG. 10 is an explanatory diagram showing a captured image including a noise area caused by external light. [Diagram 5] FIG. 1A is an explanatory diagram showing in a plan view the state in which a driver views a virtual image with his / her face facing forward, and FIG. 1B is an explanatory diagram showing in a plan view the state in which the driver's face is facing diagonally. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described with reference to the drawings.

[0015] 1, a head-up display device (HUD) 1 as a vehicle display device (vehicle-mounted display device) according to this embodiment is provided below a front windshield 2 of the vehicle, and projects display light L1, which is visible light, onto a part of the front windshield 2. The display light L1 is reflected by the front windshield 2 to generate a virtual image V, which is visually recognized by a driver D of the vehicle as being superimposed on an actual scene transmitted through the front windshield 2.

[0016] In addition, the HUD 1 has the function of a DMS (driver monitoring system) that monitors the condition of the driver D, and can project infrared light L2 onto the driver D to capture an image of the driver D and detect the position of the driver D's eyes E based on the captured image.

[0017] In detail, the HUD 1 is covered by a housing 3 molded from a black ABS resin or the like to prevent external light from entering, and is partitioned from the outside, and a light-transmitting section 4 covered with a transparent resin such as polycarbonate (not shown) is formed in the housing 3. Inside the housing 3, a display unit 5, a folding mirror 6, a concave mirror 7, an infrared light irradiation unit 8, a camera 9, and a control section 10 are held and housed.

[0018] The display unit 5 here is provided with a light source consisting of a chip-type light-emitting diode and a liquid crystal panel, and the liquid crystal panel projects and displays visible light (display light L1) by two-dimensionally modulating the light emitted from the light source, but a reflective device such as a DMD (digital micromirror display) may also be used. The folding mirror 6 is made of resin such as polycarbonate molded to have a flat portion and has a metal such as aluminum vapor-deposited on it, and simply reflects light. The concave mirror 7 is made of resin such as polycarbonate molded to have a concave portion and has a metal such as aluminum vapor-deposited on it, and has the property of magnifying and reflecting visible light while transmitting infrared light.

[0019] The infrared light irradiation unit 8 is provided on the rear side of the concave mirror 7 (the side opposite the light-transmitting section 4 and the folding mirror 6 with respect to the concave mirror 7), and irradiates infrared light (near-infrared light) emitted by a light source consisting of a light-emitting diode toward the concave mirror 7. The camera 9 includes an imaging element sensitive to the infrared light in the wavelength band irradiated from the infrared irradiation unit 8, and a lens capable of transmitting the infrared light and forming an image on the imaging element, and captures a near-infrared image.

[0020] The control unit 10 is composed of a microprocessor, a memory, and various electronic components, a board, and a case for operating them, and controls the display unit 5 so as to appropriately display the image of the HUD 1 based on vehicle information and input information from the driver D.

[0021] 2, the control unit 10 detects the positions of the pupils C (or the centers of the irises I) of the left and right eyes E of the driver D based on the contrast of the image P captured by the camera 9, and detects the position of the midpoint C1 between the left and right pupils C as the viewpoint position. The detected viewpoint position can be used to detect the state of the driver D (looking aside, falling asleep, etc.) and to control the display of the HUD 1.

[0022] Furthermore, based on the contrast of the captured image P captured by the camera 9, the control unit 10 detects the positions of feature points T such as the contour F1 (including the tip of the chin F2) of the driver D's face F, the left and right ears Y, the centers Y1 of the left and right ears Y, the left and right eyebrows B, the centers B1 of the left and right eyebrows B, the space between the eyebrows Q, the contours E1 of the left and right eyes E (including the outer corners E2 and inner corners E3 of the eyes), the contour N1 of the nose N (including the left and right nostrils N2), the center N3 of the left and right nostrils N2, the left and right nostrils N4, the center N5 of the left and right nostrils N4, the bridge of the nose N6, the tip of the nose N7, the philtrum R, the contour M1 of the mouth M (including the left and right corners of the mouth M2), and the center M3 of the left and right corners of the mouth M2, and stores the positional relationship between the detected position of the pupil C and the detected position of the feature point T in an internal memory (hereinafter, the captured image used to acquire the positional relationship stored in the control unit 10 is also referred to as the "reference image").

[0023] In the HUD 1, the display light L1 from the display unit 5 is reflected by the folding mirror 6, then is magnified and reflected by the concave mirror 7, passes through the light-transmitting portion 4, and is projected onto the windshield 2. The display light L1 projected onto the windshield 2 is magnified and reflected toward the driver D, generating a virtual image V, which is superimposed on the real view transmitted through the windshield 2 and displayed to the driver D.

[0024] On the other hand, infrared light L2 from the infrared light irradiation unit 8 transmits through the concave mirror 7, passes through the light-transmitting portion 4, is projected onto the windshield 2, and is reflected by the windshield 2 toward the driver D to irradiate the driver D. Then, when reflected by the driver D, part of the infrared light L2 follows the reverse path, transmits through the concave mirror 7, and is incident on the camera 9, whereby the driver D is imaged, and the image P is input to the control unit 10. This imaging is performed periodically or irregularly while the virtual image V is displayed, and here, while the virtual image V is displayed, video imaging is performed at a regular frame rate.

[0025] When the control unit 10 receives an image P of the driver D, it detects the positions of the pupils C of the left and right eyes E of the driver D and detects the position of the midpoint C1 between the left and right pupils C as the viewpoint position. However, while an image P of the driver D as shown in Fig. 2 should be obtained, as shown in Fig. 3, when infrared components of strong external light L3 such as sunlight are transmitted through the front windshield 2, the light-transmitting portion 4, and the concave mirror 7 and enter the camera 9, a blown-out noise region S as shown in Fig. 4 appears in the image P, which may interfere with the detection of the position of the driver D's eye E (the right eye E in Fig. 4).

[0026] Therefore, the control unit 10 acquires and stores the positional relationship between the positions of the pupils C and the positions of the feature points T from an image P that has been captured normally (with the face F not obscured by the noise areas S) in advance, and when it is difficult to detect the position of the pupils C in an image P captured thereafter due to the presence of the noise areas S, the control unit 10 estimates the positions of the left and right pupils C, and ultimately the viewpoint position (the position of the midpoint C1), based on the position of the feature points T detected from the image P and the previously stored positional relationship.

[0027] In the HUD 1 according to this embodiment, the control unit 10 detects the positions of the eyes E (pupils C) of the driver D in the captured image P, and detects the viewpoint position based on the detected positions of the eyes E. Furthermore, the control unit 10 detects the positions of feature points T (preferably the positions of a plurality of feature points T) of the face F of the driver D in the captured image P (reference image), stores the positional relationship between the detected positions of the eyes E and the detected positions of the feature points T, and when it is difficult to detect the position of the eyes E, estimates the viewpoint position based on the position of the feature points T detected at that time and the previously stored positional relationship.

[0028] Therefore, even if it is difficult to detect the position of one or both of the left and right eyes E due to the influence of external light L3, if the position of the feature point T can be obtained by imaging, the position of the eye E can be estimated, reducing the probability of failing to detect the position of the eye E, and making it possible to stably detect the position of the eye E even when external light L3 is incident.

[0029] In addition, since the image of the driver D is captured while the virtual image V is being displayed, it can be assumed that the driver D is facing the front of the vehicle (the imaging direction of the camera 9) on the eye box plane of the HUD 1 and viewing the virtual image V (otherwise, the driver D cannot view the virtual image V). On the other hand, since it is easier to estimate the viewpoint position with high accuracy by using the captured image P in which the driver D is facing as far forward as possible as the reference image, the accuracy of estimating the viewpoint position can be improved by capturing the image of the driver D while the virtual image V is being displayed.

[0030] If the reference image is to be an image P in which the driver D is more reliably facing forward, the control unit 10 detects the positions of the left and right eyes E of the driver D in the image P obtained by regular or irregular imaging while the virtual image V is displayed, updates the reference image by using the image P at which the distance between the left and right eyes E (here, the distance between the left and right pupils C) is greatest since a predetermined reference time (which may be the time when the display of the virtual image V begins), and calculates and stores the positional relationship from the reference image.

[0031] In other words, the distance between the left and right eyes E becomes greater the more the driver D faces forward, so if the distance between the left and right eyes E in a newly captured image P is greater than the distance between the left and right eyes E in the captured image P used as the reference image, the newly captured image P is used as the reference image and the positional relationship obtained from it is overwritten in the control unit 10, updating the positional relationship stored in the control unit 10, making it possible to accurately estimate the viewpoint position based on the positional relationship when the driver D is more reliably facing forward.

[0032] Alternatively, if the control unit 10 acquires information on the line of sight of the driver D based on the captured image P and it is considered that the driver D is facing forward based on the acquired line of sight information, the captured image P at that time is used as a reference image and the positional relationship obtained therefrom is stored, thereby making it possible to accurately estimate the viewpoint position based on the positional relationship in a situation where the driver D is more surely facing forward. If it is necessary to increase the accuracy of the driver D facing forward in the reference image, when it is considered that the line of sight of the driver D in a newly captured captured image P is more facing forward than the line of sight of the driver D in the captured image P used as the reference image at a certain point in time, the new captured image P may be used as a reference image and the positional relationship obtained therefrom may be overwritten in the control unit 10 to update the positional relationship stored in the control unit 10.

[0033] In addition, the control unit 10 may also detect the direction of the face F of the driver D from the position of the feature point T in the captured image P, and when it is difficult to detect the position of one or both of the left and right eyes E due to the influence of external light L3, the control unit 10 may estimate the viewpoint position based on the position of the feature point T detected at that time, the pre-stored positional relationship, and the direction of the face F.

[0034] For example, as shown in FIG. 5, when the control unit 10 stores the distance K between the pupil C and the ear Y of the driver D as one of the positional relationships (in principle, the positional relationship can be grasped on the captured image P, but in FIG. 5, the distance between the ear Y and the pupil C of the driver D is shown for convenience), if it is difficult to detect the position of the eyes E and the face F of the driver D is facing forward (FIG. 5(a)), the control unit 10 estimates the viewpoint position using the distance K, but if it is difficult to detect the position of the eyes E and the face F of the driver D is facing diagonally by an angle θ (FIG. 5(b)), the control unit 10 estimates the viewpoint position using the distance Kcosθ instead of the distance K. In other words, even if the left and right pupils C are shifted in the fore-and-aft direction (depth direction) of the vehicle and the distance between them (pupillary distance) is narrow in the left-right direction of the vehicle, it is possible to suppress a situation in which the estimation accuracy of the position of the pupil C decreases.

[0035] Although the embodiment of the present invention has been described above, the embodiment of the present invention is not limited to the above, and may be appropriately modified without departing from the spirit of the invention.

[0036] For example, in the above embodiment, the display light is projected onto the vehicle's front windshield as a reflective translucent member, but a combiner may be used in place of the front windshield.

[0037] In addition, the position of one eye E may be estimated using the positional relationship obtained from the other eye E. If the position of the pupil C of the right eye E cannot be obtained but the positions of the outer corner E2 and inner corner E3 of the eye can be obtained, and the positions of the pupil C, outer corner E2 and inner corner E3 of the left eye E can be obtained, the position of the pupil C of the right eye E may be estimated based on the positional relationship between the pupil C, outer corner E2 and inner corner E3 of the left eye E, and the positions of the outer corner E2 and inner corner E3 of the right eye E.

[0038] Furthermore, after the driver D sits in the driver's seat of the vehicle, an image may be captured as a calibration image for the DMS in a state where the face F faces the HUD display (virtual image V), and this captured image may be used as a reference image. [Explanation of symbols]

[0039] 1 Head-up display device (vehicle display device) 2 Front windshield (reflective and translucent material) 9. Camera 10 Control unit (eye detection means, viewpoint position detection means, feature point detection means, storage means, face direction detection means, gaze information acquisition means) C Pupil D Driver (user) E eyes F face L1 display light L2 infrared light P Captured image T feature points V Virtual Image

Claims

1. A display device for a vehicle, which projects display light onto a reflective translucent member provided in a vehicle, generates and displays a virtual image by the display light reflected by the reflective translucent member in a manner superimposed on an actual scene transmitted through the reflective translucent member, and projects infrared light onto a user of the vehicle to capture an image of the user, and detects a viewpoint position of the user based on the captured image, eye detection means for detecting the position of the user's eyes in the captured image; a viewpoint position detection means for detecting the viewpoint position based on the eye positions detected by the eye detection means; a feature point detection means for detecting the positions of feature points of the user in the captured image; a storage means for storing a positional relationship between the positions of the eyes detected by the eye detection means and the positions of the feature points detected by the feature point detection means, the viewpoint position detection means, when it is difficult for the eye detection means to detect the eye position, estimates the viewpoint position based on the feature point positions detected by the feature point detection means and the positional relationship stored in the storage means; the eye detection means detects positions of the left and right eyes of the user in a captured image obtained by periodically or irregularly capturing an image while the virtual image is being displayed; The storage means stores the positional relationship obtained from a captured image when the distance between the left and right eyes becomes maximum after a predetermined reference time.

2. A face direction detection means for detecting a face direction of the user, 2. The display device for a vehicle according to claim 1, wherein the viewpoint position detection means estimates the viewpoint position based on the face direction detected by the face direction detection means.

3. A gaze information acquisition means for acquiring gaze information of the user, the eye detection means detects the positions of the eyes in captured images obtained by periodically or irregularly capturing images while the virtual image is being displayed; 3. The vehicle display device according to claim 1, wherein the storage means stores the positional relationship obtained from a captured image when the user is facing forward based on the line-of-sight information acquired by the line-of-sight information acquisition means.

Images