Vehicle display device

Abstract

The present invention provides a vehicle display device that can stably detect the position of a user's eyes even when external light is incident. The head-up display device (1) of the present invention, as a vehicle display device, can project display light (L1) onto the front windshield (2) of the vehicle, generate a virtual image (V) by using the display light (L1) reflected by the front windshield (2) in a manner superimposed with a real scene passing through the front windshield (2), and display the virtual image, and project infrared light (L2) onto the driver (D) of the vehicle to photograph the driver (D), and detect the position of the driver (D)'s eyes (E) based on the photographed image, wherein the control unit (10) detects the position of the feature points of the driver (D)'s face in the photographed image and pre-stores the positional relationship between the feature points and the detected eye (E) position. In the case where it is difficult to detect the eye (E) position in the photographed image, the viewpoint position of the driver (D) is estimated based on the position of the feature points detected at this time and the pre-stored positional relationship.

Description

Technical Field

[0001] The present invention relates to a vehicle display device such as a head-up display device that can display a virtual image on a front windshield, combiner, etc. of a vehicle and detect the position of a user's eyes. Background Art

[0002] A head-up display device generates and displays a virtual image using display light reflected by reflective and translucent components, such as the vehicle's front windshield and combiner, superimposed on the real scene (scenery ahead of the vehicle). This image is used to provide the user with desired information using the virtual image while minimizing the need for the driver or other user to shift their line of sight. This contributes to safe and comfortable vehicle operation.

[0003] Furthermore, in a head-up display device, infrared light is irradiated onto a user to capture the user's image. The user's eye positions (such as the position of the pupil) are detected based on the captured image, thereby understanding the user's state of looking around or dozing off.

[0004] For example, the head-up display device described in Patent Document 1 forms a display image by reflecting visible light emitted from a display mechanism toward a user via a combiner component. The head-up display device includes: an infrared irradiation mechanism that irradiates infrared light toward the user; a reflector component that reflects visible light emitted from the display mechanism toward the combiner component and transmits infrared light reflected by the user and the combiner component; multiple imaging mechanisms that sense the infrared light transmitted through the reflector component and capture images of the user from different directions; and an image processing mechanism that calculates the user's eye position based on the images captured by the imaging mechanisms. This head-up display device can calculate the user's eye position with high accuracy.

[0005] Prior art literature

[0006] Technical Literature

[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 2008-126984 Summary of the Invention

[0008] Problems to be solved by the invention

[0009] However, there is a problem: when strong external light such as sunlight or streetlights enters the head-up display, it interferes with the captured image, causing failure to detect the user's eye position. In the head-up display described in Patent Document 1, when the infrared component of the external light passes through the reflector and enters the camera mechanism, it creates an overexposed interference area in the captured image due to factors such as color saturation. Even if this interference area does not necessarily coincide with the user's eye position (the position of the eye image) in the captured image, it can hinder eye position detection.

[0010] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a vehicle display device that can stably detect the position of the user's eyes even when external light is incident.

[0011] Means for solving problems

[0012] In order to solve the above-mentioned problems, the vehicle display device of the present invention is capable of projecting display light onto a reflective and translucent component provided on a vehicle, generating and displaying a virtual image by using the display light reflected by the reflective and translucent component in a manner superimposed with a real scene passing through the reflective and translucent component, and projecting infrared light onto a user of the vehicle to photograph the user, and detecting the viewpoint position of the user based on the photographed image. It is characterized in that it comprises: an eye detection mechanism that detects the position of the user's eyes in the photographed image; a viewpoint position detection mechanism that detects the viewpoint position based on the position of the eyes detected by the eye detection mechanism; a feature point detection mechanism that detects the position of the feature points of the user in the photographed image; and a storage mechanism that stores the positional relationship between the position of the eye detected by the eye detection mechanism and the position of the feature points detected by the feature point detection mechanism. When it is difficult to detect the position of the eye by the eye detection mechanism, the viewpoint position detection mechanism estimates the viewpoint position based on the position of the feature points detected by the feature point detection mechanism and the positional relationship stored in the storage mechanism. "The situation where it is difficult to detect the position of the eye by the eye detection mechanism" does not only refer to the situation where the eye detection mechanism fails to detect the position of the eye, but also includes the situation where the specified conditions that the position of the eye is considered difficult to detect due to the incidence of external light, etc. are met.

[0013] Furthermore, the vehicle display device of the present invention may include a facial orientation detection mechanism that detects the orientation of the user's face, and the viewpoint position detection mechanism may estimate the viewpoint position based on the facial orientation detected by the facial orientation detection mechanism.

[0014] Alternatively, the eye detection mechanism detects the positions of the left and right eyes of the user in a captured image obtained by periodically or irregularly capturing the virtual image during the display of the virtual image, and the storage mechanism stores the positional relationship obtained from the captured image when the distance between the left and right eyes becomes the maximum after a specified reference time.

[0015] Alternatively, the vehicle display device of the present invention may include a sight line information acquisition mechanism for acquiring sight line information of the user, the eye detection mechanism detecting the position of the eye in an image captured by periodically or irregularly capturing the user during the display of the virtual image, and the storage mechanism storing the positional relationship obtained from an image captured when the user is facing forward based on the sight line information acquired by the sight line information acquisition mechanism. "When the user is facing forward" refers to when the user is facing the direction to be captured, and does not only refer to a situation where the user's face or sight line is strictly facing the capturing direction, but also includes a situation where the user satisfies a specified condition that the user's face or sight line is facing the capturing direction.

[0016] Effects of the Invention

[0017] According to the vehicle display device of the present invention, the position of the user's eyes can be stably detected even when external light is incident. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is an explanatory diagram showing a head-up display device according to a specific embodiment.

[0019] Figure 2 It is shown together with the feature points based on Figure 1 An explanatory diagram of an image captured by a driver of a head-up display device.

[0020] Figure 3 It shows the intrusion of external light Figure 1 An explanatory diagram of a head-up display device.

[0021] Figure 4 1 is an explanatory diagram showing a captured image including an interference region caused by external light.

[0022] Figure 5 (a) is an explanatory diagram showing a case where the driver visually recognizes a virtual image and faces forward when viewed from above, and (b) is an explanatory diagram showing a case where the driver faces obliquely when viewed from above. DETAILED DESCRIPTION

[0023] Modes for carrying out the present invention will be described with reference to the drawings.

[0024] like Figure 1 As shown, a head-up display device (HUD) 1 serving as a vehicle display device (on-board display) according to this embodiment is installed below a front windshield 2 of a vehicle and projects display light L1, which is visible light, onto a portion of the front windshield 2. The display light L1 is reflected by the front windshield 2 to generate a virtual image V, which is visually recognized by a driver D of the vehicle in a manner superimposed on a real scene transmitted through the front windshield 2.

[0025] Furthermore, the HUD 1 has a DMS (Driver Monitoring System) function for monitoring the state of the driver D, can project infrared light L2 toward the driver D to capture an image of the driver D, and can detect the position of the driver D's eyes E based on the captured image.

[0026] Specifically, the HUD 1 is shielded from the outside by a housing 3, which is molded from black ABS resin or other materials to prevent the intrusion of external light. A light-transmitting portion 4 (not shown) is formed on the housing 3 and is covered with a transparent resin such as polycarbonate. Housing 3 holds and houses a display unit 5, a folding mirror 6, a concave mirror 7, an infrared light irradiation unit 8, a camera 9, and a control unit 10.

[0027] Here, the display unit 5 is provided with a light source composed of a chip-type light-emitting diode and a liquid crystal panel. The liquid crystal panel performs two-dimensional modulation on the light emitted by the light source to project and display visible light, i.e., image light (display light L1). However, a reflective device such as a DMD (digital micromirror display) can also be used. The folding mirror 6 is formed by vapor-depositing a metal such as aluminum on a resin such as polycarbonate having a flat portion, and simply reflects light. The concave mirror 7 is formed by vapor-depositing a metal such as aluminum on a resin such as polycarbonate having a concave portion, and has the characteristics of amplifying and reflecting visible light while allowing infrared light to pass through.

[0028] The infrared light irradiation unit 8 is disposed on the back side of the concave mirror 7 (on the opposite side of the concave mirror 7 from the light-transmitting portion 4 and the folding mirror 6). It irradiates the concave mirror 7 with infrared light (near-infrared light) emitted by a light source composed of a light-emitting diode. The camera 9 includes an imaging element sensitive to infrared light in the wavelength band irradiated from the infrared light irradiation unit 8 and a lens capable of transmitting infrared light to form an image on the imaging element. The camera 9 captures a near-infrared image.

[0029] The control unit 10 is composed of a microprocessor, a memory, various electronic components for operating them, a substrate, and a housing, and controls the display unit 5 to appropriately display images of the HUD 1 based on vehicle information and input information from the driver D.

[0030] In addition, if Figure 2 As shown, the control unit 10 detects the positions of the pupils C (or the center of the iris I) of the left and right eyes E of the driver D based on the contrast of the image P captured by the camera 9, and detects the position of the middle point C1 between the left and right pupils C as the viewpoint position. The detected viewpoint position can be used to detect the state of the driver D (looking around, dozing off, etc.) and to control the display of the HUD 1.

[0031] Furthermore, based on the contrast of the captured image P of the camera 9, the control unit 10 detects the positions of characteristic points T of the driver D, such as the contour F1 of the face F (including the front end F2 of the mandible), the left and right ears Y, the centers Y1 of the left and right ears Y, the left and right eyebrows B, the centers B1 of the left and right eyebrows B, the space between the eyebrows Q, the contours E1 of the left and right eyes E (including the outer corners E2 and the inner corners E3), the contour N1 of the nose N (including the left and right nostrils N2), the centers N3 of the left and right nostrils N2, the left and right nostrils N4, the centers N5 of the left and right nostrils N4, the bridge of the nose N6, the tip of the nose N7, the philtrum R, the contour M1 of the mouth M (including the left and right corners M2 of the mouth), the centers M3 of the left and right corners of the mouth M2, and stores the positional relationship between the position of the detected pupil C and the position of the detected characteristic point T in the internal memory (hereinafter, the captured image used to obtain the positional relationship stored in the control unit 10 will also be referred to as the "reference image").

[0032] In the HUD 1, display light L1 from the display unit 5 is reflected by the folding mirror 6, then amplified and reflected by the concave mirror 7, and projected onto the front windshield 2 through the light-transmitting portion 4. The display light L1 projected onto the front windshield 2 is amplified and reflected toward the driver D, generating a virtual image V that is displayed to the driver D in a manner superimposed on the real scene transmitted through the front windshield 2.

[0033] On the other hand, infrared light L2 from the infrared light irradiation unit 8 passes through the concave mirror 7, is projected onto the front windshield 2 via the light-transmitting portion 4, and is reflected by the front windshield 2 toward the driver D, irradiating the driver D. Furthermore, when reflected by the driver D, a portion of the infrared light L2 travels along the opposite path, and the infrared light L2 that passes through the concave mirror 7 and enters the camera 9 captures the driver D. This captured image P is input to the control unit 10. This capture is performed periodically or irregularly while the virtual image V is being displayed. Here, while the virtual image V is being displayed, dynamic image capture is performed at a regular frame rate.

[0034] When the captured image P of the driver D is input, the control unit 10 detects the positions of the pupils C of the left and right eyes E of the driver D and detects the position of the middle point C1 between the left and right pupils C as the viewpoint position. Figure 2 The captured image P of the driver D is shown, but Figure 3 As shown in FIG. 1 , when the infrared component of strong external light L3 such as sunlight passes through the front windshield 2, the light-transmitting portion 4 and the concave mirror 7 and enters the camera 9, the following image is generated on the captured image P: Figure 4 The overexposed interference area S shown may obstruct the driver's D's eyes E (in Figure 4 The middle is the detection of the position of the right eye E).

[0035] Therefore, the control unit 10 obtains and stores the positional relationship between the position of the pupil C and the position of the feature point T from the captured image P that is captured normally (when the face F is not blocked by the interference area S) in advance. Then, for the captured image P, when it is difficult to detect the position of the pupil C due to the existence of the interference area S, the positions of the left and right pupils C are estimated based on the position of the feature point T detected from the captured image P and the above-mentioned positional relationship stored in advance, and then the viewpoint position (the position of the middle point C1) is estimated.

[0036] In the HUD 1 of this embodiment, the control unit 10 detects the position of the driver D's eye E (pupil C) in the captured image P and detects the viewpoint position based on the detected position of the eye E. Furthermore, the control unit 10 detects the positions of feature points T (preferably, the positions of a plurality of feature points T) on the driver D's face F in the captured image P (reference image), stores the positional relationship between the detected position of the eye E and the positions of the detected feature points T, and, if it is difficult to detect the position of the eye E, estimates the viewpoint position based on the positions of the detected feature points T and the pre-stored positional relationship.

[0037] Therefore, even if it is difficult to detect the position of one or both of the left and right eyes E due to the influence of external light L3, the position of the eye E can be estimated as long as the position of the feature point T is obtained through shooting, thereby reducing the probability of failure in detecting the position of the eye E. Even if external light L3 is incident, the position of the eye E can be stably detected.

[0038] Furthermore, since the driver D is photographed while the virtual image V is being displayed, it can be assumed that the driver D is visually recognizing the virtual image V while facing the front of the vehicle (the photographing direction of the camera 9) within the eye movement range plane of the HUD 1 (otherwise, the driver D would not be able to visually recognize the virtual image V). On the other hand, using the captured image P of the driver D facing as far forward as possible as a reference image makes it easier to estimate the viewpoint position with high accuracy. Therefore, by photographing the driver D while the virtual image V is being displayed, the accuracy of the viewpoint position estimation can be improved.

[0039] If the captured image P of the driver D facing forward more accurately is used as the reference image, the control unit 10 detects the positions of the left and right eyes E of the driver D in the captured image P obtained by regular or irregular shooting during the display of the virtual image V, and uses the captured image P when the distance between the left and right eyes E (here, the distance between the left and right pupils C) becomes the largest after a specified reference time (which can also be the time point when the virtual image V starts to be displayed) as the reference image, thereby updating the reference image, and calculating and storing the positional relationship based on the reference image.

[0040] That is, the more accurately the driver D faces the front, the greater the distance between the left and right eyes E. Therefore, if the distance between the left and right eyes E in the newly captured image P is greater than the distance between the left and right eyes E in the image P serving as the reference image, the new image P is used as the reference image and the positional relationship obtained therefrom is rewritten to the control unit 10, and the positional relationship stored in the control unit 10 is updated. As a result, the viewpoint position can be accurately estimated based on the positional relationship when the driver D faces the front more accurately.

[0041] Alternatively, if the control unit 10 acquires information related to the driver D's line of sight based on the captured image P and, based on this acquired line of sight information, deems the driver D to be facing forward, the captured image P at that time is used as a reference image and the positional relationship obtained therefrom is stored. This allows accurate estimation of the viewpoint position based on the positional relationship under conditions where the driver D is more reliably facing forward in the reference image. If the accuracy of the driver D's frontal orientation is improved in the reference image, then if it is determined that the driver D's line of sight in a newly captured captured image P is more forward than in a captured image P that served as the reference image at a certain point in time, the newly captured image P can be used as the reference image and the positional relationship obtained therefrom can be overwritten in the control unit 10, thereby updating the positional relationship stored in the control unit 10.

[0042] In addition, the control unit 10 also detects the orientation of the driver D's face F based on the position of the feature point T in the captured image P. When it is difficult to detect the position of one or both of the left and right eyes E due to the influence of external light L3, the viewpoint position is estimated based on the position of the feature point T detected at this time, the pre-stored positional relationship, and the orientation of the face F.

[0043] For example, Figure 5 As shown, the control unit 10 stores the distance K between the pupil C and the ear Y as one of the positional relationships for the driver D (originally, it is sufficient to grasp the positional relationship on the captured image P, but in Figure 5 For convenience, the distance between the ear Y and pupil C of the driver D is shown). If the position of the eye E is difficult to detect and the driver D's face F is facing forward ( Figure 5 (a)), the distance K is used to estimate the viewpoint position, but if the position of the eye E is difficult to detect, the driver D's face F is tilted at an angle θ ( Figure 5 (b)) The viewpoint position is estimated using the distance Kcosθ instead of the distance K. That is, even if the left and right pupils C are located at positions offset in the vehicle's fore-and-aft direction (depth direction), the distance between them (the distance between the pupils) decreases in the vehicle's left-right direction, thus minimizing the decrease in the estimation accuracy of the pupil C position.

[0044] While the embodiments for carrying out the present invention have been exemplified above, the embodiments of the present invention are not limited to the above-described embodiments, and appropriate changes and the like can be made without departing from the spirit of the invention.

[0045] For example, in the above-described embodiment, the display light is projected using the front windshield of the vehicle as a reflective and translucent member, but a combiner may be used instead of the front windshield.

[0046] In addition, the positional relationship obtained from the other eye E can also be used to estimate the position of one eye E. If the position of the pupil C of the right eye E cannot be obtained but the positions of the outer corner E2 and the inner corner E3 can be obtained, and the positions of the pupil C, the outer corner E2 and the inner corner E3 of the left eye E can be obtained, then the position of the pupil C of the right eye E can also be estimated based on the positional relationship between the pupil C and the outer corner E2 and the inner corner E3 obtained for the left eye E and the positions of the outer corner E2 and the inner corner E3 of the right eye E.

[0047] Furthermore, after the driver D sits in the driver's seat of the vehicle, a photograph may be taken with the face F facing the HUD display (virtual image V) for DMS calibration, and the photographed image may be used as a reference image.

[0048] Description of Reference Numerals

[0049] 1: Head-up display device (vehicle display device);

[0050] 2: Front windshield (reflective and light-transmitting component);

[0051] 9: Camera;

[0052] 10: Control unit (eye detection mechanism, viewpoint position detection mechanism, feature point detection mechanism, storage mechanism, face orientation detection mechanism, sight line information acquisition mechanism);

[0053] C: pupil;

[0054] D: driver (user);

[0055] E: Eye;

[0056] F: face;

[0057] L1: display light;

[0058] L2: infrared light;

[0059] P: capture image;

[0060] T: feature point;

[0061] V: virtual image.

Claims

1. A vehicle display device capable of projecting display light onto a reflective and translucent component provided on a vehicle, generating and displaying a virtual image using the display light reflected by the reflective and translucent component in a manner superimposed on a real scene transmitted through the reflective and translucent component, projecting infrared light onto a user of the vehicle to capture the user, and detecting the user's viewpoint position based on the captured image, characterized in that: have: an eye detection mechanism for detecting the position of the user's eyes in the captured image; a viewpoint position detecting mechanism for detecting the viewpoint position based on the eye position detected by the eye detecting mechanism; a feature point detection mechanism for detecting positions of feature points of the user in the captured image; as well as a storage unit that stores a positional relationship between the position of the eye detected by the eye detection unit and the position of the feature point detected by the feature point detection unit, When it is difficult to detect the eye position by the eye detection means, the viewpoint position detection means estimates the viewpoint position based on the position of the feature point detected by the feature point detection means and the positional relationship stored in the storage means. The eye detection mechanism detects the positions of the left and right eyes of the user in a captured image obtained by capturing the image regularly or irregularly during the display of the virtual image. The storage unit stores the positional relationship obtained from an image captured when the distance between the left and right eyes becomes maximum after a predetermined reference time.

2. The vehicle display device according to claim 1, wherein: A facial orientation detection mechanism is provided for detecting the orientation of the user's face. The viewpoint position detection unit estimates the viewpoint position based on the orientation of the face detected by the face orientation detection unit.

3. The vehicle display device according to claim 1 or 2, wherein: A sight line information acquisition unit for acquiring the sight line information of the user is provided. The eye detection means detects the position of the eye in a captured image obtained by capturing the image regularly or irregularly during the display of the virtual image. The storage unit stores the positional relationship obtained from an image captured when the user faces forward based on the line of sight information acquired by the line of sight information acquisition unit.

Images