Vehicle head-up display system

Abstract

The present invention relates to an improved head-up display device for a vehicle. A head-up display system (10) of a vehicle (1) comprises: a head-up display device (70) having a projection device (71) for projecting three-dimensional images (91, 92) in front of an occupant of the vehicle (1), projecting images (93, 94) of objects outside the vehicle in front of the occupant; and an occupant monitoring device (60) having a photographing device (61) for photographing an occupant of the vehicle (1), generating personal data (80) about the occupant's vision based on the image photographed by the photographing device (61). The head-up display device (70) projects three-dimensional images (91, 92) obtained by adjusting the projected images (93, 94) in the three-dimensional images using the personal data (80) about the occupant's vision generated by the occupant monitoring device (60) from the projection device (71).

Description

Technical Field

[0001] The present invention relates to a head-up display system for a vehicle. Background Art

[0002] In automobiles, research on technological innovations for assisting occupants in driving is actively underway.

[0003] For example, a head-up display device that projects an image onto a vehicle's windshield or the like is disclosed. The head-up display device projects, for example, vehicle speed information onto the windshield or the like in front of the occupants. This allows occupants looking ahead to identify the vehicle's speed, etc., without having to significantly shift their gaze.

[0004] Furthermore, in such a head-up display device for a vehicle, it is also conceivable to repeatedly project an attention-calling mark on an object such as a moving object outside the vehicle in front of the vehicle, thereby making the occupant's attention to the object on which the attention-calling mark overlaps.

[0005] However, since the attention-drawing mark is projected so as to overlap with objects outside the vehicle, the occupant may feel uncomfortable or nauseous.

[0006] For example, when projecting a warning sign, the occupant focuses on it. However, because the warning sign is actually projected onto the windshield, for example, which is at a different distance than objects outside the vehicle, the visual line of sight in which the warning sign is easily visible differs from the visual line of sight in which the object outside the vehicle to which the warning sign is intended to be drawn is easily visible. Therefore, if the occupant focuses on one direction, it becomes difficult to see the other. In such a situation, the occupant tends to focus on both directions. The occupant's consciousness tends to focus on these two directions. If even this fails to achieve a successful focus, the occupant may experience nausea due to frequent eye movements.

[0007] Prior art literature

[0008] Patent Literature

[0009] Patent Document 1: Japanese Patent Application Laid-Open No. 2017-207551

[0010] Patent Document 2: Japanese Patent Application Laid-Open No. 2019-026198 Summary of the Invention

[0011] Problems to be solved by the invention

[0012] Therefore, in a head-up display device for a vehicle, it is considered to project the attention-drawing mark not in a two-dimensional manner on a position such as a windshield, but as a three-dimensional projection image utilizing parallax, for example (Patent Documents 1 and 2).

[0013] However, even if the projected image with the attention-drawing mark set to three dimensions is projected overlapping with objects outside the vehicle or projected close to them, if there is a residual offset in the occupant's sense of distance between them, the occupant will still feel uncomfortable and move frequently between them.

[0014] In Patent Documents 1 and 2, the position of the projected image is adjusted up and down, left and right according to the physique or riding posture of the passenger. However, even if the projection position of the image is adjusted up and down, left and right, the difference in the sense of distance between the three-dimensional projected image and objects outside the vehicle cannot be appropriately eliminated.

[0015] Thus, there is a need for improvement in the head-up display apparatus of a vehicle.

[0016] Technical solutions to solve problems

[0017] The head-up display system of a vehicle of the present invention comprises: a head-up display device having a projection device for projecting a three-dimensional image in front of an occupant of the vehicle, projecting an image of an object outside the vehicle in front of the occupant; and an occupant monitoring device having a shooting device for shooting an occupant of the vehicle, generating personal data about the occupant's vision based on the image shot by the shooting device, wherein the head-up display device projects a three-dimensional image from the projection device, the projected image in the three-dimensional image being adjusted using the personal data about the occupant's vision generated by the occupant monitoring device.

[0018] Preferably, the occupant monitoring device generates position information of the face or both eyes of the occupant in the vehicle, and generates information of the distance between the right eye and the left eye of the occupant, that is, the interpupillary distance, as personal data about the occupant's vision based on the image captured by the capture device, and the head-up display device switches the projection of a right-eye image including the projected image and a left-eye image including the projected image as the three-dimensional image, adjusts the position difference or size difference between the projected image in the right-eye image and the projected image in the left-eye image based on the interpupillary distance, and adjusts the position of the projected image in the right-eye image and the position of the projected image in the left-eye image in the same direction based on the position information of the face or both eyes of the occupant in the vehicle.

[0019] Preferably, the occupant monitoring device may capture the face or eyes of the occupant when the vehicle is in a safe state, determine the interpupillary distance based on the shooting positions of the eyes in the captured image of the occupant's eyes, and generate the information on the interpupillary distance.

[0020] Preferably, the occupant monitoring device generates information on the occupant's dominant eye (dominant eye) as personal data about the occupant's vision based on the captured image of the occupant's face formed by the capturing device, and the head-up display device adjusts the position of the projected image so that the projected image in the image on the dominant eye side of the right eye image and the left eye image serving as the three-dimensional image is at a position based on the line of sight between the object outside the vehicle and the dominant eye.

[0021] Preferably, the occupant monitoring device generates information about the occupant's dominant eye as personal data about the occupant's vision based on the captured image of the occupant's face formed by the capturing device, and the head-up display device is adjusted so that the projected image in the image on the dominant eye side of the right eye image and the left eye image as the three-dimensional image is easier to identify than the projected image in the other image.

[0022] Preferably, when the vehicle is in a safe state, the occupant monitoring device causes the head-up display device to project a projection image for determining the dominant eye, captures the face or eyes of the occupant in the state of projecting the projection image for determining the dominant eye, and judges the dominant eye of the occupant based on the state of the eyes in the captured image, thereby generating information on the dominant eye of the occupant.

[0023] Preferably, the occupant monitoring device may generate information on the visual acuity or refractive power of the occupant's eyes as personal data about the occupant's vision based on the captured images of the occupant's eyes formed by the capturing device, and the head-up display device may adjust the difference in position or size between the projected image in the right-eye image and the projected image in the left-eye image serving as the three-dimensional image based on the information on the visual acuity or refractive power of the occupant's eyes.

[0024] Preferably, when the vehicle is in a safe state, the head-up display device projects a projection image for visual acuity determination, captures the face or eyes of the occupant in the state of projecting the projection image for visual acuity determination, and judges the visual acuity or refractive power of the occupant's eyes based on the state of the eyes in the captured image, thereby generating information on the visual acuity or refractive power of the occupant's eyes.

[0025] Preferably, the head-up display device may change and project the projection image for vision assessment in a manner such that the projection image is blurred and then restored, and the head-up display device may capture the face or eyes of the occupant in the state where the projection image for vision assessment restored from blur is projected.

[0026] Effects of the Invention

[0027] In the present invention, a passenger monitoring device generates personal data regarding the occupant's visual sense for a head-up display based on images captured by a camera device that captures the vehicle's occupants. Furthermore, the head-up display uses the personal data regarding the occupant's visual sense generated by the passenger monitoring device to adjust the projected image in the three-dimensional image, and projects the adjusted three-dimensional image from the projection device. This eliminates the difference in distance between the projected image formed by the three-dimensional image and objects outside the vehicle. From the occupant's perspective, the projected image of an object outside the vehicle in front of the occupant appears to be near or overlapping the object in front of the occupant.

[0028] If there is a difference in distance between the three-dimensional projected image and objects outside the vehicle, the occupants may feel uncomfortable and may also feel nauseous due to frequent shifting of their gaze between them. However, the present invention can effectively suppress such situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is an explanatory diagram of a traveling state of a car according to an embodiment of the present invention.

[0030] Figure 2 yes Figure 1 A schematic illustration of the seating state of passengers in a car.

[0031] Figure 3 yes Figure 1 An illustration of the projection state of a three-dimensional image in a car.

[0032] Figure 4 As Figure 3 An explanatory diagram showing a method for adjusting the projection method, projection size, and projection position of a projection image in a three-dimensional image, using a frame mark for attracting attention.

[0033] Figure 5 As Figure 3 An illustration of the right-eye image and left-eye image generated from a three-dimensional image.

[0034] Figure 6 yes Figure 1 A schematic illustration of a control system of an automobile.

[0035] Figure 7 It is recorded in Figure 6 An illustration of an example of personal data stored in the occupant monitoring memory.

[0036] Figure 8 yes Figure 6 Flowchart of the process of registering passengers in the passenger monitoring device.

[0037] Figure 9 yes Figure 6Flowchart of the occupant identification processing performed by the occupant monitoring device.

[0038] Figure 10 yes Figure 6 Flowchart of the occupant monitoring process performed by the occupant monitoring device.

[0039] Figure 11 yes Figure 6 Flowchart of the projection processing of three-dimensional images performed by the HUD device.

[0040] Figure 12 yes Figure 8 FIG. 1 is a flowchart showing details of the process of generating the interpupillary distance information in step ST4.

[0041] Figure 13 yes Figure 8 This is a detailed flowchart of the process of generating the dominant eye information in step ST5.

[0042] Figure 14 yes Figure 8 A detailed flowchart of the process of generating the visual acuity and diopter information in step ST6 is shown.

[0043] Explanation of symbols

[0044] 1: Automobile (Vehicle), 6: Windshield, 10: Control System (Head-Up Display System), 16: Detection ECU, 17: Occupant Monitoring ECU, 18: HUD ECU, 43: Stereo Camera, 60: Occupant Monitoring Device, 61: Interior Camera, 62: Infrared Light, 63: Timer, 64: Occupant Monitoring Memory, 70: HUD Device, 71: Projection Device, 72: HUD Memory, 80: Personal Data, 90: Frame Mark for Attention Calling (Projected Image), 91: Right Eye Image, 92: Left Eye Image, 93: Frame Mark for Attention Calling (Projected Image), 94: Frame Mark for Attention Calling (Projected Image) DETAILED DESCRIPTION

[0045] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0046] Figure 1 It is an explanatory diagram of the traveling state of the automobile 1 according to the embodiment of the present invention.

[0047] Figure 1The car 1 is an example of a vehicle. The car 1 is provided with a passenger compartment 3 in the center of the vehicle body 2. The driver and other passengers riding in the car 1 operate the driving of the car 1. Thus, the car 1 can travel along the lane by automatic driving or manual driving. The car 1 can perform driving assistance when driving based on the driving-related operations performed by the passengers. Such a car 1 assists both driving by automatic driving and driving by manual driving. Figure 1 In the example, another car 1 is driving in the opposite lane, and pedestrians are walking in the roadside area.

[0048] Figure 2 yes Figure 1 Schematic illustration of a seating state of passengers in a car 1. Figure 2 It is from Figure 1 A perspective view of a motor vehicle 1 as viewed from the side.

[0049] exist Figure 2 A seat 4 for a passenger is provided in a vehicle interior 3 of a vehicle 1. A dashboard 5 is provided in front of the seat 4. A windshield 6 is provided above the dashboard 5. While seated in the seat 4, the passenger can see the front of the vehicle 1 through the windshield 6. The passenger can also see other vehicles 1 or pedestrians in front of the vehicle through the windshield 6.

[0050] Figure 3 yes Figure 1 FIG. 1 is an explanatory diagram of a projection state of a three-dimensional image in a car 1. FIG.

[0051] Figure 3 The three-dimensional image is projected onto the windshield 6 by a projection device 71 of a head-up display device (HUD device) 70 provided in the automobile 1 .

[0052] Figure 3 The three-dimensional image is overlapped with the frame mark 90 for calling attention to the objects such as the moving body in front of the vehicle that can be identified. Figure 1 In the case of the driving state, the three-dimensional image is superimposed on the other cars 1 and the frame mark 90 for calling attention is superimposed on the pedestrians.

[0053] For example, even when it is completely dark outside the vehicle, the occupant can easily recognize other cars 1 or pedestrians by superimposing the attention-attracting frame mark 90. ​​The occupant can notice the object on which the attention-attracting frame mark 90 is superimposed.

[0054] However, since the attention-drawing frame mark 90 is projected so as to overlap with objects outside the vehicle, the occupant may feel uncomfortable or nauseous.

[0055] For example, if a frame mark 90 for calling attention is projected, the passenger will focus on the frame mark 90 for calling attention. However, the frame mark 90 for calling attention is actually projected on the windshield 6, which has a different sense of distance from the objects outside the vehicle. Therefore, from the passenger's perspective, the line of sight state in which the frame mark 90 for calling attention is easy to see and the line of sight state in which the object outside the vehicle to which attention is to be called is easy to see are different states. As a result, if the passenger focuses on one side, it is difficult to see the other side. In such a situation, the passenger tries to focus on both sides. The passenger's consciousness often tends to focus on these two sides. When the line of sight cannot be focused even in this way, the passenger feels nauseous due to frequent movement of the line of sight, etc.

[0056] Thus, the HUD device 70 of the automobile 1 needs to be further improved.

[0057] Next, a method of projecting a projected image such as a frame mark for attracting attention, which is improved in this embodiment, will be described.

[0058] Figure 4 As Figure 3 An explanatory diagram showing a method for adjusting the projection method, projection size, and projection position of a projection image in a three-dimensional image, using a frame mark for attracting attention.

[0059] exist Figure 4 In the embodiment of the present invention, the occupant recognizes pedestrians outside the vehicle through the windshield 6 on which a frame mark for calling attention is projected.

[0060] In this case, the passenger's right eye recognizes the pedestrian outside the vehicle through the center of the windshield 6. The attention-attracting frame mark 93 recognized by the passenger's right eye is projected on the center of the windshield 6 intersecting with the right eye's line of sight.

[0061] The occupant's left eye recognizes pedestrians outside the vehicle through the left portion of the windshield 6. The attention-attracting frame mark 94 recognized by the occupant's left eye is projected on the left portion of the windshield 6 intersecting with the left eye's line of sight.

[0062] By projecting the attention-calling frame mark 93 recognized by the right eye and the attention-calling frame mark 94 recognized by the left eye with the projection adjusted in this way, the occupant can easily recognize whether the attention-calling frame mark 90 exists in an overlapping manner so as to physically surround the pedestrian outside the vehicle.

[0063] Figure 5 As Figure 3 An explanatory diagram of a right-eye image 91 and a left-eye image 92 generated from a three-dimensional image.

[0064] Figure 5 The three-dimensional image includes a right-eye image 91 to be recognized by the right eye and a left-eye image 92 to be recognized by the left eye.

[0065] The right-eye image 91 to be recognized by the right eye has a projected image of a frame mark 93 for attracting attention in the center of the image. The remaining portion is projected as a transparent image.

[0066] The left-eye image 92 to be viewed by the left eye has a projected image of a frame mark 94 for drawing attention on the left side of the image. The remaining portion is projected as a transparent image.

[0067] The projection device 71 of the HUD apparatus 70 can project a three-dimensional image on the windshield 6 by, for example, switching between the right-eye image 91 and the left-eye image 92 and projecting them alternately.

[0068] Figure 6 yes Figure 1 A schematic diagram illustrating a control system 10 of a car 1. Figure 6 , a traffic information system 100 is shown together with a control system 10 of a car 1 and communicates with the control system 10 .

[0069] In the automobile 1 , the passenger monitoring device 60 may identify and monitor the passenger in the vehicle during automatic driving without the passenger such as a driver or manual driving by the passenger.

[0070] Furthermore, in the automobile 1 , the HUD device 70 may be capable of projecting an attention-attracting image onto a moving object outside the vehicle and the like ahead of the moving direction of the automobile 1 , thereby performing the attention-attracting operation.

[0071] It is conceivable that these technologies can organically cooperate to identify, for example, the driver and provide advanced driving assistance tailored to the driver, thereby improving the convenience and safety of the automobile 1.

[0072] exist Figure 6 In FIG. 1 , a plurality of control devices constituting a control system 10 of the automobile 1 are represented by respectively assembled control ECUs (Electronic Control Units).

[0073] exist Figure 6Specifically, the control system 10 of automobile 1 includes a drive ECU 11, a steering ECU 12, a brake ECU 13, an automatic driving / driving assistance ECU 14 for the driving control unit, a driving operation ECU 15, a detection ECU 16, an occupant monitoring ECU 17 for the occupant monitoring device 60, a head-up display (HUD) ECU 18 for the head-up display (HUD) 70, an external communication ECU 19, and a UI ECU 20. These multiple control ECUs are directly connected to a cable 26 and are connected to a central gateway (CGW) 27, serving as a relay device, via a vehicle network 25, such as a CAN (Controller Area Network) or a LIN (Local Interconnect Network) employed in automobile 1. Each control ECU can communicate with one another using encrypted data containing its own ID (the source) and the ID of the device being sent to it. Each control ECU can simultaneously broadcast encrypted data to multiple other control ECUs without specifying a destination. Furthermore, each control ECU receives encrypted data with its own ID as the destination for use in its own control. Furthermore, a control ECU can also receive encrypted data with a specific ID other than its own as the destination for use in its own control. When the encrypted data received from each of the plurality of cables 26 is used for the control ECU connected to another cable 26, the central gateway 27 outputs the received encrypted data to the other cable 26. Figure 6 The multiple control ECUs shown are capable of sending and receiving encrypted data to each other.

[0074] The drive ECU 11 receives the encrypted data via the vehicle network 25 and controls a driving source such as an engine or an electric motor (not shown) and a transmission provided in the automobile 1. As a result, the automobile 1 can accelerate.

[0075] The steering ECU 12 receives the encrypted data via the vehicle network 25 and controls a steering device (not shown) provided in the automobile 1. This allows the automobile 1 to change its traveling direction.

[0076] The brake ECU 13 receives encrypted data via the vehicle network 25 and controls a braking device (not shown) installed on the vehicle 1. This allows the vehicle 1 to slow down and stop while in motion. Furthermore, the brake ECU 13 can communicate with the drive ECU 11 via the vehicle network 25 to suppress the rotation of the drive source or change the transmission ratio to decelerate the vehicle 1 while in motion.

[0077] The driving operation ECU 15 is connected to operating components such as a steering wheel 31, a brake pedal 32, an accelerator pedal 33, and a shift lever 34 for operating the vehicle 1. The driving operation ECU 15 outputs encrypted data including operational information regarding the operated operating components to the automatic driving / driving assistance ECU 14 and the like via the vehicle network 25.

[0078] The detection ECU 16 is connected to, for example, a speed sensor 41, an acceleration sensor 42, a stereo camera 43, and a hand-operated sensor 44. The speed sensor 41 detects the speed of the car 1. The acceleration sensor 42 detects the acceleration of the car 1. The stereo camera 43 captures the surroundings of the outside of the car 1. The stereo camera 43 can capture, for example, the front, side, and rear of the car 1. The hand-operated sensor 44 is, for example, provided on the steering wheel 31 and detects changes in electrostatic capacitance or pressure caused by the hand of the passenger operating the steering wheel 31. The detection ECU 16 obtains physical quantities of the connected sensors and the like regarding the driving of the car 1 as physical quantities detected in the car 1, and outputs them to the automatic driving / driving assistance ECU 14 and the like via the vehicle network 25. Alternatively, the detection ECU 16 may analyze the external image formed by the stereo camera 43, detect the presence of other moving objects around the automobile 1, and output information about the type of other moving objects detected, the relative distance from the vehicle, and the direction to the automatic driving / driving assistance ECU 14, HUD ECU 18, etc. through the vehicle network 25.

[0079] The external communication ECU 19 communicates with the outside world of the vehicle 1. For example, the external communication ECU 19 communicates with the communication relay station 101 of the traffic information system 100 and transmits and receives communication data with the server device 102. Furthermore, the external communication ECU 19 also transmits and receives communication data with, for example, other vehicles 110 traveling near the vehicle 1 or pedestrians' (not shown) mobile devices, for example, through V2X communication. When the external communication ECU 19 receives navigation data to the vehicle 1's destination, traffic information, and information about the surrounding conditions of the vehicle 1 through these communications, it can output this information to the autonomous driving / driving assistance ECU 14 and other devices via the vehicle network 25.

[0080] The UIECU 20 is connected to, for example, a display device 51 and an operating device 52. The operating device 52 may include a touch panel or multiple keys superimposed on the display screen of the display device 51. Upon receiving encrypted data related to display, the UIECU 20 causes the display device 51 to display an image. Based on the operation of the operating device 52 in response to the display on the display device 51, the UIECU 20 can generate navigation data, for example, and output this data to the autonomous driving / driving assistance ECU 14 and the like via the vehicle network 25.

[0081] The autonomous driving / driving assistance ECU 14 switches between autonomous driving and driving assistance to control the driving of the vehicle 1. The autonomous driving / driving assistance ECU 14 controls the driving of the vehicle 1 based on various information acquired through the vehicle network 25. For example, in autonomous driving, the autonomous driving / driving assistance ECU 14 outputs control information to the drive ECU 11, steering ECU 12, and brake ECU 13, enabling the vehicle to reach its destination while confirming surrounding safety based on navigation data. In driving assistance, the autonomous driving / driving assistance ECU 14 outputs control information to the drive ECU 11, steering ECU 12, and brake ECU 13 based on operational information, adjusting the amount of operation to account for surrounding safety and other factors.

[0082] The occupant monitoring ECU 17 is connected to an interior camera 61 , an infrared lamp 62 , a timer 63 , and an occupant monitoring memory 64 .

[0083] In order to shoot the interior of the car 1, an interior camera 61 is installed in the cabin 3. The interior camera 61 can also be a stereo camera. The interior camera 61 is installed on the inner surface of the cabin 3, for example, Figure 2 As shown, it is installed on the instrument panel 5 in front of the occupants. This allows the driver and other passengers in the vehicle 1 to be projected onto the image captured by the interior camera 61. To capture the entire interior of the vehicle 1, the interior camera 61 can be a wide-angle camera or a 360-degree camera. The interior camera 61 repeatedly captures the occupants of the vehicle interior 3 at a predetermined interval.

[0084] Infrared light 62 projects infrared light toward the driver and other passengers. The frequency of the light emitted by infrared light 62 may also be a frequency found in natural light or exterior lighting. In this case, infrared light 62 can be distinguished from natural light by adjusting the intensity and pattern of the light emitted. By subtracting the infrared image of interior camera 61 when not projecting infrared light 62 from the infrared image of interior camera 61 when projecting infrared light 62, an infrared image captured by the light projected by infrared light 62 can be obtained.

[0085] The timer 63 measures time.

[0086] The passenger monitoring memory 64 stores programs, setting data, etc. for passenger monitoring. The setting data includes personal data 80 of each passenger registered in advance, which will be described later.

[0087] The occupant monitoring ECU 17 reads and executes a program from the occupant monitoring memory 64 .

[0088] The occupant monitoring ECU 17 , which serves as a control unit of the occupant monitoring device 60 , controls the light projection of the infrared lamp 62 and acquires an image captured by the interior camera 61 .

[0089] The occupant monitoring ECU 17 recognizes and monitors occupants including the driver who are riding in the car 1 based on the acquired captured images.

[0090] The occupant monitoring ECU 17 can generate setting data such as personal data 80 related to the vision of the recognized occupant and output it to the automatic driving / driving assistance ECU 14 , the HUD ECU 18 , and the like via the vehicle network 25 .

[0091] The occupant monitoring ECU 17 can monitor the occupant's riding status based on repeatedly captured images of the identified occupant, determine the position of the occupant's face or eyes in the vehicle, and output the determined information to the HUDECU 18 or the like via the vehicle network 25 .

[0092] Regardless of whether the interior camera 61 is a stereo camera or another type of camera, the occupant monitoring ECU 17 can determine the position of the occupant's face or eyes in the interior 3 of the car 1 based on the camera's installation position, direction, viewing angle, etc. in the interior 3.

[0093] Figure 7 It is recorded in Figure 6 An explanatory diagram of an example of personal data 80 stored in the occupant monitoring memory 64.

[0094] Figure 7 The personal data 80 includes information of a plurality of passengers registered in the passenger monitoring device 60 .

[0095] Figure 7 Each row corresponds to each person.

[0096] Each person's information includes their ID (identification information), a registered image of their face captured by the interior camera 61, and information about the occupant's vision. Information about the occupant's vision includes pupil distance, dominant eye, binocular visual acuity, and binocular diopter. This information is determined based on captured images such as the registered image.

[0097] Figure 6 The HUDECU 18 is connected to the projection device 71 and the HUD memory 72 .

[0098] like Figure 3 As shown, the projection device 71 is arranged in the instrument panel 5 toward the windshield 6. The projection device 71 is as shown in FIG. Figure 5As shown, right-eye image 91 and left-eye image 92 are switched and alternately projected onto windshield 6. Thus, projection device 71 projects a three-dimensional image onto windshield 6 in front of the occupants of automobile 1. A frame mark 90, which draws attention to an object outside the vehicle in front of the occupants, is projected onto windshield 6 at a position in the direction from which the object is viewed through windshield 6. Alternatively, projection device 71 may project an image, such as an arrow mark, drawing attention to the object outside the vehicle, onto windshield 6 as a three-dimensional image near the object.

[0099] The HUD memory 72 stores programs, setting data, etc. for the head-up display. The setting data includes a frame mark 90 for calling attention and an arrow mark for calling attention.

[0100] The HUDECU 18 reads and executes a program from the HUD memory 72 , thereby allowing the HUDECU 18 to function as a control unit of the HUD device 70 .

[0101] When the HUDECU 18 , which serves as a control unit of the HUD device 70 , acquires information about another moving object in front of the automobile 1 from the detection ECU 16 , it selects, for example, a frame mark 90 for calling attention based on the type of the moving object.

[0102] The HUDECU 18 determines and determines the position and size of the attention-drawing frame mark in the right-eye image 91 and the left-eye image 92 so that the frame mark overlaps with other moving objects or is close to other moving objects when viewed from the passenger's perspective.

[0103] The HUDECU 18 generates a right-eye image 91 including the adjusted attention-attracting frame mark 93 and a left-eye image 92 including the adjusted attention-attracting frame mark 94 , and outputs the images to the projection device 71 .

[0104] As a result, the projection device 71 projects the frame mark 93 for attracting attention of the right-eye image 91 and the frame mark 94 for attracting attention of the left-eye image 92 onto the windshield 6 .

[0105] so, Figure 6 The HUD ECU 18 of the HUD device 70 receives information from, for example, a detection device including a stereo camera 43 for capturing images of the exterior of the vehicle and a detection ECU 16, and an occupant monitoring device 60 including an interior camera 61 for capturing images of occupants within the vehicle and an occupant monitoring ECU 17. Based on this information, the HUD ECU 18 projects a three-dimensional image suitable for the occupants onto the windshield 6. In this case, the head-up display system of the automobile 1 is composed of at least the detection device, the occupant monitoring device 60, and the HUD device 70.

[0106] Figure 8 yes Figure 6Flowchart of the process of registering an occupant in the occupant monitoring device 60.

[0107] When a new passenger is registered, the passenger monitoring ECU 17 repeatedly executes Figure 8 processing.

[0108] In step ST1, the occupant monitoring ECU 17 determines whether the car 1 is parked. In this way, the occupant monitoring ECU 17 determines whether the car 1 is in a safe state. If the car 1 is not parked, the occupant monitoring ECU 17 ends the Figure 8 When the automobile 1 is parked, the occupant monitoring ECU 17 advances the process to step ST2.

[0109] In step ST2 , the occupant monitoring ECU 17 issues a new ID for the new occupant.

[0110] In step ST3, the occupant monitoring ECU 17 captures the occupant's face to establish a new occupant registration image. In this case, the occupant monitoring ECU 17 uses the interior camera 61 to obtain an image of at least the occupant's face from the front. The occupant monitoring ECU 17 may also capture an image of the occupant's profile facing the front of the vehicle 1, along with the frontal image of the occupant's face.

[0111] In step ST4, the occupant monitoring ECU 17 generates information on the distance between the new occupant's right and left eyes, i.e., the interpupillary distance. For example, the occupant monitoring ECU 17 determines the imaging positions of the right and left eyes in an image of the occupant's face taken from the front, and generates the interpupillary distance information based on the imaging distance between them.

[0112] In step ST5, the occupant monitoring ECU 17 generates information about the dominant eye of the new occupant. For example, the occupant monitoring ECU 17 causes the projection device 71 of the HUD device 70 to project a projection image for determining the dominant eye, and determines the occupant's dominant eye based on the projected captured image. The projection image for determining the dominant eye can be, for example, a projection image of two bars that appear to overlap in the dominant eye.

[0113] In step ST6, the occupant monitoring ECU 17 generates information on the new occupant's binocular vision and diopter. For example, the occupant monitoring ECU 17 causes the projection device 71 of the HUD device 70 to project a projection image for vision determination, and determines the occupant's binocular vision and diopter based on the projected captured image.

[0114] In step ST7, the passenger monitoring ECU 17 records the personal data 80 on the passenger's vision generated by the above process in the passenger monitoring memory 64 together with the issued ID or the registered image. Figure 7 The personal data 80 of the newly registered user is additionally recorded in the personal data 80.

[0115] Figure 9 yes Figure 6 Flowchart of the occupant identification process performed by the occupant monitoring device 60.

[0116] When a passenger gets on the vehicle 1, the passenger monitoring ECU 17 repeatedly executes Figure 9 processing.

[0117] In step ST11, the passenger monitoring ECU 17 determines whether the passenger is on board the vehicle 1. If the passenger is not on board, the passenger monitoring ECU 17 repeats this determination process. If the passenger is on board, the passenger monitoring ECU 17 advances the process to step ST12.

[0118] In step ST12, the occupant monitoring ECU 17 determines whether a captured image of the vehicle occupant has been acquired by the interior camera 61. If no captured image has been acquired, the occupant monitoring ECU 17 repeats this determination process. If a captured image has been acquired, the occupant monitoring ECU 17 proceeds to step ST13.

[0119] In step ST13, the occupant monitoring ECU 17 recognizes the occupant based on the acquired captured image. The occupant monitoring ECU 17 compares the acquired captured image with the image registered in Figure 7 The plurality of registered images of the personal data 80 are generated, a value of the degree of consistency is generated for each registered image, and the registered user corresponding to the largest value of the degree of consistency is identified as the passenger.

[0120] In step ST14, the passenger monitoring ECU 17 obtains the personal data 80 of the identified registered user from the personal data 80 of the passenger monitoring memory 64 and sets it to be sent. Figure 7 The personal data 80 related to the vision of the occupant is sent to the HUDECU 18. Thus, the HUDECU 18 acquires the registered information related to the vision of the occupant.

[0121] Figure 10 yes Figure 6 Flowchart of the occupant monitoring process performed by the occupant monitoring device 60.

[0122] When a passenger gets on the car 1, the passenger monitoring ECU 17 and Figure 9 The processing is repeated separately Figure 10 processing.

[0123] In step ST21, the occupant monitoring ECU 17 determines whether a new captured image has been acquired by the interior camera 61. If no new captured image has been acquired, the occupant monitoring ECU 17 repeats this determination process. If a new captured image has been acquired, the occupant monitoring ECU 17 proceeds to step ST22.

[0124] In step ST22, the occupant monitoring ECU 17 monitors the occupant. Based on the newly captured image, the occupant monitoring ECU 17 determines the occupant's state of being in the vehicle. For example, if the occupant is dozing off or looking away, the ECU 17 performs appropriate monitoring. The occupant monitoring ECU 17 uses the UIECU 20 to display an image on the display device 51 indicating a forward direction.

[0125] In step ST23 , the occupant monitoring ECU 17 determines the position of the face or eyes of the occupant in the vehicle 1 based on the new captured image formed by the interior camera 61 , and generates position information thereof.

[0126] In step ST24 , the occupant monitoring ECU 17 transmits the generated information on the current position of the occupant's face or both eyes to the HUDECU 18 as personal data 80 .

[0127] In step ST25, the passenger monitoring ECU 17 determines whether to end the passenger monitoring. For example, when the vehicle 1 is parked and the passenger gets off the vehicle, the passenger monitoring ECU 17 determines that the passenger monitoring is to be ended and ends the passenger monitoring. Figure 10 If the monitoring of the occupant is not completed, the occupant monitoring ECU 17 returns the process to step ST21 and repeats the above-mentioned process.

[0128] Figure 11 yes Figure 6 Flowchart of the projection processing of the three-dimensional image performed by the HUD device 70.

[0129] When the passenger is in the car 1, the HUDECU 18 repeatedly executes Figure 11 processing.

[0130] In step ST31, the HUDECU 18 determines whether to obtain information about the moving object in front of the car 1. The HUDECU 18 obtains information about the moving object in front of the car 1 from the detection ECU 16, for example. In this case, the HUDECU 18 determines that the information about the moving object in front of the car 1 has been obtained and advances the processing to step ST32. In other cases, the HUDECU 18 ends the process. Figure 11 processing.

[0131] In step ST32, the HUDECU 18 selects a projection image to be projected in a manner appropriate to the moving object based on the acquired information about the moving object in front of the vehicle 1. For example, the HUDECU 18 selects a mark corresponding to the type of the moving object from among a plurality of attention-calling marks stored in the HUD memory 72 as the projection image to be projected.

[0132] In step ST33, the HUDECU 18 acquires the latest personal data 80. The latest personal data 80 may include Figure 9 The personal data 80 sent by the passenger monitoring ECU 17 in step ST14 and Figure 10 In step ST24, the passenger monitoring ECU 17 sends the personal data 80. In this case, the HUD ECU 18 can obtain the information of the position of the current passenger's face or eyes as the passenger's personal data 80, along with the registered passenger's interpupillary distance, dominant eye, binocular vision, and binocular diopter.

[0133] In step ST34, the HUDECU 18 adjusts and determines the display position, size, brightness, and color of, for example, a frame mark used to draw attention in the right-eye image 91, which is a 3D image, based on the current positions of the occupant's eyes. Furthermore, the HUDECU 18 adjusts and determines the display position, size, brightness, and color of, for example, a frame mark used to draw attention in the left-eye image 92, which is a 3D image.

[0134] In step ST35 , based on the determination in step ST34 , the HUDECU 18 generates the right-eye image 91 including the attention-drawing frame mark 93 and the left-eye image 92 including the attention-drawing frame mark 94 as three-dimensional images.

[0135] In step ST36, the HUDECU 18 outputs the generated three-dimensional image to the projection device 71. As a result, the right-eye image 91 including the attention-calling frame mark 93 at a predetermined position and the left-eye image 92 including the attention-calling frame mark 94 at a predetermined position are alternately displayed on the windshield 6.

[0136] Thus, the HUDECU 18 of the HUD device 70 uses the personal data 80 on the occupant's vision generated by the occupant monitoring device 60 to adjust the position, size, brightness, color, etc. of the projection image in the three-dimensional image, and projects the three-dimensional image from the projection device 71 .

[0137] Here, the HUDECU 18 is as follows, for example Figure 4 The current positions of the right and left eyes of the occupant are determined based on the current position of the face and the pupil distance. If the position of the face shifts forward, backward, left, right, or up or down, the positions of the right and left eyes will also shift forward, backward, left, right, or up or down.

[0138] The HUDECU 18 adjusts and determines the position of the projected image in the right-eye image 91 and the left-eye image 92 so that the projected image overlaps with the right or left eye's line of sight to an object, such as a moving object, outside the vehicle. Based on the relative distance and direction of the object outside the vehicle to the windshield 6 and the current position of the face inside the vehicle relative to the windshield 6, the HUDECU 18 can calculate the line of sight from the right eye to the object outside the vehicle and the line of sight from the left eye to the object outside the vehicle. The HUDECU 18 can adjust the position of the projected image so that, for example, the projected image in the projected image on the dominant eye side is located in the line of sight between the object outside the vehicle and the dominant eye. Furthermore, the HUDECU 18 determines the position of the projected image in the projected image on the non-dominant eye side based on the interpupillary distance.

[0139] The HUDECU 18 adjusts and determines the size of the projected image in the right eye image 91 and the size of the projected image in the left eye image 92 based on the ratio of the distance from the right eye to the object outside the vehicle and the distance from the left eye to the object outside the vehicle. For example, the HUDECU 18 uses the ratio of the distance from the right eye to the object outside the vehicle and the distance from the left eye to the object outside the vehicle as a reference to scale the size of the projected image in the left eye image 92 on the non-dominant eye side. Figure 4 In the figure, since the pedestrian, which is an object outside the vehicle, is on the left, the projected image in the left-eye image 92 is larger than the projected image in the right-eye image 91. The longer the relative distance to the object such as the moving object outside the vehicle, the smaller the projected image.

[0140] Thus, the positional difference or size difference between the projected image in the right-eye image 91 and the projected image in the left-eye image 92 is adjusted. In addition, the projected image and objects outside the vehicle can be recognized in a desired overlapping state or at a desired distance in the dominant eye.

[0141] Furthermore, the HUDECU 18 adjusts the positional and size differences between the projected images in the right-eye image 91 (3D image) and the projected images in the left-eye image 92 (3D image) based on the occupant's binocular vision or diopter. Thus, the positions and sizes of the projected images in the right-eye image 91 and the left-eye image 92 (3D image) are adjusted based on the occupant's physical characteristics, namely, binocular vision or diopter. The projected images are projected in a manner that is easily visible to the occupant.

[0142] Furthermore, the HUDECU 18 adjusts the brightness, color, and clarity of the projected image so that the projected image in the image on the dominant eye side is more clearly visible than the projected image in the other image on the non-dominant eye side.

[0143] Next, Figure 8Each process of step ST4 to step ST6 will be described in detail.

[0144] Figure 12 yes Figure 8 FIG. 1 is a flowchart showing details of the process of generating the interpupillary distance information in step ST4.

[0145] In step ST41, the occupant monitoring ECU 17 outputs an instruction to the occupant to face the interior camera 61. The occupant monitoring ECU 17 requests the UIECU 20 to display a facing instruction screen. The UIECU 20 displays the facing instruction screen on the display device 51, instructing the occupant to face the interior camera 61.

[0146] In step ST42 , the occupant monitoring ECU 17 turns on the infrared lamp 62 , thereby irradiating the face of the occupant facing the interior camera 61 with infrared rays.

[0147] In step ST43 , the occupant monitoring ECU 17 uses the interior camera 61 to capture an image of the occupant's face or both eyes while irradiating infrared rays.

[0148] In step ST44, the occupant monitoring ECU 17 analyzes the image of the occupant's face or eyes captured from the front, and determines the position of the occupant based on the position of the right eye and the position of the left eye. Figure 4 The distance between the right and left eyes of the occupant is the interpupillary distance.

[0149] In step ST45 , the occupant monitoring ECU 17 turns off the infrared rays and ends the light projection.

[0150] Through the above processing, the passenger monitoring ECU 17 generates information on the pupil distance of the passenger as information on the vision of the registered passenger. Figure 7 A portion of the personal data 80 is recorded in the occupant monitoring memory 64 .

[0151] The occupant monitoring ECU 17 can accurately generate information on the interpupillary distance of the occupant using a clear, unblurred frontal image of the occupant captured in a safe state where the automobile 1 is not moving.

[0152] Figure 13 yes Figure 8 Detailed flowchart of the process of generating the dominant eye information in step ST5.

[0153] In step ST51, the occupant monitoring ECU 17 starts projection for determining the dominant eye. The occupant monitoring ECU 17 instructs the HUD ECU 18 to perform this projection. The HUD ECU 18 displays a projection image for determining the dominant eye from the projection device 71. This projection image for determining the dominant eye can be, for example, an image of two bars that appear to overlap in the dominant eye. In this image, the bars appear to be offset to the left and right in the non-dominant eye.

[0154] In step ST52, the occupant monitoring ECU 17 turns on the infrared lamp 62. This causes infrared light to be irradiated toward the face of the occupant who is viewing the projected image for dominant eye determination.

[0155] In step ST53 , the occupant monitoring ECU 17 uses the interior camera 61 to capture an image of the occupant's face or both eyes while projecting the projection image for dominant eye determination.

[0156] In step ST54, the occupant monitoring ECU 17 analyzes the image of the occupant's face or eyes and determines the dominant eye based on the state of the eyes in the captured image. Instead of analyzing the state of the eyes in the captured image, the occupant monitoring ECU 17 may determine the dominant eye based on a selection operation or gesture made by the captured occupant.

[0157] In step ST55 , the occupant monitoring ECU 17 turns off the infrared rays and ends the projection by the HUDECU 18 .

[0158] Through the above processing, the occupant monitoring ECU 17 generates information on the occupant's dominant eye based on the image of the occupant's face captured by the interior camera 61. The generated information on the occupant's dominant eye is used as Figure 7 A portion of the personal data 80 is recorded in the occupant monitoring memory 64 .

[0159] The occupant monitoring ECU 17 can accurately generate information on the dominant eye of the occupant using a clear, unblurred image of the occupant captured in a safe state where the vehicle 1 is not moving.

[0160] Figure 14 yes Figure 8 Detailed flowchart of the process of generating vision and diopter information in step ST6.

[0161] In step ST61, the occupant monitoring ECU 17 starts projection for vision assessment. The occupant monitoring ECU 17 instructs the HUD ECU 18 to start projection for vision assessment. The HUD ECU 18 displays a projection image for vision assessment from the projection device 71. This projection image for vision assessment can be, for example, a balloon moving forward and backward toward the occupant. While projecting this image, the occupant moves their eyes so that their line of sight is aligned with the front and rear position of the balloon.

[0162] In step ST62, the occupant monitoring ECU 17 outputs an instruction to blur the projected image to the HUDECU 18. The HUDECU 18 deforms the projected image so that the outline of the projected image is temporarily blurred.

[0163] In step ST63, the occupant monitoring ECU 17 outputs an instruction to restore the blur of the projected image to the HUDECU 18. The HUDECU 18 restores the outline of the blurred projected image.

[0164] In step ST64, the occupant monitoring ECU 17 turns on the infrared lamp 62. This irradiates the face of the occupant who is viewing the projected image for vision determination with infrared rays.

[0165] In step ST65 , the occupant monitoring ECU 17 uses the interior camera 61 to capture an image of the occupant's face or both eyes while projecting the projection image for visual acuity determination.

[0166] In step ST66, the occupant monitoring ECU 17 analyzes the image of the occupant's face or eyes and determines the visual acuity and diopter of both eyes based on the state of both eyes in the image. The visual acuity and diopter of both eyes can be determined using methods used in conventional vision tests.

[0167] In step ST67 , the occupant monitoring ECU 17 turns off the infrared rays and ends the projection by the HUDECU 18 .

[0168] Through the above processing, the occupant monitoring ECU 17 generates information on the visual acuity and diopter of the occupant's eyes based on the image of the occupant's face or eyes captured by the interior camera 61. The generated information on the visual acuity and diopter of the occupant's eyes is used as Figure 7 A portion of the personal data 80 is recorded in the occupant monitoring memory 64 .

[0169] The occupant monitoring ECU 17 can accurately generate information on the visual acuity and diopter of the occupant's eyes using a clear, unblurred image of the occupant captured in a safe state where the vehicle 1 is not moving.

[0170] Furthermore, since the outline of the projected image is temporarily blurred before shooting, the measurement is performed in a good state after the occupant's eyes are relaxed during shooting.

[0171] As described above, in this embodiment, the occupant monitoring device 60 generates personal data 80 regarding the occupant's vision for the HUD device 70 based on images captured by the interior camera 61 of the vehicle 1 occupant. Furthermore, the HUD device 70 uses the personal data 80 regarding the occupant's vision generated by the occupant monitoring device 60 to adjust the position, size, brightness, color, and other aspects of the projected image within the three-dimensional image, and projects the adjusted three-dimensional image from the projection device 71. This embodiment effectively eliminates any discrepancies in the perceived distance between the projected image formed by the three-dimensional image and objects outside the vehicle. From the occupant's perspective, the projected image, such as the frame mark 90 or the attention-drawing arrow mark, projected onto the three-dimensional image projected onto objects outside the vehicle in front of the occupant appears close to or overlaps with the object outside the vehicle in front of the occupant.

[0172] On the other hand, if there is a difference in the sense of distance between the three-dimensional projected image and objects outside the vehicle, for example, the occupant may feel uncomfortable between them and may even feel nauseous due to frequent shifting of the eyes between them.

[0173] In this embodiment, the positional difference between the projected images of right-eye image 91 and left-eye image 92, which serve as 3D images, is adjusted based on the occupant's physical characteristics, namely, the interpupillary distance, and the current position of the face or eyes in vehicle 1. This makes it easier for the occupant to see the projected images positioned close to or overlapping with objects outside the vehicle in front of the occupant. This effectively reduces the difference in distance between the projected images formed by the 3D images and the objects outside the vehicle.

[0174] Furthermore, in this embodiment, the difference in size between the projected images of the right-eye image 91 and the left-eye image 92, which serve as 3D images, is adjusted based on the occupant's physical characteristics, namely, the interpupillary distance, and the current position of the face or eyes in vehicle 1. For example, adjustment is made based on the interpupillary distance and the position of the face or eyes so that the relative size of the projected image relative to the size of the external object in the right eye's field of view is the same as the relative size of the projected image relative to the size of the external object in the left eye's field of view. This ensures that, from the occupant's perspective, the projected images appear identical on both sides relative to the external object in front of the occupant. If the right-eye and left-eye projected images represent the same object, they are easier to discern.

[0175] Through these visual adjustments, passengers can easily perceive the projected image as being close to or overlapping with objects outside the vehicle. Passengers perceive the projected image as being close to or overlapping with objects outside the vehicle, making it less likely that they will feel uncomfortable.

[0176] The above-mentioned embodiments are examples of preferred embodiments of the present invention, but the present invention is not limited thereto, and various modifications and changes can be made without departing from the spirit of the invention.

Claims

1. A head-up display system for a vehicle, comprising: A head-up display device includes a projection device for projecting a three-dimensional image in front of a vehicle occupant, and projects images related to objects outside the vehicle in front of the occupant; as well as An occupant monitoring device includes a camera for capturing an occupant of the vehicle, and generates personal data on the occupant's vision based on images captured by the camera. The occupant monitoring device captures the face or both eyes of the occupant by the camera when the vehicle is in a safe state, and determines the distance between the right and left eyes of the occupant, i.e., the interpupillary distance, based on the shooting positions of both eyes in the captured image of the occupant's eyes, and generates information on the interpupillary distance, thereby generating position information of the face or both eyes of the occupant in the vehicle and information on the interpupillary distance as the personal data. wherein the head-up display device switches between projecting a right-eye image including the projected image and a left-eye image including the projected image as the three-dimensional image, and adjusts a position difference or a size difference between the projected image in the right-eye image and the projected image in the left-eye image based on the interpupillary distance as the personal data generated by the occupant monitoring device, thereby adjusting the position of the projected image in the right-eye image and the position of the projected image in the left-eye image so that the respective projected images overlap in the right eye's or left eye's line of sight toward an object outside the vehicle; The three-dimensional image is projected from the projection device based on the position information of the face or the position information of both eyes of the occupant in the vehicle, with respect to the projection image in the three-dimensional image being adjusted.

2. The head-up display system for a vehicle according to claim 1, wherein: The occupant monitoring device generates information on the dominant eye of the occupant as personal data on the occupant's vision based on the image captured by the camera, based on the image captured by the camera. The head-up display device adjusts the position of the projected image so that the projected image of the dominant eye side image of the right eye image and the left eye image as the three-dimensional image is located at a position based on the line of sight between the object outside the vehicle and the dominant eye.

3. The head-up display system for a vehicle according to claim 1, wherein: The occupant monitoring device generates information on the dominant eye of the occupant as personal data on the occupant's vision based on the image captured by the camera, based on the image captured by the camera. The head-up display device is adjusted so that the projected image in the image on the dominant eye side of the right-eye image and the left-eye image, which are the three-dimensional images, is more easily visible than the projected image in the other image.

4. The head-up display system for a vehicle according to claim 1, wherein: When the vehicle is in a safe state, the occupant monitoring device causes the head-up display device to project a projection image for determining the dominant eye, captures the face or both eyes of the occupant while the projection image for determining the dominant eye is being projected, and determines the dominant eye of the occupant based on the state of both eyes in the captured image, thereby generating information on the dominant eye of the occupant.

5. The head-up display system for a vehicle according to claim 1, wherein: The occupant monitoring device generates information on the visual acuity or diopter of the occupant's eyes based on the captured images of the occupant's eyes formed by the capturing device as personal data on the occupant's vision based on the captured images of the occupant's eyes. The head-up display device adjusts a positional or size difference between a projected image in a right-eye image and a projected image in a left-eye image, which are the three-dimensional images, based on information on the visual acuity or diopter of both eyes of the occupant.

6. The head-up display system for a vehicle according to claim 5, wherein: When the vehicle is in a safe state, the occupant monitoring device causes the head-up display device to project an image for visual acuity assessment. The face or eyes of the passenger are photographed while the image for visual acuity assessment is being projected. Based on the states of the eyes in the captured image, the visual acuity or diopter of the occupant's eyes is determined, and information on the visual acuity or diopter of the occupant's eyes is generated.

7. The head-up display system for a vehicle according to claim 6, wherein: The head-up display device projects a projection image for visual acuity assessment in a manner such that the projection image is blurred and then restored. The head-up display device captures the face or both eyes of the occupant in a state where a projection image for visual acuity assessment that has recovered from blur is projected.

Images