vehicle

Abstract

To provide a technique of a head-up display device, which can achieve more preferable usability than before, so as to contribute to "3. Good Health and Well-Being" of sustainable development objectives.SOLUTION: A vehicle has a head-up display device (HUD device) 1 including: a housing 102 storing a light source unit for generating image light, a main display unit, and an optical system; a sub display unit (sub LCD2) arranged outside the housing 102; and a control unit for controlling display of image information on the main display unit and display of the image information on the sub display unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for a head-up display device (sometimes referred to as a head-up display: HUD). [Background technology]

[0002] For example, International Publication No. 2018 / 229961 (Patent Document 1) describes a light source device that is small, lightweight, has a high light utilization rate, is modularized, and can be easily used as a planar light source, as well as a head-up display device equipped with the same.

[0003] When mounted on a vehicle, a head-up display device such as that disclosed in Patent Document 1 can display various information, such as driving information such as vehicle speed and engine RPM, and navigation information, as a virtual image on the vehicle's windshield (in other words, the front glass). By using a HUD, the driver can obtain information necessary for driving without having to shift their line of sight to an instrument panel built into the dashboard. For this reason, HUDs contribute to safe driving of automobiles and the like. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 229961 Summary of the Invention [Problem to be solved by the invention]

[0005] Conventional head-up display devices project image light generated based on an image displayed on an image display device (display panel) using light from a light source device onto a predetermined area (sometimes referred to as a display area) such as a windshield, thereby providing a virtual image to the driver. The virtual image can realize functions such as augmented reality (AR), which superimposes various types of image information on the real scene. A head-up display device with such functions is also called an AR-HUD.

[0006] The virtual image in a head-up display device of a conventional technology example such as that disclosed in Patent Document 1 has directionality. That is, the driver can view the virtual image favorably only when viewing the display area of the windshield from a predetermined viewpoint (including the eyebox). Because the virtual image in the display area of an AR-HUD has such characteristics, it may be difficult for users such as the driver to view, and there is room for improvement in terms of usability.

[0007] An object of the present invention is to provide a technology for a head-up display device that can achieve more suitable usability. [Means for solving the problem]

[0008] A representative embodiment of the present disclosure has the following configuration: A vehicle of the embodiment displays a virtual image in a display area based on the projection of image light, and includes a housing that houses a light source device for generating the image light, a main display device, and an optical system, a sub-display device provided outside the housing, and a control device that controls the display of image information on the main display device and the display of image information on the sub-display device. [Effects of the Invention]

[0009] According to representative embodiments of the present disclosure, it is possible to realize more suitable usability, etc., with respect to the technology of a head-up display device. Problems, configurations, effects, etc. other than those described above will be described in the description of the preferred embodiment of the present invention. [Brief explanation of the drawings]

[0010] [Figure 1] 1 shows an image of a driver in a vehicle using a HUD device according to an embodiment. [Figure 2] 1 illustrates a vehicle equipped with a HUD device according to an embodiment, and an example of the configuration of the HUD device. [Figure 3] 1 illustrates a vehicle equipped with a HUD device according to an embodiment, and an example of the internal configuration of the HUD device. [Figure 4] 1 shows an example of the external configuration of the HUD device as hardware according to an embodiment. [Figure 5A] 1 shows an example of a functional block configuration of a HUD device according to an embodiment. [Figure 5B] 2 shows an example of a functional block configuration of an MCU 411 of the HUD device according to an embodiment. [Figure 6A] 2 shows an example of a functional block configuration relating to control of two types of displays in a HUD device according to an embodiment. [Figure 6B] 10 shows an example of a functional block configuration relating to control of two types of displays in a HUD device according to a modified example. [Figure 6C] 10 shows an example of a functional block configuration relating to control of two types of displays in a HUD device according to a modified example. [Figure 7] 10 shows an operational flow at the time of startup in the HUD device of one embodiment. [Figure 8] 1 shows an example of the configuration of a video display device in a HUD device according to an embodiment. [Figure 9] 10A and 10B show examples of two types of displays in a HUD device according to an embodiment. [Figure 10] 10 shows a processing flow of an example of display control of two types of displays in consideration of the type and amount of information in a HUD device according to an embodiment. [Figure 11] 10A and 10B show examples of two types of displays in a HUD device according to an embodiment, taking into consideration the type and amount of information. [Figure 12]10A to 10C show examples of how two types of displays are displayed when adjusting the initial settings in the HUD device according to an embodiment. [Figure 13] 10 shows a processing flow at the time of initial setting in the HUD device of one embodiment. [Figure 14] 10 shows another example of display when adjusting the initial settings in the HUD device according to the embodiment. [Figure 15] 10 shows an example of a display in an error state in the HUD device of one embodiment. [Figure 16] 10 shows a processing flow in an error state in a HUD device according to an embodiment. [Figure 17] 10A to 10C show examples of display according to the scene in the HUD device of the embodiment. [Figure 18] 10 shows an example of a display related to user settings for two types of display modes in a HUD device according to an embodiment. [Figure 19] 10 shows another example of a display related to user settings for two types of display modes in the HUD device according to the embodiment. [Figure 20] 10 shows a display example of brightness change control according to the scene in the HUD device of the embodiment. [Figure 21] 1A and 1B are explanatory diagrams relating to distortion correction in a HUD device according to an embodiment. [Figure 22] 1A and 1B are explanatory diagrams relating to pitching correction in a HUD device according to an embodiment; [Figure 23] 10 shows a display example in a debug mode in the HUD device of one embodiment. [Figure 24] 1A and 1B are explanatory diagrams illustrating dual display in a HUD device according to an embodiment. [Figure 25] 1 shows a process flow relating to dual display in a HUD device according to an embodiment. [Figure 26] 1A and 1B are explanatory diagrams relating to a first example of destination change control in a HUD device according to an embodiment; [Figure 27] 10A and 10B are explanatory diagrams relating to a second example of destination change control in the HUD device according to the embodiment. [Figure 28] 10A and 10B are explanatory diagrams relating to a third example of destination change control in the HUD device according to the embodiment. [Figure 29] 10 shows a processing flow relating to destination change control in the HUD device of one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the drawings, the same parts are generally designated by the same reference numerals, and repeated explanations will be omitted. In the drawings, the representation of components may not represent their actual positions, sizes, shapes, ranges, etc., in order to facilitate understanding of the invention.

[0012] For the sake of explanation, when describing processing by a program, the program, function, processing unit, etc. may be described as the main body, but the main hardware body for these is a processor, or a controller, device, computer, system, etc. that is composed of the processor, etc. A computer executes processing according to a program read into memory using resources such as memory and communication interfaces as appropriate through the processor. This realizes predetermined functions, processing units, etc. A processor is composed of, for example, semiconductor devices such as a CPU or GPU. A processor is composed of devices or circuits that are capable of performing predetermined calculations. Processing is not limited to software program processing, but can also be implemented using dedicated circuits. Dedicated circuits such as FPGAs, ASICs, and CPLDs can be used.

[0013] The program may be pre-installed as data on the target computer, or may be distributed as data from a program source to the target computer. The program source may be a program distribution server on a communication network, or a non-transitory computer-readable storage medium (e.g., a memory card). The program may be composed of multiple modules. Various data and information may be composed of structures such as tables and lists, but are not limited to these. Expressions such as identification information, identifiers, IDs, names, and numbers are interchangeable.

[0014] <First Embodiment> A head-up display device according to a first embodiment of the present disclosure will be described with reference to FIGS. 1 to 29. For the purpose of description, symbols such as (X, Y, Z) in FIG. 1 may be used as directions and coordinate systems where appropriate. The Z direction is the vertical direction, and the X and Y directions are two perpendicular directions that form a horizontal plane. The X direction is the horizontal direction in the display area 105 and the sub-LCD2, and the left-right direction as seen from the driver. The Z direction is the vertical direction in the display area 105 and the sub-LCD2, and the up-down direction as seen from the driver. The Y direction is the front-rear direction as seen from the driver.

[0015] The HUD device 1 of the first embodiment shown in FIG. 1 etc. is a so-called AR-HUD, which projects image light from an image display device 403 in a housing 102 in FIG. 2 etc. onto a display area 105 of a windshield 103 of a vehicle 100 through an optical system such as a mirror 3 a, thereby forming and displaying a virtual image 106 (an image corresponding to the virtual image) corresponding to augmented reality (AR) etc. at a driver's viewpoint 108.

[0016] In the HUD device 1 of the first embodiment, a display area 105 in which a virtual image 106 is formed based on the video display device 403 (particularly the display panel 4) is used as a main display, and a sub-LCD2 is provided in the housing 102 as a sub-display. In the first embodiment, the main display is a main display screen, and the sub-display or the sub-LCD2 is a sub-display device. The HUD device 1 of the first embodiment assists the driver in driving by displaying a virtual image using the AR-HUD and displaying a video using the sub-display.

[0017] The HUD device 1 of the first embodiment has a structure and installation that provides a sub-LCD 2 having a physical display screen 20 as a sub-display in the housing 102. The HUD device 1 also has a circuit that drives and controls both the AR-HUD and the sub-display. The HUD device 1 also has software that controls how to use the two types of displays, the AR-HUD and the sub-display, and so on.

[0018] [Usage image] FIG. 1 shows an example of installation and usage of the HUD device 1 of the first embodiment in a vehicle. The vehicle in FIG. 1 is an example of a vehicle with a steering wheel 104 on the left side, and most of a housing 102 of the HUD device 1 is housed within a dashboard 101. A portion of the housing 102 including an opening (a part of the convex shape in FIG. 2, a housing part 102B in FIG. 4) protrudes upward outside the dashboard 101, and a sub-LCD2 (LCD: liquid crystal display) serving as a sub-display (in other words, a sub-display device) is provided in a part of the housing 102. The sub-LCD2 is disposed on the dashboard 101, in front of the steering wheel 104, with the display screen 20 facing the driver. A display area 105 (in other words, a HUD area) indicates an area on a windshield 103 of the vehicle where a virtual image 106 is formed and displayed. The virtual image 106 is displayed within the display area 105 so as to be visible from the driver's viewpoint.

[0019] The sub-LCD 2 is attached to the housing 102 of the HUD device 1. In other words, the housing 102 of the HUD device 1 also serves as the housing of the sub-LCD 2. In conventional HUD devices, the entire housing is housed within the dashboard 102. In contrast, in this HUD device 1, part of the housing 102 is housed within the dashboard 102, and another part protrudes outside the dashboard 102, where the sub-LCD 2 is provided (FIGS. 2 to 4). The installation position of this HUD device 1 is substantially the same as that of the conventional device, which satisfies the condition that the virtual image 106 of the display area 105 is correctly seen from the driver's viewpoint (the eyebox including the viewpoint). The installation position of the sub-LCD 2 is a position facing the driver directly, near the bottom side of the display area 105.

[0020] In the display area 105 of the AR-HUD, an image such as AR superimposed on the real scene is displayed as a virtual image 106 as needed. Information to assist driving is constantly displayed as a real image on the sub-LCD2. Examples of AR images include warning icons for collision prevention, icons for warning other vehicles or people, and arrow images for navigation (route guidance). Examples of information displayed on the sub-LCD2 include vehicle speed, left / right turn information, driving guides, character images, and various other information.

[0021] The projection destination of the image light for forming the display area 105 of the virtual image 106 is not limited to the windshield 103, but may be another object such as a combiner (projection panel).

[0022] [Vehicles and HUD Devices (1)] FIG. 2 shows an example of how the HUD device 1 is installed in a vehicle 100 and an example of the configuration of the HUD device 1. In FIG. 2, the HUD device 1 includes an image display device 403 (see FIG. 5A described below) and optical systems such as mirrors 3a and 3b arranged and fixed in a predetermined positional relationship within a housing 102. As shown in FIG. 2, the image display device 403 is attached within the housing 102, and may be attached to a portion of the outer periphery of the housing 102. The housing 102 in FIG. 2 has a convex shape, for example, with a portion of the convex shape projecting above the dashboard 101. A sub-LCD 2 is fixed on the outside of the housing 102, on the front side (the side facing the driver) of the convex portion projecting upward from the dashboard 101. Image light from the image display device 403 is reflected by the mirrors 3a and 3b and emitted to the outside of the housing 102 through openings provided in the convex portions. The image light is reflected by the display area 105 and directed toward the driver's viewpoint 108, forming a virtual image 106 as seen by the driver.

[0023] [Vehicles and HUD Devices (2)] FIG. 3 shows an example of the detailed configuration of FIG. 2 , including an example of the installation of the HUD device 1 in a vehicle 100 and an outline of an example of the internal configuration of the HUD device 1. The HUD device 1 includes a light source device 5, a display panel 4, and optical systems, such as mirrors 3a and 3b, arranged and fixed in a predetermined positional relationship within a housing 102. The mirror 3b is, for example, a free-form mirror or a mirror with an asymmetrical optical axis, and is a reflective mirror in this example. In the example of FIG. 3 , the housing 102 is roughly rectangular (see FIG. 4 , described later). A sub-LCD 2 is fixed to the front side (facing the driver) in the Y direction of the housing section 102B ( FIG. 4 ), which protrudes upward from the dashboard 101, on the outside of the housing 102. The light source device 5 and the display panel 4 constitute the image display device 403 in FIG. 5A. The image display device 403 (the light source device 5 and the display panel 4) and optical systems, such as mirrors 3a and 3b, constitute the image display unit 402 in FIG. 5A. Image display device 403 is a projection type image display device that projects an image formed on display panel 4 using light emitted from light source device 5 (in other words, light source light).

[0024] The light source device 5 includes a semiconductor light source element, typically an LED (Light Emitting Diode) light source, as a light source. The display panel 4 is typically a liquid crystal display (LCD) panel. The display panel 4 emits image light based on the light from the light source device 5. The display panel 4 creates an image based on image information from the control device (FIG. 5A) and displays it on the display screen of the display panel 4. The display panel 4 modulates the transmittance of the light from the light source device 5 for each pixel in accordance with the image information, thereby forming an image to be projected onto the display region 105 and emitting it as image light.

[0025] Image light from the display panel 4 passes through an optical system including a mirror 3a and a reflecting mirror 3b and is emitted from an opening on the upper side of the housing 102. The mirror 3a reflects the image light from the display panel 4 toward the reflecting mirror 3b. The reflecting mirror 3b is, for example, a concave mirror. The reflecting mirror 3b magnifies and reflects the image light from the mirror 3a toward the opening of the housing 102 at a set angle. The image light emitted from the opening of the housing 102 is reflected by the surface of a display area 105 of the windshield 103 and heads toward the driver's viewpoint 108.

[0026] As a result, when the driver looks ahead from viewpoint 108 (viewpoint 108 within a predetermined eyebox), a virtual image 106 formed by the image light can be viewed in display area 105, superimposed on the real scene (e.g., road, vehicle, person, etc.) ahead of windshield 103. The virtual image 106 can be image information that is superimposed and displayed in alignment with the object in the real scene, or image information that is displayed independently of the object in the real scene. Various types of image information can be used as virtual image 106, such as vehicle speed information, warnings, cautions, and route guidance information. The virtual image 106 can realize functions such as augmented reality (AR).

[0027] In the exemplary configuration of the image display unit 402 in FIG. 5A, a driving mechanism 404 such as a motor is provided on the reflecting mirror 3b. The driving mechanism 404 can adjust the angle of the reflecting mirror 3b (FIG. 3). This makes it possible to adjust the direction in which image light is projected from the reflecting mirror 3b onto the display area 105 on the windshield 103. In other words, the position of the display area 105 where the virtual image 106 is formed can be adjusted. The driving mechanism 404 changes the angle of the reflecting mirror 3b based on control from the control device 401 or based on manual operation by the user. This makes it possible to suitably adjust the height position, etc. of the display area 105 as seen by the driver during initial setup, which will be described later.

[0028] [Case] FIG. 4 shows an example of the configuration of the sub-LCD 2 in the housing 102 of the HUD device 1. (A) of FIG. 4 shows the HUD device 1 when in use, and (B) shows it when not in use. The housing 102 has a housing section 102A that is housed within the dashboard 101 and a housing section 102B above it that extends outside the dashboard 101. The housing section 102B has an opening 107 on its top surface. The opening 107 is a portion through which image light passes toward the display area 105, and is provided with a glare trap or the like. The sub-LCD 2 is attached to the front surface of the housing section 102B via an attachment device or the like (not shown). As a modified example, the sub-LCD 2 may be mounted so as to be embedded in the housing section 102B.

[0029] In this example, the sub-LCD2 is a horizontally elongated LCD that is wider than the width of the housing unit 102B and has a horizontally elongated display screen 20. The horizontally elongated display screen 20 of the sub-LCD2 is designed to be positioned near the bottom edge of the display area 105 of the AR-HUD, as shown in FIG. 1 . The sub-LCD2 is connected to a control device (see FIG. 5A, described below) in the housing 102 via wiring. The top edge of the sub-LCD2 may be located at the same level as the top surface of the housing unit 102B or may extend above the top surface. The sub-LCD2 is preferably positioned so that the opening 107 of the housing unit 102B is hidden by the sub-LCD2 from the driver's perspective. During use, the sub-LCD2 is mainly visible from the driver's perspective, so the housing unit 102B protruding above the dashboard 101 is not noticeable. Alternatively, the width of the sub-LCD2 may be equal to or smaller than the width of the housing unit 102B.

[0030] This HUD device 1 is not configured so that the entire housing 102 is housed within the dashboard 101, but rather so that a portion of the housing 102, a housing portion 102B, protrudes outside the dashboard 101. Therefore, the volume of the housing portion 102B within the dashboard 101 can be reduced by the amount of the housing portion 102B, and the HUD device 1 can be easily housed within the dashboard 101 even when the space within the dashboard 101 is relatively narrow. This makes it easy to install this HUD device 1 in various types of vehicles.

[0031] Furthermore, since the housing 102B may be located outside the dashboard 101, the degree of freedom in the structure of the housing 102, such as the dimensions and shape, is increased. HUD devices generally require an optical distance within the housing to form a virtual image. For this reason, a reflecting mirror or the like is provided within the housing as an optical system. Since the HUD device 1 does not require the housing 102A to be overly compact, it may be possible to reduce the optical system (optical components such as a reflecting mirror) for increasing the distance of the optical path within the housing 102. Alternatively, even if the volume of the housing 102A is the same as that of a conventional device, the HUD device 1 can ensure a longer optical path by the optical system within the housing 102, due to the housing 102B located outside the dashboard 101, thereby enabling a virtual image to be displayed at a greater distance.

[0032] In FIG. 4B, the HUD device 1 can also function as a shutter that hides the opening 107 by changing the position of the sub-LCD2 using a predetermined mechanism when not in use. In this example, a hinge 102C is provided between the upper edge of the sub-LCD2 and one edge of the front side of the upper surface of the housing 102B, and a rotation mechanism allows the sub-LCD2 to rotate around the hinge 102C as a rotation axis. When not in use, the sub-LCD2 rotates as shown in the figure, so that the display screen 20 of the sub-LCD2 faces the opening 107 of the housing 102B. As a result, the opening 107 is hidden by the sub-LCD2, and the back side of the sub-LCD2 is positioned as the upper surface. For example, a film for suppressing external light reflection (in other words, a light-shielding film) may be formed on the back of the sub-LCD2. As a result, when not in use, the sub-LCD2 functions as a shutter, preventing or reducing internal panel burn and dust accumulation caused by external light such as sunlight entering the opening 107.

[0033] Furthermore, when the display screen 20 of the sub-LCD 2 is not in contact with the opening 107, this is undesirable as it may damage the components. Therefore, the rotation mechanism is designed to stop the rotation just before the opening 107 comes into contact with the display screen 20 of the sub-LCD 2, or a spacer is provided between the opening 107 and the display screen 20 of the sub-LCD 2. For example, spacers are provided near the outside of the four corners of the opening 102 on the top surface of the housing 102B. When the display screen 20 of the sub-LCD 2 is not in use, part of the display screen 20 of the sub-LCD 2 is supported by the spacers, preventing the opening 107 from coming into contact with the display screen 20.

[0034] Furthermore, during use, the orientation of the display screen 20 of the sub-LCD 2 may be adjusted to suit the driver by changing the rotation angle using a mechanism such as the hinge portion 102C.

[0035] The movement of the sub LCD 2 as described above may be performed manually by the user, or a drive mechanism such as a motor may be provided so that the sub LCD 2 is moved automatically when the HUD device 1 is started up or shut down. For example, when the HUD device 1 is started up, the sub LCD 2 is driven to move from state (B) to state (A). When the HUD device 1 is shut down, the sub LCD 2 is driven to move from state (A) to state (B).

[0036] In (B), even if the sub-LCD 2 does not cover the entire opening 107, the effect of preventing external light from entering can be obtained for at least a part of the covered portion.

[0037] The configuration in which the opening 107 can be hidden by the sub-LCD 2 is not limited to the mechanism described above (B). As a modified example, the housing 102 may be provided with a mechanism that allows the sub-LCD 2 to slide, for example, by translation in the Y direction. As another modified example, the sub-LCD 2 itself may be configured as a foldable display.

[0038] When a mechanism that can close the opening 107 with the sub-LCD 2 is provided as in (B) above, it is possible to eliminate a mechanism for preventing external light from entering the housing 102, such as a mechanism for rotating a reflecting mirror.

[0039] FIG. 4C shows a modified example of the housing 102. A slide mechanism 102D to which the sub-LCD 2 is attached is provided on the front side of the housing part 102B. The slide mechanism 102D allows the sub-LCD 2 to slide so as to move parallel to the Z direction, for example. This makes it possible to adjust the height position of the sub-LCD 2. In another modified example, a rotation angle adjustment mechanism may be further provided to make it possible to change the orientation of the sub-LCD 2 (the direction of the optical axis of the display screen 20).

[0040] [Example of functional block configuration of HUD device (1)] 5A shows an example of a functional block configuration of the HUD device 1. The HUD device 1 includes a control device 401, a video display unit 402, a sub-LCD 2, a speaker 405, a sensor 409, etc. The control device 401 is configured, for example, by an electronic control unit (ECU). The video display unit 402 has a configuration as shown in FIG. 2 or 3 above.

[0041] The control device 401 corresponds to a controller that controls the entire HUD device 1 and each unit thereof, and mainly controls the video display and audio output of the HUD device 1. The control device 401 is configured, for example, with a wiring board or the like. The control device 401 is mounted, for example, in the housing 102 of FIG. 2 or 3. Note that the control device 401 is not limited to being mounted inside the housing 102, but may also be mounted outside the housing 102. The control device 401 includes a vehicle information acquisition unit 407, a communication unit 408, a sensor 409 (which may be located outside the control device 401 or the housing 102), a microcontroller (MCU) 411, a nonvolatile memory 211, a volatile memory 212, a display driver 223, an audio driver 324, etc.

[0042] The HUD device 1 acquires vehicle information 406 from various sensors (in other words, information acquisition devices) installed in various parts of the vehicle 100 ( FIG. 2 ) via the vehicle information acquisition unit 407. The various sensors periodically detect, for example, parameters related to conditions such as driving conditions inside and outside the vehicle 100. An example of the various sensors is the camera 109 (exterior camera or interior camera) installed in the vehicle 100 of FIG. 3 . The HUD device 1 acquires detection information from the various sensors via the vehicle information acquisition unit 407, and can detect or determine various events related to the vehicle 100 based on the detection information. The HUD device 1 may also acquire detection information from a sensor 409 installed in the HUD device 1. The HUD device 1 may detect or determine the state of the HUD device 1 or the state near the HUD device 1 based on the detection information from the sensor 409.

[0043] The vehicle information acquisition unit 407 acquires the vehicle information 406 based on a communication protocol compatible with, for example, a CAN (Controller Area Network) interface or a LIN (Local Interconnect Network) interface.

[0044] Vehicle information 406 is a general term for information related to the driving status of vehicle 100. Vehicle information 406 includes, for example, speed information and gear information of vehicle 100, steering wheel steering angle information, lamp illumination information, external light information, distance information, infrared information, engine ON / OFF information, camera image information, acceleration gyro information, GPS (Global Positioning System) information, navigation information, vehicle-to-vehicle communication information, and road-to-vehicle communication information. Camera image information is image information from camera 109 or the like, and includes in-vehicle camera image information and vehicle-exterior camera image information. GPS information includes latitude and longitude as well as current time information.

[0045] Examples of various sensors (information acquisition devices) installed in the vehicle 100 or the HUD device 1 are shown below. The various sensors (information acquisition devices) include a vehicle speed sensor, a shift position sensor, a steering wheel angle sensor, a headlight sensor, an illuminance sensor, a chromaticity sensor, a distance measurement sensor, an infrared sensor, an engine start sensor, an acceleration sensor, a gyro sensor, a temperature sensor, an in-vehicle camera, an exterior-vehicle camera, a wireless transceiver for road-to-vehicle communication, a wireless transceiver for vehicle-to-vehicle communication, a GPS receiver, a VICS (Vehicle Information and Communication System, registered trademark) receiver, etc. The various sensors are not limited to these, and can be added, deleted, replaced, etc.

[0046] The vehicle speed sensor detects the speed of the vehicle 100 (also referred to as vehicle speed) and generates speed information as the detection result. The shift position sensor detects the current gear and generates gear information as the detection result. The steering wheel steering angle sensor detects the current steering wheel steering angle and generates steering wheel steering angle information as the detection result. The headlight sensor detects whether the headlights are on or off and generates lamp illumination information as the detection result. The illuminance sensor and chromaticity sensor detect external light and generate external light information as the detection result.

[0047] The distance measurement sensor detects the distance between the vehicle 100 and an external object, and generates distance information as the detection result. The infrared sensor detects the presence or absence of an object and its distance within close range of the vehicle 100, and generates infrared information as the detection result. The engine start sensor detects whether the engine is on or off, and generates ON / OFF information as the detection result. The acceleration sensor and gyro sensor detect the acceleration and angular velocity of the vehicle 100, and generate acceleration gyro information representing the attitude and behavior of the vehicle 100 as the detection result. The temperature sensor detects the temperature inside and outside the vehicle, and generates temperature information as the detection result.

[0048] The in-vehicle camera captures images of the interior of the vehicle 100 to generate in-vehicle camera video information. The exterior camera captures images of the exterior of the vehicle 100 to generate outside camera video information. In a specific example, the camera 109 in FIG. 3 is an in-vehicle camera that captures, for example, the driver's posture, eye position, and movement, constituting a DMS (Driver Monitoring System). By analyzing the images captured by the in-vehicle camera, it is possible to ascertain the driver's fatigue level and line of sight. The exterior camera also captures images of the surrounding conditions, such as the front and rear of the vehicle 100. By analyzing the images captured by the exterior camera, it is possible to ascertain the presence or absence of other vehicles or people around the vehicle 100, buildings, terrain, road conditions such as rain, snow, ice, and unevenness, and road signs. The exterior camera also includes a drive recorder that records the driving conditions as video.

[0049] The wireless transceiver for road-to-vehicle communication generates road-to-vehicle communication information through road-to-vehicle communication between the vehicle 100 and roads, signs, traffic lights, etc. The wireless transceiver for vehicle-to-vehicle communication generates vehicle-to-vehicle communication information through vehicle-to-vehicle communication between the vehicle 100 and other vehicles in the vicinity. The GPS receiver generates GPS information by receiving GPS signals from GPS satellites. For example, the current time, latitude, and longitude can be acquired as GPS information. The VICS receiver generates VICS information obtained by receiving VICS signals. The GPS receiver and VICS receiver may be provided as part of a navigation system.

[0050] The control device 401 of the HUD device 1 controls the display of the image display unit 402 based on the vehicle information 406, etc., and forms a virtual image 106 by image light in the display area 105. The control device 401 of the HUD device 1 also controls the display of the sub-LCD 2 based on the vehicle information 406, etc.

[0051] The MCU 411 includes a processor such as a CPU (Central Processing Unit), a memory, and various peripheral functions. The MCU 411 in the control device 401 may have a configuration as shown in FIG. 5B. In FIG. 5B, functional blocks implemented by the MCU 411 include a video data generation unit 412, a distortion correction unit 413, a pitching correction unit 414, a light source adjustment unit 415, a mirror adjustment unit 416, and an audio data generation unit 417. These units are mainly implemented by the CPU of the MCU 411 reading and executing a program stored in the nonvolatile memory 421 or the volatile memory 422. The control device 401 is not limited to being implemented using the MCU 411, and may be implemented using an ECU or other semiconductor device.

[0052] The MCU 411 receives and acquires vehicle information 406 via the vehicle information acquisition unit 407. Based on the vehicle information 406 and the like, the MCU 411 generates video information (video data) for the video display device 403, audio data for the speaker 405, and the like.

[0053] In the configuration of FIG. 5B , video data generation unit 412 generates video data (an original image, described later) that determines the content of the video to be displayed on display panel 4 to form virtual image 106 in display area 105, based on vehicle information 406 and the like. Distortion correction unit 413 performs distortion correction processing, described later, on the video data from video data generation unit 412, and generates corrected video data. Distortion correction corrects video distortion that occurs in accordance with the curvature of windshield 103 when video light from video display device 403 is projected onto windshield 103. Furthermore, pitch correction unit 414 performs pitch correction processing, described later, on the video data from video data generation unit 412, and generates corrected video data. In the first embodiment, both distortion correction and pitch correction are performed on the video data.

[0054] Display driver 423 drives the display elements of display panel 4 of video display device 403 based on the video data after distortion correction and pitch correction. As a result, display panel 4 creates and displays on the display screen an image to be projected onto display area 105.

[0055] The light source adjustment unit 415 controls the brightness of an LED element (to be described later) of the light source device 5 of the video display device 403 .

[0056] When adjusting the position of the display area 105, the mirror adjustment section 416 controls the drive mechanism 404 provided on the reflecting mirror 3b of the image display unit 403 to change the angle of the reflecting mirror 3b.

[0057] The voice data generating unit 417 generates voice data based on the vehicle information 406, etc., as necessary. The voice data is generated, for example, when providing voice guidance for the navigation system or when issuing a warning to the driver using the AR function. The voice driver 424 drives the speaker 405 based on the voice data, causing the speaker 405 to output voice.

[0058] 5A, non-volatile memory 421 mainly stores in advance programs executed by the CPU in MCU 411, setting parameters used in the processing of each unit in MCU 411, specified video data, audio data, etc. Volatile memory 422 mainly stores acquired vehicle information 406 and various data used in the processing of each unit in MCU 411 as appropriate.

[0059] The communication unit 408 is a device equipped with a communication interface, and communicates with the outside of the HUD device 1 based on a communication protocol such as CAN or LIN. The communication unit 408 and the vehicle information acquisition unit 407 may be integrated. Each unit of the control device 401 in FIG. 5A may be implemented by a dedicated circuit such as an FPGA (Field Programmable Gate Array).

[0060] [HUD device functional block configuration example (2)] 6A shows an example of a functional block configuration for controlling two types of displays (AR-HUD and sub-LCD 2) in the HUD device 1 based on FIG. 5A. In the example configuration of FIG. 6A according to the first embodiment, the control device 401 has one MCU 411, and this one MCU 411 controls both the display panel 4 and the sub-LCD 2. The display panel 4 is a main display device made up of, for example, an LCD, and the sub-LCD 2 is a sub-display device.

[0061] 6B is a modified example of the configuration of FIG. 6A, in which the control device 401 has one control microcomputer 503, and this one control microcomputer 503 controls both the display panel 4 and the sub-LCD 2. In FIG.

[0062] The MCU 411 in FIG. 6A (similar to the control microcomputer 503 in FIG. 6B) has, as functional blocks, a display control unit 500, a first video data generation unit 501, a second video data generation unit 502, etc. The video control unit 500 determines how video data to be displayed, generated based on the vehicle information 406, etc., should be displayed, including two types of display destinations (for example, FIG. 10 described below). Then, the video control unit 500 controls the first video data generation unit 501, the second video data generation unit 502, etc., based on the determination.

[0063] The first video data generation unit 501 generates first video data to be displayed on the display panel 4 when the display destination is an AR-HUD, and drives and controls the first display driver 511. The second video data generation unit 502 generates second video data to be displayed on the sub-LCD 2 when the display destination is the sub-LCD 2, and drives and controls the second display driver 512. The first display driver 511 drives the display panel 4 based on the first video data. The second display driver 512 drives the sub-LCD 2 based on the second video data.

[0064] The MCU 411 reads and writes video data (first video data and second video data) from and to a memory 520 that is internal or external to the MCU 411. The memory 520 may be the non-volatile memory 421 in FIG. 5A or the like. The example in FIG. 6A shows a configuration example in which both first video data 521 to be provided to the display panel 4 and second video data 522 to be provided to the sub-LCD 2 are stored in the same memory 520.

[0065] 6A, the first video data 521 is specifically stored and managed separately as image data (e.g., first image data d11) and character data (e.g., first character data d12). Similarly, the second video data 522 is stored and managed separately as image data (e.g., first image data d21) and character data (e.g., first character data d22). The image data here refers to data of two-dimensional or three-dimensional images (still images or moving images) in a format such as a bitmap. The character data here refers to data containing character information such as character codes, character fonts, and enlargement / reduction rates from which character images are generated.

[0066] The speech bubble also illustrates an example of a detailed data structure of the video data. For example, the first image data d11 has an ID, event information, a display destination, display conditions, attributes, and the video data itself. The ID is identification information for the video data. The event information is information indicating the event that triggered the generation of the video data. The display destination is information indicating the display destination (two types of displays) when the display destination (two types of displays) for the video data has been determined (for example, 0 for the main display panel 4 and 1 for the sub LCD 2). The display conditions are information indicating the conditions and methods (for example, display time and display end conditions) for how the video data is displayed on the display destination (for example, double display and moving display, which will be described later). The attributes of the video data include various attribute information such as type, priority or importance, data volume, information volume, and size (display size). The type is information such as the classification of the video information, such as warning, caution, route guidance, vehicle speed, and facility information.

[0067] Fig. 6C shows an example of a functional block configuration of the control device 401 in a modification of Fig. 6A. In this modification, the control device 401 has two control microcomputers (601, 602) independently arranged in parallel, with the control microcomputer 601 controlling the display panel 4 and the control microcomputer 602 controlling the sub-LCD 2. In the control device 401, the control microcomputer 601, the control microcomputer 602, the vehicle information acquisition unit 407, the communication unit 408, and the like are interconnected via a bus 630.

[0068] The control microcomputer 601 has, as functional blocks, a display control unit 610 and a first video data generation unit 501. The display control unit 610 determines how the video data to be displayed should be displayed, including two types of display destinations, and controls the first video data generation unit 501 when the display destination is the display panel 4. The display control unit 610 communicates with the control microcomputer 602 and gives instructions to the display control unit 620 when the display destination is the sub-LCD2 for the video data to be displayed. The display control unit 620 controls the second video data generation unit 502 when the display destination is the sub-LCD2 for the video data to be displayed.

[0069] 6C , as an example of a memory configuration, a memory 641 used by the control microcomputer 601 and a memory 642 used by the control microcomputer 602 are provided independently and in parallel. The control microcomputer 601 accesses the memory 641 to read and write the first video data 521. The control microcomputer 602 accesses the memory 642 to read and write the second video data 522.

[0070] As another example of memory configuration, the memory used by the control microcomputer 601 and the memory used by the control microcomputer 602 may be one common memory, and the first video data 521 and the second video data 522 may be stored in that common memory, similar to memory 520 in FIG. 6A.

[0071] The configuration example in FIG. 6B is a configuration example in which one control microcomputer 503 controls two systems of video output (two types of display destinations). In this configuration example, since there is only one control microcomputer, it saves space and is low cost. However, this single control microcomputer is required to have the performance to control the two systems of video output and output them in sync without delay. In cases such as synchronizing the video display on two types of displays, this single control microcomputer configuration example can easily accommodate this.

[0072] The configuration example in Figure 6C is an example in which two control microcomputers (601, 602) control the video output of each system. Each control microcomputer basically only needs to control one corresponding type of video output. However, when synchronizing the video output of two systems, the two control microcomputers must communicate with each other to synchronize them.

[0073] In the memory configuration example of FIG. 6B, one control microcomputer 503 is provided with one memory 520, and data is allocated and stored in memory 520 with first video data 521 for display panel 4 and second video data 522 for sub-LCD 2 at separate addresses. While it would be possible to store all data in a single block without separating them, storing data separately according to the display destination, and separating the data into image data and character data, offers significant advantages. For example, when only second video data 522 for sub-LCD 2 needs to be replaced or updated, only the second video data 522 can be updated efficiently. When the block of all data mixed together is large, updating takes time because data that does not need to be updated is also rewritten.

[0074] 6C has two control microcomputers (601, 602) each having their own memory (641, 642), and the data is arranged such that the first video data 521 for the display panel 4 is arranged in the memory 641, and the second video data 522 for the sub-LCD 2 is arranged in the memory 642. Storing data separately according to the display destination in this way, and dividing the data into image data and character data, offers significant advantages.

[0075] [HUD device activates] FIG. 7 shows an example of an operation flow at the start-up of the HUD device 1. In step S1, the control device 401 of the HUD device 1 first starts display preparation on the sub-LCD2. Next, in step S2, the control device 401 of the HUD device 1 starts display preparation on the video display unit 402 including the display panel 4 corresponding to the AR-HUD. In step S3, the HUD device 1 waits until display preparation on the sub-LCD2 is complete. In step S4, the HUD device 1 starts display on the sub-LCD2. In step S5, the HUD device 1 waits until display preparation on the AR-HUD is complete. In step S6, the HUD device 1 starts display on the display panel 4 corresponding to the AR-HUD.

[0076] In step S7, the HUD device 1 checks whether the conditions for transitioning the HUD device 1 to an off state or a sleep state are met. Examples of such conditions include when the user presses the power-off button or when a certain period of time has passed without any display change. If the conditions are met, in step S8, the HUD device 1 first stops the display on the sub-LCD 2. Next, in step S9, the HUD device 1 stops the display on the display panel 4 corresponding to the AR-HUD. In step S10, the HUD device 1 starts a display termination process for the AR-HUD. In step S11, the HUD device 1 starts a display termination process for the sub-LCD 2. In step S12, the HUD device 1 waits until the display termination process for the sub-LCD 2 is completed. In step S13, the HUD device 1 waits until the display termination process for the AR-HUD is completed. After these display termination processes are completed, the HUD device 1 transitions to an off state or a sleep state in step S14.

[0077] Generally, preparations related to the AR-HUD take more time than preparations related to the sub-LCD 2, so the above flow is used.

[0078] [Internal configuration example of HUD device] Fig. 8 shows the display panel 4 and light source device 5 as an example of the configuration of the image display device 403 (Fig. 5A) of the HUD device 1. Fig. 8 shows a schematic cross-sectional view taken along the YZ plane. The light source device 5 (the portion other than the display panel 4 in Fig. 8) and the display panel 4 are disposed within the housing 102A (Fig. 4).

[0079] The light source device 5 includes, in order from the light source side, an LED substrate 801, a collimator 802, a polarization conversion element 803, a light guide 804, and a diffuser 805. The display panel 4 is arranged behind (on the output side of) the diffuser 805. The reflecting mirror 3a of FIG. 3 is arranged behind the display panel 4. In this configuration example, as shown in the figure, the LED substrate 801, collimator 802, polarization conversion element 803, and light guide 804 are arranged along the Y direction, and the diffuser 805 and display panel 4 are arranged from the light guide 804 along the Z direction. Note that such a light source device 5 is merely an example and is not limited to this.

[0080] The LED substrate 801 is a substrate having a plurality of LED elements 801A as semiconductor light source elements. Note that, since Fig. 8 is a YZ cross section, only one LED element 801A and one collimator element are shown, but a plurality of LED elements 801A are similarly provided in the X direction on the main surface (XZ plane) of the LED substrate 801, and correspondingly, a plurality of collimator elements are similarly provided in the X direction on the collimator 802.

[0081] A collimator 802 is provided on the output side (Y direction) of light from the LED elements 801A of the LED substrate 801. The collimator 802 is an element that controls the traveling direction of light, and converts the light from the LED elements 801A into approximately parallel light and outputs it. A polarization conversion element 803 is provided on the output side (Y direction) of the light from the collimator 802. The polarization conversion element 803 is an element that aligns the polarization characteristics, and converts light having random polarization as the approximately parallel light from the collimator 802 into light having linear polarization. The polarization conversion element 803 is configured by combining a polarization conversion prism and a wave plate.

[0082] A light guide 804 is provided on the exit side (Y direction) of light from the polarization conversion element 803. Light guide 804 receives linearly polarized light in the Y direction from the polarization conversion element 803 at an entrance portion, and performs light distribution control by reflecting the light by a reflecting portion in a Z direction different from the Y direction, i.e., toward the display panel 4, and then emits the light from the exit portion. Light guide 804 has a reflecting portion that performs reflection and light distribution control. The reflecting portion is formed by alternating reflection surfaces and connecting surfaces. The exit surface of the exit portion has, for example, a free-form curved surface shape for light distribution control.

[0083] Light emitted from the exit portion of the light guide 804 travels generally upward in the Z direction, and in this example, slightly diagonally to the upper right. A diffuser 805 is provided on the light exit side of the light guide 804. The light from the light guide 804 is diffused by the diffuser 805 and enters the back side of the display panel 4. The display panel 4 uses this incident light as backlight to generate image light. This image light from the display panel 4 is directional image light.

[0084] [Two types of displays] In the HUD device 1 according to the first embodiment, the basic characteristics of the two types of displays, the display area 105 of the AR-HUD and the sub-LCD 2, are as follows.

[0085] The virtual image 106 displayed in the display area 105 of the AR-HUD based on the display panel 4 is a directional image, and therefore can be accurately viewed only from a viewpoint 108 within a set eye box when the driver is in the driving posture seated in the driver's seat of the vehicle 100 shown in Fig. 3. The virtual image 106 in the display area 105 cannot be easily viewed by passengers in the vehicle.

[0086] The virtual image 106 is inferior to the real image on the sub-LCD2 in terms of image clarity and resolution. Conversely, the image on the sub-LCD2 is clearer than the virtual image 106. The visibility of the virtual image 106 is also easily affected by the surrounding brightness and the content of the real scene. Furthermore, because the windshield 103 has a curved surface, distortion occurs in the image projected onto the windshield 103 according to the curvature. This distortion requires some kind of correction (for example, the distortion correction described above), and can be eliminated by such correction. However, this distortion is not always completely eliminated, and in particular, fine display content such as text images on the virtual image 106 may be difficult to see.

[0087] The display screen 20 of the sub-LCD2 is disposed, for example, in a position directly facing the driver in the driver's seat, as shown in FIG. 1 etc. The image on the sub-LCD2 is a diffused image, so it is visible even if the driver's viewpoint position moves within the space. This image is visible even when the driver is not in a predetermined driving posture. Passengers in the vehicle can also view the image on the sub-LCD2. Naturally, the image on the sub-LCD2 cannot achieve effects such as AR by superimposing it on a real scene. The image on the sub-LCD2 takes less time to become ready to display than the virtual image 106. The image on the sub-LCD2 requires less preparation time to start displaying than the virtual image 106. The image on the sub-LCD2 does not lose visibility even when displaying small details such as characters.

[0088] The HUD device 1 according to the first embodiment has a function of controlling the use of the two types of displays in consideration of the characteristics of the two types of displays described above. This control is performed by, for example, the display control unit 500 in Fig. 6B or the display control unit 610 in Fig. 6C.

[0089] [How to use it (1)] An example of controlling the use of the above two types of displays is described below. The display of virtual image 106 in display area 105 based on display panel 4 is used for AR-only or AR-focused display. This virtual image 106 display is effective when used for three-dimensional stereoscopic image representation. When displaying image information other than AR, two-dimensional image representation is used. The display area 105 or sub-LCD2 is used to display two-dimensional image information. For image information that is temporarily required, the virtual image 106 in display area 105 is mainly used. For image information that needs to be displayed at all times, the sub-LCD2 is mainly used.

[0090] The sub-LCD2 displays two-dimensional image information as non-AR image information. The sub-LCD2 displays image information that should be displayed to the driver at all times. The sub-LCD2 clearly displays two-dimensional image information (such as text images) in color against a black background, for example.

[0091] [Display example (1)] FIG. 9 shows a display example in which two types of displays are used in the HUD device 1 of the first embodiment. (A) of FIG. 9 shows an image including an actual scene through the windshield 103, including the display area 105 seen by the driver, and an image on the display screen 20 of the sub-LCD 2. In the display area 105, an AR virtual image 901 is displayed superimposed on an actual scene such as a road. This AR virtual image 901 is, for example, route guidance information (e.g., an image consisting of multiple triangles) for navigating a right-turn route. On the other hand, the sub-LCD 2 mainly displays video information 902 including text images such as the current vehicle speed ("40 km / h") and the distance to the right-turn point (right-turn guidance). Note that the video information 902 is displayed on the display screen 20 in a specified color, such as white on a black background, but the text is illustrated in black in this drawing. The display content of the display screen 20 is not limited to this and various other contents are possible.

[0092] 9B shows another example of display. When a preceding vehicle is detected in the vehicle's own lane, an AR virtual image 903 (e.g., a ring-shaped image) is displayed in the display area 105 to call attention to the preceding vehicle. Meanwhile, the sub-LCD 2 displays video information 904 including the vehicle speed, road name, and a text image such as "vehicle ahead."

[0093] 9A and the like illustrates a gap region 900 in the vertical direction (Z direction) between the display region 105 and the sub-LCD 2. When the driver views the front from the viewpoint of the driver's driving posture, such a gap region 900 may be present. This gap region 900 occurs depending on the mounting configuration of the HUD device 1 itself including the sub-LCD 2, the mounting configuration of the HUD device 1 in the vehicle as shown in FIG. 1 and the like. The present invention is not limited to a configuration in which the gap region 900 is present, and a configuration in which the gap region 900 is not present may also be possible.

[0094] [How to Use It (2)] When displaying certain video information, the HUD device 1 determines the display destination depending on the type of the video information, etc. For example, if the video information to be displayed can be displayed either as a virtual image 106 in the display area 105 of the AR-HUD or as a video display on the sub-LCD 2, the HUD device 1 selects between displaying the video information in the display area 105 and displaying the video information on the sub-LCD 2, taking into consideration the type of the video information, the amount of information, etc. The HUD device 1 selects and determines whether to display the video information on the display area 105 or the sub-LCD 2 based on the driving conditions at the time and the amount of information in the display area 105 and the display screen 20 within the driver's field of view. An example of such control is described below.

[0095] In a conventional HUD device having only one display (AR-HUD), an image with an information volume of 100 is displayed on the display. In contrast, the HUD device of the first embodiment can distribute and display the image with an information volume of 100 between two types of displays: the display area 105 and the sub-LCD 2. Note that the information volume here is an abstract concept, but is an amount associated with the magnitude of the load required for processing, such as recognition, on the driver. In the first embodiment, the image with an information volume of 100 can be distributed, for example, by displaying it on the display area 105 as a virtual image 106 with an information volume of 60 and on the sub-LCD 2 as an image with an information volume of 40. This control can limit the information volume of the image displayed as the virtual image 106 on the display area 106 to a certain information volume or less.

[0096] This ensures the field of view necessary for safe driving in the driver's field of vision, including the display area 105, while providing support through the virtual image 106 and sub-LCD2 to an extent that does not place too much strain on the driver, thereby reducing the effects of sickness, dizziness, fatigue, etc. on the driver.

[0097] [Example of control taking into account the amount of information] FIG. 10 shows a processing flow of the HUD device 1 relating to a control example that takes into account the amount of information, etc. In step S101, for example, the control device 401 of FIG. 6B receives video information to be displayed at various times based on the vehicle information 406, etc. The control device 401 can grasp the vehicle and surrounding conditions from the vehicle information 406, etc. The control device 401 obtains information about the video information to be displayed, such as the type, data amount, size, priority, or importance. This information may be information attached to the video information in the form of attribute information, etc., or information associated with and stored, as in the example of FIG. 6B, or may be newly generated by the control device 401. The control device 401 may, for example, calculate the amount of information of the video information.

[0098] The "type" is information indicating, for example, whether the video information is AR or non-AR, whether it is a three-dimensional (3D) image or a two-dimensional (2D) image, whether it is text information, or whether it is a GUI (graphical user interface such as a menu). The "type" may be classified as a warning, caution, route guidance, vehicle speed, facility information, or advertising information. The "type" may also be information indicating whether the type should be displayed as a virtual image 106 in the display area 105, whether it should be displayed on the sub-LCD2, or whether it can be displayed on either of the two types of displays. One type of "type" is text information, which is expressed, for example, by character codes. Using the text information as source data, a text image (e.g., a bitmap-like text image having two-dimensional pixel values) to be displayed on the AR-HUD or the sub-LCD2 is created.

[0099] "Data volume" refers to the amount of data stored in memory for that video information. "Size" refers to the size, vertical and horizontal dimensions, etc., of the display when that video is displayed on the display area 105 or the display screen 20 of the sub-LCD 2. Priority or importance is not essential, but if provided, it is information that indicates the level of priority or importance that video information has among all other video information. For example, video information such as an emergency warning is assigned the highest priority value. Video information that alerts the driver to objects around the vehicle is assigned a lower priority value than warnings. Route guidance information is assigned an even lower priority value. Facility information, advertising information, etc. are assigned lower priority values.

[0100] In step S102, the control device 401 determines the display destination (one of two types of displays) for each piece of video information to be displayed that is currently being received, using the various types of information in step S101. First, for a type of video information for which a single display destination has been determined in advance, the control device 401 provisionally determines that display destination. That is, for a predetermined type of video information such as AR, the control device 401 provisionally determines the display destination to be the display panel 4 (the corresponding display area 105). For a predetermined type of video information such as text information, the control device 401 provisionally determines the display destination to be the sub-LCD 2.

[0101] In step S103, when there are multiple pieces of video information to be displayed, the control device 401 sequentially determines which piece of video information to actually display. At this time, the control device 401 determines the video information to be displayed one by one and the display destination, for example, in order of type or priority. The control device 401 determines the display destination for each piece of video information, for example, in order of priority. At this time, the control device 401 also counts the amount of display information for the display area 105 based on the display panel 4 and the sub-LCD 2. For example, the amount of display information for the display area 105 is the total amount (e.g., sum) of the amounts of information for one or more virtual images 106 within the display area 105. Similarly, the amount of display information for the sub-LCD 2 is the total amount (e.g., sum) of the amounts of information for one or more pieces of video information within the display screen 20. In addition, for video information of a type that can be displayed on either display, the control device 401 selects the display panel 4 or the sub-LCD 2 as the display destination so as to satisfy the restriction that the amount of display information on each of the display panel 4 and the sub-LCD 2 is equal to or less than a predetermined amount.

[0102] In step S103, when the control device 401 calculates the amount of information of the video information, the data amount or size of the video information may be taken into consideration. Alternatively, the amount of information may be set in advance for each piece of video information. For example, the larger the data amount or size of the video information, the larger the amount of information may be. The amount of information and the amount of displayed information represent the magnitude of the burden and ease / difficulty of the driver when viewing or recognizing the video information or the screen, and may also be referred to as the amount of load or index value.

[0103] FIG. 11 shows a display example according to the control example of FIG. 10. At a certain time, an AR virtual image 1101 indicating caution against a preceding vehicle and an AR virtual image 1102 warning about the distance between vehicles are displayed in the display area 105. The control device 401 displays one or more of these virtual images 106 within a range in which the amount of display information in the display area 105 is equal to or less than a predetermined amount. At this time, it is assumed that the control device 401 also receives right-turn route guidance information similar to that shown in FIG. 9A as the video information to be displayed. However, when displaying the right-turn route guidance information as AR in the display area 105, the control device 401 determines that the amount of display information in the display area 105 exceeds the predetermined amount and that the order of priority is warning, caution, and route guidance. Therefore, the control device 401 determines to prioritize the display of the virtual images 1101 and 1102 and display them as they are, but not to display the AR of the right-turn route guidance.

[0104] Meanwhile, the control device 401 is displaying video information 1103, 1104 using character images on the display screen 20 of the sub-LCD2, within the limited range of the amount of display information on the sub-LCD2. The video information 1104 is the vehicle speed. The control device 401 determines and displays non-AR video information 1103 (an image showing a right turn 50 m ahead) on the sub-LCD2 for route guidance information for a right turn. In this way, the HUD device 1 of the first embodiment can provide driving assistance using suitable displays by using two types of displays appropriately.

[0105] [Use: Initial setting] FIG. 12 shows an example of a display example of the initial setting of the display area 105 of the HUD device 1 as an example of control for selectively using two types of displays. When a driver uses the HUD device 1 for the first time, adjustments (also called calibrations) are performed to appropriately initialize the height of the display area 105 and other settings to match the driver's driving posture and eye position (including the eyebox). The HUD device 1 allows this adjustment when the vehicle is safely stopped, but does not allow it while the vehicle is moving. In an AR-HUD, such adjustment of the display area 105 for each driver is essential. If this adjustment is neglected, the driver will not be able to properly view the virtual image 106 in the display area 105. For example, it will be impossible to properly align and superimpose the AR virtual image 106 on an object such as a vehicle or person in the real scene. In more serious cases, the driver will not be able to view the virtual image 106 at all.

[0106] Therefore, when adjusting the display area of an AR-HUD, conventional HUD devices display image information for adjusting the position of the display area (including a virtual image) in the display area, and the user adjusts the position of the display area (including a virtual image) in response to an operational input from the user. However, because the image information for adjustment itself is displayed in the display area as an AR virtual image, there are cases where the image information itself is not visible to the driver. In such cases, adjustment is time-consuming.

[0107] Therefore, in the HUD device 1 of the first embodiment, during adjustment of the initial settings, not only does it display image information for adjustment in the display area 105, but it also displays image information for guiding and supporting the adjustment on the sub-LCD 2. In other words, the HUD device 1 uses two types of displays to display image information for adjusting the display area 105 and for guiding and supporting the driver, who is the user. The user performs adjustment operations according to the image information.

[0108] In the example of FIG. 12, first, adjustment virtual images 1201 and 1202 are displayed in the display area 105. The virtual image 1201 is an image representing the four corners of a rectangular frame, and the virtual image 1202 is a grid image. Conventionally, the display area 105 is adjusted so that the grid virtual image 1202 fits within the frame virtual image 1201 when viewed from the driver's viewpoint. The adjustment operation can be performed, for example, by operating a remote control attached to the HUD device 1.

[0109] The HUD device 1 according to the first embodiment further displays video information 1203 for guiding and assisting the adjustment on the display screen 20 of the sub-LCD2. The video information 1203 may be, for example, a text image of a guide message such as "Does the grid fit within the frame ahead? Yes / No," or "Adjust the height so that the grid fits within the frame ahead," or other GUI components. Even if the user cannot clearly see the adjustment images (virtual images 1201 and 1202) in the display region 105, the user can still see the video information 1203 on the sub-LCD2, and can therefore more easily make adjustments by following the guidance provided by the video information 1203. After adjusting the virtual image 1202 to match the virtual image 1201, the driver ends the adjustment by operating a remote control, for example. The HUD device 1 saves and sets the state of the display region 105 after the adjustment.

[0110] The sub-LCD 2 may be a touch panel, in which case the user can also perform touch input operations on the display screen 20. For example, the user can also select "Yes / No" in the video information 1203 on the display screen 20 and then perform a touch operation to confirm the adjustment.

[0111] The user can also operate the HUD device 1 via an operation interface (e.g., a button, switch, key, etc.) installed on the steering wheel 104 ( FIG. 1 ) of the vehicle or the like. This operation can also be used for the above adjustment. In this case, a higher-level controller (e.g., the vehicle's ECU) of the HUD device 1 receives information operated through the operation interface, and the HUD device 1 operates by receiving operation information from the higher-level controller via CAN communication or the like. In this case, the processing flow is, for example, as follows: When an adjustment is selected by a user operation on a menu screen displayed on a control panel or the like of the vehicle, the HUD device 1 receives operation information of the operation from the higher-level controller, transitions to adjustment mode, and starts displaying a virtual image related to the adjustment. The user performs an operation, such as up, down, left, or right, on the operation interface to make the adjustment. The HUD device 1 receives the operation information of the operation from the higher-level controller and adjusts the display in the display area 105 (e.g., the virtual image 1201) according to the operation. When the adjustment is completed, the user performs a confirmation operation. The HUD device 1 receives operation information of the decision operation from the upper controller, reflects and saves the adjustment result, and ends the adjustment mode.

[0112] FIG. 13 shows a processing flow for adjusting the initial settings. In step S201, the HUD device 1 determines whether or not to adjust the display area 105 of the AR-HUD, for example, at startup. If adjustment is to be performed, the process proceeds to steps S202 and S203. If adjustment is not to be performed, the process proceeds to steps S206 and S207. In steps S202 and S203, the HUD device 1 displays a virtual image 106 for adjusting the display area 102, as shown in FIG. 12, in the display area 105 of the AR-HUD, and displays image information for adjustment support on the sub-LCD 2. Next, in step S204, the HUD device 1 adjusts the display area 105 in accordance with the adjustment operation by the driver. In step S205, the HUD device 1 confirms whether readjustment is to be performed (Y) or whether the adjustment is complete (N). For example, if the driver has not input an operation to complete the adjustment, the process determines that readjustment is to be performed, and returns to steps S202 and S203 to repeat the same process. If the adjustment is complete (N), the process proceeds to steps S206 and S207. In steps S206 and S207, the HUD device 1 starts normal display of the virtual image 106 in the display area 105 of the adjusted AR-HUD, and starts normal display of the image on the sub-LCD 2.

[0113] [Adjustment using the sub LCD] FIG. 14 shows another control example and display example relating to adjustment support using the sub-LCD 2 for adjustment during the initial setup. The HUD device 1 of the first embodiment uses the sub-LCD 2 to support adjustment of the height position of the display region 105 of the AR-HUD (in other words, the AR display position). (A) of FIG. 14 shows a display example during adjustment. In the case of initial setup using this function, the HUD device 1 displays, as shown in (A), image information for adjustment of the display region 105 (e.g., a frame virtual image 1201 and a grid virtual image 1202) similar to that shown in FIG. 12 on the display region 105, while displaying image information 1401 for adjustment support of the display region 105 on the display screen 20 of the sub-LCD 2. Note that FIG. 14 shows an enlarged view of the sub-LCD 2.

[0114] The video information 1401 includes, as example contents, information indicating that the AR-HUD is in adjustment mode (a mode for performing adjustments during initial setup), as well as information such as a menu, a guide message, and a cursor. An example of a guide message is a text image urging the user to adjust the adjustment image (virtual images 1201, 1202) in the display area 105, such as "↑ Adjust so that the grid fits within the frame." In this example, the HUD device 1 also displays, on the sub-LCD 2, video information 1402 indicating the content currently displayed as the adjustment image (virtual images 1201, 1202) in the display area 105. The HUD device 1 also displays, as guide messages, text images indicating the content currently displayed in the display area 105 of the AR-HUD, such as "← Adjustment image currently displayed on the AR-HUD" and "※ If you cannot see, please adjust your posture, etc.", as well as text images urging the user to take action if the content cannot be seen.

[0115] In response to the adjustment operation input by the driver, the control device 401 of the HUD device 1 updates the display state of the adjustment image information (virtual images 1201, 1202) in the display area 105 to reflect the adjustment result, and also updates the corresponding image information 1401, 1402 on the sub-LCD 2. The driver makes adjustments according to the adjustment support image information 1401, 1402 on the sub-LCD 2 so that the grid virtual image 1202 fits within the frame virtual image 1201 of the display area 105 when viewed from the viewpoint in the driving posture. After the image information (virtual images 1201, 1202) of the display area 105 has been adjusted to a suitable state, the driver makes an input indicating completion of the adjustment. In response to the input, the HUD device 1 saves the adjustment state of the display area 105 at that time and ends the adjustment mode.

[0116] As a result, even if the driver cannot clearly see the images for adjustment (virtual images 1201, 1202) in the display area 105, the driver can make adjustments more easily by looking at the adjustment support image information 1401, 1402 on the sub-LCD2. Even if the driver cannot see the images for adjustment in the display area 105 at all, the driver can recognize discrepancies by looking at the image information 1402 on the sub-LCD2 and easily determine whether adjustments are necessary. Even if the driver cannot clearly see the image information in the display area 105, the driver can adjust his / her seat or posture according to the image information 1401, 1402 on the sub-LCD2 so that the image information in the display area 105 is visible.

[0117] 14B shows another example of the display of adjustment support related to (A). The control device 401 of the HUD device 1 displays an example of the AR virtual image 106 (in this example, a virtual image 1403 corresponding to an AR object for route guidance) in the display area 105, and also displays image information 1404 for adjustment related to the AR virtual image 106 in the display area 105 on the sub-LCD 2. The driver can select the AR to be adjusted. Multiple AR virtual images may be displayed in the display area 105, and each may be adjustable.

[0118] Examples of content that can be adjusted or changed using the video information 1404 on the sub-LCD2 include the color (or brightness), size, shape, tilt, and position of the displayed AR virtual image 1403. GUI components are displayed on the sub-LCD2 for each adjustable item, such as color. Examples of GUI components include a bar or palette for changing the color (or brightness), a button for changing the size or ratio, a list box for selecting a shape (e.g., a triangle, an arrow, etc.), a button for changing the tilt (such as the tilt relative to a horizontal plane), and a button for changing the position. The driver can adjust the virtual image 1403 for each adjustable item by operating a remote control, etc. The control device 401 updates the display state of the AR virtual image 1403 in response to the adjustment operation input and saves the determined adjustment state.

[0119] When using the above function, the adjustment support video information is provided using the sub-LCD2, compared to conventional cases where the video information for adjustment is displayed only in the display area of the AR-HUD, so a wider screen area for adjustment can be used, making adjustments easier and more detailed.

[0120] [Usage: When an error occurs] 15 shows another example of display differentiation in the event of an error. When one of the two types of displays (the display area 105 and the sub-LCD 2) experiences an error such as being unable to display or displaying poorly, the HUD device 1 notifies the user of the error using the display on the other display.

[0121] 15A, when an error occurs in the display of the display area 105 of the AR-HUD, the control device 401 displays, on the sub-LCD 2, video information 1501 notifying the user that an error has occurred in the display of the display area 105. The video information 1501 is, for example, a text image stating, "Due to poor communication conditions, the HUD cannot be displayed. It will be displayed when communication conditions improve."

[0122] The control device 401 detects an error state in which the virtual image 106 cannot be correctly displayed in the display area 105 due to various circumstances and causes. Examples of circumstances and causes include a failure of a component related to the AR-HUD display, a sunlight protection function that stops the display, or a CAN communication problem that prevents ADAS information (ADAS: advanced driving assistance system) from being acquired and the display from being unable to be updated.

[0123] When an error state occurs, the control device 401 notifies the user that the AR-HUD cannot correctly display a virtual image due to the error state in each of the above cases (circumstances and causes) by displaying on the sub-LCD 2. Furthermore, if the HUD device 1 knows details of the error state, the prospects for recovery from the error state, and how to deal with the error, it also notifies the user by displaying on the sub-LCD 2.

[0124] The example in (B) of FIG. 15 shows a case where, when the display on the display screen 20 of the sub-LCD2 goes into an error state due to a wire break or other cause, a virtual image 1502 is displayed and notified in the display area 105 of the AR-HUD to indicate the error state of the sub-LCD2. The virtual image 1502 is, for example, an AR text image such as "Sub-LCD display is not possible due to a wire break. Please check the connection." When the control device 401 detects an error state of the sub-LCD2, it displays a virtual image 1501 in the display area 105 of the AR-HUD to notify the user of the error state of the sub-LCD2, its cause, and how to deal with the error. Note that such a display on the AR-HUD is only performed, for example, while the vehicle is stopped.

[0125] With the above-described function, even if one of the two types of displays is in an error state, the error state can be communicated to the user using the other display, thereby improving user convenience.

[0126] FIG. 16 shows the processing flow when an error occurs. In step S301, the HUD device 1 determines and detects whether the display in the display area 105 of the AR-HUD is in an error state. If it is in an error state, the process proceeds to step S302, and if it is normal, the process proceeds to step S305. In step S302, the HUD device 1 determines and detects whether the display on the sub-LCD 2 is in an error state. If it is in an error state, the process proceeds to step S303, and if it is normal, the process proceeds to step S304. In step S305, the HUD device 1 determines and detects whether the display on the sub-LCD 2 is in an error state. If it is in an error state, the process proceeds to step S306, and if it is normal, the process proceeds to the end of the flow.

[0127] The process proceeds to step S303 when both of the two types of displays are in an error state. In this case, the HUD device 1 cannot display on either display, and therefore notifies a higher-level controller for the HUD device 1, such as a vehicle electronic control unit (ECU), via communication that both of the two types of displays of the HUD device 1 are in an error state and therefore cannot display.

[0128] The process proceeds to step S304 when only the AR-HUD is in an error state. In this case, the HUD device 1 displays and notifies the user that the AR-HUD is in an error state on the sub-LCD 2, as in the example of (A) in Fig. 15 .

[0129] The process proceeds to step S306 when only the sub-LCD 2 is in an error state. In this case, the HUD device 1 displays and notifies the user that the sub-LCD 2 is in an error state in the display area 105 of the AR-HUD, as in the example of Fig. 15(B).

[0130] [Use: Situation] 17 shows another example of different display modes for different scenes. The HUD device 1 controls the display destination (two types of displays) and display contents of the video information to be changed depending on the vehicle's driving conditions and surrounding conditions (sometimes collectively referred to as the scene).

[0131] The differences in driving conditions include urban areas and highways. For example, in urban areas, on general roads, vehicles travel at relatively low speeds and there are many points to pay attention to when driving, making it easy to utilize AR displays. On the other hand, on highways, vehicles travel at relatively high speeds, and a driver's lack of concentration can lead to a serious accident. For this reason, AR displays are less effective on highways than on general roads.

[0132] Furthermore, if too many AR virtual images 106 are displayed in the display area 105 of the AR-HUD, they may obstruct the driver's field of vision and cause cognitive confusion due to the large amount of information. This must be avoided as it could cause an accident. Compared to ordinary roads, it is preferable to limit the amount of AR displayed in the display area 105 of the AR-HUD on expressways.

[0133] Therefore, for example, in a scene on an ordinary road, the HUD device 1 of the first embodiment gives top priority to displaying the AR virtual image 106 of a warning in the display area 105 of the AR-HUD, and does not display AR such as route guidance, which has a lower priority, or displays it within a predetermined limit of the amount of display information. Furthermore, when there is low-priority image information, such as route guidance or facility information, the HUD device 1 displays it mainly on the sub-LCD 2.

[0134] Furthermore, in a scene on a highway, for example, the HUD device 1 gives the highest priority to displaying a warning AR virtual image 106 in the AR-HUD display area 105, at a minimum, and does not display other ARs with lower priorities, or displays them in a limited manner within a predetermined limit. The HUD device 1 mainly displays low-priority image information on the sub-LCD 2. Route guidance information is useful even on highways, but limited AR display, such as displaying it when the vehicle is near an exit to exit the highway, is sufficient. The HUD device 1 normally displays route guidance information on the sub-LCD 2.

[0135] The HUD device 1 controls the display destination and display contents according to the various scenes as described above and the type, priority, amount of information, etc. of the video information as described above.

[0136] The HUD device 1 obtains information such as registered destination information, the vehicle's current driving position, and vehicle speed based on the vehicle information 406. Based on this information, the HUD device 1 determines and calculates the distance between the vehicle's current position and the destination (e.g., a right turn point, a highway exit, etc.), the estimated arrival time, etc. For example, when the vehicle approaches within a predetermined distance to the next destination (e.g., a highway exit), the HUD device 1 starts displaying AR such as route guidance information in the AR-HUD display area 105.

[0137] In addition, the AR display for warning other vehicles may be displayed only when another vehicle approaches in front of or around the vehicle. When a vehicle approaches from behind, the driver may be notified by the AR display or the display on the sub-LCD2.

[0138] The example in Figure 17 shows a scene on a highway, where (A) shows a display example at a first point in time while driving straight, and (B) shows a display example at a second point in time thereafter when approaching the highway exit. In (A), in order to ensure the driver's field of view, the HUD device 1 minimizes the display of the virtual image 106 in the AR-HUD display area 105, displaying nothing under normal circumstances and displaying a warning AR in the event of an emergency. The HUD device 1 displays various predetermined information, such as vehicle speed, on the sub-LCD 2.

[0139] In (B), when the vehicle approaches an expressway exit, which is the next destination, for example, when the distance between the vehicle's current position and the destination becomes equal to or shorter than a predetermined distance, the HUD device 1 starts displaying an AR virtual image 1701 of route guidance (e.g., a plurality of triangular images encouraging a left turn) in the display area 105. The HUD device 1 also starts displaying video information 1702 of text images for route guidance (e.g., "Exit 200 m ahead") on the sub-LCD 2 in association with the display of the virtual image 1701. When the vehicle reaches the destination, the HUD device 1 ends the display of the virtual image 1701 and the video information 1702 on the sub-LCD 2.

[0140] Another example of a situation is traffic congestion. During traffic congestion, the vehicle is traveling at a low speed or is stopped, so it is easy to utilize the display on the sub-LCD2 in addition to the AR display on the AR-HUD. During traffic congestion, the HUD device 1 displays video information related to the traffic congestion using the display area 105 and the sub-LCD2. For example, the HUD device 1 displays video information such as the traffic congestion situation, an estimate of the length of the traffic congestion, suggested detours, and the distance and time to rest areas. The HUD device 1 displays this video information preferentially in the display area 105 of the AR-HUD, and displays information that cannot be displayed in the display area 105 on the sub-LCD2. The AR warning display for the vehicle ahead is difficult to utilize when the vehicle ahead is too close due to traffic congestion, so it is possible to disable the AR display.

[0141] Another example of a situation is the weather. For example, when it is snowing or foggy, the visibility of the display area 105 of the AR-HUD decreases. The HUD device 1 determines the weather, etc. based on the vehicle information 406, and when it determines that the visibility is insufficient when displaying the AR virtual image 106 in the display area 105, it does not display the AR, and instead displays alternative image information only on the sub-LCD 2.

[0142] The HUD device 1 may also use two different display destinations depending on the destination. For example, in a normal situation such as when the driver is dropping off or picking up family members at a station (when the destination is known), the driver already knows the route, so the AR for route guidance is omitted in the display area 105 of the AR-HUD, and warning AR is the main display. The HUD device 1 displays information such as information about arriving trains and emails from family members on the sub-LCD 2. When the destination is unknown, the HUD device 1 displays the AR for route guidance in the display area 105 of the AR-HUD.

[0143] [User selection of display destination] The HUD device 1 according to the first embodiment has a function that allows the user to select and set whether to turn on or off the display of the virtual image 106 in the display area 105 of the AR-HUD and whether to turn on or off the video display on the sub-LCD 2. The display states of the two types of displays are controlled according to the selection and setting. The user may also be able to select and set what type of information is displayed on each of the AR-HUD and the sub-LCD 2.

[0144] FIG. 18 shows display examples related to this function. First, (A) shows an example in which the on / off of the AR-HUD display and the like are displayed in a menu on the sub-LCD2, allowing the user to select. Video information 1801 of the function menu is displayed on the display screen 20 of the sub-LCD2. In this menu, the AR-HUD display options include "Yes," "Yes (Partially)," and "No," and the user can select from these options. The selection operation may be performed by touch input as described above, or by using a remote control or the like. For example, when "Yes" is selected, the display of the virtual image 106 in the display area 105 is turned on, and when "No" is selected, the display of the virtual image 106 in the display area 105 is turned off. When "Yes (Partially)" is selected, the display of the virtual image 106 is turned on for only some of the types of information that have been set. The user can also set some of the types of information in another menu to which the user is redirected.

[0145] (B) shows an example in which the AR-HUD menu is displayed, allowing the user to select whether to turn the display on or off on the sub-LCD2. A function menu is displayed in the form of a virtual image 1802 in the display area 105. This menu offers options for the display on the sub-LCD2, "on" and "off," from which the user can select. The selection can be made using a remote control, voice input, or the like. For example, when "on" is selected, the display on the sub-LCD2 is turned on.

[0146] Furthermore, for the above-described functions, combinations recommended to the user may be defined in advance and presented in a menu, allowing the user to select. FIG. 19 shows a display example related to this function. In the example of FIG. 19, a display method setting menu is displayed in the display area 105, and the user can select from a plurality of setting combinations. Similarly, a display method setting menu is also displayed on the sub-LCD2. The user can operate and set either on the AR-HUD side or the sub-LCD2 side, whichever they prefer.

[0147] As an example of a combination of display methods, a first combination is such that the AR display is on, the sub LCD display is on, destination guidance is displayed in AR, and vehicle speed is displayed on the sub LCD. A second combination is such that the AR display is on, the sub LCD display is off, destination guidance is displayed in AR, and vehicle speed is displayed in AR. A third combination is such that the AR display is off, the sub LCD display is on, destination guidance is displayed on the sub LCD, and vehicle speed is displayed on the sub LCD. Detailed settings are also possible for each combination. In the detailed settings, the type and priority of the video information to be displayed on the display destination can be set. For example, it is possible to set the type of video information to be displayed on the AR-HUD by turning destination guidance on / off, vehicle speed on / off, and prioritizing each type.

[0148] Note that a remote control for AR-HUD input and a remote control for sub-LCD2 input may be provided as input devices and operation input means for the HUD device 1. Alternatively, only one remote control may be provided for common input to the AR-HUD and the sub-LCD2. In this case, the user may be able to select whether to use the remote control for input operations to the AR-HUD or the sub-LCD2 as appropriate. Alternatively, the HUD device 1 may automatically determine, based on the situation, which display the remote control is to be used for input to. For example, the HUD device 1 may control the remote control so that the input destination is the AR-HUD when only the AR-HUD display is on, as in the second combination, and control the remote control so that the input destination is the sub-LCD2 when only the sub-LCD2 display is on, as in the third combination. The above two types of input control may be similarly applied to the aforementioned operation interfaces, etc., as well as the remote control. The HUD device 1 may also be operated via an operation IF installed on the steering wheel or the like of a vehicle.

[0149] [Use: Brightness control] As another example of different usage, the HUD device 1 can control and set the display brightness separately for the AR-HUD side and the sub-LCD 2 side. This setting can be automatic or manual. As shown in FIG. 3 , the virtual image 106 in the display area 105 of the AR-HUD is formed corresponding to a predetermined position in front of the vehicle through the windshield 103 when viewed from the driver's viewpoint. The brightness of this virtual image 106 can be dynamically changed depending on the brightness of the display position of the virtual image 106 (for example, 10 m ahead of the vehicle). Note that brightness information of such a virtual image display position is obtained, for example, as one of the vehicle information 406. Alternatively, the HUD device 1 may acquire the brightness information of the virtual image display position using a sensor 409.

[0150] In addition, the HUD device 1 may acquire headlight ON / OFF information from the vehicle information 406, and increase the brightness of the virtual image 106 of the AR-HUD when the headlights are turned ON (lit) in a scene such as at night or inside a tunnel.

[0151] On the other hand, the sub-LCD 2 does not need to change its brightness dynamically in this manner, but the HUD device 1 may change the brightness. When changing the brightness, the HUD device 1 changes the brightness of the display screen 20 of the sub-LCD 2 in accordance with the brightness of the surroundings of the sub-LCD 2 (e.g., the interior of the vehicle) determined from the vehicle information 406 or the sensor 409, for example.

[0152] 20 shows display examples of the above-described display luminance change control. (A) shows a case where the surroundings, including the area ahead of the vehicle, are relatively bright, such as daytime. (B) shows a case where the surroundings, including the area ahead of the vehicle, are relatively dark, such as nighttime. In (A), the HUD device 1 controls the display of the virtual image 2001 in the display region 105 of the AR-HUD so that the luminance of the virtual image 2001 is relatively high (compared to, for example, nighttime). This improves the visibility of the virtual image 2001. In (B), the HUD device 1 controls the display of the virtual image 2002 in the display region 105 of the AR-HUD so that the luminance of the virtual image 2002 is relatively low (compared to, for example, daytime). This reduces glare while ensuring the visibility of the virtual image 2002.

[0153] In addition, in the example of FIG. 20, the HUD device 1 also controls the sub-LCD2 side so that the brightness on the display screen 20 is higher during the daytime (A) than at nighttime (B) (shown as whiter in the drawing).

[0154] In the above function, the HUD device 1 controls the display brightness of the display area 105 of the AR-HUD and the display screen 20 of the sub-LCD 2 so as to improve overall visibility as much as possible in relation to each other. The HUD device 1 controls the brightness of the display area 105 and the brightness of the display screen 20 so that they have a predetermined brightness relationship.

[0155] Also, in relation to the above function, it is preferable that the HUD device 1 unifies the display style between the display of the virtual image 106 of the AR-HUD and the display of the sub-LCD 2. Elements that constitute the style include, for example, color (e.g., warm colors / cool colors), font, GUI design, etc. When the driver shifts his or her gaze between the display area 105 and the display screen 20, a unified style reduces the sense of incongruity. The user may be able to select and set the style of the display of the AR-HUD and the display of the sub-LCD 2 from multiple themes.

[0156] [Distortion Correction] The HUD device 1 performs distortion correction on the display of the virtual image 106 of the AR-HUD using predetermined hardware or software, such as the distortion correction unit 413 shown in FIG. 5B. FIG. 21 shows a display example related to distortion correction. Distortion correction is a process of correcting distortion of the virtual image 106, which occurs due to the curvature of the vehicle's windshield 103 (FIG. 1), so as to reduce the distortion. Because the HUD device 1 directly projects image light onto the display area 105 of the windshield 103, such distortion correction is necessary to present a suitable virtual image 106. The windshield 103 has different curvatures, for example, in the horizontal and vertical directions. Therefore, if the image information to be displayed is projected directly onto the display area 105, distortion will occur in the image depending on the curvature. For example, if the image information to be displayed is an original image 2100 (e.g., a star image within a square area), and the original image 2100 is projected and displayed as is, distortion such as virtual image 2101 will occur.

[0157] The control device 401 of the HUD device 1 performs, for example, a correction process 2102 (correction process for canceling distortion according to the curvature) on an original image 2100 of video information to be displayed, taking into account the curvature of the windshield 103. Image light corresponding to the corrected image 2103 is projected onto the display area 105. As a result, when viewed from the driver's viewpoint, a virtual image 2104 corresponding to the corrected image 2103 can be visually recognized as being undistorted and close to the original image 2100.

[0158] On the other hand, the above-described distortion correction is not required on the sub-LCD2 side. In the first embodiment, the display screen 20 of the sub-LCD2 is flat and has no curvature. Therefore, distortion correction is not required. When the original image 2100 is displayed on the sub-LCD2 as is, an image 2105 that is close to the original image 2100 is obtained.

[0159] As a modified example, the sub-LCD 2 may be a curved display in which the display screen 20 has a predetermined curvature. In this case, the HUD device 1 may perform distortion correction on the video information displayed on the sub-LCD 2, taking into account the curvature of the display screen 20.

[0160] [Pitching Compensation] The HUD device 1 performs pitching correction on the display of the virtual image 106 of the AR-HUD using predetermined hardware or software, for example, the pitching correction unit 414 shown in FIG. 5B. FIG. 22 shows a display example related to pitching correction. Pitching correction involves correcting the display position of the virtual image 106 in the display area 105 in the vertical direction in accordance with the sway of the vehicle while traveling. When there is sway, the virtual image 106 in the display area 105 is displayed shifted relative to the position of the target vehicle ahead, as shown, for example, in the image (ring-shaped image) indicated by the dashed line. In this example, the image indicated by the dashed line shows a case where the image is shifted downward relative to the position of the target vehicle ahead. Pitching correction reduces the misalignment of the superimposition of the real image and the virtual image 106. The HUD device 1 grasps the sway of the vehicle based on the vehicle information 406 and corrects the display position of the video information to be displayed within the display area 105 in accordance with the sway.

[0161] The control device 401 of the HUD device 1 performs pitching correction processing 2202, for example, on an original image 2200 of video information to be displayed, taking into account the state of vehicle sway. The original image 2200 has a display position (e.g., point p1) in the display area 205. In the pitching correction processing 2202, the display position is corrected (e.g., to point p2), and the size, tilt, and the like may also be corrected as necessary. In this example, the corrected display position is corrected to point p2 above point p1 so that the AR image is closer to the position of the target forward vehicle, i.e., higher up. Video light corresponding to the corrected image 2203 is projected onto the display area 105. As a result, from the driver's viewpoint, a virtual image 2201 corresponding to the corrected image 2203 can be visually recognized as being superimposed and displayed in accordance with the position of the target forward vehicle.

[0162] On the other hand, the pitching correction described above is not required on the sub-LCD2 side. Because the driver and the sub-LCD2 also sway with the sway of the vehicle and because the sub-LCD2 is not AR, the driver can view the video information on the sub-LCD2 without pitching correction. In the example of Figure 22, the video information on the display screen 20 of the sub-LCD2 is displayed at a predetermined position.

[0163] [Usage: Debug mode] As another example of different uses, a display corresponding to the debug mode of the HUD device 1 is also possible. The HUD device 1 transitions from the normal mode to the debug mode in response to a predetermined operation. The debug mode is a mode in which developers, maintainers, and other personnel on the provider side of the HUD device 1 debug and maintain the HUD device 1. In the debug mode, the HUD device 1 displays information (debug information, maintenance information) useful to developers and maintainers, such as logs of CAN information received from the vehicle via CAN communication as vehicle information 406, software operation logs, and error codes, on at least one of the two types of displays. Conventional HUD devices do not have a sub-LCD 2, so even if such information is displayed, it is displayed on the AR-HUD or on a device externally connected to the HUD device.

[0164] For example, the HUD device 1 stores an error code when an error occurs. The HUD device 1 may store, for example, the error codes for the past several times in association with date and time information in the nonvolatile storage memory 421. In the debug mode, the HUD device 1 displays information including the error code on a display selected from two types of displays.

[0165] FIG. 23 shows examples of displays in the debug mode. (A) shows a case where an error code is displayed in the display area 105 of the AR-HUD during an actual driving test. (B) shows a case where an error code is displayed on the sub-LCD 2 during an actual driving test. In (A), a virtual image 2301 of the error code is displayed in the display area 105. While it is possible to display the error code as a virtual image 2301 as in (A), this virtual image 2301 can only be seen from a viewpoint within the driver's eyebox. Therefore, this display method is not very suitable for displaying fine details, especially text information. Therefore, the HUD device 1 displays video information 2302 of the error code on the display screen 20 of the sub-LCD 2 as in (B). This allows developers and maintenance personnel to efficiently perform tasks such as debugging by viewing the fine video information 2302 of the error code on the sub-LCD 2 even if there is a problem with the AR-HUD display, thereby improving the efficiency of development and maintenance.

[0166] The display in debug mode in FIG. 23 is controlled in a similar manner to the control example of the error state notification in FIG. 15 described above.

[0167] [Double display] The HUD device 1 of the first embodiment also has a function of displaying the same information in dual mode on two types of displays. This function can be used, for example, as a demo mode for developers, and can also be used during normal use by general users. The HUD device 1 transitions to dual display mode upon a predetermined trigger, such as a mode specification input by the user. Alternatively, the HUD device 1 automatically performs dual display depending on the type of video information to be displayed, etc.

[0168] Fig. 24 shows a display example of dual display. When the HUD device 1 is in dual display mode, for example, it also displays information similar to the AR virtual image 106 displayed in the display area 105 of the AR-HUD on the display screen 20 of the sub-LCD2. In the example of Fig. 24, route guidance virtual images 2401 and 2402 are displayed in the display area 105. The virtual image 2401 is a triangle image indicating a right turn, and the virtual image 2402 is a text image indicating "turn right in 100m." Meanwhile, the display screen 20 of the sub-LCD2 displays video information 2403 corresponding to the virtual image 2401 and video information 2404 corresponding to the virtual image 2402.

[0169] Note that the size, brightness, and other characteristics of the display area 105 are fundamentally different from the size, brightness, and other characteristics of the display screen 20, and an image cannot be displayed on the display screen 20 in exactly the same manner as the virtual image 106. Therefore, the HUD device 1 creates the AR virtual image 106 based on the original image, and also creates a roughly identical or similar image to be displayed on the sub-LCD 2. The control device 401 displays two roughly identical images, taking into account the difference in characteristics between the two types of displays.

[0170] The control device 401 generates video information 2411 for the display area 105 and video information 2412 for the sub-LCD2, for example, using the first video data generation unit 501 and the second video data generation unit 502 in FIG. 6B based on the original image 2400. The video information 2411 is data having a display position (for example, point q1) and size within the display area 105. The video information 2412 is data having a display position (for example, point q2) and size within the display screen 20. The control device 401 displays each piece of video information on each display destination in accordance with the timing based on the data.

[0171] The virtual image 2401 etc. on the AR-HUD side has directionality, so it can only be seen from the viewpoint of the driver in a driving position in the driver's seat, and cannot be seen by passengers etc. On the other hand, the video information 2403 etc. on the sub-LCD2 side can be seen from the viewpoint of people other than the driver, such as passengers.

[0172] For example, such dual display can be used during development or in demo mode for maintenance. This allows a person other than the driver to see an image on the sub-LCD2 that is roughly the same as the virtual image 106 displayed on the AR-HUD. For example, a passenger such as a developer can easily see what kind of virtual image 106 is currently being displayed on the AR-HUD by looking at the image on the sub-LCD2. This can improve development efficiency.

[0173] This dual display function can also be used in various ways by general users in their normal use. The following description will be given using the example of FIG. 24 . For example, assume that a driver specifies a destination, and the HUD device 1 displays AR route guidance for the route to the destination in the display area 105. The HUD device 1 displays, for example, virtual images 2401 and 2402 in the display area 105 of the AR-HUD, depending on the relationship between the current position of the vehicle and the destination, etc. The driver can see the virtual images. On the other hand, passengers cannot see the virtual images, but can recognize the content of the route guidance and the like that the HUD device 1 is currently presenting to the driver by looking at the image information 2403 and 2404 on the sub-LCD 2.

[0174] From the passenger's perspective, there may be cases where the route guidance being presented is incorrect or inefficient, or the passenger wishes to change their destination. In these cases, the passenger can immediately notify the driver. This allows for quick route changes starting from the passenger, improving convenience.

[0175] When displaying an image similar to the virtual image 106 of the AR-HUD on the sub-LCD 2, the HUD device 1 may perform image processing such as processing based on the original image to create and display an image (e.g., image information 2412) that matches the characteristics of the display screen 20, such as the size. Examples of processing include enlarging / reducing the image, changing the aspect ratio, and cropping. Even if the images displayed dually on the two types of displays are not the same, this is effective as long as the meaning of the image is conveyed to people. In addition, a text image (e.g., "Route guidance is currently being displayed on the AR-HUD") that represents the meaning of the virtual image 106 may be displayed on the sub-LCD 2.

[0176] 25 shows a processing flow of the HUD device 1 for controlling selection of the display destination of video information from dual display on the AR-HUD only, the sub-LCD only, or both, in relation to the dual display function. In step S401, the control device 401 of the HUD device 1 determines the occurrence of an event based on the vehicle information 406, and obtains video information to be displayed according to the event. In step S402, the HUD device 1 determines the event, scene, type of video information, etc., and decides the display destination of the video information, etc. Step S403 is a branch depending on the display destination; if only the AR-HUD is selected, the process proceeds to step S404; if only the sub-LCD2 is selected, the process proceeds to step S405; and if both the AR-HUD and the sub-LCD2 are selected (dual display), the process proceeds to step S406.

[0177] In step S404, the HUD device 1 displays the virtual image 106 in the display region 105 of the AR-HUD based on the image information generated by the first image data generation unit 501. In step S405, the HUD device 1 displays the image information on the sub-LCD 2 based on the image information generated by the second image data generation unit 502. In step S406, the HUD device 1 displays the virtual image 106 in the display region 105 of the AR-HUD based on the image information generated by the first image data generation unit 501, and also displays the image information on the sub-LCD 2 based on the image information generated by the second image data generation unit 502.

[0178] In step S407, the HUD device 1 determines whether a predetermined display termination condition corresponding to the virtual image 106 (image information) displayed in the display area 105 of the AR-HUD is met. For example, in the case of the route guidance in Fig. 24, this condition is whether the current position of the vehicle has reached the destination. If the condition is met, in step S408, the HUD device 1 terminates the display of the virtual image 106 in the display area 105 of the AR-HUD.

[0179] In step S409, the HUD device 1 determines whether a predetermined display termination condition corresponding to the video information displayed on the sub-LCD 2 is met. This condition is, for example, the same as the condition on the virtual image side in step S407. If the condition is met, the HUD device 1 ends the display of the video information on the sub-LCD 2 in step S410.

[0180] In step S411, the HUD device 1 determines whether a predetermined display termination condition is met for both the virtual image 106 of the AR-HUD and the video information on the sub-LCD 2. This condition is, for example, the same as the condition in step S407. If the condition is met, the HUD device 1 terminates the display of the video information on both the AR-HUD and the sub-LCD 2 in step S412.

[0181] In the above control example, with regard to dual display, the display start and end timings of the virtual image 106 of the AR-HUD and the image on the sub-LCD 2 are the same, and the display end conditions are the same. This is not limiting, and the following modifications are also possible. That is, the display start and end timings of the virtual image 106 of the AR-HUD and the image on the sub-LCD 2 may be made different depending on the type of video information to be displayed, the scene, etc. For example, the HUD device 1 may start displaying video information with corresponding content on two different display destinations at the same timing, and end the display of each piece of video information at different timings under different display end conditions for each display destination. In another example, the HUD device 1 may start displaying video information with corresponding content on two different display destinations under different display start conditions, and end the display of each piece of video information at the same timing. It is sufficient that the video information with corresponding content on the two display destinations be in a dual display state for at least a certain period of time.

[0182] [Change display destination (1)] The HUD device 1 of the first embodiment also has a function of controlling the transition, change, movement, etc. of the display destination of video information from one to the other using two types of displays. Such display is also referred to as display destination change control or moving display. The HUD device 1 controls such moving display depending on the type of video information, the scene, etc.

[0183] 26 shows a display example in which the display destination of certain video information is changed or moved from the AR-HUD to the sub-LCD 2. (A) shows a state in which, at a first point in time, the video information for notification, for example, a right turn guide, is displayed as a virtual image 106 in the display area 105 of the AR-HUD. (B) shows a state in which, at a second point in time, the video information for notification, for example, a right turn guide, is displayed on the sub-LCD 2.

[0184] For example, suppose an event occurs that requires the driver to be notified. As an example, the event is that the vehicle is approaching a destination, and the notification is a notification that the vehicle is approaching the destination. Suppose the video information of the notification is expressed in a non-AR format (for example, a text image). Even in this case, the HUD device 1 first displays the video information of the notification as an AR virtual image 106 in the display area 105 of the AR-HUD, which is easily within the field of view of the driver and can be viewed without moving the driver's viewpoint.

[0185] The HUD device 1 first displays the video information on the display panel 4 at a first point in time. As a result, an AR virtual image 106 is displayed in the display area 105 of the AR-HUD. Thereafter, when a certain time has passed, or when a predetermined condition is met, such as when it is confirmed based on a sensor or the like (for example, the camera 109 in FIG. 3 ) that the driver has visually recognized the AR virtual image 106, the HUD device 1 displays video information of the same content as the virtual image 106 on the sub-LCD 2 at a second point in time. At this time, the HUD device 1 erases the video information on the display panel 4, thereby erasing the AR virtual image 106. As a result, for example, the driver is given a visual effect in which the display destination of the notification video information changes or moves from the display area 105 of the AR-HUD to the sub-LCD 2, as in the example of FIG. 26 .

[0186] A detailed example will be described with reference to Fig. 26. In Fig. 26A, image information (more specifically, a combination of a text image and an arrow image) for guiding a right turn, such as "Turn right in 300m," is displayed as a virtual image 2601 in the display area 105 of the AR-HUD. This virtual image 2601 is generated when the current position of the vehicle approaches within a predetermined first distance from the destination (e.g., a right turn point). At this moment, the HUD device 1 first selects the AR-HUD as the display destination (first display destination) for this image information, and displays the image information on the display panel 4. As a result, the virtual image 2601 is displayed in the display area 105 of the AR-HUD for a first period from a first time point.

[0187] In this control example, the reason why the first display destination is the AR-HUD is that an image displayed on the AR-HUD is more likely to be in the driver's field of view, making it easier for the driver to notice the image, and reducing the driver's line of sight. By viewing the virtual image 2601 that appears in the display area 105, the driver can recognize that there is a "right turn 300 meters ahead."

[0188] Next, in (B), at a second time point after a time has elapsed since (A), the display of the virtual image 2601 in the display area 105 of the AR-HUD is erased, and video information 2603 having content corresponding to the virtual image 2601 is displayed on the display screen 20 of the sub-LCD 2. The HUD device 1 terminates the display of the AR-HUD virtual image 2601 as shown in an area 2602 and erases it from the display area 105 at a predetermined trigger. The predetermined trigger may be the passage of a predetermined time or traveling a predetermined distance. For example, a condition may be when the current position of the vehicle approaches within a predetermined second distance (shorter than the first distance) from the destination (the point where the vehicle will turn right).

[0189] Along with the erasure, the HUD device 1 displays video information 2603 (more specifically, a combination of a character image and an arrow image) having content corresponding to the virtual image 2601 on the display screen 20 of the sub-LCD 2. In other words, the HUD device 1 changes the display destination of the video information of the right turn guidance from the first display destination to the second display destination. In other words, the HUD device 1 moves the video of the right turn guidance from the first display destination to the second display destination.

[0190] At the second time point (B), the vehicle is closer to the destination than at the first time point (A), so the importance of the right-turn guidance video information to the driver is reduced. In addition, when making a right or left turn, there are many things to pay attention to in the vicinity, so maintaining a clear field of vision allows for safer driving. If an AR display or the like is present in the display area 105 at this time, the field of vision may be obstructed, making it difficult for the driver to see what they want or need to see. Therefore, in this example, it is effective to change the display destination of this video information to the sub-LCD2 and move the video information to the sub-LCD2, thereby reducing the number of images displayed in the display area 105 of the AR-HUD and reducing the amount of displayed information.

[0191] During a second period from the second time point, the right turn guidance continues on the sub-LCD2 using the image information 2603. The driver can appropriately shift his / her gaze to the sub-LCD2 to check the image information 2601. The HUD device 1 ends the display of the image information 2603 on the sub-LCD2 at a predetermined trigger. The predetermined trigger may be, for example, the passage of a predetermined time from the second time point or the driving of a predetermined distance. For example, the condition may be when the current position of the vehicle reaches the destination.

[0192] Note that the video information for right turn guidance in the example of FIG. 26 (virtual image 2601 and video information 2603) is information determined according to the relationship between the distance between the current position of the vehicle and the destination. This video information may be displayed as a static image, or may be video that changes over time. For example, this video information may change in sequence according to the distance, such as "300 m ahead," "200 m ahead," and "100 m ahead." In the example of FIG. 26, for the virtual image 2601, the text in the video information 2603 is updated to "100 m ahead" according to the traveled distance.

[0193] In addition, in the above-mentioned moving display control, as with the dual display control described above, the AR-HUD and the sub-LCD 2 basically have different characteristics, so it is sufficient to display video information of roughly the same content or related content on both.

[0194] As a modified example, the HUD device 1 may control the moving display of AR or the like to return to its original state after completing a right or left turn, etc. For example, the HUD device 1 displays certain video information as AR in the display area 105 at a first time point before turning right, moves the video information to the sub-LCD 2 and displays it thereon at a second time point during the right turn, and then displays the video information as AR again in the display area 105 at a third time point after the right turn. In other words, the HUD device 1 may control the video information to temporarily change from the usual first display destination to the second display destination only when and if a predetermined condition (e.g., turning right) is satisfied.

[0195] As in the above example, triggers and conditions for when the video information displayed in the display area 105 of the AR-HUD as the first display destination is moved to the sub-LCD2 as the second display destination, or for how long the video information displayed on the second display destination is displayed, can be controlled according to the type of video information, the situation, and the like. For example, the triggers and conditions may be set to occur after a certain time has passed since the occurrence of an event. Furthermore, for example, in response to an event in which a seatbelt is detected as not fastened, a warning may be displayed on the AR-HUD as the first display destination and the sub-LCD2 as the second display destination, and then, in response to detection of fastening of the seatbelt, the warning display may be terminated. In another example, the display of video information may basically continue until the AR-HUD or the sub-LCD2 is turned off. Alternatively, when new video information is generated on the AR-HUD or the sub-LCD2, the older video information may be terminated. Alternatively, the AR-HUD or the sub-LCD2 may continuously display multiple pieces of video information within a range that satisfies the amount of information displayed on each.

[0196] Furthermore, when controlling the moving display, there may be a period when the corresponding video information is temporarily displayed on both of the two types of displays, or a period when it is temporarily not displayed on either of the two displays. In either case, the visual effect of changing the display destination of certain video information can be achieved.

[0197] As a variation of the above moving display, it is also possible to set the first display destination to the sub-LCD 2 and the second display destination to the AR-HUD depending on the type of video information, etc.

[0198] FIG. 27 shows a display example in which the display destination of certain video information is changed and moved from the sub-LCD2 to the AR-HUD. In (A), at a first time point, video information 2701 of "turn right in 500 m" is displayed on the sub-LCD2, which is the first display destination. In (B), at a second time point, a virtual image 2702 of "turn right in 300 m" is displayed in the display area 105 of the AR-HUD, which is the second display destination. The display content at the second time point is the same as the example in (A) of FIG. 26. The concept of the control example in FIG. 27 is that at the first time point, the destination is far away and the importance of right-turn guidance is still low, so the video information is displayed on the sub-LCD2. At the second time point, the importance of right-turn guidance has increased, so the video information is displayed on the AR-HUD.

[0199] [Change display destination (2)] FIG. 28 shows a display example in a modified example of the display destination change control in FIGS. 26 and 27. The HUD device 1 initially displays video information of a certain display target (a star-shaped image in this example) as a virtual image 2801 in the display area 105 of the AR-HUD as a first display target at a first time point (A). The virtual image 2801 is an AR having a predetermined function, but is illustrated in an abstract form in this example. The right turn guidance in FIG. 27 described above can also be applied as an example of the target video information.

[0200] At a predetermined trigger, the HUD device 1 changes the display destination of this video information (virtual image 2801) to the sub-LCD2 as a second display destination. At that time, the HUD device 1 continuously moves the video information (virtual image 2801) toward the sub-LCD2 (downward) on the display, as shown at a second point in time in (B). That is, the display position of the virtual image 2801 moves downward on the time axis within the display area 105. (B) shows an example of a state in which the display position of the video information (virtual image 2801) reaches the bottom edge of the display area 105 and is in the process of going beyond the bottom edge and framing out. At the same time, the HUD device 1 continuously moves this video information within the display screen 20 of the sub-LCD2. In (B), this video information is framing in from the top edge of the rectangle on the display screen 20. Naturally, the video information in the portion 2802 displayed on the display screen 20 is a non-AR image.

[0201] Then, (C) shows a state at a third time point after this video information has moved to a predetermined position on display screen 20 of sub-LCD 2. This video information is displayed as video information 2803 on display screen 20 of sub-LCD 2.

[0202] As described above, the HUD device 1 controls the display so that the display destination of the video information is continuously moved from the first display position of the first display destination to the second display position of the second display destination. During this movement, the appearance of the video information changes from the virtual image 106 to an image that is not the virtual image 106. In this control example, the target video information continuously moves from the driver's viewpoint, making it easy to recognize the change in the display destination of the video information. It is also possible to continuously move the display destination of the video information from the sub-LCD 2 to the display area 105.

[0203] FIG. 29 shows a processing flow of the HUD device 1 when controlling the moving display of FIG. 26 and the like. In step S501, the control device 401 of the HUD device 1 obtains event information and target video information based on the vehicle information 406 and the like. In step S502, the HUD device 1 determines the event, scene, type of video information, and the like, and determines the first display destination and second display destination of the video information when performing moving display. In step S503, the HUD device 1 uses the first video data generation unit 501 (FIG. 6A) to create first video information to be displayed on the first display destination, and displays the first video information (e.g., virtual image 106) on the first display destination (e.g., AR-HUD). In step S504, the HUD device 1 determines the conditions for moving the first video information of the first display destination to the second display destination. If the condition is satisfied, in step S505, the HUD device 1 uses the second video data generating unit 502 (FIG. 6A) to create second video information to be displayed on the second display destination, displays the second video information on the second display destination (e.g., sub LCD 2), and erases the first video information on the first display destination. Note that, when performing the continuous moving display of FIG. 28, the processing content of step S505 is display control of continuous moving from the first display destination to the second display destination.

[0204] The present invention has been specifically described above based on the embodiments, but the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the spirit of the invention. In each embodiment, components can be added, deleted, or replaced, except for essential components. Unless otherwise specified, each component may be singular or plural. A combination of each embodiment is also possible.

[0205] The technology according to this embodiment allows the driver to conveniently view images of information necessary for driving, such as alert information when an oncoming vehicle or pedestrian is detected, in addition to navigation information such as destination and speed projected onto the windshield, etc., and provides an information display device (head-up display device) that is user-friendly for users such as drivers and reduces the driver's eye movement, thereby contributing to the prevention of traffic accidents. This contributes to the achievement of "Good health and well-being for all," one of the Sustainable Development Goals (SDGs) advocated by the United Nations. [Explanation of symbols]

[0206] 1...HUD device, 2...sub LCD, 20...display screen, 101...dashboard, 102...casing, 103...windshield, 104...steering wheel, 105...display area, 106...virtual image

Claims

1. A vehicle that displays a virtual image in a display area based on projection of image light, a housing that houses a light source device for generating the image light, a main display device, and an optical system; a sub-display device provided outside the housing; a control device that controls the display of video information on the main display device and the display of video information on the sub-display device; Equipped with the control device displays, on the sub-display device, video information for user setting of a display format including on / off of a display in the display area based on the main display device, and displays, on the main display device, video information for user setting of a display format including on / off of a display of the sub-display device; The type of video information to be displayed on the main display device is controlled by operating the sub-display device. vehicle.

2. 2. The vehicle of claim 1, the control device determines at least one of the main display device and the sub-display device as a display destination for the video information to be displayed, based on at least one of the surrounding conditions of the vehicle, the type of the video information, the data amount, the information amount, the display size, and the priority. vehicle.

3. 2. The vehicle of claim 1, the control device determines at least one of the main display device and the sub-display device as a display destination for the video information to be displayed, based on an amount of information to be displayed on the main display device and an amount of information to be displayed on the sub-display device. vehicle.

4. 2. The vehicle of claim 1, When adjusting the position of the display area in the initial setting, the control device displays image information for adjustment on the main display device and image information for adjustment support on the sub-display device. vehicle.

5. 5. The vehicle according to claim 4, the control device displays an AR image as the adjustment image information on the main display device, and displays image information for adjusting at least one of color, size, shape, inclination, and position of the AR image as adjustment support image information on the sub-display device; vehicle.

6. 2. The vehicle of claim 1, When the display of the display area based on the main display device is in an error state, the control device displays, on the sub-display device, video information indicating that the display area is in an error state, and when the display of the sub-display device is in an error state, the control device displays, on the main display device, video information indicating that the sub-display device is in an error state. vehicle.

7. 2. The vehicle of claim 1, the control device determines at least one of the main display device and the sub-display device as a display destination for the video information to be displayed, depending on a scene including a situation around the vehicle. vehicle.

8. 2. The vehicle of claim 1, the control device adjusts the luminance of the display on the main display device and the luminance of the display on the sub-display device according to a scene including a situation around the vehicle and a situation inside the vehicle. vehicle.

9. 2. The vehicle of claim 1, the control device determines, for the video information to be displayed, the display destination as dual display on both the main display device and the sub-display device based on the situation around the vehicle or the type of the video information; vehicle.

10. 2. The vehicle of claim 1, the control device determines one of the main display device and the sub-display device as a first display destination for the video information to be displayed, based on the situation around the vehicle or the type of the video information, and determines the other of the main display device and the sub-display device as a second display destination, and causes the first video information to be displayed on the first display destination first on the time axis, and then causes the second video information having content corresponding to the first video information to be displayed on the second display destination; vehicle.

11. 11. The vehicle of claim 10, the control device displays the video information to be displayed at a first display position of the first display destination, and displays the video information to be displayed so as to move continuously from the first display position to a second display position of the second display destination, vehicle.

12. 2. The vehicle of claim 1, The housing has a first housing portion accommodated in a dashboard of the vehicle and a second housing portion extending outside the dashboard, The sub-display device is provided in the second housing portion. vehicle.

13. 13. The vehicle of claim 12, the second housing portion has an opening through which the image light is emitted, the housing has a mechanism for changing the position of the sub-display device so that the sub-display device covers at least a part of the opening of the second housing portion when the vehicle is not in use; vehicle.

14. 2. The vehicle of claim 1, Equipped with a remote control as an input device, the remote controller can be used in common for input operations of the main display device and the sub-display device, When the display of the main display device is on, the remote control is controlled so that an input destination of the remote control is the main display device, and when the display of the sub-display device is on, the remote control is controlled so that an input destination of the remote control is the sub-display device; The type of video information to be displayed on the main display device is controlled by operating the sub-display device using the remote control. vehicle.

Images