Display device and virtual image display device

Abstract

To provide a more preferably display technique for a display device, which can be corresponded to the case where a video image is generated in an external part, and the video image is received and displayed as a video image input by the display device, and contributes to "3: Good Health and Well-Being" in a sustainable development goals (SDGs) in accordance with the present invention.SOLUTION: A virtual image display device 1 comprises: a video image processing part that processes video image information input from an external part; and a video image display device that displays a video image on the basis of a video image data processed by the video image processing part, and outputs a video image light corresponded to the video image, and adjusts an information transmission path for transmitting video information of the video image from the external part in accordance with a display of the video image. For instance, the virtual image display device 1 and a controller 100 switch the path to a second transmission path 202 in the case where a first transmission path 201 is in an abnormal or in a failure.SELECTED DRAWING: Figure 14

Description

[Technical field]

[0001] The present disclosure relates to a display device and a virtual image display device technology. [Background technology]

[0002] In the past, images were often generated and displayed inside the virtual image display device, but in the future, it is expected that images will be generated outside the virtual image display device and the virtual image display device will receive and display the generated images as image input. An example of an external image generator is a controller for an in-vehicle system.

[0003] An example of the prior art is JP 2010-70066 A (Patent Document 1). Patent Document 1 describes an example of a HUD (Head-up display: HUD) having a display information generating means for generating projection information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2010-70066 A Summary of the Invention [Problem to be solved by the invention]

[0005] Depending on the image provided from outside, there is a problem that it is difficult for the user to see depending on the display device. Examples of images displayed by mounting a display device on a vehicle include delays in the image provided from outside, images that completely cover the entire vehicle, images that block most of the field of vision, images that are too dazzling, images that are in sync with or buried in the brightness of the surroundings making it difficult to see the necessary information, images that contain information that is different from the actual situation or incorrect information (for example, incorrect vehicle speed), etc., which hinder safe driving.

[0006] An object of the present disclosure is to provide a display device technology that is more suitable for cases in which an image is generated externally and the display device receives and displays the image as a video input. [Means for solving the problem]

[0007] In order to solve the above problem, for example, the configuration described in the claims is adopted. The present application includes a plurality of means for solving the above problem, and an example thereof is as follows. The virtual image display device includes an image processing unit that processes image information input from the outside, and an image display device that displays an image based on the image data processed by the image processing unit and outputs image light corresponding to the image, and adjusts an information transmission path for transmitting image information from the outside in accordance with the display of the image. Effect of the Invention

[0008] According to a representative embodiment of the present disclosure, it is possible to provide a more suitable display device technology that can handle a case where an image is generated externally and the display device receives and displays the image as an image input. Problems, configurations, effects, and the like other than those described above are described in the description of the embodiment of the invention. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a first example of a configuration in which the display device according to the first embodiment is mounted on a vehicle. [Diagram 2] FIG. 4 is a diagram showing a second example of a configuration in which the display device according to the first embodiment is mounted on a vehicle. [Diagram 3] FIG. 1 illustrates an example of the configuration of an in-vehicle system including the display device of the first embodiment. [Figure 4] 2 is a diagram showing a configuration example of an information transmission path between the display device of the first embodiment and a vehicle controller. [Diagram 5] FIG. 2 is a diagram showing an example of a functional block configuration of a controller of a vehicle in the first embodiment. [Figure 6] FIG. 2 is a diagram showing a first example of a functional block configuration of the display device in the first embodiment. [Figure 7] FIG. 4 is a diagram showing a second example of a functional block configuration of the display device in the first embodiment. [Figure 8] FIG. 13 is a diagram showing a third example of a functional block configuration of the display device in the first embodiment. [Figure 9] FIG. 2 is a diagram showing an overview of main functions in the first and second embodiments. [Figure 10] FIG. 11 is a diagram showing a case where one information transmission path is normal in the first embodiment. [Figure 11A] FIG. 2 is a diagram showing an example in which nothing is displayed in a display area of ​​the display device in the first embodiment. [Figure 11B] FIG. 2 is a diagram showing an example of normal display in a display area of ​​the display device in the first embodiment. [Figure 12] FIG. 11 is a diagram showing a case where one information transmission path is abnormal in the first embodiment. [Figure 13] FIG. 11 is a diagram showing a case where one information transmission path is malfunctioning in the first embodiment. [Figure 14] FIG. 2 is a diagram showing switching between two information transmission paths in the first embodiment. [Figure 15] FIG. 2 is a diagram showing the combined use of two information transmission paths in the first embodiment. [Figure 16] FIG. 11 is a diagram showing information restrictions in the case where one information transmission path is malfunctioning in the first embodiment. [Figure 17] FIG. 11 is a diagram showing a display example of a display area when information is restricted in the first embodiment. [Figure 18A] FIG. 11 is a diagram showing a display example of a control state in a display area when information is restricted in the first embodiment. [Figure 18B] FIG. 11 is a diagram showing a display example of a control state in a display area when display is stopped in the first embodiment. [Figure 18C] FIG. 11 is a diagram showing an example of a display of a control state in a display area when a transmission path is switched in the first embodiment. [Figure 18D] FIG. 11 is a diagram showing an example of displaying a control state in a display area when transmission paths are used in combination in the first embodiment. [Figure 18E] FIG. 11 is a diagram showing an example of a display of a control state in a display area when a transmission path is switched in the first embodiment. [Figure 18F]FIG. 4 is a diagram showing an example of displaying the state of a transmission path in a display area in the first embodiment. [Figure 18G] FIG. 4 is a diagram showing an example of displaying the state of a transmission path in a display area in the first embodiment. [Figure 18H] FIG. 4 is a diagram showing an example of displaying the state of a transmission path in a display area in the first embodiment. [Figure 18I] FIG. 4 is a diagram showing an example of displaying the state of a transmission path in a display area in the first embodiment. [Figure 19] FIG. 11 is a diagram showing another example of switching between two information transmission paths in the first embodiment. [Figure 20] FIG. 4 is a diagram showing detection of the state of a transmission path between a controller and a device in the first embodiment. [Figure 21] FIG. 4 is a diagram showing a first example of a processing flow of the in-vehicle system in the first embodiment. [Figure 22] FIG. 11 is a diagram showing a second example of the processing flow of the in-vehicle system in the first embodiment. [Figure 23] FIG. 11 is a diagram showing a third example of a processing flow of the in-vehicle system in the first embodiment. [Figure 24] FIG. 4 is a diagram showing an application example of a combination of multiple controls in the first embodiment. [Diagram 25] FIG. 11 is a diagram showing a case where a display is appropriate in checking the appropriateness of the display in the second embodiment. [Figure 26] FIG. 11 is a diagram showing a case of an inappropriate display in checking the appropriateness of the display in the second embodiment. [Figure 27] 13A and 13B are diagrams showing examples of inappropriate displays in a display area of ​​a display device in the second embodiment. [Figure 28] 13 is a diagram showing an example of display in the display area when display prohibition / restriction control is performed based on confirmation of the appropriateness of display in the second embodiment. FIG. [Figure 29] FIG. 11 is a diagram showing a processing flow of the in-vehicle system for checking the appropriateness of the display in the second embodiment. [Diagram 30] FIG. 11 is a diagram showing details of control relating to confirmation of the appropriateness of display in the second embodiment. [Diagram 31] FIG. 11 is an explanatory diagram relating to a display ratio in a display area in the second embodiment. [Diagram 32]FIG. 11 is another explanatory diagram relating to the display ratio in the display area in the second embodiment. [Diagram 33] FIG. 11 is an explanatory diagram relating to a luminance abnormality in a display region in the second embodiment. [Diagram 34] FIG. 11 is an explanatory diagram relating to incomplete information in a display area in the second embodiment. [Figure 35A] 13 is a diagram showing an example of a display control state in a display area when control is performed to disable or restrict display based on confirmation of the appropriateness of display in the second embodiment. FIG. [Figure 35B] FIG. 11 is a diagram showing another example of the display of the control state in the display area when control is performed to disable / restrict display based on confirmation of the appropriateness of display in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. In the drawings, the same parts are generally designated by the same reference numerals, and repeated explanations will be omitted. In the drawings, the representation of components may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention.

[0011] For the sake of explanation, when describing processing by a program, the program, functions, processing units, etc. may be described as the main components, but the main hardware components of these are the processor, or a controller, device, computer, system, etc. that is configured with the processor. The computer executes processing according to the program read into the memory by the processor, appropriately using resources such as memory and communication interfaces. This realizes predetermined functions, processing units, etc. The processor is configured with semiconductor devices such as a CPU / MPU or GPU, for example. Processing is not limited to software program processing, and can also be implemented by dedicated circuits. Dedicated circuits such as FPGA, ASIC, CPLD, etc. can be used.

[0012] The program may be pre-installed as data in the target computer, or may be distributed as data from a program source to the target computer. The program source may be a program distribution server on a communication network, or a non-transient computer-readable storage medium, such as a memory card or a disk. The program may be composed of multiple modules. The computer system may be composed of multiple devices. The computer system may be composed of a client-server system, a cloud computing system, an IoT system, etc. Various data and information are composed of structures such as, for example, tables and lists, but are not limited to these. Expressions such as identification information, identifiers, IDs, names, and numbers are mutually replaceable.

[0013] [Solution methods, etc.] To address the above-mentioned problems, the embodiments have the following measures and solutions. Fig. 9 is a table summarizing the solutions, and shows an overview of the main functions in the first and second embodiments.

[0014] Solution 1: Display control of a display device according to the state of an information transmission path of an external video input.

[0015] Solution 1-1. Switching information transmission paths: The information transmission path for external video input between the outside and the display device, in other words the communication path, is configured to have two or more paths for redundancy. In addition, the transmission path used for transmitting video data of the external video input is controlled to be switched from the first transmission path under normal circumstances to the second transmission path when the first transmission path is abnormal or malfunctioning. For example, when the first transmission path under normal circumstances is malfunctioning and there is a large delay, the system including the display device switches to the second transmission path to transmit the video data. A user viewing the display device's display does not mind the switching of the transmission path as long as the same display is obtained.

[0016] Solution 1-2. Use of multiple transmission paths: In a redundant configuration of information transmission paths, for example, when a first transmission path that is normally used is malfunctioning, the second transmission path is controlled as a main path and the first transmission path is controlled as a sub-path, so that video data is transmitted using both of the two transmission paths.

[0017] Solution 1-3. Display restriction on display devices by reducing the amount of transmitted data: Display restriction control of the display device, which reduces and limits the amount of display on the display device by reducing and limiting the amount of data transmitted on the information transmission path. This reduces the load related to the transmission of external video input, ensures real-time display on the display device, and reduces delays.

[0018] Solution 1-4. Display stop of display device due to transmission stop: This is a control that stops / pauses the display on the display device by stopping the transmission of external video output-input when lags or discrepancies occur due to delays in the transmission of video data from external video input in the information transmission path. As a last resort, the video output itself is stopped in this way rather than allowing display lags to occur.

[0019] However, if stopping / pausing the display results in a state where the information necessary for driving cannot be provided to the user, this increases the risk, and separate considerations and measures are necessary. For example, if there is no means of providing information other than the display device, such as a meter cluster that provides information such as vehicle speed, in other words, the instruments on the instrument panel, stopping the display device's display will result in a state where information such as vehicle speed cannot be provided.

[0020] In the embodiment, an automobile has a meter cluster on an instrument panel, and information such as vehicle speed is displayed on the meter cluster. That is, there is also information providing means other than a display device. When vehicle speed information is displayed in a display area of ​​a display device, if there is a transmission delay, the display of the vehicle speed information on the display device may be stopped / paused. In this case, the vehicle speed information is displayed on the meter cluster at a minimum, so safe driving is not hindered. Depending on the vehicle, there may be no meter cluster on an instrument panel, and a display interface for vehicle speed information, etc. may be integrated into the display device. In this case, if the display of vehicle speed information on the display device is stopped / paused, the driver will not be able to be provided with information necessary for driving, which will hinder safe driving. Therefore, in this case, it is desirable to apply other solutions rather than applying solutions 1-4.

[0021] Solution 2: Display control of a display device according to the contents of video information of an external video input. In other words, display control of a display device according to the appropriateness or validity of a display image of the display device when a display of the display device is assumed based on video information of an external video input.

[0022] Solution 2-1. Check the appropriateness of the video display content: When displaying on a display device based on external video input, the appropriateness and validity of the video display content is confirmed and checked, and the video is output only if there is no problem, and not output if there is a problem. A system including a display device checks whether the video display content that the display device is about to display is video that may lead to interference with safe driving, video that may reduce driving safety, etc. Such video may be described as "inappropriate video," etc. In other words, the system checks whether the video content that the display device is about to display is appropriate video that can ensure driving safety.

[0023] Examples of such inappropriate images include images that completely cover the entire screen, images that block most of the field of vision, images that are too dazzling, images that blend in with or are lost in the brightness of the surroundings making it difficult to see necessary information, images that contain information that differs from the actual situation or incorrect information, etc.

[0024] Even if there is no problem with the transmission and communication of video data, a system including a display device judges the appropriateness of the video to be displayed on the display device as described above. The system judges whether the video to be displayed corresponds to an example of an inappropriate video as described above based on predetermined rules and conditions. For example, the display area of ​​the display device, which is the range in which the display device can display a virtual image, is set to 100 in its entirety. If the ratio of the total area occupied by the displayed video to the entire area is equal to or exceeds a threshold value, the system judges that the video interferes with safe driving and is inappropriate.

[0025] For example, the system determines whether the image to be displayed corresponds to the above-mentioned full-screen image, image that blocks most of the field of view, etc., based on the area of ​​the area occupied by the image to be displayed, brightness, etc. The system also compares the vehicle surroundings situation, for example, a brightness reference value, with the brightness of the image to be displayed, etc., to determine whether the brightness of the image to be displayed is abnormal.

[0026] Also, for example, if the system has an information source separate from the external device that provides the external video, such as an in-vehicle controller, the system may compare the information of the external video input with information from the other source to determine whether the information of the external video input contains an error, differs from the actual situation, or contains a discrepancy.

[0027] The embodiment of the present invention will be described using an example in which a HUD device, which is a virtual image display device, is mounted on a vehicle. The system including the HUD device of the embodiment has a function corresponding to the above-mentioned solution, thereby realizing that the HUD display based on the external image input does not display anything that hinders safe driving.

[0028] <Embodiment 1> The HUD device and the like according to the first embodiment of the present disclosure will be described with reference to Fig. 1 to Fig. 24. The first embodiment has a function corresponding to the first solution.

[0029] [HUD device] Fig. 1 shows a configuration example of a HUD device 1 which is a virtual image display device according to the first embodiment. In the configuration example of Fig. 1, the HUD device 1 is stored in a dashboard 2A in a vehicle 2. In other words, the HUD device 1 is mounted so as to be embedded in the dashboard 2A. The dashboard 2A also includes an instrument panel 2C including a meter cluster, for example, in front of a steering wheel 2B. The vehicle 2 is typically an automobile, but is not limited thereto, and may be a railroad car, an airplane, or the like.

[0030] Image light 1a emitted from the HUD device 1 passes through an opening 7 in the dashboard 2A and is irradiated onto the inner surface of the windshield 3 and reflected. The reflected image light 1b is incident on the driver's viewpoint 6. When viewed from the driver's viewpoint 6, a virtual image 9 corresponding to the image light 1b is formed and visible beyond the HUD display area 5 corresponding to the irradiation area of ​​the image light 1a. In the present invention, the projection of the image light onto the windshield 3 will be described, but the image light may also be projected onto a projection member such as a combiner.

[0031] FIG. 2 shows another example of the configuration of the HUD device 1, which is the virtual image display device of the first embodiment. In the example of the configuration of FIG. 2, the HUD device 1 is installed outside the dashboard 2A in the vehicle 2. The image light 1a emitted from the HUD device 1 is irradiated to the inner surface of the windshield 3 and reflected. The reflected image light 1b is incident on the driver's viewpoint 6. When viewed from the driver's viewpoint 6, a virtual image 9 corresponding to the image light 1b is formed and visible beyond the HUD display area 5 corresponding to the irradiation area of ​​the image light 1a. As shown in FIG. 2, the HUD device 1 is integrated, but it does not have to be integrated. For example, the light source device is mounted on the roof of the vehicle 2, and the image light projection unit is arranged on the dashboard 2A.

[0032] [In-vehicle system] 3 shows a configuration example of an in-vehicle system 300 that is a system including a HUD device 1. The in-vehicle system 300 is provided in a vehicle 2. The in-vehicle system 300 includes a controller 100 and a HUD device 1. The controller 100 and the HUD device 1 are connected by a predetermined information transmission path 200, in other words, a communication path 200, and are capable of communicating with each other. The controller 100 is a control device / control unit also called an ECU (Electronic Control Unit), for example, and controls the entire in-vehicle system 300.

[0033] The controller 100 acquires / generates vehicle information 4 from various sensors (FIG. 5) installed in the vehicle 2. The various sensors detect various events that occur in the vehicle 2 and detect various parameter values ​​related to the driving conditions of the vehicle 2. The controller 100 uses the vehicle information 4 to control the driving of the vehicle 2. The controller 100 may provide the vehicle information 4 to the HUD device 1, for example, by communication via CAN.

[0034] In this embodiment, the controller 100 is a source of external video information and the like, in other words, an image source, for the HUD device 1, and transmits video data and the like corresponding to external video input to the HUD device 1 via an information transmission path 200, and controls the video output using the HUD device 1.

[0035] 3, a car navigation system 150 (abbreviated as car navigation) is also included as a part of the in-vehicle system 100, and the car navigation 150 is connected to the controller 100 via communication. Since the HUD device 1 is connected to the car navigation 150 via the controller 100, it is also possible to obtain information from the car navigation 150. A case where the HUD device 1 obtains video data of an external video input and other information from the controller 100 via the information transmission path 200 is indicated by a dashed arrow as information input 2E. A case where the HUD device 1 obtains information from the car navigation 150 via the controller 100 and the information transmission path 200 is indicated by a dashed arrow as information input 2F. Note that the car navigation 150 may be integrally included in the controller 100.

[0036] In the example of Fig. 3, the information transmission path 200 is illustrated as a single line, and is assumed to be a single path. In the example of Fig. 3, it is assumed that all information including video data of an external video input is transmitted via this single path.

[0037] The well-known FPD-LinkIII is a representative connection form of the information transmission path 200. FPD-LinkIII is an in-vehicle video communication interface standard, and has the feature that only one cable is required for connection between devices.

[0038] The controller 100 may be connected to another device such as a car navigation system 150. For example, the car navigation system 150 may provide map information or navigation information to the controller 100, and the controller 100 may generate video information based on the provided information. The controller 100 then transmits the video information to the HUD device 1 via the information transmission path 200, and the HUD device 1 displays a video as a virtual image based on the video information. The HUD device 1 may also acquire information from the car navigation system 150 via the controller 100 and the information transmission path 200.

[0039] In addition, in FIG. 3, the controller 100 and the car navigation system 150 are separate components, but the car navigation system 150 may be integrally included in the controller 100 as a function.

[0040] FIG. 4 shows an example of a dual-path configuration of the information transmission path 200 between the controller 100 and the HUD device 1. In the example of FIG. 4, as shown in (A), the information transmission path 200 is composed of two information transmission paths, a first information transmission path 201 and a second information transmission path 202, which are illustrated by two lines, forming two paths. (B) shows an example of normal use, in which only the first information transmission path 201 is used for video data transmission. (C) shows an example of use during switching, in which only the second information transmission path 202 is used for video data transmission. The path used for video data transmission is illustrated by a solid line, and the path not used for video data transmission is illustrated by a dashed line. In another example of use, both the first information transmission path 201 and the second information transmission path 202 can be used for video data transmission simultaneously and in parallel.

[0041] 3 and 4 are paths capable of bidirectional communication, but in this embodiment, they are mainly used for external video input, and therefore are mainly used for video data transmission in the downlink direction from the controller 100 to the HUD device 1. In addition to video data transmission, the information transmission path 200 is capable of control communication and the like. This is possible without being limited to the configuration examples of the information transmission path 200 in FIG. 3 and FIG. 4.

[0042] As will be described later, the HUD device 1 and the controller 100 each have a communication unit mounted thereon that corresponds to the communication interface of the information transmission path 200. In this embodiment, the information transmission path 200 is configured as a wired connection using FPD-LinkIII, but is not limited thereto, and may be a wireless connection or a combination of wired and wireless connections.

[0043] In the case of the duplicated information transmission path 200 of Fig. 4, as an implementation example, it may be a configuration in which two FPD-LinkIII systems are provided, or a configuration in which different connection forms are combined. For example, it may be a combination of FPD-LinkIII and CAN, or a combination of FPD-LinkIII and Ethernet (registered trademark). It may be a triple or more configuration. It may also be a configuration in which the path used for video data transmission and the path used for control communication are separated. For example, a configuration in which one path is used only for control communication, another path is used only for video data transmission, and another path can be used for both control communication and video data transmission.

[0044] In one example of the present embodiment, the first transmission path 201 and the second transmission path 202, which are dual information transmission paths, are configured such that, for example, the first transmission path 201 is configured with FPD-Link III and the second transmission path 202 is configured with CAN. The first transmission path 201 is configured with a wired cable according to FPD-Links III. The CAN of the second transmission path 202 is configured with a wireless path.

[0045] In the case where there are multiple information transmission paths, such as a redundant system, only some of the transmission paths may be used in normal operation, and other transmission paths may be used in abnormal operation. Alternatively, multiple transmission paths may be used simultaneously in parallel. For example, the first transmission path 201 may transmit a relatively large amount of video data, and the second transmission path 202 may transmit a relatively small amount of other information, such as vehicle information 4 and control information.

[0046] [Vehicle information] 3, the controller 100 may display the vehicle information 4, etc., as a virtual image 9, which is an image on the HUD device 1. In this case, the image data / image information transmitted from the controller 100 to the HUD device 1 via the information transmission path 200 includes the vehicle information 4 or vehicle information image data that visualizes the vehicle information 4. Examples of the vehicle information 4 acquired / generated by the controller 100 / in-vehicle system 300 include the following information:

[0047] The vehicle information 4 includes, for example, speed information (also referred to as vehicle speed) of the vehicle 2, gear information, steering wheel angle information, lamp illumination information, external light information, distance information, infrared information, engine ON / OFF information, camera image information, acceleration gyro information, GPS (Global Positioning System) information, navigation information, vehicle-to-vehicle communication information, road-to-vehicle communication information, input information from the driver, and the like.

[0048] The input information from the driver is any information input by the driver through an operation input device. In the in-vehicle system 300 in this embodiment, the means for inputting an operation to the HUD device 1 by the driver, who is the user, is not on the HUD device 1 side but on the controller 100 side. Therefore, the input information from the driver is one of the vehicle information 4 generated by the controller 100. An operation input device provided in the vehicle 2 or the controller 100, for example, an operation input device such as a button installed near the steering wheel 2B, a car operation switch 511 (steering switch, etc.) in FIG. 5, accepts an operation by the driver and generates the input information from the driver. The input information from the driver may be an instruction to the controller 100 or an instruction to the HUD device 1.

[0049] [Video source] In the first embodiment, the video to be displayed on the HUD device 1 (in other words, the video to be displayed, the video information, and the video data) is generated by the controller 100. In the present embodiment, the video source and the external video input source are the controller 100, and the video destination and the external video input destination are the HUD device 1.

[0050] In the first embodiment, the only image provider for the display target image, which is the image to be displayed on the HUD device 1, is the controller 100. The controller 100 generates image data for the display target image and transmits it to the HUD device 1 via an information transmission path 200. The HUD device 1 receives and inputs the image data as an external image input, and displays an image based on the image data. The HUD device 1 displays the image as a virtual image 9 in the HUD display area 5 of FIG. 1.

[0051] [Controller-related function blocks] Fig. 5 shows an example of a functional block configuration of the controller 100. In Fig. 5, examples of various sensors and devices installed in the vehicle 2 or the controller 100 include the following: a vehicle speed sensor 501, a shift position sensor 502, a steering wheel steering angle sensor 503, a headlight sensor 504, an illuminance sensor 505, a chromaticity sensor 506, a distance measurement sensor 507, an infrared sensor 508, an engine start sensor 509, an acceleration sensor 512, a gyro sensor 513, a temperature sensor 514, a wireless transceiver for road-to-vehicle communication 515, a wireless transceiver for vehicle-to-vehicle communication 516, a GPS receiver 518, a VICS (Vehicle Information and Communication System, registered trademark) receiver 519, a camera (in-vehicle camera) 531, a camera (out-vehicle camera) 532, an audio input device 541 (including a microphone, etc.), an audio output device 542 (including a speaker, etc.), an image generating unit 510, and a communication unit 520. The sensors and devices are not limited to these, and may be added, deleted, replaced, etc.

[0052] The vehicle speed sensor 501 detects the speed of the vehicle 2 (also referred to as vehicle speed) and generates speed information as the detection result. The shift position sensor 502 detects the current gear and generates gear information as the detection result. The steering wheel steering angle sensor 503 detects the current steering wheel steering angle and generates steering wheel steering angle information as the detection result. The headlight sensor 504 detects the ON / OFF status of the headlights, etc. and generates lamp illumination information as the detection result. The illuminance sensor 505 and chromaticity sensor 506 detect external light and generate external light information as the detection result.

[0053] The distance sensor 507 detects the distance between the vehicle 2 and an external object, and generates distance information as the detection result. The infrared sensor 508 detects the presence or absence of an object in the vicinity of the vehicle 2 and the distance, and generates infrared information as the detection result. The engine start sensor 509 detects the ON / OFF of the engine, and generates ON / OFF information as the detection result. The vehicle operation switch 511 is, for example, a steering switch provided on the steering wheel 2B, and generates input information from the driver by accepting the operation by the driver. The acceleration sensor 512 and the gyro sensor 513 detect the acceleration and angular velocity of the vehicle 2, and generate acceleration gyro information representing the attitude and behavior of the vehicle 2 as the detection result. The temperature sensor 514 detects the temperature inside and outside the vehicle 2, and generates temperature information as the detection result.

[0054] The in-vehicle camera 531 captures the inside of the vehicle 2 to generate in-vehicle camera video information. The outside camera 532 captures the outside of the vehicle 2 to generate outside camera video information. In a specific example, the in-vehicle camera 531 captures the driver's posture, eye position, movement, etc., and constitutes a DMS (Driver Monitoring System). The outside camera 532 captures the surrounding situation, for example, the front of the vehicle 2. By analyzing the outside camera video information, it is possible to grasp the presence or absence of other vehicles and people around the vehicle 2, buildings and topography, road surface conditions such as rain, snow, ice, unevenness, and road signs. The outside camera 532 also includes a drive recorder that records the situation during driving as video.

[0055] The road-to-vehicle communication wireless transceiver 515 generates road-to-vehicle communication information by road-to-vehicle communication between the vehicle 2 and roads, signs, traffic lights, etc. The vehicle-to-vehicle communication wireless transceiver 516 generates vehicle-to-vehicle communication information by vehicle-to-vehicle communication between the vehicle 2 and other vehicles in the vicinity. The GPS receiver 518 generates GPS information by receiving GPS signals from GPS satellites. The GPS information includes information such as the current time, latitude, and longitude. The VICS receiver 519 generates VICS information obtained by receiving VICS signals. The GPS receiver 518 and the VICS receiver 519 may be provided as part of a navigation system.

[0056] The image generating unit 510 is a part that generates image information for HUD display on the HUD device 1. The image generating unit 510 may be implemented inside the controller 100.

[0057] The communication unit 520 is a device or circuit in which a communication interface corresponding to the information transmission path 200 is implemented. In this example, the communication unit 520 corresponds to the configuration of the duplexed information transmission path 200 in FIG. 4, and at least a communication interface of CAN and FPD-LinkIII is implemented. The communication unit 520 includes a first communication unit 521 and a second communication unit 522. The first communication unit 521 is implemented with FPD-LinkIII as a first communication interface, and the second communication unit 522 is implemented with CAN (Controller Area Network) as a second communication interface. Although the communication unit 520 is illustrated as one block, each communication interface may be provided as a separate block. Not limited to this, the communication unit 520 may also be implemented with another communication interface. In another example, the first communication interface of the first communication unit 521 may be GMSL (Gigabit Multimedia Serial Link) or the like, and the second communication interface of the second communication unit 522 may be in-vehicle Ethernet or the like.

[0058] The temperature sensor 514 may be used in the HUD device 1 in the following manner, depending on the location where it is installed. In the case where the HUD device 1 is stored in the dashboard 2A as shown in FIG. 1, the temperature sensor 514 is used to detect the temperature around the HUD device 1 in the dashboard 2A. If the temperature becomes abnormally high, the operation of the HUD device 1 may be affected. If the temperature becomes abnormally high, the controller 100 stops the operation of the HUD device 1. In the case where the HUD device 1 is installed on the dashboard 2A as shown in FIG. 2, the temperature sensor 514 is used to detect the temperature around the HUD device 1 outside the dashboard 2A. If the temperature becomes abnormally high due to exposure to direct sunlight, for example, the operation of the HUD device 1 may be affected. If the temperature becomes abnormally high, the controller 100 stops the operation of the HUD device 1.

[0059] In this embodiment, the HUD device 1 does not include an audio input device or an audio output device, and when performing audio input and output, an audio input device 541 and an audio output device 542 provided on the controller 100 side are used. However, in this embodiment, since the HUD device 1 cooperates with the controller 100, it is possible to use sensors and devices provided on the controller 100 side under the control of the controller 100.

[0060] [Functional block of HUD device (1)] FIG. 6 is a diagram showing a first example of a functional block configuration of the HUD device 1 when no image is generated inside the HUD, and when an image is generated by a controller 100 outside the HUD.

[0061] 6 includes an MCU 1010, a non-volatile memory 1011, a volatile memory 1012, a communication unit 1015, a light source drive unit 1021, a display driver 1022, a mirror drive unit 1020, an image display device 10, a mirror 22 functioning as an image projection unit, and the like, and these components are interconnected by an architecture such as a bus 1090. On the other hand, in the case of a display device that does not include the mirror drive unit 1020 and the mirror 22 functioning as an image projection unit, the connections of the components and the associated controls are the same as those described below.

[0062] The image display device 10 includes a light source device 11 and a display panel 12. The light source device 11 generates light source light based on driving from a light source driving unit 1021 and supplies it to the display panel 12. The display panel 12 is composed of, for example, a liquid crystal display panel, and forms an image on a display screen based on driving from a display driver 1022. The display panel 12 generates and emits image light obtained by modulating an image based on the light source light. The image light is reflected by a mirror 22. The reflected image light is reflected by, for example, the windshield 3 in FIG. 1 to form a virtual image 9 in the HUD display area 5. The mirror 22 is capable of changing its orientation based on driving from a mirror driving unit 1020. In addition, when the HUD device 1 has an optical element 21, the optical element 21 has a function of controlling a light flux and optically correcting the light flux.

[0063] The MCU 1010 is a unit including a processor, and controls the HUD device 1. The non-volatile memory 1011 stores control programs, setting information, and the like. The volatile memory 1012 stores data for processing. The data includes display data for HUD display. The HUD device 1 may also include internal / external storage, not shown, and programs and data may be stored in the storage. The MCU 1010 executes processing according to a program read from the non-volatile memory 1011, etc. This constitutes an execution module corresponding to each function.

[0064] The communication unit 1015 includes a video processing unit 1013 and a communication processing unit 1014. The communication unit 1015 is equipped with a communication interface of the information transmission path 200 in FIG. 4. In this embodiment, in correspondence with the redundant configuration of the information transmission path 200 in FIG. 4, at least an FPD-LinkIII as a first communication interface and a CAN as a second communication interface are equipped. The communication unit 1015 includes a video processing unit 1013 corresponding to the first transmission path 201 of the first communication interface, and a communication processing unit 1014 corresponding to the second transmission path 202 of the second communication interface.

[0065] The video processor 1013 receives video data of an external video input from the controller 100, in other words, video information 250 (see FIG. 10 described later, etc.), via the first transmission path 201. The video processor 1013 temporarily stores the received video information 250 in, for example, the volatile memory 1012, and performs necessary processing on the video information 250. A buffer memory for the video information 250 and the corresponding display data is secured in the volatile memory 1012. Note that the buffer memory may be provided in the communication unit 1015 or the video processor 1013. The video processor 1013 transfers the processed video information 250 stored in the volatile memory 1012, in other words, the display data created according to the specifications of the video display device 10, to the display driver 1022. This causes the display driver 1022 to display the video on the display panel 12. The display driver 1022 drives the display panel 12 based on the display data read from the volatile memory 1012, thereby causing the display panel 12 to display an image. The image processor 1013 may also have a function of generating image data.

[0066] The image processor 1013 may perform image data conversion processing in addition to the reception processing of the image information 250. The image data conversion processing includes converting the image information 250 in a format handled by the controller 100 into data information in a format handled by the image display device 10. The type of image data conversion processing required varies depending on the format of the image information 250 handled by the controller 100 and the format handled by the image display device 10, but generally includes, for example, decoding of the image, conversion of the resolution, and distortion correction so that the image is displayed without distortion when projected onto the windshield 3. Depending on the image information 250 received by the HUD device 1, conversion of the resolution, size, brightness, etc. may be required, and the image data conversion processing includes such conversion. Depending on the image information 250, correction such as distortion correction and position correction may be required so that the image is displayed as a suitable image without distortion or positional deviation when displayed as a virtual image 9 on the HUD display area 5. The image data conversion processing includes such correction. The video data conversion process may also be performed by the controller 100 depending on the system configuration.

[0067] The communication processing unit 1014 receives the vehicle information 4, control information, and other information from the controller 100 through a second transmission path 202 corresponding to the second communication interface. The communication processing unit 1014 stores the received vehicle information 4, etc., in, for example, a volatile memory 1012, and performs necessary processing. Note that a buffer memory for storing the vehicle information 4, etc., may be provided in the communication unit 1015 or the communication processing unit 1014. When the vehicle information 4 is to be displayed on the HUD, the communication processing unit 1014 transfers the processed vehicle information 4 stored in the volatile memory 1012 to the display driver 1022. As a result, the display driver 1022 causes the display panel 12 to display the vehicle information 4 as an image.

[0068] 6 shows a case where the information to be displayed on the HUD is divided into the video information 250, the vehicle information 4, and the control information, and transmitted and processed through different information transmission paths 200, but is not limited thereto. For example, the vehicle information 4 may be transmitted and processed as part of the video information 250 through the same information transmission path 200, for example, the first transmission path 201.

[0069] Although not shown, the HUD device 1 may be provided with a sensor such as a sunshine sensor. When external light such as sunlight enters the HUD device 1 and is irradiated onto the display panel 12 such as a liquid crystal display panel, the panel may be burned, deteriorated, or damaged, which is not preferable. In order to prevent such a situation, a sunshine sensor or the like may be provided instead of the vehicle-side illuminance sensor 505 in FIG. 5. The HUD device 1 performs an operation to protect the panel according to the detection value of the sunshine sensor. For example, when the HUD device 1 is not in use, the mirror 22 is rotated to a predetermined state by the mirror drive unit 1020 so that external light is not irradiated onto the panel or the like in the housing. When the HUD device 1 is in use, the mirror drive unit 1020 rotates the mirror 22 to a predetermined state so that the image light is irradiated onto the windshield 3 corresponding to the HUD display area 5. Furthermore, when the HUD device 1 determines that the panel should be protected based on the detection value of the sunlight sensor, it causes the mirror driving unit 1020 to rotate the mirror 22 to a predetermined state so that external light is not irradiated onto the panel inside the housing.

[0070] 6 does not have a dedicated operation input device for the HUD device 1, but the driver can input input information from operation input devices such as the vehicle operation switch 511 on the vehicle 2 and controller 100 side. Then, the input information can be given to the HUD device 1 via the controller 100. In a modified example, the HUD device 1 may be provided with a dedicated operation input device.

[0071] [Functional block of HUD device (2)] Fig. 7 is a configuration diagram showing a second example of a functional block configuration diagram of the HUD device 1, in which an image is generated inside the HUD. In this configuration, the controller 100 outside the HUD does not generate an image. The configuration in Fig. 7 differs from the configuration in Fig. 6 mainly in that the communication unit 1015 does not include an image processing unit 1013, but includes an image processing unit 1016 connected to a bus 1090. A communication processing unit 1014 in the communication unit 1015 performs communication processing corresponding to the communication interface of the information transmission path 200, and receives vehicle information 4 and the like.

[0072] The video processor 1016 generates video information 250 to be displayed based on the control of the MCU 1010, instructions from the controller 100, the vehicle information 4, and the like. The video information 250 here corresponds to display data for displaying on the video display device 10. The video processor 1016 stores the video information 250, in other words, the display data, in, for example, the volatile memory 1012. The video processor 1016 transfers the video information 250, in other words, the display data, from the volatile memory 1012 to the display driver 1022. This causes the display driver 1022 to display the video on the display panel 12. The video processor 1016 may be provided with a memory for storing the video information 250.

[0073] In the configuration example of FIG. 7, the information of the external video input from the controller 100 is limited to the vehicle information 4 and the like.

[0074] [HUD device function block (3)] FIG. 8 is a third example of a functional block diagram of the HUD device 1, which is a configuration in which the configuration and functions of FIG. 6 and the configuration and functions of FIG. 7 are integrated into one. This configuration is capable of handling both a case where an image is generated by the controller 100 outside the HUD and the image is input and displayed on the HUD, and a case where an image is generated inside the HUD and displayed on the HUD. This configuration includes both an image processing unit 1013 in a communication unit 1015 and another image processing unit 1016. When image information 250 generated by the controller 100 is input from the outside and displayed on the HUD, the image processing unit 1013 performs the same process as described above. When image information 250 is generated by the HUD device 1 and displayed on the HUD, the image processing unit 1016 performs the same process as described above. These can be switched as appropriate.

[0075] [One information transmission path, normal case] Fig. 10 is an explanatory diagram showing a case where one information transmission path 200 as in Fig. 3 is in a normal state. Fig. 10 shows a flow in which the controller 100 generates video information 250, transmits the generated video information 250 to the HUD device 1 via one information transmission path 200, the HUD device 1 receives the video information 250, and displays an image as a virtual image 9 in the HUD display area 5 based on the received video information 250.

[0076] The controller 100 and the HUD device 1 confirm and recognize each other's specifications, etc. For example, the controller 100 recognizes in advance specifications such as the size and coordinate system of the HUD display area 5 of the HUD device 1, and generates the video information 250 based on the recognition.

[0077] Fig. 11A is a schematic diagram of an example of an actual view seen when looking ahead from the viewpoint 6 of a driver in a vehicle 2 through a HUD display area 5 of a windshield 3. In Fig. 11A, nothing is displayed in the HUD display area 5. In Fig. 11A, the driver can see the scenery outside the vehicle, and for example, there is a pedestrian 11A1 on the road on the left side.

[0078] FIG. 11B is a schematic diagram of information visually recognized by the driver when the HUD device 1 displays a virtual image 9 in the HUD display area 5 on the actual scene of FIG. 11A based on the image information 250 of FIG. 10. In this example, the images that are the virtual images 9 include images 11B1, 11B2, 11B3, and 11B4. The image 11B1 is an image that displays the vehicle speed based on the vehicle information 4 (a text image of "30 km / h"). The image 11B2 is an image that displays navigation information such as "turn right 100 m ahead." The image 11B3 is an image that displays navigation information such as "keep straight ahead." The image 11B4 is an image that displays a warning alert superimposed on the position of the pedestrian 11A1.

[0079] As shown in FIG. 10, when one information transmission path 200 is normal, for example when there is no transmission delay, an appropriate HUD display is provided as shown in FIG. 11B, thereby supporting safe driving.

[0080] [In the case of an abnormality in one information transmission path] Fig. 12 is an explanatory diagram of a case where one information transmission path 200 is in an abnormal state. The abnormality is also indicated by an x ​​mark. Due to the abnormality in the information transmission path 200, the HUD device 1 cannot receive the video information 250 from the controller 100, and since there is no video information to display, the virtual image 9 cannot be displayed. As shown in Fig. 12, when there is only one information transmission path 200, if an abnormality occurs, the HUD device 1 cannot display the video. The information visually recognized by the driver is the same as that in Fig. 11A, for example, since there is no HUD display.

[0081] [In the case of a malfunction of one information transmission path] FIG. 13 is an explanatory diagram of a case where one information transmission path 200 is in a malfunction state. A case where a transmission delay occurs due to a malfunction of the information transmission path 200 is shown. The malfunction here refers to any malfunction such as a communication malfunction, and although the transmission path is alive and data transmission itself is possible, the transmission speed is slow and the normal transmission speed cannot be ensured. Malfunctions are also shown by triangle marks. Although the HUD device 1 can receive video data frames and the like as the video information 250 from the controller 100, the video information is received with a delay, so that real-time performance of the HUD video display cannot be ensured. In the case of video information 250 for which real-time performance should be ensured, the HUD device 1 cannot ensure real-time performance, so that the HUD device 1 controls not to display the HUD rather than causing a delay in the HUD display. In the case of video information 250 for which real-time performance is not required, the HUD device 1 may display the video information as it is even if a delay occurs in the HUD display.

[0082] Furthermore, when a transmission delay occurs in the information transmission path 200, a display update stagnation may occur in the HUD display. For example, it is assumed that the HUD device 1 starts to display a virtual image 9 in the HUD display area 5 as shown in FIG. 11B based on the reception of the video information 250. Then, it is assumed that the reception of the chronologically succeeding video information 250 is delayed due to a transmission delay. During that time, the HUD display content remains in the same state as shown in FIG. 11B. The HUD device 1 may determine such a display update stagnation and take measures. For example, when the HUD device 1 determines that the HUD display content has remained the same and has not been updated for a predetermined time or more, it may erase the HUD display.

[0083] [Two information transmission paths, dealing with anomalies (1)] Fig. 14 is an explanatory diagram showing an example of control for dealing with an abnormality when an abnormality occurs in one of the transmission paths, assuming that there are a first transmission path 201 and a second transmission path 202 as two information transmission paths 200 as shown in Fig. 4. This control corresponds to the solution 1-1.

[0084] Assume that initially, the controller 100 was transmitting or was about to transmit the video information 250 to the HUD device 1 using the first transmission path 201, but the first transmission path 201 becomes abnormal. An abnormal state here is, for example, a state in which even the minimum transmission speed cannot be achieved due to a communication malfunction or error. In that case, the controller 100 and the HUD device 1 switch the path used for transmitting the video information 250 to the second transmission path 202, which is normal. The HUD device 1 appropriately displays an image based on the video information 250 transmitted and received via the second transmission path 202. The image displayed on the HUD is, for example, the same as that shown in FIG. 11B.

[0085] In this way, when there are two or more information transmission paths 200, even if some paths are abnormal, the transmission of video information and the HUD display can be continued by switching to use other normal paths. This ensures real-time performance of the HUD display. Note that, when the first transmission path 201 subsequently recovers to a normal state, the HUD device 1 and the controller 100 may switch the path used for transmitting the video information 250 back from the second transmission path 202 to the first transmission path 201.

[0086] [Two information transmission paths, dealing with anomalies (2)] Fig. 15 is an explanatory diagram showing an example of control for dealing with a case where a malfunction occurs in one of the transmission paths, assuming that there are two information transmission paths 200, similar to Fig. 14. This control corresponds to the solution 1-2.

[0087] Assume that initially, the controller 100 was transmitting or was attempting to transmit the video information 250 to the HUD device 1 using the first transmission path 201, but the first transmission path 201 went into a malfunction. The malfunction here means that the transmission itself is possible and the minimum transmission speed is realized, but the transmission speed is low, for example, below a certain threshold. In that case, the controller 100 and the HUD device 1 control the second transmission path 202, which is normal, as the main path for transmitting the video information 250, and the first transmission path 201 as the sub-path, so that these two paths are used in parallel. This control can be said to change the main path to the second transmission path 202, and is therefore a type of path switching control.

[0088] The controller 100 and the HUD device 1 may transmit the video information 250 with respective data amounts according to the transmission speed of the main path and the transmission speed of the sub-path. In the example of FIG. 15, the video information 250A is transmitted on the main path, and the video information 250B is transmitted on the sub-path. Alternatively, the controller 100 and the HUD device 1 may transmit the contents of the video information 250 by dividing them into information to be transmitted on the main path and information to be transmitted on the sub-path. For example, among the contents of the video information 250, information with a high priority may be transmitted on the main path, and information with a low priority may be transmitted on the sub-path.

[0089] The HUD device 1 appropriately displays an image based on the image information 250A transmitted and received via the second transmission path 202, which is the main path, and the image information 250B transmitted and received via the first transmission path 201, which is the sub-path. The image displayed on the HUD is, for example, similar to that shown in FIG. 11B.

[0090] [Information Restriction] 16 is an explanatory diagram of a case where information is restricted, in other words, HUD display is restricted by reducing the amount of transmission data, as an example of control to deal with a malfunction occurring in one of the information transmission paths 200. This control corresponds to solution 1-3.

[0091] In the example of FIG. 16, it is assumed that one information transmission path 200 is malfunctioning, and although transmission itself is possible, the normal transmission speed cannot be ensured, and a transmission delay occurs. In this malfunction state, the controller 100 and the HUD device 1 control to limit the information transmitted through the information transmission path 200 and limit the information displayed on the HUD device 1. In this malfunction state, the controller 100 adjusts the display target image by reviewing the display contents and reducing the data amount, and transmits only information with a high priority. For example, the image information before review 250a is changed to the image information after review 250b. The image information after review 250b is carefully selected and limited to only the image information 1601 of the vehicle speed (text image of "30km / h"), thereby reducing the data amount and information amount. The controller 100 limits the contents of the image to be displayed to information with a high priority, so that the data amount of the image information 250b to be transmitted is equal to or less than a certain amount according to the state of the transmission path.

[0092] In this example, the case where the information with high priority is limited to the video information 1601 of the vehicle speed has been shown, but the present invention is not limited to this. The type of video information with what priority may vary depending on the situation, etc. Priorities according to the type of video, etc. may be set in advance. For example, when a video of an alert for a vehicle or pedestrian ahead is generated, the video of the alert may become the information with the highest priority.

[0093] The HUD device 1 cooperates with the controller 100 to implement HUD display restrictions on the reception and display of video to be displayed. Restrictions on the HUD device 1 side correspond to restrictions on the controller 100 side. It is preferable that such information restriction control is implemented after the controller 100 and the HUD device 1 mutually confirm that there is an abnormality or malfunction in the information transmission path 200, that information restriction is necessary, and the like.

[0094] The controller 100 transmits the revised video information 250b through the information transmission path 200. The HUD device 1 receives the video information 250b through the information transmission path 200 while the information restriction is being executed. The video processing unit 1013 of the HUD device 1 outputs display data based on the video information 250b to the video display device 10, and causes the video to be displayed in the HUD display area 5.

[0095] 17 shows an example of a display during HUD display restriction, in which only the vehicle speed image information 11B1 is displayed. This makes it possible to prevent / reduce lag in the HUD display even if the transmission is not working properly.

[0096] [Control status display] Regarding the various control examples (Solutions 1-1 to 1-4) described above, the HUD device 1 may display information for informing the user of the control state in the HUD display area 5. A specific example will be described below.

[0097] FIG. 18A shows an example in which an image representing a control state is displayed as a virtual image 9 in the HUD display area 5 when information restriction and HUD display restriction control as shown in FIG. 16 are performed due to a malfunction or transmission delay of the information transmission path 200, and is an example of the image viewed by the driver.

[0098] The HUD device 1 displays, in the HUD display area 5, the image information 1601 (FIG. 16) of the vehicle speed corresponding to the restricted image information 250b. At the same time, the HUD device 1 displays, for example, an OSD (on-screen display) display of an image 1801 (FIG. 18A) indicating that information restriction is currently being executed as a control state, by adding or superimposing it. That is, this image 1801 is not information transmitted from the controller 100, but is image information displayed with a transmission delay. In this example, the image 1801 of "information restricted (transmission delay)" is displayed, for example, in the upper left part of the HUD display area 5 by the OSD. The "transmission delay" in parentheses indicates the reason for executing "information restriction", but may be omitted. The image 1801 is not limited to a character image, and various forms such as a mark, an icon, a figure, and an animation are possible, and for example, a warning mark is displayed by blinking.

[0099] By viewing the image 1801 of "information restricted", the driver, who is the user, can easily recognize that the HUD device 1 is in an information restricted state and a HUD display restricted state. Although displaying this image 1801 is not essential, such display is helpful and preferable since it makes it easy for the user to know that the information is currently restricted. Furthermore, displaying this image 1801 is not limited to being constantly displayed while the information restriction is being executed, and may be a temporary display. For example, after the information restriction state is first entered, this image 1801 may be displayed for a certain period of time and then erased. Furthermore, this image 1801 may be displayed periodically thereafter. Furthermore, when the information restriction state is improved and restored to a normal state, an image indicating that the information restriction has ended may be displayed.

[0100] Furthermore, the display of the image representing the control status is not limited to the OSD display, and the image information 250 transmitted from the controller 100 to the HUD device 1 may include image information representing the control status, such as “information restricted”.

[0101] 18A, the control state may be output using not only the HUD display but also audio output. The HUD device 1 or the controller 100 may output audio 1811 such as "HUD display is being restricted" using a speaker (e.g., audio output device 542 in FIG. 5) provided in the HUD device 1 or the controller 100.

[0102] FIG. 18B shows another example of the display of the control state in the case of control to stop / pause the HUD display when an abnormality or malfunction occurs in the information transmission path 200 as in FIG. 12 or FIG. 13. In FIG. 18B, an image 1802 of "display stopped (transmission abnormality)" or an icon is displayed on the OSD, for example, in the upper left part of the HUD display area 5. The "transmission abnormality" in parentheses indicates the reason for executing "display stop", but may be omitted. In addition, the HUD device 1 or the controller 100 may use a speaker (for example, the audio output device 542 in FIG. 5) provided in the HUD device 1 or the controller 100 to output an audio output 1812 such as "HUD display has been stopped".

[0103] Fig. 18C shows another example of the display of the control state when the information transmission path 200 is switched as in Fig. 14. In Fig. 18C, an image 1803 of "transmission path 2" is displayed by OSD, for example, in the upper left part of the HUD display area 5. "Transmission path 2" represents the second transmission path 202 to which the switch is made.

[0104] Fig. 18D shows another example of the display of the control state when a main path and a sub-path are used in combination as the information transmission path 200 as in Fig. 15, as another example of the display of the control state. In Fig. 18D, an image 1804 of "transmission path 1-sub" and "transmission path 2-main" is displayed on the OSD in, for example, the upper left portion of the HUD display area 5.

[0105] Fig. 18E shows a variation of the display in Fig. 18C as another example of the display of the control state. An image 1805 in Fig. 18E is an image that expresses the state of switching, etc., of the two information transmission paths 200 (201, 202) between the controller 100 ("CNT") and the HUD device 1 ("display device") with icons, figures, etc. In this example, each transmission path is expressed by an arrow, the abnormal state (or non-use) of the first transmission path 201 is expressed by a cross mark, and the normal state (or in use) of the second transmission path 202, which is the switching destination, is expressed by a circle mark.

[0106] FIG. 18F shows another example of the display of the control state, which shows the state of the information transmission path 200 as shown in FIG. 10, FIG. 12, and FIG. 13, particularly the transmission performance. The image 1806 in FIG. 18F includes an image 1806a and an image 1806b. The image 1806a shows the connection form of the information transmission path 200, and for example, in the case of a wired connection such as FPD-LinkIII, "Wired" is displayed. The image 1806b shows the smoothness of transmission, the transmission speed, etc. as the transmission performance of the information transmission path 200, in other words, the state related to the transmission delay. For example, the image 1806b shows the smoothness of transmission as a value ranging from 0 to 100 in a bar. A value of 100 corresponds to the maximum speed being achieved in the communication interface of the transmission path. This display allows the driver to easily recognize the state of the transmission performance, etc. related to the HUD display of the external image input.

[0107] Fig. 18G is another example of a display of the control state, which is another display example of the state of the information transmission path 200 such as those in Figs. 10, 12, and 13. In Fig. 18G, when the information transmission path 200 is a wired connection, for example, by FPD-LinkIII, "Wired" and "FPD-LinkIII" are displayed. In addition, in the image 1807, the state of transmission performance / transmission delay is displayed in a pie chart.

[0108] Fig. 18H is another example of a display of a control state, which is another display example of the state of the information transmission path 200 as shown in Figs. 10, 12, and 13. When the information transmission path 200 is a wireless connection, the image 1808 in Fig. 18H is displayed with an icon representing the wireless connection as the base. When the wireless connection is abnormal or malfunctioning, a slash ( / ) representing the abnormality or malfunction is superimposed on the icon. Such an image 1808 is displayed only when the information transmission path 200 is in an abnormal or malfunctioning state.

[0109] Fig. 18I is another example of the display of the control state, which is a display example when the transmission path to be used is switched as in Fig. 14. For example, assume that the transmission path is switched from the second transmission path 202, which is a transmission path for a wireless connection, to the first transmission path 201, which is a transmission path for a wired connection. An image 1809 in Fig. 18I is displayed for a certain period of time at the time of the switching. The image 1809 in Fig. 18I includes an icon representing a wireless connection and an icon representing a wired connection, and the switching from the wireless connection to the wired connection is represented by an arrow image.

[0110] The various images representing the above-mentioned control state and the state of the information transmission path 200 may be displayed constantly or for a certain period of time. The images may be displayed for a certain period of time only when a change in the situation occurs. Moreover, the images may be selectively set as to whether or not to display them. When the images are set to be displayed, the position in the HUD display area 5 at which they are displayed may also be set. Moreover, the display of the images is not limited to being realized by OSD display, and the images may be created so as to be embedded as a part of the image information 250 of the image to be displayed. Moreover, the display of the images may be replaced by an audio output mode, or both display and audio may be output.

[0111] The display of images of the control state and the like as in the examples of FIG. 18A and FIG. 18B can also be similarly applied to the function of the second embodiment described later.

[0112] [Variation: Route information display] FIG. 19 shows a modified example of FIG. 14 and FIG. 18C. FIG. 19 shows a control example of a case where the first transmission path 201 of two information transmission paths 200 (201, 202) has an abnormality and is switched to the normal second transmission path 202. As in FIG. 14, the controller 100 and the HUD device 1 cooperate to switch from the first transmission path 201 to the second transmission path 202. The controller 100 transmits information on the transmission path to be used and information indicating which transmission path is used to transmit the video information 250 or is being transmitted, together with the transmission of the video information 250. In this example, the video information 250 includes information indicating the second transmission path 202 as used transmission path information 1901. Such used transmission path information 1901 may be described in the attribute information or metadata of the video information 250. In addition, when the communication units of the controller 100 and the HUD device 1 know which transmission path is being used, the separate transmission of the used transmission path information 1901 may be omitted.

[0113] The HUD device 1 adds and displays an image showing the transmission path state and the transmission path in use to the image displayed in the HUD display area 5 by OSD display or by using the transmission path in use information 1901 itself, based on the transmission path in use information 1901 accompanying the image information 250. As a result, a display similar to that shown in FIG. 18C can be obtained.

[0114] [Detection of abnormalities and malfunctions in information transmission paths] Examples of technical means for determining and detecting an abnormality or malfunction in the information transmission path 200 include the following. Fig. 20 is an explanatory diagram of a configuration for determining and detecting an abnormality or malfunction in the information transmission path 200 in the in-vehicle system 300. In the controller 100, the communication unit 520 described above (Fig. 5) performs communication processing at each communication interface of the information transmission path 200. In the HUD device 1, the communication unit 1015 described above (Fig. 6) performs communication processing at each communication interface of the information transmission path 200.

[0115] First, the first communication unit 521 of the communication unit 520 transmits the video information 250 through the first transmission path 201, and the video processing unit 1013 of the communication unit 1015 receives the video information 250 through the first transmission path 201. For example, each unit sequentially transmits the video information 250 for each video data frame. Assume that the communication conditions on the first transmission path 201 deteriorate. In this case, the first communication unit 521 can determine and detect a decrease in the transmission speed, in other words, a transmission delay, or an abnormality / error in the first transmission path 201, based on the fact that, even if the first communication unit 521 transmits a video data frame, an acknowledgement (ACK) frame indicating that the data has been normally received is not returned from the video processing unit 1013. The first communication unit 521 may inform the video processing unit 1013 of such a condition through control communication.

[0116] Similarly, from the perspective of the HUD device 1, the video processing unit 1013 can determine and detect that the video data frame does not arrive from the first communication unit 521, or arrives slowly, which indicates a decrease in the transmission speed, in other words, a transmission delay, or an abnormality / error in the first transmission path 201. The video processing unit 1013 may notify the first communication unit 521 of such a state by control communication.

[0117] The first communication unit 521 on the controller 100 side and the video processor 1013 on the HUD device 1 side cooperate to perform control so as to switch to the second transmission path 202, for example, as shown in Fig. 14, when a transmission delay or abnormality is detected in the first transmission path 201 as described above. For example, a request or instruction to switch to the second transmission path 202 may be transmitted from the first communication unit 521 side to the video processor 1013 side by control communication on the first transmission path 201 or the second transmission path 202. When the second transmission path 202 is available for transmitting the video information 250, the first communication unit 521 and the video processor 1013 switch to the second transmission path 202 and perform transmission. Conversely, a request or instruction to switch to the second transmission path 202 may be transmitted from the video processing unit 1013 to the first communication unit 521 via control communication on the first transmission path 201 or the second transmission path 202.

[0118] As described above, the state of the information transmission path 200 may be judged or detected by either the controller 100 side or the HUD device 1 side, or may be judged or detected by both, or the state may be appropriately notified from one side to the other. Between the HUD device 1 and the controller 100, the HUD device 1 and the controller 100 judge and confirm with each other whether there is an abnormality or malfunction in the information transmission path 200, whether switching or restriction is necessary, etc., and cooperate based on the judgment and execute the various controls described above.

[0119] Furthermore, the video data transmission and the control communication may be transmitted in a mixed manner on one information transmission path 200, or may be transmitted separately on different information transmission paths 200. Examples of control communication include communication related to determining the state of the transmission path and controlling switching of the transmission path to be used.

[0120] Also, a method may be used in which different types of video content of the video information 250 are provided, and each type of video is transmitted through a different information transmission path 200. For example, in the example of Fig. 6, when video related to the vehicle information 4 is displayed on the HUD, the second transmission path 202 is used. When video information other than the vehicle information 4 is displayed on the HUD, the first transmission path 201 is used.

[0121] [Processing flow (1)] FIG. 21 shows a process flow in the case where the switching of the information transmission path 200 is controlled as in the example of FIG. 14 in the in-vehicle system 300 in the first embodiment. The in-vehicle system 300 judges whether or not to switch the information transmission path 200 used for transmitting the video information 250 in cooperation between the HUD device 1 and the controller 100. As described above, this judgment may be made by either the HUD device 1 or the controller 100. In addition, the control process including this judgment may be performed by the controller 100 as the main and the HUD device 1 as the secondary, or by the HUD device 1 as the main and the controller 100 as the secondary. The in-vehicle system 300 may judge the availability and usage status, and the status of transmission performance such as transmission speed / transmission delay as the status of each of the multiple information transmission paths 200.

[0122] In step S100, the in-vehicle system 300 starts a process of checking the status of the information transmission path 200. In the on-state of the function related to the first embodiment, a process flow as shown in FIG. 21 is executed. In step S101, the in-vehicle system 300 monitors the information transmission status in the information transmission path 200. The monitoring can be performed by the respective communication units of the controller 100 and the HUD device 1 as shown in FIG. 20. In step S102, the in-vehicle system 300 checks whether the transmission speed of the video information 250 in the information transmission path 200 is equal to or higher than a predetermined speed. The predetermined speed can be set as a threshold. The predetermined speed is a setting value for judging whether the transmission performance at the transmission speed is sufficient. If the speed is equal to or higher than the predetermined speed (Yes), the process proceeds to step S109, and if the speed is not equal to or higher than the predetermined speed (No), the process proceeds to step S103.

[0123] In step S103, the in-vehicle system 300 checks whether there is an available alternative transmission path for the currently used transmission path, in other words, a candidate for switching. If there is an alternative path (Yes), the process proceeds to step S105, and if there is no alternative path (No), the process proceeds to step S104. For example, when the first transmission path 201 is in use and the second transmission path 202 is available, the second transmission path 202 can be used as an alternative path. For example, when the first transmission path 201 is in use and the second transmission path 202 is being used to transmit important information, the second transmission path 202 cannot be used as an alternative path.

[0124] The processing flow of this embodiment is adapted to select and determine, based on the situation such as an abnormality in the information transmission path 200, whether to take measures such as switching to an alternative path, limiting the HUD display by reducing the amount of transmission data, or stopping the HUD display by stopping transmission, etc. In other words, a plurality of functions corresponding to the above-mentioned plurality of solving means are appropriately selected and applied.

[0125] In step S104, since the transmission performance of the currently used transmission path is low and there is no alternative path, the in-vehicle system 300 determines whether to stop or suspend information transmission on the currently used transmission path, i.e., to stop the HUD display as described above, or to reduce the amount of data and information to be transmitted and continue transmission, i.e., to limit the HUD display as described above, and executes the determined countermeasure. Various countermeasure determinations are possible in step S104, and specific examples will be described later. In addition, when executing the countermeasure, the controller 100 and the HUD device 1 cooperate with each other by notifying and confirming with each other, as described above, to execute the countermeasure.

[0126] In step S108, the in-vehicle system 300 outputs to the user the status of the action determined in step S104. Examples of the output include the above-mentioned Fig. 18A, etc. After step S108, the process proceeds to step S109.

[0127] In step S105, the in-vehicle system 300 selects one route to be used as a switching destination from among one or more alternative routes for the currently used transmission route.

[0128] In step S106, the in-vehicle system 300 switches the information transmission path 200. When dealing with this switching, the controller 100 and the HUD device 1 cooperate with each other, such as by notifying and confirming with each other, to carry out the handling. A specific example of cooperation will be described later.

[0129] In step S107, the in-vehicle system 300 outputs to the user the switching state of the information transmission path 200. Examples of the output include the above-mentioned Fig. 18C etc. After step S107, the process proceeds to step S109.

[0130] In step S109, the in-vehicle system 300 ends the information transmission status confirmation process. The flow of Fig. 21 is similarly repeatedly executed while the function is in the ON state.

[0131] [Processing flow (2)] The process flow in FIG. 22 is a modified example of the process flow in FIG. 21, a detailed process example, and is an example in which main decisions are made on the controller 100 side. A specific example of cooperation between the controller 100 and the HUD device 1 is also shown.

[0132] In step S201, the controller 100 monitors the information transmission status of the information transmission path 200. In step S211, the HUD device 1 monitors the information reception status of the information transmission path 200.

[0133] In steps S202 and S203, the controller 100 makes the same judgment as described above, and proceeds to step S204, step S207, or step S211 depending on the judgment. In step S204, the controller 100 decides on an action to take, whether to stop transmission or reduce the amount of data, and in step S205 notifies the HUD device 1 of the decided action. In step S212, the HUD device 1 receives the notification of the action. In step S206, the controller 100 takes action by stopping transmission or by performing transmission with reduced data amount. In step S213, the HUD device 1 takes action according to the notification, by stopping HUD display corresponding to the stop of transmission, or limiting HUD display corresponding to transmission with reduced data amount.

[0134] In step S207, the controller 100 selects a route to switch to, and in step S208, the controller 100 notifies the HUD device 1 of a course of action for switching the route. In step S214, the HUD device 1 receives the notification of the action, and in step S215, the HUD device 1 prepares to switch the route as a course of action in accordance with the notification, and when preparation is complete, transmits a preparation completion notification to the controller 100. In step S209, the controller 100 receives the preparation completion notification, and in step S210, the controller 100 executes route switching. In step S211, the information transmission status confirmation process ends.

[0135] As described above, in the flow of FIG. 22, the controller 100 takes the lead in controlling the situation judgment and the decision to deal with the situation, and the HUD device 1 performs corresponding control in accordance with notifications and instructions from the controller 100.

[0136] [Processing flow (3)] The process flow of FIG. 23 is a modified example of the process flow of FIG. 21, a detailed process example, and is an example in which main decisions are made on the HUD device 1 side. A specific example of cooperation between the controller 100 and the HUD device 1 is also shown.

[0137] In step S301, the HUD device 1 monitors the information reception status of the information transmission path 200. In step S311, the controller 100 monitors the information transmission status of the information transmission path 200.

[0138] In steps S302 and S303, the HUD device 1 makes the same judgment as described above, and proceeds to step S304, step S307, or step S311 depending on the judgment. In step S304, the HUD device 1 determines the action to be taken, whether to stop the HUD display by stopping the transmission or to limit the HUD display by transmitting a reduced amount of data, and in step S305, notifies the controller 100 of the determined action. In step S312, the controller 100 receives the notification of the action. In step S313, the controller 100 executes the transmission stop or the transmission with the reduced amount of data as the action according to the notification. Correspondingly, in step S306, the HUD device 1 executes the HUD display stop corresponding to the transmission stop or the HUD display limit corresponding to the transmission with the reduced amount of data as the action.

[0139] In step S307, the HUD device 1 selects a route to switch to, and in step S308, the HUD device 1 notifies the controller 100 of a course of action for switching the route. In step S314, the controller 100 receives the notification of the action, and in step S315, the controller 100 prepares to switch the route as a course of action in accordance with the notification, and when the preparation is complete, transmits a preparation completion notification to the HUD device 1. In step S309, the HUD device 1 receives the preparation completion notification, and in step S310, the HUD device 1 executes the route switching. In step S311, the information transmission status confirmation process ends.

[0140] As described above, in the flow of FIG. 23, the HUD device 1 takes the lead in controlling the situation judgment and the response decision, and the controller 100 performs the corresponding control according to notifications and instructions from the HUD device 1.

[0141] [Examples of control combinations] FIG. 24 is a table summarizing examples of application of the above-mentioned multiple types of control, for example, switching of the information transmission path 200, HUD display restriction by transmission with reduced data amount, and HUD display stop by transmission stop. The table in FIG. 24 also shows an example of selecting and switching the application of multiple types of control using control priority. In the table in FIG. 24, rows indicated by #1 to #4 show each control example. The column items from the left are "path in use", "abnormal case", and "path to be switched and switching content", and "path to be switched and switching content" has columns of priority 1 to priority 5. The priority here is the priority regarding the application of multiple types of control. Here, the highest priority is priority 1, and the lowest priority is priority 5. In this example, it is assumed that the information transmission path 200 has a first transmission path 201 and a second transmission path 202 as shown in FIG. 4 and FIG. 14.

[0142] (#1) The first example is as follows. First, assume that the information transmission path 200 being used for transmitting the video information 250 of the external video input from the controller 100 to the HUD device 1 is the first transmission path 201. As an abnormal case, assume that the first transmission path 201 goes into a "transmission unavailable" state. In this case, the first priority 1 measure is to switch to the second transmission path 202. If the second transmission path 202 is available, this switching is executed. If it is not available, the next candidate is a priority 2 measure. That is, the priority 2 measure is to stop the HUD display by stopping the transmission.

[0143] (#2) The second example is a case where, compared to the first example, the information transmission path 200 being used for transmitting the video information 250 is the second transmission path 202. As an abnormal case, it is assumed that the second transmission path 202 is in a "transmission not possible" state. In this case, the first priority 1 measure is to switch to the first transmission path 201. If the first transmission path 201 is usable, this switching is executed. If it is not usable, the next candidate is a priority 2 measure. That is, the priority 2 measure is to stop the HUD display by stopping the transmission.

[0144] (#3) The third example is as follows. First, assume that the information transmission path 200 being used for transmitting video data is the first transmission path 201. As an abnormal case, assume that the first transmission path 201 is in a state of "below the transmission speed threshold", in other words, in a state where the transmission performance has deteriorated, or in the above-mentioned malfunction state. In this case, the first priority 1 measure is to switch the second transmission path 202 to the main path and the first transmission path 201 to the sub-path, and to use these two paths in parallel for transmission. FIG. 15 above is an example. If the second transmission path 202 is usable and transmission is possible with the transmission speed of the first transmission path 201 being greater than the threshold, this priority 1 measure is executed. In order to obtain the maximum efficiency of transmission, first, the combined use of such two paths is considered as the first priority. The second transmission path 202, which allows normal transmission, is used as a main path, and the first transmission path 201, which has a transmission abnormality but is still capable of transmission, is used as a sub-path, and the two paths are used in combination.

[0145] Next, when the transmission speed of the first transmission path 201 used in combination at priority level 1 is equal to or lower than the threshold and is very slow, for example close to 0, as a measure for priority level 2, a switch is made to the second transmission path 202, and transmission is performed only via the second transmission path 202.

[0146] Next, when the second transmission path 202 is unavailable in the priority 2, that is, when transmission is not possible or the transmission speed is equal to or lower than a threshold, for example, when it is slower than the transmission speed of the first transmission path 201, the control of priority 3 is performed to transmit data via the first transmission path 201, but with a reduced amount of data. This limits the HUD display.

[0147] Next, when the transmission speed of the second transmission path 202 at priority level 3 is equal to or lower than the threshold but is still faster than the transmission speed of the first transmission path 201, the amount of data is reduced and transmitted via the second transmission path 202 as control at priority level 4. This limits the HUD display.

[0148] Next, in the case where a delay occurs even if the data volume is reduced to the minimum necessary in both the first transmission path 201 and the second transmission path 202 under priority level 4, the HUD display is stopped by stopping transmission under priority level 5 control.

[0149] (#4) In the fourth example, compared to the third example, the information transmission path used for transmitting the video information 250 at first is the second transmission path 202, and the selected path is reversed. As an abnormal case, it is assumed that the second transmission path 202 is in a "transmission speed below threshold" state. In this case, the first priority 1 measure is to switch the first transmission path 201 as the main path and the second transmission path 202 as the sub path, and perform transmission using these two paths in parallel. If the first transmission path 201 is usable and transmission is possible in which the transmission speed of the second transmission path 202 is greater than the threshold, this priority 1 measure is executed.

[0150] Next, in the case where the transmission speed of the second transmission path 202 used in conjunction at priority level 1 is equal to or lower than the threshold and is very slow, for example close to 0, as a measure for priority level 2, a switch is made to the first transmission path 201, and transmission is performed only via the first transmission path 201.

[0151] Next, in the case where the first transmission path 201 is unavailable in the priority 2, that is, where transmission is not possible or the transmission speed is equal to or lower than a threshold, for example, where the transmission speed is slower than that of the second transmission path 202, the control of priority 3 is performed so that the second transmission path 202 is used for transmission, but with a reduced amount of data. This limits the HUD display.

[0152] Next, in the case where the transmission speed of the first transmission path 201 is equal to or lower than the threshold value in the priority level 3 but is still higher than the transmission speed of the second transmission path 202, the amount of data is reduced and transmitted via the first transmission path 201 as the control of priority level 4. This limits the HUD display.

[0153] Next, in the case where a delay occurs even if the data volume is reduced to the minimum necessary in both the second transmission path 202 and the first transmission path 201 under priority level 4, the HUD display is stopped by stopping transmission under priority level 5 control.

[0154] [Effects of the first embodiment] As described above, according to the first embodiment, it is possible to provide a more suitable HUD technology that can handle the case where an image is generated outside the HUD and the HUD receives and displays the image as an image input. According to the first embodiment, even if the information transmission path 200 between the controller 100 and the HUD device 1 is in an abnormal or malfunctioning state, the HUD display is improved to ensure real-time HUD display by taking measures such as switching or combining transmission paths, reducing the amount of transmitted data, or stopping transmission. This makes it possible to prevent safe driving from being impeded.

[0155] <Embodiment 2> The second embodiment will be described with reference to Fig. 25 and subsequent figures. The basic configuration of the second embodiment is similar to and common to the first embodiment, and the following mainly describes the components of the second embodiment that are different from the first embodiment. The second embodiment has a function corresponding to the above-mentioned solution means 2.

[0156] [2-1. HUD display control according to the content of video information from external video input] As outlined in Fig. 9 above, in the second embodiment, the in-vehicle system 300 performs control as a solution 2-1 to check the appropriateness of the HUD display based on the video information of the external video input and to prevent an inappropriate image from being displayed. Various control examples included in the solution 2-1 will be described below.

[0157] [Appropriateness of HUD display] 25 and 26 are explanatory diagrams of a specific example regarding the appropriateness of HUD display. This example shows a case where there is one information transmission path 200 between the controller 100 and the HUD device 1, as in Fig. 9. In this example, a case is described in which the HUD device 1 checks the appropriateness of the HUD display image assuming HUD display for the image information 250 provided from the controller 100 by transmission over the information transmission path 200, and performs control to limit the HUD display according to the check result.

[0158] Fig. 25 shows a normal case, i.e., a case where the HUD display is appropriate. The HUD device 1 checks the appropriateness, in other words, appropriateness, of the HUD display for the video information 250 received from the controller 100 via the information transmission path 200. The HUD device 1, for example the video processor 1013 in Fig. 6, performs this checking and judgment process based on predetermined standards, conditions, etc. If the video processor 1013 judges that there is no problem with displaying the target video in the HUD display area 5, in other words, that the appropriateness and appropriateness are sufficiently high and the standard is met, it outputs the target video to the video display device 10 for display.

[0159] In the case of Fig. 6, the image processor 1013 stores display data of the target image in the volatile memory 1012. The display driver 1022 reads out the display data of the target image from the volatile memory 1012 and drives the display panel 12 according to the display data. As a result, the target image is displayed on the screen of the display panel 12, and the image light forms a virtual image 9 in the HUD display area 5 via the mirror 22. The display in the HUD display area 5 is similar to that in Fig. 11B, for example.

[0160] Furthermore, in such a normal case, although this is not essential, the HUD device 1 may transmit / notify the controller 100 of a response 270. The response 270 is at least one of information indicating that the target image provided by the controller 100 is to be displayed on the HUD and information indicating that there is no problem with displaying the image on the HUD, i.e., information indicating that the condition of appropriateness is met.

[0161] FIG. 26 shows an abnormal case, that is, a case where the HUD display is inappropriate. The HUD device 1 checks the appropriateness of the HUD display, in other words, the appropriateness, of the HUD display, assuming that the HUD display is performed, for the video information 250 received from the controller 100 via the information transmission path 200. The HUD device 1, for example, the video processing unit 1013 in FIG. 6 performs this checking and judgment process based on predetermined standards, conditions, and the like. When the video processing unit 1013 judges that there is a problem with the HUD display of the target video, in other words, that the appropriateness and appropriateness are low and the standard is not met, the video processing unit 1013 does not output the target video to the video display device 10 and does not display it. In the case of FIG. 6, the video processing unit 1013 does not store display data of the target video in the volatile memory 1012. As a result, the target video is not displayed on the screen of the display panel 12, and the target video is not formed in the HUD display area 5.

[0162] The example of the image information 250 in Fig. 26 is an example of an image with a certain brightness and color and a solid color. In the case of such an image, the entire area of ​​the HUD display area 5 is covered with the image, so the degree to which the image blocks the driver's view is high. Therefore, the image processor 1013 of the HUD device 1 determines that the image is not suitable for HUD display and is problematic, and performs control so that the image cannot be displayed.

[0163] FIG. 27 is a schematic diagram showing an overview of a case where a virtual image 9 is displayed in the HUD display area 5 based on the above-mentioned full-color solid image information 250. In this case, the driver can see the road ahead through the full-color solid virtual image 9, but the degree of obstruction of the driver's field of vision is high, making it difficult to see the road ahead. In addition, for example, in a case where a full-color solid white virtual image 9 is displayed in a dark environment such as at night, the display of the virtual image is too bright compared to the darkness of the surroundings, making it difficult for the driver to see the road ahead, which is very dangerous. If this image is not displayed, the result will be the same as FIG. 11A.

[0164] Furthermore, in such an abnormal case, the HUD device 1 must transmit / notify the controller 100 of a response 270. The response 270 is at least one of information indicating that the target image provided by the controller 100 will not be displayed on the HUD and information indicating that there is a problem with the HUD display of the image, i.e., information indicating that the image does not satisfy the conditions of appropriateness / validity.

[0165] FIG. 28 shows an example in which, when the HUD device 1 checks the appropriateness of a target image and determines that it is problematic and cannot be displayed, the image is not displayed in the HUD display area 5 (this can also be called display stop or display restriction), and an image 2801 indicating "display not possible", in other words, that HUD display is restricted / stopped, is displayed, for example, on the OSD. In the example of FIG. 28, the image 2801 is "HUD display not possible (display rate)", and the "display rate" in parentheses indicates the reason for "not being able to display". It is preferable that the driver be able to select whether or not to display the reason for HUD display not possible in this example. This is because some drivers may want to know the details of the reason why it is not possible to display, and conversely, some drivers may feel uneasy when the reason why it is not possible to display is displayed, which may prevent them from concentrating on driving.

[0166] It should be noted that it is also possible to restrict the display so that a part of the image is displayed and a part is not displayed, other than completely hiding the image in the HUD display area 5. For example, the display may be restricted so that the display ratio of the image in the HUD display area 5 is kept below a predetermined ratio.

[0167] [Process flow: Video confirmation] 29 shows a process flow in which the HUD device 1 checks the appropriateness and validity of the HUD display and controls the HUD display based on the video information 250 provided to the HUD device 1 from the controller 100 in the in-vehicle system 300 in the second embodiment. The main processing of the process flow in FIG. 29 is performed by the video processing unit 1013 (FIG. 6) of the HUD device 1.

[0168] In step S400, the in-vehicle system 300 starts the video confirmation process. With the function of the second embodiment on, the flow of Fig. 29 is performed. In step S401, the controller 100 transmits the video information 250 to the HUD device 1 via the information transmission path 200. The HUD device 1, particularly the video processing unit 1013 in Fig. 6, receives the video information 250 and temporarily stores it in a memory, the volatile memory 1012 in Fig. 6.

[0169] In step S402, the video processor 1013 of the HUD device 1 judges whether the received video information 250 meets the conditions for disabling HUD display, in other words, inappropriate conditions. Examples of the conditions will be described later. If it is judged that the conditions are not met and display is possible (No), the process proceeds to step S403, and if it is met and display is not possible (Yes), the process proceeds to step S404.

[0170] In step S403, the image processor 1013 of the HUD device 1 controls so as to display the received image information 250 on the HUD. The image processor 1013 outputs the image information 250 to the image display device 10. The image processor 1013 in FIG. 6 creates display data for the image information 250 and stores it in the volatile memory 1012. The display driver 1022 reads out the display data from the volatile memory 1012 and drives the display panel 12. As a result, a virtual image 9 corresponding to the image information 250 is formed in the HUD display area 5.

[0171] In step S405, the video processor 1013 of the HUD device 1 transmits a response 270 indicating that the received video information 250 has been displayed on the HUD to the controller 100. The controller 100 can recognize that the video information 250 has been displayed on the HUD from the response 270. Step S405 can be omitted.

[0172] In step S404, the image processor 1013 of the HUD device 1 performs control so as not to display the received image information 250 on the HUD. The image processor 1013 does not output the image information 250 to the image display device 10. The image processor 1013 in FIG. 6 does not store display data of the image information 250 in the volatile memory 1012. As a result, the virtual image 9 corresponding to the image information 250 is not formed in the HUD display area 5.

[0173] In step S406, the video processor 1013 of the HUD device 1 transmits a response 270 indicating that the received video information 250 will not be displayed on the HUD to the controller 100. The controller 100 can recognize from the response 270 that the video information 250 was not displayed on the HUD.

[0174] In step S407, the video confirmation process ends. The flow of Fig. 29 is repeated in the same manner as in the second embodiment in response to the generation of video information 250 while the function of the second embodiment is in an ON state.

[0175] [Conditions for display not possible] Fig. 30 is a table summarizing the above-mentioned conditions for which display is not possible or inappropriate, and corresponds to the various control examples in Fig. 9. The table lists the criteria for determining whether display is not possible or not.

[0176] [2-1-1. Judgment and restriction of HUD display amount / display rate] An example of a case where the display is not possible is when there is too much HUD display content. As a criterion for the determination, the ratio of the display content (referred to as the display ratio) in the HUD display area 5, i.e., the entire range in which the virtual image 9 can be displayed, is equal to or greater than a threshold. If the display ratio assuming HUD display based on the video information 250 to be displayed is equal to or greater than a threshold, it is determined that the display is not possible or inappropriate. As a specific example of a method of determination, the video processor 1013 of the HUD device 1 calculates the display ratio by comparing the total area of ​​the display content based on the display data of the video information 250 to be displayed with the area of ​​the HUD display area 5. For example, if the display ratio is 30% or more, it is determined that the display is not possible.

[0177] In this processing example, in order to check the undisplayability and appropriateness, the video processor 1013 creates provisional display data for the case where the target video information 250 is displayed on the HUD, and calculates the display rate and the like using the provisional display data. If the result of the check is that the HUD is not to be displayed, the provisional display data is discarded. As another example of the determination method in step S402, it may be determined that the HUD is not to be displayed when a feature corresponding to the undisplayable case in FIG. 30 is extracted by analyzing the binary data of the target video information 250.

[0178] In the above example, the display rate is used to determine whether or not display is possible, but this is not limiting. Instead, the amount of display / amount of data / amount of information on the HUD display may be calculated to determine whether or not display is possible.

[0179] 31 shows a specific example of calculation of the display ratio corresponding to the control example of 2-1-1 above. Based on the target video information 250, the rectangle of the HUD display area 5 has areas 3101 and 3102 as areas in which display contents are arranged, for example, in the bottom side area or the area near the center. When the entire area of ​​the HUD display area 5 is taken as 100, if the total area of ​​the areas 3101 and 3102 is equal to or greater than, for example, a threshold value of 30%, it is determined that display is not possible.

[0180] As an example of control when it is determined that display is not possible based on the above conditions, there is a basic control to hide all the video information 250 corresponding to the areas 3101 and 3102. Not limited to this, it is also possible to control to reduce the content to be displayed so that the display ratio is less than a threshold. For example, if the display ratio is less than the threshold by hiding one of the area 3101 of the central content and the area 3102 of the bottom side content, such a display may be used. Furthermore, if display priorities are set according to the display position in the HUD display area 5 or the type of display content, it may be controlled to display the content with the highest priority and hide the content with the lowest priority.

[0181] FIG. 32 shows another example of the calculation of the display ratio. Based on the target image information 250, in the rectangle of the HUD display area 5, for example, display contents by the virtual image 9 may be arranged in areas and places shown by A1, A2, A3, B1, B2, B3, C1, C2, and C3. For example, A1, A2, and A3 are examples of areas near the center in the vertical direction, B1, B2, and B3 are examples of lower side areas, and C1, C2, and C3 are examples of upper side areas. For example, display contents are arranged simultaneously in a plurality of areas, including areas A1 and A2, areas B2 and B3, and area C1. If the display ratio based on the total area of ​​the areas of these display contents is equal to or greater than a threshold value, it is determined that the display contents cannot be displayed.

[0182] As a further modification, a weight may be added to each display position in the HUD display area 5, and the display rate may be calculated by multiplying the area by the weight value. For example, the central area A1 may have a relatively large weight value, and the peripheral areas B2, B3, C2, C3, etc. may have relatively small weight values. That is, the weight may be large near the center of the HUD display area 5 since it has a large effect on the driver's visibility and safety, and the weight may be small near the periphery since it has a small effect on the driver's visibility and safety. As a result, even if the display content has the same area, the weight that affects the display indisplayability and appropriateness changes depending on the display position, and the calculation result of the display rate using the area and weight value will differ.

[0183] [2-1-2. Judgment and restrictions on brightness of HUD display] An example of a case where display is not possible is when the brightness of the HUD display content is abnormal. A criterion for this determination is that the brightness specified in the target video information 250 is abnormal with respect to a daytime brightness reference value or a nighttime brightness reference value. As a specific example of a determination method, the HUD device 1 determines whether it is daytime or nighttime from time information, and refers to a brightness reference value corresponding to whether it is daytime or nighttime. The HUD device 1 compares the brightness (which may be, for example, an average brightness) of the display content of the target video information 250 with the brightness reference value, and if the difference therebetween is equal to or greater than a threshold value, determines that the brightness is abnormal.

[0184] Fig. 33 is an explanatory diagram of a specific example of the control example of 2-1-2 above. In this case, a navigation arrow image is displayed as display content 3301 at a position near the center of the HUD display area 5. It is assumed that it is daytime, and a daytime brightness reference value 3302 is set in advance. In the video information 250 from the controller 100, a brightness 3303 of the display content 3301 is specified. For control purposes, for example, a threshold range 3304 including the daytime brightness reference value 3302 is set. The threshold range 3304 is, in other words, a brightness reference range. If the brightness 3303 of the display content 3301 is outside the threshold range 3304, the HUD device 1 determines that it is not possible to display it.

[0185] The above control example is a case where the relative difference between the brightness of the HUD display content and a brightness reference value corresponding to the ambient brightness is determined, but this is not limited to this, and it is also possible to determine the absolute brightness of the display content.

[0186] The above example is an example in which the judgment is made by dividing the time into two broad categories, daytime and nighttime, but the present invention is not limited to this example, and judgment may be made by taking into account the influence of the characteristics of a dark place, such as inside a tunnel or under an overpass, even during the day. In addition, judgment may be made by taking into account the influence of weather, such as sunny, cloudy, rainy, or snowy. This allows for more accurate control.

[0187] As a variant example, when referring to the brightness reference value, the HUD device 1 may use a sensor of itself or the controller 100 to detect the surrounding brightness, the brightness near the HUD display area 5, etc., and calculate the brightness reference value according to the current time.

[0188] An example of control when it is determined based on the above conditions that display is not possible is basically to hide the target video information 250. Another example of control is not limited to this, and it is also possible to control the brightness of the display content to be changed so that it falls within a threshold range of the daytime brightness reference value, for example, as a HUD display restriction.

[0189] [2-1-3. Judgment and restrictions on information errors and discrepancies with the actual situation] An example of a case where display is not possible is when insufficient information causes a discrepancy between the actual situation and the HUD display. An example of a criterion for determination is an error in information provided to the user. The HUD device 1 determines whether or not there is an error in information provided as the video information 250 from the controller 100, such as the vehicle speed of the vehicle and route information of the navigation system, in light of the actual situation. For example, the HUD device 1 estimates that there is an error when the actual situation and the provided information do not completely match, or when the difference between the actual situation and the provided information is equal to or greater than a predetermined value.

[0190] As a specific example of the determination method, the HUD device 1, particularly the video processor 1013 in Fig. 6, analyzes the video content information provided as the video information 250 from the controller 100, or refers to attribute information or metadata attached to the video information 250, to grasp the video content information to be determined as the error. At the same time, the HUD device 1 refers to information acquired from itself or another information source as information corresponding to the target video content information. The HUD device compares the target video content information with the corresponding information, and determines whether the target video content information is free of errors, in other words, whether there is a discrepancy with the actual situation.

[0191] An example of another information providing source is the car navigation system 150 in Fig. 3. The HUD device 1 obtains information from the car navigation system 150 via the controller 100.

[0192] FIG. 34 shows a specific example of the control example of 2-1-3 above. For example, assume that the display content 3401 based on the video information 250 from the controller 100 is a navigation arrow image as shown in the figure. This navigation arrow image is an image that encourages a right turn 100 m ahead. However, in the current actual scenery, the road ahead is a straight road, and there is no road to turn right. In other words, when this display content 3401 is displayed on the HUD, a discrepancy occurs with the actual situation. The HUD device 1 also refers to navigation information from the car navigation system 150, for example, and compares the information of the display content 3401 with the information of the car navigation system 150. For example, when the difference between the information is large, the HUD device 1 determines that the information of the display content 3401 may be erroneous and therefore cannot be displayed.

[0193] As a control example when it is determined based on the above conditions that the information cannot be displayed, the target video information 250 is basically hidden. The HUD device 1 may also display an image in the HUD display area 5 indicating that the target video information 250 is erroneous and has been hidden. As another control example, the target video information 250 may be displayed as is as a HUD display restriction, and an image indicating that the target video information 250 may be erroneous may be displayed in the HUD display area 5.

[0194] As a modified example, the HUD device 1 may use a sensor of the HUD device 1 itself or the controller 100, for example, image information captured by an outside camera, to determine the actual situation in the driver's field of vision, i.e., the actual situation related to the display content. For example, the HUD device 1 may determine, from a camera image, whether the road ahead is a straight road or whether there is a road to turn right.

[0195] In regard to the various control examples described above, the video content of the video information 250 may change over time. Even in this case, similar control can be applied to each piece of video data at each point in time.

[0196] 35A and 35B show examples of display of the control state regarding the function in the second embodiment, and an image 3501 of "display restricted (inappropriate)" is displayed in the HUD display area 5, for example, by OSD. This image 3501 is an image indicating that the HUD display is restricted or stopped because it is determined to be inappropriate as a result of checking the appropriateness. In other words, this image 3501 is an image indicating that the display rate of the image displayed on the HUD is suppressed for safe driving. In the example of FIG. 35A, when there is an image to be displayed, for example, as shown in FIG. 34 based on the image information 250, the navigation information 3401 of the right turn is deleted as a result of checking the appropriateness, and the image to be displayed is limited to only the image information 11B1 of the vehicle speed. The reason why the image information 11B1 of the vehicle speed continues to be displayed when the display is restricted is that the vehicle speed information is considered to be the most basic and important information for safe driving. It is preferable that the driver can select whether or not to display the reason for the HUD display restriction as in this example. Some drivers may want to know the details of why the display is restricted, while others may feel anxious and unable to concentrate on driving if negative words such as "inappropriate" are displayed as the reason for the display restriction.

[0197] The example in FIG. 35B shows a state in which all images to be displayed are restricted, in other words, the display of the display device is stopped. As mentioned above, vehicle speed information is the most basic and important information for safe driving, but continuing to display incorrect information does not contribute to safe driving, so in this example, the vehicle speed image information 11B1 is also deleted. Such measures can be taken when there is a means in the vehicle for the driver to check vehicle speed information other than the HUD display, for example, when vehicle speed information is displayed on the meter cluster. The driver is informed that the display is restricted or stopped by audio output from the speaker.

[0198] [Effects of the second embodiment] As described above, according to the second embodiment, it is possible to provide a more suitable HUD technology that can handle cases where an image is generated outside the HUD and the HUD receives and displays the image as an image input. According to the second embodiment, when attempting to display an image on the HUD based on image information of an external image input, the appropriateness of the HUD display is confirmed, and if it is not appropriate, the display is disabled. This makes it possible to avoid impeding safe driving.

[0199] It is also possible to implement both the functions of the first embodiment and the second embodiment. It is also possible to implement a plurality of functions as various functions, including modified examples. In this case, it is possible to set which function to use in the in-vehicle system.

[0200] Although the embodiments of the present disclosure have been specifically described above, they are not limited to the above-described embodiments and can be modified in various ways without departing from the gist of the present disclosure. In each embodiment, components can be added, deleted, or replaced, except for essential components. Unless otherwise specified, each component can be singular or plural. A combination of each embodiment and its modified examples is also possible.

[0201] The technology according to the embodiment of the present invention can provide a more suitable display device technology that can handle cases where a display device receives and displays an image transmitted from an external source as an image input. By being able to check the appropriateness of the display on the display device and not displaying it if it is inappropriate, it is possible to provide a display device that contributes to supporting safe driving and prevent traffic accidents. This contributes to "Good health and well-being for all," one of the Sustainable Development Goals (SDGs) advocated by the United Nations. [Explanation of symbols]

[0202] 1...HUD device (virtual image display device, display device), 5...display area, 9...virtual image, 100...controller, 10...image display device, 200...information transmission path, 201...first transmission path, 202...second transmission path, 250...image information.

Claims

1. A virtual image display device, an image processing unit that processes image information input from an external device; an image display device that displays an image based on the image data processed by the image processing unit and outputs image light corresponding to the image; Equipped with adjusting an information transmission path for transmitting the video information from the outside in response to the display of the video; Virtual image display device.

2. 2. The virtual image display device according to claim 1, the information transmission path is provided as two or more paths when the video information is input from the outside, and when transmitting the video information through a first transmission path, if the state of the first transmission path is abnormal or malfunctioning, the video information is transmitted through a second transmission path. Virtual image display device.

3. 2. The virtual image display device according to claim 1, the information transmission path is provided when the video information is input from the outside, and when the video information is transmitted through the first transmission path, if the first transmission path is in an abnormal or malfunctioning state, the second transmission path is used as a main path and the first transmission path is used as a sub-path, and the video information is transmitted using both the main path and the sub-path. Virtual image display device.

4. 2. The virtual image display device according to claim 1, a transmission path is provided as the information transmission path when the video information is input from the outside, and when transmitting the video information via the one transmission path, if the state of the one transmission path is poor, the data amount of the video information is reduced and transmitted, thereby limiting the video display on the video display device; Virtual image display device.

5. 2. The virtual image display device according to claim 1, a transmission path is provided as the information transmission path when the video information is input from the outside, and when the video information is transmitted through the one transmission path, if a state of the one transmission path is abnormal, the transmission of the video information is stopped and the video display on the video display device is stopped. Virtual image display device.

6. A virtual image display device, an image processing unit that processes image information input from an external device; an image display device that displays an image based on the image data processed by the image processing unit and outputs image light corresponding to the image; Equipped with adjusting an information transmission path for transmitting the video information from the outside in response to the display of the video; Virtual image display device.

7. 2. The virtual image display device according to claim 1, when displaying the video information on the video display device, checking the appropriateness of the video information, and if the result of the checking is that the video information is inappropriate, disabling the video display of the video information on the video display device; Virtual image display device.

8. 8. The virtual image display device according to claim 7, When the display ratio or the display amount of the video information with respect to the display area is equal to or greater than a threshold value, it is determined that the video information is not displayable. Virtual image display device.

9. 8. The virtual image display device according to claim 7, When the luminance of the video information in the display area is abnormal with respect to a reference luminance, it is determined that the video information is not displayable. Virtual image display device.

10. 8. The virtual image display device according to claim 7, If there is an error in the image information for the display area, or if there is a discrepancy between the actual situation and the image information, it is determined that the image cannot be displayed. Virtual image display device.

11. 2. The virtual image display device according to claim 1, The virtual image display device is connected to a controller of an in-vehicle system via communication, and the external device is the controller. Virtual image display device.

Images