Vehicle-mounted system and virtual image display device

Abstract

Provided is a more suitable display technology for a display device, the display technology being capable of handling the case where an image generated on the outside is received and displayed by the display device as an image input. The present invention contributes to "3. Good-Health and Well-Being" of Sustainable Development Goals. A vehicle-mounted system 300 is provided with a controller 100 and a virtual image display device 1. The virtual display device 1 displays an image as a virtual image on the basis of image information from a portable terminal 400 as the outside.

Description

In-vehicle system and virtual image display device

[0001] The present disclosure relates to technologies for an in-vehicle system and a virtual image display device.

[0002] In the past, images were often generated and displayed inside a virtual image display device, but in the future, it is expected that images will be generated outside the virtual image display device, and the virtual image display device will receive and display the generated images as video input. An example of an external image generator is a controller for an in-vehicle system.

[0003] An example of the prior art is Japanese Patent Laid-Open No. 2010-70066 (Patent Document 1), which describes an example of a head-up display (HUD) having a display information generating means that generates projection information.

[0004] JP 2010-70066 A

[0005] An example of an external image generator (i.e., an image provider) is a mobile terminal (i.e., a mobile information terminal device) such as a smartphone owned by a user. The mobile terminal of the image provider is connected to the in-vehicle system for communication, and data and information such as video content are provided from the mobile terminal of the image provider to the virtual image display device, which displays the video as a virtual image. In this case, it is necessary to ensure that the virtual image display is good and does not interfere with safe driving.

[0006] An object of the present disclosure is to provide a more suitable display device technology that can accommodate cases where an image is generated externally and the display device receives and displays the image as a video input.

[0007] In order to solve the above problems, for example, the configuration described in the claims is adopted. The present application includes multiple means for solving the above problems, and an example thereof is as follows: An in-vehicle system includes a controller and a virtual image display device communicatively connected to the controller, the virtual image display device including an image processing unit that processes image information input from an external device, and an image display device that displays an image based on the image information processed by the image processing unit and outputs image light corresponding to the image, the virtual image display device displays an image as a virtual image based on image information from a mobile terminal as the external device.

[0008] According to a representative embodiment of the present disclosure, it is possible to provide a more suitable display device technology that can handle cases where an image is generated externally and the display device receives and displays that image as a video input. Problems, configurations, effects, etc. other than those described above are described in the description of the embodiment.

[0009] 1 is a diagram showing a first example of a configuration in which a display device according to a first embodiment is mounted on a vehicle. FIG. 2 is a diagram showing a second example of a configuration in which a display device according to a first embodiment is mounted on a vehicle. FIG. 3 is a diagram showing an example of a configuration of an in-vehicle system including a display device according to a first embodiment. FIG. 4 is a diagram showing an example of a configuration of an information transmission path between a display device according to a first embodiment and a vehicle controller. FIG. 5 is a diagram showing an example of a functional block configuration of a vehicle controller according to a first embodiment. FIG. 6 is a diagram showing a first example of a functional block configuration of a display device according to a first embodiment. FIG. 7 is a diagram showing a second example of a functional block configuration of a display device according to a first embodiment. FIG. 8 is a diagram showing a third example of a functional block configuration of a display device according to a first embodiment. FIG. 9 is a diagram showing an overview of main functions in the first and second embodiments. FIG. 10 is a diagram showing a case in which one information transmission path is normal according to a first embodiment. FIG. 11 is a diagram showing an example in which nothing is displayed in the display area of ​​the display device according to a first embodiment. FIG. 12 is a diagram showing an example in which an image is normally displayed in the display area of ​​the display device according to a first embodiment. FIG. 13 is a diagram showing a case in which one information transmission path is abnormal according to a first embodiment. FIG. 14 is a diagram showing a case in which one information transmission path is malfunctioning according to a first embodiment. FIG. 15 is a diagram showing switching between two information transmission paths according to a first embodiment. 1 is a diagram showing the combined use of two information transmission paths in the first embodiment; FIG. 2 is a diagram showing information restriction in the case of a malfunction of one information transmission path in the first embodiment; FIG. 3 is a diagram showing an example of display in the display area when information is restricted in the first embodiment; FIG. 4 is a diagram showing an example of display of the control state in the display area when information is restricted in the first embodiment; FIG. 5 is a diagram showing an example of display of the control state in the display area when display is stopped in the first embodiment; FIG. 6 is a diagram showing an example of display of the control state in the display area when transmission paths are switched in the first embodiment; FIG. 7 is a diagram showing an example of display of the control state in the display area when transmission paths are used in combination in the first embodiment; FIG. 8 is a diagram showing an example of display of the control state in the display area when transmission paths are switched in the first embodiment; FIG. 9 is a diagram showing an example of display of the status of the transmission paths in the display area in the first embodiment; FIG. 10 is a diagram showing an example of display of the status of the transmission paths in the display area in the first embodiment; 1 is a diagram showing detection of a state of a transmission path between a controller and a device in embodiment 1. FIG. 2 is a diagram showing a first example of a processing flow of an in-vehicle system in embodiment 1. FIG. 3 is a diagram showing a second example of a processing flow of an in-vehicle system in embodiment 1.1 is a diagram showing a third example of the processing flow of the in-vehicle system in the first embodiment. FIG. 2 is a diagram showing an application example of a combination of multiple controls in the first embodiment. FIG. 3 is a diagram showing a case of appropriate display in confirmation of the appropriateness of display in the second embodiment. FIG. 4 is a diagram showing a case of inappropriate display in confirmation of the appropriateness of display in the second embodiment. FIG. 5 is a diagram showing an example of inappropriate display in the display area of ​​the display device in the second embodiment. FIG. 6 is a diagram showing an example of display in the display area when control is exercised to disable / limit display based on confirmation of the appropriateness of display in the second embodiment. FIG. 7 is a diagram showing the processing flow of the in-vehicle system related to confirmation of the appropriateness of display in the second embodiment. FIG. 8 is a diagram showing details of control related to confirmation of the appropriateness of display in the second embodiment. FIG. 9 is an explanatory diagram related to the display ratio in the display area in the second embodiment. FIG. 10 is another explanatory diagram related to the display ratio in the display area in the second embodiment. FIG. 11 is an explanatory diagram related to an abnormal brightness in the display area in the second embodiment. FIG. 12 is an explanatory diagram related to insufficient information in the display area in the second embodiment. FIG. 13 is a diagram showing an example of a display state of the control in the display area when control is exercised to disable / limit display based on confirmation of the appropriateness of display in the second embodiment. 10 is a diagram showing another example of the display of the control state in the display area when display is disabled / restricted based on confirmation of the appropriateness of the display in embodiment 2. FIG. 11 is a diagram showing an example of the configuration of an in-vehicle system of a vehicle in embodiment 3. FIG. 12 is a diagram showing an example of mounting a mirror-type display device in a vehicle in embodiment 3. FIG. 13 is a diagram showing an example of the configuration (1) of the display area by a mirror-type display device in a vehicle in embodiment 3. FIG. 14 is a diagram showing an example of the configuration (2) of the display area by a mirror-type display device in a vehicle in embodiment 3. FIG. 15 is a diagram showing an example of the configuration (3) of the display area by a mirror-type display device in a vehicle in embodiment 3. FIG. 16 is a diagram showing an example of the functional block configuration of a controller in embodiment 3. FIG. 17 is a diagram showing an example of the functional block configuration of a virtual image display device in embodiment 3. FIG. 18 is a diagram showing an example of the communication connection (1) between an in-vehicle system and a mobile terminal in embodiment 3. FIG. 19 is a diagram showing an example of the communication connection (2) between an in-vehicle system and a mobile terminal in embodiment 3. FIG. 19 is a diagram showing an example of the communication connection (3) between an in-vehicle system and a mobile terminal in embodiment 3. FIG. 19 is a diagram showing an example of the state of a mobile terminal in embodiment 3. 10 is a diagram showing an example (2) of the state of a mobile terminal in embodiment 3. FIG. 11 is a diagram showing a configuration example (1) in the case where two virtual image display devices are provided in embodiment 3. FIG. 12 is a diagram showing a configuration example (2) in the case where two virtual image display devices are provided in embodiment 3.10 is a diagram showing an example (1) of a case where video transmission from a video provider and video display are normal in the third embodiment. FIG. 11 is a diagram showing an example (2) of a case where video transmission from a video provider and video display are normal in the third embodiment. FIG. 12 is a diagram showing an example of a case where video transmission from a video provider and video display are malfunctioning or an error occurs in the third embodiment. FIG. 13 is a diagram showing an example of a case where a mobile terminal acquires driving assistance information from a controller as a modified example in the third embodiment. FIG. 14 is a diagram showing an example of a desirable state of an in-vehicle system and a mobile terminal in the third embodiment. FIG. 15 is a diagram showing an example of an undesirable state of an in-vehicle system and a mobile terminal in the third embodiment. FIG. 16 is a diagram showing an example (1) of a display of a notification of the status of a mobile terminal on a virtual image display device in the third embodiment. FIG. 17 is a diagram showing an example (2) of a display of a notification of the status of a mobile terminal on a virtual image display device in the third embodiment. FIG. 18 is a diagram showing an example of a controller grasping the battery status of a mobile terminal in the third embodiment. FIG. 19 is a diagram showing an example of a virtual image display grasping the battery status of a mobile terminal in the third embodiment. FIG. 19 is a diagram showing control of switching of a video provider using priority in the third embodiment. 10 is a diagram showing a detailed case of a mobile terminal that is a candidate for switching the video provider in embodiment 3. FIG. 11 is a diagram showing a method for selecting one device from a plurality of candidate mobile terminals in embodiment 3. FIG. 12 is a diagram showing conditions and triggers for switching the video provider in embodiment 3. FIG. 13 is a diagram showing an example of a GUI screen for selecting a video provider in a virtual image display device in embodiment 3. FIG. 14 is a diagram showing an example of a GUI screen for selecting a video provider in a mobile terminal in embodiment 3. FIG. 15 is a diagram showing a configuration example when approval for switching the video provider is given to the mobile terminal before switching in embodiment 3. FIG. 16 is a diagram showing an example of a screen for user settings of a mobile terminal that is a video provider in a virtual image display device in embodiment 3. FIG. 17 is a diagram showing an example of a screen for user settings of the priority of a video provider in a virtual image display device in embodiment 3. FIG. 18 is a diagram showing an example of switching the video provider depending on the state of a communication path with a mobile terminal in embodiment 3. FIG. 19 is a diagram showing an example (1) of switching the video provider depending on the battery state of a mobile terminal in embodiment 3. FIG. 19 is a diagram showing an example (2) of switching the video provider depending on the battery state of a mobile terminal in embodiment 3. 10 is a diagram showing an example of selecting a video provider in accordance with the state of a communication path with a mobile terminal and the battery state of the mobile terminal in embodiment 3. FIG. 11 is a diagram showing example (1) of switching a video provider in accordance with the occurrence of a priority event in the mobile terminal in embodiment 3.FIG. 10 is a diagram showing an example (2) of switching a video provider in response to the occurrence of a priority event on a mobile terminal in the third embodiment. FIG. 11 is a diagram showing an example (3) of switching a video provider in response to the occurrence of a priority event on a mobile terminal in the third embodiment. FIG. 12 is a diagram showing an example of a configuration in which consent to switching of the video provider is obtained from the switching destination mobile terminal in the third embodiment. FIG. 13 is a diagram showing an example of a configuration for switching of the video provider in the case where the controller is the video provider in the third embodiment. FIG. 14 is a diagram showing an example of a configuration for switching of the video provider in the case where the virtual image display device is the video provider in the third embodiment. FIG. 15 is a diagram showing a main control flow related to selection and switching of the video provider by the in-vehicle system in the third embodiment. FIG. 16 is a diagram showing an example (1) of a control sequence related to selection and switching of the video provider in the in-vehicle system in the third embodiment. FIG. 17 is a diagram showing an example (2) of a control sequence related to selection and switching of the video provider in the in-vehicle system in the third embodiment. FIG. 18 is a diagram showing an example of a screen display when a priority event occurs on a mobile terminal in the third embodiment. FIG. 19 is a diagram showing an example of a screen display on the virtual image display device in the case where a priority event occurs on the mobile terminal in the third embodiment. A diagram showing an example of a screen display regarding a notice of switching of the video provider in the virtual image display device in embodiment 3. A diagram showing an example of a screen display regarding completion of switching of the video provider in the virtual image display device in embodiment 3. A diagram showing an example of a screen display regarding selection of a destination of the video provider in the virtual image display device in embodiment 3. A diagram showing an example of a screen display regarding failure of switching of the video provider in the virtual image display device in embodiment 3.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the drawings, the same parts are generally designated by the same reference numerals, and repeated explanations will be omitted. In the drawings, the representation of components may not represent their actual positions, sizes, shapes, ranges, etc., in order to facilitate understanding of the invention.

[0011] For the purpose of explanation, when describing processing by a program, the program, functions, processing units, etc. may be described as the main components, but the main hardware components are the processor, or a controller, device, computer, system, etc. that is configured with the processor, etc. The computer executes processing according to the program read into memory using resources such as memory and communication interfaces as appropriate through the processor. This realizes predetermined functions, processing units, etc. The processor is configured, for example, with semiconductor devices such as a CPU / MPU or GPU. Processing is not limited to software program processing, and can also be implemented using dedicated circuits. Dedicated circuits such as FPGAs, ASICs, and CPLDs can be used.

[0012] The program may be pre-installed as data on the target computer, or may be distributed as data from a program source to the target computer. The program source may be a program distribution server on a communication network, or a non-transitory computer-readable storage medium, such as a memory card or disk. The program may be composed of multiple modules. The computer system may be composed of multiple devices. The computer system may be composed of a client-server system, a cloud computing system, an IoT system, etc. Various data and information may be composed of structures such as tables and lists, for example, but are not limited to these. Expressions such as identification information, identifiers, IDs, names, and numbers are interchangeable.

[0013] [Solutions, etc.] To address the above-described problems, the embodiments have the following measures, solutions, etc. Fig. 9 is a table summarizing the solutions, and shows an overview of the main functions in the first and second embodiments.

[0014] Solution 1: Display control of a display device according to the state of an information transmission path of an external video input.

[0015] Solution 1-1. Switching of information transmission paths: The information transmission path for external video input between the outside and the display device, in other words, the communication path, is configured to be redundant by providing two or more paths. Then, the transmission path used for transmitting video data of the external video input is controlled so that, for example, a first transmission path under normal circumstances is switched to a second transmission path when an abnormality or malfunction occurs in the first transmission path. For example, a system including a display device switches to the second transmission path to transmit video data when the first transmission path under normal circumstances malfunctions and there is a large delay, etc. A user viewing the display device's display does not mind the switching of the transmission path as long as the same display is obtained.

[0016] Solution 1-2: Use of multiple transmission paths in combination: In a redundant configuration of information transmission paths, for example, when a first transmission path that is normally used is malfunctioning, control is performed so that video data is transmitted using both the second transmission path as the main path and the first transmission path as the sub-path.

[0017] Solution 1-3. Display restriction on display devices by reducing the amount of transmitted data: Control of display restriction on display devices, which reduces and suppresses the amount of video data transmitted on information transmission paths, thereby reducing and suppressing the amount of display on the display device. This reduces the load related to the transmission of external video input, ensuring real-time display on the display device and reducing delays.

[0018] Solution 1-4. Stopping display on display device by stopping transmission: When lag or misalignment occurs due to delays in the transmission of video data from external video input on the information transmission path, control is performed to stop / pause the display on the display device by stopping the transmission of external video output-input. As a last resort, the video output itself is stopped in this way rather than allowing display lag to occur.

[0019] However, if this stopping / pausing results in a state in which the user cannot be provided with information necessary for driving, it increases the risk, and therefore separate considerations and measures are required. For example, if there is no information providing means other than the display device, such as a meter cluster (in other words, instruments on the instrument panel) that provides information such as vehicle speed, stopping the display device's display will result in a state in which information such as vehicle speed cannot be provided.

[0020] In the embodiment, an automobile has a meter cluster on an instrument panel, and information such as vehicle speed is displayed on the meter cluster. That is, there are information providing means other than a display device. When vehicle speed information is displayed in the display area of ​​the display device, if there is a transmission delay or the like, the display of the vehicle speed information on the display device may be stopped / paused. In this case, at the very least, the vehicle speed information is displayed on the meter cluster, so safe driving is not hindered. Some vehicles may not have a meter cluster on the instrument panel, and a display interface for vehicle speed information, etc., may be integrated into the display device. In this case, stopping / pausing the display of vehicle speed information on the display device will prevent the driver from being provided with information necessary for driving, thereby hindering safe driving. Therefore, in this case, it is desirable to not apply solutions 1-4, but to apply other solutions.

[0021] Solution 2: Display control of a display device according to the content of video information of an external video input. In other words, display control of a display device according to the appropriateness and validity of the video displayed on the display device when assuming that the display on the display device will be based on video information of an external video input.

[0022] Solution 2-1. Confirmation of the appropriateness of the video display content: When displaying on a display device based on external video input, the appropriateness and validity of the video display content is confirmed and checked, and the video is output only if there are no problems, and is not output if there are problems. The system including the display device checks whether the video display content that the display device is about to display is video that could lead to interference with safe driving, video that could reduce driving safety, etc. Such video may be described as inappropriate video, etc. In other words, the system checks whether the video content that the display device is about to display is appropriate video that can ensure driving safety.

[0023] Examples of such inappropriate images include completely blacked-out images, images that block a large part of the field of view, images that are too bright, images that blend in with or are lost in the surrounding brightness, making necessary information difficult to see, and images that contain information that differs from the actual situation or is incorrect.

[0024] A system including a display device determines the appropriateness of the image to be displayed on the display device as described above, even if there is no problem with the transmission and communication of the image data. The system determines whether the image to be displayed corresponds to the above-mentioned example of inappropriate image based on predetermined rules and conditions. For example, the entire display area of ​​the display device, which is the range in which the display device can display a virtual image, is set to 100. If the ratio of the total area occupied by the displayed image to the entire area is equal to or greater than a threshold, the system determines that the displayed image interferes with safe driving and is inappropriate.

[0025] Furthermore, for example, the system determines whether the image to be displayed corresponds to the above-mentioned full-screen image, image that blocks most of the field of view, etc., based on the area and brightness of the area occupied by the image to be displayed. The system also compares the vehicle surroundings, for example, a brightness reference value, with the brightness of the image to be displayed, etc., to determine whether the brightness of the image to be displayed is abnormal.

[0026] Also, for example, if the system has an information source other than the external device that provides the external video, such as an in-vehicle controller, the system may compare the information from the external video input with information from the other source to determine whether the information from the external video input contains any errors, differs from the actual situation, or contains any discrepancies.

[0027] The embodiment of the present invention will be described using an example in which a HUD device, which is a virtual image display device, is mounted on a vehicle. The system including the HUD device of the embodiment has functions corresponding to the above-mentioned solution, thereby realizing that the HUD display based on external video input does not display anything that hinders safe driving.

[0028] 1 to 24, a HUD device and the like according to a first embodiment of the present disclosure will be described. The first embodiment has a function corresponding to the first solution.

[0029] [HUD Device] Fig. 1 shows a configuration example of a HUD device 1 that is a virtual image display device according to a first embodiment. In the configuration example of Fig. 1, the HUD device 1 is stored in a dashboard 2A in a vehicle 2. In other words, the HUD device 1 is mounted so as to be embedded in the dashboard 2A. The dashboard 2A also includes an instrument panel 2C including a meter cluster, for example, in front of a steering wheel 2B. The vehicle 2 is typically an automobile, but is not limited thereto and may also be a railroad vehicle, an airplane, or the like.

[0030] Image light 1a emitted from HUD device 1 passes through an opening 7 in dashboard 2A, is irradiated onto the inner surface of windshield 3, and is reflected therefrom. The reflected image light 1b is incident on a driver's viewpoint 6. When viewed from the driver's viewpoint 6, a virtual image 9 corresponding to image light 1b is formed and visible beyond a HUD display area 5 corresponding to the irradiation area of ​​image light 1a. In the present invention, projection of image light onto windshield 3 is described, but image light may also be projected onto a projection member such as a combiner.

[0031] FIG. 2 shows another example configuration of the HUD device 1, which is the virtual image display device of the first embodiment. In the example configuration of FIG. 2 , the HUD device 1 is installed outside the dashboard 2A in a vehicle 2. Image light 1a emitted from the HUD device 1 is irradiated onto the inner surface of the windshield 3 and reflected therefrom. The reflected image light 1b is incident on the driver's viewpoint 6. When viewed from the driver's viewpoint 6, a virtual image 9 corresponding to the image light 1b is formed and visible beyond the HUD display area 5 corresponding to the irradiation area of ​​the image light 1a. As shown in FIG. 2 , the HUD device 1 is an integrated device, but it need not be an integrated device. For example, the light source device may be mounted on the roof of the vehicle 2 and the image light projection unit may be disposed on the dashboard 2A.

[0032] [In-Vehicle System] Fig. 3 shows a configuration example of an in-vehicle system 300 that is a system including the HUD device 1. The in-vehicle system 300 is provided in a vehicle 2. The in-vehicle system 300 includes a controller 100 and the HUD device 1. The controller 100 and the HUD device 1 are connected via a predetermined information transmission path 200, in other words, a communication path 200, and are capable of communicating with each other. The controller 100 is a control device / control unit also called an ECU (Electronic Control Unit), for example, and controls the entire in-vehicle system 300.

[0033] The controller 100 acquires / generates vehicle information 4 from various sensors ( FIG. 5 ) installed in the vehicle 2. The various sensors detect various events that occur in the vehicle 2 and detect various parameter values ​​related to the driving conditions of the vehicle 2. The controller 100 uses the vehicle information 4 to control the driving of the vehicle 2. The controller 100 may provide the vehicle information 4 to the HUD device 1, for example, by communication via a CAN.

[0034] In this embodiment, the controller 100 is a provider of external video information and the like, in other words, a video source, for the HUD device 1, and transmits video data and the like corresponding to external video input to the HUD device 1 via an information transmission path 200, and controls video output using the HUD device 1.

[0035] 3, the in-vehicle system 100 also includes a car navigation system 150 (abbreviated as "car navigation"), which is connected to the controller 100 via communication. The HUD device 1 is connected to the car navigation system 150 via the controller 100, and is therefore also able to obtain information from the car navigation system 150. The case in which the HUD device 1 obtains external video input video data and other information from the controller 100 via the information transmission path 200 is indicated by a dashed arrow as information input 2E. The case in which the HUD device 1 obtains information from the car navigation system 150 via the controller 100 and the information transmission path 200 is indicated by a dashed arrow as information input 2F. Note that the car navigation system 150 may be integrated into the controller 100.

[0036] 3, the information transmission path 200 is illustrated as a single line, and is a single path. In the example of Fig. 3, all information including video data of external video input is transmitted via this single path.

[0037] The well-known FPD-Link III is a typical connection form for the information transmission path 200. FPD-Link III is an in-vehicle video communication interface standard, and is characterized by the fact that only one cable is required for connection between devices.

[0038] The controller 100 may be connected to another device such as a car navigation system 150. For example, the car navigation system 150 may provide map information or navigation information to the controller 100, and the controller 100 may generate video information based on the provided information. The controller 100 then transmits the video information to the HUD device 1 via the information transmission path 200, and the HUD device 1 displays a virtual image based on the video information. The HUD device 1 may also acquire information from the car navigation system 150 via the controller 100 and the information transmission path 200.

[0039] Although the controller 100 and the car navigation system 150 are shown as separate components in FIG. 3, the car navigation system 150 may be integrally included in the controller 100 as a function.

[0040] FIG. 4 shows an example of a dual-path configuration for the information transmission path 200 between the controller 100 and the HUD device 1. In the example of FIG. 4, as shown in (A), the information transmission path 200 is composed of two information transmission paths, a first information transmission path 201 and a second information transmission path 202, which are illustrated by two lines, forming a dual-path configuration. (B) shows an example of normal use, in which only the first information transmission path 201 is used for video data transmission. (C) shows an example of use during switching, in which only the second information transmission path 202 is used for video data transmission. The path used for video data transmission is illustrated by a solid line, and the path not used for video data transmission is illustrated by a dashed line. In other use examples, both the first information transmission path 201 and the second information transmission path 202 can be used for video data transmission simultaneously or in parallel.

[0041] 3 and 4 are paths capable of bidirectional communication, but in this embodiment, they are mainly used for external video input, and therefore are mainly used for video data transmission in the downlink direction from the controller 100 to the HUD device 1. In addition to video data transmission, the information transmission path 200 is also capable of control communication and the like. This is possible without being limited to the configuration examples of the information transmission path 200 in FIGS. 3 and 4.

[0042] As will be described later, the HUD device 1 and the controller 100 each have a communication unit mounted thereon that corresponds to the communication interface of the information transmission path 200. In this embodiment, the information transmission path 200 is configured as a wired connection using FPD-Link III, but is not limited to this and may be configured as a wireless connection or a combination of wired and wireless connections.

[0043] In the case of the duplicated information transmission path 200 shown in Figure 4, as an example of implementation, two FPD-Link III systems may be provided, or different connection configurations may be combined. For example, a combination of FPD-Link III and CAN may be used, or a combination of FPD-Link III and Ethernet (registered trademark) may be used. A triplex or more configuration may also be used. Also, a configuration in which the path used for video data transmission and the path used for control communication are separate may be used. For example, a configuration in which one path is used only for control communication, another path is used only for video data transmission, and another path can be used for both control communication and video data transmission.

[0044] In one example of this embodiment, the first transmission path 201 and the second transmission path 202, which are dual information transmission paths, are configured such that, for example, the first transmission path 201 is configured with an FPD-Link III and the second transmission path 202 is configured with a CAN. The first transmission path 201 is configured with a wired cable using FPD-Link III. The CAN of the second transmission path 202 is configured with a wireless path.

[0045] When there are multiple information transmission paths, such as redundant paths, only some of the transmission paths may be used under normal circumstances, and other transmission paths may be used in the event of an abnormality. Alternatively, multiple transmission paths may be used simultaneously in parallel. For example, a relatively large amount of video data may be transmitted over the first transmission path 201, and a relatively small amount of other information, such as vehicle information 4 or control information, may be transmitted over the second transmission path 202.

[0046] 3 , the controller 100 may display the vehicle information 4 and the like as a virtual image 9, which is an image on the HUD device 1. In this case, the image data / image information transmitted from the controller 100 to the HUD device 1 via the information transmission path 200 includes the vehicle information 4 or vehicle information image data that visualizes the vehicle information 4. The vehicle information 4 acquired / generated by the controller 100 / in-vehicle system 300 may include, for example, the following information:

[0047] The vehicle information 4 includes, for example, speed information (also referred to as vehicle speed) of the vehicle 2, gear information, steering wheel steering angle information, lamp illumination information, external light information, distance information, infrared information, engine ON / OFF information, camera image information, acceleration gyro information, GPS (Global Positioning System) information, navigation information, vehicle-to-vehicle communication information, road-to-vehicle communication information, input information from the driver, and the like.

[0048] The driver's input information is any information input by the driver through an operation input device. In the in-vehicle system 300 of this embodiment, the means for inputting an operation to the HUD device 1 by the driver as a user is provided on the controller 100 side, not on the HUD device 1 side. Therefore, the driver's input information is one of the vehicle information 4 generated by the controller 100. An operation input device provided in the vehicle 2 or the controller 100, for example, an operation input device such as a button installed near the steering wheel 2B, such as the vehicle operation switch 511 (steering switch, etc.) in FIG. 5, accepts an operation by the driver and generates the driver's input information. The driver's input information may be an instruction to the controller 100 or an instruction to the HUD device 1.

[0049] [Video Source] In the first embodiment, the video to be displayed on the HUD device 1 (in other words, the video to be displayed, video information, and video data) is generated by the controller 100. In the present embodiment, the video source and the external video input source are the controller 100, and the video destination and the external video input destination are the HUD device 1.

[0050] In the first embodiment, the only image provider for the display target image, which is the image to be displayed on the HUD device 1, is the controller 100. The controller 100 generates image data for the display target image and transmits it to the HUD device 1 via an information transmission path 200. The HUD device 1 receives and inputs the image data as an external image input, and displays an image based on the image data. The HUD device 1 displays the image as a virtual image 9 in the HUD display area 5 of FIG. 1 .

[0051] [Functional Blocks Related to Controller] Fig. 5 shows an example of a functional block configuration related to the controller 100. In Fig. 5, examples of various sensors and devices installed in the vehicle 2 or the controller 100 include the following: a vehicle speed sensor 501, a shift position sensor 502, a steering wheel angle sensor 503, a headlight sensor 504, an illuminance sensor 505, a chromaticity sensor 506, a distance measurement sensor 507, an infrared sensor 508, an engine start sensor 509, an acceleration sensor 512, a gyro sensor 513, a temperature sensor 514, a wireless transceiver for road-to-vehicle communication 515, a wireless transceiver for vehicle-to-vehicle communication 516, a GPS receiver 518, a VICS (Vehicle Information and Communication System, registered trademark) receiver 519, a camera (in-vehicle camera) 531, a camera (out-vehicle camera) 532, an audio input device 541 (including a microphone, etc.), an audio output device 542 (including a speaker, etc.), an image generation unit 510, and a communication unit 520. The sensors and devices are not limited to these, and can be added, deleted, replaced, etc.

[0052] The vehicle speed sensor 501 detects the speed of the vehicle 2 (also referred to as vehicle speed) and generates speed information as the detection result. The shift position sensor 502 detects the current gear and generates gear information as the detection result. The steering wheel steering angle sensor 503 detects the current steering wheel steering angle and generates steering wheel steering angle information as the detection result. The headlight sensor 504 detects whether the headlights are on or off and generates lamp illumination information as the detection result. The illuminance sensor 505 and chromaticity sensor 506 detect external light and generate external light information as the detection result.

[0053] The distance sensor 507 detects the distance between the vehicle 2 and an external object and generates distance information representing the detection result. The infrared sensor 508 detects the presence and distance of an object in the vicinity of the vehicle 2 and generates infrared information representing the detection result. The engine start sensor 509 detects whether the engine is on or off and generates on or off information representing the detection result. The vehicle operation switch 511 is, for example, a steering switch provided on the steering wheel 2B, and receives operation by the driver and generates input information from the driver. The acceleration sensor 512 and gyro sensor 513 detect the acceleration and angular velocity of the vehicle 2 and generate acceleration gyro information representing the attitude and behavior of the vehicle 2 as the detection result. The temperature sensor 514 detects the temperature inside and outside the vehicle 2 and generates temperature information representing the detection result.

[0054] The in-vehicle camera 531 captures images of the interior of the vehicle 2 to generate in-vehicle camera video information. The exterior camera 532 captures images of the exterior of the vehicle 2 to generate exterior camera video information. In a specific example, the in-vehicle camera 531 captures the driver's posture, eye position, and movement, constituting a DMS (Driver Monitoring System). The exterior camera 532 captures images of the surrounding conditions, such as the area ahead of the vehicle 2. By analyzing the exterior camera video information, it is possible to determine the presence or absence of other vehicles or people around the vehicle 2, buildings and terrain, road conditions such as rain, snow, ice, and unevenness, and road signs. The exterior camera 532 also includes a drive recorder that records the driving conditions on video.

[0055] The road-to-vehicle communication wireless transceiver 515 generates road-to-vehicle communication information through road-to-vehicle communication between the vehicle 2 and roads, signs, traffic lights, etc. The vehicle-to-vehicle communication wireless transceiver 516 generates vehicle-to-vehicle communication information through vehicle-to-vehicle communication between the vehicle 2 and other vehicles in the vicinity. The GPS receiver 518 generates GPS information by receiving GPS signals from GPS satellites. The GPS information includes information such as the current time, latitude, and longitude. The VICS receiver 519 generates VICS information by receiving VICS signals. The GPS receiver 518 and the VICS receiver 519 may be provided as part of a navigation system.

[0056] The image generating unit 510 is a part that generates image information to be displayed on the HUD device 1. The image generating unit 510 may be implemented inside the controller 100.

[0057] The communication unit 520 is a device or circuit that implements a communication interface corresponding to the information transmission path 200. In this example, the communication unit 520 corresponds to the dual configuration of the information transmission path 200 in FIG. 4 and is implemented with at least CAN and FPD-Link III communication interfaces. The communication unit 520 includes a first communication unit 521 and a second communication unit 522. The first communication unit 521 implements FPD-Link III as a first communication interface, and the second communication unit 522 implements CAN (Controller Area Network) as a second communication interface. Although the communication unit 520 is illustrated as a single block, each communication interface may be provided as a separate block. However, the communication unit 520 may also be implemented with other communication interfaces. In another example, the first communication interface of the first communication unit 521 may be a Gigabit Multimedia Serial Link (GMSL) or the like, and the second communication interface of the second communication unit 522 may be an in-vehicle Ethernet or the like.

[0058] Depending on the installation location, the temperature sensor 514 may be used in the HUD device 1 in the following manner. In a configuration in which the HUD device 1 is housed within a dashboard 2A as shown in FIG. 1 , the temperature sensor 514 is used to detect the temperature near the HUD device 1 within the dashboard 2A. If the temperature becomes abnormally high, the operation of the HUD device 1 may be affected. The controller 100 stops the operation of the HUD device 1 when the temperature becomes abnormally high. In a configuration in which the HUD device 1 is installed on top of the dashboard 2A as shown in FIG. 2 , the temperature sensor 514 is used to detect the temperature near the HUD device 1 outside the dashboard 2A. If the temperature becomes abnormally high due to exposure to direct sunlight, for example, the operation of the HUD device 1 may be affected. The controller 100 stops the operation of the HUD device 1 when the temperature becomes abnormally high.

[0059] In this embodiment, the HUD device 1 does not include an audio input device or an audio output device, and when performing audio input and audio output, an audio input device 541 and an audio output device 542 provided on the controller 100 side are used. However, in this embodiment, the HUD device 1 cooperates with the controller 100, and therefore, under the control of the controller 100, it is possible to use sensors and devices provided on the controller 100 side.

[0060] [Functional Blocks of HUD Device (1)] FIG. 6 is a diagram showing a first example of the functional block configuration of the HUD device 1 when no image is generated inside the HUD, and when an image is generated by the controller 100 outside the HUD.

[0061] 6 includes an MCU 1010, a non-volatile memory 1011, a volatile memory 1012, a communication unit 1015, a light source drive unit 1021, a display driver 1022, a mirror drive unit 1020, an image display device 10, a mirror 22 that functions as an image projection unit, and the like, and these components are interconnected by an architecture such as a bus 1090. On the other hand, in the case of a display device that does not include the mirror drive unit 1020 and the mirror 22 that functions as an image projection unit, the connections of the components and related controls are the same as those described below.

[0062] The image display device 10 includes a light source device 11 and a display panel 12. The light source device 11 generates light source light based on driving from a light source driver 1021 and supplies the light to the display panel 12. The display panel 12 is composed of, for example, a liquid crystal display panel, and forms an image on a display screen based on driving from a display driver 1022. The display panel 12 generates and emits image light obtained by modulating the image based on the light source light. The image light is reflected by a mirror 22. The reflected image light is reflected, for example, by the windshield 3 in FIG. 1 to form a virtual image 9 in the HUD display area 5. The orientation of the mirror 22 can be changed based on driving from a mirror driver 1020. Furthermore, if the HUD device 1 includes an optical element 21, the optical element 21 has the function of controlling light flux and optical correction.

[0063] The MCU 1010 is a unit including a processor and controls the HUD device 1. The non-volatile memory 1011 stores control programs, setting information, and the like. The volatile memory 1012 stores processing data, including display data for HUD display. The HUD device 1 may also include internal or external storage, not shown, in which programs and data may be stored. The MCU 1010 executes processing in accordance with programs read from the non-volatile memory 1011, etc. This constitutes an execution module corresponding to each function.

[0064] The communication unit 1015 includes a video processing unit 1013 and a communication processing unit 1014. The communication interface of the information transmission path 200 in FIG. 4 is implemented in the communication unit 1015. In this embodiment, in correspondence with the redundant configuration of the information transmission path 200 in FIG. 4, at least an FPD-Link III as a first communication interface and a CAN as a second communication interface are implemented. The communication unit 1015 includes a video processing unit 1013 corresponding to the first transmission path 201 of the first communication interface, and a communication processing unit 1014 corresponding to the second transmission path 202 of the second communication interface.

[0065] The video processing unit 1013 receives video data of an external video input from the controller 100, in other words, video information 250 (see FIG. 10 , etc., described below), via the first transmission path 201. The video processing unit 1013 temporarily stores the received video information 250 in, for example, the volatile memory 1012 and performs necessary processing on the video information 250. The volatile memory 1012 has a buffer memory for storing the video information 250 and corresponding display data. Note that the buffer memory may be located within the communication unit 1015 or the video processing unit 1013. The video processing unit 1013 transfers the processed video information 250 stored in the volatile memory 1012, in other words, display data created according to the specifications of the video display device 10, to the display driver 1022. This causes the display driver 1022 to display the video on the display panel 12. The display driver 1022 drives the display panel 12 based on the display data read from the volatile memory 1012, thereby displaying an image on the display panel 12. The image processing unit 1013 may also have a function of generating image data.

[0066] In addition to receiving the video information 250, the video processing unit 1013 may also perform video data conversion processing. The video data conversion processing includes converting the video information 250 in a format handled by the controller 100 into data and information in a format handled by the video display device 10. The type of video data conversion processing required depends on the format of the video information 250 handled by the controller 100 and the format handled by the video display device 10. Generally, the video data conversion processing includes, for example, video decoding, resolution conversion, and distortion correction to ensure that the image is displayed without distortion when projected onto the windshield 3. Depending on the video information 250 received by the HUD device 1, conversion of resolution, size, brightness, and the like may be required. The video data conversion processing includes such conversions. Depending on the video information 250, corrections such as distortion correction and position correction may be required to ensure that the image is displayed as a virtual image 9 in the HUD display area 5 without distortion or misalignment. The video data conversion processing includes such corrections. The video data conversion process can also be performed by the controller 100 depending on the system configuration.

[0067] The communication processing unit 1014 receives the vehicle information 4, control information, and other information from the controller 100 via the second transmission path 202 corresponding to the second communication interface. The communication processing unit 1014 stores the received vehicle information 4, etc., in, for example, the volatile memory 1012, and performs necessary processing. Note that the communication unit 1015 or the communication processing unit 1014 may have a buffer memory for storing the vehicle information 4, etc. When the vehicle information 4 is to be displayed on the HUD, the communication processing unit 1014 transfers the processed vehicle information 4 stored in the volatile memory 1012 to the display driver 1022. This causes the display driver 1022 to display the vehicle information 4 as an image on the display panel 12.

[0068] 6 shows a case in which the information to be displayed on the HUD, i.e., the video information 250, the vehicle information 4, and the control information, are separated and transmitted and processed via different information transmission paths 200. However, this is not limiting. For example, the vehicle information 4 may also be transmitted and processed as part of the video information 250 via the same information transmission path 200, e.g., the first transmission path 201.

[0069] Although not shown, the HUD device 1 may include a sensor such as a sunshine sensor. If external light, such as sunlight, enters the HUD device 1 and irradiates the display panel 12, such as a liquid crystal display panel, it may burn, deteriorate, or break the panel, which is undesirable. To prevent this, a sunshine sensor or the like may be provided instead of the vehicle-side illuminance sensor 505 shown in FIG. 5 . The HUD device 1 performs operations to protect the panel in response to the detection value of the sunshine sensor. For example, when the HUD device 1 is not in use, the mirror driver 1020 rotates the mirror 22 to a predetermined position to prevent external light from irradiating the panel or the like within the housing. When the HUD device 1 is in use, the mirror driver 1020 rotates the mirror 22 to a predetermined position so that image light is irradiated onto the windshield 3 corresponding to the HUD display area 5. Furthermore, when the HUD device 1 determines that the panel needs to be protected based on the detection value of the sunlight sensor, it rotates the mirror 22 to a predetermined position via the mirror driving unit 1020 to prevent external light from irradiating the panel inside the housing.

[0070] 6 does not have a dedicated operation input device for the HUD device 1, but the driver can input input information from operation input devices such as the vehicle operation switch 511 on the vehicle 2 and controller 100 side. Then, the input information can be provided to the HUD device 1 via the controller 100. In a modified example, the HUD device 1 may be provided with a dedicated operation input device.

[0071] [Functional Block (2) of HUD Device] Figure 7 is a configuration diagram showing a second example of a functional block configuration diagram of the HUD device 1, in which an image is generated inside the HUD. In this configuration, the controller 100 outside the HUD does not generate an image. The configuration of Figure 7 differs from the configuration of Figure 6 mainly in that the communication unit 1015 does not include an image processing unit 1013, but instead includes an image processing unit 1016 connected to the bus 1090. The communication processing unit 1014 in the communication unit 1015 performs communication processing corresponding to the communication interface of the information transmission path 200, and receives vehicle information 4, etc.

[0072] The video processing unit 1016 generates video information 250 to be displayed based on control of the MCU 1010, instructions from the controller 100, vehicle information 4, etc. The video information 250 here corresponds to display data for display on the video display device 10. The video processing unit 1016 stores the video information 250, in other words, the display data, in, for example, the volatile memory 1012. The video processing unit 1016 transfers the video information 250, in other words, the display data, from the volatile memory 1012 to the display driver 1022. This causes the display driver 1022 to display the video on the display panel 12. The video processing unit 1016 may also include a memory for storing the video information 250.

[0073] In the configuration example of FIG. 7, the information of the external video input from the controller 100 is limited to the vehicle information 4 and the like.

[0074] [Functional Block (3) of HUD Device] FIG. 8 is a third example of a functional block diagram of the HUD device 1, integrating the configuration and functions of FIG. 6 and the configuration and functions of FIG. 7 into one. This configuration is compatible with both a case where an image is generated by the controller 100 outside the HUD and inputted for HUD display, and a case where an image is generated internally in the HUD and displayed on the HUD. This configuration includes both an image processing unit 1013 in the communication unit 1015 and a separate image processing unit 1016. When image information 250 generated by the controller 100 is externally input and displayed on the HUD, the image processing unit 1013 performs the same processing as described above. When image information 250 is generated by the HUD device 1 and displayed on the HUD, the image processing unit 1016 performs the same processing as described above. These can be switched between as needed.

[0075] [One Information Transmission Path, Normal Case] ​​Fig. 10 is an explanatory diagram showing a case where one information transmission path 200 such as that shown in Fig. 3 is in a normal state. Fig. 10 shows a flow in which the controller 100 generates video information 250, transmits the generated video information 250 to the HUD device 1 via the one information transmission path 200, the HUD device 1 receives the video information 250, and displays an image as a virtual image 9 in the HUD display area 5 based on the received video information 250.

[0076] The controller 100 and the HUD device 1 confirm and recognize each other's specifications, etc. For example, the controller 100 recognizes specifications such as the size and coordinate system of the HUD display area 5 of the HUD device 1 in advance, and generates the video information 250 based on the recognition.

[0077] 11A is a schematic diagram of an example of an actual view seen from the driver's viewpoint 6 inside the vehicle 2 through the HUD display area 5 of the windshield 3. In Fig. 11A, nothing is displayed in the HUD display area 5. In Fig. 11A, the driver can see the scenery outside the vehicle, and for example, there is a pedestrian 11A1 on the left side of the road.

[0078] 11B is a schematic diagram of information visually recognized by the driver when the HUD device 1 displays a virtual image 9 in the HUD display area 5 on the actual scene of FIG. 11A based on the image information 250 of FIG. 10. In this example, the images serving as the virtual image 9 include images 11B1, 11B2, 11B3, and 11B4. Image 11B1 is an image displaying the vehicle speed (a text image indicating "30 km / h") based on the vehicle information 4. Image 11B2 is an image displaying navigation information such as "turn right 100 meters ahead." Image 11B3 is an image displaying navigation information such as "go straight ahead." Image 11B4 is an image displaying a cautionary alert superimposed on the position of pedestrian 11A1.

[0079] As shown in FIG. 10, when one information transmission path 200 is normal, for example, when there is no transmission delay, an appropriate HUD display is provided as shown in FIG. 11B, thereby supporting safe driving.

[0080] [In the Case of an Abnormality in One Information Transmission Path] Figure 12 is an explanatory diagram of a case where one information transmission path 200 is in an abnormal state. The abnormality is also indicated by an x ​​mark. Due to the abnormality in the information transmission path 200, the HUD device 1 cannot receive the video information 250 from the controller 100, and there is no video information to display, so the virtual image 9 cannot be displayed. As shown in Figure 12, when an abnormality occurs in only one information transmission path 200, the HUD device 1 cannot display the video. Since there is no HUD display, the information visually recognized by the driver is the same as, for example, Figure 11A.

[0081] [In the Case of Malfunction of One Information Transmission Path] FIG. 13 is an explanatory diagram illustrating a case where one information transmission path 200 is malfunctioning. This illustrates a case where a transmission delay occurs due to a malfunction of the information transmission path 200. The malfunction here refers to any malfunction, such as a communication malfunction, in which the transmission path is active and data transmission itself is possible, but the transmission speed is slow and the normal transmission speed cannot be ensured. Malfunctions are also indicated by triangles. Although the HUD device 1 can receive video data frames and the like as video information 250 from the controller 100, the reception is delayed, making it impossible to ensure real-time performance of the HUD video display. Since the HUD device 1 cannot ensure real-time performance for video information 250 that requires real-time performance, the HUD device 1 controls the HUD not to display the video information 250 rather than causing a delay in the HUD display. For video information 250 that does not require real-time performance, the HUD device 1 may display the video information 250 as is, even if a delay occurs in the HUD display.

[0082] Furthermore, if a transmission delay occurs in the information transmission path 200, display update stagnation may occur in the HUD display. For example, assume that the HUD device 1 begins to display a virtual image 9 in the HUD display area 5 as shown in FIG. 11B based on the reception of video information 250. Then, assume that the reception of subsequent video information 250 in chronological order is delayed due to a transmission delay. During this time, the HUD display content remains the same as shown in FIG. 11B. The HUD device 1 may determine such a display update stagnation and take appropriate action. For example, the HUD device 1 may temporarily erase the HUD display if it determines that the HUD display content has remained the same without being updated for a predetermined period of time or longer.

[0083] [Two Information Transmission Paths, Dealing with Abnormalities (1)] Fig. 14 is an explanatory diagram showing an example of control for dealing with abnormalities when an abnormality occurs in one of the transmission paths, assuming that there are two information transmission paths 200, a first transmission path 201 and a second transmission path 202, as shown in Fig. 4. This control corresponds to solution 1-1.

[0084] Suppose that initially, the controller 100 was transmitting or attempting to transmit video information 250 to the HUD device 1 using the first transmission path 201, but then the first transmission path 201 went into an abnormal state. An abnormal state here refers to, for example, a state in which even the minimum transmission speed cannot be achieved due to a communication malfunction or error. In this case, the controller 100 and the HUD device 1 switch the path used to transmit the video information 250 to the second transmission path 202, which is normal. The HUD device 1 appropriately displays an image based on the video information 250 transmitted and received via the second transmission path 202. The image displayed on the HUD is, for example, similar to that shown in FIG. 11B .

[0085] In this way, when there are two or more information transmission paths 200, even if one path is abnormal, the transmission of video information and the HUD display can be continued by switching to another normal path. This ensures real-time performance of the HUD display. Note that, if the first transmission path 201 subsequently recovers to a normal state, the HUD device 1 and the controller 100 may switch the path used for transmitting the video information 250 back from the second transmission path 202 to the first transmission path 201.

[0086] [Two Information Transmission Paths, Dealing with Abnormalities (2)] Fig. 15 is an explanatory diagram showing an example of control to deal with a malfunction occurring in one of the transmission paths, assuming there are two information transmission paths 200, similar to Fig. 14. This control corresponds to solution 1-2.

[0087] Suppose that the controller 100 initially transmitted or attempted to transmit the video information 250 to the HUD device 1 using the first transmission path 201, but then the first transmission path 201 malfunctioned. The malfunction here refers to a state in which transmission itself is possible and a minimum transmission speed is achieved, but the transmission speed is low, for example, below a certain threshold. In this case, the controller 100 and the HUD device 1 control the second transmission path 202, which is normal, as the main path for transmitting the video information 250, and the first transmission path 201 as the sub-path, so that these two paths are used in parallel. This control can also be said to change the main path to the second transmission path 202, and is therefore a type of path switching control.

[0088] The controller 100 and the HUD device 1 may transmit the video information 250 at respective data amounts according to the transmission speed of the main path and the transmission speed of the sub-path. In the example of FIG. 15 , the video information 250A is transmitted via the main path, and the video information 250B is transmitted via the sub-path. Alternatively, the controller 100 and the HUD device 1 may transmit the contents of the video information 250 divided into information to be transmitted via the main path and information to be transmitted via the sub-path. For example, among the contents of the video information 250, information with a high priority may be transmitted via the main path, and information with a low priority may be transmitted via the sub-path.

[0089] The HUD device 1 appropriately displays an image based on the image information 250A transmitted and received via the second transmission path 202, which is the main path, and the image information 250B transmitted and received via the first transmission path 201, which is the sub-path. The image displayed on the HUD is similar to that shown in FIG. 11B, for example.

[0090] 16 is an explanatory diagram of an example of information restriction, in other words, HUD display restriction by reducing the amount of transmission data, as a control example for dealing with a malfunction in one information transmission path 200. This control corresponds to solution 1-3.

[0091] In the example of FIG. 16 , assume that one information transmission path 200 is malfunctioning. While transmission itself is possible, the normal transmission speed cannot be ensured, resulting in a transmission delay. In this malfunctioning state, the controller 100 and the HUD device 1 control the information transmitted through the information transmission path 200 and the information displayed on the HUD device 1 to be limited. In this malfunctioning state, the controller 100 adjusts the display content of the image to be displayed, such as by reviewing the display content and reducing the data volume, and transmits only high-priority information. For example, the unreviewed image information 250a is changed to the reviewed image information 250b. The reviewed image information 250b is carefully selected and limited to only vehicle speed image information 1601 (a text image of "30 km / h"), thereby reducing the data volume and information volume. By limiting the content of the image to be displayed to high-priority information, the controller 100 ensures that the data volume of the transmitted image information 250b is below a certain amount according to the state of the transmission path.

[0092] In this example, the case where high priority information is limited to the vehicle speed image information 1601 has been shown, but this is not limiting. The priority of the image information may vary depending on the situation, etc. Priorities may be set in advance according to the type of image, etc. For example, when an image of an alert for a vehicle or pedestrian ahead is generated, the image of the alert may become the information with the highest priority.

[0093] The HUD device 1 cooperates with the controller 100 to impose HUD display restrictions on the reception and display of the video to be displayed. The restrictions on the HUD device 1 correspond to the restrictions on the controller 100. The controller 100 and the HUD device 1 may mutually confirm that there is an abnormality or malfunction in the information transmission path 200, that information restriction is necessary, and the like, before executing such information restriction control.

[0094] The controller 100 transmits the revised video information 250b via the information transmission path 200. While the information restriction is being executed, the HUD device 1 receives the video information 250b via the information transmission path 200. The video processing unit 1013 of the HUD device 1 outputs display data based on the video information 250b to the video display device 10, causing the video to be displayed in the HUD display area 5.

[0095] 17 shows an example of a display during HUD display restriction, in which only the vehicle speed image information 11B1 is displayed. This makes it possible to prevent or reduce lag in the HUD display even if transmission is poor.

[0096] [Control Status Display] With regard to the various control examples (solutions 1-1 to 1-4) described above, the HUD device 1 may display information to inform the user about the control status in the HUD display area 5. A specific example is shown below.

[0097] FIG. 18A shows an example in which an image representing the control status is displayed as a virtual image 9 in the HUD display area 5 when information restriction and HUD display restriction control as shown in FIG. 16 are performed due to a malfunction or transmission delay in the information transmission path 200, and is an example of an image that is visible to the driver.

[0098] The HUD device 1 displays, in the HUD display area 5, image information 1601 ( FIG. 16 ) of the vehicle speed corresponding to the restricted image information 250b. At the same time, the HUD device 1 displays, for example, an on-screen display (OSD) display of an image 1801 ( FIG. 18A ) indicating that information restriction is currently being implemented as the current control status. This image 1801 is not information transmitted from the controller 100, but rather image information displayed with a transmission delay. In this example, the image 1801 indicating "Information Restriction (Transmission Delay)" is displayed by OSD in the upper left corner of the HUD display area 5. The "Transmission Delay" in parentheses indicates the reason for the "information restriction," but may be omitted. The image 1801 is not limited to a text image, but may be a mark, icon, graphic, animation, or other various forms, such as a flashing warning mark.

[0099] By viewing the "information restricted" image 1801, the driver, who is the user, can easily recognize that the HUD device 1 is currently restricting information and HUD display. While displaying this image 1801 is not essential, such display is helpful and preferable because it makes it easier for the user to understand that information is currently restricted. Furthermore, the display of this image 1801 does not need to be constantly displayed while the information restriction is in effect, but may be a temporary display. For example, after the information restriction state is first entered, this image 1801 may be displayed for a certain period of time and then erased. Furthermore, this image 1801 may be displayed periodically thereafter. Furthermore, when the information restriction state improves or returns to a normal state, an image indicating that the information restriction has ended may be displayed.

[0100] Furthermore, the display of the image representing the control status is not limited to an OSD display, and the image information 250 transmitted from the controller 100 to the HUD device 1 may include image information representing the control status, such as "information restricted."

[0101] 18A , the control status may be output not only by HUD display but also by audio output. The HUD device 1 or the controller 100 may output audio 1811, such as "HUD display is being restricted," using a speaker (e.g., audio output device 542 in FIG. 5 ) provided in the HUD device 1 or the controller 100.

[0102] FIG. 18B shows another example of a control status display, in which the HUD display is stopped / paused when an abnormality or malfunction occurs in the information transmission path 200, as shown in FIGS. 12 and 13 . In FIG. 18B , an image 1802 of "Display stopped (transmission abnormality)" or an icon is displayed by OSD, for example, in the upper left corner of the HUD display area 5. The "transmission abnormality" in parentheses indicates the reason for the "display stopped," but may be omitted. Furthermore, the HUD device 1 or the controller 100 may use a speaker (e.g., the audio output device 542 in FIG. 5 ) provided in the HUD device 1 or the controller 100 to output an audio output 1812, such as "HUD display stopped."

[0103] 18C shows another example of the display of the control state when the information transmission path 200 is switched as shown in FIG. 14. In FIG. 18C, an image 1803 of "transmission path 2" is displayed by OSD in, for example, the upper left portion of the HUD display area 5. "Transmission path 2" represents the second transmission path 202 to which the switch has been made.

[0104] Fig. 18D shows another example of the display of the control state when a main path and a sub-path are used together as the information transmission path 200 as in Fig. 15. In Fig. 18D, an image 1804 of "transmission path 1-sub" and "transmission path 2-main" is displayed by OSD in, for example, the upper left part of the HUD display area 5.

[0105] Fig. 18E shows a variation of the display in Fig. 18C as another example of a control state display. An image 1805 in Fig. 18E is an image that uses icons, graphics, or the like to represent the switching and other states of the two information transmission paths 200 (201, 202) between the controller 100 ("CNT") and the HUD device 1 ("display device"). In this example, each transmission path is represented by an arrow, an abnormal state (or non-use) of the first transmission path 201 is represented by an x, and a normal state (or in use) of the second transmission path 202, to which the switch is made, is represented by a circle.

[0106] FIG. 18F shows another example of a control state display, illustrating the status of the information transmission path 200, particularly transmission performance, as shown in FIGS. 10, 12, and 13. Image 1806 in FIG. 18F includes images 1806a and 1806b. Image 1806a represents the connection type of the information transmission path 200, displaying "Wired" for a wired connection such as FPD-Link III. Image 1806b represents the transmission performance of the information transmission path 200, such as transmission smoothness and transmission speed, or in other words, the status of transmission delay. Image 1806b, for example, displays the transmission smoothness as a bar value ranging from 0 to 100. A value of 100 corresponds to the maximum speed being achieved by the communication interface of that transmission path. This display allows the driver to easily recognize the status of transmission performance and other aspects of the HUD display of external video input.

[0107] Fig. 18G shows another example of a display of the control state, which is another display example of the state of the information transmission path 200 such as those shown in Figs. 10, 12, and 13. In Fig. 18G, an image 1807 displays "Wired" and "FPD-Link III" when the information transmission path 200 is a wired connection using, for example, FPD-Link III. In addition, the image 1807 displays the transmission performance / transmission delay state in a pie chart.

[0108] Fig. 18H shows another example of a display of the control state, which is another display example of the state of the information transmission path 200 such as those shown in Figs. 10, 12, and 13. In the image 1808 shown in Fig. 18H, when the information transmission path 200 is a wireless connection, an icon representing the wireless connection is displayed as the base. Furthermore, when the wireless connection is abnormal or malfunctioning, a slash ( / ) representing the abnormality or malfunction is superimposed on the icon. Such an image 1808 is displayed only when the information transmission path 200 is abnormal or malfunctioning.

[0109] Fig. 18I shows another example of a display of the control state when the transmission path being used is switched, as in Fig. 14. For example, assume that the second transmission path 202, which is a transmission path for a wireless connection, is switched to the first transmission path 201, which is a transmission path for a wired connection. Image 1809 in Fig. 18I is displayed for a certain period of time during the switching. Image 1809 in Fig. 18I includes an icon representing a wireless connection and an icon representing a wired connection, and an arrow image represents the switching from the wireless connection to the wired connection.

[0110] The various images representing the control state, the state of the information transmission path 200, and the like may be displayed constantly or for a fixed period of time. The images may be displayed for a fixed period of time only when a change in the situation occurs. Furthermore, the images may be selectively set to be displayed or not. When the images are set to be displayed, the position at which they are displayed within the HUD display area 5 may also be set. Furthermore, the display of the images is not limited to being realized by OSD display, but may also be created by embedding the images as part of the image information 250 of the image to be displayed. Furthermore, the display of the images may be replaced with audio output, or both display and audio may be output.

[0111] The display of images of control states and the like as in the examples of FIGS. 18A and 18B can also be applied to the functions of the second embodiment described later.

[0112] [Modification: Route Information Display] FIG. 19 shows a modification of FIGS. 14 and 18C . FIG. 19 shows a control example of how to handle a situation where an abnormality occurs in the first transmission path 201 of two information transmission paths 200 (201, 202) and switching is made to the normal second transmission path 202. As in FIG. 14 , the controller 100 and the HUD device 1 cooperate to switch from the first transmission path 201 to the second transmission path 202. The controller 100 transmits, along with the transmission of the video information 250, information about the transmission path to be used and information indicating which transmission path is being used to transmit the video information 250 or information indicating the transmission path being used. In this example, the video information 250 includes information indicating the second transmission path 202 as used transmission path information 1901. Such used transmission path information 1901 may be described in the attribute information or metadata of the video information 250. In addition, if the communication units of the controller 100 and the HUD device 1 know which transmission path they are using, the separate transmission of the used transmission path information 1901 may be omitted.

[0113] Based on the used transmission path information 1901 accompanying the video information 250, the HUD device 1 adds and displays an image showing the transmission path status and the used transmission path to the video displayed in the HUD display area 5 by using an OSD display or the used transmission path information 1901 itself. As a result, a display similar to that shown in Fig. 18C can be obtained.

[0114] [Detection of Abnormality or Malfunction in Information Transmission Path] Examples of technical means for determining and detecting an abnormality or malfunction in the information transmission path 200 include the following. Fig. 20 is an explanatory diagram of a configuration for determining and detecting an abnormality or malfunction in the information transmission path 200 in the in-vehicle system 300. In the controller 100, the communication unit 520 described above (Fig. 5) performs communication processing using each communication interface of the information transmission path 200. In the HUD device 1, the communication unit 1015 described above (Fig. 6) performs communication processing using each communication interface of the information transmission path 200.

[0115] First, the first communication unit 521 of the communication unit 520 transmits the video information 250 via the first transmission path 201, and the video processing unit 1013 of the communication unit 1015 receives the video information 250 via the first transmission path 201. For example, each unit sequentially transmits the video information 250 for each video data frame. Suppose the communication conditions on the first transmission path 201 deteriorate. In this case, even if the first communication unit 521 transmits a video data frame, the first communication unit 521 may determine and detect a decrease in the transmission speed, in other words, a transmission delay, or an abnormality / error in the first transmission path 201, based on the fact that the video data frame has not been received correctly from the video processing unit 1013. The first communication unit 521 may inform the video processing unit 1013 of such a condition via control communication.

[0116] Similarly, from the perspective of the HUD device 1, if a video data frame does not arrive from the first communication unit 521 or arrives slowly, the video processing unit 1013 can determine and detect a decrease in the transmission speed, in other words, a transmission delay, or an abnormality / error in the first transmission path 201. The video processing unit 1013 may notify the first communication unit 521 of such a state through control communication.

[0117] The first communication unit 521 on the controller 100 side and the video processing unit 1013 on the HUD device 1 side cooperate to perform control to switch to the second transmission path 202, for example, as shown in Fig. 14 , when detecting a transmission delay or abnormality on the first transmission path 201 as described above. For example, the first communication unit 521 side may transmit a request or instruction to switch to the second transmission path 202 to the video processing unit 1013 side via control communication on the first transmission path 201 or the second transmission path 202. When the second transmission path 202 is available for transmitting the video information 250, the first communication unit 521 and the video processing unit 1013 switch to the second transmission path 202 and perform transmission. Conversely, a request or instruction to switch to the second transmission path 202 may be sent from the video processing unit 1013 to the first communication unit 521 via control communication on the first transmission path 201 or the second transmission path 202.

[0118] As described above, the state of the information transmission path 200 may be determined or detected by either the controller 100 side or the HUD device 1 side, or by both, or one may notify the other of the state as appropriate. The HUD device 1 and the controller 100 mutually determine and confirm the abnormality or malfunction of the information transmission path 200, whether switching or restriction is necessary, and the like, and based on that, they cooperate to execute the various controls described above.

[0119] Furthermore, the video data transmission and the control communication may be transmitted together on one information transmission path 200, or may be transmitted separately on separate information transmission paths 200. Examples of control communication include communication related to determining the state of the transmission path and controlling switching of the transmission path to be used.

[0120] Further, a method may be used in which different types of video content are provided for the video information 250, and each type of video is transmitted via a different information transmission path 200. For example, in the example of Fig. 6, when video related to vehicle information 4 is to be displayed on the HUD, the second transmission path 202 is used. When video information other than vehicle information 4 is to be displayed on the HUD, the first transmission path 201 is used.

[0121] [Processing Flow (1)] FIG. 21 shows a processing flow for controlling switching of the information transmission path 200 in the in-vehicle system 300 according to the first embodiment, as in the example shown in FIG. 14 . In the in-vehicle system 300, the HUD device 1 and the controller 100 cooperate to determine whether to switch the information transmission path 200 used for transmitting the video information 250. As described above, this determination may be made by either the HUD device 1 or the controller 100. Furthermore, the control process including this determination may be performed by the controller 100 acting as the primary and the HUD device 1 acting as the secondary, or by the HUD device 1 acting as the primary and the controller 100 acting as the secondary. The in-vehicle system 300 may determine the status of each of the multiple information transmission paths 200, such as availability, usage status, and transmission performance (e.g., transmission speed) / transmission delay status.

[0122] In step S100, the in-vehicle system 300 starts processing to check the status of the information transmission path 200. When the function related to the first embodiment is in an on state, a processing flow such as that shown in FIG. 21 is executed. In step S101, the in-vehicle system 300 monitors the information transmission status on the information transmission path 200. The monitoring can be performed by the respective communication units of the controller 100 and the HUD device 1, as shown in FIG. 20 . In step S102, the in-vehicle system 300 checks whether the transmission speed of the video information 250 on the information transmission path 200 is equal to or greater than a predetermined speed. The predetermined speed can be set as a threshold value. The predetermined speed is a setting value for determining whether the transmission performance at the transmission speed is sufficient. If the speed is equal to or greater than the predetermined speed (Yes), the process proceeds to step S109. If the speed is not equal to or greater than the predetermined speed (No), the process proceeds to step S103.

[0123] In step S103, the in-vehicle system 300 checks whether there is an available alternative transmission path for the currently used transmission path, in other words, whether there is a candidate for switching. If there is an alternative path (Yes), the process proceeds to step S105. If there is no alternative path (No), the process proceeds to step S104. For example, if the first transmission path 201 is in use and the second transmission path 202 is available, the second transmission path 202 can be used as an alternative path. For example, if the first transmission path 201 is in use and the second transmission path 202 is being used to transmit important information, the second transmission path 202 cannot be used as an alternative path.

[0124] The processing flow of this embodiment selects and determines which of the following measures should be taken based on the situation, such as an abnormality in the information transmission path 200: switching to an alternative path, limiting the HUD display by reducing the amount of transmission data, or stopping the HUD display by stopping transmission. In other words, multiple functions corresponding to the multiple solution means described above are appropriately selected and applied.

[0125] In step S104, since the transmission performance of the currently used transmission path is low and there is no alternative path, the in-vehicle system 300 determines whether to stop or pause information transmission over the currently used transmission path, i.e., stop the HUD display as described above, or to continue transmission while reducing the amount of data and information to be transmitted, i.e., limit the HUD display as described above, and executes the determined response. Various response determinations are possible in step S104, and specific examples will be described later. Furthermore, when executing a response, the controller 100 and the HUD device 1 cooperate with each other, for example, by mutually notifying and confirming, as described above.

[0126] In step S108, the in-vehicle system 300 outputs to the user the status of the measures determined in step S104. Examples of the output include the above-mentioned Fig. 18A etc. After step S108, the process proceeds to step S109.

[0127] In step S105, the in-vehicle system 300 selects one route to be used as a switching destination from one or more alternative routes for the currently used transmission route.

[0128] In step S106, the in-vehicle system 300 switches the information transmission path 200. When this switching is performed, the controller 100 and the HUD device 1 cooperate with each other, for example, by notifying and confirming with each other. A specific example of cooperation will be described later.

[0129] In step S107, the in-vehicle system 300 outputs to the user the switching state of the information transmission path 200. Examples of the output include the above-mentioned Fig. 18C etc. After step S107, the process proceeds to step S109.

[0130] In step S109, the in-vehicle system 300 ends the information transmission status confirmation process. The flow of Fig. 21 is similarly repeatedly executed while the function is in the ON state.

[0131] [Processing Flow (2)] The processing flow in FIG. 22 is a modified example of the processing flow in FIG. 21 , a detailed processing example, and is an example in which the main determination is made on the controller 100 side, and also shows a specific example of cooperation between the controller 100 and the HUD device 1.

[0132] In step S201, the controller 100 monitors the information transmission status of the information transmission path 200. In step S211, the HUD device 1 monitors the information reception status of the information transmission path 200.

[0133] In steps S202 and S203, the controller 100 makes the same determination as described above, and proceeds to step S204, step S207, or step S211 depending on the determination. In step S204, the controller 100 determines whether to stop transmission or reduce the amount of data to be transmitted, and in step S205, notifies the HUD device 1 of the determined action. In step S212, the HUD device 1 receives the notification of the action. In step S206, the controller 100 takes action by stopping transmission or transmitting with a reduced amount of data. In step S213, the HUD device 1 takes action in accordance with the notification, by stopping the HUD display corresponding to the transmission stop, or by limiting the HUD display corresponding to transmission with a reduced amount of data.

[0134] In step S207, the controller 100 selects a route to switch to, and in step S208, the controller 100 notifies the HUD device 1 of an action to take regarding the route switching. In step S214, the HUD device 1 receives the notification of the action, and in step S215, the HUD device 1 prepares to switch the route as an action in accordance with the notification, and when preparation is complete, transmits a preparation completion notification to the controller 100. In step S209, the controller 100 receives the preparation completion notification, and in step S210, the controller 100 executes the route switching. In step S211, the information transmission status confirmation process ends.

[0135] As described above, in the flow of FIG. 22, the controller 100 takes the lead in controlling the situation assessment and response decision, and the HUD device 1 performs corresponding control in accordance with notifications and instructions from the controller 100.

[0136] [Processing Flow (3)] The processing flow in FIG. 23 is a modified example of the processing flow in FIG. 21 , a detailed processing example, and is an example in which the main determination is made on the HUD device 1 side, and also illustrates a specific example of cooperation between the controller 100 and the HUD device 1.

[0137] In step S301, the HUD device 1 monitors the information reception status of the information transmission path 200. In step S311, the controller 100 monitors the information transmission status of the information transmission path 200.

[0138] In steps S302 and S303, the HUD device 1 makes the same determination as described above, and proceeds to step S304, step S307, or step S311 depending on the determination. In step S304, the HUD device 1 determines the appropriate action to take: stopping the HUD display by stopping transmission or limiting the HUD display by transmitting a reduced amount of data. In step S305, the HUD device 1 notifies the controller 100 of the determined action. In step S312, the controller 100 receives the notification of the action. In step S313, the controller 100 performs transmission by stopping transmission or transmitting a reduced amount of data as the action in accordance with the notification. Correspondingly, in step S306, the HUD device 1 performs the action of stopping the HUD display corresponding to the transmission stop or limiting the HUD display corresponding to the transmission by transmitting a reduced amount of data.

[0139] In step S307, the HUD device 1 selects a route to switch to, and in step S308, the HUD device 1 notifies the controller 100 of an action to take regarding the route switching. In step S314, the controller 100 receives the notification of the action, and in step S315, the controller 100 prepares to switch the route as an action in accordance with the notification, and when preparation is complete, transmits a preparation completion notification to the HUD device 1. In step S309, the HUD device 1 receives the preparation completion notification, and in step S310, the HUD device 1 executes the route switching. In step S311, the information transmission status confirmation process ends.

[0140] As described above, in the flow of FIG. 23, the HUD device 1 takes the lead in controlling situation assessment and response decisions, and the controller 100 performs corresponding control in accordance with notifications and instructions from the HUD device 1.

[0141] [Example of Control Combination] FIG. 24 is a table summarizing examples of combined application of the above-described multiple types of control, including, for example, switching the information transmission path 200, limiting HUD display by transmission with reduced data volume, and stopping HUD display by stopping transmission. The table in FIG. 24 also illustrates an example in which application of multiple types of control is selected and switched using control priorities. In the table in FIG. 24, rows labeled #1 to #4 represent each control example. From left to right, the columns are "Currently in Use," "Abnormal Case," and "Destination Route and Switching Details." The "Destination Route and Switching Details" column has columns numbered Priority 1 to Priority 5. The priority here refers to the priority regarding the application of multiple types of control. Here, the highest priority is Priority 1, and the lowest priority is Priority 5. In this example, it is assumed that the information transmission path 200 includes a first transmission path 201 and a second transmission path 202, as shown in FIGS. 4 and 14 .

[0142] (#1) A first example is as follows. First, assume that the information transmission path 200 currently being used to transmit the video information 250 of the external video input from the controller 100 to the HUD device 1 is the first transmission path 201. As an abnormal case, assume that the first transmission path 201 goes into a "transmission unavailable" state. In this case, the first priority 1 response is to switch to the second transmission path 202. If the second transmission path 202 is available, this switching is executed. If it is not available, the next candidate is a priority 2 response. That is, the priority 2 response is to stop the HUD display by stopping transmission.

[0143] (#2) In the second example, compared to the first example, the information transmission path 200 that is initially being used to transmit the video information 250 is the second transmission path 202. As an abnormal case, it is assumed that the second transmission path 202 enters a "transmission unavailable" state. In this case, the first priority 1 response is to switch to the first transmission path 201. If the first transmission path 201 is available, this switching is executed. If it is not available, the next candidate is a priority 2 response. That is, the priority 2 response is to stop the HUD display by stopping transmission.

[0144] (#3) A third example is as follows. First, assume that the information transmission path 200 currently being used for transmitting video data is the first transmission path 201. In an abnormal case, assume that the first transmission path 201 is in a "transmission speed threshold or below" state, in other words, a state in which transmission performance has deteriorated, or the aforementioned malfunction. In this case, the first priority 1 measure is to switch the second transmission path 202 as the main path and the first transmission path 201 as the sub-path, and perform transmission using these two paths in parallel. The aforementioned FIG. 15 shows an example. If the second transmission path 202 is available and transmission is possible with the first transmission path 201 at a transmission speed greater than the threshold, this priority 1 measure is executed. To achieve maximum transmission efficiency, the combined use of these two paths is considered the first priority. The second transmission path 202, which allows normal transmission, is used as the main path, and the first transmission path 201, which has a transmission abnormality but is still able to transmit, is used as the sub-path, and the two paths are used in combination.

[0145] Next, if the transmission speed of the first transmission path 201 used in combination at priority 1 is below the threshold and is very slow, for example close to 0, then as a measure for priority 2, the system switches to the second transmission path 202 and transmits only via the second transmission path 202.

[0146] Next, when the second transmission path 202 is unavailable at priority 2, that is, when transmission is not possible or the transmission speed is equal to or lower than a threshold, for example, when the transmission speed is even slower than the transmission speed of the first transmission path 201, priority 3 control is performed, whereby transmission is continued via the first transmission path 201, but with a reduced data volume, thereby restricting the HUD display.

[0147] Next, when the transmission speed of the second transmission path 202 at priority 3 is equal to or lower than the threshold but is still higher than the transmission speed of the first transmission path 201, the amount of data is reduced and transmitted via the second transmission path 202 as control at priority 4. This limits the HUD display.

[0148] Next, if delays occur even when the data volume is reduced to the minimum necessary for both the first transmission path 201 and the second transmission path 202 at priority level 4, the HUD display is stopped by stopping transmission as control at priority level 5.

[0149] (#4) In the fourth example, compared to the third example, the information transmission path initially used for transmitting the video information 250 is the second transmission path 202, and the selected path is reversed. As an abnormal case, assume that the second transmission path 202 is in a "transmission speed below the threshold" state. In this case, the first priority 1 action is to switch the first transmission path 201 as the main path and the second transmission path 202 as the sub-path, and perform transmission using these two paths in parallel. If the first transmission path 201 is usable and transmission is possible on the second transmission path 202 at a transmission speed greater than the threshold, this priority 1 action is executed.

[0150] Next, if the transmission speed of the second transmission path 202 used in combination at priority 1 is below the threshold and is very slow, for example close to 0, then as a measure for priority 2, the system switches to the first transmission path 201 and transmits only via the first transmission path 201.

[0151] Next, when the first transmission path 201 is unavailable at priority 2, that is, when transmission is not possible or the transmission speed is equal to or lower than a threshold, for example, when the transmission speed is slower than the transmission speed of the second transmission path 202, priority 3 control is performed, whereby transmission is continued via the second transmission path 202, but with a reduced amount of data, thereby restricting the HUD display.

[0152] Next, when the transmission speed of the first transmission path 201 is equal to or lower than the threshold value at priority 3 but is still higher than the transmission speed of the second transmission path 202, the amount of data is reduced and transmitted via the first transmission path 201 as control at priority 4. This limits the HUD display.

[0153] Next, if delays occur even when the data volume is reduced to the minimum necessary for both the second transmission path 202 and the first transmission path 201 at priority 4, the HUD display is stopped by stopping transmission as control at priority 5.

[0154] [Effects of First Embodiment] As described above, the first embodiment provides a more suitable HUD technology that can handle cases where an image is generated outside the HUD and the HUD receives and displays that image as a video input. According to the first embodiment, even if the information transmission path 200 between the controller 100 and the HUD device 1 is abnormal or malfunctioning, the HUD display is improved to ensure real-time HUD display by taking measures such as switching or combining transmission paths, reducing the amount of transmitted data, or stopping transmission. This prevents impediments to safe driving.

[0155] <Second Embodiment> The second embodiment will be described using Figure 25 and subsequent figures. The basic configuration of the second embodiment is the same as that of the first embodiment, and the following mainly describes the components of the second embodiment that are different from those of the first embodiment. The second embodiment has a function corresponding to the second solution described above.

[0156] 9, in the second embodiment, the in-vehicle system 300 performs a control as a solution 2-1 in which the appropriateness of the HUD display based on the video information of the external video input is confirmed and an inappropriate image is not displayed. Various control examples included in the solution 2-1 are described below.

[0157] 25 and 26 are explanatory diagrams of a specific example regarding the appropriateness of HUD display. Similar to FIG. 9 , this example shows a case where there is one information transmission path 200 between the controller 100 and the HUD device 1. This example explains a case where the HUD device 1 checks the appropriateness of the HUD display image assuming HUD display for video information 250 provided from the controller 100 via transmission over the information transmission path 200, and performs control to limit the HUD display based on the check result.

[0158] Figure 25 shows a normal case, i.e., a case where the HUD display is appropriate. The HUD device 1 checks the appropriateness, or in other words, the validity, of the HUD display for the video information 250 received from the controller 100 via the information transmission path 200. The HUD device 1, for example, the video processing unit 1013 in Figure 6, performs this checking and judgment process based on predetermined criteria, conditions, and the like. If the video processing unit 1013 determines that there is no problem with displaying the target video in the HUD display area 5, in other words, if it determines that the appropriateness and validity are sufficiently high and meet the criteria, it outputs the target video to the video display device 10 for display.

[0159] 6, the image processor 1013 stores display data of the target image in the volatile memory 1012. The display driver 1022 reads the display data of the target image from the volatile memory 1012 and drives the display panel 12 in accordance with the display data. As a result, the target image is displayed on the screen of the display panel 12, and image light passes through the mirror 22 to form a virtual image 9 in the HUD display area 5. The display in the HUD display area 5 is similar to that shown in FIG. 11B, for example.

[0160] Furthermore, in such a normal case, although not required, the HUD device 1 may transmit or notify the controller 100 of a response 270. The response 270 is at least one of information indicating that the target image provided by the controller 100 will be displayed on the HUD and information indicating that there is no problem with displaying the image on the HUD, i.e., information indicating that the appropriateness condition is met.

[0161] FIG. 26 illustrates an abnormal case, i.e., a case where the HUD display is inappropriate. The HUD device 1 confirms the appropriateness, or appropriateness, of the HUD display, assuming HUD display, for the video information 250 received from the controller 100 via the information transmission path 200. The HUD device 1, for example, the video processing unit 1013 in FIG. 6 , performs this confirmation and determination process based on predetermined criteria and conditions. If the video processing unit 1013 determines that the target video would be problematic if displayed on the HUD—in other words, if the video processing unit 1013 determines that the target video is of low appropriateness or does not meet the criteria—then the target video is not output to the video display device 10 and is not displayed. In FIG. 6 , the video processing unit 1013 does not store display data for the target video in the volatile memory 1012. As a result, the target video is not displayed on the screen of the display panel 12, and the target video is not formed in the HUD display area 5.

[0162] The example of the image information 250 in Figure 26 is an example of a solid image with a certain brightness and color. In the case of such an image, the entire HUD display area 5 is covered with that image, which significantly obstructs the driver's view. Therefore, the image processing unit 1013 of the HUD device 1 determines that this image is not suitable for HUD display and is problematic, and controls the image so that it cannot be displayed.

[0163] FIG. 27 is a schematic diagram showing an overview of a case where a virtual image 9 is displayed in the HUD display area 5 based on the above-described solid-color image information 250. In this case, the driver can see the road ahead, etc. through the solid-color virtual image 9, but the driver's field of vision is obstructed to a large extent, making it difficult to see the road ahead, etc. Furthermore, for example, in a case where a solid white virtual image 9 is displayed in a dark environment, such as at night, the virtual image display is too bright compared to the darkness of the surroundings, making it difficult for the driver to see the road ahead, etc., which is extremely dangerous. If this image is not displayed, the result will be the same as FIG. 11A.

[0164] Furthermore, in such an abnormal case, the HUD device 1 must transmit and notify the controller 100 of a response 270. The response 270 is at least one of information indicating that the target image provided by the controller 100 will not be displayed on the HUD and information indicating that there is a problem with the HUD display of the image, i.e., information indicating that the image does not satisfy the conditions of appropriateness and validity.

[0165] FIG. 28 illustrates an example in which, when the HUD device 1 checks the appropriateness of a target image and determines that the image is problematic and cannot be displayed, the HUD device 1 prevents the image from being displayed in the HUD display area 5 (this can also be referred to as display stop or display restriction), and displays, for example, an image 2801 indicating "display not possible," or in other words, that HUD display is restricted or stopped, via an OSD. In the example of FIG. 28 , the image 2801 is marked "HUD display not possible (display rate)," and the "display rate" in parentheses indicates the reason for the "display not possible." It is preferable to allow the driver to select whether or not to display the reason for the HUD display not possible, as in this example. This is because some drivers may want to know the details of the reason for the display not possible, while others may feel anxious and be unable to concentrate on driving when the reason for the display not possible is displayed.

[0166] It should be noted that it is also possible to restrict the display so that some of the image is displayed and some is not, other than completely hiding the image in the HUD display area 5. For example, the display may be restricted so that the display ratio of the image in the HUD display area 5 is kept below a predetermined ratio.

[0167] 29 shows a process flow in the in-vehicle system 300 according to the second embodiment, in which the HUD device 1 checks the appropriateness and validity of the HUD display based on the video information 250 provided to the HUD device 1 from the controller 100, and controls the HUD display. The main processing in the process flow in FIG. 29 is performed by the video processing unit 1013 ( FIG. 6 ) of the HUD device 1.

[0168] In step S400, the in-vehicle system 300 starts the video confirmation process. The flow of FIG. 29 is performed when the function of the second embodiment is on. In step S401, the controller 100 transmits the video information 250 to the HUD device 1 via the information transmission path 200. The HUD device 1, particularly the video processing unit 1013 in FIG. 6, receives the video information 250 and temporarily stores it in a memory, specifically the volatile memory 1012 in FIG. 6.

[0169] In step S402, the video processing unit 1013 of the HUD device 1 determines whether the received video information 250 satisfies a condition for disabling HUD display, in other words, an inappropriate condition. Examples of the conditions will be described later. If it is determined that the condition does not apply and the video information 250 is displayable (No), the process proceeds to step S403. If it is determined that the condition applies and the video information 250 is displayable (Yes), the process proceeds to step S404.

[0170] In step S403, the image processing unit 1013 of the HUD device 1 controls the HUD to display the received image information 250. The image processing unit 1013 outputs the image information 250 to the image display device 10. The image processing unit 1013 in FIG. 6 creates display data for the image information 250 and stores it in the volatile memory 1012. The display driver 1022 reads the display data from the volatile memory 1012 and drives the display panel 12. As a result, a virtual image 9 corresponding to the image information 250 is formed in the HUD display area 5.

[0171] In step S405, the video processing unit 1013 of the HUD device 1 transmits a response 270 indicating that the received video information 250 has been displayed on the HUD to the controller 100. The controller 100 can recognize from the response 270 that the video information 250 has been displayed on the HUD. Step S405 can be omitted.

[0172] In step S404, the image processing unit 1013 of the HUD device 1 performs control so as not to display the received image information 250 on the HUD. The image processing unit 1013 does not output the image information 250 to the image display device 10. The image processing unit 1013 in FIG. 6 does not store display data for the image information 250 in the volatile memory 1012. As a result, the virtual image 9 corresponding to the image information 250 is not formed in the HUD display area 5.

[0173] In step S406, the video processing unit 1013 of the HUD device 1 transmits a response 270 indicating that the received video information 250 will not be displayed on the HUD to the controller 100. The controller 100 can recognize from the response 270 that the video information 250 was not displayed on the HUD.

[0174] In step S407, the video confirmation process ends. The flow of Fig. 29 is repeated in the same manner as in the second embodiment in accordance with the generation of video information 250, while the function of the second embodiment is in an ON state.

[0175] [Conditions for Display Unavailable] Fig. 30 is a table summarizing the conditions for display unavailability and inappropriateness described above, which correspond to the various control examples in Fig. 9. The table lists the criteria for determining whether a display is unavailable.

[0176] [2-1-1. Determination and Limitation of HUD Display Amount / Display Rate] An example of a case where display is not possible is when there is an excessive amount of HUD display content. The determination criterion is whether the proportion of the display content (referred to as the display rate) in the HUD display area 5, i.e., the entire range in which the virtual image 9 can be displayed, is equal to or greater than a threshold. If the display rate assuming HUD display based on the video information 250 to be displayed is equal to or greater than a threshold, the display is determined to be inappropriate. As a specific example of a determination method, the video processing unit 1013 of the HUD device 1 calculates the display rate by comparing the total area of ​​the display content based on the display data of the video information 250 to be displayed with the area of ​​the HUD display area 5. For example, a display rate of 30% or more is determined to be not possible.

[0177] In this processing example, to confirm whether the video information 250 is displayable or not and whether it is appropriate, the video processing unit 1013 creates temporary display data for when the target video information 250 is displayed on a HUD, and calculates the display rate and the like using the temporary display data. If the confirmation result indicates that the HUD should not be displayed, the temporary display data is discarded. As another example of a determination method in step S402, the binary data of the target video information 250 may be analyzed to determine that the HUD should not be displayed if a feature corresponding to the display-not-permitted case in FIG. 30 is extracted.

[0178] In the above example, the display rate is used to determine whether or not display is possible, but this is not limited to this. Instead, the amount of display / amount of data / amount of information on the HUD display may be calculated to determine the amount of display.

[0179] 31 shows a specific example of calculation of the display ratio corresponding to the control example of 2-1-1 above. Based on the target video information 250, the rectangular HUD display area 5 has areas 3101 and 3102 as areas where the display content is arranged, for example, in the bottom area or near the center. If the total area of ​​the HUD display area 5 is taken as 100, and the total area of ​​areas 3101 and 3102 is equal to or greater than, for example, a threshold value of 30%, it is determined that the content cannot be displayed.

[0180] As a control example when it is determined that display is not possible based on the above conditions, a basic control is to hide all of the video information 250 corresponding to the areas 3101 and 3102. This is not limiting, and control to reduce the amount of content to be displayed is also possible. For example, if hiding either the area 3101 of the central content or the area 3102 of the bottom content would result in the display rate being less than the threshold, such display may be possible. Furthermore, if display priorities are set according to the display position within the HUD display area 5 or the type of displayed content, control may be performed so that items with higher priorities are displayed and items with lower priorities are hidden.

[0181] 32 shows another example of the calculation of the display rate. Based on the target video information 250, display content using virtual images 9 may be arranged in the rectangular HUD display area 5 in areas or locations indicated by A1, A2, A3, B1, B2, B3, C1, C2, and C3. For example, A1, A2, and A3 are examples of areas near the center in the vertical direction, B1, B2, and B3 are examples of lower edge areas, and C1, C2, and C3 are examples of upper edge areas. For example, display content may be arranged simultaneously in multiple areas, including areas A1 and A2, areas B2 and B3, and area C1. If the display rate based on the total area of ​​these display content areas is equal to or greater than a threshold, it is determined that the content cannot be displayed.

[0182] As a further modification, a weight may be added to each display position within the HUD display area 5, and the display rate may be calculated by multiplying the area by the weight value. For example, the central area A1 may be assigned a relatively large weight value, while the peripheral areas B2, B3, C2, C3, etc. may be assigned relatively small weight values. That is, the weight may be larger toward the center of the HUD display area 5, since it is assumed that the weight has a greater impact on the driver's visibility and safety, and smaller toward the periphery, since it is assumed that the weight has a smaller impact on the driver's visibility and safety. As a result, even for display content with the same area, the weight that affects display inapplicability and appropriateness will vary depending on the display position, resulting in different display rate calculation results using the area and weight value.

[0183] [2-1-2. Determination and Restriction of Brightness, etc. of HUD Display] An example of a case where display is not permitted is when the brightness of the HUD display content is abnormal. The determination criterion is that the brightness specified in the target video information 250 is abnormal with respect to a daytime brightness reference value or a nighttime brightness reference value. As a specific example of a determination method, the HUD device 1 determines whether it is currently daytime or nighttime based on time information, and references a brightness reference value corresponding to daytime or nighttime. The HUD device 1 compares the brightness (which may be, for example, average brightness) of the display content of the target video information 250 with the brightness reference value, and if the difference therebetween is equal to or greater than a threshold, determines that the brightness is abnormal.

[0184] FIG. 33 is an explanatory diagram of a specific example of the control example of 2-1-2 above. In this example, a navigation arrow image is displayed as display content 3301 near the center of the HUD display area 5. It is assumed that it is daytime, and a daytime brightness reference value 3302 is set in advance. In the video information 250 from the controller 100, a brightness 3303 of the display content 3301 is specified. For control purposes, for example, a threshold range 3304 is set that includes the daytime brightness reference value 3302. The threshold range 3304 is, in other words, a brightness reference range. If the brightness 3303 of the display content 3301 falls outside the threshold range 3304, the HUD device 1 determines that the display content 3301 cannot be displayed.

[0185] The above control example is a case where the relative difference between the brightness of the HUD display content and a brightness reference value corresponding to the ambient brightness is determined, but this is not limited to this, and it is also possible to determine the absolute brightness of the display content.

[0186] The above example is an example in which the determination is made by roughly dividing the time into two periods, daytime and nighttime, but the present invention is not limited to this. For example, it is also possible to make a determination by taking into account the influence of the characteristics of a dark place, such as inside a tunnel or under an overpass, even during the day. It is also possible to make a determination by taking into account the influence of weather, such as sunny, cloudy, rainy, or snowy. This allows for more accurate control.

[0187] As a variant example, when referring to the brightness reference value, the HUD device 1 may use a sensor of itself or the controller 100 to detect the ambient brightness, the brightness near the HUD display area 5, etc., and calculate a brightness reference value according to the current time.

[0188] An example of control when it is determined that display is not possible based on the above conditions is to basically hide the target video information 250. However, other examples of control are possible, such as limiting the HUD display by changing the brightness of the display content so that it falls within a threshold range of the daytime brightness reference value.

[0189] [2-1-3. Determination and Limitation of Information Errors and Discrepancies with Actual Situation] An example of a case where information cannot be displayed is when incomplete information causes a discrepancy between the actual situation and the HUD display. An example of a criterion for this determination is an error in the information provided to the user. The HUD device 1 determines whether or not there is an error in the information provided as video information 250 from the controller 100, such as the vehicle speed of the vehicle or navigation route information, in light of the actual situation. For example, the HUD device 1 determines that there is an error when the actual situation and the provided information do not completely match, or when the difference between the actual situation and the provided information is equal to or greater than a predetermined value.

[0190] As a specific example of the determination method, the HUD device 1, particularly the video processing unit 1013 in FIG. 6, analyzes the video content information provided as the video information 250 from the controller 100 or refers to attribute information or metadata attached to the video information 250 to determine the video content information to be determined for errors. At the same time, the HUD device 1 refers to information obtained from itself or another information source as information corresponding to the target video content information. The HUD device compares the target video content information with the corresponding information to determine whether the target video content information is error-free, in other words, whether there is a discrepancy with the actual situation.

[0191] Another example of an information source is the car navigation system 150 in Fig. 3. The HUD device 1 acquires information from the car navigation system 150 via the controller 100.

[0192] FIG. 34 shows a specific example of the control example 2-1-3 above. For example, suppose that the display content 3401 based on the video information 250 from the controller 100 is a navigation arrow image as shown in the figure. This navigation arrow image is an image encouraging a right turn 100 meters ahead. However, in the current actual scenery, the road ahead is a straight road, and there is no road to turn right. In other words, if this display content 3401 is displayed on the HUD, a discrepancy will occur with the actual situation. The HUD device 1 also references navigation information from, for example, the car navigation system 150, and compares the information of the display content 3401 with the information of the car navigation system 150. For example, if there is a large difference between the information, the HUD device 1 determines that the information of the display content 3401 may be erroneous and therefore cannot be displayed.

[0193] A control example when it is determined based on the above conditions that the information cannot be displayed is basically to hide the target video information 250. Alternatively, the HUD device 1 may display an image in the HUD display area 5 indicating that the target video information 250 is erroneous and has been hidden. Another control example is to restrict HUD display by displaying the target video information 250 as is, and then displaying an image in the HUD display area 5 indicating that the target video information 250 may be erroneous.

[0194] As a modified example, the HUD device 1 may determine the actual situation in the driver's field of view, i.e., the actual situation related to the displayed content, by using a sensor of the HUD device 1 itself or the controller 100, for example, by using image information captured by an external camera. For example, the HUD device 1 may determine, from the camera image, whether the road ahead is a straight road or whether there is a road where a right turn is required.

[0195] In the above-described various control examples, the video content of the video information 250 may change over time. Even in this case, the same control can be applied to each piece of video data at each point in time.

[0196] 35A and 35B show examples of control status displays related to the functions of the second embodiment. In the HUD display area 5, a "display restricted (inappropriate)" image 3501 is displayed, for example, via OSD. This image 3501 indicates that the HUD display is restricted or stopped because the appropriateness check results indicate inappropriateness. In other words, this image 3501 indicates that the display rate of the image displayed on the HUD has been reduced for safe driving. In the example of FIG. 35A , when an image to be displayed, such as that shown in FIG. 34 , is present based on the image information 250, the right-turn navigation information 3401 is deleted as a result of the appropriateness check, and the image displayed is limited to only the vehicle speed image information 11B1. The vehicle speed image information 11B1 continues to be displayed when the display is restricted because vehicle speed information is considered to be the most basic and important information for safe driving. It is preferable that the driver be able to select whether or not to display the reason for the HUD display restriction, as in this example. Some drivers may want to know the details of why the display is restricted, while others may feel anxious and unable to concentrate on driving if negative words such as "inappropriate" are displayed as the reason for the display restriction.

[0197] The example in Figure 35B shows a state in which all images to be displayed are restricted, in other words, the display on the display device is stopped. As mentioned above, vehicle speed information is the most basic and important information for safe driving, but continuing to display incorrect information does not contribute to safe driving, so in this example, the vehicle speed image information 11B1 is also deleted. Such a measure can be taken when there is another means in the vehicle for the driver to check vehicle speed information other than the HUD display, for example, when vehicle speed information is displayed on the meter cluster. Note that the driver is notified by audio output from the speaker that the display is restricted or stopped.

[0198] [Effects of Second Embodiment] As described above, the second embodiment provides a more suitable HUD technology that can handle cases where an image is generated outside the HUD and the HUD receives and displays that image as a video input. According to the second embodiment, when attempting to display an image on the HUD based on video information from an external video input, the appropriateness of the HUD display is confirmed, and if the image is inappropriate, the display is disabled. This prevents impediments to safe driving.

[0199] It is also possible to implement both the functions of the first embodiment and the second embodiment. It is also possible to implement a plurality of functions as various functions, including modifications. In this case, it is possible to set which function to use in the in-vehicle system.

[0200] <Third Embodiment> A third embodiment will be described with reference to Figure 36 and subsequent figures. The basic configuration of the third embodiment is the same as or common to the first and second embodiments, and the following description will mainly focus on the components of the third embodiment that are different from the first and second embodiments.

[0201] In the third embodiment, when a user's mobile terminal is connected to the in-vehicle system for communication, the mobile terminal can be used as a video source to display an image on the virtual image display device. In addition, the third embodiment has control over selection and switching of the video source in this case.

[0202] [3-1. In-Vehicle System] FIG. 36 shows a configuration example of an in-vehicle system 300, which is a system including a virtual image display device 1, according to the third embodiment. The in-vehicle system 300 is provided in a vehicle 2. The in-vehicle system 300 includes a controller 100 and a virtual image display device 1. The virtual image display device 1 in FIG. 36 is a display device having a function of displaying a virtual image 9 (FIGS. 1 and 2). The virtual image display device 1 may be a head-up display (HUD) device 1 such as that shown in FIG. 3, or may be another type of display device, such as a mirror-type display device (FIG. 37) described below. The mirror-type display device is a compact display device suitable for in-vehicle use and is equipped with image light control technology that enables image light to be directed and projected in any direction. Note that, although the mirror-type display device is considered a different concept from a HUD device here, the mirror-type display device may also be considered a type of HUD device.

[0203] The controller 100 and the virtual image display device 1 are connected via a predetermined information transmission path 200, in other words, a communication path 200, and are capable of communicating with each other. The controller 100 is, for example, a control device / control unit also called an ECU, and controls the entire in-vehicle system 300. The main difference between FIG. 36 and FIG. 36 in relation to FIG. 3 is that a user's mobile terminal 400 is included. The mobile terminal 400 is a general term that includes, for example, a smartphone, a tablet terminal, a wearable terminal, etc. The mobile terminal 400 is communicatively connected to the virtual image display device 1 and the controller 100. A communication path 601 indicated by an arrow indicates the communication connection and communication path between the mobile terminal 400 and the virtual image display device 1. A communication path 602 indicated by an arrow indicates the communication connection and communication path between the mobile terminal 400 and the controller 100. A user who uses the mobile terminal 400 is, for example, a driver or a passenger. The communication interfaces of the communication paths 601 and 602 are not particularly limited, but may include wireless LAN, Bluetooth (registered trademark), etc. The communication interfaces of the communication paths 601 and 602 are not limited to wireless, and may be wired.

[0204] In the third embodiment, for example, one or more mobile terminals 400 serving as video providers (in other words, content providers) may be connected to the virtual image display device 1 via a controller 100 as shown in FIG. 36 , or may be connected directly to the virtual image display device 1 without going through the controller 100. In FIG. 36 , the mobile terminal 400 serving as the video provider holds video data 3601. The mobile terminal 400 transmits video data 650 corresponding to the video data 3601 to the virtual image display device 1 via a communication path 601, or transmits the video data 650 to the controller 100 via a communication path 602. In the latter case, the controller 100 transmits video data corresponding to the video data 650 to the virtual image display device 1 via an information transmission path 200. The virtual image display device 1 displays an image as a virtual image 9 in the display area 5 based on the video data.

[0205] In the third embodiment, in the in-vehicle system 300, when an image is displayed as a virtual image 9 ( FIG. 1 , etc.) by the virtual image display device 1, the device that provides the image can be selected / switched between the virtual image display device 1, the controller 100, the mobile terminal 400, etc. This selection / switching may be fixed in the system design / settings, but it is also possible to perform variable selection / switching automatically determined by the system or variable selection / switching based on settings or instructions by the user.

[0206] In the third embodiment, particularly when the mobile terminal 400 is used as the video provider, as will be described later, there may be cases where the video cannot be provided properly from the mobile terminal 400, and in such cases, the system has a control function for making appropriate selections, switching, and dealing with the situation.

[0207] 36 , the mobile terminal 400 may include an application program for using the controller 100 of the in-vehicle system 300 or the virtual image display device 1. As an example, the mobile terminal 400 includes a car navigation application 411, a map application 412, and the like. Furthermore, the mobile terminal 400 can use not only these applications but also functions such as a web browser to access websites and services on the Internet, acquire desired information, and display it on the touch panel screen 405. Images and information stored in the mobile terminal 400, or images and information from an external source (e.g., a server 3600 on the Internet) designated by the mobile terminal 400, can be transmitted to the controller 100 or the virtual image display device 1 via communication, thereby displaying the images and information as a virtual image 9 on the virtual image display device 1.

[0208] In the third embodiment, the mobile terminal 400 can be a provider of external video information and the like, in other words, a video provider or video generator, for the controller 100 and the virtual image display device 1. The mobile terminal 400 transmits video data / video information corresponding to an external video input to the controller 100 or the virtual image display device 1 via a communication path 601 or a communication path 602. Whether the mobile terminal 400 is to be communicatively connected to the controller 100 or the virtual image display device 1 may be fixed in system design / settings or may be variable by selection. Furthermore, the mobile terminal 400 may be communicatively connected to both the controller 100 and the virtual image display device 1.

[0209] The virtual image display device 1 can output an image, in other words, display a virtual image, based on image data from the mobile terminal 400 that provides the image. In one embodiment, when the controller 100 is the main control entity, the mobile terminal 400 transmits image data 650 to the controller 100 via the communication path 602, and the controller 100 transmits the image data 650 or corresponding image data to the virtual image display device 1 via the information transmission path 200. In another embodiment, when the virtual image display device 1 is the main control entity, the image data 650 may be transmitted from the mobile terminal 400 that provides the image to the virtual image display device 1 via the communication path 601. The control entity refers to an entity that performs control that is characteristic of the third embodiment, including control of selecting and switching the image provider.

[0210] The virtual image display device 1 and the controller 100 each include a communication unit corresponding to the communication interface of the communication paths 601 and 602 with the mobile terminal 400 (see FIGS. 39 and 40 described below). The communication paths may be wired, wireless, or a combination of both. In the example of FIG. 36, the communication paths 601 and 602 are each illustrated as a single line, each representing a single path. However, this is not limited to this, and multiple paths may be multiplexed and parallelized, as in the first and second embodiments. In this case, multiple paths may be provided using a single communication interface, or multiple paths using multiple communication interfaces. The communication paths may be separated depending on the application. For example, a communication path for video data transmission and a communication path for control communication may be provided separately. The multiple paths may be selected or switched so that some paths are used normally and other paths are used in an abnormal situation. The performance of each path may be different.

[0211] [3-2. Mirror-Type Display Device] FIG. 37 shows an example of an in-vehicle configuration in which a display device 1M is applied as the virtual image display device 1 in the third embodiment. When the display device 1M is mounted on a vehicle 2, as in FIG. 2 , the display device 1M is disposed on a dashboard 2A, for example, near a portion of the upper surface of the dashboard 2A. The display device 1M may be a mirror-type display device or a display device other than a mirror-type display device. In this embodiment, a mirror-type display device will be used for explanation. Image light 1a emitted from the mirror-type display device 1M is directly reflected by a portion 3M of the windshield 3. The reflected image light 1b is incident on the driver's viewpoint 6, and the driver views the image light 1b as a virtual image 9. In other words, the portion 3M of the windshield 3 is a projection member 3M onto which the image light 1a is projected, and functions as a mirror (reflecting member) 3M. This mirror (reflecting member) 3M is one element constituting the mirror-type display device 1M or a mirror-type display system including the mirror-type display device 1M.

[0212] The mirror-type display device 1M includes a display panel 12 having an image display element and a light source device 11 having a light source. The light source device 11 emits light generated based on the light source (in other words, light source light) to the display panel 12. The display panel 12 forms and displays an image on a display screen based on image data, and generates and emits image light 1a corresponding to the image based on the light from the light source device 11. The mirror-type display device 1M projects the image light 1a (in other words, projection light) from the display panel 12 onto an area 3. The area 3 may be a projection member or a mirror, and is not particularly limited.

[0213] [3-3. Display Area of ​​Mirror-Type Display Device] FIG. 38A shows a display image of a display area 5M of a mirror-type display device 1M in a vehicle 2. The image light 1b ( FIG. 37 ) from the mirror-type display device 1M forms and displays a virtual image 9 in a predetermined display area 5M that corresponds to a partial area (projection member, mirror) 3M of the windshield 3 as seen by the driver. In this example, a text image of "40 km / h" indicating the vehicle speed is displayed as the virtual image 9 in the display area 5M. In the example of FIG. 38A , the display area 5M (in other words, the display range) is configured as a horizontally elongated screen, and the screen is divided into 10 5x2 regions. In the example of FIG. 38A , a text image as the virtual image 9 is displayed across the entire 5x2 divided area in the display area 5M.

[0214] 38A , a user's mobile terminal 400 is placed on a dashboard 2A. The mobile terminal 400 serves as an image provider for the mirror-type display device 1M. In this example, the display area 5M is formed in the lower right part of the front windshield 3 (in other words, the windshield) of the vehicle 2, in a position close to the dashboard 2A and the right handle (steering wheel) 2B.

[0215] FIG. 38B shows a display image of a display area 5M displayed by a mirror-type display device 1M in a vehicle 2 of another example. In this example, the vehicle 2 has a black ceramic portion (referred to as black ceramic portion 3B) on the top, bottom, left, and right sides of the front windshield 3. This black ceramic portion 3B appears black to the driver. This black ceramic portion 3B serves to protect the adhesive that adheres the windshield 3 to the vehicle body from external ultraviolet light and other factors. In the example of FIG. 38B, the display area 5M corresponding to the area (projection member, mirror) 3M has the same screen as that of FIG. 38A, but differs in that it is positioned so that it partially overlaps the black ceramic portion 3B located on the lower edge of the windshield 3. Specifically, the upper half of the display area 5M is positioned above the black ceramic portion 3B, and the lower half is positioned so that it overlaps the black ceramic portion 3B. 38B , in the display area 5M, only the lower half area overlapping the black ceramic portion 3B displays a text image of "40 km / h" and an arrow image as the virtual image 9. For example, if a bright image such as white is displayed as the virtual image 9 on top of the black area of ​​the black ceramic portion 3B, the image is easy to see.

[0216] However, it is also possible to display the virtual image 9 in an area that does not overlap with the black ceramic portion 3B. A specific example is shown in the lower part of Fig. 38B. The virtual image 9 is displayed only in the upper half area that does not overlap with the black ceramic portion 3B.

[0217] FIG. 38C shows a display image of a display area 5M by a mirror-type display device 1M in a vehicle 2 of another example. In this example, multiple, for example, two, display areas 5M (5M1, 5M2) are configured in the front windshield 3. In this example, the left display area 5M1 and the right display area 5M2 are arranged adjacent to each other. In this example, these two display areas 5M (5M1, 5M2) are configured using two mirror-type display devices 1M, which will be described later. In the example of FIG. 38C, the two display areas 5M1, 5M2 can be used as a single horizontally elongated display area 5M as a whole. It is possible to display the virtual image 9 in both of the two display areas 5M1, 5M2, or in only one of the two display areas 5M1, 5M2, as appropriate.

[0218] The positions of the two mirror-type display devices 1M relative to the dashboard 2A are not limited. The two devices may be disposed adjacent to each other or spaced apart. In the example of FIG. 38C , the two display areas 5M1 and 5M2 are disposed adjacent to each other, but this is not limiting. The two display areas 5M1 and 5M2 may be disposed spaced apart on the curved surface of the windshield 3. For example, a display area 5M for the driver and a display area 5M for a passenger may be provided.

[0219] [3-4. Functional Blocks Related to the Controller] FIG. 39 shows an example of a functional block configuration related to the controller 100 in the third embodiment. The difference between FIG. 39 and FIG. 5 is that the controller 100 includes a wired wireless communication unit 517 for mobile terminal-to-vehicle communication (also referred to as a mobile terminal communication unit) connected to or built into the controller 100. The mobile terminal communication unit 517 performs communication processing via a communication interface with the mobile terminal 400 in FIG. 36. The wired wireless communication unit 517 for mobile terminal-to-vehicle communication may be provided as part of the communication unit 520, for example, as a third communication unit. In other words, the wired wireless communication unit 517 for mobile terminal-to-vehicle communication may be eliminated and implemented as an integrated unit as the communication unit 520. In terms of the correspondence with FIG. 36, the controller 100 establishes a communication path 602 with the mobile terminal 400 via the mobile terminal communication unit 517 and performs communication processing via the communication path 602. Controller 100 can cooperate with portable terminal 400 based on communication with portable terminal 400 via portable terminal communication unit 517. Controller 100 can receive data and information such as requests and images from portable terminal 400 and can transmit data and information to portable terminal 400.

[0220] [3-5. Functional Blocks Related to the Virtual Image Display Device] FIG. 40 shows an example of a functional block configuration related to the virtual image display device 1 in embodiment 3. The example of FIG. 40 is based on the example of FIG. 9 , with the difference being the addition of a mobile terminal communication unit 4001. The mobile terminal communication unit 4001 is equipped with a communication interface with the mobile terminal 400 and performs communication processing via a communication path 600 with the mobile terminal 400. For example, the MCU 1010 or the video processing unit 1016 controls the mobile terminal communication unit 4001 and stores images and information received from the mobile terminal 400 via the mobile terminal communication unit 4001 in the volatile memory 1012. Note that the mobile terminal communication unit 4001 may be provided with a memory, and the images and information may be stored in that memory. Note that when the virtual image display device 1 receives or acquires image data, etc. from the mobile terminal 400 via the controller 100, the above-described communication unit 1015 may be used. The mobile terminal communication unit 4001 may be integrated and implemented within the communication unit 1015.

[0221] [3-6. Communication Path with Portable Terminal] FIG. 41 shows a configuration example of a communication path (in other words, an information transmission path) 600 between the portable terminal 400 and the controller 100 and the virtual image display device 1. In the example of FIG. 41A, the communication path 600 includes a communication path 601 with the virtual image display device 1 and a communication path 602 with the controller 100. In this example, the communication paths 601 and 602 use a direct connection communication interface such as Bluetooth, for example, and are connected to each other for communication without an intermediate communication network. The communication path 602 shown by a solid line here indicates a connection with the controller 100, and the communication path 601 shown by a dashed line indicates a connection with the virtual image display device 1. As a usage example, in a first example, video data 650 is transmitted using only the communication path 601 with the virtual image display device 1. In a second example, video data 650 is transmitted using only the communication path 602 with the controller 100.

[0222] In the third example, the two communication paths, the communication path 601 and the communication path 602, may be used simultaneously in parallel. For example, the communication path 602 with the controller 100 may be used as a main (first) communication path, and the communication path 601 with the virtual image display device 1 may be used as a secondary (second) communication path. Alternatively, the communication path 601 with the virtual image display device 1 may be used as a main (first) communication path, and the communication path 602 with the controller 100 may be used as a secondary (second) communication path. Furthermore, for example, the communication path 602 with the controller 100 may be used for transmitting image data, and the communication path 601 with the virtual image display device 1 may be used for control communication and operation input. Alternatively, the communication path 601 with the virtual image display device 1 may be used for transmitting image data, and the communication path 602 with the controller 100 may be used for control communication and operation input.

[0223] The example in FIG. 41A shows a case where one portable terminal 400 is connected, while the example in FIG. 41B shows a case where two portable terminals 400 are connected. Here, one portable terminal 400 is designated as portable terminal 400A, and the other portable terminal 400 is designated as portable terminal 400B. For example, portable terminal 400B is connected to the virtual image display device 1 via communication path 603, or to the controller 100 via communication path 604. As an example, user U1, who is the driver, possesses and uses portable terminal 400A, and user U2, who is a passenger, possesses and uses portable terminal 400B. In another example, one person may use multiple portable terminals 400. As in FIG. 41A, various communication connections are possible for each portable terminal 400.

[0224] FIG. 42 shows another configuration example of the communication path 600. In the example of FIG. 42A , the communication path 600 is connected to the virtual image display device 1 via a communication network 609. The communication network 609 is an in-vehicle network, such as a CAN or a wireless LAN, but is not limited to these. In this example, the communication path 600 includes communication paths 610 and 611 with the virtual image display device 1 and communication paths 610 and 612 with the controller 100. The communication path with the virtual image display device 1 includes the communication path 610 from the mobile terminal 400 to the communication network 609 and the communication path 611 from the communication network 609 to the virtual image display device 1. The communication path with the controller 100 includes the communication path 610 from the mobile terminal 400 to the communication network 609 and the communication path 612 from the communication network 609 to the controller 100. As a usage example, in a first example, video data 650 is transmitted using only the communication paths 610 and 611 with the virtual image display device 1. In the second example, video data 650 is transmitted using communication paths 610 and 612 to the controller 100 .

[0225] The example of (B) in Figure 42 similarly illustrates a case where two mobile terminals 400 (400A, 400B) are connected to communication network 609 via communication network 609. Various communication connections are possible for each mobile terminal 400. In this example, mobile terminal 400A is connected to communication network 609 via communication path 613, and mobile terminal 400B is connected to communication network 609 via communication path 614. Note that in various examples, multiple mobile terminals 400 connected to in-vehicle system 300 may communicate with each other.

[0226] Fig. 43 shows another example of the configuration of the communication path 600. In the example of Fig. 43 (A), the mobile terminal 400 is directly connected to the virtual image display device 1 via a communication path 601, and is connected to the controller 100 via communication paths 610 and 612 via a communication network 609. In the example of Fig. 43 (B), the mobile terminal 400 is connected to the virtual image display device 1 via communication paths 610 and 611 via the communication network 609, and is directly connected to the controller 100 via a communication path 602.

[0227] Regarding the various communication connection examples described above, each communication path is not limited to one-way communication, but can be two-way communication. For example, the communication path 601 in Fig. 41A can be used for inputting images and instructions from the mobile terminal 400 to the virtual image display device 1, and for outputting information from the virtual image display device 1 to the mobile terminal 400. Examples of information output include transmission of information such as confirmation and response in a communication protocol when establishing a communication connection, and transmission of control information for predetermined control.

[0228] Furthermore, with regard to the various examples of communication connections described above, a user may use the mobile terminal 400 as an operation input means / device for the virtual image display device 1. In other words, the mobile terminal 400 may be used as a so-called remote control. A graphical user interface (GUI) for operation input may be displayed on a screen of an app or the like on the mobile terminal 400. For example, the communication path 601 between the mobile terminal 400 and the virtual image display device 1 shown in FIG. 36 or FIG. 41A may be used for operation input communication. The mobile terminal 400 generates operation input information (in other words, instruction information) based on an operation input by a user via a touch panel, hardware buttons, or the like, and transmits the information to the virtual image display device 1 via the communication path 601. The virtual image display device 1 performs a corresponding predetermined operation, such as power on / off, brightness adjustment, volume adjustment, cursor movement, confirmation / cancellation, etc., in accordance with the received operation input information / instruction information.

[0229] 44A and 44B are explanatory diagrams illustrating the power supply and battery states of the portable terminal 400. In FIG. 44A, the portable terminal 400 is electrically connected to a power supply (i.e., a battery) 420, and the portable terminal 400 is fully charged. For example, the portable terminal 400 has a remaining battery capacity (charged power) of 90%. The power supply 420 may be an external or internal power supply, such as a mobile battery device. In the third embodiment, the portable terminal 400 is used as an image provider to display an image on the virtual image display device 1 using a communication channel 600. The portable terminal 400 in FIG. 44A holds image data (i.e., content) 4401, and transmits image data 650 corresponding to the image data 4401 to the virtual image display device 1 or the controller 100 via the communication channel 600. At this time, it is important to ensure that the portable terminal 400 does not run out of battery. At this time, if the battery of the mobile terminal 400 runs out, the image information to be displayed as the virtual image 9 on the virtual image display device 1 will not be provided, and the image transmission or virtual image display will malfunction or an error will occur, as described below.

[0230] FIG. 44B illustrates a case in which the battery of the portable terminal 400 runs low or runs out at the start or during transmission of video data 650 over the communication path 600 in an attempt to display an image on the virtual image display device 1 based on video data 4401. In the state of FIG. 44B , the portable terminal 400 is not connected to the power source 420, and the remaining battery level is, for example, 5%. For example, the portable terminal 400 transmits the video data 650 to the controller 100 over the communication path 602, and the controller 100 transmits the video data 650 (or other video data generated based on the video data 650) to the virtual image display device 1 over the information transmission path 200. If the battery of the portable terminal 400 runs low or runs out, the transmission of the video data 650 is not performed or is interrupted, which may result in a malfunction or an error in the virtual image display device 1 displaying the virtual image based on the video data 650.

[0231] 44B , the communication state of the communication path 602 and the information transmission path 200, for example, the radio wave state of the wireless connection, is good, but the battery is low. When the battery is low, the mobile information terminal 400 may change its operation to reduce battery consumption (for example, transition to a power-saving mode) and / or change the order of priority processing, which may result in malfunction or error in the video transmission over the communication path 600 and the virtual image display on the virtual image display device 1, as shown in the figure. This malfunction or error may cause problems with the virtual image display, which is the final output, such as an inability to ensure real-time display.

[0232] [3-8. Communication Path with Mobile Terminal: In the Case of Two Mirror-Type Display Devices] FIG. 45 shows an example of the configuration of an in-vehicle system 300 in which two mirror-type display devices 1M (FIG. 37) are used to configure two display areas 5M (5M1, 5M2) as in the example of FIG. 38C. The in-vehicle system 300 of FIG. 45(A) has two mirror-type display devices 1M (1M1, 1M2) as two virtual image display devices 1, each of which is connected to the controller 100 via an information transmission path 200 (200-1, 200-2). Furthermore, one mobile terminal 400 is communicatively connected to each of the mirror-type display devices 1M (1M1, 1M2). For example, the mobile terminal 400 may be connected to each of the mirror-type display devices 1M (1M1, 1M2) via a communication path 602 with the controller 100. Alternatively, the mobile terminal 400 may be connected to each of the mirror-type display devices 1M (1M1, 1M2) via a direct connection communication path 601 (601-1, 601-2).

[0233] For example, video data 650 may be transmitted from the portable terminal 400, which is a video provider, to the controller 100 via the communication path 602. Then, from the controller 100, video data corresponding to the video data 650 (e.g., video data 650-1, 650-2) may be transmitted to each of the mirror-type display devices 1M (1M1, 1M2) via the respective information transmission paths 200 (200-1, 200-2). For example, the first mirror-type display device 1M1 may project first video light generated based on the video data 650-1 onto the region 3M of the windshield 3 shown in FIG. 38C to form a virtual image 9 in the first display region 5M1 on the left side. At the same time, the second mirror-type display device 1M2 may project second video light generated based on the video data 650-2 onto the region 3M of the windshield 3 shown in FIG. 38C to form a virtual image 9 in the second display region 5M2 on the right side.

[0234] In another example, similar to the above, the mobile terminal 400 may transmit each of the video data 650 (650-1, 650-2) to each of the mirror-type display devices 1M (1M1, 1M2) via each of the communication paths 601 (601-1, 601-2). Also, the mobile terminal 400 may transmit the video data 650 to only one of the selected mirror-type display devices 1M, so that the virtual image 9 is displayed only in one of the display ranges 5M.

[0235] In the example of Fig. 45 (B), one mobile terminal 400 is connected to each of the mirror-type display devices 1M (1M1, 1M2) by a communication path 600 via a communication network 609. For example, video data 650 may be transmitted from the mobile terminal 400 to the controller 100 via the communication network 609 (communication paths 610, 612). Furthermore, the video data 650 may be transmitted from the mobile terminal 400 to one or both of the mirror-type display devices 1M (1M1, 1M2) via a communication path 611 (611-1, 611-2).

[0236] In the example of FIG. 46A , two mobile terminals 400 (400A, 400B) are provided, and each mobile terminal 400 is connected to the controller 100 and / or each mirror-type display device 1M (1M1, 1M2) via a direct connection communication path 600, similar to FIG. 45A . For example, video data 650 (content A) may be transmitted from one mobile terminal 400A to one mirror-type display device 1M1, with or without the controller 100, to display a virtual image 9 in one display area 5M1. Video data 650 (content B) may be transmitted from the other mobile terminal 400B to the other mirror-type display device 1M2, with or without the controller 100, to display a virtual image 9 in the other display area 5M2.

[0237] Similarly, it is possible for a plurality of mobile terminals 400 to be connected to a plurality of mirror-type display devices 1M via the communication network 609. Furthermore, with regard to these communication connections, a direct connection method and a method involving the communication network 609 may be mixed.

[0238] In the example of FIG. 46B , for example, two mobile terminals 400 (400A, 400B) are connected to the controller 100. One of the mobile terminals 400, 400A, is connected to the power source 420 and has sufficient battery power (e.g., 90%). Therefore, the mobile terminal 400A acts as a video source and transmits video data 650 to the controller 100. Based on the video data 650, the controller 100 transmits corresponding video data 650-1 and 650-2 to the respective mirror-type display devices 1M (1M1 and 1M2), causing the respective display areas 5M (5M1 and 5M2) to display virtual images 9. The other mobile terminal 400B is not connected to the power source 420 and has a low battery power (e.g., 10%). Therefore, the in-vehicle system 300 controls the mobile terminal 400B so that it does not act as a video source (described below).

[0239] 47 shows an example of video data of the video source mobile terminal 400 and an example of video display on the virtual image display device 1 under normal conditions. The video source mobile terminal 400 generates and stores video data (in other words, content) 4701. Note that the video data 4701 may be video data that the mobile terminal 400 references and acquires from an external server 3600 ( FIG. 36 ), etc. When the mobile terminal 400 is used as the video source to display video on the virtual image display device 1, it is basically unnecessary for the controller 100 to generate video in FIGS. 3 and 36 , in other words, for the controller 100 to be the video source. As a modified example, it is also possible to use the mobile terminal 400 as the video source for some of the video displayed in the display area 5 of the virtual image display device 1, and use the controller 100 as the video source for other video.

[0240] As a first example of video data 4701, video 4701A is navigation information created by car navigation application 411 ( FIG. 36 ) of mobile terminal 400. This video 4701A is generally similar to, for example, FIG. 11B , but also includes an image a1 of vehicle speed, an image a2 of an alert for pedestrians and the like, an image a3 representing driving straight ahead, and an image a4 representing a right turn.

[0241] When navigation information provided by the car navigation application 411 (FIG. 36) of the mobile terminal 400 is displayed as the virtual image 9, the car navigation system 150 in FIG. 3 may be used in combination, but is not essential.

[0242] As a second example of video data 4701, video 4701B is similarly navigation information created by car navigation application 411 ( FIG. 36 ) of mobile terminal 400. This video 4701A does not include images of vehicle speed or alerts, but does include images representing driving straight ahead and images representing turning right. In the second example, controller 100 is the video provider for the images of vehicle speed and alerts.

[0243] When the navigation information (image 4701A) of the first example is displayed as a virtual image 9, video data 650 corresponding to the video 4701A is transmitted from the mobile terminal 400 to the virtual image display device 1 via the controller 100 (communication path 602 and information transmission path 200). At this time, if the mobile terminal 400 is functioning normally and video transmission via the communication path 600 and information transmission path 200 is normal, the virtual image display device 1 can receive and acquire all of the video data 650 (video 4701A) in approximately real time. "Approximately real time" means that the time difference (delay) between the time of data transmission from the mobile terminal 400 and the time of data reception and display by the virtual image display device 1 is sufficiently small. The video data received and acquired by the virtual image display device 1 corresponding to the video 4701A from the video provider, i.e., the video to be displayed on the screen of the video display device 10, is shown as, for example, video 4711. As a result, the virtual image display device 1 can display the image 4711 based on the image data 650 on the screen of the image display device 10 in approximately real time. Then, image light corresponding to the image 4711 on the screen is emitted, and the virtual image 9 corresponding to the image 4711 can be displayed in the display area 5.

[0244] When the navigation information (image 4701B) of the second example is displayed as the virtual image 9, the video data 650 corresponding to the video 4701B is transmitted from the portable terminal 400 to the virtual image display device 1 via the controller 100 (communication path 602 and information transmission path 200). In the second example, the video 4702 is transmitted from the controller 100 simultaneously with the transmission of the video 4701B from the portable terminal 400. The video 4702 includes, for example, an image of the vehicle speed and an alert image. At this time, if the portable terminal 400 is functioning normally and the video transmission over the communication path 600 and the information transmission path 200 is normal, the virtual image display device 1 can receive and acquire all of the video data 650 (video 4701B and video 4702) in approximately real time. The video data received and acquired by the virtual image display device 1 corresponding to the video 4701B and video 4702 from the video provider, i.e., the video to be displayed on the screen of the video display device 10, is shown as, for example, video 4712. As a result, in the virtual image display device 1, the image 4712 based on the image data 650 can be displayed in approximately real time on the screen of the image display device 10. Then, image light corresponding to the image 4712 is emitted, and the virtual image 9 corresponding to the image 4712 can be displayed in the display area 5.

[0245] The first example above is an example in which the mobile terminal 400 can generate the same video information as the controller 100 or the car navigation system 150 ( FIG. 3 ) (for example, video information similar to the video information 250 in FIG. 10 ). The second example above is an example in which the mobile terminal 400 can generate some but not all of the same video information as the controller 100 or the car navigation system 150 (for example, cannot generate vehicle speed or alerts). The mobile terminal 400 can provide data and information such as video content that it can generate itself as information for driving assistance as a video source instead of the controller 100 or the like.

[0246] Furthermore, when the mobile terminal 400 generates video data such as driving assistance information, the generation is not limited to being performed only by its own application, etc. The mobile terminal 400 may appropriately acquire data and information required for generation (for example, the above-mentioned vehicle information 4, FIG. 3 ) from the controller 100 of the in-vehicle system 300, etc., and generate the video data from the acquired information (described later).

[0247] Note that the format and details, such as image size, of the navigation video information generated and displayed by, for example, the navigation application 411 ( FIG. 36 ) on the screen 405 of the mobile terminal 400 may differ from the navigation video information generated by the controller 100 or the virtual image display device 1 and displayed on the screen of the display area 5 of the virtual image display device 1. When such different navigation video information of the mobile terminal 400 is to be displayed on the screen of the display area 5 of the virtual image display device 1, the mobile terminal 400, the controller 100, or the virtual image display device 1 may appropriately convert data and information to match the format and details.

[0248] FIG. 50 shows a modified example in which the mobile terminal 400 acquires driving assistance information (vehicle information 4) from the controller 100 and generates video data. In the example of FIG. 50 , the mobile terminal 400 is connected to the virtual image display device 1 via a communication path 601 and to the controller 100 via a communication path 602. The controller 100 has the vehicle information 4 described above. The mobile terminal 400 acquires the vehicle information 4 (e.g., vehicle speed and alert information) from the controller 100 via the communication path 602 as information 5001 necessary for driving assistance. Based on the acquired information 5001, the mobile terminal 400, as an image provider, creates video data 5002 (e.g., similar to the video 4701A in FIG. 47 ) for displaying a virtual image 9 on the virtual image display device 1. In this example, a vehicle speed image a1 and an alert image a2 are created based on the information 5001. The mobile terminal 400 transmits the created video data 5002 to the virtual image display device 1 via the communication path 601. The virtual image display device 1 displays a virtual image 9 in the display area 5 based on the received video data 5002 .

[0249] FIG. 48 shows another example of video data from the mobile terminal 400, which is the video provider, and an example of a video display on the virtual image display device 1 under normal conditions. The example in FIG. 48 shows a third example of video data 4701 from the mobile terminal 400. The video 4701C is map information created by the map application 412 ( FIG. 36 ). The map information in the video 4701C is, for example, a map of the area surrounding destination c1. As an example of use, when a passenger user U2, such as in FIG. 41B, wants to visit a store or other location, the map application 412 of the mobile terminal 400B displays map information with the store or other location as the destination. The passenger then operates the virtual image display device 1 or the controller 100 (the controller 100 in this example) to request display of the mobile terminal 400B as the video provider on the virtual image display device 1. This causes the map image to be displayed in the display area 5 of the virtual image display device 1, allowing the passenger to show the map image to the driver, user U1.

[0250] When the map information (image 4701C) of the third example is displayed as the virtual image 9, the image data 650 corresponding to the image 4701C is transmitted from the mobile terminal 400 to the virtual image display device 1 via the controller 100 (communication path 602 and information transmission path 200). At this time, if the mobile terminal 400 is normal and the image transmission over the communication path 600 is normal, the virtual image display device 1 can receive and acquire all of the image data 650 (image 4701C) in approximately real time. The image data received and acquired by the virtual image display device 1 corresponding to the image 4701C from the image provider is the map image 4801. Based on this image 4801, the virtual image display device 1 creates and displays an image 4802 to be displayed on the screen of the image display device 10. Specifically, the map image 4801 is appropriately enlarged / reduced and arranged within the landscape-oriented image 4802. In this example, a map image 4801 is placed at the center of the image (screen) 4802 so as to fit the vertical width of the image (screen) 4802. As a result, the virtual image display device 1 can display the map image as a virtual image 9 in the display area 5 in approximately real time.

[0251] In the third example, the virtual image display device 1 performs image processing to display the target image in the display area 5, but this is not limitative, and the controller 100 may perform image processing.

[0252] As another example, a fourth example is a case in which an image 4803 of vehicle speed or the like is transmitted from the controller 100 simultaneously with transmission of an image 4701C of map information from the portable terminal 400. In the fourth example, the controller 100 combines the image 4701C of map information received from the portable terminal 400 with the image 4803 of vehicle speed or the like to create an image 4804 to be transmitted to the virtual image display device 1, and transmits the image 4804 to the virtual image display device 1 via the information transmission path 200. As shown in the figure, in the combined image 4804, the controller 100 places the image 4701C of map information as an image 4805 in a portion of the image area (screen) corresponding to the display area 5 of the virtual image display device 1. The area in which the image 4701C of map information is placed should preferably be an empty area (the upper left corner in this example) so as not to interfere with other image information necessary for driving (such as vehicle speed, alerts, and navigation information). The controller 100 transmits the synthesized image 4804 to the virtual image display 1 , and the virtual image display 1 displays the virtual image 9 in the display area 5 based on the image 4804 .

[0253] Not limited to the above example, for example, the virtual image display device 1 may combine the image received from the controller 100 as the image provider with the image received from the mobile terminal 400 as the image provider to create display data for displaying the virtual image 9 in the display area 5.

[0254] [3-10. Malfunction / Error in Virtual Image Display Due to Battery Deadness of Mobile Terminal] FIG. 49 illustrates an example in which, when a virtual image 9 is displayed on the virtual image display device 1 based on video data from the mobile terminal 400 that provides the video, as in the examples of FIGS. 47 and 48 , a malfunction or error occurs in the video transmission from the mobile terminal 400 and / or the virtual image display on the virtual image display device 1 due to a dead battery in the mobile terminal 400. For example, as in FIG. 44B , assume that the mobile terminal 400 that provides the video runs out of battery. In the example of FIG. 49 , the same video data 4701A (navigation information) as in the first example of FIG. 47 is used. In the example of FIG. 49 , the mobile terminal 400 transmits video data 650 corresponding to the video data 4701A to the virtual image display device 1 via the controller 100.

[0255] In the example of Figure 49, the mobile terminal 400 has a low battery level and changes its operation to reduce battery consumption (for example, transitions to power saving mode) and / or changes the order of priority processing, resulting in malfunction or an error in the transmission of video data 650 from the mobile terminal 400 to the virtual image display device 1 via the controller 100 and the display of the virtual image on the virtual image display device 1.

[0256] In the case of an error, the virtual image display device 1 cannot receive or acquire part or all of the video data 650. In the illustrated example, all of the video data 650 cannot be acquired (indicated by "none" in the video area (screen) 4900 of the video display device 10). In this case, the virtual image 9 corresponding to the video data 650 cannot be displayed in the display area 5 (indicated by "no display").

[0257] In the event of a malfunction, although some or all of the video data 650 can be transmitted via the communication path 602 or the information transmission path 200, due to a decrease in transmission speed or data loss, the virtual image display device 1 cannot receive or acquire the video data 650 at the ideal timing of approximately real time. In the illustrated example, the transmission of the video data 650 from the mobile terminal 400 to the controller 100 via the communication path 602 is delayed, and corresponding to this delay, the transmission of the video data 650 from the controller 100 to the virtual image display device 1 via the information transmission path 200 is also delayed. As a result, the virtual image display device 1 receives the video data 650 at a timing delayed from the ideal and displays the virtual image 9 in the display area 5. In other words, the virtual image display is delayed, and real-time performance cannot be ensured.

[0258] In the third embodiment, as in the first and second embodiments, such malfunctions or error states of the video transmission and virtual image display may be displayed in the display area 5.

[0259] [3-11. Notification Regarding the Status of the Portable Terminal] There are issues regarding the status of the battery, etc. of the portable terminal 400 as described above. Therefore, it is desirable that the portable terminal 400, which is a video source connected to the in-vehicle system 300, is always used while connected to the power source 420 ( FIG. 44A ), or that it be connected to the power source 420 to charge when the battery level is low. In response to this, in the third embodiment, the in-vehicle system 300 notifies the user of the status of the portable terminal 400 and urges the user to charge the battery, so as to minimize the risk of the portable terminal 400 running out of battery power. Specific examples are shown below.

[0260] FIG. 51A shows an example of a desirable state of the in-vehicle system 300 and the portable terminal 400. In FIG. 51A, the portable terminal 400, which is the video source, is connected to the power supply 420 and has sufficient battery power. Because the portable terminal 400 is connected to the power supply 420, the in-vehicle system 300 controls the portable terminal 400 to be permitted as the video source, and the virtual image display device 1 displays in the display area 5 or outputs audio to indicate that the portable terminal 400 is connected to the power supply 420. In the example of FIG. 51A, "Smartphone: Power Supply Connected" is displayed in a portion of the display area 5.

[0261] FIG. 51B illustrates an example of an undesirable state of the in-vehicle system 300 and the portable terminal 400, in which the portable terminal 400 is not connected to the power source 420 and the remaining battery power is low, resulting in a battery shortage. The in-vehicle system 300 controls the portable terminal 400 not to be accepted as a video source because the portable terminal 400 is not connected to the power source 420 and the remaining battery power is, for example, below a threshold value. The virtual image display device 1 also displays or outputs audio in the display area 5 to indicate that the portable terminal 400 is not connected to the power source 420 and that the remaining battery power is low, and to urge the user to charge the device. In the example of FIG. 51B, the message "Smartphone: Please charge" is displayed in a portion of the display area 5.

[0262] FIG. 52A illustrates an example of a virtual image 9 displayed in the display area 5 of the virtual image display device 1 to display a status notification regarding the remaining battery charge of the mobile terminal 400. The mobile terminal 400 is connected to the power source 420 and has sufficient battery charge. Image 5201 is an image of the status notification, displayed, for example, in the upper left corner of the display area 5. Image 5201 represents the remaining battery charge (e.g., 80%) of the mobile terminal 400 that is communicatively connected to the virtual image display device 1 and the power connection status of the mobile terminal 400, which is connected to the power source 420 (power connected). The details of image 5201 are not limited, and may be represented, for example, as a battery icon, similar to a smartphone. In addition, in response to the display of image 5201, a status notification such as "The smartphone's battery charge is 80%," "The smartphone is fully charged," or "The smartphone is connected to a power source" is output as audio output from a speaker 5211 provided in the virtual image display device 1, the controller 100, or the mobile terminal 400. Only one of display and audio output may be provided.

[0263] Furthermore, the display of the image 5201 and the audio output of the status notification do not need to be limited to constant output, and may be output temporarily when necessary. For example, they may be displayed when the remaining battery charge of the mobile terminal 400 falls below a threshold (lower limit) or exceeds a threshold (upper limit). Furthermore, this display may be displayed when the mobile terminal 400 is connected to the in-vehicle system 300 for communication, when the communication connection is released, at a fixed time interval during the communication connection, when the user performs an operation input, or when a fixed time has passed since the user performed no operation input.

[0264] FIG. 52B shows an example of a virtual image 9 displayed in the display area 5 of the virtual image display device 1 as a status notification regarding the remaining battery charge of the mobile terminal 400. The example shows a case in which the mobile terminal 400 is not connected to the power source 420 and the remaining battery charge is low. Image 5202 is an image of the status notification. Image 5202 represents the remaining battery charge (e.g., 20%) of the mobile terminal 400 that is communicatively connected to the virtual image display device 1 and the power connection status of the mobile terminal 400. The power connection status, for example, indicates a state in which the mobile terminal 400 is not connected to the power source 420, and is represented by an x ​​mark. Correspondingly, a status notification and a prompt to charge (in other words, an alert) are output from the speaker 5211 as audio output, such as "The smartphone's battery charge is 20%," "The smartphone's charge is low," or "Please promptly connect to a power source and charge."

[0265] 52B , for example, when the portable terminal 400 is not connected to the power source 420, power is consumed by use of the portable terminal 400, resulting in the remaining battery level dropping to, for example, 20%. This may occur if the user forgets to connect the portable terminal 400 to the power source 420, or if the user connects the portable terminal 400 to the power source 420 via a wired connection and then the wired cable becomes unplugged. When the in-vehicle system 300, for example, the controller 100, the virtual image display device 1, or the portable terminal 400, determines that the remaining battery level of the portable terminal 400 is below a set threshold (described later), it controls the in-vehicle system 300 to output a message urging the user to charge the portable terminal 400, as in the above example.

[0266] As in the above example, in the third embodiment, it is possible to clearly inform the user by displaying or outputting audio that the mobile terminal 400 needs to be sufficiently charged in order to properly display the video from the mobile terminal 400 as the virtual image 9. In response to this output, the user can take action such as charging the mobile terminal 400. As a result, it is possible to prevent malfunctions or errors in the virtual image display caused by a dead battery in the mobile terminal 400.

[0267] 53 is an explanatory diagram illustrating an example of control for grasping and determining the remaining battery charge of the portable terminal 400. The in-vehicle system 300 has a function for performing processing to grasp and determine the status of the portable terminal 400, such as the remaining battery charge, for the above-mentioned status notification and the like. In the example of FIG. 53 , for example, the controller 100, which serves as the control entity for the in-vehicle system 300, determines the remaining battery charge of the portable terminal 400. The controller 100 has a control function 5300 for this purpose. In this example, the portable terminal 400 has video data 5301 as a video providing source. Assume that the portable terminal 400 transmits video data 650 corresponding to the video data 5301 to the controller 100 via the communication path 602, and the controller 100 transmits the corresponding video data 650 to the virtual image display device 1 via the information transmission path 200, and the virtual image display device 1 displays a virtual image 9 based on the video data 650.

[0268] When the controller 100 is connected to the portable terminal 400 via the communication path 602, the controller 100 first confirms and determines whether the portable terminal 400 will be an image provider for the virtual image display device 1. For example, when the controller 100 receives a request / instruction 5311 to become an image provider from the portable terminal 400 or receives image data 650 from the portable terminal 400 during communication connection, the controller 100 may determine that the portable terminal 400 will be an image provider for the virtual image display device 1.

[0269] Furthermore, the control function 5300 of the controller 100 grasps and checks the status of the portable terminal 400, such as the remaining battery level. For example, the control function 5300 of the controller 100 performs communication 5312 with the portable terminal 400 to check the status of the portable terminal 400, including the remaining battery level. As a result, the control function 5300 of the controller 100 grasps the status of the portable terminal 400, such as the remaining battery level and whether or not the power source 420 is connected. The controller 100 may perform processing to check the status of the portable terminal 400, such as the remaining battery level, using a communication device (e.g., the configuration in FIG. 39 ), for example, before or after the portable terminal 400 is connected via the communication path 602. This timing may be periodically, for example. The status to be checked may include at least the remaining battery level / charging status of the portable terminal 400, and may also include whether or not the portable terminal 400 is connected to the power source 420, i.e., whether or not the portable terminal 400 is charging. In this example, it is assumed that at a certain point in time, the remaining battery level of the mobile terminal 400 is, for example, 50% and the power source 420 is not connected (disconnected).

[0270] As an example of status confirmation communication 5312, controller 100 transmits a request for status confirmation to portable device 400. In response to the request, portable device 400 transmits a response indicating its status, including its own battery remaining capacity / charge status (50% in this example), to controller 100. From the response, controller 100 grasps the status of portable device 400 as information. Control function 5300 of controller 100 stores in memory the status of portable device 400 grasped based on communication 5312, here portable device battery remaining capacity status 5322 (in other words, portable device status information).

[0271] As another example, information describing the state of portable terminal 400 may be included as part of video data 650 transmitted from portable terminal 400 or in attribute information accompanying video data 650.

[0272] When the mobile terminal 400 is used as the video source, the in-vehicle system 300 has a preset value (in other words, control setting information) for a threshold value (e.g., a lower limit value) related to the remaining battery capacity of the mobile terminal 400. In this example, the control function 5300 of the controller 100 stores a mobile terminal remaining battery capacity threshold value (lower limit value) 5321 in the memory as the setting value.

[0273] The control function 5300 of the controller 100 compares the mobile device battery remaining capacity status 5322 confirmed and grasped at a certain point in time with the mobile device battery remaining capacity threshold (lower limit value) 5321, and if it is below the threshold, determines that it is unsuitable as a video provider, and if it is above the threshold, determines that it is suitable as a video provider.

[0274] The lower part of Figure 53 (A) shows an example of a comparison between a mobile terminal battery remaining capacity threshold (lower limit) 5321 and a mobile terminal battery remaining capacity state 5322. In this example, the lower limit threshold 5321 is 25%. In this case, the remaining capacity state 5322, 50%, is equal to or greater than the threshold 5321. Therefore, the mobile terminal 400 is determined to be appropriate and acceptable as an image providing source. In this case, the mobile terminal 400 is controlled to output a status notification in the virtual image display device 1, as in the example of Figure 52A described above.

[0275] The control function of the controller 100 performs control processing related to image display on the virtual image display device 1 based on the grasped state of the portable terminal 400. In a specific example, the control function 5300 of the controller 100 cooperates with the virtual image display device 1 to transmit, to the virtual image display device 1, instructions / information 5313 for control corresponding to the "appropriate" state. The instructions / information 5313 may include information representing the state of the portable terminal 400. The instructions / information 5313 may also include image information for displaying a status notification on the virtual image display device 1. The virtual image display device 1 (particularly the image processing unit 1016 in FIG. 40 , etc.) outputs, for example, a status notification on the display area 5 based on the instructions / information 5313.

[0276] As another example, in (B), when the threshold value 5321, which is the lower limit, is 25%, the remaining charge state 5322, which is 20%, is less than the threshold value 5321. Therefore, the mobile terminal 400 is determined to be inappropriate and unacceptable as a video provider. In this case, as in the example of FIG. 52B described above, the virtual image display device 1 is controlled to output a status notification such as a prompt to charge. The control function 5300 of the controller 100 transmits, to the virtual image display device 1, an instruction / information 5313 for control corresponding to the "inappropriate" state. Based on the instruction / information 5313, the virtual image display device 1 outputs the above-described status notification, a prompt to charge, etc.

[0277] The fact that the mobile terminal 400 is "unsuitable" as a video source corresponds to the fact that the video data transmission based on the video source and the virtual image display on the virtual image display device 1 are likely to cause the malfunctions or errors described above (e.g., FIG. 49 ), that real-time performance cannot be ensured, etc. Furthermore, this "unsuitability" corresponds to the fact that the mobile terminal 400 should be charged.

[0278] In the above example, the remaining battery level is used as a condition for determining whether the mobile terminal 400 is suitable, but other conditions for determination may also be used, such as whether the power source 420 is connected. For example, if the power source 420 is connected, it may be determined to be suitable, and if the power source 420 is not connected, it may be determined to be unsuitable. Both of these conditions may also be used.

[0279] The control example by the controller 100 in FIG. 53 can also be applied to the case where the mobile terminal 400 transmits the video data 650 to the virtual image display device 1 via the communication path 601.

[0280] Fig. 54 shows a variation of Fig. 53 in which the virtual image display device 1 is the main control entity. The virtual image display device 1 has a control function 5400. In the example of Fig. 54, the mobile terminal 400 has video data 5401, is connected to the virtual image display device 1 via a communication path 601, and transmits video data 650 corresponding to the video data 5401. This transmission does not involve the controller 100.

[0281] When the virtual image display device 1 is connected to the mobile terminal 400 via the communication path 601, the virtual image display device 1 confirms and determines whether the mobile terminal 400 will be an image provider for the virtual image display device 1. For example, when the virtual image display device 1 receives a request / instruction 5411 to become an image provider from the mobile terminal 400 or receives image data 650 from the mobile terminal 400 during communication connection, the virtual image display device 1 may determine that the mobile terminal 400 will be an image provider for the virtual image display device 1.

[0282] Furthermore, the control function 5400 of the virtual image display device 1 grasps and checks the remaining battery level of the mobile terminal 400. For example, the control function 5400 of the virtual image display device 1 performs communication 5412 with the mobile terminal 400 to check the status of the mobile terminal 400, including the remaining battery level. As a result, the control function 5400 of the virtual image display device 1 grasps the status of the mobile terminal 400, such as the remaining battery level. The control function 5400 of the virtual image display device 1 stores the mobile terminal battery remaining level status 5422 grasped based on the communication 5412 in a memory.

[0283] In this example, the control function 5400 of the virtual image display device 1 stores a mobile device battery remaining capacity threshold (lower limit) 5421 in memory as a preset value. The control function 5400 of the virtual image display device 1 compares the mobile device battery remaining capacity status 5422 grasped at a certain point in time with the mobile device battery remaining capacity threshold (lower limit) 5421, and if the remaining capacity status is less than the threshold, determines that the device is unsuitable as an image providing source (similar to FIG. 53 ). The control function 5400 of the virtual image display device 1 performs control according to the determination. If determined to be appropriate, the control is performed so that the virtual image display device 1 outputs a status notification, such as the example of FIG. 52A described above. If determined to be unsuitable, the control is performed so that the virtual image display device 1 outputs a status notification, such as a prompt to charge the device, as the example of FIG. 52B described above.

[0284] The control example of the virtual image display device 1 in FIG. 54 can also be applied to the case where the mobile terminal 400 transmits the video data 650 via the controller 100 over the communication path 602.

[0285] In addition, the virtual image display device 1 in Figure 54 may notify the controller 100 of the state of the mobile terminal 400 that has been grasped in connection with the mobile terminal 400 by sending information 5313 to the controller 100.

[0286] The thresholds 5321 and 5421 described above are not limited to fixed values ​​and may be variable. For example, assume that the in-vehicle system 300 is capable of transmitting the video data 650 and displaying a virtual image in approximately real time when the mobile terminal 400 has a certain remaining battery charge, but that continuing to transmit the video data 650 may predict a malfunction or error. In this case, the thresholds 5321 and 5421 described above are set to relatively large values ​​to prompt charging earlier. This can prevent malfunctions and errors from occurring. On the other hand, although this depends on the content of the target video data, the thresholds 5321 and 5421 may be set to relatively small values ​​to maximize the use of the battery power of the mobile terminal 400.

[0287] 53 and 54 show the case where the controller 100 or the virtual image display device 1 is the control entity and grasps the status of the mobile terminal 400, such as the remaining battery level, but the status of the mobile terminal 400, such as the remaining battery level, may be grasped by an application or the like on the mobile terminal 400, and the status may be notified from the mobile terminal 400 to the controller 100 or the virtual image display device 1. Similarly, the controller 100, the virtual image display device 1, or the mobile terminal 400 may grasp the communication status, such as the radio wave conditions and communication speed, related to communication on the communication path 600 of the mobile terminal 400. The in-vehicle system 300 may use the grasped communication status to perform control related to selection and switching of the video provider.

[0288] [3-13. Control of Switching of Video Providers] FIG. 55 is a table summarizing the control of switching of video providers in the third embodiment. In this example, at least the mobile terminal 400, the controller 100, and the virtual image display device 1 are candidates / options for the video provider when the virtual image display device 1 displays the virtual image 9. The in-vehicle system 300, particularly the controller 100 or the virtual image display device 1 as the control entity, appropriately selects and controls which of these devices will be used as the video provider to display the virtual image 9 at each point in time. In this case, the in-vehicle system 300 determines one video provider device to actually use from among multiple candidate video provider devices based on control priority. Note that the candidate device must be a device equipped with the functionality to become a video provider and must be permitted to become a video provider in terms of system design / settings. For example, as shown in FIG. 6 above, a display device 1 that does not have the functionality to generate images internally will naturally not be a candidate video provider.

[0289] The table in FIG. 55A shows a portion of the case where the "current video provider" is the mobile terminal 400. As the "video provider switching destination," candidates are listed for each priority. In this example, the priority is 1, 2, 3, and 4, from high to low, from left to right. One of the candidates is "display stop," shown with priority 4, which represents control to stop the virtual image display on the virtual image display device 1 as a final measure, in other words, control to hide the image. For ease of identification, the control for each row is referred to as control 5501, etc.

[0290] In the in-vehicle system 300, the control to be applied for the control of selecting and switching the video provider using the above priority is designed and set in advance. The in-vehicle system 300 may apply multiple controls. In this case, the in-vehicle system 300 is equipped with functions for multiple controls, and the control to be applied is selected and switched as appropriate. The control to be applied may be selected and switched as appropriate based on settings or instructions by the user. For example, a GUI screen provided to the user may allow the user to select from each control in a table such as that shown in FIG. 55.

[0291] For example, in control 5501, priority 1 is assigned to mobile terminal 400, priority 2 is assigned to controller 100, priority 3 is assigned to virtual image display device 1, and priority 4 is assigned to stop display. When switching the video provider from a situation where the "current video provider" is mobile terminal 400, mobile terminal 400 is first selected as a candidate for the destination of the video provider, with priority 1. In table (A), mobile terminal 400 as the "current video provider" and mobile terminal 400 with priority 1 as a candidate for the destination of the video provider are different devices. In a situation where there is an appropriate mobile terminal 400 as a destination of the priority 1, the appropriate mobile terminal 400 is determined as the destination of the video provider. "Appropriate" refers to meeting predetermined conditions, such as having sufficient remaining battery power, i.e., not causing malfunctions or errors in the video transmission or virtual image display, as in the example described above. There are various detailed cases in which the mobile terminal 400 is a candidate for the video provider, but in this table they are all illustrated together as a single "mobile terminal" cell, which will be discussed later (Figure 56A).

[0292] In a situation where there is no suitable mobile terminal 400 as a switching destination for the video provider with priority 1, the controller 100 becomes a candidate for the switching destination for the video provider with priority 2. In a situation where there is a suitable controller 100 with priority 2, that controller 100 is determined as the switching destination.

[0293] In a situation where there is no appropriate controller 100, in other words, when the condition of the controller 100 is not good, the virtual image display device 1 becomes the next candidate for switching to the image provider destination with priority 3. In a situation where there is an appropriate virtual image display device 1 with priority 3, that virtual image display device 1 is determined as the switching destination.

[0294] In a situation where there is no appropriate virtual image display device 1, in other words, when the virtual image display device 1 is not in good condition, the last decision is to stop the display as a priority of 4. Note that, for example, it is assumed that the mobile terminal 400, the controller 100, and the virtual image display device 1 that are candidates in control 5501 all have a function to become an image provider. Devices that do not have that function can be considered not to be candidates.

[0295] For the controls in each row from control 5502 onwards, candidates are determined in order of priority, similar to control 5501. For control 5502, priority 1 is the controller 100, priority 2 is the portable terminal 400, priority 3 is the virtual image display 1, and priority 4 is display stop. For control 5503, priority 1 is the controller 100, priority 2 is the virtual image display 1, priority 3 is the portable terminal 400, and priority 4 is display stop. For control 5504, priority 1 is the virtual image display 1, priority 2 is the portable terminal 400, priority 3 is the controller 100, and priority 4 is display stop. For control 5505, priority 1 is the virtual image display 1, priority 2 is the controller 100, priority 3 is the portable terminal 400, and priority 4 is display stop.

[0296] The table in Fig. 55 (B) shows a portion of the case where the "current video provider" is the controller 100. In control 5511, priority 1 is the mobile terminal 400, priority 2 is the virtual image display device 1, and priority 3 is display stop. In control 5512, priority 1 is the virtual image display device 1, priority 2 is the mobile terminal 400, and priority 3 is display stop.

[0297] For example, in control 5511, first, the mobile terminal 400 becomes a candidate with priority 1. In a situation where an appropriate mobile terminal 400 with priority 1 is available, the appropriate mobile terminal 400 is determined as the switching destination. In a situation where an appropriate mobile terminal 400 with priority 1 is not available, the virtual image display device 1 becomes a candidate with priority 2. In a situation where an appropriate virtual image display device 1 with priority 2 is available, the virtual image display device 1 is determined as the switching destination. In a situation where an appropriate virtual image display device 1 with priority 2 is not available, finally, the response of stopping the display is determined with priority 3.

[0298] 55C shows a portion of the case where the "current video provider" is the virtual image display device 1. In control 5521, priority 1 is the mobile terminal 400, priority 2 is the controller 100, and priority 3 is display stop. In control 5522, priority 1 is the controller 100, priority 2 is the mobile terminal 400, and priority 3 is display stop.

[0299] For example, in control 5521, first, the portable terminal 400 becomes a candidate with priority 1. In a situation where an appropriate portable terminal 400 with priority 1 is available, that appropriate portable terminal 400 is determined as the switching destination. In a situation where an appropriate portable terminal 400 with priority 1 is not available, next, the controller 100 becomes a candidate with priority 2. In a situation where an appropriate controller 100 with priority 2 is available, that controller 100 is determined as the switching destination. In a situation where an appropriate controller 100 with priority 2 is not available, finally, a response of stopping the display is determined with priority 3.

[0300] Note that, for example, as shown by priority 4 in the table (A), if the candidates with priorities 1 to 3 cannot become the video provider, the last resort is to "stop display." However, for example, because the controller 100 is designed to ensure high reliability, a situation in which the controller 100 cannot become the video provider almost never actually occurs. Furthermore, other than "stop display," control may be performed to continue display. As an example, control may be performed such that, after comparing the communication and remaining battery status of the mobile device that is the current video provider with the status of the candidates with priorities 1 to 3, video provision from the mobile device that is the current video provider is continued rather than switching to a candidate with priorities 1 to 3.

[0301] [3-14. Multiple Mobile Terminals as Video Providers] Fig. 56A shows various detailed examples of the "Mobile Terminal" cell as a candidate for switching to the video provider in Fig. 55. In detail, the "Mobile Terminal" cell in Fig. 55 has at least four cases shown in Fig. 56A. For example, when control 5501 in Fig. 55 is applied, in detail, there can be four controls corresponding to the four cases.

[0302] In case 5601, when a search by the in-vehicle system 300 finds a newly connected mobile terminal 400 (e.g., a second smartphone) other than the mobile terminal 400 (e.g., a first smartphone) that is the current video provider, the single mobile terminal 400 (second smartphone) is selected as a candidate for switching the video provider.

[0303] In case 5602, when the search finds two or more mobile terminals 400 (e.g., second, third, etc. smartphones) other than the mobile terminal 400 (e.g., a first smartphone) that is the current video provider, these two or more mobile terminals 400 are set as candidates for the destination of the video provider, and one mobile terminal 400 is selected from them.

[0304] In case 5603, the user selects and designates one mobile terminal 400 connected to the in-vehicle system 300 as the destination of the video provider switch. For example, when the current video provider is a first smartphone, the user designates a second smartphone as the destination of the switch by a predetermined operation or a pre-set setting. When the second smartphone is communicatively connected to the in-vehicle system 300, the second smartphone is determined as the destination of the switch based on this designation. Note that case 5603 can also be considered as an example or part of the method of selecting one mobile terminal 400 for the other cases 5601, 5602, and 5604.

[0305] Case 5604 is a case where the current video provider mobile terminal 400 is prioritized over one or more other mobile terminals 400 found as a result of the search, and is maintained as the video provider. In other words, this is a case where a plurality of mobile terminals 400, including the current video provider mobile terminal 400, are selected as candidates, and the current video provider mobile terminal 400 is selected from these candidates. For example, in the case of control 5501 in FIG. 55A, a search finds another mobile terminal 400 (e.g., a second smartphone) with priority 1 for the current video provider mobile terminal 400 (e.g., a first smartphone). In this case, the first smartphone is selected as the video provider as a result of comparing the two mobile terminals 400, for example, as a result of determining which is more appropriate as a video provider.

[0306] When the case of FIG. 56A is considered in combination with the tables of FIG. 55B and FIG. 55C, the result is as follows. In the tables of FIG. 55B and FIG. 55C, the current video source is not the mobile terminal 400, so the "current video source" described in each case of FIG. 56A can be replaced with the controller 100 or the virtual image display device 1. For example, in the control 5511 of FIG. 56B, the current video source is the controller 100. As a result of the search, suppose two mobile terminals 400 are found as mobile terminals 400 with priority 1, as in case 5602. If an appropriate mobile terminal 400 can be selected from the two, that one becomes the switching destination. If no selection is possible, the virtual image display device 1 with priority 2 becomes a candidate.

[0307] There are various methods for selecting one mobile terminal 400 for case 5602 in Fig. 56A and the like. Fig. 56B shows various methods and processing examples for selecting one mobile terminal 400 from multiple candidate mobile terminals 400. Each row represents method 5611 and the like. This selection of one mobile terminal may be performed automatically by the in-vehicle system 300 based on a predetermined decision, or the user may be asked to make the selection.

[0308] For example, as shown in method 5611, the in-vehicle system 300 outputs (by display or sound) to the user a notification that two or more mobile terminals 400 (including the mobile terminal 400 if the current video provider is a mobile terminal 400) have been found as a result of the search, along with information about the multiple mobile terminals 400. The in-vehicle system 300 then asks the user which one of the mobile terminals 400 to use as the destination for switching the video provider, and allows the user to select one. At this time, if the user chooses to maintain the current video provider mobile terminal 400 (e.g., the first smartphone) as is, in other words, not to switch, this is acceptable.

[0309] Alternatively, as shown in method 5612, a method may be used in which a user pre-configures one or more specific mobile terminals 400 that can serve as video providers. When multiple mobile terminals 400 are configured, a priority order for the mobile terminals 400 to serve as video providers is set. Each mobile terminal 400 is identified by its terminal ID, communication address, or the user ID of the associated user (e.g., the owner). This method selects a mobile terminal 400 based on the user's mobile terminal. Based on this user configuration, the in-vehicle system 300 can automatically select and determine one mobile terminal 400 as the video provider in accordance with the user configuration. For example, the mobile terminal 400 of a specific user (e.g., the driver) is permitted to serve as the video provider. Furthermore, for example, when multiple family members use multiple mobile terminals 400 and are riding in the vehicle 2, a priority order for the video provider is set for each family member. For example, the mother's mobile terminal may be set as the first priority, the father's mobile terminal may be set as the second priority, and so on.

[0310] As an example of a case where the in-vehicle system 300 (e.g., the controller 100) makes an automatic decision to select a mobile terminal 400 that has a history of communication connection with the in-vehicle system 300 in the past, as shown in method 5613. For example, the selection may be based on the fact that the mobile terminal 400 has been an appropriate video source at least once. The mobile terminal with the most history may be selected.

[0311] In another example, as shown in method 5614, among multiple mobile terminals 400, the one with the most remaining battery power or the one currently connected to power source 420 may be selected based on confirmation and understanding of the remaining battery power, etc. (Figure 53, etc.).

[0312] In another example, as shown in method 5615, for multiple mobile terminals 400, the one with the best and most stable communication status, or the one with the fastest communication speed, may be selected based on checking and understanding the communication status (e.g., radio wave conditions).

[0313] Furthermore, when switching the video provider from mobile terminal 400 to another device, confirmation and approval may be obtained from the user of mobile terminal 400, the previous video provider (described later). Alternatively, when the switching destination is mobile terminal 400, confirmation and approval may be obtained from the user of mobile terminal 400, the switching destination (described later). For example, in-vehicle system 300 outputs (by display or sound) a message that switching is about to be performed from previous mobile terminal 400 to new mobile terminal 400, and whether or not the switching is approved. In response to this output, the user of previous mobile terminal 400 inputs an operation as to whether or not the switching is approved. If approval is input, in-vehicle system 300 executes the switching.

[0314] The cases in FIG. 56A and the methods in FIG. 56B can also be applied in combination.

[0315] [3-15. Conditions and Triggers for Switching Video Providers] In the third embodiment (FIG. 36 and subsequent figures), when, for example, the portable terminal 400 provides video content as a video provider and causes the virtual image display device 1 to display a virtual image 9, the in-vehicle system 300 selects and switches the video provider based on predetermined conditions and triggers and control such as that shown in FIG. 55 . As described above, the predetermined conditions and triggers may be when a new portable terminal 400 is connected for communication, but other examples include the following. Control is possible using at least one of the conditions and triggers.

[0316] FIG. 56C is a table summarizing predetermined conditions and triggers for switching the video provider. Condition #1 occurs when a new mobile terminal 400 is communicatively connected to the in-vehicle system 300. Condition #2 occurs when there is a malfunction or error in the communication path in communication between the mobile terminal 400 and the virtual image display device 1 or the controller 100. In other words, there is a malfunction or error in the video transmission, such as a decrease in communication speed or a communication interruption. Condition #3 occurs when the mobile terminal 400 is in a state where the battery is low or is not connected to a power source. Condition #4 occurs when the mobile terminal 400 is used to respond to a priority event (described below), such as an incoming call.

[0317] The in-vehicle system 300, particularly the controller 100 and the virtual image display device 1 as the control entities, controls the selection and switching of the image provider based on the above-described predetermined conditions and triggers while applying control using priorities such as those shown in Fig. 55. This allows the virtual image display device 1 to continue displaying a virtual image based on the image provider even if there is a problem with the communication path, a low battery, an incoming call, or the like.

[0318] 57A shows an example of the method of selecting one mobile terminal 400 in FIG. 56B above, in which a user selects one mobile terminal 400 from a plurality of mobile terminals 400 on a GUI screen displayed in the display area 5 of the virtual image display device 1. Note that screen displays and user operation inputs related to user settings and the like are permitted while the vehicle 2 is stopped.

[0319] FIG. 57A illustrates a case in which the controller 100 or the virtual image display device 1, acting as the control entity of the in-vehicle system 300, checks and searches for the mobile terminal 400 communicatively connected to the in-vehicle system 300, and finds, for example, smartphone A, smartphone B, and smartphone C. For example, smartphone A is the current video provider. This screen displays a guide such as, "Please select a video provider." This screen displays "Smartphone A," "Smartphone B," and "Smartphone C" as candidates / options for the video provider. The user selects one of the options on this screen by inputting a predetermined operation to the virtual image display device 1. This predetermined operation can be input, for example, by an operation switch 2D provided on the steering wheel 2B ( FIG. 37 ) of the vehicle 2. For example, when "Smartphone B" is selected, the new video provider is switched to smartphone B. The UI is not limited to such a display UI, and may be a UI for audio input / output.

[0320] 57B is a similar example, but shows a GUI screen displayed on the display panel of the mobile terminal 400, in which the user is asked to select one mobile terminal 400 from multiple mobile terminals 400. The mobile terminal 400 shown in the figure corresponds to, for example, smartphone A, but the same applies to other mobile terminals 400. The user selects one of the options on this screen by inputting a predetermined operation to the mobile terminal 400. For example, if "smartphone B" is selected, the new video provider is switched to smartphone B.

[0321] 58 shows another example of a screen display. Assume that there are multiple users, such as user U1 who is the driver and user U2 who is a passenger, in a vehicle 2, as in (B) of FIG. 41 . User U1 possesses smartphone A (portable terminal 400A), and user U2 possesses smartphone B (portable terminal 400B). Assume that initially, smartphone A of user U1 is used as the image provider, and a virtual image is displayed on the virtual image display device 1. In this example, the image is transmitted without the intervention of the controller 100, and the controller 100 is not shown.

[0322] Next, assume that user U2 connects smartphone B to the virtual image display device 1 for communication. As shown in the figure, user U2 inputs an instruction to the virtual image display device 1 on a screen 5802 of smartphone B to display an image of the virtual image 9 using the user's own smartphone B as a new image provider. In the example of screen 5802, it is displayed that the current image provider is smartphone A of user U1, and a button 5802b is provided for displaying an image of the virtual image 9 using the user's own smartphone B as the image provider. In addition, a field 5802c is provided in which the image to be displayed can be confirmed and specified.

[0323] When user U2 selects the target video and presses button 5802b, smartphone B transmits an instruction / request 5803 to the virtual image display device 1 to display the video of the virtual image 9 using itself as a new video provider. The virtual image display device 1 makes a decision to select and switch the video provider in response to the instruction / request 5803. In this example, the virtual image display device 1 transmits information 5804 to smartphone A (corresponding user U1), the current video provider, for confirmation and approval of whether or not it is OK to switch to smartphone B (corresponding user U2). Based on the information 5804, smartphone A displays information on screen 5801, as shown in the figure, for confirmation and approval of whether or not it is OK to switch to smartphone B as the video provider. In the example of screen 5801, information such as a request from smartphone B and a YES / NO button for confirmation and approval of whether or not it is OK to switch smartphone B as the video provider is displayed. User U1 views the information and presses the YES button if they agree. When the YES button is pressed, the smartphone A transmits information 5805 representing YES (approval) to the virtual image display device 1. Based on the information 5805 representing YES (approval), the virtual image display device 1 switches the image provider from the smartphone A to the smartphone B.

[0324] For example, the virtual image display device 1 switches the image provider from smartphone A to smartphone B in information (management information) 5800 stored in the virtual image display device 1 and indicating which device is the image provider. The virtual image display device 1 transmits information 5806 to smartphone B indicating that smartphone B has been accepted as the image provider. Based on the information 5806, smartphone B becomes the new image provider and transmits image data 650 of the target image to the virtual image display device 1. The virtual image display device 1 displays an image of a virtual image 9 in the display area 5 based on the image data 650 received from smartphone B.

[0325] The above-described example of switching control is a case where the virtual image display device 1 is the control subject, but it can also be applied to a case where the controller 100 is the control subject. In addition, the above-described example of the screen of the mobile terminal 400 may be similarly applied to the screen of the display area 5 of the virtual image display device 1.

[0326] 59A shows an example of a case where a mobile terminal 400 that will be a video provider is set by user setting on a GUI screen of the virtual image display device 1, for example, corresponding to method 5612 of FIG. 56B . In this example screen, a user can assign a priority 5901 and set in advance one or more specific mobile terminals 400 (corresponding mobile terminal information) that will be candidates for the video provider. Although not shown, information such as an application that will be the video provider may be set in advance. Similar user setting may be possible on the GUI screen of each mobile terminal 400. The in-vehicle system 300 (e.g., the controller 100) selects and switches the video provider based on such user setting information.

[0327] 59B shows an example of the control using the priority of FIG. 55 , in which the priority of the video provider is set by the user on the GUI screen of the virtual image display device 1. In this example screen, the user can set in advance the priority 5901B for the controller 100, the virtual image display device 1, and the mobile terminal 400 as types of devices that are candidates for the video provider. Similar user settings may be possible on the GUI screen of each mobile terminal 400. The in-vehicle system 300 (for example, the controller 100) selects and switches the video provider based on such user setting information.

[0328] [3-18. Terminating Video Provision from a Video Provider] Furthermore, with regard to control using priorities as shown in FIG. 55 , for example, when the video provider is switched from one mobile terminal 400 (e.g., smartphone A) to another mobile terminal 400 (e.g., smartphone B), and then the video provision from the other mobile terminal 400 (e.g., smartphone B) is to be terminated, for example, the following control method can be applied.

[0329] One control method is as follows. For example, when control 5501 is applied, it is assumed that the display of the virtual image 9 has been completed using another mobile terminal 400 (e.g., smartphone B) with priority 1 as the video provider. In this case, the in-vehicle system 300, for example, the controller 100, updates smartphone B to a state where it is not a video provider. At this point, there is no video provider. After that, if a request to become a new video provider is issued from any device, that device can be set as the video provider.

[0330] Another control method is as follows. For example, when control 5501 is applied, it is assumed that the display of the virtual image 9 using another mobile terminal 400 (e.g., smartphone B) with priority 1 as the video provider has ended. In this case, the in-vehicle system 300, for example, the controller 100, updates the state of smartphone B to a state where it is not the video provider. At the same time, the controller 100 selects and determines one video provider. For this purpose, the controller 100 refers to the video provider before the switch. In the case of control 5501, the video provider before the switch is the mobile terminal 400 (e.g., smartphone A). The controller 100 checks whether it is possible to make smartphone A the video provider again, and if it is possible, in other words, if smartphone A is in an appropriate state, sets smartphone A as the video provider again. This returns the video provider to the mobile terminal 400 (400A) before the switch. This control method is designed to avoid a state where there is no video provider.

[0331] Another control method is as follows. For example, when control 5501 is applied, it is assumed that the display of the virtual image 9 using another mobile terminal 400 (e.g., smartphone B) with priority 1 as the video provider has been completed. In this case, the in-vehicle system 300, for example, the controller 100, updates the state of smartphone B to a state where it is not the video provider. At the same time, the controller 100 selects and determines the next video provider in accordance with the priority of control 5501. In the case of control 5501, the controller 100 with the next priority 2 is referenced, and if the controller 100 is in an appropriate state, the controller 100 is set as the video provider.

[0332] The above description is based on the basic premise that there is only one video provider per time point in the in-vehicle system 300. As a modified example, two or more video providers can be simultaneously displayed per time point. For example, as shown in FIG. 38C , when the display area 5 (5M) can be divided into multiple screens (e.g., 5M1 and 5M2), it is also possible to use different video providers for each divided screen (e.g., two mobile terminals 400 in FIG. 46 ). Furthermore, as shown in the second example of FIG. 47 , it is also possible to display a composite video within a single display area 5, with both the controller 100 and the mobile terminal 400 serving as video providers. The composite display may be a side-by-side display or a superimposed display.

[0333] [3-19. Switching of Video Provider: #2: Communication State] FIG. 60 shows an example of switching of the video provider when the condition #2 in FIG. 56C is applied. The in-vehicle system 300 in FIG. 60 shows a case where two mobile terminals 400 (400A, 400B) are connected. Initially, in state A, the mobile terminal 400A is used as the video provider, and a virtual image 9 is displayed on the virtual image display device 1 via the controller 100 based on video data A (6001). In this example, the controller 100 is the control entity. The mobile terminal 400A transmits video data 650 corresponding to the video data 6001 to the controller 100 via the communication path 602. The controller 100 transmits the corresponding video data 650 to the virtual image display device 1 via the information transmission path 200. The virtual image display device 1 displays the virtual image 9 in the display area 5 based on the video data 650. In state A, the communication path 602 and the information transmission path 200 are in a good communication state (indicated by a circle), and there are no malfunctions or errors. The virtual image display device 1 displays a virtual image in a good state, i.e., the image data 6001 is displayed in approximately real time in response to transmission from the mobile terminal 400A.

[0334] On the other hand, mobile terminal 400B is connected to controller 100 via, for example, communication path 604. Here, when limiting the video provider to one, controller 100 performs control so that mobile terminal 400B is not set as a video provider.

[0335] State B is, for example, a case where the communication path 602 with the mobile terminal 400A malfunctions (for example, a decrease in communication speed) or experiences an error (for example, a communication interruption). Malfunctions are indicated by a triangle, and errors are indicated by an x. The controller 100 uses the communication function to detect malfunctions or errors in the communication path 602. Based on this detection, the controller 100 performs control to switch the video provider. The controller 100 selects a device to switch to. For example, when the control 5501 in FIG. 55 is applied, the controller 100 assigns priority 1 to the mobile terminal 400B as a candidate for the switching destination. Note that in this example, the mobile terminal 400B holds video data B (6002) and can be a video provider. In other words, it is acceptable to switch to a virtual image display using video data B instead of video data A.

[0336] Controller 100 determines whether portable terminal 400B is appropriate. To do so, controller 100 checks the communication state of communication path 604 with portable terminal 400B. In state B, the check results show that communication path 604 and information transmission path 200 are in good communication states, with no malfunctions or errors. Therefore, controller 100 determines that portable terminal 400B is appropriate and decides to switch the video provider from portable terminal 400A to portable terminal 400B.

[0337] The controller 100 transmits, for example, an instruction / request to switch the video provider to the mobile terminal 400B, and when the mobile terminal 400B becomes the video provider based on the instruction / request, the mobile terminal 400B transmits video data 650 corresponding to video data B (6002) to the controller 100 as the video provider. The controller 100 transmits the corresponding video data 650 to the virtual image display device 1 via the information transmission path 200, and the virtual image display device 1 displays a virtual image 9 in the display area 5 based on the video data 650. As a result, the virtual image display on the virtual image display device 1 is switched from the image based on video data A to the image based on video data B. The virtual image display on the virtual image display device 1 is in a good state. That is, the video data 6002 is displayed in approximately real time in response to the transmission of the video data 6002 from the mobile terminal 400B.

[0338] When the video data B has the same video content (e.g., navigation information) as the video data A, the same video content continues to be provided to the user. Note that the control as shown in Fig. 60 can also be applied to the case where the virtual image display device 1 is the main control entity.

[0339] [3-20. Switching of Video Providers: #3: Battery Status] FIG. 61 shows an example of switching of video providers when the condition #3 in FIG. 56C is applied. The in-vehicle system 300 in FIG. 61 shows a case where two mobile terminals 400 (400A, 400B) are connected. Initially, in state A, the mobile terminal 400A is used as the video provider, and a virtual image 9 is displayed on the virtual image display device 1 via the controller 100 based on video data A (6101). In this example, the controller 100 is the control entity. The mobile terminal 400A transmits video data 650 corresponding to video data 6001 to the controller 100 via the communication path 602. The controller 100 transmits the corresponding video data 650 to the virtual image display device 1 via the information transmission path 200. The virtual image display device 1 displays the virtual image 9 in the display area 5 based on the video data 650. In state A, the mobile terminal 400A is connected to the power source 420 (indicated by a circle), in other words, is being charged, and the remaining battery level is, for example, 90%. The virtual image display on the virtual image display device 1 is in a good state.

[0340] State B is, for example, a case where portable terminal 400A is disconnected from power source 420 (indicated by an x) and the remaining battery level has dropped to, for example, 25%. Controller 100 grasps the status of portable terminal 400A, such as the remaining battery level, as described above. Based on this grasp, controller 100 performs control to switch the video provider. Controller 100 selects a switching destination device. For example, when control 5501 in FIG. 55 is applied, controller 100 assigns priority 1 to portable terminal 400B as a switching destination candidate. Controller 100 determines whether portable terminal 400B is appropriate. To this end, controller 100 checks, for example, the remaining battery level of portable terminal 400B. In state B, the check results show that portable terminal 400B is connected to power source 420 and the remaining battery level is, for example, 75%. Controller 100 compares the remaining battery level with a threshold value and determines that the result is equal to or greater than the threshold value. Therefore, controller 100 determines that portable terminal 400B is appropriate, and decides to switch the video provider from portable terminal 400A to portable terminal 400B.

[0341] The controller 100 switches the image provider from the mobile terminal 400A to the mobile terminal 400B, for example, in the same manner as in the case of Fig. 60. As a result, the virtual image display on the virtual image display device 1 is switched from the image based on the image data A to the image based on the image data B. The virtual image display on the virtual image display device 1 is in a good state. Note that the control as shown in Fig. 61 can also be applied to the case where the virtual image display device 1 is the main control entity.

[0342] FIG. 62 shows another example of control for switching the video provider by determining the battery state of the portable terminal 400, similar to FIG. 61 . In state A, one portable terminal 400 (400A) is connected to the controller 100, for example, via a communication path 602. The portable terminal 400A has 100% remaining battery power. The portable terminal 400A has video data A (6201). The controller 100 sets the portable terminal 400A, which has sufficient battery power, as the video provider. The portable terminal 400A, as the video provider, transmits video data 650 corresponding to the video data A (6201) to the controller 100. The controller 100 transmits the corresponding video data 650 to the virtual image display device 1, and the virtual image display device 1 displays a virtual image 9 in the display area 5 based on the video data 650. The video data transmission and virtual image display are in good condition.

[0343] State B is a state in which the remaining battery power of the portable terminal 400A is low, for example, 15%. The controller 100 grasps the remaining battery power of the portable terminal 400A and determines from that state that the portable terminal 400A is unsuitable as a video source, i.e., that good video data transmission and virtual image display cannot be continued. Therefore, the controller 100 considers switching the video source from the portable terminal 400A to another device. Here, a case in which switching to another portable terminal 400A is described. Also, a case in which switching is performed by automatic determination by the controller 100 rather than by user selection is described here.

[0344] State C indicates a search for a switching destination. In this example, it is assumed that the controller 100 uses communication to search for a portable terminal 400 other than the portable terminal 400A, and as a result, portable terminals 400B and 400C are discovered. The portable terminal 400B is, for example, a second portable terminal of user U1, and the portable terminal 400C is, for example, a portable terminal of user U2. The controller 100 checks the remaining battery levels of the respective candidate portable terminals 400B and 400C. In this example, the remaining battery level of the portable terminal 400B is 30%, and the remaining battery level of the portable terminal 400C is 80%.

[0345] For example, when the remaining battery charge of the portable terminal 400C is equal to or greater than a threshold, the controller 100 selects and determines the portable terminal 400C as an appropriate switching destination. In this example, the portable terminal 400C, which has the largest remaining battery charge, is selected. The controller 100 switches the video source from the portable terminal 400A to the portable terminal 400C. The portable terminal 400C has video data C (6202). After the switching, the portable terminal 400C transmits video data 650 corresponding to the video data C (6202) to the controller 100, and the controller 100 transmits the corresponding video data 650 to the virtual image display device 1. The virtual image display device 1 displays the virtual image 9 in the display area 5 based on the video data 650. This allows the virtual image display device 1 to continue displaying the virtual image.

[0346] As described above, when an abnormality occurs in the portable terminal 400 that is the video source, the in-vehicle system 300, for example, the controller 100, controls the video source to be switched to another device, as in the example of Fig. 62. In the example of Fig. 62, the abnormality is a low remaining battery charge. The controller 100 selects another device that is not abnormal, for example, the portable terminal 400 that has sufficient remaining battery charge, as the switching destination.

[0347] In the above example, the in-vehicle system 300 (e.g., the controller 100) may switch the video provider for the mobile terminal 400A whose remaining battery charge is low, and may also prompt the user U1 to charge the mobile terminal 400A as shown in the example of FIG. 52B . In response to the prompt to charge, the user U1 can connect the mobile terminal 400A to the power source 420 and start charging. In addition, when the in-vehicle system 300 detects the response to start charging, the in-vehicle system 300 may perform control so as not to switch the video provider. Alternatively, when the in-vehicle system 300 detects the response to start charging, the in-vehicle system 300 may perform control so as to switch the video provider back to the mobile terminal 400 that has been prompted to start charging.

[0348] The automatic control example shown in Fig. 62 is effective in a situation where, for example, while driving vehicle 2 on a highway or the like, there is no time to stop at a red light, making it difficult for the user to perform an operation such as connecting portable terminal 400 to power source 420. Furthermore, even in the case where switching is user-selected in the example shown in Fig. 62, for example, a GUI screen may be displayed in display area 5 of virtual image display device 1 as shown in Fig. 57A, and user U1, who is the driver, may select the switching destination and switch to another portable terminal 400 simply by operating operation switch 2D (Fig. 37) provided on steering wheel 2B at hand, which is convenient.

[0349] 63 shows an example of determining switching of the video provider by combining the multiple conditions shown in FIG. 56C. In this example, a case where the communication status condition #2 and the battery status condition #3 are combined is described.

[0350] In state A of FIG. 63 , two new mobile terminals 400 (400A, 400B) are connected to the in-vehicle system 300 (virtual image display device 1 and controller 100), or two mobile terminals 400 (400A, 400B) are found as a result of the in-vehicle system 300 searching using communication. The image provider has not been determined, and virtual image display device 1 is not displaying a virtual image. The controller 100 is assumed to be the control entity, for example. The control function of the controller 100 manages information (management information) 6300 indicating which device is the image provider. In this example, the image provider to the virtual image display device 1 is limited to one mobile terminal 400. Both of the two candidate mobile terminals 400 (400A, 400B) have image data and can be image providers. Each mobile terminal 400 may issue an image display request.

[0351] In state A, the controller 100 selects and determines one of two mobile terminals 400 as the video provider. In this example, a case will be described in which the selection is made automatically by the controller 100, rather than by a user. In this case, the controller 100 combines two conditions, condition #2 and condition #3, in other words, two perspectives, to make a comprehensive decision and select an appropriate mobile terminal. The controller 100 checks and determines both the communication status ( FIG. 60 ) and the battery status ( FIG. 61 ) of each of the two mobile terminals 400 (400A, 400B). In the example of state A, the mobile terminal 400A has a relatively high remaining battery level (compared to the mobile terminals 400A and 400B) but a relatively poor communication status, while the mobile terminal 400B has a relatively low remaining battery level but a relatively good communication status.

[0352] There are various methods for making a comprehensive judgment, and for example, a condition may be used that is constructed by combining each condition with AND (logical product) or OR (logical sum). Priorities may also be set for multiple conditions (i.e., viewpoints). Furthermore, an evaluation value for each condition may be calculated, and a comprehensive evaluation value may be calculated, and a judgment may be made based on the comprehensive evaluation value.

[0353] As an example of a comprehensive judgment, the controller 100 prioritizes the remaining battery level and selects the mobile terminal 400A as an appropriate video provider, provided that the remaining battery level is sufficient (for example, above a threshold) and the communication path condition is sufficient (for example, the communication speed is above a threshold).

[0354] State B shows the state after the selected mobile terminal 400A has been set as the video provider. The controller 100 sets the selected mobile terminal 400A as the video provider. As the video provider, the mobile terminal 400A transmits video data 650 corresponding to video data A (6301) to the controller 100 via the communication path 602. The controller 100 transmits the corresponding video data 650 to the virtual image display device 1, and the virtual image display device 1 displays a virtual image 9 in the display area 5 based on the video data 650.

[0355] [3-22. Switching of Video Providers: #4: Priority Event] FIG. 64 is an explanatory diagram of switching of video providers when the condition #4 in FIG. 56C is applied. In this embodiment, when an event such as an incoming call (sometimes referred to as a priority event) occurs in the mobile terminal 400 that is the video provider, the in-vehicle system 300 controls the system so that processing and operations related to the event take priority over providing video content as the video provider. As part of this control, when the user responds to the event (e.g., answers an incoming call) on the mobile terminal 400 where the event occurred, the in-vehicle system 300 controls the video provider to switch from the mobile terminal 400 where the event occurred to another device, for example, another mobile terminal 400. When the user simply receives an incoming call to the mobile terminal 400 but does not respond to the call, the video provider does not need to be switched.

[0356] As a control example, when an incoming call (priority event) occurs to the mobile terminal 400 that is currently providing video content from a video source, the in-vehicle system 300, particularly the controller 100 or the virtual image display device 1 as a control entity, outputs / notifies the user that the incoming call (priority event) has occurred based on cooperation between the mobile terminal 400, the controller 100, and the virtual image display device 1. The output / notification is not limited to output from the mobile terminal 400, but may also be, for example, a display of a virtual image 9 in the display area 5 or an audio output (described later). When the user confirms the notification and decides to respond to the incoming call (priority event), the in-vehicle system 300, based on this understanding, considers switching the video provider from the mobile terminal 400 that responded to the incoming call (priority event) to another device. The in-vehicle system 300 searches for candidate destinations for the video provider and performs control using priorities, such as those described above in FIG. 55. The in-vehicle system 300 communicates with the candidate devices, and when a suitable candidate device is found, it determines the device as the switching destination and switches the video provider to that device.

[0357] While the in-vehicle system 300 switches the video provider, the user responds to the incoming call (priority event) using the mobile terminal 400 that was used before the switch. Note that the user responds to the call using the mobile terminal 400 while the vehicle 2 is stopped, for example.

[0358] Furthermore, if the user decides not to respond to the call (priority event) in response to the notification, the in-vehicle system 300, based on the understanding of this, controls the video provider not to be switched.

[0359] FIG. 64 shows an example of the above control. In the example of FIG. 64, one mobile terminal 400 (400A) of user U1 is connected to the virtual image display device 1 or the controller 100 via a direct connection communication path. Here, a case where the mobile terminal 400 is connected to the controller 100 via a direct connection communication path 602 will be described. The controller 100 is the main control entity. State A is the normal state, that is, the state before the video provider is switched. The mobile terminal 400A transmits video data 650 corresponding to video data A (6401) to the controller 100, and the controller 100 transmits the corresponding video data 650 to the virtual image display device 1, and the virtual image display device 1 displays an image of the virtual image 9 in the display area 5 based on the video data 650. In this normal state, the virtual image display is in a good state.

[0360] State B is when an incoming call occurs as an example of a priority event in mobile terminal 400A, the video providing source. This is based on the premise that mobile terminal 400 is equipped with telephone application 413. In the control function of controller 100, it is set in advance as part of information (management information) 6400 that an incoming call is to be treated as a priority event in mobile terminal 400. Mobile terminal 400A outputs sound and vibration of the incoming call based on telephone application 413, and a notification video in response to the incoming call (e.g., the caller's phone number, buttons for answering, etc.) is displayed on screen 6402 of mobile terminal 400A.

[0361] FIG. 65 is a continuation of FIG. 64. State C is a state in which the in-vehicle system 300 (e.g., the controller 100) is searching for a candidate for the switching destination. In this example, the controller 100 uses the communication function with the portable terminal 400 to search for a nearby portable terminal 400. The search range 6501 corresponds to the communication range of the communication function. As a result of the search, it is assumed that a portable terminal 400B (e.g., smartphone B) and a portable terminal 400C (e.g., smartphone C) are found as portable terminals 400 other than the portable terminal 400A (e.g., smartphone A) before switching. For example, the portable terminal 400B is a second portable terminal 400 owned by user U1, and the portable terminal 400C is a portable terminal 400 owned by user U2. At this stage, a communication connection (establishment of a communication connection) with the found portable terminal 400 is not required, but the communication connection may be established.

[0362] In this example, controller 100 selects one switching destination from the two found candidate portable terminals 400B and 400C. Here, this selection is not an automatic decision by controller 100, but a selection operation by the user. Furthermore, controller 100 may confirm and determine the suitability of the candidate portable terminals 400 (400B, 400C) at this stage, and leave only suitable devices as candidates. In this example, it is assumed that both two portable terminals 400 (400B, 400C) are in an appropriate state to become video providers.

[0363] State D indicates candidate switching destinations and a user selection of the switching destination. Two mobile terminals 400 (400B, 400C) in an appropriate state are also referred to as candidate 1 and candidate 2. The controller 100 cooperates with the virtual image display device 1 by transmitting an instruction 6502 or the like to the virtual image display device 1, and causes a GUI for selecting a switching destination of the video provider to be displayed on the screen of the display area 5 of the virtual image display device 1.

[0364] Figure 66 is a continuation of Figure 65. State E is a display example of a GUI screen for a user to select a switching destination in the display area 5 of the virtual image display device 1. On the screen in this example, a guide such as "Please select a switching destination for the video provider" is output, and as options, information on smartphone A, which is currently being used as the video provider and has received the call, and information on smartphone B (candidate 1) and smartphone C (candidate 2), which are two smartphones found as a result of the search in state D, are displayed. Note that in this example, the currently being used smartphone A can also be selected as the video provider, i.e., smartphone A can be maintained as the video provider, but display of information on the currently being used smartphone A on the screen may be omitted.

[0365] A user, for example, user U1, can check and understand the situation by viewing the notification on this screen, and selects the switching destination if he or she wants to prioritize answering the incoming call on mobile terminal 400A (smartphone A). For example, assume that user U1 selects mobile terminal 400B (smartphone B) as the switching destination. Note that in this example, this switching destination mobile terminal 400B (smartphone B) is user U1's second mobile terminal 400, and therefore, at this point, user U1 has confirmed and agreed to the switching destination to mobile terminal 400B.

[0366] State E is an example in which a GUI screen is displayed on the virtual image display device 1, but this is not limiting, and a similar GUI screen may be displayed on the mobile terminal 400A of the user U1 or the like.

[0367] State F indicates switching of the image provider and the state after the switching. In response to the selection of the mobile terminal 400B as the user selection, the virtual image display device 1 cooperates with the controller 100 and transmits information 6601 indicating the selection. Based on the information 6601, the controller 100 controls the switching of the image provider from the mobile terminal 400A (smartphone A) to the mobile terminal 400B (smartphone B). For example, the controller 100 establishes a communication connection with the mobile terminal 400B (smartphone B) if one has not yet been established, and transmits an instruction / request 6603 via a communication path 6602 of the communication connection to designate the mobile terminal 400B (smartphone B) as the image provider. Based on the instruction / request 6603, the mobile terminal 400B (smartphone B) transmits image data 650 corresponding to the image data B (6404) to the controller 100 as the image provider. The controller 100 transmits the corresponding image data 650 to the virtual image display device 1. The virtual image display device 1 displays a virtual image 9 in the display area 5 based on the video data 650 .

[0368] On the other hand, the user U1 answers the incoming call (talks) at the mobile terminal 400A of the user U1. A display 6605 of the mobile terminal 400A indicates that the call is in progress.

[0369] 67 shows, as another example, a control example of switching the video provider when the user U1 selects the mobile terminal 400C (smartphone C) as the switching destination on the screen of state E in FIG. 66 . The virtual image display device 1 (which may be the mobile terminal 400A) cooperates with the controller 100 in response to the selection of the mobile terminal 400C as the user selection, and transmits information 6701 representing the selection. Based on the information 6701, the controller 100 performs control to switch the video provider from the mobile terminal 400A (smartphone A) to the mobile terminal 400C (smartphone C). For example, the controller 100 establishes a communication connection with the mobile terminal 400C (smartphone C) if one has not yet been established, and transmits an instruction / request 6703 via a communication path 6702 of the communication connection to specify the mobile terminal 400C (smartphone C) as the video provider. Based on the instruction / request 6703, the mobile terminal 400C (smartphone C), as a video provider, transmits video data 650 corresponding to the video data C (6704) to the controller 100. The controller 100 transmits the corresponding video data 650 to the virtual image display device 1. The virtual image display device 1 displays a virtual image 9 in the display area 5 based on the video data 650.

[0370] Note that, at the stage where the selection is made by the user U1, consent as to whether or not to switch may be obtained from the user U2 of the mobile terminal 400C (smartphone C) to which the selection is to be made. Obtaining consent from the user of the switching destination is the opposite of obtaining consent from the user before switching in the example of Fig. 58, but can be realized as a control process that is generally similar.

[0371] In the example of Fig. 67 , when an instruction / request 6703 is received from the controller 100, the switching destination mobile terminal 400C (smartphone C) displays a screen 6705 for obtaining approval from the user U2. The mobile terminal 400C (smartphone C) displays the screen 6705, and the user U2 checks the contents of the notification on the screen 6705. If the user U2 approves of switching to the user U2's own device (smartphone C), the user presses the YES button. The mobile terminal 400C transmits information indicating approval (YES) to the controller 100, and the controller 100 decides to switch the video provider to the mobile terminal 400C based on that information.

[0372] The control of switching between video providers in a plurality of mobile terminals 400 described above can be similarly applied to the case of connection via a communication network 609 as shown in FIG.

[0373] The priority event is not limited to an incoming call, but can be applied to any event that requires a response action using the mobile terminal 400, such as an incoming email.

[0374] [3-23. When the Controller is the Video Provider] FIG. 68 shows another example of control for switching the video provider when the controller 100 is the video provider. In this example, the controller 100 is the control subject. The control function of the controller 100 manages which device is the video provider as information 6800. Initially, as shown in state A, the controller 100 is the video provider and has video data 6801. The controller 100 transmits video data 650 corresponding to the video data 6801 to the virtual image display device 1 via the information transmission path 200. The virtual image display device 1 displays a virtual image 9 in the display area 5 based on the video data 650. Furthermore, two mobile terminals 400 (400A, 400B) are connected to the in-vehicle system 300.

[0375] Here, a case will be described in which the video provider is switched from the controller 100 to another device in accordance with the control using the priority shown in Fig. 55 described above. For example, a case will be described in which the control 5511 in the table of Fig. 55 (B) is applied. In the control 5511, priority 1 is the mobile terminal 400, and priority 2 is the virtual image display device 1. In this example, a case will be described in which the mobile terminal 400 (the corresponding user) issues an instruction / request to become the video provider and display video, and the controller 100 prioritizes the instruction / request and controls to switch the video provider to the mobile terminal 400.

[0376] The controller 100 recognizes that there are two mobile terminals 400 with priority 1, mobile terminal 400A and mobile terminal 400B. First, for example, if there is a request only from mobile terminal 400A, the controller 100 switches the video provider to mobile terminal 400A. If there is a request only from mobile terminal 400B, the controller 100 switches the video provider to mobile terminal 400B. If there are requests from both mobile terminal 400A and mobile terminal 400B, the controller 100 selects one of them. As described above, the selection may be made by a user or automatically by the controller 100. In this example, the controller 100 automatically selects an appropriate mobile terminal 400 using a predetermined method ( FIG. 56B ). Here, it is assumed that the controller 100 selects mobile terminal 400A as the switching destination.

[0377] State B shows the state after switching. Controller 100 switches the video provider to the selected mobile terminal 400A. Mobile terminal 400A, as the video provider, transmits video data 650 corresponding to video data A (6802) to controller 100 via communication path 602. Controller 100 transmits the corresponding video data 650 to virtual image display device 1, and virtual image display device 1 displays a virtual image 9 in display area 5 based on the video data 650.

[0378] [3-24. When the virtual image display device is the image provider] Fig. 69 shows, as another example, an example of control for switching the image provider when the virtual image display device 1 is the image provider. In this example, the virtual image display device 1 is the control entity. The control function of the virtual image display device 1 manages which device is the image provider as information 6900. Initially, as shown in state A, the virtual image display device 1 has image data 6901 as the image provider, and displays a virtual image 9 in the display area 5 based on the image data 6901. Furthermore, two mobile terminals 400 (400A, 400B) are connected to the in-vehicle system 300.

[0379] Here, a case will be described in which the image provider is switched from the virtual image display device 1 to another device in accordance with the control using the priority shown in Fig. 55 described above. For example, a case will be described in which the control 5521 in the table of Fig. 55 (C) is applied. In the control 5521, priority 1 is the mobile terminal 400, and priority 2 is the controller 100. In this example, a case will be described in which the mobile terminal 400 (the corresponding user) issues an instruction / request to become the image provider and display an image, and the virtual image display device 1 prioritizes the instruction / request and controls to switch the image provider to the mobile terminal 400.

[0380] The virtual image display device 1 recognizes that there are two mobile terminals 400, namely, mobile terminal 400A and mobile terminal 400B, as mobile terminals 400 with priority 1. First, for example, if there is a request only from mobile terminal 400A, the virtual image display device 1 switches the mobile terminal 400A as the image provider. If there is a request only from mobile terminal 400B, the virtual image display device 1 switches the mobile terminal 400B as the image provider. If there are requests from both mobile terminal 400A and mobile terminal 400B, the virtual image display device 1 selects one of them. As described above, the selection may be user selection, or may be automatic determination by the virtual image display device 1. In this example, the virtual image display device 1 automatically selects one appropriate mobile terminal 400 using a predetermined method ( FIG. 56B ). Here, it is assumed that the virtual image display device 1 selects the mobile terminal 400A as the switching destination.

[0381] State B shows the state after the switch. The virtual image display device 1 switches the image provider to the selected mobile terminal 400A. The mobile terminal 400A, as the image provider, transmits image data 650 corresponding to image data A (6902) to the virtual image display device 1 via the communication path 601, and the virtual image display device 1 displays a virtual image 9 in the display area 5 based on the image data 650.

[0382] 3-25. Control Flow FIG. 70 shows a main control flow by the in-vehicle system 300 in the third embodiment. In step S701, the in-vehicle system 300 checks the video provider (content provider) that will display the video on the virtual image display device 1, and checks whether there are multiple candidates for that video provider. If there is only one candidate (No), that candidate is selected, and the process proceeds to step S703. If there are multiple candidates (Yes), the process proceeds to step S702. In step S702, the in-vehicle system 300 checks and selects one video provider from the multiple candidates. One method for selecting a video provider from the candidates is for the user to select one via a UI, and another method is for the in-vehicle system 300 to automatically select one based on predetermined conditions. Once one video provider device has been selected, the process proceeds to step S703.

[0383] In step S703, the in-vehicle system 300 causes one image source device to provide (transmit) an image to the virtual image display device 1 and causes the virtual image display device 1 to display a virtual image.

[0384] In step S704, the in-vehicle system 300 checks and determines whether to switch the video provider based on predetermined conditions or triggers. Examples of conditions are the battery state of the mobile terminal 400 and a priority event, as described above. If it is determined that the video provider should be switched (Yes), the process proceeds to step S705. If it is determined that the video provider should not be switched, i.e., the current video provider should be maintained (No), the process proceeds to step S708.

[0385] In step S705, the in-vehicle system 300 searches for a destination (candidate) for switching to the video provider. In step S706, the in-vehicle system 300 checks whether multiple candidate destinations have been found as a result of the search. If there is one candidate (No), that candidate is selected, and the process proceeds to step S708. If there are multiple candidates (Yes), the process proceeds to step S707. Note that if no candidate destination is found as a result of the search, the process also proceeds to step S708 as a No result. Furthermore, if one of the candidate destinations found as a result of the search is inappropriate as a switching destination, the process also proceeds to step S708 as a No result. In these cases, the current video provider is maintained, or the display is stopped as described above.

[0386] In step S707, the in-vehicle system 300 checks and selects one video provider from the multiple candidates. One method is for the user to select one via a UI, and another method is for the in-vehicle system 300 to automatically select one based on predetermined conditions. Once one video provider device has been determined, the process proceeds to step S708.

[0387] In this flow (steps S704 to S707), the current video provider and the candidate switching destinations found in the search results are considered separately, but this is not limiting. It is also possible to consider a group including the current video provider and the candidate switching destinations found in the search results, and then select and determine one switching destination from that group based on predetermined conditions.

[0388] In step S708, the in-vehicle system 300 causes the determined one image provider device to provide (transmit) an image to the virtual image display device 1 and display a virtual image on the virtual image display device 1. After step S708, this flow ends. From then on, the same process may be repeated, for example, from step S704.

[0389] Fig. 71 shows a more detailed example of the control flow of the in-vehicle system 300. Fig. 71 shows an example of a sequence of processes and operations between three elements: (A) the virtual image display device 1, (B) the controller 100, and (C) the mobile terminal 400. Steps in each element are indicated as step A1, etc. Fig. 71 shows a case where the main control entity is the controller 100. In the example of Fig. 71, video data is transmitted via the controller 100.

[0390] In step A1, the virtual image display device 1 is started. In step B1, the controller 100 is started. In step B2, the controller 100 transmits a notification to the virtual image display device 1 to confirm the display preparation status. In step A2, the virtual image display device 1 receives the notification from the controller 100. In step A3, the virtual image display device 1 performs preparation processing for displaying an image as a virtual image 9 in the display area 5, and when preparation is complete, transmits a preparation completion notification to the controller 100. In step B3, the controller 100 receives the preparation completion notification.

[0391] In this example, it is assumed that the controller 100 is the image provider in the initial and default settings. In step B4, the controller 100 transmits the target image (image data, content information) to the virtual image display device 1. In step A4, the virtual image display device 1 receives the image data from the controller 100. In step A5, the virtual image display device 1 displays an image as a virtual image 9 in the display area 5 based on the image data.

[0392] In step B6, controller 100 checks and determines whether or not to switch the video provider. If not (No), the process returns to, for example, step B4 and repeats the same process. If switching is to be performed (Yes), the process proceeds to step B7. In this example, it is assumed that a mobile terminal 400 is a candidate for the switching destination from controller 100. In step B7, controller 100 searches for a mobile terminal 400 as a candidate for the switching destination. If a mobile terminal 400 is found as a candidate for the switching destination in step B8 (Yes), the process proceeds to step B9, and if not found (No), the process returns to, for example, step B4. In step B8, if there are multiple candidate mobile terminals 400, one of them is selected.

[0393] In step B9, controller 100 transmits a notification to portable terminal 400, which is a candidate for switching destination, to confirm the video provision preparation status of that portable terminal 400. In step C9, portable terminal 400 receives the notification. In step C10, portable terminal 400 performs preparation processing for transmitting the target video with itself as the video provider, and when preparation is complete, transmits a preparation completion notification to controller 100. In step B10, controller 100 receives the preparation completion notification.

[0394] In step B11, the controller 100 transmits a video provision instruction to the mobile terminal 400. This video provision instruction instructs the mobile terminal 400 on the timing to transmit the target video. In this example, video data is transmitted in the order of the mobile terminal 400, the controller 100, and the virtual image display device 1, and the virtual image display device 1 is controlled to display the target video in approximately real time. The video provision instruction is control information for controlling the timing for this. In step C11, the mobile terminal 400 receives the video provision instruction.

[0395] In step C12, the portable terminal 400 transmits video data to the controller 100 in accordance with the video provision instruction. In step B12, the controller 100 receives the video data from the portable terminal 400. The controller 100 and the like buffer the video data in memory as appropriate. In step B13, the controller 100 transmits corresponding video data to the virtual image display device 1 at a timing controlled by the controller 100. In step A13, the virtual image display device 1 receives the video data from the controller 100. In step A14, the virtual image display device 1 displays an image as a virtual image 9 in the display area 5 based on the video data.

[0396] In step B15, the controller 100 checks and determines whether to end the virtual image display on the virtual image display device 1 based on the image provider, and if not (No), returns to step B6, for example, and repeats the same process. If it is to be ended (Yes), proceeds to step B14, and the controller 100 performs end processing.

[0397] Fig. 72 shows a detailed example of a control sequence as a variation of Fig. 71. In this example, video data is transmitted via the controller 100 in the same manner as in Fig. 71, except that the virtual image display device 1 makes decisions and performs control regarding the selection and switching of the video provider. The following mainly describes the differences.

[0398] Initially, the image provider is assumed to be controller 100. In step A2, virtual image display 1 transmits a display preparation status confirmation notification to controller 100, and in step B2, controller 100 receives the notification. In step B3, controller 100 prepares for display and transmits a preparation completion notification to virtual image display 1. Virtual image display 1 receives the preparation completion notification. In step B4, controller 100 transmits image data of the target image to virtual image display 1, and in step A4, virtual image display 1 receives the image data, and in step A5, virtual image display 1 displays the image as a virtual image 9 in display area 5 based on the image data.

[0399] In step A6, the virtual image display 1 confirms and determines whether to switch the video provider. If the video provider is to be switched (Yes), the virtual image display 1 searches for a mobile terminal 400 that is a candidate for the switching destination in step A7. If a candidate for the switching destination is found in step A8 (Yes), the virtual image display 1 transmits a video provision preparation status confirmation notification to the controller 100 in step A9. In step B9, the controller 100 receives the notification and transmits a corresponding notification to the candidate mobile terminal 400. In step C9, the mobile terminal 400 receives the notification. In step C10, the mobile terminal 400 performs a preparation process for video provision using itself as the video provider and transmits a preparation completion notification to the controller 100. In step B10, the controller 10 receives the preparation completion notification and transmits a corresponding preparation completion notification to the virtual image display 1, and in step A10, the virtual image display 1 receives the preparation completion notification.

[0400] In step B11, controller 100 transmits an instruction to provide a video to mobile terminal 400, which is the source of the video, and in step C11, mobile terminal 400 receives the instruction to provide a video. In step C12, mobile terminal 400 transmits video data of the target video to controller 100. In step B12, controller 100 receives the video data. In step B13, it transmits corresponding video data to virtual image display device 1. In step A13, virtual image display device 1 receives the video data. In step A14, virtual image display device 1 displays the video as a virtual image 9 in display area 5 based on the video data.

[0401] In step A15, the virtual image display device 1 checks and determines whether to end the virtual image display, and if not (No), returns to step A6, for example, and repeats the same process. If it is to be ended (Yes), in step A16, the virtual image display device 1 performs an end process.

[0402] [3-26. Other Screen Display Examples] FIG. 73 shows another screen display example, different from FIG. 66 , when an incoming call event (priority event) occurs while video is being provided by the mobile terminal 400, which is the video provider. In the example of state A in FIG. 73 , the mobile terminal 400A of the user U1, who is the driver, is the video provider, and navigation video information from the navigation application 411 ( FIG. 36 ) is displayed on the screen of the mobile terminal 400A. The mobile terminal 400A is installed in the vehicle 2, and the user U1, who is the driver, does not need to look at the screen of the mobile terminal 400A itself. Furthermore, the display on the screen of the mobile terminal 400A may be performed as background processing. In other words, as long as the corresponding video data is transmitted from the mobile terminal 400A to the virtual image display device 1, the video information does not need to be displayed on the screen of the mobile terminal 400A.

[0403] An example of state B is an example in which an incoming call occurs on the screen of state A, and an image notifying the incoming call is displayed in a partial area of ​​the screen. Such a screen display may also be used.

[0404] An example of state C is a case where map information is displayed by the map application 412 ( FIG. 36 ) on the screen of the portable terminal 400B of the passenger user U2. As described above ( FIG. 48 ), the portable terminal 400B can be used as an image provider to display this map information in the display area 5 of the virtual image display device 1. The passenger user U2 holds and operates the portable terminal 400B in his / her hand and views the map information on the screen.

[0405] An example of state D is an example in which an incoming call occurs on the screen of state C, and an image notifying the incoming call is displayed in a partial area of ​​the screen. Such a screen display may also be used.

[0406] The screen of the above state B or state D may transition to a GUI for confirming or selecting the video provider, as described above ( FIG. 57B ). The transition destination may be the screen of the portable terminal 400 or the screen of the display area 5 of the virtual image display device 1. When the user operates while looking at the GUI on the screen of the portable terminal 400 (for example, FIG. 57B ), the above-mentioned video provider can be selected or switched based on the operation of the GUI on that screen. Furthermore, when the user operates while looking at the GUI on the screen of the display area 5 of the virtual image display device 1 (for example, FIGS. 57A and 66 ), the above-mentioned video provider can be selected or switched based on the operation of the GUI on that screen.

[0407] FIG. 74 shows an example in which, when an incoming call, which is a priority event, occurs on the mobile terminal 400 (e.g., the mobile terminal 400A) that is the video source (e.g., state B in FIG. 73 ), a display is displayed on the screen of the display area 5 of the virtual image display device 1 to inform the user of the incoming call. The virtual image display device 1 or the controller 100 cooperates to recognize the occurrence of an incoming call on the mobile terminal 400. For example, when an incoming call (priority event) occurs on the mobile terminal 400A, the mobile terminal 400A transmits information 7401 indicating the occurrence of the incoming call (priority event) to the controller 100. The controller 100 recognizes the occurrence of the incoming call (priority event) on the mobile terminal 400A from the information 7401. Based on the information 7401, the controller 100 transmits information 7402 indicating the occurrence of the incoming call (priority event) on the mobile terminal 400A to the virtual image display device 1. This information 7402 may be considered as an instruction to cause the virtual image display device 1 to perform a display informing the user of the occurrence of the incoming call (priority event) on the mobile terminal 400A. Based on the information 7402, the virtual image display device 1 displays an image 7400 informing the user of the occurrence of an incoming call (priority event) on the mobile terminal 400A as a virtual image 9 in the display area 5. This allows the user to recognize the occurrence of an incoming call (priority event) on the mobile terminal 400A.

[0408] In another example, the mobile terminal 400A may transmit information 7401 indicating the occurrence of an incoming call (priority event) to the virtual image display device 1.

[0409] Furthermore, the display of the image 7400 of the incoming call in the display area 5 may be transitioned to a GUI for checking or selecting the video provider as described above. Furthermore, when the user performs a predetermined operation input on the image 7400 of the incoming call, the display may be transitioned to that GUI.

[0410] FIG. 75A shows an example of a notice of advance notice regarding switching of the video provider as another example of a screen display. When the in-vehicle system 300 (e.g., the controller 100) decides to switch the video provider as described above, it displays an image 7501 indicating advance notice regarding switching of the video provider, for example, in the display area 5 of the virtual image display device 1, as shown in the figure. Image 7501 in this example illustrates a case in which the video provider (portable terminal for virtual image display) is switched from smartphone A to smartphone B. By viewing image 7501, the user can easily recognize the situation immediately before switching of the video provider. After displaying image 7501, the in-vehicle system 300 switches the video provider. A similar notice can be displayed when switching the video provider to the controller 100 or the virtual image display device 1. The various notification images are not limited to text, but may also include icons, animations, and the like, and audio output may also be used.

[0411] 75B shows an example of a notification of completion of switching of the video provider as another example of a screen display. When the in-vehicle system 300 (e.g., the controller 100) has completed switching of the video provider (completed normally, in other words, successfully), it displays an image 7502 indicating the completion of switching of the video provider, for example, in the display area 5 of the virtual image display device 1, as shown in the figure. In this example, the image 7502 is an image that conveys that switching of the video provider from smartphone A to smartphone B has been completed, and that smartphone A is receiving an incoming call and smartphone B is the video provider. By looking at the image 7502, the user can easily recognize the situation immediately after the completion of switching of the video provider.

[0412] FIG. 75C shows another example of a screen display for selecting a destination for switching the video provider. In this example, the in-vehicle system 300 (e.g., the controller 100) displays a GUI image 7503 for selecting a destination for switching the video provider in a portion of the display area 5 of the virtual image display device 1. In this example, the image 7503 shows smartphones B and C as candidate destinations, and also displays user information for the owners of each mobile terminal 400. The user selects one of the candidate destinations in the GUI image 7503 by operating an operation input device provided in the in-vehicle system 300. For example, the driver can move the cursor in the GUI image 7503 to select and confirm the destination by operating the operation switch 2D (e.g., a dial key, button, etc., shown in FIG. 37 ) provided on the steering wheel 2B. For safety reasons, the display prompting the user to perform a selection operation, such as that shown in FIG. 75C, is preferably performed while the vehicle is stopped. For example, when an event occurs that requires switching the video provider, the user may first be prompted to stop the vehicle while taking safety into consideration, and after it is determined that the vehicle has stopped, the display shown in Figure 75C may be displayed, allowing the user to select the video provider to switch to.

[0413] 75D shows another example of a screen display, which is a notification of a failure in switching the video provider. In this example, when the in-vehicle system 300 (e.g., the controller 100) fails to switch the video provider (in other words, terminates due to an error or the like), the in-vehicle system 300 displays an image 7504 indicating the failure of switching the video provider in the display area 5 of the virtual image display device 1. In this example, the image 7504 indicates that the in-vehicle system 300 attempted to switch to smartphone B, but the user of smartphone B, for example, refused the switch, resulting in a failure / error.

[0414] As described above, according to the third embodiment, even when the user's mobile terminal 400 is connected to the in-vehicle system 300 as an external video provider, the video provided from the mobile terminal 400 can be suitably displayed as the virtual image 9 on the virtual image display device 1. Furthermore, according to the third embodiment, even when there are multiple candidates for the video provider, the suitable video provider can be selected and switched.

[0415] Although the embodiments of the present disclosure have been specifically described above, they are not limited to the above-described embodiments and can be modified in various ways without departing from the spirit of the present disclosure. Except for essential components, components can be added, deleted, or replaced in each embodiment. Unless otherwise specified, each component may be singular or plural. A combination of each embodiment and its variations is also possible.

[0416] The technology according to the embodiment of the present invention can accommodate cases where a display device receives and displays an externally transmitted image as a video input, thereby providing a more suitable display device technology. By being able to check the appropriateness of the display on the display device and not displaying an image if it is inappropriate, a display device can be provided that contributes to supporting safe driving, thereby preventing traffic accidents. This contributes to "Good health and well-being for all," one of the Sustainable Development Goals (SDGs) advocated by the United Nations.

[0417] 1...HUD device, virtual image display device, display device, 1M...mirror-type display device, 2...vehicle, 3...windshield, 4...vehicle information, 5...display area, 9...virtual image, 100...controller, 150...car navigation system, 200...information transmission path, 400...mobile terminal, 405...screen, 411...car navigation system application, 412...map application, 601, 602...communication path, 650...video data, 3600...server, 3601...video data

Claims

1. A virtual image display device comprising: an image processing unit that processes image information input from the outside; and an image display device that displays an image based on the image data processed by the image processing unit and outputs image light corresponding to the image, and the virtual image display device adjusts an information transmission path for transmitting the image information from the outside in response to the display of the image.

2. A virtual image display device as described in claim 1, comprising two or more transmission paths as the information transmission path when inputting the image information from the outside, and when transmitting the image information via a first transmission path, if the state of the first transmission path is abnormal or malfunctioning, the virtual image display device switches to a second transmission path to transmit the image information.

3. A virtual image display device as described in claim 1, comprising two or more transmission paths as the information transmission path when the image information is input from the outside, and when transmitting the image information via the first transmission path, if the state of the first transmission path is abnormal or malfunctioning, the second transmission path is used as a main path and the first transmission path is used as a sub-path, and the image information is transmitted using both the main path and the sub-path.

4. A virtual image display device as described in claim 1, comprising a single transmission path as the information transmission path when the image information is input from the outside, and when transmitting the image information via the single transmission path, if the state of the single transmission path is poor, the amount of data in the image information is reduced and transmitted, thereby restricting the image display on the image display device.

5. A virtual image display device as claimed in claim 1, comprising a single transmission path as the information transmission path when the image information is input from the outside, and when transmitting the image information via the single transmission path, if the state of the single transmission path is abnormal, the transmission of the image information is stopped and image display on the image display device is stopped.

6. A virtual image display device comprising: an image processing unit that processes image information input from the outside; and an image display device that displays an image based on the image data processed by the image processing unit and outputs image light corresponding to the image, and which adjusts an information transmission path for transmitting the image information from the outside in response to the display of the image.

7. A virtual image display device as claimed in claim 1, wherein when the image information is displayed on the image display device, the appropriateness is checked, and if the result of the check is that the image information is inappropriate, the image display of the image information on the image display device is made undisplayable.

8. A virtual image display device according to claim 7, wherein when a display rate or a display amount of the video information relative to the display area is equal to or greater than a threshold value, it is determined that the video information is not displayable.

9. A virtual image display device according to claim 7, wherein when the luminance of the video information in the display area is abnormal with respect to a reference luminance, it is determined that the display is not possible.

10. A virtual image display device according to claim 7, wherein when there is an error in the image information for the display area or when there is a discrepancy between the actual situation and the image information, it is determined that the display is not possible.

11. A virtual image display device according to claim 1, wherein the virtual image display device is connected via communication to a controller of an in-vehicle system, and the external device is the controller.

12. An in-vehicle system comprising: a controller; and a virtual image display device communicatively connected to the controller, wherein the virtual image display device comprises: an image processing unit that processes image information input from an external source; and an image display device that displays an image based on the image information processed by the image processing unit and outputs image light corresponding to the image, wherein the virtual image display device displays an image as a virtual image based on image information from a mobile terminal as the external source.

13. An in-vehicle system according to claim 12, comprising the controller, the virtual image display device, and the mobile terminal as candidates for image providers when displaying an image on the virtual image display device, and controlling the selection of the image provider.

14. An in-vehicle system according to claim 13, wherein the virtual image display device controls the selection of the image provider.

15. An in-vehicle system according to claim 13, wherein the controller controls the selection of the video provider.

16. An in-vehicle system according to claim 13, wherein priorities are set for the candidate video provider, and selection of the video provider is controlled based on the priorities.

17. An in-vehicle system according to claim 13, wherein, in a situation where an appropriate image provider cannot be selected from the candidate image providers, the display of the virtual image on the virtual image display device is stopped.

18. An in-vehicle system according to claim 12, further comprising at least the mobile terminal as a candidate for an image provider when an image is displayed on the virtual image display device, and the image provider is selected based on an operational input or setting by a user.

19. An in-vehicle system according to claim 12, comprising at least the mobile terminal as a candidate for an image provider when an image is displayed on the virtual image display device, and selecting the image provider based on automatic judgment under predetermined conditions.

20. An in-vehicle system as described in claim 12, comprising at least the portable terminal as a candidate for an image provider when an image is displayed on the virtual image display device, and when there are a plurality of portable terminals, the image provider is selected from the plurality of portable terminals.

21. An in-vehicle system as described in claim 12, comprising at least the portable terminal as a candidate for an image providing source when an image is displayed on the virtual image display device, the battery status of the portable terminal is checked, and the portable terminal with a battery in good condition is selected as the image providing source.

22. An in-vehicle system as described in claim 12, comprising at least the portable terminal as a candidate for an image providing source when an image is displayed on the virtual image display device, the system confirms a communication state with the portable terminal, and selects the portable terminal with a good communication state as the image providing source.

23. An in-vehicle system according to claim 12, further comprising: a state of the portable terminal being confirmed; and the virtual image display device displaying an image representing the state of the portable terminal as the virtual image.

24. An in-vehicle system as described in claim 12, further comprising at least the mobile terminal as a candidate for an image provider when displaying an image on the virtual image display device, and confirming historical information regarding the image provider, and selecting, as the image provider, the mobile terminal that has previously been a video provider.

25. An in-vehicle system as described in claim 12, comprising at least the mobile terminal as a candidate for an image provider when an image is displayed on the virtual image display device, and when a new mobile terminal is connected for communication, determining whether or not to switch the image provider to the new mobile terminal, and performing the switching based on the result of the determination.

26. An in-vehicle system as described in claim 12, comprising at least the mobile terminal as a candidate for an image provider when an image is displayed on the virtual image display device, and when a communication path to the mobile terminal as the image provider is malfunctioning or in an error state, the in-vehicle system determines whether to switch the image provider to another device, and performs the switch based on the result of the determination.

27. An in-vehicle system as described in claim 12, comprising at least the portable terminal as a candidate for an image providing source when an image is displayed on the virtual image display device, and determining whether to switch the image providing source to another device when the battery condition of the portable terminal, which is the image providing source, is not good, and making the switch based on the result of the determination.

28. An in-vehicle system as described in claim 12, comprising at least the portable terminal as a candidate for an image provider when an image is displayed on the virtual image display device, and when a priority event occurs in the portable terminal that is the image provider, the in-vehicle system determines whether or not to switch the image provider to another device, and performs the switch based on the result of the determination.

29. An in-vehicle system as described in claim 28, wherein, when the priority event occurs on the mobile terminal that is the image providing source, the virtual image display device displays, as the virtual image, an image representing a state in which the priority event has occurred on the mobile terminal.

30. An in-vehicle system as described in claim 12, comprising at least the mobile terminal as a candidate for an image provider when an image is displayed on the virtual image display device, and when switching the image provider from the mobile terminal to another device, a user of the mobile terminal from which the image provider is to be switched is confirmed for consent to the switch, or when switching the image provider from a device to the mobile terminal, a user of the mobile terminal to which the image provider is to be switched is confirmed for consent to the switch.

31. An in-vehicle system as described in claim 12, further comprising at least the mobile terminal as a candidate for an image provider when the virtual image display device displays an image, and when the image provider is switched from the mobile terminal to another device, or when the image provider is switched from one device to the mobile terminal, the virtual image display device displays an image representing the switch in the image provider as the virtual image.

32. An in-vehicle system as described in claim 12, further comprising at least the mobile terminal as a candidate for an image provider when the image is displayed on the virtual image display device, and when switching the image provider, the virtual image display device displays an image representing a candidate for a destination of the image provider as the virtual image, and selects the destination of the image provider based on a user's operation input for the image representing the candidate.

33. An in-vehicle system according to claim 12, wherein the virtual image display device includes a communication unit that is communicatively connected to the portable terminal and receives the image information from the portable terminal.

34. An in-vehicle system according to claim 12, wherein the virtual image display device includes a communication unit that is communicatively connected to the controller, and receives the image information from the portable terminal from the controller via the controller.

35. A virtual image display device comprising: an image processing unit that processes image information input from the outside; and an image display device that displays an image based on the image information processed by the image processing unit and outputs image light corresponding to the image, wherein the virtual image display device displays an image as a virtual image based on image information from a mobile terminal as the outside.

36. A virtual image display device according to claim 35, comprising a controller of an in-vehicle system, the virtual image display device, and the mobile terminal as candidates for an image provider when an image is displayed on the virtual image display device, and controls the selection of the image provider.

37. A virtual image display device according to claim 36, wherein priorities are set for the candidates for the image provider, and selection of the image provider is controlled based on the priorities.

38. A virtual image display device according to claim 35, comprising at least the mobile terminal as a candidate for an image provider when an image is displayed on the virtual image display device, and the image provider is selected based on an operational input or setting by a user.

39. A virtual image display device according to claim 35, comprising at least the mobile terminal as a candidate for an image provider when an image is displayed on the virtual image display device, and selecting the image provider based on automatic determination under predetermined conditions.

40. A virtual image display device as described in claim 35, comprising at least the mobile terminal as a candidate for an image provider when an image is displayed on the virtual image display device, and when there are a plurality of mobile terminals, the image provider is selected from the plurality of mobile terminals.

41. A virtual image display device according to claim 35, comprising at least the portable terminal as a candidate for an image providing source when an image is displayed on the virtual image display device, the battery status of the portable terminal is confirmed, and the portable terminal having a battery in good condition is selected as the image providing source.

42. A virtual image display device as described in claim 35, comprising at least the mobile terminal as a candidate for an image providing source when an image is displayed on the virtual image display device, the state of communication with the mobile terminal is confirmed, and the mobile terminal with a good communication state is selected as the image providing source.

43. A virtual image display device according to claim 35, further comprising: a state of said portable terminal being confirmed; and an image representing the state of said portable terminal being displayed as said virtual image.

44. A virtual image display device according to claim 35, further comprising a communication unit that is communicatively connected to said mobile terminal and receives said image information from said mobile terminal.

45. A virtual image display device according to claim 35, further comprising a communication unit that is communicatively connected to a controller of an in-vehicle system, and receives the image information from the portable terminal from the controller via the controller.

Images