vehicle

Abstract

We provide vehicles that can flexibly accommodate various display specifications. Furthermore, we contribute to Sustainable Development Goal 3: "Ensure healthy lives and promote well-being for all." [Solution] The system comprises a video display unit that displays an image and emits video light from the displayed image; a video light projection unit that projects the emitted video light onto a display area so that the projected video light can be seen as a virtual image; and a control unit that determines the display content, prepares video data, and displays the image based on the data on the video display unit. The control unit changes the provisionally determined display content if the preparation time required to prepare the video data for the provisionally determined display content before determining the display content is longer than a predetermined processing cycle.

Description

Technical Field

[0001] The present invention relates to a vehicle, for example, to a vehicle technology using AR (Augmented Reality).

Background Art

[0002] Patent Document 1 discloses a display system that extracts an object to be an AR display target based on imaging data, generates AR image data of the object, and sets the frame rate of the AR image data based on the importance of the object. For example, when the overall frame rate is 60 fps and three AR image data are displayed in time division, the frame rates of the three AR image data are set to 30 fps, 20 fps, and 10 fps in order according to the importance. Thereby, the higher the importance of the AR image data, the more the flicker can be suppressed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] A head-up display device prepares various video data based on information obtained from ADAS (Advanced Driver Assistance Systems) or the like, and projects video light based on the video data onto a display area to be visually recognized as a virtual image. At this time, the preparation time required for preparing the video data varies depending on differences such as the type and number of videos, the size of the video according to the virtual image distance, the display format such as 2D / 3D, and the presence or absence of a graphics effect. In the specification, the head-up display device is also referred to as a HUD device.

[0005] In HUD devices, processing performance such as the maximum frame rate is typically fixed and determined primarily based on the hardware specifications. However, there are times when users want to change the display specifications of a HUD device through software updates, without changing the hardware itself. In such cases, the required display specifications for the HUD device often involve an increased processing load. As a result, the hardware processing performance may become insufficient, potentially leading to issues such as dropped frames during AR display, or other problems that prevent the device from meeting the required display specifications.

[0006] The present invention has been made in view of the above, and one of its objectives is to provide a vehicle that satisfies the required display specifications as much as possible within the limits of the hardware processing performance, and that can flexibly respond to the display specifications.

[0007] The aforementioned and other objectives and novel features of the present invention will become apparent from the description herein and the accompanying drawings. [Means for solving the problem]

[0008] A brief overview of some of the representative inventions disclosed in this application is as follows:

[0009] A typical vehicle includes a video display unit that displays an image and emits video light from the displayed image; a video light projection unit that projects the emitted video light onto a display area, causing the projected video light to be perceived as a virtual image; and a control unit that determines the display content based on acquired vehicle information, prepares video data based on the determined display content, and displays the image based on the prepared video data on the video display unit. The control unit changes the provisionally determined display content if the preparation time required to prepare the video data for the provisionally determined display content before determining the display content is longer than a predetermined processing cycle. [Effects of the Invention]

[0010] To briefly explain the effects obtained by representative inventions disclosed in this application, it becomes possible to flexibly accommodate display specifications in vehicles. [Brief explanation of the drawing]

[0011] [Figure 1] This is a schematic diagram showing an example of the configuration of a vehicle equipped with a head-up display device according to Embodiment 1. [Figure 2A] Figure 1 is a schematic diagram showing an example of the main components of the HUD device. [Figure 2B] Figure 1 is a schematic diagram showing an example of the main components of the HUD device, different from those shown in Figure 2A. [Figure 3A] Figures 2A and 2B show a block diagram illustrating an example of the configuration of the main components of the control system responsible for control in the HUD device. [Figure 3B] This block diagram shows an example of the main configuration of the control system responsible for control in the HUD device shown in Figures 2A and 2B, which differs from that shown in Figure 3A. [Figure 4] Figures 3A and 3B are block diagrams showing examples of the configuration of the control unit. [Figure 5] Figure 1 is a schematic diagram showing an example of the display content of the HUD device. [Figure 6] Figure 3A is a flowchart illustrating an example of the processing procedure for displaying video in the HUD device shown. [Figure 7] This timing chart schematically illustrates an example of a problem that may arise when displaying video using the flow shown in Figure 6. [Figure 8A] This is a schematic diagram showing how the displayed content changes when no frame drops occur. [Figure 8B] This is a schematic diagram showing how the displayed content changes when frame drops occur. [Figure 9] This figure shows an example of the internal state of the control unit shown in Figure 3A. [Figure 10] Figure 3A shows an example of the monitoring results of the preparation time by the control unit shown. [Figure 11A] It is a schematic diagram showing an example of the display content of the HUD device in the normal state shown in FIG. 9. [Figure 11B] It is a schematic diagram showing an example of the display content of the HUD device in the suppression transition state shown in FIG. 9. [Figure 11C] It is a schematic diagram showing an example of the display content of the HUD device in the suppression state shown in FIG. 9. [Figure 12] It is a flowchart showing an example of the processing content of the control unit shown in FIG. 3A. [Figure 13A] It is a schematic diagram showing an example of the relationship between the video data written in the frame buffer and the display content displayed on the video display unit in the HUD device shown in FIGS. 3A and 3B. [Figure 13B] It is a schematic diagram showing an example of a relationship different from that in FIG. 13A. [Figure 14] In FIG. 6, it is a diagram showing an example of items that affect the preparation time of video data. [Figure 15] It is a flowchart showing an example of the processing procedure when displaying a video in the HUD device according to the second embodiment. [Figure 16] It is a flowchart showing an example of the detailed processing content of the display content adjustment process in FIG. 15. [Figure 17A] In FIG. 16, it is a diagram showing a specific example of the prediction method (step S412) of the preparation time. [Figure 17B] It is a diagram showing a different specific example from that in FIG. 17A. [Figure 18] In FIG. 16, it is a diagram for explaining an example of the method of changing the display content (step S414). [Figure 19] In FIG. 18, it is a schematic diagram showing an example of the display content of the HUD device obtained as a result of changing the display content. [Figure 20] It is a timing chart when displaying a video using the flowcharts shown in FIGS. 15 and 16. [Figure 21A] It is a schematic diagram showing an example of the main processing content of the control unit in the HUD device according to the third embodiment. [Figure 21B] It is a supplementary diagram of FIG. 21A. [Figure 22A] It is a schematic diagram showing an example different from FIG. 21A. [Figure 22B] It is a supplementary diagram of FIG. 22A. [Figure 23] It is a diagram showing a configuration example of a display setting table stored by the control unit in the HUD device according to Embodiment 3. [Figure 24A] It is a flowchart showing an example of the processing content of the control unit in the HUD device according to Embodiment 3. [Figure 24B] It is a flowchart showing an example of processing content different from FIG. 24A.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described in detail based on the drawings. In all the drawings for explaining the embodiments, the same members are generally denoted by the same reference numerals, and the repeated explanations thereof are omitted.

[0013] (Embodiment 1) <Overview of HUD Device> FIG. 1 is a schematic diagram showing a configuration example of a vehicle equipped with a head-up display device according to Embodiment 1. The head-up display (HUD) device 1 shown in FIG. 1 is mounted on a vehicle 2 which is one type of vehicle. The vehicle 2 is typically an automobile, but is not necessarily limited thereto, and may be a railway vehicle or the like. Also, the vehicle is not limited to a vehicle, and may be an aircraft or the like. Further, the vehicle 2 is equipped with a control unit 21 called an ECU (Electronic Control Unit), for example.

[0014] The control unit 21 acquires vehicle information 4 from various sensors installed in various parts of the vehicle 2, as well as from a navigation system, etc. The various sensors detect various events that occur in the vehicle 2, and also detect various parameter values ​​related to the driving situation. The HUD device 1 acquires the vehicle information 4 acquired by the control unit 21, for example, using CAN (Controller Area Network) communication, etc.

[0015] Vehicle information 4 includes, for example, vehicle 2's speed information, gear information, steering angle information, lamp illumination information, ambient light information, distance information, infrared information, engine ON / OFF information, camera images inside and outside the vehicle, acceleration gyro information, GPS (Global Positioning System) information, navigation information, vehicle-to-vehicle communication information, and vehicle-to-infrastructure communication information. GPS information also includes information such as the current time. Vehicle information 4 also includes various warning information. Based on this vehicle information 4, the HUD device 1 projects image light onto a display area such as the windshield 3. In this way, the HUD device 1 allows the driver or other user to perceive the projected image light onto the display area as a virtual image, more specifically as a virtual image superimposed on the scenery in front of the vehicle 2.

[0016] Figure 2A is a schematic diagram showing an example of the main components of the HUD device shown in Figure 1. The HUD device 1 shown in Figure 2A includes, for example, a video display unit 11, mirrors M1 and M2, and a mirror drive unit 14, all housed within a housing 12. The video display unit 11 is, for example, a display panel such as an LCD (Liquid Crystal Display) or a projector, which displays an image based on input video data and emits video light from the displayed image.

[0017] Mirror M2 reflects the video light from the video display unit 11 toward mirror M1. Mirror M2 is effective in saving space while ensuring a long optical path length. Depending on the space within the HUD housing and the required optical path length, mirror M2 may be omitted or multiple mirrors may be placed. Mirror M1 functions as a video light projection unit. Mirror M1, being the video light projection unit, projects the video light emitted from the video display unit 11 and reflected by mirror M2 onto the display area 5 of the windshield 3 through an opening 7 provided in the dashboard 10. In this way, the video light projection unit allows the user 6 to see the projected video light as a virtual image.

[0018] In detail, mirror M1 is, for example, a concave mirror (magnifying mirror), which reflects and magnifies the image light reflected by mirror M2 and projects it onto the display area 5 through the opening 7. The image light projected onto the display area 5 is reflected by the display area 5 and enters the eyes of the user 6. As a result, the user 6 sees the image light projected onto the display area 5 as a virtual image 9 existing beyond the transparent windshield 3, superimposed on the scenery outside the vehicle (roads, buildings, people, etc.). The information represented by the virtual image 9 includes various things, such as road signs, the current speed of the vehicle, and various information added to objects in the scenery, i.e., AR information.

[0019] Furthermore, the mirrors M1 and M2 may be, for example, free-form mirrors or mirrors with an asymmetrical optical axis shape. Here, the installation angle of mirror M2 is fixed. On the other hand, a mirror drive unit 14 is installed on mirror M1. The mirror drive unit 14 adjusts the installation angle of mirror M1. In detail, the mirror drive unit 14 includes, for example, a motor, and rotates mirror M1 by the rotational operation of the motor.

[0020] By adjusting the installation angle of the mirror M1, the position of the display area 5 on the windshield 3, that is, the vertical position of the virtual image viewed by the user 6, can be adjusted. Furthermore, by adjusting the installation angle of the mirror M1, it becomes possible to protect the image display unit 11 from sunlight. Specifically, sunlight can travel in the opposite direction to the optical path of the image light and enter the image display unit 11. If the possibility of damage to the image display unit 11 due to the entry of such sunlight increases, the installation angle of the mirror M1 can be changed so that sunlight does not reach the image display unit 11.

[0021] Figure 2B is a schematic diagram showing an example of the main components of the HUD device in Figure 1, but with a different configuration than that shown in Figure 2A. The HUD device 1 shown in Figure 2B differs from the configuration shown in Figure 2A in that a lens LS is provided inside the housing 12 instead of a mirror M2. The image light from the image display unit 11 enters the mirror M1 via the lens LS. The mirror M1 projects the incident image light onto the display area 5 through the opening 7, as in the case of Figure 2A. A mirror drive unit may also be installed in the mirror M1, as in the case of Figure 2A. The configuration shown in Figure 2B can be applied, for example, when the windshield 3 is installed at a near-vertical angle, such as in a minivan or truck.

[0022] Figure 3A is a block diagram showing an example configuration of the main components of the control system responsible for control in the HUD device shown in Figures 2A and 2B. The HUD device 1 shown in Figure 3A comprises a mirror drive unit 14, a display drive unit 15, a communication unit 16, a memory 17, a frame buffer 18, and a control unit 20, all connected to each other by a bus 13.

[0023] The communication unit 16 receives and transmits vehicle information and is implemented, for example, by a communication interface circuit, and functions as an information acquisition unit. The communication unit 16 acquires or receives information about the vehicle from the control unit 21 using CAN communication, etc., and transmits the received vehicle information to the control unit 20. The control unit 20 controls the mirror drive unit 14 and the display drive unit 15 based on the information from the communication unit 16. The mirror drive unit 14 adjusts the installation angle of the mirror M1, for example, in response to a command from the control unit 20, as shown in Figure 2A. The mirror drive unit 14 may be implemented by a motor as shown in Figure 2A, as well as a motor driver circuit that drives the motor.

[0024] The frame buffer 18 is composed of, for example, volatile memory and stores video data. The display driver unit 15 reads the video data stored in the frame buffer 18 via the bus 13 and drives the video display unit 11 based on the video data. The video display unit 11 is, for example, a liquid crystal display comprising a light source and a display panel. The display panel displays video by modulating the backlight irradiated from the light source pixel by pixel based on the video data. In this case, the display driver unit 15 can be implemented by an LCD driver circuit or the like.

[0025] Memory 17 is composed of, for example, a combination of volatile memory and non-volatile memory, and stores programs and data used by the control unit 20. The control unit 20 is implemented by a processor such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit), and controls the entire HUD device 1 by executing the programs stored in memory 17. As one example, the control unit 20 prepares video data, including the creation of video data, based on information about the vehicle acquired by the communication unit 16, i.e., the information acquisition unit, and displays the video based on the prepared video data on the video display unit 11.

[0026] The communication unit 16, memory 17, frame buffer 18, and control unit 20 shown in Figure 3A can be mounted on a microcontroller or the like. However, the implementation is not limited to this configuration; for example, an implementation combining an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) may also be used.

[0027] Figure 3B is a block diagram showing an example of the main configuration of the control system responsible for control in the HUD device shown in Figures 2A and 2B, but different from that shown in Figure 3A. The HUD device 1 shown in Figure 3B differs from the configuration example shown in Figure 3A in that it does not have a memory 17 and a control unit 20. In this case, the control unit 21 creates video data instead of the control unit 20 shown in Figure 3A, and writes the created video data to the frame buffer 18 via the communication processing unit 16a.

[0028] In the configuration example shown in Figure 3B, the control unit 20 described in Figure 3A may function as a communication processing unit 16a. Alternatively, the control unit 21 and the communication processing unit 16a may each handle some of the functions. In the configuration example in Figure 3B, a mirror drive unit 14, a display drive unit 15, and a communication processing unit 16a are provided. Unlike the communication unit 16 in Figure 3A, the communication processing unit 16a receives information about the vehicle from the control unit 21 using CAN communication or the like, processes the received information, and adjusts the operation of the mirror drive unit 14 and the display drive unit 15 based on the processing results.

[0029] Figure 4 is a block diagram showing an example configuration of the control unit in Figures 3A and 3B. The control unit 21 acquires vehicle information 4 as described in Figure 1. The vehicle information 4 is generated by various information acquisition devices such as sensors connected to the control unit 21, as shown in Figure 4. An example of such information acquisition device is shown in Figure 4.

[0030] In Figure 4, for example, the vehicle speed sensor 101 detects the speed of vehicle 2 in Figure 1 and generates speed information as the detection result. The shift position sensor 102 detects the current gear and generates gear information as the detection result. The steering angle sensor 103 detects the current steering angle and generates steering angle information as the detection result. The headlight sensor 104 detects whether the headlights are ON or OFF and generates lamp illumination information as the detection result. The illuminance sensor 105 and the chromaticity sensor 106 detect ambient light and generate ambient light information as the detection result.

[0031] The distance measuring sensor 107 detects the distance between the vehicle 2 and an external object and generates distance information as the detection result. The infrared sensor 108 detects the presence and distance of an object in close proximity to the vehicle 2 and generates infrared information as the detection result. The engine start sensor 109 detects whether the engine is ON or OFF and generates ON / OFF information as the detection result. The acceleration sensor 110 and the gyro sensor 111 detect the acceleration and angular velocity of the vehicle 2, respectively, and generate acceleration gyro information representing the attitude and behavior of the vehicle 2 as the detection result. The temperature sensor 112 detects the temperature inside and outside the vehicle and generates temperature information as the detection result.

[0032] The vehicle-to-infrastructure wireless receiver 113 generates vehicle-to-infrastructure communication information through vehicle-to-infrastructure communication between vehicle 2 and roads, signs, traffic lights, etc. The vehicle-to-vehicle wireless receiver 114 generates vehicle-to-vehicle communication information through vehicle-to-vehicle communication between vehicle 2 and other surrounding vehicles. The in-vehicle camera 115 and the exterior camera 116 generate in-vehicle camera image information and exterior camera image information by capturing images of the inside and outside of the vehicle, respectively. The in-vehicle camera 115 is, for example, a camera for a Driver Monitoring System (DMS) that captures the posture, eye position, movement, etc., of user 6 as shown in Figure 2A, etc. In this case, by analyzing the captured images, it is possible to understand the fatigue level and gaze position of user 6.

[0033] On the one hand, the external camera 116 of the vehicle, for example, captures the surrounding situation such as the front and rear of the vehicle 2. In this case, by analyzing the captured video, it becomes possible to grasp the presence or absence of obstacles such as other vehicles and people existing in the vicinity, the road surface conditions such as buildings, terrain, rain, snow accumulation, freezing, unevenness, etc., and road signs, etc. In addition, the external camera 116 may include, for example, a drive recorder that records the situation during driving in video.

[0034] The GPS receiver 117 generates GPS information obtained by receiving GPS signals. For example, the current time can be acquired by the GPS receiver 117. The VICS (Vehicle Information and Communication System, registered trademark) receiver 118 generates VICS information obtained by receiving VICS signals. The GPS receiver 117 and the VICS receiver 118 may be provided as part of the navigation device. Note that regarding the various information acquisition devices shown in FIG. 4, it is possible to appropriately delete them, add other types of devices, or replace them with other types of devices.

[0035] <Regarding the display of the HUD device> FIG. 5 is a schematic diagram showing an example of the display content of the HUD device shown in FIG. 1. The display content shows an example of AR display and shows an example of the virtual image 9 shown in FIG. 2A etc. In the example shown in FIG. 5, five videos VDa to VDe are displayed. In the specification, the plurality of videos VDa to VDe are collectively referred to as video VD, or image VD, or video object VD.

[0036] The video VDa is one of the landscapes and is displayed so as to overlap the object OB, which is an object of AR and here is a person. The video VDa means that the object OB has been detected by the various information acquisition devices shown in FIG. 4. That is, the HUD device 1 acquires information representing the detection result of the object OB from the control unit 21. In addition, the video VDa represents warning information for prompting the user 6, for example, the driver, to pay attention to the object OB.

[0037] Video VDb is displayed on a road, which is part of the scenery, and represents the direction of travel of vehicle 2. Video VDc represents navigation information. Video VDd represents, for example, a road sign, which is part of vehicle-to-infrastructure communication information. Video VDe represents the speed information of vehicle 2. Note that videos VDa and VDb are 3D graphics, while videos VDc to VDe are 2D graphics.

[0038] Figure 6 is a flowchart showing an example of the processing procedure when displaying video in the HUD device shown in Figure 3A. In Figure 6, first, the communication unit 16, i.e., the information acquisition unit, acquires information about the vehicle from the control unit 21 (step S11). Next, the control unit 20 performs video data preparation processing including steps S121 to S124 (step S12). For example, the processor executes the processing in steps S121 to S123 by executing the video processing program in memory 17.

[0039] In step S121, the control unit 20 determines the display content based on the vehicle information acquired in step S11, for example, the vehicle information 4 shown in Figure 1. Specifically, the control unit 20 selects the vehicle information 4 that corresponds to the display and determines where, what size, and in what layout the video VD representing the selected vehicle information 4 will be displayed. In the example shown in Figure 5, the control unit 20 decides to display five video VDa to VDe as the display content. In step S122, the control unit 20 creates video data for each video VD based on the display content determined in step S121.

[0040] In step S123, the control unit 20 writes the multiple video data created in step S122 to the storage area in the frame buffer 18 corresponding to the position to be displayed. In step S124, the control unit 20 performs distortion correction on the entire video data in the frame buffer 18, for example, according to the curvature of the windshield 3. Distortion correction may be achieved, for example, by the processor executing a distortion correction program in the memory 17, or by dedicated hardware.

[0041] After completing the preparation process for the video data (step S12), the video display process is performed (step S13). In step S13, the display drive unit 15 reads the video data stored in the frame buffer 18 and drives the video display unit 11 based on the video data to display the video VD on the video display unit 11. The processing procedure shown in Figure 6 is executed in synchronization with a processing cycle determined based on the frame rate. For example, if the frame rate is 60fps, the processing cycle is 16.6ms. Furthermore, the processing procedure for displaying the video VD is not limited to that shown in Figure 6, but can be replaced with various generally known procedures.

[0042] Figure 7 is a timing chart that schematically illustrates an example of a problem that may arise when displaying video using the flow shown in Figure 6. Figure 7 shows the operations performed in the first to fifth processing cycles or control cycles Tc[1] to Tc[5]. In this specification, the processing cycles or control cycles Tc[1] to Tc[5] are collectively referred to as the processing cycle or control cycle Tc. The control cycle Tc is, for example, 16.6 ms. Figure 7 also shows the operation when the processes of steps S11 and S12 in Figure 6 and the process of step S13 are executed in a pipeline.

[0043] In Figure 7, during the first control cycle Tc[1], information about the vehicle is acquired (step S11), and video data is prepared based on this information (step S12). The preparation time Tp[1] for the video data required for steps S11 and S12 is shorter than the control cycle Tc[1]. This is the case when, for example, the number of AR objects, and therefore the number of video VDs to be displayed, is small, and the preparation time Tp[1] fits within the control cycle Tc[1]. Similarly, during the second control cycle Tc[2], information about the vehicle is acquired (step S11), and video data is prepared based on this information (step S12). In parallel with this, during the second control cycle Tc[2], the video VD display process is performed based on the video data prepared during the first control cycle Tc[1] (step S13).

[0044] Here, in the second control cycle Tc[2], the preparation time Tp[2] for the video data required for processing steps S11 and S12 is longer than the control cycle Tc[2]. The preparation time Tp[2] can be prolonged, for example, by increasing the number of objects to be used for AR, and consequently the number of video VDs to be displayed. As a result, this video data is not reflected in the display processing in the third control cycle Tc[3], but is reflected in the display processing in the fourth control cycle Tc[4]. Consequently, frame drops occur in the third control cycle Tc[3].

[0045] Furthermore, in the fourth control cycle Tc[4], the preparation time Tp[3] for the video data required for processing steps S11 and S12 is shorter than the control cycle Tc[4]. Therefore, this video data is reflected in the display processing in the fifth control cycle Tc[5]. However, depending on the method, due to frame drops, the video data prepared in preparation time Tp[3] may actually be the video data that should have been prepared in the third control cycle Tc[3]. In this specification, the multiple preparation times Tp[1] to Tp[3] are collectively referred to as preparation time Tp.

[0046] Figure 8A is a schematic diagram showing how the displayed content changes when no frame drops occur. Figure 8B is a schematic diagram showing how the displayed content changes when frame drops occur. For example, as shown in Figures 8A and 8B, let's assume that an object OB, in this case a person, moves from time t1 to time t2. When no frame drops occur, as shown in Figure 8A, at time t1, the image VDa(t1) is superimposed on the object OB(t1) at that position, and at time t2, the image VDa(t2) is superimposed on the object OB(t2) at that position.

[0047] Thus, when no frame drops occur, a highly real-time video VD that tracks the AR object can be displayed. On the other hand, when frame drops occur, especially when they occur consecutively over a continuous control cycle Tc, as shown in Figure 8B, the video VDa(t2) displayed on object OB(t2) at time t2 is shifted towards object OB(t1). In this case, video VDa(t2) is not completely superimposed on object OB(t2). Thus, when frame drops occur, a video VD with low real-time capabilities that does not track the AR object may be displayed.

[0048] <Overview of Control Unit Operation> Therefore, in order to display highly real-time video VD, the control unit 20 performs control to reduce the processing load required to prepare the video data, so that the preparation of the video data is completed within a predetermined processing cycle or control cycle Tc. In other words, the control unit 20 prepares the video data based on the vehicle information received by the communication unit 16, and if the conditions for preparing the video data are not met within a predetermined processing cycle, it modifies some of the content of the video data to be prepared and prepares it. More specifically, the control unit 20 monitors the preparation time Tp required for preparing the video data, and when the preparation time Tp meets predetermined conditions, it starts control to reduce the processing load. The predetermined processing cycle or control cycle Tc is determined based on the frame rate.

[0049] The preparation time Tp is the time required for processing steps S11 and S12 shown in Figure 6, assuming the operation method described in Figure 7. On the other hand, assuming an operation method in which the processing of step S11 and the processing of step S12 are executed in a pipeline, the preparation time Tp may be the time required for processing step S12. However, to improve real-time performance, it is preferable to use the operation method described in Figure 7.

[0050] The control unit 20 continuously monitors the preparation time Tp. For example, the preparation time can be monitored in conjunction with the vehicle's time, or using a timer. Assuming the operation method described in Figure 7, as shown in Figure 3A, the control unit 20 monitors the preparation time Tp required from the start of acquiring vehicle-related information using the communication unit 16 until the creation of the video data is written to the frame buffer 18, using a timer. When using the configuration shown in Figure 3B, unlike in Figure 3A, the communication processing unit 16a acquires information from the control unit 21, processes the acquired information, and monitors the preparation time Tp required until the processing results are written to the frame buffer 18, using a timer.

[0051] Figure 9 shows an example of the internal states of the control unit shown in Figure 3A. Figure 10 shows an example of the monitoring results of the preparation time by the control unit shown in Figure 3A. As shown in Figure 9, the control unit 20 has four internal states: a normal state ST0, a recovery transition state ST1, a suppression transition state ST2, and a suppression state ST3. In other words, the control unit 20 has four operating modes: a normal mode ST0, a recovery transition mode ST1, a suppression transition mode ST2, and a suppression mode ST3. Having four states is just one example; in another example, the control unit 20 may have only two states: a normal state ST0 and a suppression state ST3.

[0052] In the normal state ST0, the control unit 20 prepares video data as usual based on the information acquired using the communication unit 16 and displays the resulting video VD on the video display unit 11. In the normal state ST0, the control unit 20 transitions to the suppression transition state ST2 if (A) the preparation time Tp for the monitoring result is longer than the control cycle Tc, or (B) the preparation time Tp is longer than the first threshold time Tth1 for multiple consecutive times. The number of consecutive times is set to a value of, for example, 2 to 10 times.

[0053] As a specific example, in Figure 10, the preparation time Tp4 obtained at monitoring time tm4 is longer than the control period Tc. Also, the preparation times Tp2 and Tp3 obtained at consecutive monitoring times tm2 and tm3 are both shorter than the control period Tc, but longer than the first threshold time Tth1. The first threshold time Tth1 is a time shorter than the control period Tc, which is 16.6 ms, for example, 15.0 ms.

[0054] The control unit 20 transitions to the suppression transition state ST2 in the normal state ST0 if (A) a monitoring result such as preparation time Tp4 is obtained, or (B) multiple monitoring results such as preparation times Tp2, Tp3 are obtained consecutively, for example, twice in a row. Condition (A) is for quickly resolving frame drops. On the other hand, condition (B) is for preventing in advance a situation in the near future where the preparation time Tp becomes longer than the control period Tc, and consequently a frame drop occurs.

[0055] In the suppression transition state ST2, the control unit 20 starts control to reduce the processing load, and then gradually increases the amount of processing load reduction at each control cycle Tc within a predetermined suppression transition period. The suppression transition period is set to, for example, 5 seconds. As will be described in detail later, the control unit 20 reduces the amount of video data to be prepared by reducing the amount of video data to be prepared, for example, by not creating some video data or by simplifying some video data, thereby reducing the processing load. In this case, if the amount of video data is reduced drastically, the displayed content will also change drastically, which is undesirable from the user's perspective 6. Therefore, the control unit 20 gradually increases the amount of data to be reduced. Then, after the suppression transition period, for example, 5 seconds, the control unit 20 transitions to the suppression state ST3.

[0056] In the suppression state ST3, the control unit 20 prepares video data with the reduced processing load. Furthermore, in the suppression state ST3, if the preparation time Tp is shorter than the second threshold time Tth2 for a predetermined threshold duration TthD or longer, the control unit 20 transitions to the recovery transition state ST1. The second threshold time Tth2 is shorter than the control cycle Tc, for example, the same as the first threshold time Tth1, or shorter than the first threshold time Tth1. The threshold duration TthD is several times the control cycle Tc, for example, 5 seconds.

[0057] As a specific example, in Figure 10, the second threshold time Tth2 is set to a shorter time than the first threshold time Tth1. The preparation times Tp5 and Tp6 obtained at monitoring times tm5 and tm6 are both shorter than the second threshold time Tth2. The control unit 20 transitions to the recovery transition state ST1 when monitoring results such as preparation times Tp5 and Tp6 are obtained for a period of TthD or longer.

[0058] Thus, if the state of “preparation time Tp < second threshold time Tth2” occurs stably, it is expected that even if the system returns to the normal state ST0, the aforementioned conditions (A) and (B) will no longer be met. In some cases, the control unit 20 may, anticipating that the processing load will naturally decrease after a predetermined period of time, not perform such condition checks and simply transition to the return transition state ST1 after a period of time such as 5 seconds.

[0059] In the recovery transition state ST1, the control unit 20 gradually reduces the amount of processing load reduction at each control cycle Tc within a predetermined recovery transition period. The recovery transition period is set to, for example, 5 seconds. After the recovery transition period, for example 5 seconds, the control unit 20 transitions to the normal state ST0. By providing the recovery transition state ST1, undesirable situations from the user's perspective can be avoided, similar to the case of the suppression transition state ST2.

[0060] <Methods to reduce processing load> Figure 11A is a schematic diagram showing an example of the display content of the HUD device in the normal state shown in Figure 9. In the normal state ST0, for example, as shown in Figure 11A, seven images VDa1 to VDa3 and VDb to VDe are displayed. Images VDa1 and VDa2 are displayed superimposed on objects OB1 and OB2, respectively, in this case a person. Image VDa3 is displayed superimposed on object OB3, in this case a vehicle. Images VDb to VDe represent navigation information, direction of travel, road signs, and speed information, respectively, as in the case of Figure 5.

[0061] Figure 11B is a schematic diagram showing an example of the display content of the HUD device in the suppression transition state shown in Figure 9. In suppression transition state ST2, based on the display content shown in Figure 11A, two images VDa2 and VDd in Figure 11A are deleted, as indicated by reference numerals 201 and 203 in Figure 11B. Specifically, image VDa2, which is superimposed on object OB2 which is farther away than object OB1, and image VDd, which represents a road sign, are deleted. In addition, as indicated by reference numeral 202 in Figure 11B, image VDa3 in Figure 11A is simplified by making it uncolored, etc.

[0062] Furthermore, Figure 11B shows a video VDm1, such as a mark, to notify the user 6 that the suppression transition state ST2, or in other words, the suppression transition period, is being observed. In the suppression transition state ST2, the control unit 20 writes the video data of the video VDm1 as a template to a fixed storage area in the frame buffer 18. By displaying such a mark, the user 6 can recognize that the change in display content is due to the activation of the display suppression function, and not due to a malfunction.

[0063] When displaying the image shown in Figure 11B, the control unit 20 reduces the amount of video data prepared in step S12 by setting video VDa2 and VDd to be hidden and applying a simplified display to video VDa3 in step S121 shown in Figure 6. This reduces the processing load required for preparing the video data for the control unit 20. In other words, the control unit 20 shortens the time required for processing in step S122 and step S123. Furthermore, the control unit 20 gradually reduces the amount of data in this way.

[0064] Figure 11C is a schematic diagram showing an example of the display content of the HUD device in the suppression state shown in Figure 9. In suppression state ST3, the two images VDa1 and VDb in Figure 11B are further simplified, as indicated by reference numerals 301 and 302 in Figure 11C, based on the display content shown in Figure 11B. Specifically, the two images VDa1 and VDb are simplified by being uncolored or reduced in size. Furthermore, in Figure 11C, an image VDm2, such as a mark, is displayed to notify the user 6 that the suppression state ST3, or in other words, the suppression period, is being observed.

[0065] As an example of specific processing, the control unit 20 pre-stores a suppression table in memory 17 that defines the correspondence between the type and priority of vehicle information 4. For example, in the suppression transition state ST2, the control unit 20 selects vehicle information 4 in order of decreasing priority based on the suppression table, and gradually increases the number of selected vehicle information 4. Then, the control unit 20 sets the video VD representing the selected vehicle information 4 to be hidden, or simplifies it in a predetermined manner.

[0066] Here, the priority in the suppression table is determined by criteria such as the following: First, vehicle information 4 that contributes more to safe driving is given a higher priority. Also, if vehicle information 4 represents a warning and the object of the warning is a person or a vehicle, the person is given a higher priority. However, in this case, the priority may be weighted according to the distance between the vehicle and the object. For example, if the person is far away and the vehicle is extremely close, the vehicle may be given priority. Also, a person approaching the vehicle may be given a higher priority than a person moving away from the vehicle.

[0067] Furthermore, regarding the video VDb showing the direction of travel as shown in Figures 11B and 11C, for example, based on navigation information, the priority may be lower when there is a straight line ahead and higher when the timing for a right or left turn is approaching. Also, regarding the video VDb, based on past driving history, the priority may be lower when driving on frequently used roads and higher when driving on roads that have not been used before. In addition, regarding the video VDd showing road signs, i.e., speed limits, as shown in Figures 11A and 11B, the priority may be changed based on the difference between the vehicle's current speed and the road sign.

[0068] Furthermore, the control unit 20 may reduce the processing load by, for example, displaying the video VD once every multiple control cycles Tc, rather than completely deleting the video VD. In addition, the control unit 20 may reduce the processing load not only by reducing the amount of video data in this way, but also by, for example, simplifying the distortion correction process in step S124 in Figure 6, and consequently reducing its accuracy.

[0069] <Detailed operation of the control unit> Figure 12 is a flowchart showing an example of the processing content of the control unit shown in Figure 3A. The control unit 20 executes the flow shown in Figure 12 when, for example, the adjustment of the installation angle of the mirror M1 shown in Figure 2A is completed and the environment for projecting images is ready. In Figure 12, the control unit 20 starts monitoring the preparation time Tp using a timer or the like (step S20). Subsequently, the control unit 20 waits for a start trigger to occur (step S21). A start trigger is generated for each control cycle Tc.

[0070] When the start trigger occurs in step S21, the control unit 20 uses the communication unit 16, i.e., the information acquisition unit, to acquire information about the vehicle (step S22). Next, the control unit 20 checks the current internal state as shown in Figure 9 (step S23). Then, the control unit 20 performs the video data preparation process as shown in step S12 in Figure 6 (step S24). At this time, if the internal state is any state other than the normal state ST0, the control unit 20 performs control to reduce the processing load required for preparing the video data, as shown in Figures 9, 11B, and 11C.

[0071] When the control unit 20 completes the preparation processing of the video data in step S24, it generates a ready signal (step S25). The control unit 20 outputs a display start command to the display drive unit 15 in response to a start trigger after generating the ready signal. The display drive unit 15 performs the display processing of the video VD in response to the display start command, as described in step S13 in Figure 6 and in Figure 7. The control unit 20 also evaluates the preparation time Tp, which is the monitoring result associated with step S21, when it has completed the preparation processing of the video data (step S26).

[0072] Then, based on the monitoring result, the preparation time Tp, the control unit 20 determines whether or not the state transition described in Figure 9 is necessary (step S27). If a state transition is necessary (step S27: Yes), the control unit 20 determines the transition destination (step S29), transitions to the determined destination, and updates the internal state (step S30). On the other hand, if a state transition is not necessary (step S27: No), the control unit 20 returns to step S21 and waits for the next start trigger until a request to terminate the HUD display occurs, and repeats the same process (step S28).

[0073] <Main effects of Embodiment 1> As described above, in the method of Embodiment 1, control is performed to reduce the processing load required for preparing video data so that the preparation of video data is completed within the control cycle Tc, thereby suppressing the occurrence of dropped frames and ensuring a minimum level of display quality. In detail, even if the HUD device 1 is designed to prevent dropped frames, etc., the processing load may increase due to software updates involving changes in display specifications, which may cause dropped frames. Using the method of Embodiment 1, even in such cases, the occurrence of dropped frames can be suppressed. As a result, it becomes possible to satisfy the required display specifications as much as possible within the processing performance of the hardware, and to respond flexibly to the display specifications.

[0074] (Embodiment 2) The method in Embodiment 1 reduced the processing load by actually monitoring the preparation time Tp required to prepare the video or image data, so that the preparation of the video or image data was completed within the control cycle Tc, or in other words, the processing cycle. Embodiment 2 describes a method that reduces the processing load by predicting the preparation time Tp, so that the preparation of the video or image data was completed within the control cycle Tc, or in other words, the processing cycle. The following explanation will use video data.

[0075] <Regarding the preparation process for video data> Figure 13A is a schematic diagram showing an example of the relationship between video data written to the frame buffer and the display content shown on the video display unit in the HUD device shown in Figures 3A and 3B. Figure 13B is a schematic diagram showing an example of a different relationship from that in Figure 13A. For example, as described in Figure 6, in the video data preparation process (step S12), video data based on the determined display content is written to the frame buffer 18 (steps S122, S123).

[0076] In Figure 13A, the six video streams VDa1-VDa3, VDb, VDc, and VDe are written as video data to the memory area within the frame buffer 18 corresponding to the position where they should be displayed. As described in Figure 5, video streams VDa1-VDa3 are warning videos superimposed on the detected object. Video stream VDb represents the direction of travel of the vehicle, video stream VDc represents navigation information, and video stream VDe represents the vehicle's speed information. For example, video streams VDa1-VDa3 and VDb are 3D graphics, while video streams VDc and VDe are 2D graphics.

[0077] In the example shown in Figure 13A, the size of the frame buffer 18 is equal to the size of the video display unit 11. In this case, the video display unit 11 displays all of each video VD, in other words, all of each video object VD, written to the frame buffer 18, in their original positions. On the other hand, in the example shown in Figure 13B, the size of the frame buffer 18 is larger than the size of the video display unit 11. In this case, the video display unit 11 displays only a portion of each video VD written to the frame buffer 18.

[0078] In Figure 13B, for example, to accommodate cases where the display moves up and down due to pitch correction, video VD is also placed in the area outside the display area of ​​the video display unit 11 within the frame buffer 18. In this case, the amount of processing load required for preparation is determined by how much video VD is placed in the entire frame buffer 18, including the area outside the display area of ​​the video display unit 11.

[0079] Figure 14 shows an example of the factors that affect the preparation time of video data in Figure 6. Factors that affect the preparation time include the number of video VDs and the display format for each video VD. Regarding the number of video VDs, the more VDs there are, the greater the processing load, and consequently, the longer the preparation time Tp. The display format for each video VD includes items such as size, display position, design type, and distortion correction.

[0080] Regarding size, the larger the size, the longer the preparation time Tp. Regarding display position, generally, the closer the display position is, the larger the size, and therefore the longer the preparation time Tp. Design types include, for example, 2D / 3D graphics and types such as with / without gradients. For example, the more complex the design, such as using 3D graphics with a large polygon count and gradients, the longer the preparation time Tp will be. Also, as mentioned in Figure 6, the preparation time Tp associated with distortion correction is shorter when distortion correction is performed in hardware and longer when distortion correction is performed in software.

[0081] Figure 15 is a flowchart showing an example of the processing procedure for displaying video in the HUD device according to Embodiment 2. The HUD device 1 according to Embodiment 2 is implemented with the configuration shown in Figure 3A or Figure 3B described above. Figure 15 shows the same flow as in Figure 6. That is, as in Figure 6, the control unit 20 determines the display content based on the vehicle information acquired by the communication unit 16, i.e., the information acquisition unit, prepares video data based on the determined display content, and displays the video based on the prepared video data on the video display unit 11.

[0082] However, in Figure 15, unlike in Figure 6, the control unit 20 determines the display content after appropriately adjusting it. For this reason, the content of the video data preparation process (step S12A) in Figure 15 is slightly different from that in Figure 6. Specifically, in step S12A shown in Figure 15, the control unit 20 first determines the display content by performing a display content adjustment process (step S41A), and then creates video data based on the adjusted display content (step S42A). After that, the control unit 20 writes the created video data to the frame buffer 18 (step S123A), as in Figure 6, and performs distortion correction using hardware processing or software processing (step S124A).

[0083] Figure 16 is a flowchart showing an example of the detailed processing of the display content adjustment process (step S41A) in Figure 15. As will be described in detail later, the control unit 20 pre-stores prediction time information in memory 17, etc., which defines the relationship between the display content, for example, the difference in display content, and the predicted time required to prepare the video data. Based on this premise, the control unit 20 first provisionally determines the display content based on the vehicle information acquired in step S11 in Figure 15, before determining the display content (step S411). Subsequently, the control unit 20 predicts the preparation time Tp required to prepare the video data based on the provisionally determined display content, also called the provisional display content, based on the prediction time information (step S412).

[0084] Next, the control unit 20 determines whether the preparation time Tp predicted in step S412 is shorter than a predetermined control cycle Tc, in other words, the processing cycle (step S413). If the preparation time Tp is longer than the control cycle Tc (step S413: No), the control unit 20 changes the provisionally determined display content and then returns to step S412 to repeat the same process (step S414). As a result, the control unit 20 changes the provisionally determined display content, i.e., the temporary display content, so that the preparation time Tp is shorter than the control cycle Tc. On the other hand, if the preparation time Tp is shorter than the control cycle Tc (step S413: Yes), the control unit 20 determines the provisionally determined display content, i.e., the current temporary display content, as the final display content (step S415).

[0085] <Method for predicting preparation time> Figure 17A shows a specific example of the preparation time prediction method (step S412) in Figure 16. Figure 17B shows a different specific example from Figure 17A. As shown in Figures 17A and 17B, the control unit 20 stores in advance prediction time information 400 which defines the relationship between the difference in display content and the prediction time required to prepare the video data. Specifically, the prediction time information 400 includes a base time, 0.8 [msec] in this example, and coefficients corresponding to each item included in the display format for each video shown in Figure 14.

[0086] In other words, the predicted time information 400 includes a size coefficient C1, a display position coefficient C2, and a design type coefficient C3. The design type coefficient C3 includes a polygon count coefficient C31 and a gradient coefficient C32. The size coefficient C1 is a coefficient proportional to the size of the video VD. The size coefficient C1 is fixed for each type of video VD, such as the video VDa1 representing a warning, the video VDb representing the direction of travel, and the video VDc representing navigation information, as shown in Figure 13A.

[0087] The display position coefficient C2 is set to increase as the display position of the video VD gets closer. In this example, the display position is divided into three stages: close, intermediate, and far. The display position coefficient C2 is set to 1.5 when the display position is close, 1.0 when it is intermediate, and 0.5 when it is close. The polygon count coefficient C31 is set to increase as the number of polygons used when rendering the video VD increases. In this example, the number of polygons is divided into two stages: above the baseline and below the baseline, i.e., many and standard. The polygon count coefficient C31 is set to 1.2 when the number of polygons is many and 1.0 when it is standard.

[0088] The gradation coefficient C32 is a coefficient that depends on whether or not there is a gradation. In this example, the gradation coefficient C32 is set to 1.1 if there is a gradation, and to 1.0 if there is no gradation. The base time and the values ​​of each coefficient are determined by taking into account the time required for the video data creation process in step S42A, in other words, the drawing process, the time required for writing to the frame buffer 18 in step S123A, and the time required for distortion correction in step S124A, as shown in Figure 15. Specific methods for determining these values ​​include, for example, methods based on simulations or methods based on measured values.

[0089] The control unit 20 uses this predicted time information 400 to predict the preparation time Tp by multiplying the base time by each coefficient. Specifically, for each video VD, the control unit 20 calculates the predicted time Tr required to prepare the video data using the formula "base time × C1 × C2 × C31 × C32". However, the method for calculating the predicted time Tr is not limited to this, and in some cases it may be calculated using "base time × C1 × C2 × C31". Furthermore, if there are other parameters that affect the predicted time Tr, the predicted time Tr may be calculated taking those parameters into consideration.

[0090] Furthermore, when calculating the predicted time Tr, instead of calculating it each time from the basic time and coefficients, information indicating how much time is required for preparation when displaying, for example, which video, at what size, at what position, and with what design, may be stored in advance as predicted time information. In this case, the preparation time required needs to be stored for each video VD, but the predicted time Tr does not need to be calculated each time. The control unit 20 predicts the preparation time Tp by summing the predicted time Tr for each video VD calculated by the above method.

[0091] As a concrete example, let's assume that in step S411 shown in Figure 16, the display content is tentatively determined to display three videos VD1 to VD3 as shown in Figure 17A. Video VD1 has a size factor C1 of 5 and is displayed at a close display position using 3D graphics with a gradient. Video VD2 has a size factor C1 of 2 and is displayed at an intermediate display position using 2D graphics. Video VD3 has a size factor C1 of 10 and is displayed at a far display position using 2D graphics with a gradient.

[0092] In this case, the control unit 20 calculates the estimated time Tr required to prepare the video data for video VD1 as 7.9 [msec] = 0.8 × C1 (=5) × C2 (=1.5) × C31 (=1.2) × C32 (=1.1). Similarly, the control unit 20 calculates the estimated time Tr required to prepare the video data for video VD2 as 1.6 [msec] and the estimated time Tr required to prepare the video data for video VD3 as 4.4 [msec].

[0093] The control unit 20 then calculates the preparation time Tp as 13.9 [msec] = 7.9 + 1.6 + 4.4 by summing the predicted times Tr for each of the calculated video VD1 to VD3. For example, if the control period Tc is 16.6 ms, the predicted preparation time Tp will be shorter than the control period Tc. Therefore, in step S415 shown in Figure 16, the control unit 20 determines the three video VD1 to VD3, which have the same display format, as the display content.

[0094] As another specific example, consider the case where the display content is tentatively determined to display two additional videos, VD4 and VD5, in addition to the three videos VD1 to VD3 shown in Figure 17A, as shown in Figure 17B. Video VD4 has a size factor C1 of 8 and is displayed in an intermediate display position using 2D graphics. Video VD5 has a size factor C1 of 2 and is displayed in a nearby display position using 3D graphics.

[0095] In this case, the control unit 20 calculates the predicted time Tr required to prepare the video data for video VD4 as 6.4 [msec] = 0.8 × C1 (=8) × C2 (=1.0) × C31 (=1.0) × C32 (=1.0). Similarly, the control unit 20 calculates the predicted time Tr required to prepare the video data for video VD5 as 2.9 [msec]. Then, the control unit 20 sums the calculated predicted times Tr for each of the videos VD1 to VD5 to predict a preparation time Tp of 23.2 [msec]. The predicted preparation time Tp is longer than the control cycle Tc. Therefore, in step S414 shown in Figure 16, the control unit 20 changes the display content, i.e., the provisionally determined display content.

[0096] <How to change the displayed content> Figure 18 illustrates an example of how to change the display content in Figure 16 (step S414). The control unit 20 changes the number of video VDs or changes at least one of the display format, i.e., size, display position, and design type for each video VD, so that the preparation time Tp is shorter than the control cycle Tc. At this time, the control unit 20 stores priority information 410, which defines the priority of each item to be changed, in the memory 17 or the like, for example, as shown in Figure 18. The control unit 20 then changes the display content by increasing the number of items to be changed based on the priority information 410 until the preparation time Tp is shorter than the control cycle Tc.

[0097] In the priority information 410 shown in Figure 18, the number of video VDs is given a lower priority than the display format for each video VD. In the example shown in Figure 18, the priorities for changing the display format for each video VD are, in descending order, design type, size, display position, and distortion correction. Based on the priority information 410, the control unit 20 first changes the design type of each video VD. Specifically, the control unit 20 changes, for example, 3D graphics to 2D graphics and gradients to no gradients.

[0098] As a result, in Figure 16, if the re-predicted preparation time Tp becomes shorter than the control cycle Tc (steps S412, S413), the control unit 20 determines the display content using the modified video VD (step S415). On the other hand, if the re-predicted preparation time Tp is still longer than the control cycle Tc (steps S412, S413), the control unit 20 reduces the size of each video VD in addition to the design type of each video VD. Specifically, the control unit 20, for example, pre-determines a lower limit for the size of each type of video VD, and sequentially reduces the size of each video VD until the lower limit is reached.

[0099] As a specific example, in FIG. 17B, assume that the lower limit value of the size for video VD3, specifically, the lower limit value of size coefficient C1 is set to 8. In this case, the control unit 20 first changes the size of video VD3 to a size where the size coefficient C1 becomes 9, and if Tp > Tc still holds, it changes the size to a size where the size coefficient C1 becomes 8. At this time, depending on the type of video VD, a video VD for which the change of the size coefficient C1 is not permitted, that is, a video VD for which the lower limit value is not set, may also be set.

[0100] Subsequently, in the same manner, based on the priority information 410, the control unit 20 sequentially adds the display position and the distortion correction to the item to be changed until Tp < Tc. When changing the display position, the control unit 20 reduces the size by moving the display position deeper. At this time, depending on the type of video VD, a video VD for which the change of the display position is not permitted may also be set. Also, when changing the item of distortion correction, the control unit 20 applies, for example, a predetermined simplified distortion correction or does not perform the distortion correction itself.

[0101] Even if the display format for each video VD is changed in this way and Tp > Tc still holds, as a last resort, the control unit 20 reduces the number of videos VD. At this time, the control unit 20 determines in which order to reduce the videos VD based on, for example, the suppression table described in FIG. 11C. In the suppression table, as described above, the priority is set so that the vehicle information 4 that contributes greatly to safe driving, in other words, the type of video VD, is less likely to be a deletion target.

[0102] Figure 19 is a schematic diagram showing an example of the display content of a HUD device obtained as a result of changing the display content in Figure 18. The upper part of Figure 19 shows the display content before the change, which is the same as in Figure 11A. That is, seven images VDa1 to VDa3 and VDb to VDe are displayed here. Images VDa1 and VDa2 are displayed superimposed on objects OB1 and OB2, respectively, in this case a person. Image VDa3 is displayed superimposed on object OB3, in this case a vehicle. Images VDb, VDc, VDd, and VDe represent the direction of travel, navigation information, road signs, and speed information, respectively. Images VDa1 to VDa3 are assumed to use 3D graphics with gradients.

[0103] The lower section of Figure 19 shows the modified display content. In the modified display content, compared to the original display content, video VDa1 to VDa3 are replaced with video VDa1x to VDa3x, respectively. Video VDa1x to VDa3x use 2D graphics and no gradients. Furthermore, the size of video VDa1x to VDa3x is slightly smaller than that of video VDa1 to VDa3.

[0104] Furthermore, in the example shown in Figure 19, even when using video VDa1x to VDa3x, the preparation time Tp was still longer than the control cycle Tc, so the video VDd representing speed information was deleted, as indicated by the reference numeral 405. In other words, in this example, the video VDd representing speed information was deleted on the assumption that its contribution to safe driving was smaller than that of the other video VDs. By using this modification method, it becomes possible to change the display content in a way that maintains the display content before the change as much as possible from the perspective of safe driving, while not causing significant discomfort to drivers or other users.

[0105] Figure 20 is a timing chart for displaying video using the flows shown in Figures 15 and 16. Similar to Figure 7, Figure 20 shows the operations performed in the first to fourth processing cycles or control cycles Tc[1] to Tc[4]. The control cycle Tc is, for example, 16.6 ms. As shown in Figure 20, if the display content is not changed, for example, the preparation time Tp[2a] in control cycle Tc[2] may be longer than the control cycle Tc[2]. In this case, as described in Figure 7, frame drops will occur.

[0106] On the other hand, by changing the display content, for example, the preparation time Tp[2b] in the control cycle Tc[2] can be made shorter than the control cycle Tc[2]. As a result, frame drops can be prevented. Note that the preparation time Tp also includes the time required for the display content adjustment process (step S41A) shown in Figures 15 and 16. Therefore, the time required for step S41A is an overhead time within the preparation time Tp, but it is usually sufficiently small compared to the time required for steps S42, S123, and S124, and can be considered negligible.

[0107] Furthermore, in the method of Embodiment 1 described above, since the actual preparation time Tp is monitored and control is performed based on the monitoring results, in other words, feedback control is performed, so a situation like the preparation time Tp[2a] shown in Figure 20 may occur momentarily. On the other hand, in the method of Embodiment 2, the preparation time Tp is predicted and control is performed, in other words, feedforward control is performed, so ideally, a situation like the preparation time Tp[2a] shown in Figure 20 will not occur.

[0108] <Main effects of Embodiment 2> As described above, the method of Embodiment 2 also provides the same effects as those described in Embodiment 1. Specifically, it suppresses or prevents frame drops and ensures a minimum level of display quality. Furthermore, it satisfies the required display specifications as much as possible within the processing capabilities of the hardware, and allows for flexible adaptation to those specifications. Moreover, by predicting the preparation time Tp, it is less likely to excessively change or suppress the display content compared to the method of Embodiment 1, thus enabling a further improvement in display quality.

[0109] Furthermore, the method of Embodiment 1 is particularly useful when there are many variable elements in the display specifications, such as in initial products, making it difficult to create the predicted time information 400 shown in Figures 17A and 17B. On the other hand, the method of Embodiment 2 is particularly useful when there are few variable elements in the display specifications, such as in somewhat mature products. Also, since the method of Embodiment 2 requires the creation of predicted time information 400, the method of Embodiment 1 may be more useful in terms of flexibility or versatility when dealing with display specifications that fluctuate greatly.

[0110] (Embodiment 3) <Details of the control unit> Figure 21A is a schematic diagram showing an example of the main processing content of the control unit in the HUD device according to Embodiment 3, and Figure 21B is a supplementary diagram to Figure 21A. Figure 22A is a schematic diagram showing an example different from Figure 21A, and Figure 22B is a supplementary diagram to Figure 22A. The HUD device 1 according to Embodiment 3 is realized with the configuration shown in Figure 3A or Figure 3B described above. The control unit 20 superimposes a warning image onto the object OB3, in this case the vehicle, as shown in the image VDa3a in Figure 21A and the image VDa3b in Figure 22A.

[0111] In this case, the control unit 20 determines the color or shape of the warning images VDa3a and VDa3b according to the distance to the object OB3. In the examples of Figures 21A and 21B, the distance between the vehicle 420 equipped with the HUD device 1 and the object OB3 located in front of the vehicle 420 is 50m. In this case, the control unit 20 sets the color of image VDa3a to, for example, green. On the other hand, in the examples of Figures 22A and 22B, the distance between the vehicle 420 and the object OB3 is 10m. In this case, the control unit 20 sets the color of image VDa3b to, for example, red.

[0112] Figure 23 shows an example of the configuration of the display setting table stored by the control unit in the HUD device according to Embodiment 3. The control unit 20 stores, for example, a display setting table 415 as shown in Figure 23 in the memory 17 beforehand. The display setting table 415 defines the correspondence between the distance to the object OB3 in front and the color or shape of the warning image, in this case, the color. In this example, the color of the warning image is set to red when the distance to the object OB3 is less than 12m, green when it is 32m or more, and yellow when it is in the range of 17m to 27m. In addition, the color for the intermediate distances, such as the range of 12m to 17m and the range of 27m to 32m, is set to an appropriate intermediate color.

[0113] The display setting table 415 is configured such that, for example, the color or shape becomes more attention-grabbing to the user, such as the driver, as the distance to the object OB3 decreases. This contributes to safer driving. However, the perception of color or shape depends on the user's subjective perception. Therefore, the display setting table 415 may be configured to allow the user to arbitrarily select the color or shape from multiple options during initial setup. Furthermore, the display setting table 415 may be configured to allow the user to arbitrarily set the distance range.

[0114] Figure 24A is a flowchart showing an example of the processing content of the control unit in the HUD device according to Embodiment 3. In Figure 24A, the control unit 20 executes the processing of steps S411 to S415 described in Figure 16 in Embodiment 2. Prior to this processing, the communication unit 16, i.e., the information acquisition unit, acquires the distance between the vehicle 420 and the object OB3 located in front of the vehicle 420 as one of the pieces of information related to the vehicle in step S11 shown in Figure 15.

[0115] The control unit 20 determines the display content by performing the processes in steps S411 to S415 shown in Figure 24A, such that the preparation time Tp is shorter than the control cycle Tc, or in other words, the processing cycle. Subsequently, the control unit 20 determines whether the display content determined in step S415 includes a warning image superimposed on object OB3 (step S416A). If the determined display content includes a warning image (step S416A: Yes), the control unit 20 refers to the distance information to object OB3 acquired by the communication unit 16 (step S417) and updates the color or shape of the warning image based on the display setting table 415 (step S418A). If there are multiple warning images, the update process is performed for each warning image.

[0116] As shown in the flow diagram in Figure 24A, if the determined display content includes a warning image, the control unit 20 updates the color or shape of the warning image without re-predicting the preparation time Tp, i.e., without going through processing such as step S412 (step S418A). For this reason, it is desirable that the color or shape of the warning image for each distance be predetermined so that even if the color or shape of the warning image is updated, the preparation time Tp remains the same as before the update.

[0117] Figure 24B is a flowchart showing an example of processing content different from that in Figure 24A. In the flowchart shown in Figure 24B, unlike the flowchart shown in Figure 24A, the processing similar to that of steps S416A, S417, and S418A described in Figure 24A is incorporated into the processing of steps S411 to S415, rather than after the processing of steps S411 to S415 described in Figure 16.

[0118] In Figure 24B, the control unit 20 provisionally determines the display content (step S411), and then determines whether the provisionally determined display content includes a warning image superimposed on object OB3 (step S416B). If the provisionally determined display content includes a warning image (step S416B: Yes), the control unit 20 refers to the distance information to object OB3 (step S417) and determines the color or shape of the warning image based on the display setting table 415 (step S418B).

[0119] Then, the control unit 20 predicts the preparation time Tp required to prepare the video data based on the provisionally determined display content (step S412). If the provisionally determined display content does not include a warning video (step S416B: No), the control unit 20 proceeds directly to step S412. After step S412, the control unit 20 modifies the provisionally determined display content through the processing in steps S413 to S415, as in the case of Figure 16, so that the preparation time Tp becomes shorter than the control cycle Tc.

[0120] However, unlike in Figure 16, the control unit 20 changes the provisionally determined display content and then proceeds to step S416B (step S414). As a result, even if the color or shape of the warning image is changed by the processing in step S414, it can be correctly restored by the processing in steps S417 and S418B, and the preparation time Tp can be predicted (step S412).

[0121] Using the flow shown in Figure 24B, the preparation time Tp can be predicted while reflecting changes in color or shape in the warning video, which may improve prediction accuracy compared to using the flow shown in Figure 24A. However, using the flow shown in Figure 24B requires processing steps S416B, S417, and S418B each time a loop from step S414 to step S416B occurs, which may increase processing overhead. Therefore, from this perspective, using the flow shown in Figure 24A is more advantageous.

[0122] Furthermore, the processing based on the display setting table 415, as described in steps S416A, S417, and S418A in Figure 24A, is not limited to the method of Embodiment 2, but may also be applied to the method of Embodiment 1. In this case, the control unit 20 may, for example, determine the display content based on the internal state in step S121 shown in Figure 6, and at the final stage, execute the processing described in steps S416A, S417, and S418A.

[0123] <Main effects of Embodiment 3> As described above, by using the method of Embodiment 3, in addition to the various effects described in Embodiments 1 and 2, it becomes possible to contribute to safer driving by changing the color or shape of the warning image according to the distance to the object.

[0124] Furthermore, using the methods of each embodiment, the user 6 can view various information necessary for driving, such as navigation information including destination and speed, as well as alert information when oncoming vehicles or pedestrians are detected, as images through the windshield 3. Even if the display specifications change, the user can view images with a minimum level of display quality ensured. This makes it possible to provide a HUD device 1 that reduces the user 6's eye movement and contributes to supporting safe driving. As a result, it becomes possible to prevent traffic accidents. Moreover, it becomes possible to contribute to "3. Good Health and Well-being" of the United Nations' Sustainable Development Goals (SDGs).

[0125] The present invention has been described in detail above based on embodiments, but the present invention is not limited to the embodiments described above and can be modified in various ways without departing from its essence. For example, the embodiments described above are described in detail in order to explain the present invention in an easy-to-understand manner and are not necessarily limited to those having all the described configurations. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add a configuration from another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with a configuration from another embodiment. [Explanation of symbols]

[0126] 1...Head-Up Display (HUD) device, 2...Vehicle, 4...Vehicle information, 5...Display area, 6...User, 11...Video display unit, 16...Communication unit (information acquisition unit), 20...Control unit, 400...Predicted time information, 410...Priority information, 415...Display setting table, C1~C3,C31,C32...Coefficients, M1...Mirror (video light projection unit), ST0...Normal state, ST1...Return transition state, ST2...Suppression transition state, ST3...Suppression state, Tc...Control cycle, Tp...Preparation time, Tr...Predicted time, Tth1...First threshold time, Tth2...Second threshold time, TthD...Threshold duration, VD...Video

Claims

1. It is a vehicle, An information acquisition unit that acquires information about the aforementioned vehicle, A video display unit that displays an image and emits video light from the displayed image, A video light projection unit projects the video light emitted from the video display unit onto a display area, thereby causing the projected video light to be perceived as a virtual image. A control unit that determines the display content based on the information about the vehicle acquired by the information acquisition unit, prepares video data based on the determined display content, and causes the video display unit to display the video based on the prepared video data, Equipped with, Before determining the display content, the control unit predicts the preparation time required to prepare the video data based on the provisionally determined display content, based on predicted time information that defines the relationship between the display content and the predicted time required to prepare the video data. If the preparation time required to prepare the video data for the provisionally determined display content before determining the display content is longer than a predetermined processing cycle, the control unit modifies the provisionally determined display content so that it becomes shorter than the processing cycle. vehicle.

2. In the vehicle described in claim 1, The items to be changed when modifying the provisionally decided display content include the number of videos, or the display format for each video. The items used to change the display format for each video include at least one of the following: size, display position, and design type. vehicle.

3. In the vehicle described in claim 2, The control unit pre-stores priority information that defines the priority of each item, and changes the provisionally determined display content while increasing the number of items to be changed based on the priority information until the preparation time becomes shorter than the processing cycle. vehicle.

4. In the vehicle described in claim 3, The items used when changing the provisionally determined display content include the number of images and the display format for each image. In the priority information, the number of videos is set to have a lower priority than the display format for each video. vehicle.

5. In the vehicle described in claim 2, The predicted time information includes a base time and a coefficient corresponding to the item included in the display format for each video, The control unit predicts the preparation time by multiplying the base time by the coefficient. vehicle.

6. In the vehicle described in claim 1, The information acquisition unit acquires the distance between the vehicle and an object located in front of the vehicle. When the control unit superimposes a warning image onto the object, it determines the color or shape of the warning image according to the distance from the object. vehicle.

7. In the vehicle described in claim 6, The control unit, after determining the display content such that the preparation time is shorter than the processing cycle, updates the color or shape of the warning video if the determined display content includes the warning video, without re-predicting the preparation time. vehicle.

Images