Head-up display device and video data processing method

Abstract

To provide a head-up display device and a method for processing video data with which it is possible to flexibly respond to various display specifications, as well as contribute to Sustainable Development Goal 3: Good Health and Well-Being.SOLUTION: The head-up display device comprises: a video display unit that displays a video and emits video light of the displayed video; a video light projection unit that projects the emitted video light to a display region so as to have the projected video light as a virtual image visually recognized; and a control unit that determines display content, prepares video data, and causes the video display unit to display the video based on the data. The control unit changes provisionally determined display content when the preparation time required for preparing the video data of provisionally determined display content before display content is determined is longer than a prescribed processing period.SELECTED DRAWING: Figure 16

Description

[Technical Field]

[0001] The present invention relates to a head-up display device and a method for processing video data, and relates to technology for a head-up display device that uses AR (Augmented Reality), for example. [Background technology]

[0002] Patent Document 1 discloses a display system that extracts an object to be displayed in AR based on imaging data, generates AR image data of the object, and sets the frame rate of the AR image data based on the importance of the object. For example, when the overall frame rate is 60 fps and three AR image data are displayed in a time-division manner, the frame rates of the three AR image data are set to 30 fps, 20 fps, and 10 fps, respectively, according to their importance. This allows for more suppressed flickering of AR image data with higher importance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-6164 Summary of the Invention [Problem to be solved by the invention]

[0004] A head-up display device prepares various image data based on information acquired from ADAS (Advanced Driver Assistance Systems) and projects image light based on the image data onto a display area to be viewed as a virtual image. The preparation time required to prepare the image data varies depending on the type and number of images, the size of the image according to the virtual image distance, the display format (2D / 3D, etc.), the presence or absence of graphics effects, etc. In this specification, a head-up display device is also referred to as a HUD device.

[0005] In a typical HUD device, processing performance such as a maximum frame rate is fixedly determined based primarily on the hardware specifications. However, there may be a desire to change the display specifications of the HUD device by updating software without changing the hardware itself. In such cases, the required display specifications of the HUD device often involve an increased processing load. As a result, there is a risk that the hardware processing performance may be insufficient, resulting in failure to meet the required display specifications, such as frame dropping during AR display.

[0006] The present invention has been made in view of the above, and one of its objects is to provide a head-up display device and a video data processing method that can satisfy required display specifications as much as possible within the range of hardware processing performance and can flexibly respond to display specifications.

[0007] The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings. [Means for solving the problem]

[0008] Among the inventions disclosed in this application, the outline of representative inventions will be briefly explained as follows.

[0009] A typical head-up display device includes an image display unit that displays an image and emits image light of the displayed image, an image light projection unit that projects the emitted image light onto a display area to make the projected image light visible as a virtual image, and a control unit that determines display content based on acquired information about the vehicle, prepares image data based on the determined display content, and displays an image based on the prepared image data on the image display unit. If the preparation time required to prepare the image data of the provisionally determined display content before determining the display content is longer than a predetermined processing cycle, the control unit changes the provisionally determined display content. [Effects of the Invention]

[0010] To briefly explain the effect obtained by a representative invention among the inventions disclosed in the present application, it becomes possible to flexibly accommodate display specifications in a head-up display device. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic diagram showing an example of the configuration of a vehicle equipped with a head-up display device according to a first embodiment. [Figure 2A] 2 is a schematic diagram showing an example of the configuration of a main part of the HUD device in FIG. 1. [Figure 2B] 2B is a schematic diagram showing an example of the configuration of the main part of the HUD device in FIG. 1, which is different from that in FIG. 2A. [Figure 3A] 2C is a block diagram showing an example of the configuration of a main part of a control system that is responsible for control in the HUD device shown in FIGS. 2A and 2B. FIG. [Figure 3B] 3A is a block diagram showing an example of the configuration of a main part of a control system that is responsible for control in the HUD device shown in FIGS. 2A and 2B, the main part being different from that shown in FIG. 3A. FIG. [Figure 4] 3A and 3B are block diagrams showing an example of the configuration of a portion related to a control unit. [Figure 5] 2 is a schematic diagram showing an example of the display content of the HUD device shown in FIG. 1. FIG. [Figure 6] 3B is a flowchart showing an example of a processing procedure for displaying an image in the HUD device shown in FIG. 3A. FIG. [Figure 7] 7 is a timing chart schematically illustrating an example of a problem that occurs when displaying video using the flow shown in FIG. 6. [Figure 8A] FIG. 10 is a schematic diagram showing how the display content changes when no dropped frames occur. [Figure 8B] 10A and 10B are schematic diagrams showing how display content changes when a frame is dropped. [Figure 9] 3B is a diagram showing an example of an internal state of a control unit shown in FIG. 3A. FIG. [Figure 10]3B is a diagram showing an example of a result of monitoring the preparation time by the control unit shown in FIG. 3A. FIG. [Figure 11A] 10 is a schematic diagram showing an example of the display content of the HUD device in the normal state shown in FIG. 9. [Figure 11B] 10 is a schematic diagram showing an example of the display content of the HUD device in the suppression transition state shown in FIG. 9. [Figure 11C] 10 is a schematic diagram showing an example of the display content of the HUD device in the suppressed state shown in FIG. 9. [Figure 12] 3B is a flowchart showing an example of processing performed by a control unit shown in FIG. 3A. FIG. [Figure 13A] 3C is a schematic diagram showing an example of the relationship between video data written to a frame buffer and display content displayed on a video display unit in the HUD device shown in FIGS. 3A and 3B. FIG. [Figure 13B] FIG. 13B is a schematic diagram showing an example of a relationship different from that shown in FIG. 13A. [Figure 14] FIG. 6 shows an example of items that affect the preparation time for video data. [Figure 15] 10 is a flowchart showing an example of a processing procedure for displaying a video in the HUD device according to the second embodiment. [Figure 16] 16 is a flowchart showing an example of detailed processing contents of the display content adjustment processing in FIG. 15. FIG. [Figure 17A] FIG. 17 is a diagram showing a specific example of a method for predicting preparation time (step S412) in FIG. [Figure 17B] FIG. 17B is a diagram showing a specific example different from FIG. 17A. [Figure 18] FIG. 17 is a diagram illustrating an example of a method for changing the display content in FIG. 16 (step S414). [Figure 19] FIG. 19 is a schematic diagram showing an example of the display content of the HUD device obtained as a result of changing the display content in FIG. 18. [Figure 20] 17 is a timing chart for displaying video using the flow shown in FIGS. 15 and 16. [Figure 21A]In the HUD device according to Embodiment 3, it is a schematic diagram showing an example of the main processing content of the control unit. [Figure 21B] It is a supplementary diagram of FIG. 21A. [Figure 22A] It is a schematic diagram showing an example different from FIG. 21A. [Figure 22B] It is a supplementary diagram of FIG. 22A. [Figure 23] In the HUD device according to Embodiment 3, it is a diagram showing a configuration example of a display setting table stored in the control unit. [Figure 24A] In the HUD device according to Embodiment 3, it is a flowchart showing an example of the processing content of the control unit. [Figure 24B] It is a flowchart showing an example of processing content different from FIG. 24A.

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described in detail based on the drawings. In all the drawings for explaining the embodiments, the same members are generally denoted by the same reference numerals, and the repeated description thereof is omitted.

[0013] (Embodiment 1) <Overview of HUD Device> FIG. 1 is a schematic diagram showing a configuration example of a vehicle equipped with a head-up display device according to Embodiment 1. The head-up display (HUD) device 1 shown in FIG. 1 is mounted on a vehicle 2 which is one of the vehicles. The vehicle 2 is typically an automobile, but is not necessarily limited thereto, and may be a railway vehicle or the like. Further, the vehicle is not limited to a vehicle, and may be an aircraft or the like. Further, the vehicle 2 is equipped with a control unit 21 called an ECU (Electronic Control Unit), for example.

[0014] The control unit 21 acquires the vehicle information 4, for example, from various sensors installed in various parts of the vehicle 2, as well as from a navigation device or the like. The various sensors detect, for example, various events that occur in the vehicle 2 and various parameter values ​​related to the driving situation. The HUD device 1 acquires the vehicle information 4 acquired by the control unit 21, for example, using CAN (Controller Area Network) communication or the like.

[0015] The vehicle information 4 includes, for example, speed information and gear information of the vehicle 2, steering wheel steering angle information, lamp illumination information, external light information, distance information, infrared information, engine ON / OFF information, camera image information inside and outside the vehicle, acceleration gyro information, GPS (Global Positioning System) information, navigation information, vehicle-to-vehicle communication information, and road-to-vehicle communication information. The GPS information also includes information such as the current time. The vehicle information 4 also includes various warning information. The HUD device 1 projects video light onto a display area such as the windshield 3 based on such vehicle information 4. As a result, the HUD device 1 allows a user such as a driver to visually recognize the video light projected onto the display area as a virtual image, specifically, as a virtual image superimposed on the scenery ahead of the vehicle 2.

[0016] Fig. 2A is a schematic diagram showing an example configuration of the main parts of the HUD device in Fig. 1. The HUD device 1 shown in Fig. 2A includes, for example, an image display unit 11 housed in a housing 12, mirrors M1 and M2, and a mirror driver 14. The image display unit 11 is, for example, a display panel such as an LCD (Liquid Crystal Display) or a projector, and displays an image based on input image data and emits image light of the displayed image.

[0017] Mirror M2 reflects the image light from image display unit 11 toward mirror M1. Mirror M2 is effective in saving space and ensuring a long optical path length. Depending on the space inside the HUD housing and the required optical path length, mirror M2 may not be provided, or multiple mirrors M2 may be provided. Mirror M1 functions as an image light projection unit. Mirror M1, which is the image light projection unit, projects the image light emitted from image display unit 11 and reflected by mirror M2 onto display area 5 of windshield 3 through opening 7 provided in dashboard 10. As a result, the image light projection unit allows user 6 to view the projected image light as a virtual image.

[0018] Specifically, mirror M1 is, for example, a concave mirror (magnifying mirror), which reflects and magnifies the image light reflected by mirror M2 and projects it onto display area 5 through opening 7. The image light projected onto display area 5 is reflected by display area 5 and enters the eyes of user 6. As a result, user 6 visually recognizes the image light projected onto display area 5 as a virtual image 9 existing beyond transparent windshield 3, superimposed on the scenery outside the vehicle (roads, buildings, people, etc.). The information represented by virtual image 9 includes various types of information, such as road signs, the vehicle's current speed, and various types of information added to objects in the scenery, i.e., AR information.

[0019] Furthermore, mirrors M1 and M2 may be, for example, free-form surface mirrors or mirrors having an asymmetric shape with respect to the optical axis. Here, the installation angle of mirror M2 is fixed. On the other hand, mirror M1 is provided with a mirror driver 14. The mirror driver 14 variably adjusts the installation angle of mirror M1. In detail, mirror driver 14 includes, for example, a motor, and rotates mirror M1 by the rotational operation of the motor.

[0020] By adjusting the installation angle of the mirror M1, it is possible to adjust the position of the display area 5 on the windshield 3, i.e., the vertical position of the virtual image viewed by the user 6. Furthermore, by adjusting the installation angle of the mirror M1, it is possible to protect the image display unit 11 from sunlight. Specifically, sunlight can travel in the opposite direction along the optical path of the image light and enter the image display unit 11. If the incidence of sunlight increases the possibility of damaging the image display unit 11, it is sufficient to change the installation angle of the mirror M1 so that sunlight does not reach the image display unit 11.

[0021] FIG. 2B is a schematic diagram showing an example of the configuration of the main parts of the HUD device in FIG. 1, which is different from that shown in FIG. 2A. The HUD device 1 shown in FIG. 2B differs from the configuration shown in FIG. 2A in that a lens LS is provided in place of the mirror M2 within the housing 12. Image light from the image display unit 11 is incident on the mirror M1 via the lens LS. As in the case of FIG. 2A, the mirror M1 projects the incident image light onto the display area 5 through the opening 7. As in the case of FIG. 2A, a mirror driver may be provided on the mirror M1. The configuration shown in FIG. 2B can be applied to cases where the windshield 3 is installed at an angle close to vertical, such as in a minivan or truck.

[0022] Fig. 3A is a block diagram showing an example of the configuration of the main parts of the control system that are responsible for control in the HUD device shown in Fig. 2A and Fig. 2B. HUD device 1 shown in Fig. 3A includes mirror driver 14, display driver 15, communication unit 16, memory 17, frame buffer 18, and control unit 20, which are connected to each other via bus 13.

[0023] The communication unit 16 receives and transmits vehicle information and is realized, for example, by a communication interface circuit or the like, and functions as an information acquisition unit. The communication unit 16 acquires or receives information about the vehicle from the control unit 21 using CAN communication or the like, and transmits the received information about the vehicle to the control unit 20. The control unit 20 controls the mirror drive unit 14 and the display drive unit 15 based on the information from the communication unit 16. The mirror drive unit 14 adjusts the installation angle of the mirror M1, as described in FIG. 2A, in response to, for example, a command from the control unit 20. The mirror drive unit 14 can be realized by a motor such as that described in FIG. 2A, as well as a motor driver circuit or the like that drives the motor.

[0024] Frame buffer 18 is configured, for example, by a volatile memory and stores video data. Display driver 15 reads the video data stored in frame buffer 18 via bus 13 and drives video display unit 11 based on the video data. Video display unit 11 is, for example, a liquid crystal display equipped with a light source and a display panel. The display panel displays video by modulating backlight emitted from the light source for each pixel based on the video data. In this case, display driver 15 can be realized by an LCD driver circuit or the like.

[0025] The memory 17 is configured, for example, by a combination of volatile memory and non-volatile memory, and stores programs, data, etc. used by the control unit 20. The control unit 20 is realized by a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and controls the entire HUD device 1 by executing the programs stored in the memory 17. As one of its functions, the control unit 20 prepares video data, including the creation of video data, based on information about the vehicle acquired by the communication unit 16, i.e., the information acquisition unit, and causes the video display unit 11 to display an image based on the prepared video data.

[0026] 3A may be mounted on a microcontroller or the like. However, the implementation is not limited to this, and may be, for example, an implementation in which an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or the like is appropriately combined.

[0027] Fig. 3B is a block diagram showing an example configuration of a main part of the control system that is responsible for control in the HUD device shown in Fig. 2A and Fig. 2B, which is different from Fig. 3A. Unlike the example configuration shown in Fig. 3A, the HUD device 1 shown in Fig. 3B is configured without the memory 17 and the control unit 20. In this case, the control unit 21 creates video data in place of the control unit 20 shown in Fig. 3A, and writes the created video data to the frame buffer 18 via the communication processing unit 16a.

[0028] In the configuration example shown in Fig. 3B, the control unit 20 described in Fig. 3A may function as the communication processing unit 16a. Alternatively, some functions may be shared between the control unit 21 and the communication processing unit 16a. The configuration example of Fig. 3B is provided with a mirror driving unit 14, a display driving unit 15, and a communication processing unit 16a. Unlike the communication unit 16 of Fig. 3A, the communication processing unit 16a receives information about the vehicle from the control unit 21 using CAN communication or the like, processes the received information, and adjusts the operations of the mirror driving unit 14 and the display driving unit 15 based on the processing results.

[0029] Fig. 4 is a block diagram showing an example of the configuration of the parts related to the control unit in Fig. 3A and Fig. 3B. The control unit 21 acquires vehicle information 4 as described in Fig. 1. As shown in Fig. 4, the vehicle information 4 is generated by information acquisition devices such as various sensors connected to the control unit 21. Fig. 4 shows an example of the information acquisition device.

[0030] In FIG. 4, for example, a vehicle speed sensor 101 detects the speed of the vehicle 2 in FIG. 1 and generates speed information as the detection result. A shift position sensor 102 detects the current gear and generates gear information as the detection result. A steering wheel angle sensor 103 detects the current steering wheel angle and generates steering wheel angle information as the detection result. A headlight sensor 104 detects whether the headlights are on or off and generates lamp illumination information as the detection result. An illuminance sensor 105 and a chromaticity sensor 106 detect external light and generate external light information as the detection result.

[0031] The distance measurement sensor 107 detects the distance between the vehicle 2 and an external object, and generates distance information that represents the detection result. The infrared sensor 108 detects the presence or absence of an object in the vicinity of the vehicle 2, as well as the distance thereto, and generates infrared information that represents the detection result. The engine start sensor 109 detects whether the engine is on or off, and generates on or off information that represents the detection result. The acceleration sensor 110 and the gyro sensor 111 detect the acceleration and angular velocity of the vehicle 2, respectively, and generate acceleration gyro information that represents the attitude and behavior of the vehicle 2 as the detection result. The temperature sensor 112 detects the temperature inside and outside the vehicle, and generates temperature information that represents the detection result.

[0032] The road-to-vehicle communication wireless receiver 113 generates road-to-vehicle communication information through road-to-vehicle communication between the vehicle 2 and roads, signs, traffic lights, etc. The vehicle-to-vehicle communication wireless receiver 114 generates vehicle-to-vehicle communication information through vehicle-to-vehicle communication between the vehicle 2 and other surrounding vehicles. The interior camera 115 and the exterior camera 116 capture images of the interior and exterior of the vehicle, respectively, to generate interior camera image information and exterior camera image information. The interior camera 115 is, for example, a camera for a DMS (Driver Monitoring System) that captures the posture, eye position, movement, etc. of the user 6 shown in FIG. 2A, etc. In this case, the fatigue level and gaze position of the user 6 can be determined by analyzing the captured images.

[0033] On the one hand, the external camera 116 of the vehicle, for example, captures the surrounding situation such as the front and rear of the vehicle 2. In this case, by analyzing the captured video, it becomes possible to grasp the presence or absence of obstacles such as other vehicles and people existing in the vicinity, road surface conditions such as buildings, terrain, rain, snow accumulation, freezing, unevenness, etc., and road signs. Further, the external camera 116 may include, for example, a drive recorder that records the situation during driving in video.

[0034] The GPS receiver 117 generates GPS information obtained by receiving GPS signals. For example, the GPS receiver 117 can acquire the current time. The VICS (Vehicle Information and Communication System, registered trademark) receiver 118 generates VICS information obtained by receiving VICS signals. The GPS receiver 117 and the VICS receiver 118 may be provided as part of the navigation device. Regarding the various information acquisition devices shown in FIG. 4, it is possible to appropriately delete them, add other types of devices, or replace them with other types of devices.

[0035] <Regarding the display of the HUD device> FIG. 5 is a schematic diagram showing an example of the display content of the HUD device shown in FIG. 1. The display content shows an example of AR display and shows an example of the virtual image 9 shown in FIG. 2A etc. In the example shown in FIG. 5, five videos VDa to VDe are displayed. In the specification, the plurality of videos VDa to VDe are collectively referred to as video VD, or image VD, or video object VD.

[0036] The video VDa is one of the landscapes and is displayed so as to be superimposed on an object OB, which is an object of AR, here a person. The video VDa means that the object OB has been detected by the various information acquisition devices shown in FIG. 4. That is, the HUD device 1 acquires information representing the detection result of the object OB from the control unit 21. Further, the video VDa represents warning information for prompting the user 6, for example, the driver, to pay attention to the object OB.

[0037] Image VDb is displayed on a road, which is one type of scenery, and shows the traveling direction of vehicle 2. Image VDc shows navigation information. Image VDd shows, for example, road signs, which are one type of road-to-vehicle communication information. Image VDe shows speed information of vehicle 2. Note that images VDa and VDb are 3D graphics, and images VDc to VDe are 2D graphics.

[0038] Fig. 6 is a flow diagram showing an example of a processing procedure when displaying an image in the HUD device shown in Fig. 3A. In Fig. 6, first, the communication unit 16, i.e., the information acquisition unit, acquires information about the vehicle from the control unit 21 (step S11). Next, the control unit 20 performs a preparation process for the image data including steps S121 to S124 (step S12). For example, the processor executes the image processing program in the memory 17 to perform the processes of steps S121 to S123.

[0039] In step S121, the control unit 20 determines the display content based on the information about the vehicle acquired in step S11, for example, the vehicle information 4 shown in FIG. 1. Specifically, the control unit 20 selects the vehicle information 4 that is compatible with the display, and determines the position, size, and layout in which the video VD representing the selected vehicle information 4 should be displayed. In the example shown in FIG. 5, the control unit 20 determines that five videos VDa to VDe should be displayed as the display content. In step S122, the control unit 20 creates video data for each video VD based on the display content determined in step S121.

[0040] In step S123, control unit 20 writes each of the multiple pieces of video data created in step S122 to a storage area in frame buffer 18 corresponding to the position where the video data should be displayed. In step S124, control unit 20 performs distortion correction on all of the video data in frame buffer 18, for example, in accordance with the curvature of windshield 3. Note that distortion correction may be achieved, for example, by a processor executing a distortion correction program in memory 17, or may be achieved by dedicated hardware.

[0041] After completing this video data preparation process (step S12), video display processing is performed (step S13). In step S13, display drive unit 15 reads out the video data stored in frame buffer 18 and drives video display unit 11 based on the video data, thereby causing video display unit 11 to display the video VD. The processing procedure shown in FIG. 6 is executed in synchronization with a processing cycle determined based on the frame rate. For example, when the frame rate is 60 fps, the processing cycle is 16.6 ms. The processing procedure for displaying the video VD is not limited to that shown in FIG. 6, and can be replaced with various commonly known procedures.

[0042] FIG. 7 is a timing chart schematically illustrating an example of a problem that occurs when displaying video using the flow shown in FIG. 6. FIG. 7 shows operations executed in the first to fifth processing or control cycles Tc[1] to Tc[5]. In the specification, the processing or control cycles Tc[1] to Tc[5] are collectively referred to as the processing cycle or control cycle Tc. The control cycle Tc is, for example, 16.6 ms. FIG. 7 also shows operations when the processes of steps S11 and S12 and the process of step S13 in FIG. 6 are executed in a pipeline.

[0043] 7, in the first control period Tc[1], information about the vehicle is acquired (step S11), and video data is prepared based on the acquired information (step S12). The video data preparation time Tp[1] required for the processing of steps S11 and S12 is shorter than the control period Tc[1]. This is the case, for example, when the number of AR objects, and therefore the number of videos VD to be displayed, is small, and the preparation time Tp[1] fits within the control period Tc[1]. Similarly, in the second control period Tc[2], information about the vehicle is acquired (step S11), and video data is prepared based on the acquired information (step S12). In parallel with this, in the second control period Tc[2], display processing of the video VD is performed based on the video data prepared in the first control period Tc[1] (step S13).

[0044] Here, in the second control period Tc[2], the preparation time Tp[2] of the video data required for the processing of steps S11 and S12 is longer than the control period Tc[2]. The preparation time Tp[2] can be lengthened, for example, due to an increase in the number of AR objects and thus the video VD to be displayed. As a result, the video data is not reflected in the display processing in the third control period Tc[3], but is reflected in the display processing in the fourth control period Tc[4]. As a result, dropped frames occur in the third control period Tc[3].

[0045] Furthermore, in the fourth control period Tc[4], the preparation time Tp[3] of the video data required for the processing of steps S11 and S12 is shorter than the control period Tc[4]. Therefore, the video data is reflected in the display processing in the fifth control period Tc[5]. However, depending on the method, due to the occurrence of frame dropping, the video data prepared in the preparation period Tp[3] may be the video data that should have been prepared in the third control period Tc[3]. In this specification, the multiple preparation periods Tp[1] to Tp[3] are collectively referred to as the preparation time Tp.

[0046] FIG. 8A is a schematic diagram showing how the display content changes when no dropped frames occur. FIG. 8B is a schematic diagram showing how the display content changes when dropped frames occur. For example, as shown in FIGS. 8A and 8B, assume that an object OB, in this case a person, moves from time t1 to time t2. When no dropped frames occur, as shown in FIG. 8A, an image VDa(t1) is displayed superimposed on the position of the object OB(t1) at time t1, and an image VDa(t2) is displayed superimposed on the position of the object OB(t2) at time t2.

[0047] In this way, when no frame dropping occurs, it is possible to display a video VD with high real-time performance that tracks the AR object. On the other hand, when frame dropping occurs, particularly when frame dropping occurs continuously in consecutive control cycles Tc, as shown in FIG. 8B, the video VDa(t2) is displayed at a position shifted toward the object OB(t1) from the position of the object OB(t2) at time t2. In this case, the video VDa(t2) is not completely superimposed on the object OB(t2). In this way, when frame dropping occurs, it is possible that a video VD with low real-time performance that does not track the AR object will be displayed.

[0048] <Outline of operation of the control unit> Therefore, in order to display the video VD with high real-time performance, the control unit 20 generally performs control to reduce the processing load required for preparing the video data so that the preparation of the video data is completed within a predetermined processing cycle or control period Tc. That is, the control unit 20 prepares the video data based on the information about the vehicle received by the communication unit 16, and if the preparation conditions for the video data do not satisfy the predetermined conditions within the predetermined processing cycle, the control unit 20 modifies some of the content of the video data to be prepared. More specifically, the control unit 20 monitors the preparation time Tp required to prepare the video data, and when the preparation time Tp satisfies a predetermined condition, it initiates control to reduce the processing load. The predetermined processing cycle or control period Tc is determined based on the frame rate.

[0049] Assuming an operation method such as that described in Fig. 7, the preparation time Tp is the time required for the processing of steps S11 and S12 shown in Fig. 6. On the other hand, assum- ing an operation method in which the processing of step S11 and the processing of step S12 are executed in a pipeline, for example, the preparation time Tp may be the time required for the processing of step S12. However, in order to improve real-time performance, it is desirable to use the operation method described in Fig. 7.

[0050] The control unit 20 continuously monitors the preparation time Tp. For example, the preparation time can be monitored in conjunction with the vehicle time or by using a timer or the like. Assuming the operation method described in FIG. 7, as shown in FIG. 3A, the control unit 20 monitors the preparation time Tp, which is required from the start of acquiring information about the vehicle using the communication unit 16 until the completion of writing the created video data to the frame buffer 18, by using a timer or the like. When the configuration shown in FIG. 3B is used, unlike the case of FIG. 3A, the communication processing unit 16a acquires information from the control unit 21, processes the acquired information, and monitors the preparation time Tp, which is required from the start of acquiring information about the vehicle using the communication unit 16 until the completion of writing the processed results to the frame buffer 18, by using a timer or the like.

[0051] FIG. 9 is a diagram showing an example of the internal state of the control unit shown in FIG. 3A. FIG. 10 is a diagram showing an example of the monitoring result of the preparation time by the control unit shown in FIG. 3A. As shown in FIG. 9, the control unit 20 has, as its internal states, a normal state ST0, a return transition state ST1, a suppression transition state ST2, and a suppression state ST3. In other words, the control unit 20 has, as its operating modes, a normal mode ST0, a return transition mode ST1, a suppression transition mode ST2, and a suppression mode ST3. Having four states is just one example, and in another example, the control unit 20 may have two states: the normal state ST0 and the suppression state ST3.

[0052] In the normal state ST0, the control unit 20 prepares video data as usual based on information acquired using the communication unit 16, and displays the video VD based on the video data on the video display unit 11. In addition, in the normal state ST0, the control unit 20 transitions to the suppression transition state ST2 if (A) the preparation time Tp, which is the monitoring result, is longer than the control period Tc, or (B) the preparation time Tp is longer than the first threshold time Tth1 multiple times in a row. The number of consecutive times is set to a value between 2 and 10, for example.

[0053] 10, the preparation time Tp4 obtained at the monitoring time tm4 is longer than the control period Tc. Furthermore, the preparation times Tp2 and Tp3 obtained at the consecutive monitoring times tm2 and tm3 are both shorter than the control period Tc but longer than the first threshold period Tth1. The first threshold period Tth1 is shorter than the control period Tc of 16.6 ms, such as 15.0 ms.

[0054] In the normal state ST0, the control unit 20 transitions to the suppression transition state ST2 when (A) a monitoring result such as preparation time Tp4 is obtained, or (B) a monitoring result such as preparation times Tp2 and Tp3 is obtained multiple times in succession, for example, twice in succession. Condition (A) is intended to quickly resolve dropped frames. On the other hand, condition (B) is intended to prevent a situation in the near future in which the preparation time Tp becomes longer than the control cycle Tc, thereby causing dropped frames.

[0055] After starting control to reduce the processing load in the suppression transition state ST2, the control unit 20 gradually increases the amount of reduction in the processing load for each control cycle Tc within a predetermined suppression transition period. The suppression transition period is set to, for example, 5 seconds. As will be described in detail later, the control unit 20 reduces the amount of video data to be prepared and the processing load by, for example, not creating some video data or simplifying some video data. In this case, if the amount of video data is suddenly reduced, the display content also suddenly changes, which is undesirable from the perspective of the user 6. Therefore, the control unit 20 gradually increases the amount of data to be reduced. Then, after the suppression transition period, for example, 5 seconds, has elapsed, the control unit 20 transitions to the suppression state ST3.

[0056] In the suppression state ST3, the control unit 20 prepares video data with the reduced processing load. Furthermore, when the state in which the preparation time Tp is shorter than the second threshold time Tth2 continues for a predetermined threshold duration TthD or longer in the suppression state ST3, the control unit 20 transitions to the return transition state ST1. The second threshold time Tth2 is shorter than the control period Tc, and is set to, for example, the same time as the first threshold time Tth1 or a time shorter than the first threshold time Tth1. The threshold duration TthD is a period that is at least multiple times the control period Tc, and is set to, for example, 5 seconds.

[0057] 10, the second threshold time Tth2 is set to be shorter than the first threshold time Tth1. The preparation times Tp5 and Tp6 obtained at the monitoring times tm5 and tm6 are both shorter than the second threshold time Tth2. The control unit 20 transitions to the return transition state ST1 when monitoring results such as the preparation times Tp5 and Tp6 are obtained continuously for a threshold duration TthD or longer.

[0058] In this way, when the state of "preparation time Tp<second threshold time Tth2" is occurring stably, even if the normal state ST0 is restored, it is expected that the above-mentioned conditions (A) and (B) will no longer be met. Note that in some cases, the control unit 20 may not perform such a condition determination, and may simply transition to the restoration transition state ST1 after a time such as 5 seconds has passed, anticipating that the processing load will naturally be reduced after a predetermined period of time has passed.

[0059] In the return transition state ST1, the control unit 20 gradually reduces the amount of processing load reduction for each control cycle Tc within a predetermined return transition period. The return transition period is set to, for example, 5 seconds. Then, the control unit 20 transitions to the normal state ST0 after the return transition period, for example, 5 seconds, has elapsed. By providing the return transition state ST1, it is possible to avoid situations that are undesirable from the perspective of the user 6, as in the case of the suppression transition state ST2.

[0060] <How to reduce processing load> Fig. 11A is a schematic diagram showing an example of the display content of the HUD device in the normal state shown in Fig. 9. In the normal state ST0, for example, as shown in Fig. 11A, seven images VDa1 to VDa3 and VDb to VDe are displayed. The images VDa1 and VDa2 are displayed so as to be superimposed on objects OB1 and OB2, respectively (people in this case). The image VDa3 is displayed so as to be superimposed on object OB3, a vehicle in this case. The images VDb to VDe represent navigation information, direction of travel, road signs, and speed information, respectively, as in Fig. 5.

[0061] Fig. 11B is a schematic diagram showing an example of the display content of the HUD device in the suppression transition state shown in Fig. 9. In the suppression transition state ST2, the display content shown in Fig. 11A is used as a reference, and as indicated by reference numerals 201 and 203 in Fig. 11B, two images VDa2 and VDd in Fig. 11A are deleted. That is, the image VDa2 superimposed on the object OB2 that is farther away than the object OB1, and the image VDd showing the road sign, are deleted. Furthermore, as indicated by reference numeral 202 in Fig. 11B, the image VDa3 in Fig. 11A is simplified by, for example, making it uncolored.

[0062] 11B, an image VDm1, such as a mark, is displayed to notify the user 6 that the display is in the suppression transition state ST2, or in other words, the display is in a suppression transition period. In the suppression transition state ST2, the control unit 20 writes the image data of the image VDm1 as a template to a fixed storage area in the frame buffer 18. By displaying such a mark, the user 6 can recognize that the change in the display content is not due to a malfunction, but is due to the activation of the display suppression function.

[0063] When performing the display shown in FIG. 11B, the control unit 20 determines in step S121 shown in FIG. 6 that videos VDa2 and VDd are not displayed and applies simplified display to video VDa3, thereby reducing the amount of video data prepared in step S12. In this way, the control unit 20 reduces the processing load required to prepare the video data. In other words, the control unit 20 shortens the time required for the processing in step S122 and the time required for the processing in step S123. Furthermore, the control unit 20 gradually reduces the amount of data.

[0064] Fig. 11C is a schematic diagram showing an example of the display content of the HUD device in the suppressed state shown in Fig. 9. In the suppressed state ST3, the two images VDa1 and VDb in Fig. 11B are further simplified as indicated by reference numerals 301 and 302 in Fig. 11C, based on the display content shown in Fig. 11B. That is, the two images VDa1 and VDb are simplified by removing color and reducing their size. Furthermore, Fig. 11C displays an image VDm2, such as a mark, to notify the user 6 that the suppressed state ST3, or in other words, the suppressed period, is in effect.

[0065] As an example of specific processing, the control unit 20 stores in advance in the memory 17 a suppression table that defines the correspondence between the types of vehicle information 4 and their priorities. For example, in the suppression transition state ST2, the control unit 20 selects vehicle information 4 in ascending order of priority based on the suppression table, and gradually increases the number of selected vehicle information 4. Then, the control unit 20 hides the video VD representing the selected vehicle information 4 or simplifies it in a predetermined manner.

[0066] Here, the priority in the suppression table is determined, for example, according to the following criteria: First, the vehicle information 4 that contributes more to safe driving is given a higher priority. Furthermore, when the vehicle information 4 indicates a warning and the object of the warning is a person or a vehicle, the priority of the person is given a higher priority. However, in this case, the priority may be weighted according to the distance between the vehicle and the object. For example, if the distance to the person is far and the distance to the vehicle is extremely close, the vehicle may be given priority. Furthermore, a person approaching the vehicle may be given a higher priority than a person moving away from the vehicle.

[0067] 11B and 11C, the priority of the image VDb showing the direction of travel may be lowered if the road is a series of straight lines, and may be higher if the timing for a right or left turn is approaching, based on navigation information. Furthermore, the priority of the image VDb may be lowered if the vehicle is traveling on a frequently used road, based on past driving history, and may be higher if the vehicle is traveling on a road that has not been used before. Furthermore, the priority of the image VDd showing the road sign, i.e., the speed limit, shown in FIGS. 11A and 11B may be changed based on the difference from the vehicle's driving speed.

[0068] Note that the control unit 20 may reduce the processing load by, for example, displaying the video VD once every multiple control cycles Tc, without completely deleting the video VD. Furthermore, the control unit 20 is not limited to such a method of reducing the amount of video data, and may reduce the processing load by, for example, simplifying the distortion correction process in step S124 in Fig. 6, thereby reducing the accuracy.

[0069] <Detailed operation of the control unit> Fig. 12 is a flow diagram showing an example of the processing contents of the control unit shown in Fig. 3A. The control unit 20 executes the flow shown in Fig. 12, for example, when adjustment of the installation angle of the mirror M1 shown in Fig. 2A and the like is completed and the environment for projecting an image is prepared. In Fig. 12, the control unit 20 starts monitoring the preparation time Tp using a timer or the like (step S20). Next, the control unit 20 waits for a start trigger to be generated (step S21). The start trigger is generated every control period Tc.

[0070] When a start trigger occurs in step S21, the control unit 20 acquires information about the vehicle using the communication unit 16, i.e., the information acquisition unit (step S22). Next, the control unit 20 checks the current internal state described in FIG. 9 (step S23). Then, the control unit 20 performs a video data preparation process as described in step S12 of FIG. 6 (step S24). At this time, if the internal state is other than the normal state ST0, the control unit 20 performs control to reduce the processing load required for preparing the video data, as described in FIGS. 9, 11B, and 11C.

[0071] When the control unit 20 completes the preparation process for the video data in step S24, it generates a preparation complete signal (step S25). For example, in response to a start trigger after generating the preparation complete signal, the control unit 20 outputs a display start command to the display drive unit 15. In response to the display start command, the display drive unit 15 performs display processing for the video VD as described in step S13 in FIG. 6 and in FIG. 7. Furthermore, when the control unit 20 completes the preparation process for the video data, it evaluates the preparation time Tp, which is the monitoring result obtained in step S21 (step S26).

[0072] 9 is necessary based on the preparation time Tp, which is the monitoring result (step S27). If a state transition is necessary (step S27: Yes), the control unit 20 determines a transition destination (step S29), performs a state transition to the determined transition destination, and also updates the internal state (step S30). On the other hand, if a state transition is not necessary (step S27: No), the control unit 20 returns to step S21 and waits for the next start trigger, repeating the same process until a request to end the HUD display is generated (step S28).

[0073] <Major Effects of the First Embodiment> As described above, the method of the first embodiment performs control to reduce the processing load required for preparing video data so that preparation of the video data is completed within the control period Tc, thereby suppressing the occurrence of dropped frames and ensuring a minimum display quality. In particular, even if the HUD device 1 is designed to prevent dropped frames, a software update or the like that involves a change in display specifications may increase the processing load and cause dropped frames. By using the method of the first embodiment, dropped frames and the like can be suppressed even in such cases. As a result, it becomes possible to satisfy the required display specifications as much as possible within the range of the processing performance of the hardware and flexibly respond to the display specifications.

[0074] (Embodiment 2) The method of the first embodiment is a method of reducing the processing load by actually monitoring the preparation time Tp required to prepare video data or image data so that the preparation of video data or image data is completed within the control period Tc, in other words, the processing period. In the second embodiment, a method of reducing the processing load by predicting the preparation time Tp so that the preparation of video data or image data is completed within the control period Tc, in other words, the processing period will be described. The following description will be given using video data.

[0075] <Video data preparation processing> Fig. 13A is a schematic diagram showing an example of the relationship between video data written to the frame buffer and the display content displayed on the video display unit in the HUD device shown in Fig. 3A and Fig. 3B. Fig. 13B is a schematic diagram showing an example of a relationship different from that shown in Fig. 13A. For example, as described in Fig. 6, in the video data preparation process (step S12), video data based on the determined display content is written to the frame buffer 18 (steps S122 and S123).

[0076] 13A, six images VDa1 to VDa3, VDb, VDc, and VDe are written as video data in storage areas in frame buffer 18 corresponding to the positions where they should be displayed. As described in FIG. 5, images VDa1 to VDa3 are warning images superimposed on detected objects. Image VDb represents the vehicle's traveling direction, image VDc represents navigation information, and image VDe represents vehicle speed information. Furthermore, for example, images VDa1 to VDa3 and VDb are 3D graphics, and images VDc and VDe are 2D graphics.

[0077] 13A, the size of the frame buffer 18 is equal to the size of the video display unit 11. In this case, each video VD written to the frame buffer 18, in other words, all of the video objects VD, are displayed in their original positions on the video display unit 11. On the other hand, in the example shown in FIG. 13B, the size of the frame buffer 18 is larger than the size of the video display unit 11. In this case, only a portion of each video VD written to the frame buffer 18 is displayed on the video display unit 11.

[0078] 13B, for example, in order to deal with cases where the display moves up and down due to pitching correction, the video VD is also arranged in an area of ​​the frame buffer 18 outside the display area of ​​the video display unit 11. In this case, the magnitude of the processing load required for the preparation process is determined by how much video VD is arranged in the entire frame buffer 18, including the area outside the display area of ​​the video display unit 11.

[0079] 14 is a diagram showing an example of items that affect the preparation time for video data in FIG. 6. Items that affect the preparation time include the number of video VDs and the display format for each video VD. With regard to the number of video VDs, the greater the number, the greater the processing load, and ultimately the longer the preparation time Tp. Items that affect the display format for each video VD include, for example, size, display position, design type, distortion correction, etc.

[0080] Regarding size, the larger the size, the longer the preparation time Tp. Regarding display position, the display position is usually closer to the front; in other words, the closer the display position, the larger the size, and therefore the longer the preparation time Tp. Design types include, for example, types such as 2D / 3D graphics and types such as whether or not there is a gradation. For example, the more complex the design, such as when using 3D graphics with a large number of polygons and gradations, the longer the preparation time Tp. Furthermore, as described in Figure 6, the preparation time Tp associated with distortion correction is shorter when distortion correction is performed by hardware and longer when distortion correction is performed by software.

[0081] Fig. 15 is a flow diagram showing an example of a processing procedure for displaying video in the HUD device according to embodiment 2. HUD device 1 according to embodiment 2 is realized by the configuration shown in Fig. 3A or 3B described above. Fig. 15 shows a flow similar to that shown in Fig. 6. That is, as in Fig. 6, control unit 20 determines display content based on information about the vehicle acquired by communication unit 16, i.e., the information acquisition unit, prepares video data based on the determined display content, and causes video display unit 11 to display video based on the prepared video data.

[0082] However, in FIG. 15, unlike the case of FIG. 6, control unit 20 determines the display content after appropriately adjusting the display content. Therefore, the content of the video data preparation process (step S12A) in FIG. 15 is slightly different from that in FIG. 6. That is, in step S12A shown in FIG. 15, control unit 20 first determines the display content by performing a display content adjustment process (step S41A), and then creates video data based on the adjusted display content (step S42A). Thereafter, control unit 20 writes the created video data to frame buffer 18 (step S123A), as in the case of FIG. 6, and performs distortion correction using hardware processing or software processing (step S124A).

[0083] FIG. 16 is a flow diagram showing an example of detailed processing of the display content adjustment processing (step S41A) in FIG. 15. As will be described in detail later, the control unit 20 stores, in advance in the memory 17 or the like, predicted time information that defines the relationship between the display content, for example, differences in the display content, and the predicted time required to prepare the video data. On this premise, the control unit 20 first provisionally determines the display content based on the information about the vehicle acquired in step S11 in FIG. 15 before determining the display content (step S411). Next, the control unit 20 predicts the preparation time Tp required to prepare the video data based on the provisionally determined display content (also referred to as provisional display content) based on the predicted time information (step S412).

[0084] Next, the control unit 20 determines whether the preparation time Tp predicted in step S412 is shorter than a predetermined control period Tc, in other words, the processing period (step S413). If the preparation time Tp is longer than the control period Tc (step S413: No), the control unit 20 changes the provisionally determined display content and then returns to step S412 to repeat the same process (step S414). As a result, the control unit 20 changes the provisionally determined display content, i.e., the provisional display content, so that the preparation time Tp is shorter than the control period Tc. On the other hand, if the preparation time Tp is shorter than the control period Tc (step S413: Yes), the control unit 20 determines the provisionally determined display content, i.e., the current provisional display content, as the final display content (step S415).

[0085] <How to estimate preparation time> FIG. 17A is a diagram showing a specific example of the preparation time prediction method (step S412) in FIG. 16. FIG. 17B is a diagram showing a specific example different from that in FIG. 17A. As shown in FIGS. 17A and 17B, control unit 20 pre-stores predicted time information 400 that defines the relationship between differences in display content and the predicted time required to prepare video data. Specifically, predicted time information 400 includes a basic time, 0.8 msec in this example, and coefficients corresponding to each item included in the display format for each video shown in FIG. 14.

[0086] That is, the predicted time information 400 includes a size coefficient C1, a display position coefficient C2, and a design type coefficient C3. The design type coefficient C3 includes a polygon count coefficient C31 and a gradation coefficient C32. The size coefficient C1 is a coefficient proportional to the size of the video VD. The size coefficient C1 is fixedly determined for each type of video VD, such as the video VDa1 showing a warning, the video VDb showing a direction of travel, and the video VDc showing navigation information shown in FIG. 13A.

[0087] The display position coefficient C2 is set to a larger value as the display position of the video VD gets closer. In this example, the display position is classified into three levels: close, medium, and far, and the display position coefficient C2 is set to 1.5 when the display position is close, 1.0 when it is medium, and 0.5 when it is close. The polygon count coefficient C31 is set to a larger value as the number of polygons used to render the video VD increases. In this example, the number of polygons is classified into two levels: above a reference value and below the reference value, i.e., high and standard, and the polygon count coefficient C31 is set to 1.2 when the number of polygons is high and 1.0 when it is standard.

[0088] The gradation coefficient C32 is a coefficient that depends on whether or not a gradation is present. In this example, the gradation coefficient C32 is set to 1.1 when a gradation is present, and 1.0 when a gradation is absent. The basic time and the values ​​of each coefficient are determined taking into consideration all of the time required for the video data creation process (in other words, the drawing process) in step S42A shown in FIG. 15, the time required for the writing process to the frame buffer 18 in step S123A, and the time required for distortion correction in step S124A. Specific methods for determining the basic time and each coefficient include, for example, a method based on simulations or a method based on actual measurements.

[0089] The control unit 20 uses this predicted time information 400 to predict the preparation time Tp by multiplying the basic time by each coefficient. Specifically, the control unit 20 calculates the predicted time Tr required to prepare video data for each video VD by "basic time x C1 x C2 x C31 x C32." Note that the method for calculating the predicted time Tr is not limited to this, and in some cases, it may be calculated by "basic time x C1 x C2 x C31." Furthermore, if there is another parameter that affects the predicted time Tr, that parameter may be taken into consideration when calculating the predicted time Tr.

[0090] Furthermore, instead of calculating the predicted time Tr each time from the basic time and coefficients, information indicating the amount of time required for preparation for displaying a particular video, at what size, at what position, and in what design, may be stored in advance as predicted time information. In this case, the preparation time must be stored for each video VD, but the predicted time Tr does not need to be calculated each time. The control unit 20 predicts the preparation time Tp by adding up the predicted times Tr for each video VD calculated using the above method.

[0091] As a specific example, assume that in step S411 shown in Fig. 16, the display content is provisionally determined to display three videos VD1 to VD3 as shown in Fig. 17A. Video VD1 has a size coefficient C1 of 5 and is displayed in a close display position using 3D graphics with gradation. Video VD2 has a size coefficient C1 of 2 and is displayed in an intermediate display position using 2D graphics. Video VD3 has a size coefficient C1 of 10 and is displayed in a far display position using 2D graphics with gradation.

[0092] In this case, the control unit 20 calculates the predicted time Tr required to prepare the video data for video VD1 as 7.9 [msec] = 0.8 × C1 (= 5) × C2 (= 1.5) × C31 (= 1.2) × C32 (= 1.1). Similarly, the control unit 20 calculates the predicted time Tr required to prepare the video data for video VD2 as 1.6 [msec], and the predicted time Tr required to prepare the video data for video VD3 as 4.4 [msec].

[0093] Then, the control unit 20 adds up the calculated predicted times Tr for each of the videos VD1 to VD3 to predict the preparation time Tp as 13.9 [msec] = 7.9 + 1.6 + 4.4. For example, when the control period Tc is 16.6 ms, the predicted preparation time Tp is shorter than the control period Tc. Therefore, in step S415 shown in FIG. 16, the control unit 20 determines the three videos VD1 to VD3, which have the same display format, as the display content.

[0094] As another specific example, assume that the display content is provisionally determined to display two videos VD4 and VD5 in addition to the three videos VD1 to VD3 shown in Figure 17A, as shown in Figure 17B. Video VD4 has a size coefficient C1 of 8 and is displayed in a middle display position using 2D graphics. Video VD5 has a size coefficient C1 of 2 and is displayed in a nearby display position using 3D graphics.

[0095] In this case, the control unit 20 calculates the predicted time Tr required to prepare the video data for video VD4 as 6.4 [msec] = 0.8 × C1 (= 8) × C2 (= 1.0) × C31 (= 1.0) × C32 (= 1.0). Similarly, the control unit 20 calculates the predicted time Tr required to prepare the video data for video VD5 as 2.9 [msec]. Then, the control unit 20 sums the calculated predicted times Tr for each of videos VD1 to VD5 to predict the preparation time Tp as 23.2 [msec]. The predicted preparation time Tp is longer than the control cycle Tc. Therefore, the control unit 20 changes the display content, i.e., the provisionally determined display content, in step S414 shown in FIG. 16.

[0096] <How to change the displayed content> 18 is a diagram illustrating an example of a method for changing the display content (step S414) in FIG. 16. The control unit 20 changes the number of images VD or changes at least one of the display format of each image VD, i.e., the size, display position, and design type, so that the preparation time Tp becomes shorter than the control period Tc. In this case, the control unit 20 stores, in advance in the memory 17, priority information 410 that defines the priority of each item to be changed, as shown in FIG. 18, for example. The control unit 20 then changes the display content while increasing the number of items to be changed based on the priority information 410 until the preparation time Tp becomes shorter than the control period Tc.

[0097] In the priority information 410 shown in Fig. 18, the number of video VDs is set to a lower priority than the display format of each video VD. In the example shown in Fig. 18, the priority when changing the display format of each video VD is, in descending order, design type, size, display position, and distortion correction. The control unit 20 first changes the design type of each video VD based on the priority information 410. Specifically, the control unit 20 changes, for example, 3D graphics to 2D graphics and changes a display with gradation to a display without gradation.

[0098] 16, if the re-estimated preparation time Tp is shorter than the control period Tc (steps S412 and S413), the control unit 20 uses the changed video VD to determine the display content (step S415). On the other hand, if the re-estimated preparation time Tp is still longer than the control period Tc (steps S412 and S413), the control unit 20 reduces the size of each video VD in addition to the design type of each video VD. Specifically, the control unit 20, for example, predetermines a lower limit value for the size of each video VD, and sequentially reduces the size of each video VD until the lower limit value is reached.

[0099] As a specific example, in FIG. 17B, assume a case where the lower limit value of the size for the video VD3, specifically, the lower limit value of the size coefficient C1 is set to 8. In this case, the control unit 20 first changes the size of the video VD3 to a size where the size coefficient C1 becomes 9, and if Tp>Tc still holds, it changes the size to a size where the size coefficient C1 becomes 8. At this time, depending on the type of the video VD, a video VD for which the change of the size coefficient C1 is not permitted, that is, a video VD for which the lower limit value is not set, may also be set.

[0100] Subsequently, in the same manner, the control unit 20 adds the display position and the distortion correction to the item to be changed in order until Tp<Tc based on the priority information 410. When changing the display position, the control unit 20 reduces the size by moving the display position backward. At this time, depending on the type of the video VD, a video VD for which the change of the display position is not permitted may also be set. Also, when changing the item of the distortion correction, the control unit 20 applies, for example, a predetermined simplified distortion correction or does not perform the distortion correction itself.

[0101] Even if the display format for each video VD is changed in this way and Tp>Tc still holds, as a final measure, the control unit 20 reduces the number of videos VD. At this time, the control unit 20 determines in which order to reduce the videos VD based on, for example, the suppression table described in FIG. 11C. In the suppression table, as described above, the priority is set so that the vehicle information 4 that contributes greatly to safe driving, in other words, the type of the video VD, is less likely to be a deletion target.

[0102] Fig. 19 is a schematic diagram showing an example of the display content of the HUD device obtained as a result of changing the display content in Fig. 18. The upper part of Fig. 19 shows the same display content as in Fig. 11A as the display content before the change. That is, seven images VDa1 to VDa3 and VDb to VDe are displayed here. Images VDa1 and VDa2 are displayed so as to be superimposed on objects OB1 and OB2, respectively (people in this case). Image VDa3 is displayed so as to be superimposed on object OB3, a vehicle in this case. Images VDb, VDc, VDd, and VDe represent the traveling direction, navigation information, road signs, and speed information, respectively. It is assumed that 3D graphics with gradation are used for images VDa1 to VDa3.

[0103] The lower part of Fig. 19 shows the display content after the change. In the display content after the change, compared to the display content before the change, the images VDa1 to VDa3 have been replaced with images VDa1x to VDa3x, respectively. The images VDa1x to VDa3x use 2D graphics without gradation. Furthermore, the size of the images VDa1x to VDa3x is slightly smaller than that of the images VDa1 to VDa3.

[0104] 19, even when the images VDa1x to VDa3x are used, the preparation time Tp is still longer than the control period Tc, and therefore the image VDd representing the speed information is deleted, as shown at 405. That is, in this example, the image VDd representing the speed information is deleted on the assumption that it contributes less to safe driving than the other images VD. By using such a modification method, it is possible to modify the display content in a manner that does not cause significant discomfort to users, such as drivers, while maintaining the display content before the modification as much as possible from the perspective of safe driving.

[0105] FIG. 20 is a timing chart for displaying video using the flows shown in FIGS. 15 and 16. Similar to the case of FIG. 7, FIG. 20 shows the operations executed in the first to fourth processing or control cycles Tc[1] to Tc[4]. The control cycle Tc is, for example, 16.6 ms. As shown in FIG. 20, if the display content is not changed, for example, the preparation time Tp[2a] in the control cycle Tc[2] may be longer than the control cycle Tc[2]. In this case, as described in FIG. 7, dropped frames will occur.

[0106] On the other hand, if the display content is changed, for example, the preparation time Tp[2b] in the control cycle Tc[2] can be made shorter than the control cycle Tc[2]. As a result, it is possible to prevent dropped frames. The preparation time Tp also includes the time required for the display content adjustment process (step S41A) shown in FIGS. 15 and 16. Therefore, the time required for step S41A is an overhead time within the preparation time Tp, but is usually sufficiently small compared to the time required for steps S42, S123, and S124, and is considered to be at a negligible level.

[0107] Furthermore, in the method of the first embodiment described above, the actual preparation time Tp is monitored and control is performed based on the monitoring results, i.e., feedback control is performed, so that a situation like the preparation time Tp[2a] shown in Fig. 20 may occur momentarily. On the other hand, in the method of the second embodiment, the preparation time Tp is predicted and control is performed, i.e., feedforward control is performed, so that ideally, a situation like the preparation time Tp[2a] shown in Fig. 20 will not occur.

[0108] <Major Effects of the Second Embodiment> As described above, the method of the second embodiment can also achieve the same effects as those described in the first embodiment. That is, it is possible to suppress or prevent the occurrence of dropped frames and ensure a minimum display quality. Furthermore, it is possible to satisfy the required display specifications as much as possible within the range of the processing performance of the hardware and to flexibly respond to the display specifications. Furthermore, by predicting the preparation time Tp, it is less likely that the display content will be excessively changed or suppressed compared to the method of the first embodiment, making it possible to further improve the display quality.

[0109] The method of the first embodiment is particularly useful when the display specifications have many variables, such as in the case of an initial product, and it is difficult to generate predicted time information 400 as shown in FIGS. 17A and 17B. On the other hand, the method of the second embodiment is particularly useful when the display specifications have few variables, such as in the case of a relatively mature product. Furthermore, since the method of the second embodiment requires the generation of predicted time information 400, the method of the first embodiment may be more useful in terms of flexibility or versatility in dealing with display specifications that vary greatly.

[0110] (Embodiment 3) <Details of the control unit> FIG. 21A is a schematic diagram showing an example of main processing contents of a control unit in a HUD device according to embodiment 3, and FIG. 21B is a supplementary diagram of FIG. 21A. FIG. 22A is a schematic diagram showing an example different from FIG. 21A, and FIG. 22B is a supplementary diagram of FIG. 22A. HUD device 1 according to embodiment 3 is realized by the configuration shown in FIG. 3A or 3B described above. Control unit 20 superimposes a warning image on object OB3, here a vehicle, as shown in image VDa3a in FIG. 21A and image VDa3b in FIG. 22A.

[0111] At this time, the control unit 20 determines the color or shape of the warning images VDa3a and VDa3b according to the distance to the object OB3. In the example of FIGS. 21A and 21B, the distance between the host vehicle 420 equipped with the HUD device 1 and the object OB3 present in front of the host vehicle 420 is 50 m. In this case, the control unit 20 determines the color of the image VDa3a to be green, for example. On the other hand, in the example of FIGS. 22A and 22B, the distance between the host vehicle 420 and the object OB3 is 10 m. In this case, the control unit 20 determines the color of the image VDa3b to be red, for example.

[0112] FIG. 23 is a diagram showing an example of the configuration of a display setting table stored in the control unit of the HUD device according to the third embodiment. The control unit 20 stores, for example, a display setting table 415 as shown in FIG. 23 in advance in the memory 17 or the like. The display setting table 415 defines the correspondence between the distance to the forward object OB3 and the color or shape of the warning image, in this case, color. In this example, the color of the warning image is set to red when the distance to the object OB3 is less than 12 m, green when the distance is 32 m or more, and yellow when the distance is in the range of 17 m to 27 m. Furthermore, the color for the intermediate distance ranges of 12 m to 17 m and 27 m to 32 m is appropriately set to an intermediate color.

[0113] The display setting table 415 is set, for example, so that the closer the distance to the object OB3, the more the color or shape attracts the attention of the user, for example, the driver. This can contribute to safe driving. However, how the color or shape is perceived depends on the user's subjective opinion. For this reason, the display setting table 415 may be configured so that the user can arbitrarily select the color or shape from a plurality of options through initial setting. Furthermore, the display setting table 415 may be configured so that the distance range can be arbitrarily set.

[0114] Fig. 24A is a flow diagram showing an example of processing details of a control unit in the HUD device according to Embodiment 3. In Fig. 24A, the control unit 20 executes the processing of steps S411 to S415 described in Fig. 16 in Embodiment 2. Prior to this processing, the communication unit 16, i.e., the information acquisition unit, acquires the distance between the host vehicle 420 and an object OB3 present in front of the host vehicle 420 as one piece of information related to the vehicle in step S11 shown in Fig. 15.

[0115] 24A, the control unit 20 determines the display content so that the preparation time Tp is shorter than the control cycle Tc, in other words, the processing cycle. Thereafter, the control unit 20 determines whether the display content determined in step S415 includes a warning video to be superimposed on the object OB3 (step S416A). If the determined display content includes a warning video (step S416A: Yes), the control unit 20 refers to the distance information from the object OB3 acquired by the communication unit 16 (step S417) and updates the color or shape of the warning video based on the display setting table 415 (step S418A). Note that if multiple warning videos are included, the update process is performed for each warning video.

[0116] 24A, when the determined display content includes a warning image, the control unit 20 updates the color or shape of the warning image (step S418A) without re-estimating the preparation time Tp, i.e., without going through the process of step S412, etc. For this reason, it is desirable that the color or shape of the warning image for each distance be determined in advance so that the preparation time Tp remains the same as before the update, even when the color or shape of the warning image is updated.

[0117] Fig. 24B is a flow diagram showing an example of processing content different from that of Fig. 24A. Unlike the flow shown in Fig. 24A, the flow shown in Fig. 24B incorporates processing similar to steps S416A, S417, and S418A described in Fig. 24A into the processing of steps S411 to S415, rather than after the processing of steps S411 to S415 described in Fig. 16.

[0118] 24B, the control unit 20 tentatively determines the display content (step S411), and then determines whether the tentatively determined display content includes a warning image to be superimposed on the object OB3 (step S416B). If the tentatively determined display content includes a warning image (step S416B: Yes), the control unit 20 refers to the distance information from the object OB3 (step S417) and determines the color or shape of the warning image based on the display setting table 415 (step S418B).

[0119] Thereafter, the control unit 20 predicts the preparation time Tp required to prepare video data based on the provisionally determined display content (step S412). Note that if the provisionally determined display content does not include a warning video (step S416B: No), the control unit 20 proceeds directly to step S412. After step S412, the control unit 20 changes the provisionally determined display content by the processes of steps S413 to S415, as in the case of FIG. 16, so that the preparation time Tp is shorter than the control cycle Tc.

[0120] 16, the control unit 20 changes the provisionally determined display content and then proceeds to step S416B (step S414). As a result, even if the color or shape of the warning image is changed by the processing of step S414, it can be restored correctly by the processing of steps S417 and S418B, and the preparation time Tp can be predicted (step S412).

[0121] Using the flow shown in Fig. 24B allows the preparation time Tp to be predicted while reflecting changes in color or shape in the warning video, which may improve prediction accuracy compared to using the flow shown in Fig. 24A. However, using the flow shown in Fig. 24B may increase processing overhead because steps S416B, S417, and S418B must be processed each time a loop from step S414 to step S416B occurs. Therefore, from this perspective, using the flow shown in Fig. 24A is beneficial.

[0122] 24A based on display setting table 415 as described in steps S416A, S417, and S418A may be applied not only to the method of embodiment 2 but also to the method of embodiment 1. In this case, control unit 20 may execute the processing as described in steps S416A, S417, and S418A at the final stage when determining the display content based on the internal state in step S121 shown in FIG.

[0123] <Major Effects of the Third Embodiment> As described above, by using the method of embodiment 3, in addition to the various effects described in embodiments 1 and 2, it is possible to contribute to safer driving by changing the color or shape of the warning image depending on the distance from the object.

[0124] Furthermore, by using the methods of each embodiment, the user 6 can view various information necessary for driving, such as navigation information such as destination and speed, as well as alert information when an oncoming vehicle or pedestrian is detected, as an image through the windshield 3. Even if the display specifications change, the image can be viewed with a minimum display quality ensured. This makes it possible to provide a HUD device 1 that reduces the need for the user 6 to move their viewpoint and contributes to supporting safe driving. As a result, it becomes possible to prevent traffic accidents. Furthermore, it becomes possible to contribute to "Good health and well-being for all" of the Sustainable Development Goals (SDGs) advocated by the United Nations.

[0125] The invention made by the inventor has been specifically described above based on the embodiments, but the present invention is not limited to the above embodiments and can be modified in various ways without departing from the spirit of the invention. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. [Explanation of symbols]

[0126] 1...Head-up display (HUD) device, 2...Vehicle, 4...Vehicle information, 5...Display area, 6...User, 11...Video display unit, 16...Communication unit (information acquisition unit), 20...Control unit, 400...Predicted time information, 410...Priority information, 415...Display setting table, C1 to C3, C31, C32...Coefficients, M1...Mirror (video light projection unit), ST0...Normal state, ST1...Return transition state, ST2...Suppression transition state, ST3...Suppression state, Tc...Control period, Tp...Preparation time, Tr...Predicted time, Tth1...First threshold time, Tth2...Second threshold time, TthD...Threshold duration, VD...Video

Claims

1. A head-up display device mounted on a vehicle, an information acquisition unit that acquires information about the vehicle; an image display unit that displays an image and emits image light of the displayed image; an image light projection unit that projects the image light emitted from the image display unit onto a display area, thereby allowing the projected image light to be viewed as a virtual image; a control unit that determines display content based on information about the vehicle acquired by the information acquisition unit, prepares video data based on the determined display content, and displays video based on the prepared video data on the video display unit; and Equipped with the control unit, before determining the display content, predicts a preparation time required to prepare the video data based on the provisionally determined display content based on predicted time information that defines a relationship between the display content and a predicted time required to prepare the video data, and, if the preparation time required to prepare the video data of the provisionally determined display content before determining the display content is longer than a predetermined processing cycle, changes the provisionally determined display content so that the preparation time is shorter than the processing cycle. Head-up display device.

2. The head-up display device according to claim 1, The items for changing the provisionally determined display content include the number of images or the display format for each image, The items for changing the display format for each image include at least one of size, display position, and design type. Head-up display device.

3. The head-up display device according to claim 2, the control unit pre-stores priority information that defines a priority for each of the items, and changes the provisionally determined display content while increasing the number of items to be changed based on the priority information until the preparation time becomes shorter than the processing cycle. Head-up display device.

4. The head-up display device according to claim 3, the items when changing the provisionally determined display content include the number of images and a display format for each image; In the priority information, the number of images is set to a lower priority than the display format of each image. Head-up display device.

5. The head-up display device according to claim 2, the predicted time information includes a basic time and a coefficient according to the item included in the display format of each of the videos; the control unit predicts the preparation time by multiplying the basic time by the coefficient. Head-up display device.

6. The head-up display device according to claim 1, the information acquisition unit acquires a distance between the vehicle equipped with the head-up display device and an object present in front of the vehicle; When the control unit superimposes a warning image on the object, the control unit determines a color or a shape of the warning image depending on a distance from the object. Head-up display device.

7. The head-up display device according to claim 6, the control unit determines the display content so that the preparation time is shorter than the processing cycle, and then, if the determined display content includes the warning video, updates the color or shape of the warning video without re-estimating the preparation time. Head-up display device.

8. A method for processing video data for a head-up display device mounted on a vehicle, comprising: obtaining information about the vehicle; determining display content based on the acquired information about the vehicle, preparing video data based on the determined display content, and displaying a video based on the prepared video data on a video display unit of the head-up display device; predicting a preparation time required for preparing the video data based on the provisionally determined display content based on predicted time information that defines a relationship between the display content and a predicted time required for preparing the video data, before determining the display content; and, if the preparation time required for preparing the video data of the provisionally determined display content before determining the display content is longer than a predetermined processing cycle, changing the provisionally determined display content so that the preparation time is shorter than the processing cycle. How video data is processed.

9. 9. The video data processing method according to claim 8, The items for changing the provisionally determined display content include the number of images or the display format for each image, The items for changing the display format for each image include at least one of size, display position, and design type. How video data is processed.

10. 10. The video data processing method according to claim 9, storing in advance priority information that defines a priority for each of the items, and changing the provisionally determined display content while increasing the number of items to be changed based on the priority information until the preparation time becomes shorter than the processing cycle; How video data is processed.

11. 11. The video data processing method according to claim 10, the items when changing the provisionally determined display content include the number of images and a display format for each image; In the priority information, the number of images is set to a lower priority than the display format of each image. How video data is processed.

12. 10. The video data processing method according to claim 9, the predicted time information includes a basic time and a coefficient according to the item included in the display format of each of the videos; predicting the preparation time by multiplying the basic time by the coefficient; How video data is processed.

13. 9. The video data processing method according to claim 8, acquire a distance between the vehicle equipped with the head-up display device and an object present in front of the vehicle; When a warning image is superimposed on the object, a color or a shape of the warning image is determined according to a distance from the object. How video data is processed.

14. 14. The video data processing method according to claim 13, after determining the display contents so that the preparation time is shorter than the processing cycle, if the warning image is included in the determined display contents, updating the color or shape of the warning image without re-estimating the preparation time; How video data is processed.

Images