Head-up display device and video data processing method

Abstract

To provide a head-up display device and a processing method of video data that are capable of flexibly dealing with various display specifications, and to contribute to the sustainable development goal "3. Good health and well-being."SOLUTION: A head-up display device comprises: a video display unit 11 that displays a video and emits video light of the displayed video; a video light projection unit that projects the emitted video light to a display region to allow the projected video light to be recognized as a virtual image; and a control unit 20 that prepares video data based on acquired information on a vehicle and causes the video display unit 11 to display the video based on the prepared video data. The control unit 20 performs control for reducing a processing load required to prepare the video data so that the preparation of the video data is completed within a predetermined processing period.SELECTED DRAWING: Figure 3A

Description

[Technical Field]

[0001] The present invention relates to a head-up display device and a method for processing video data, and relates to technology for a head-up display device that uses AR (Augmented Reality), for example. [Background technology]

[0002] Patent Document 1 discloses a display system that extracts an object to be displayed in AR based on imaging data, generates AR image data of the object, and sets the frame rate of the AR image data based on the importance of the object. For example, when the overall frame rate is 60 fps and three AR image data are displayed in a time-division manner, the frame rates of the three AR image data are set to 30 fps, 20 fps, and 10 fps, respectively, according to their importance. This allows for more suppressed flickering of AR image data with higher importance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-6164 Summary of the Invention [Problem to be solved by the invention]

[0004] A head-up display device prepares various image data based on information acquired from ADAS (Advanced Driver Assistance Systems) and projects image light based on the image data onto a display area to be viewed as a virtual image. The preparation time required to prepare the image data varies depending on the type and number of images, the size of the image according to the virtual image distance, the display format (2D / 3D, etc.), the presence or absence of graphics effects, etc. In this specification, a head-up display device is also referred to as a HUD device.

[0005] In a typical HUD device, processing performance such as a maximum frame rate is fixedly determined based primarily on the hardware specifications. However, there may be a desire to change the display specifications of the HUD device by updating software without changing the hardware itself. In such cases, the required display specifications of the HUD device often involve an increased processing load. As a result, there is a risk that the hardware processing performance may be insufficient, resulting in failure to meet the required display specifications, such as frame dropping during AR display.

[0006] The present invention has been made in view of the above, and one of its objects is to provide a head-up display device and a video data processing method that can satisfy required display specifications as much as possible within the range of hardware processing performance and can flexibly respond to display specifications.

[0007] The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings. [Means for solving the problem]

[0008] Among the inventions disclosed in this application, the outline of representative inventions will be briefly explained as follows.

[0009] A typical head-up display device includes an image display unit that displays an image and emits image light of the displayed image, an image light projection unit that projects the emitted image light onto a display area so that the projected image light is visually recognized as a virtual image, and a control unit that prepares image data based on acquired information about the vehicle and displays an image based on the prepared image data on the image display unit. The control unit performs control to reduce the processing load required for preparing the image data so that preparation of the image data is completed within a predetermined processing cycle. [Effects of the Invention]

[0010] To briefly explain the effect obtained by a representative invention among the inventions disclosed in the present application, it becomes possible to flexibly accommodate display specifications in a head-up display device. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic diagram showing an example of the configuration of a vehicle equipped with a head-up display device according to an embodiment. [Figure 2A] 2 is a schematic diagram showing an example of the configuration of a main part of the HUD device in FIG. 1. [Figure 2B] 2B is a schematic diagram showing an example of the configuration of the main part of the HUD device in FIG. 1, which is different from that in FIG. 2A. [Figure 3A] 2C is a block diagram showing an example of the configuration of a main part of a control system that is responsible for control in the HUD device shown in FIGS. 2A and 2B. FIG. [Figure 3B] 3A is a block diagram showing an example of the configuration of a main part of a control system that is responsible for control in the HUD device shown in FIGS. 2A and 2B, the main part being different from that shown in FIG. 3A. FIG. [Figure 4] 3A and 3B are block diagrams showing an example of the configuration of a portion related to a control unit. [Figure 5] 2 is a schematic diagram showing an example of the display content of the HUD device shown in FIG. 1. FIG. [Figure 6] 3B is a flowchart showing an example of a processing procedure for displaying an image in the HUD device shown in FIG. 3A. FIG. [Figure 7] 7 is a timing chart schematically illustrating an example of a problem that occurs when displaying video using the flow shown in FIG. 6. [Figure 8A] FIG. 10 is a schematic diagram showing how the display content changes when no dropped frames occur. [Figure 8B] 10A and 10B are schematic diagrams showing how display content changes when a frame is dropped. [Figure 9] 3B is a diagram showing an example of an internal state of a control unit shown in FIG. 3A. FIG. [Figure 10]It is a diagram showing an example of the monitoring result of the preparation time by the control unit shown in FIG. 3A. [Figure 11A] It is a schematic diagram showing an example of the display content of the HUD device in the normal state shown in FIG. 9. [Figure 11B] It is a schematic diagram showing an example of the display content of the HUD device in the suppression transition state shown in FIG. 9. [Figure 11C] It is a schematic diagram showing an example of the display content of the HUD device in the suppression state shown in FIG. 9. [Figure 12] It is a flowchart showing an example of the processing content of the control unit shown in FIG. 3A.

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described in detail based on the drawings. In all the drawings for explaining the embodiments, the same members are basically denoted by the same reference numerals, and the repeated explanations thereof are omitted.

[0013] <Outline of HUD Device> FIG. 1 is a schematic diagram showing a configuration example of a vehicle equipped with a head-up display device according to an embodiment. The head-up display (HUD) device 1 shown in FIG. 1 is mounted on a vehicle 2 which is one of the vehicles. The vehicle 2 is typically an automobile, but is not necessarily limited thereto, and may be a railway vehicle or the like. Further, the vehicle is not limited to a vehicle, and may be an aircraft or the like. Further, the vehicle 2 is equipped with a control unit 21 called an ECU (Electronic Control Unit), for example.

[0014] The control unit 21 acquires vehicle information 4 from various sensors installed in each part of the vehicle 2 and, in addition, a navigation device or the like. The various sensors detect various events occurring in the vehicle 2 and also detect various parameter values regarding the running state. The HUD device 1 acquires the vehicle information 4 acquired by the control unit 21 using, for example, CAN (Controller Area Network) communication or the like.

[0015] The vehicle information 4 includes, for example, speed information and gear information of the vehicle 2, steering wheel steering angle information, lamp illumination information, external light information, distance information, infrared information, engine ON / OFF information, camera image information inside and outside the vehicle, acceleration gyro information, GPS (Global Positioning System) information, navigation information, vehicle-to-vehicle communication information, and road-to-vehicle communication information. The GPS information also includes information such as the current time. The vehicle information 4 also includes various warning information. The HUD device 1 projects video light onto a display area such as the windshield 3 based on such vehicle information 4. As a result, the HUD device 1 allows a user such as a driver to visually recognize the video light projected onto the display area as a virtual image, specifically, as a virtual image superimposed on the scenery ahead of the vehicle 2.

[0016] Fig. 2A is a schematic diagram showing an example configuration of the main parts of the HUD device in Fig. 1. The HUD device 1 shown in Fig. 2A includes, for example, an image display unit 11 housed in a housing 12, mirrors M1 and M2, and a mirror driver 14. The image display unit 11 is, for example, a display panel such as an LCD (Liquid Crystal Display) or a projector, and displays an image based on input image data and emits image light of the displayed image.

[0017] Mirror M2 reflects the image light from image display unit 11 toward mirror M1. Mirror M2 is effective in saving space and ensuring a long optical path length. Depending on the space inside the HUD housing and the required optical path length, mirror M2 may not be provided, or multiple mirrors M2 may be provided. Mirror M1 functions as an image light projection unit. Mirror M1, which is the image light projection unit, projects the image light emitted from image display unit 11 and reflected by mirror M2 onto display area 5 of windshield 3 through opening 7 provided in dashboard 10. As a result, the image light projection unit allows user 6 to view the projected image light as a virtual image.

[0018] In detail, mirror M1 is, for example, a concave mirror (magnifying mirror), which reflects and magnifies the image light reflected by mirror M2 and projects it onto display area 5 through opening 7. The image light projected onto display area 5 is reflected by display area 5 and enters the eyes of user 6. As a result, user 6 visually recognizes the image light projected onto display area 5 as a virtual image 9 existing beyond transparent windshield 3, superimposed on the scenery outside the vehicle (roads, buildings, people, etc.). The information represented by virtual image 9 includes various information, such as road signs, the vehicle's current speed, and various information added to objects in the scenery, i.e., AR information.

[0019] Furthermore, mirrors M1 and M2 may be, for example, free-form surface mirrors or mirrors having an asymmetric shape with respect to the optical axis. Here, the installation angle of mirror M2 is fixed. On the other hand, mirror M1 is provided with a mirror driver 14. The mirror driver 14 variably adjusts the installation angle of mirror M1. In detail, mirror driver 14 includes, for example, a motor, and rotates mirror M1 by the rotational operation of the motor.

[0020] By adjusting the installation angle of the mirror M1, it is possible to adjust the position of the display area 5 on the windshield 3, i.e., the vertical position of the virtual image viewed by the user 6. Furthermore, by adjusting the installation angle of the mirror M1, it is possible to protect the image display unit 11 from sunlight. Specifically, sunlight can travel in the opposite direction along the optical path of the image light and enter the image display unit 11. If the incidence of sunlight increases the possibility of damaging the image display unit 11, it is sufficient to change the installation angle of the mirror M1 so that sunlight does not reach the image display unit 11.

[0021] FIG. 2B is a schematic diagram showing an example of the configuration of the main parts of the HUD device in FIG. 1, which is different from that shown in FIG. 2A. The HUD device 1 shown in FIG. 2B differs from the configuration shown in FIG. 2A in that a lens LS is provided in place of the mirror M2 within the housing 12. Image light from the image display unit 11 is incident on the mirror M1 via the lens LS. As in the case of FIG. 2A, the mirror M1 projects the incident image light onto the display area 5 through the opening 7. As in the case of FIG. 2A, a mirror driver may be provided on the mirror M1. The configuration shown in FIG. 2B can be applied to cases where the windshield 3 is installed at an angle close to vertical, such as in a minivan or truck.

[0022] Fig. 3A is a block diagram showing an example of the configuration of the main parts of the control system that are responsible for control in the HUD device shown in Fig. 2A and Fig. 2B. HUD device 1 shown in Fig. 3A includes mirror driver 14, display driver 15, communication unit 16, memory 17, frame buffer 18, and control unit 20, which are connected to each other via bus 13.

[0023] The communication unit 16 receives and transmits vehicle information and is realized, for example, by a communication interface circuit or the like, and functions as an information acquisition unit. The communication unit 16 acquires or receives information about the vehicle from the control unit 21 using CAN communication or the like, and transmits the received information about the vehicle to the control unit 20. The control unit 20 controls the mirror drive unit 14 and the display drive unit 15 based on the information from the communication unit 16. The mirror drive unit 14 adjusts the installation angle of the mirror M1, as described in FIG. 2A, in response to, for example, a command from the control unit 20. The mirror drive unit 14 can be realized by a motor such as that described in FIG. 2A, as well as a motor driver circuit or the like that drives the motor.

[0024] Frame buffer 18 is configured, for example, by a volatile memory and stores video data. Display driver 15 reads the video data stored in frame buffer 18 via bus 13 and drives video display unit 11 based on the video data. Video display unit 11 is, for example, a liquid crystal display equipped with a light source and a display panel. The display panel displays video by modulating backlight emitted from the light source for each pixel based on the video data. In this case, display driver 15 can be realized by an LCD driver circuit or the like.

[0025] The memory 17 is configured, for example, by a combination of volatile memory and non-volatile memory, and stores programs, data, etc. used by the control unit 20. The control unit 20 is realized by a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and controls the entire HUD device 1 by executing the programs stored in the memory 17. As one of its functions, the control unit 20 prepares video data, including the creation of video data, based on information about the vehicle acquired by the communication unit 16, i.e., the information acquisition unit, and causes the video display unit 11 to display an image based on the prepared video data.

[0026] 3A may be mounted on a microcontroller or the like. However, the implementation is not limited to this, and may be, for example, an implementation in which an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or the like is appropriately combined.

[0027] Fig. 3B is a block diagram showing an example configuration of a main part of the control system that is responsible for control in the HUD device shown in Fig. 2A and Fig. 2B, which is different from Fig. 3A. Unlike the example configuration shown in Fig. 3A, the HUD device 1 shown in Fig. 3B is configured without the memory 17 and the control unit 20. In this case, the control unit 21 creates video data in place of the control unit 20 shown in Fig. 3A, and writes the created video data to the frame buffer 18 via the communication processing unit 16a.

[0028] In the configuration example shown in Fig. 3B, the control unit 20 described in Fig. 3A may function as the communication processing unit 16a. Alternatively, some functions may be shared between the control unit 21 and the communication processing unit 16a. The configuration example of Fig. 3B is provided with a mirror driving unit 14, a display driving unit 15, and a communication processing unit 16a. Unlike the communication unit 16 of Fig. 3A, the communication processing unit 16a receives information about the vehicle from the control unit 21 using CAN communication or the like, processes the received information, and adjusts the operations of the mirror driving unit 14 and the display driving unit 15 based on the processing results.

[0029] Fig. 4 is a block diagram showing an example of the configuration of the parts related to the control unit in Fig. 3A and Fig. 3B. The control unit 21 acquires vehicle information 4 as described in Fig. 1. As shown in Fig. 4, the vehicle information 4 is generated by information acquisition devices such as various sensors connected to the control unit 21. Fig. 4 shows an example of the information acquisition device.

[0030] In FIG. 4, for example, a vehicle speed sensor 101 detects the speed of the vehicle 2 in FIG. 1 and generates speed information as the detection result. A shift position sensor 102 detects the current gear and generates gear information as the detection result. A steering wheel angle sensor 103 detects the current steering wheel angle and generates steering wheel angle information as the detection result. A headlight sensor 104 detects whether the headlights are on or off and generates lamp illumination information as the detection result. An illuminance sensor 105 and a chromaticity sensor 106 detect external light and generate external light information as the detection result.

[0031] The distance measurement sensor 107 detects the distance between the vehicle 2 and an external object, and generates distance information that represents the detection result. The infrared sensor 108 detects the presence or absence of an object in the vicinity of the vehicle 2, as well as the distance thereto, and generates infrared information that represents the detection result. The engine start sensor 109 detects whether the engine is on or off, and generates on or off information that represents the detection result. The acceleration sensor 110 and the gyro sensor 111 detect the acceleration and angular velocity of the vehicle 2, respectively, and generate acceleration gyro information that represents the attitude and behavior of the vehicle 2 as the detection result. The temperature sensor 112 detects the temperature inside and outside the vehicle, and generates temperature information that represents the detection result.

[0032] The road-to-vehicle communication wireless receiver 113 generates road-to-vehicle communication information through road-to-vehicle communication between the vehicle 2 and roads, signs, traffic lights, etc. The vehicle-to-vehicle communication wireless receiver 114 generates vehicle-to-vehicle communication information through vehicle-to-vehicle communication between the vehicle 2 and other surrounding vehicles. The interior camera 115 and the exterior camera 116 capture images of the interior and exterior of the vehicle, respectively, to generate interior camera image information and exterior camera image information. The interior camera 115 is, for example, a camera for a DMS (Driver Monitoring System) that captures the posture, eye position, movement, etc. of the user 6 shown in FIG. 2A, etc. In this case, the fatigue level and gaze position of the user 6 can be determined by analyzing the captured images.

[0033] On the other hand, the exterior vehicle camera 116 captures images of the surroundings, such as the areas in front of and behind the vehicle 2. In this case, by analyzing the captured images, it becomes possible to determine the presence or absence of obstacles such as other vehicles or people in the vicinity, buildings, topography, road conditions such as rain, snow, ice, unevenness, and road signs. The exterior vehicle camera 116 may also include, for example, a drive recorder that records images of the situation while driving.

[0034] The GPS receiver 117 generates GPS information obtained by receiving GPS signals. For example, the GPS receiver 117 can acquire the current time. The VICS (Vehicle Information and Communication System, registered trademark) receiver 118 generates VICS information obtained by receiving VICS signals. The GPS receiver 117 and the VICS receiver 118 may be provided as part of the navigation device. Regarding the various information acquisition devices shown in FIG. 4, it is possible to appropriately delete them, add other types of devices, or replace them with other types of devices.

[0035] <Regarding the display of the HUD device> FIG. 5 is a schematic diagram showing an example of the display content of the HUD device shown in FIG. 1. The display content shows an example of AR display and shows an example of the virtual image 9 shown in FIG. 2A etc. In the example shown in FIG. 5, five videos VDa to VDe are displayed. In the specification, the plurality of videos VDa to VDe are collectively referred to as video VD.

[0036] Video VDa is one of the landscapes and is displayed so as to overlap an object OB, which is an object of AR, here a person. Video VDa means that the object OB has been detected by the various information acquisition devices shown in FIG. 4. That is, the HUD device 1 acquires information representing the detection result of the object OB from the control unit 21. Also, video VDa represents warning information for prompting the user 6, for example, the driver, to pay attention to the object OB.

[0037] Video VDb is displayed on a road, which is one of the landscapes, and represents the traveling direction of the vehicle 2. Video VDc represents navigation information. Video VDd represents, for example, a road sign, which is one of the vehicle-to-vehicle communication information. Video VDe represents the speed information of the vehicle 2. Note that videos VDa and VDb are 3D graphics, and videos VDc to VDe are 2D graphics.

[0038] Fig. 6 is a flow diagram showing an example of a processing procedure when displaying an image in the HUD device shown in Fig. 3A. In Fig. 6, first, the communication unit 16, i.e., the information acquisition unit, acquires information about the vehicle from the control unit 21 (step S11). Next, the control unit 20 performs a preparation process for the image data including steps S121 to S124 (step S12). For example, the processor executes the image processing program in the memory 17 to perform the processes of steps S121 to S123.

[0039] In step S121, the control unit 20 determines the display content based on the information about the vehicle acquired in step S11, for example, the vehicle information 4 shown in FIG. 1. Specifically, the control unit 20 selects the vehicle information 4 that is compatible with the display, and determines the position, size, and layout in which the video VD representing the selected vehicle information 4 should be displayed. In the example shown in FIG. 5, the control unit 20 determines that five videos VDa to VDe should be displayed as the display content. In step S122, the control unit 20 creates video data for each video VD based on the display content determined in step S121.

[0040] In step S123, control unit 20 writes each of the multiple pieces of video data created in step S122 to a storage area in frame buffer 18 corresponding to the position where the video data should be displayed. In step S124, control unit 20 performs distortion correction on all of the video data in frame buffer 18, for example, in accordance with the curvature of windshield 3. Note that distortion correction may be achieved, for example, by a processor executing a distortion correction program in memory 17, or may be achieved by dedicated hardware.

[0041] After completing this video data preparation process (step S12), video display processing is performed (step S13). In step S13, display drive unit 15 reads out the video data stored in frame buffer 18 and drives video display unit 11 based on the video data, thereby causing video display unit 11 to display the video VD. The processing procedure shown in FIG. 6 is executed in synchronization with a processing cycle determined based on the frame rate. For example, when the frame rate is 60 fps, the processing cycle is 16.6 ms. The processing procedure for displaying the video VD is not limited to that shown in FIG. 6, and can be replaced with various commonly known procedures.

[0042] FIG. 7 is a timing chart schematically illustrating an example of a problem that occurs when displaying video using the flow shown in FIG. 6. FIG. 7 shows operations executed in the first to fifth processing or control cycles Tc[1] to Tc[5]. In the specification, the processing or control cycles Tc[1] to Tc[5] are collectively referred to as the processing cycle or control cycle Tc. The control cycle Tc is, for example, 16.6 ms. FIG. 7 also shows operations when the processes of steps S11 and S12 and the process of step S13 in FIG. 6 are executed in a pipeline.

[0043] 7, in the first control period Tc[1], information about the vehicle is acquired (step S11), and video data is prepared based on the acquired information (step S12). The video data preparation time Tp[1] required for the processing of steps S11 and S12 is shorter than the control period Tc[1]. This is the case, for example, when the number of AR objects, and therefore the number of videos VD to be displayed, is small, and the preparation time Tp[1] fits within the control period Tc[1]. Similarly, in the second control period Tc[2], information about the vehicle is acquired (step S11), and video data is prepared based on the acquired information (step S12). In parallel with this, in the second control period Tc[2], display processing of the video VD is performed based on the video data prepared in the first control period Tc[1] (step S13).

[0044] Here, in the second control period Tc[2], the preparation time Tp[2] of the video data required for the processing of steps S11 and S12 is longer than the control period Tc[2]. The preparation time Tp[2] can be lengthened, for example, due to an increase in the number of AR objects and thus the video VD to be displayed. As a result, the video data is not reflected in the display processing in the third control period Tc[3], but is reflected in the display processing in the fourth control period Tc[4]. As a result, dropped frames occur in the third control period Tc[3].

[0045] Furthermore, in the fourth control period Tc[4], the preparation time Tp[3] of the video data required for the processing of steps S11 and S12 is shorter than the control period Tc[4]. Therefore, the video data is reflected in the display processing in the fifth control period Tc[5]. However, depending on the method, due to the occurrence of frame dropping, the video data prepared in the preparation period Tp[3] may be the video data that should have been prepared in the third control period Tc[3]. In this specification, the multiple preparation periods Tp[1] to Tp[3] are collectively referred to as the preparation time Tp.

[0046] FIG. 8A is a schematic diagram showing how the display content changes when no dropped frames occur. FIG. 8B is a schematic diagram showing how the display content changes when dropped frames occur. For example, as shown in FIGS. 8A and 8B, assume that an object OB, in this case a person, moves from time t1 to time t2. When no dropped frames occur, as shown in FIG. 8A, an image VDa(t1) is displayed superimposed on the position of the object OB(t1) at time t1, and an image VDa(t2) is displayed superimposed on the position of the object OB(t2) at time t2.

[0047] In this way, when no frame dropping occurs, it is possible to display a video VD with high real-time performance that tracks the AR object. On the other hand, when frame dropping occurs, particularly when frame dropping occurs continuously in consecutive control cycles Tc, as shown in FIG. 8B, the video VDa(t2) is displayed at a position shifted toward the object OB(t1) from the position of the object OB(t2) at time t2. In this case, the video VDa(t2) is not completely superimposed on the object OB(t2). In this way, when frame dropping occurs, it is possible that a video VD with low real-time performance that does not track the AR object will be displayed.

[0048] <Outline of operation of the control unit> Therefore, in order to display the video VD with high real-time performance, the control unit 20 generally performs control to reduce the processing load required for preparing the video data so that the preparation of the video data is completed within a predetermined processing cycle or control period Tc. That is, the control unit 20 prepares the video data based on the information about the vehicle received by the communication unit 16, and if the preparation conditions for the video data do not satisfy the predetermined conditions within the predetermined processing cycle, the control unit 20 modifies some of the content of the video data to be prepared. More specifically, the control unit 20 monitors the preparation time Tp required to prepare the video data, and when the preparation time Tp satisfies a predetermined condition, it initiates control to reduce the processing load. The predetermined processing cycle or control period Tc is determined based on the frame rate.

[0049] Assuming an operation method such as that described in Fig. 7, the preparation time Tp is the time required for the processing of steps S11 and S12 shown in Fig. 6. On the other hand, assum- ing an operation method in which the processing of step S11 and the processing of step S12 are executed in a pipeline, for example, the preparation time Tp may be the time required for the processing of step S12. However, in order to improve real-time performance, it is desirable to use the operation method described in Fig. 7.

[0050] The control unit 20 continuously monitors the preparation time Tp. For example, the preparation time can be monitored in conjunction with the vehicle time or by using a timer or the like. Assuming the operation method described in FIG. 7, as shown in FIG. 3A, the control unit 20 monitors the preparation time Tp, which is required from the start of acquiring information about the vehicle using the communication unit 16 until the completion of writing the created video data to the frame buffer 18, by using a timer or the like. When the configuration shown in FIG. 3B is used, unlike the case of FIG. 3A, the communication processing unit 16a acquires information from the control unit 21, processes the acquired information, and monitors the preparation time Tp, which is required from the start of acquiring information about the vehicle using the communication unit 16 until the completion of writing the processed results to the frame buffer 18, by using a timer or the like.

[0051] FIG. 9 is a diagram showing an example of the internal state of the control unit shown in FIG. 3A. FIG. 10 is a diagram showing an example of the monitoring result of the preparation time by the control unit shown in FIG. 3A. As shown in FIG. 9, the control unit 20 has, as its internal states, a normal state ST0, a return transition state ST1, a suppression transition state ST2, and a suppression state ST3. In other words, the control unit 20 has, as its operating modes, a normal mode ST0, a return transition mode ST1, a suppression transition mode ST2, and a suppression mode ST3. Having four states is just one example, and in another example, the control unit 20 may have two states: the normal state ST0 and the suppression state ST3.

[0052] In the normal state ST0, the control unit 20 prepares video data as usual based on information acquired using the communication unit 16, and displays the video VD based on the video data on the video display unit 11. In addition, in the normal state ST0, the control unit 20 transitions to the suppression transition state ST2 if (A) the preparation time Tp, which is the monitoring result, is longer than the control period Tc, or (B) the preparation time Tp is longer than the first threshold time Tth1 multiple times in a row. The number of consecutive times is set to a value between 2 and 10, for example.

[0053] 10, the preparation time Tp4 obtained at the monitoring time tm4 is longer than the control period Tc. Furthermore, the preparation times Tp2 and Tp3 obtained at the consecutive monitoring times tm2 and tm3 are both shorter than the control period Tc but longer than the first threshold period Tth1. The first threshold period Tth1 is shorter than the control period Tc of 16.6 ms, such as 15.0 ms.

[0054] In the normal state ST0, the control unit 20 transitions to the suppression transition state ST2 when (A) a monitoring result such as preparation time Tp4 is obtained, or (B) a monitoring result such as preparation times Tp2 and Tp3 is obtained multiple times in succession, for example, twice in succession. Condition (A) is intended to quickly resolve dropped frames. On the other hand, condition (B) is intended to prevent a situation in the near future in which the preparation time Tp becomes longer than the control cycle Tc, thereby causing dropped frames.

[0055] After starting control to reduce the processing load in the suppression transition state ST2, the control unit 20 gradually increases the amount of reduction in the processing load for each control cycle Tc within a predetermined suppression transition period. The suppression transition period is set to, for example, 5 seconds. As will be described in detail later, the control unit 20 reduces the amount of video data to be prepared and the processing load by, for example, not creating some video data or simplifying some video data. In this case, if the amount of video data is suddenly reduced, the display content also suddenly changes, which is undesirable from the perspective of the user 6. Therefore, the control unit 20 gradually increases the amount of data to be reduced. Then, after the suppression transition period, for example, 5 seconds, has elapsed, the control unit 20 transitions to the suppression state ST3.

[0056] In the suppression state ST3, the control unit 20 prepares video data with the reduced processing load. Furthermore, when the state in which the preparation time Tp is shorter than the second threshold time Tth2 continues for a predetermined threshold duration TthD or longer in the suppression state ST3, the control unit 20 transitions to the return transition state ST1. The second threshold time Tth2 is shorter than the control period Tc, and is set to, for example, the same time as the first threshold time Tth1 or a time shorter than the first threshold time Tth1. The threshold duration TthD is a period that is at least multiple times the control period Tc, and is set to, for example, 5 seconds.

[0057] 10, the second threshold time Tth2 is set to be shorter than the first threshold time Tth1. The preparation times Tp5 and Tp6 obtained at the monitoring times tm5 and tm6 are both shorter than the second threshold time Tth2. The control unit 20 transitions to the return transition state ST1 when monitoring results such as the preparation times Tp5 and Tp6 are obtained continuously for a threshold duration TthD or longer.

[0058] In this way, when the state of "preparation time Tp<second threshold time Tth2" is occurring stably, even if the normal state ST0 is restored, it is expected that the above-mentioned conditions (A) and (B) will no longer be met. Note that in some cases, the control unit 20 may not perform such a condition determination, and may simply transition to the restoration transition state ST1 after a time such as 5 seconds has passed, anticipating that the processing load will naturally be reduced after a predetermined period of time has passed.

[0059] In the return transition state ST1, the control unit 20 gradually reduces the amount of processing load reduction for each control cycle Tc within a predetermined return transition period. The return transition period is set to, for example, 5 seconds. Then, the control unit 20 transitions to the normal state ST0 after the return transition period, for example, 5 seconds, has elapsed. By providing the return transition state ST1, it is possible to avoid situations that are undesirable from the perspective of the user 6, as in the case of the suppression transition state ST2.

[0060] <How to reduce processing load> Fig. 11A is a schematic diagram showing an example of the display content of the HUD device in the normal state shown in Fig. 9. In the normal state ST0, for example, as shown in Fig. 11A, seven images VDa1 to VDa3 and VDb to VDe are displayed. The images VDa1 and VDa2 are displayed so as to be superimposed on objects OB1 and OB2, respectively (people in this case). The image VDa3 is displayed so as to be superimposed on object OB3, a vehicle in this case. The images VDb to VDe represent navigation information, direction of travel, road signs, and speed information, respectively, as in Fig. 5.

[0061] Fig. 11B is a schematic diagram showing an example of the display content of the HUD device in the suppression transition state shown in Fig. 9. In the suppression transition state ST2, the display content shown in Fig. 11A is used as a reference, and as indicated by reference numerals 201 and 203 in Fig. 11B, two images VDa2 and VDd in Fig. 11A are deleted. That is, the image VDa2 superimposed on the object OB2 that is farther away than the object OB1, and the image VDd showing the road sign, are deleted. Furthermore, as indicated by reference numeral 202 in Fig. 11B, the image VDa3 in Fig. 11A is simplified by, for example, making it uncolored.

[0062] 11B, an image VDm1, such as a mark, is displayed to notify the user 6 that the display is in the suppression transition state ST2, or in other words, the suppression transition period. In the suppression transition state ST2, the control unit 20 writes the image data of the image VDm1 as a template to a fixed storage area in the frame buffer 18. By displaying such a mark, the user 6 can recognize that the change in the display content is not due to a malfunction, but is due to the activation of the display suppression function.

[0063] When performing the display shown in FIG. 11B, the control unit 20 determines in step S121 shown in FIG. 6 that videos VDa2 and VDd are not displayed and applies simplified display to video VDa3, thereby reducing the amount of video data prepared in step S12. In this way, the control unit 20 reduces the processing load required to prepare the video data. In other words, the control unit 20 shortens the time required for the processing in step S122 and the time required for the processing in step S123. Furthermore, the control unit 20 gradually reduces the amount of data.

[0064] Fig. 11C is a schematic diagram showing an example of the display content of the HUD device in the suppressed state shown in Fig. 9. In the suppressed state ST3, the two images VDa1 and VDb in Fig. 11B are further simplified as indicated by reference numerals 301 and 302 in Fig. 11C, based on the display content shown in Fig. 11B. That is, the two images VDa1 and VDb are simplified by removing color and reducing their size. Furthermore, Fig. 11C displays an image VDm2, such as a mark, to notify the user 6 that the suppressed state ST3, or in other words, the suppressed period, is in effect.

[0065] As an example of specific processing, the control unit 20 stores in advance in the memory 17 a suppression table that defines the correspondence between the types of vehicle information 4 and their priorities. For example, in the suppression transition state ST2, the control unit 20 selects vehicle information 4 in ascending order of priority based on the suppression table, and gradually increases the number of selected vehicle information 4. Then, the control unit 20 hides the video VD representing the selected vehicle information 4 or simplifies it in a predetermined manner.

[0066] Here, the priority in the suppression table is determined, for example, according to the following criteria: First, the vehicle information 4 that contributes more to safe driving is given a higher priority. Furthermore, when the vehicle information 4 indicates a warning and the object of the warning is a person or a vehicle, the priority of the person is given a higher priority. However, in this case, the priority may be weighted according to the distance between the vehicle and the object. For example, if the distance to the person is far and the distance to the vehicle is extremely close, the vehicle may be given priority. Furthermore, a person approaching the vehicle may be given a higher priority than a person moving away from the vehicle.

[0067] 11B and 11C, the priority of the image VDb showing the direction of travel may be lowered if the road is a series of straight lines, and may be higher if the timing for a right or left turn is approaching, based on navigation information. Furthermore, the priority of the image VDb may be lowered if the vehicle is traveling on a frequently used road, based on past driving history, and may be higher if the vehicle is traveling on a road that has not been used before. Furthermore, the priority of the image VDd showing the road sign, i.e., the speed limit, shown in FIGS. 11A and 11B may be changed based on the difference from the vehicle's driving speed.

[0068] Note that the control unit 20 may reduce the processing load by, for example, displaying the video VD once every multiple control cycles Tc, without completely deleting the video VD. Furthermore, the control unit 20 is not limited to such a method of reducing the amount of video data, and may reduce the processing load by, for example, simplifying the distortion correction process in step S124 in Fig. 6, thereby reducing the accuracy.

[0069] <Detailed operation of the control unit> Fig. 12 is a flow diagram showing an example of the processing contents of the control unit shown in Fig. 3A. The control unit 20 executes the flow shown in Fig. 12, for example, when adjustment of the installation angle of the mirror M1 shown in Fig. 2A and the like is completed and the environment for projecting an image is prepared. In Fig. 12, the control unit 20 starts monitoring the preparation time Tp using a timer or the like (step S20). Next, the control unit 20 waits for a start trigger to be generated (step S21). The start trigger is generated every control period Tc.

[0070] When a start trigger occurs in step S21, the control unit 20 acquires information about the vehicle using the communication unit 16, i.e., the information acquisition unit (step S22). Next, the control unit 20 checks the current internal state described in FIG. 9 (step S23). Then, the control unit 20 performs a video data preparation process as described in step S12 of FIG. 6 (step S24). At this time, if the internal state is other than the normal state ST0, the control unit 20 performs control to reduce the processing load required for preparing the video data, as described in FIGS. 9, 11B, and 11C.

[0071] When the control unit 20 completes the preparation process for the video data in step S24, it generates a preparation complete signal (step S25). For example, in response to a start trigger after generating the preparation complete signal, the control unit 20 outputs a display start command to the display drive unit 15. In response to the display start command, the display drive unit 15 performs display processing for the video VD as described in step S13 in FIG. 6 and in FIG. 7. Furthermore, when the control unit 20 completes the preparation process for the video data, it evaluates the preparation time Tp, which is the monitoring result obtained in step S21 (step S26).

[0072] 9 is necessary based on the preparation time Tp, which is the monitoring result (step S27). If a state transition is necessary (step S27: Yes), the control unit 20 determines a transition destination (step S29), performs a state transition to the determined transition destination, and also updates the internal state (step S30). On the other hand, if a state transition is not necessary (step S27: No), the control unit 20 returns to step S21 and waits for the next start trigger, repeating the same process until a request to end the HUD display is generated (step S28).

[0073] <Major Effects of the Embodiment> As described above, the method of the embodiment performs control to reduce the processing load required for preparing video data so that preparation of the video data is completed within the control period Tc, thereby suppressing the occurrence of dropped frames and ensuring a minimum display quality. In particular, even if the HUD device 1 is designed to prevent dropped frames, a software update or the like that involves a change in display specifications can increase the processing load and cause dropped frames. By using the method of the embodiment, dropped frames and the like can be suppressed even in such cases. As a result, it becomes possible to satisfy the required display specifications as much as possible within the range of the hardware's processing performance and flexibly respond to the display specifications.

[0074] Furthermore, by using the method of the embodiment, the user 6 can view various information necessary for driving, such as navigation information such as destination and speed, as well as alert information when an oncoming vehicle or pedestrian is detected, as an image through the windshield 3. Even if the display specifications change, the image can be viewed with a minimum display quality. This makes it possible to provide a HUD device 1 that reduces the need for the user 6 to move their viewpoint and contributes to supporting safe driving. As a result, it becomes possible to prevent traffic accidents. Furthermore, it becomes possible to contribute to "Good health and well-being for all" of the Sustainable Development Goals (SDGs) advocated by the United Nations.

[0075] The invention made by the inventor has been specifically described above based on the embodiments, but the present invention is not limited to the above embodiments and can be modified in various ways without departing from the spirit of the invention. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. [Explanation of symbols]

[0076] 1...Head-up display (HUD) device, 2...Vehicle, 4...Vehicle information, 5...Display area, 6...User, 11...Video display unit, 16...Communication unit (information acquisition unit), 20...Control unit, M1...Mirror (video light projection unit), ST0...Normal state, ST1...Return transition state, ST2...Suppression transition state, ST3...Suppression state, Tc...Control period, Tp...Preparation time, Tth1...First threshold time, Tth2...Second threshold time, TthD...Threshold duration, VD...Video

Claims

1. A head-up display device mounted on a vehicle, an information acquisition unit that acquires information about the vehicle; an image display unit that displays an image and emits image light of the displayed image; an image light projection unit that projects the image light emitted from the image display unit onto a display area, thereby allowing the projected image light to be viewed as a virtual image; a control unit that prepares video data based on the information about the vehicle acquired by the information acquisition unit, and displays a video based on the prepared video data on the video display unit; Equipped with the control unit compares a preparation time required for preparing the video data with a predetermined processing cycle, and uses a first threshold time shorter than the processing cycle, and when the preparation time is longer than the first threshold time multiple times in succession, performs control to reduce the processing load required for preparing the video data by reducing part of the content of the video data so that the preparation of the video data is completed within the processing cycle. Head-up display device.

2. The head-up display device according to claim 1, After starting the control to reduce the processing load, the control unit sets a suppression transition time which is a time from the start of the control to the time when a preset target value for the amount of reduction in the processing load is reached, and displays the image while gradually increasing the amount of reduction in the processing load for each processing cycle during the suppression transition time. Head-up display device.

3. 3. The head-up display device according to claim 2, the control unit sets a return transition time, which is a time required for returning the processing load that has been reduced to the target value to the original processing load, and after starting control to return the processing load to the original processing load, displays the video while gradually reducing the amount of reduction in the processing load for each processing cycle during the return transition time. Head-up display device.

4. 4. The head-up display device according to claim 3, After the suppression transition time has elapsed, the control unit performs control during the return transition time after a suppression period in which the video data is prepared with the reduced processing load. Head-up display device.

5. 5. The head-up display device according to claim 4, The control unit, during the suppression period, uses a second threshold time shorter than the processing cycle and a threshold duration that is a period that is multiple times or more of the processing cycle, and starts control with the return transition time when a state in which the preparation time is shorter than the second threshold time continues for the threshold duration or longer. Head-up display device.

6. 3. The head-up display device according to claim 2, the control unit generates the video data for notifying the user that the suppression transition time has arrived, and displays the video during the suppression transition time. Head-up display device.

7. The head-up display device according to claim 1, the control unit reduces the processing load by reducing the amount of the video data to be prepared based on a table that defines a correspondence between information about the vehicle and a priority. Head-up display device.

8. A head-up display device mounted on a vehicle, an information acquisition unit that acquires information about the vehicle; an image display unit that displays an image and emits image light of the displayed image; an image light projection unit that projects the image light emitted from the image display unit onto a display area, thereby allowing the projected image light to be viewed as a virtual image; Equipped with preparing video data based on the information about the vehicle acquired by the information acquisition unit, and if the preparation conditions for the video data do not satisfy predetermined conditions in a predetermined processing cycle, displaying a mark indicating a change in the content of the video data to be prepared, and not creating part of the video data or preparing video data that simplifies part of the video data; Head-up display device.

9. A method for processing video data for a head-up display device mounted on a vehicle, comprising: obtaining information about the vehicle; preparing image data based on the acquired information about the vehicle, and displaying an image based on the prepared image data on an image display unit of the head-up display device; When preparing the video data, a preparation time required for preparing the video data is compared with a predetermined processing cycle, and a first threshold time shorter than the processing cycle is used. When the preparation time is longer than the first threshold time multiple times in succession, control is performed to reduce the processing load required for preparing the video data by reducing part of the content of the video data so that the preparation of the video data is completed within the processing cycle. How video data is processed.

10. 10. The video data processing method according to claim 9, After starting the control to reduce the processing load, a suppression transition time is set, which is a time from the start of the control to reduce the processing load until a preset target value for the amount of reduction in the processing load is reached, and the image is displayed while the amount of reduction in the processing load is increased stepwise for each processing cycle during the suppression transition time. How video data is processed.

11. 11. The video data processing method according to claim 10, a return transition time, which is a time required for the processing load reduced to the target value to be returned to the original processing load, is set, and after control for returning the processing load to the original processing load is started, the image is displayed while gradually reducing the amount of reduction in the processing load for each processing cycle during the return transition time. How video data is processed.

Images