Head-up display device and image data processing method

Abstract

Provided are a head-up display device and a video data processing method that can flexibly cope with various display specifications. In addition, a contribution is made to sustainable development goals' 3, guarantee of health and welfare for everyone '. The present invention is provided with: an image display unit that displays an image and emits image light of the displayed image; a video light projection unit that projects the projected video light to the display area so as to visually recognize the projected video light as a virtual image; and a control unit that determines display content, prepares video data, and causes the video display unit to display a video based on the data. The control unit performs control to reduce the processing load required for preparation of the video data so that the preparation of the video data is completed within a predetermined processing cycle. Alternatively, if the preparation time required for preparing the video data of the temporarily determined display content before the determination of the display content is longer than the predetermined processing cycle, the control unit changes the temporarily determined display content.

Description

Technical Field

[0001] The present invention relates to a head-up display device and a method for processing image data, and for example, to a technology of a head-up display device using AR (Augmented Reality). Background Art

[0002] Patent document 1 shows a display system that extracts an object as an AR display object from camera data, generates AR image data of the object, and sets the frame rate of the AR image data according to the importance of the object. For example, when the overall frame rate is 60fps and three AR image data are displayed in time division, the frame rates of the three AR image data are set to 30fps, 20fps, and 10fps, respectively, according to the importance. As a result, the more important the AR image data is, the more flicker can be suppressed.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-6164 Summary of the invention

[0006] The head-up display device prepares various image data based on information obtained from ADAS (Advanced Driver Assistance Systems) and projects image light based on the image data to the display area, so that the image is visually recognized as a virtual image. At this time, the preparation time required for the preparation of the image data varies depending on the type and number of images, the size of the image corresponding to the distance from the virtual image, the display format such as 2D / 3D, the presence or absence of graphic effects, etc. In addition, in the specification, the head-up display device is also referred to as a HUD device.

[0007] Here, in the HUD device, the processing performance such as the maximum frame rate is usually fixedly determined mainly based on the hardware specifications. On the other hand, for example, sometimes it is desired not to change the hardware of the HUD device itself, but to change the display specifications of the HUD device by updating the software. In this case, there are many cases where the required display specifications of the HUD device are accompanied by an increase in the processing load. As a result, there is a possibility that the processing performance of the hardware is insufficient, for example, frame loss occurs during AR display, and the required display specifications are not met.

[0008] The present invention has been made in view of such circumstances, and one object of the present invention is to provide a head-up display device and a method for processing image data that can meet required display specifications as much as possible within the processing performance of hardware and can flexibly cope with various display specifications.

[0009] The above and other objects and novel features of the present invention will become more apparent from the description of this specification and the accompanying drawings.

[0010] Among the inventions disclosed in this application, the outline of representative inventions will be briefly described as follows.

[0011] A typical head-up display device includes: an image display unit that displays an image and emits image light of the displayed image; an image light projection unit that projects the emitted image light onto a display area so that the projected image light is visually recognized as a virtual image; and a control unit that determines display content based on information obtained about a vehicle, prepares image data based on the determined display content, and displays an image based on the prepared image data on the image display unit. The control unit performs control to reduce the processing load required for the preparation of the image data in such a manner that the preparation of the image data is completed within a predetermined processing cycle. In addition, when the preparation time required for the preparation of the image data of the temporarily determined display content before the display content is determined is longer than the predetermined processing cycle, the control unit changes the temporarily determined display content.

[0012] If the effects obtained by the representative invention among the inventions disclosed in this application are briefly described, the head-up display device can flexibly cope with various display standards. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a schematic diagram showing a configuration example of a vehicle equipped with the head-up display device according to the first embodiment.

[0014] Figure 2A It is shown Figure 1 Schematic diagram of a structural example of the main parts of the HUD device.

[0015] Figure 2B It is shown Figure 1 The HUD device in Figure 2A Schematic diagram of the structure of different main parts.

[0016] Figure 3A It is shown in Figure 2A as well as Figure 2B A block diagram of an example of the configuration of a main portion of a control system that is responsible for controlling the HUD device shown.

[0017] Figure 3B It is shown in Figure 2A as well as Figure 2B The control system responsible for controlling the HUD device shown in the figure is Figure 3A Block diagram of an example of the structure of the different main parts.

[0018] Figure 4 It is shown in Figure 3A as well as Figure 3B A block diagram showing an example of the structure of a portion related to a control unit.

[0019] Figure 5 It is shown Figure 1 A schematic diagram showing an example of display content of the HUD device shown.

[0020] Figure 6 It is shown in Figure 3A A flowchart showing an example of a processing procedure when an image is displayed in the HUD device shown.

[0021] Figure 7 It is a schematic diagram showing the use of Figure 6 The flow shown is a timing diagram of an example of a problem when displaying images.

[0022] Fig. 8A This is a schematic diagram showing how the display content changes when no frame loss occurs.

[0023] Figure 8B This is a schematic diagram showing how display contents change when frame loss occurs.

[0024] Fig. 9 It is shown Figure 3A A diagram showing an example of an internal state of a control unit shown.

[0025] Fig.10 It shows the use of Figure 3A FIG. 1 is a diagram showing an example of the monitoring result of the preparation time executed by the control unit shown.

[0026] Fig.11A It is shown Fig. 9 Schematic diagram of an example of display content of the HUD device in the normal state shown.

[0027] Fig. 11B It is shown Fig. 9 Schematic diagram of an example of display content of the HUD device in the suppressed transition state shown.

[0028] Fig. 11C It is shown Fig. 9 Schematic diagram of an example of display content of the HUD device in the suppressed state shown.

[0029] Fig.12 It is shown Figure 3A A flowchart showing an example of the processing contents of the control unit shown.

[0030] Fig.13A It is shown in Figure 3A as well as Figure 3BFIG. 1 is a schematic diagram showing an example of the relationship between image data written into a frame buffer and display contents displayed on an image display unit in the HUD device shown.

[0031] Fig. 13B is shown with Fig.13A Schematic diagram of an example of different relationships.

[0032] Fig.14 It is shown in Figure 6 FIG. 1 is a diagram showing an example of items that affect the preparation time of image data.

[0033] Fig.15 This is a flowchart showing an example of a processing procedure when displaying an image in the HUD device according to the second embodiment.

[0034] Fig.16 It is shown Fig.15 A flowchart showing an example of detailed processing contents of the display content adjustment process in FIG.

[0035] Fig.17A It is shown Fig.16 FIG. 4 is a diagram showing a specific example of the method for predicting the preparation time (step S412 ).

[0036] Fig. 17B is shown with Fig.17A Figures of different specific examples.

[0037] Fig.18 Yes Description Fig.16 FIG. 4 is a diagram showing an example of a method for changing display content (step S414 ) in FIG.

[0038] Fig.19 It is shown in Fig.18 Schematic diagram of an example of display content of the HUD device obtained as a result of changing the display content.

[0039] Fig. 20 is used Fig.15 as well as Fig.16 The process shown is a timing diagram for displaying images.

[0040] Fig.21A This is a schematic diagram showing an example of main processing contents of the control unit in the HUD device according to the third embodiment.

[0041] Fig. 21B yes Fig.21A Supplementary Figure.

[0042] Fig.22A is shown with Fig.21A Schematic diagram of a different example.

[0043] Fig. 22BYes Fig.22A is a supplementary figure.

[0044] Fig.23 is a diagram showing a structural example of a display setting table stored in the control unit in the HUD device of Embodiment 3.

[0045] Fig.24A is a flowchart showing an example of the processing content of the control unit in the HUD device of Embodiment 3.

[0046] Fig. 24B is a flowchart showing an example of processing content different from Fig.24A . Detailed Embodiments

[0047] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In addition, in all the drawings for explaining the embodiments, the same reference numerals are attached to the same components in principle, and repeated explanations thereof are omitted. (Embodiment 1)

[0048] <Outline of HUD Device>

[0049] Figure 1 is a schematic diagram showing a structural example of a vehicle equipped with the head-up display device of Embodiment 1. Figure 1 The head-up display (HUD) device 1 shown is mounted on a vehicle 2 which is one of the means of transportation. The vehicle 2 is typically an automobile, but is not necessarily limited thereto, and may be a railway vehicle or the like. In addition, the means of transportation is not limited to a vehicle, and may be an aircraft or the like. In addition, on the vehicle 2, for example, a control unit 21 called an ECU (Electronic Control Unit) is mounted.

[0050] The control unit 21 obtains vehicle information 4 from, for example, various sensors provided in each part of the vehicle 2 and also from a navigation device or the like. The various sensors detect various events generated in the vehicle 2 and also detect various parameter values related to the driving condition. The HUD device 1 obtains the vehicle information 4 obtained by the control unit 21 using, for example, CAN (Controller Area Network) communication or the like.

[0051] The vehicle information 4 includes, for example, the speed information of the vehicle 2, gear information, steering wheel steering angle information, light lighting information, external light information, distance information, infrared information, engine ON / OFF information, camera image information inside and outside the vehicle, accelerometer gyro information, GPS (Global Positioning System) information, navigation information, vehicle-to-vehicle communication information, and road-to-vehicle communication information. The GPS information also includes information such as the current time. In addition, the vehicle information 4 also includes various warning information. The HUD device 1 projects image light onto a display area such as the windshield 3 based on such vehicle information 4. As a result, the HUD device 1 enables the driver and other users to visually recognize the image light projected onto the display area as a virtual image, specifically, to visually recognize it as a virtual image superimposed on the scenery in front of the vehicle 2.

[0052] Figure 2A It is shown Figure 1 Schematic diagram of a structural example of the main parts of the HUD device. Figure 2A The HUD device 1 shown in the figure includes, for example, an image display unit 11, mirrors M1 and M2, and a mirror driving unit 14, etc., which are accommodated in a housing 12. The image display unit 11 is, for example, a display panel such as an LCD (Liquid Crystal Display), a projector, etc., and displays an image based on input image data and emits image light of the displayed image.

[0053] The reflector M2 reflects the image light from the image display unit 11 toward the reflector M1. The reflector M2 is a mirror that is effective in order to save space and ensure a longer optical path. Depending on the space in the HUD frame and the required optical path value, the reflector M2 may not be configured, or multiple reflectors M2 may be configured. The reflector M1 functions as an image light projection unit. The reflector M1, which serves as an image light projection unit, projects the image light emitted from the image display unit 11 and reflected by the reflector M2 to the display area 5 of the windshield 3 via the opening 7 provided in the instrument panel 10. Thus, the image light projection unit enables the user 6 to visually recognize the projected image light as a virtual image.

[0054] Specifically, the reflector M1 is, for example, a concave mirror (magnifying glass), which reflects and magnifies the image light reflected by the reflector M2, and projects it to the display area 5 through the opening 7. The image light projected to the display area 5 is reflected by the display area 5 and enters the eyes of the user 6. As a result, the user 6 visually recognizes the image light projected to the display area 5 as a virtual image 9 existing in front of the transparent windshield 3 in the form of superimposing the image light projected to the display area 5 on the scenery outside the vehicle (roads, buildings, people, etc.). The information represented by the virtual image 9 includes, for example, road signs, the current speed of the vehicle, various information on objects attached to the scenery, that is, AR information, and other examples.

[0055] In addition, the reflectors M1 and M2 may be, for example, free-form reflectors, reflectors having an asymmetric shape of the optical axis, etc. Here, the setting angle of the reflector M2 is fixed. On the other hand, the reflector M1 is provided with a reflector drive unit 14. The reflector drive unit 14 variably adjusts the setting angle of the reflector M1. In detail, the reflector drive unit 14 includes, for example, a motor, and the reflector M1 is rotated by the rotation of the motor.

[0056] By variably adjusting the setting angle of the reflector M1, the position of the display area 5 on the windshield 3, that is, the position of the virtual image visually recognized by the user 6 in the up-down direction, can be adjusted. Furthermore, by variably adjusting the setting angle of the reflector M1, it is possible to prevent the image display unit 11 from being irradiated by sunlight. Specifically, sunlight may move in the reverse direction in the optical path of the image light and enter the image display unit 11. In the case where the possibility of damage to the image display unit 11 due to the incidence of the sunlight increases, the setting angle of the reflector M1 can be changed in a manner to prevent the sunlight from reaching the image display unit 11.

[0057] Figure 2B It is shown Figure 1 The HUD device in Figure 2A Schematic diagram of the structure of different main parts. Figure 2B The HUD device 1 shown is Figure 2A The structure shown in FIG. 1 is different from that shown in FIG. 1 in that a lens LS is provided in place of the reflector M2 in the frame 12. The image light from the image display unit 11 is incident on the reflector M1 via the lens LS. Figure 2A Similarly, the incident image light is projected onto the display area 5 through the opening 7. Figure 2A Similarly to the case of , a mirror driving unit is provided at the mirror M1. Figure 2B The structure shown can be applied to a case where the windshield 3 is arranged at a nearly vertical angle, such as in a van or a truck.

[0058] Figure 3A It is shown in Figure 2A as well as Figure 2B A block diagram of an example of the configuration of a main portion of a control system that is responsible for controlling the HUD device shown. Figure 3A The HUD device 1 shown includes a mirror driving unit 14 , a display driving unit 15 , a communication unit 16 , a memory 17 , a frame buffer 18 , and a control unit 20 , which are connected to each other via a bus 13 .

[0059] The communication unit 16 is a device for receiving and sending vehicle information, and is implemented, for example, by a communication interface circuit, and functions as an information acquisition unit. The communication unit 16 acquires or receives information related to the vehicle from the control unit 21 using CAN communication, and sends the received information related to the vehicle to the control unit 20. The control unit 20 controls the mirror drive unit 14 and the display drive unit 15 based on the information from the communication unit 16. The mirror drive unit 14, for example, controls the display drive unit 15 based on the command from the control unit 20. Figure 2A As described in the above, the setting angle of the reflector M1 is adjusted. The reflector drive unit 14 can be operated as described in Figure 2A The motor described in and the motor driver circuit that drives the motor are implemented.

[0060] The frame buffer 18 is composed of, for example, a volatile memory, and stores image data. The display driver 15 reads the image data stored in the frame buffer 18 via the bus 13, and drives the image display unit 11 according to the image data. The image display unit 11 is, for example, a liquid crystal display having a light source and a display panel. The display panel displays an image by modulating the backlight irradiated from the light source according to the image data for each pixel. In this case, the display driver 15 can be implemented by an LCD driver circuit or the like.

[0061] The memory 17 is composed of, for example, a combination of a volatile memory and a nonvolatile memory, and stores programs, data, and the like used in the control unit 20. The control unit 20 is implemented by, for example, a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and controls the entire HUD device 1 by executing the program stored in the memory 17. As one of the steps, the control unit 20 prepares image data based on the information related to the vehicle acquired by the communication unit 16, that is, the information acquisition unit, including creating the image data, and causing an image based on the prepared image data to be displayed on the image display unit 11.

[0062] also, Figure 3A The communication unit 16, memory 17, frame buffer 18, and control unit 20 shown in the figure can be mounted on a microcontroller, etc. However, it is not limited to such an installation method, and for example, an installation method that appropriately combines FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), etc. may also be used.

[0063] Figure 3B It is shown in Figure 2A as well as Figure 2BThe control system responsible for controlling the HUD device shown in the figure is Figure 3A Block diagram of an example of the structure of the different main parts. Figure 3B The HUD device 1 shown is Figure 3A The configuration example shown is different from that in which the memory 17 and the control unit 20 are not provided. In this case, the control unit 21 replaces Figure 3A The control unit 20 shown in FIG. 1 creates video data, and writes the created video data into the frame buffer 18 via the communication processing unit 16 a .

[0064] In addition, Figure 3B In the case of the structural example shown in Figure 3A The control unit 20 described in the above may also function as the communication processing unit 16a. Alternatively, the control unit 21 and the communication processing unit 16a may each share some functions. Figure 3B In the structural example, a mirror driving unit 14, a display driving unit 15, and a communication processing unit 16a are provided. The communication processing unit 16a and Figure 3A Different from the communication unit 16, the information related to the vehicle is received from the control unit 21 using CAN communication or the like, the received information is processed, and the operation of the mirror drive unit 14 and the display drive unit 15 is adjusted according to the processing result.

[0065] Figure 4 It is shown in Figure 3A as well as Figure 3B The control unit 21 is a block diagram of a configuration example of a portion related to the control unit. Figure 1 The vehicle information 4 is obtained as described in Figure 4 The data is generated by various sensors and other information acquisition devices connected to the control unit 21. Figure 4 An example of the information acquisition device is shown in FIG.

[0066] exist Figure 4 For example, the vehicle speed sensor 101 detects Figure 1 The speed of the vehicle 2 is detected, and speed information is generated as a detection result. The gear position sensor 102 detects the current gear and generates gear information as a detection result. The steering wheel steering angle sensor 103 detects the current steering wheel steering angle and generates steering wheel steering angle information as a detection result. The headlight sensor 104 detects the ON / OFF of the headlight and generates light lighting information as a detection result. The illuminance sensor 105 and the chromaticity sensor 106 detect external light and generate external light information as a detection result.

[0067] The distance measuring sensor 107 detects the distance between the vehicle 2 and an external object, and generates distance information as a detection result. The infrared sensor 108 detects the presence or absence of an object near the vehicle 2, the distance, etc., and generates infrared information as a detection result. The engine start sensor 109 detects the ON / OFF of the engine, and generates ON / OFF information as a detection result. The acceleration sensor 110 and the gyro sensor 111 detect the acceleration and angular velocity of the vehicle 2, respectively, and generate acceleration gyro information indicating the posture and behavior of the vehicle 2 as a detection result. The temperature sensor 112 detects the temperature inside and outside the vehicle, and generates temperature information as a detection result.

[0068] The wireless receiver 113 for road-to-vehicle communication generates road-to-vehicle communication information through road-to-vehicle communication between the vehicle 2 and the road, signs, traffic lights, etc. The wireless receiver 114 for vehicle-to-vehicle communication generates vehicle-to-vehicle communication information through vehicle-to-vehicle communication between the vehicle 2 and other surrounding vehicles. The in-vehicle camera 115 and the out-vehicle camera 116 generate in-vehicle camera image information and out-vehicle camera image information by shooting the inside and outside of the vehicle, respectively. The in-vehicle camera 115 is, for example, Figure 2A The camera for DMS (Driver Monitoring System) captures the posture, eye position, movement, etc. of the user 6 as shown in FIG. In this case, the fatigue state of the user 6 and the position of the line of sight can be understood by analyzing the captured image.

[0069] On the other hand, the vehicle exterior camera 116 captures the surrounding conditions such as the front and rear of the vehicle 2. In this case, by analyzing the captured images, it is possible to understand whether there are obstacles such as other vehicles and people in the surrounding area, buildings or terrain, road conditions such as rain or snow, ice, bumps, etc., and road signs. In addition, for example, the vehicle exterior camera 116 may also include a driving recorder that uses images to record the conditions during driving.

[0070] The GPS receiver 117 generates GPS information obtained by receiving GPS signals. For example, the current time can be obtained using the GPS receiver 117. The VICS (Vehicle Information and Communication System, a registered trademark) receiver 118 generates VICS information obtained by receiving VICS signals. The GPS receiver 117 and the VICS receiver 118 can also be set as part of the navigation device. In addition, Figure 4 The various information acquisition devices shown can be deleted, added, or replaced with other types of devices as appropriate.

[0071] <About HUD device display>

[0072] Figure 5 It is shown Figure 1 A schematic diagram of an example of display content of the HUD device shown. The display content shows an example of AR display, showing Figure 2A An example of a virtual image 9 is shown in FIG. Figure 5 In the example shown, five images VDa to VDe are displayed. In the specification, the plurality of images VDa to VDe are collectively referred to as images VD, or images VD, or image objects VD.

[0073] The image VDa is one of the landscapes and is displayed in a manner overlapping with the object OB, here a person, which is the target of AR. Figure 4 The various information acquisition devices shown detect the object OB. That is, the HUD device 1 acquires information indicating the detection result of the object OB from the control unit 21. In addition, the image VDa indicates warning information for prompting the user 6, for example, the driver, to pay attention to the object OB.

[0074] Image VDb is displayed on a road as one of the scenery, and indicates the traveling direction of vehicle 2. Image VDc indicates navigation information. Image VDd, for example, indicates a road sign as one of the road-to-vehicle communication information. Image VDe indicates speed information of vehicle 2. In addition, images VDa and VDb are 3D graphics, and images VDc to VDe are 2D graphics.

[0075] Figure 6 It is shown in Figure 3A A flowchart showing an example of a process for displaying an image in a HUD device shown in FIG. Figure 6 In the embodiment of the present invention, first, the communication unit 16, i.e., the information acquisition unit, acquires information related to the vehicle from the control unit 21 (step S11). Next, the control unit 20 performs a preparation process of the image data including steps S121 to S124 (step S12). For example, the processor executes the image processing program in the memory 17 to perform the processing of steps S121 to S123.

[0076] In step S121, the control unit 20 obtains the information related to the transportation means in step S11, such as Figure 1 Specifically, the control unit 20 selects the vehicle information 4 corresponding to the display, and determines the position, size, and layout of the image VD representing the selected vehicle information 4. Figure 5In the example shown, the control unit 20 determines to display five images VDa to VDe as display contents. In step S122, the control unit 20 creates image data for each image VD based on the display contents determined in step S121.

[0077] In step S123, the control unit 20 writes the plurality of image data generated in step S122 into storage areas in the frame buffer 18 corresponding to positions to be displayed. In step S124, the control unit 20 performs distortion correction corresponding to, for example, the curvature of the windshield 3 on the entire image data in the frame buffer 18. In addition, the distortion correction may be realized by, for example, the processor executing a distortion correction program in the memory 17 or by using dedicated hardware.

[0078] After the image data preparation process (step S12) is completed, the image display process (step S13) is performed. In step S13, the display driver 15 reads the image data stored in the frame buffer 18 and drives the image display unit 11 according to the image data, so that the image display unit 11 displays the image VD. Figure 6 The processing process shown is executed synchronously with the processing cycle determined by the frame rate. For example, when the frame rate is 60fps, the processing cycle is 16.6ms. In addition, the processing process when displaying the image VD is not particularly limited to Figure 6 The examples shown can be replaced with various generally known processes.

[0079] Figure 7 It is a schematic diagram showing the use of Figure 6 The flow chart shown is an example of a problem when displaying an image. Figure 7 In FIG. 1 , the actions performed in the first to fifth processing cycles or control cycles Tc[1] to Tc[5] are shown. In the specification, the processing cycles or control cycles Tc[1] to Tc[5] are collectively referred to as processing cycles or control cycles Tc. The control cycle Tc is, for example, 16.6 ms. Figure 7 In the pipelined manner, Figure 6 The actions in the case of the processing of steps S11, S12 and the processing of step S13.

[0080] exist Figure 7In the first control cycle Tc[1], information related to the vehicle is obtained (step S11), and image data is prepared based on the information (step S12). The preparation time Tp[1] of the image data required for the processing of steps S11 and S12 is shorter than the control cycle Tc[1]. This is, for example, a situation where the number of AR objects or even the number of displayed images VD is small, and the preparation time Tp[1] converges within the control cycle Tc[1]. Similarly, in the second control cycle Tc[2], information related to the vehicle is also obtained (step S11), and image data is prepared based on the information (step S12). In addition, in parallel with this, in the second control cycle Tc[2], the display processing of the image VD is performed based on the image data prepared in the first control cycle Tc[1] (step S13).

[0081] Here, in the second control cycle Tc[2], the preparation time Tp[2] of the image data required for the processing of steps S11 and S12 is longer than the control cycle Tc[2]. The preparation time Tp[2] may be extended, for example, due to an increase in the object of AR or even the displayed image VD. Therefore, the image data is not reflected in the display processing in the third control cycle Tc[3], but is reflected in the display processing in the fourth control cycle Tc[4]. As a result, frame loss occurs in the third control cycle Tc[3].

[0082] In addition, in the fourth control cycle Tc[4], the preparation time Tp[3] of the image data required for the processing of steps S11 and S12 is shorter than the control cycle Tc[4]. Therefore, the image data is reflected in the display processing in the fifth control cycle Tc[5]. However, depending on the method, due to frame loss, the image data prepared in the preparation time Tp[3] may be the image data that should have been prepared in the third control cycle Tc[3]. In addition, in the specification, multiple preparation times Tp[1]~Tp[3] are collectively referred to as the preparation time Tp.

[0083] Fig. 8A This is a schematic diagram showing how the display content changes when no frame loss occurs. Figure 8B FIG. 4 is a schematic diagram showing how the display content changes when frame loss occurs. Fig. 8A as well as Figure 8B As shown in FIG. 1 , it is assumed that the object OB (here, a person) moves from time t1 to time t2. If no frame loss occurs, Fig. 8A As shown, at the position of the object OB (t1) at the time t1, the image VDa (t1) is superimposed and displayed, and at the position of the object OB (t2) at the time t2, the image VDa (t2) is superimposed and displayed.

[0084] Thus, when no frame loss occurs, a high-real-time image VD of the object being tracked by AR can be displayed. On the other hand, when frame loss occurs, especially when frame loss occurs continuously in a continuous control period Tc, as shown in FIG. Figure 8B As shown, at the object OB (t2) at time t2, the image VDa (t2) is displayed at a position offset from the position of the object OB (t1). In this case, the image VDa (t2) does not completely overlap the object OB (t2). In this way, when frame loss occurs, the image VD with low real-time performance that does not track the target of AR may be displayed.

[0085] <General Operation of Control Unit>

[0086] Therefore, in order to display a highly real-time image VD, the control unit 20 performs control to reduce the processing load required for the preparation of the image data in a manner that the preparation of the image data is completed within a predetermined processing cycle or control cycle Tc. That is, the control unit 20 prepares the image data based on the information related to the vehicle received by the communication unit 16, and when the preparation conditions of the image data do not meet the predetermined conditions within the predetermined processing cycle, the control unit 20 changes a part of the content of the image data to be prepared and prepares it. In more detail, the control unit 20 monitors the preparation time Tp required for the preparation of the image data, and starts the control to reduce the processing load when the preparation time Tp meets the predetermined conditions. In addition, the predetermined processing cycle or control cycle Tc is determined based on the frame rate.

[0087] In the Figure 7 Assuming the action method described in the above, the preparation time Tp is Figure 6 On the other hand, if the operation mode of executing the processing of step S11 and the processing of step S12 in a pipeline manner is assumed, the preparation time Tp may be the time required for the processing of step S12. However, in order to improve the real-time performance, it is better to use Figure 7 The action method described in .

[0088] The control unit 20 monitors the preparation time Tp successively. For example, the preparation time can be monitored in conjunction with the vehicle time or using a timer. Figure 7 When the action method described in is the premise, such as Figure 3A As shown, the control unit 20 uses a timer or the like to monitor the preparation time Tp required from the start of acquiring the information related to the vehicle using the communication unit 16 to the writing of all the generated image data into the frame buffer 18. Figure 3B In the case of the structure shown, Figure 3AUnlike the case of , the communication processing unit 16 a uses a timer or the like to monitor the preparation time Tp required to obtain information from the control unit 21 , process the obtained information, and write all the processed results into the frame buffer 18 .

[0089] Fig. 9 It is shown Figure 3A A diagram showing an example of an internal state of a control unit shown. Fig.10 It shows the use of Figure 3A FIG. 1 is a diagram showing an example of the monitoring result of the preparation time performed by the control unit shown in FIG. Fig. 9 As shown in FIG. 1 , the control unit 20 has a normal state ST0, a recovery transition state ST1, a suppression transition state ST2, and a suppression state ST3 as internal states. In other words, the control unit 20 has a normal mode ST0, a recovery transition mode ST1, a suppression transition mode ST2, and a suppression mode ST3 as operation modes. The four states are an example, and in other examples, the control unit 20 may have two states, the normal state ST0 and the suppression state ST3.

[0090] In the normal state ST0, the control unit 20 prepares the image data as usual based on the information obtained using the communication unit 16, and displays the image VD based on the image data on the image display unit 11. In addition, in the normal state ST0, the control unit 20 transitions to the suppressed transition state ST2 when (A) the preparation time Tp as a monitoring result is longer than the control cycle Tc, or (B) the preparation time Tp is longer than the first threshold time Tth1 for a plurality of consecutive times. The number of consecutive times is determined to be, for example, a value of 2 or more and 10 or less.

[0091] As a specific example, Fig.10 , the preparation time Tp4 obtained at the monitoring time tm4 is longer than the control cycle Tc. In addition, the preparation times Tp2 and Tp3 obtained at the consecutive monitoring times tm2 and tm3 are both shorter than the control cycle Tc, but longer than the first threshold time Tth1. The first threshold time Tth1 is a time shorter than 16.6 ms as the control cycle Tc, for example, 15.0 ms.

[0092] In the normal state ST0, the control unit 20 transitions to the suppression transition state ST2 when (A) a monitoring result such as the preparation time Tp4 is obtained, or (B) the monitoring results such as the preparation time Tp2 and Tp3 are obtained multiple times in a row, for example, twice in a row. Condition (A) is a condition for quickly eliminating frame loss. On the other hand, condition (B) is a condition for preventing in advance the situation in which the preparation time Tp is longer than the control cycle Tc in the near future, or even the situation in which frame loss occurs.

[0093] In the suppressed transition state ST2, after the control of reducing the processing load is started, the control unit 20 increases the amount of reduction of the processing load for each control cycle Tc in stages within the predetermined suppressed transition period. The suppressed transition period is determined to be, for example, 5 seconds. As described in detail later, the control unit 20 reduces the processing load by reducing the amount of image data to be prepared, for example, by not producing a part of the image data or simplifying a part of the image data. In this case, when it is assumed that the amount of image data is reduced and changes rapidly, the displayed content also changes rapidly, which is not preferred from the perspective of the user 6. Therefore, the control unit 20 increases the amount of data to be reduced in stages. Moreover, after the suppressed transition period, for example, 5 seconds, has passed, the control unit 20 migrates to the suppressed state ST3.

[0094] The control unit 20 prepares image data with a reduced processing load in the suppression state ST3. In addition, in the suppression state ST3, when the state in which the preparation time Tp is shorter than the second threshold time Tth2 continues for a predetermined threshold duration period TthD or longer, the control unit 20 transitions to the recovery transition state ST1. The second threshold time Tth2 is a time shorter than the control cycle Tc, and is determined to be, for example, the same time as the first threshold time Tth1, or a time shorter than the first threshold time Tth1. The threshold duration period TthD is a period that is several times or more of the control cycle Tc, and is determined to be, for example, 5 seconds.

[0095] As a specific example, Fig.10 In the embodiment, the second threshold time Tth2 is determined to be shorter than the first threshold time Tth1. The preparation times Tp5 and Tp6 obtained at the monitoring times tm5 and tm6 are both shorter than the second threshold time Tth2. When the control unit 20 obtains monitoring results such as the preparation times Tp5 and Tp6 for more than the threshold duration TthD, it transitions to the recovery transition state ST1.

[0096] In this way, when the state of "preparation time Tp < second threshold time Tth2" is stably generated, it is estimated that even if the normal state ST0 is restored, the above conditions (A) and (B) are not met. In addition, the control unit 20 may estimate that the processing load will naturally decrease as a predetermined period passes, and may not perform such condition determination, but simply transition to the recovery transition state ST1 after a period of time such as 5 seconds has passed, depending on the situation.

[0097] In the recovery transition state ST1, the control unit 20 gradually reduces the amount of reduction in the processing load for each control cycle Tc within a predetermined recovery transition period. The recovery transition period is determined to be, for example, 5 seconds. Then, after the recovery transition period, for example, 5 seconds, has passed, the control unit 20 transitions to the normal state ST0. By setting the recovery transition state ST1, as in the case of the suppression transition state ST2, it is possible to avoid a situation that is not preferable from the perspective of the user 6.

[0098] <Methods to reduce processing load>

[0099] Fig.11A It is shown Fig. 9 FIG. 1 is a schematic diagram showing an example of the display content of the HUD device in the normal state. In the normal state ST0, for example, Fig.11A As shown, seven images VDa1 to VDa3, VDb to VDe are displayed. Images VDa1 and VDa2 are displayed in a manner overlapping objects OB1 and OB2 (here, people), respectively. Image VDa3 is displayed in a manner overlapping object OB3 (here, a vehicle). Images VDb to VDe are displayed in a manner overlapping objects OB1 and OB2 (here, people). Figure 5 Similarly, it represents navigation information, travel direction, road signs, and speed information respectively.

[0100] Fig. 11B It is shown Fig. 9 FIG. 1 is a schematic diagram showing an example of the display content of the HUD device in the transition suppression state shown in FIG. Fig.11A The displayed contents are as follows: Fig. 11B As shown in symbols 201 and 203, delete Fig.11A That is, the image VDa2 overlapping the object OB2 farther than the object OB1 and the image VDd representing the road sign are deleted. Fig. 11B As shown in the symbol 202 in FIG. Fig.11A The image VDa3 in the image is made non-colored, and the image VDa3 is simplified.

[0101] Furthermore, in Fig. 11B In the control unit 20, the image VDm1 is displayed to notify the user 6 that the transition is in the inhibited state ST2, in other words, during the inhibited transition, such as a mark. In the inhibited transition state ST2, the control unit 20 writes the image data of the image VDm1 as a template to a fixed storage area in the frame buffer 18. By displaying such a mark, the user 6 can recognize that the change in the display content is not caused by a malfunction, but caused by the operation of the display inhibiting function.

[0102] In progress Fig. 11BIn the case of the display shown in FIG. Figure 6 In step S121 shown in the figure, the image VDa2 and VDd are determined not to be displayed, and simplified display is applied to the image VDa3, thereby reducing the amount of image data prepared in step S12. As a result, the control unit 20 reduces the processing load required for the preparation of image data. That is, the control unit 20 shortens the time required for the processing in step S122 and the time required for the processing in step S123. In addition, the control unit 20 promotes such reduction in data amount in stages.

[0103] Fig. 11C It is shown Fig. 9 FIG. 1 is a schematic diagram showing an example of the display content of the HUD device in the suppressed state shown in FIG. Fig. 11B The displayed contents are as follows: Fig. 11C As shown in the symbols 301 and 302, furthermore, the simplified Fig. 11B That is, the two images VDa1 and VDb are simplified by not coloring them and reducing their size. Fig. 11C In the display, a video VDm2 such as a logo is displayed to notify the user 6 that the state is the suppression state ST3, in other words, the suppression period.

[0104] As an example of specific processing, the control unit 20 stores a suppression table that specifies the correspondence between the types and priorities of the vehicle information 4 in the memory 17 in advance. For example, in the suppression transition state ST2, the control unit 20 selects the vehicle information 4 in order of priority from low to high according to the suppression table, and increases the number of selections in stages. In addition, the control unit 20 determines that the image VD representing the selected vehicle information 4 is not displayed, or is simplified by a predetermined method.

[0105] Here, for example, the priority in the suppression table is determined according to the following criteria. First, the higher the contribution of vehicle information 4 to safe driving, the higher the priority. In addition, when the vehicle information 4 indicates a warning and the object of the warning is a person and a vehicle, the priority of the person is increased. However, at this time, the priority may be weighted according to the distance between the vehicle and the object. For example, when the distance to the person is far and the distance to the vehicle is very close, the vehicle may be given priority. In addition, the priority of a person who is getting closer to the vehicle may be increased compared to a person who is getting farther and farther away from the vehicle.

[0106] In addition, about Fig. 11B as well as Fig. 11CThe image VDb showing the direction of travel shown in the figure may, for example, be lowered in priority when continuing in a straight line, and may be raised in priority when the timing of the left or right turns is approaching, based on navigation information. In addition, regarding the image VDb, it may be lowered in priority when traveling on a frequently used road, and may be raised in priority when traveling on a road that has not been used in the past, based on past driving history. Fig.11A as well as Fig. 11B The image VDd showing the road sign, that is, the speed limit, may also change its priority according to the difference from the traveling speed of the own vehicle.

[0107] In addition, the control unit 20 may not completely delete the image VD, but may reduce the processing load by, for example, displaying the image VD once every multiple control cycles Tc. Furthermore, the control unit 20 is not limited to such a method of reducing the amount of image data. For example, the control unit 20 may also simplify Figure 6 The distortion correction processing in step S124 in the image processing may even be reduced in accuracy to reduce the processing load.

[0108] <Detailed Operation of Control Unit>

[0109] Fig.12 It is shown Figure 3A The control unit 20 is a flowchart of an example of the processing content of the control unit shown in FIG. Figure 2A When the adjustment of the setting angle of the reflector M1 shown in the figure is completed and the environment capable of projecting an image is ready, execute Fig.12 The process shown in Fig.12 In step S20 , the control unit 20 starts monitoring the preparation time Tp using a timer or the like (step S20 ). Next, the control unit 20 waits for a start trigger to be generated (step S21 ). The start trigger is generated for each control cycle Tc.

[0110] When the start trigger is generated in step S21, the control unit 20 uses the communication unit 16, that is, the information acquisition unit, to acquire information related to the vehicle (step S22). Fig. 9 The current internal state described in (step S23). Then, the control unit 20 performs the Figure 6 As described in step S12 of FIG. 1 , the image data is prepared (step S24). At this time, the control unit 20 is in an internal state other than the normal state ST0, such as in Fig. 9 , Fig. 11B as well as Fig. 11C As described in the above, control is performed to reduce the processing load required for preparing image data.

[0111] After the preparation processing of the image data in step S24 is completed, the control unit 20 generates a preparation completion signal (step S25). For example, the control unit 20 outputs a display start command to the display driver 15 according to a start trigger after the preparation completion signal is generated. The display driver 15 performs the display start command according to the display start command. Figure 6 Step S13 in Figure 7 The display process of the video VD described in . In addition, when the preparation process of the video data is completed, the control unit 20 evaluates the preparation time Tp as the monitoring result accompanying step S21 (step S26 ).

[0112] Then, the control unit 20 determines whether it is necessary to perform the preparation time Tp according to the monitoring result. Fig. 9 ) (step S27). If state transition is required (step S27: "Yes"), the control unit 20 determines the transition target (step S29), causes the state to transition to the determined transition target, and updates the internal state at the same time (step S30). On the other hand, if state transition is not required (step S27: "No"), the control unit 20 returns to step S21, waits for the next start trigger, and repeats the same process until a request to end the HUD display is generated (step S28).

[0113] <Main Effects of Embodiment 1>

[0114] As described above, in the method of embodiment 1, by performing control to reduce the processing load required for the preparation of image data in a manner that completes the preparation of image data within the control period Tc, it is possible to suppress the occurrence of frame loss, etc., and to ensure the minimum display quality. In detail, even in the case where the HUD device 1 is designed in a manner that does not cause frame loss, etc., there is a possibility that frame loss, etc. will occur due to an increase in processing load due to software updates accompanying changes in display specifications. When the method of embodiment 1 is used, even in such a case, the occurrence of frame loss, etc. can be suppressed. As a result, the required display specifications can be met as much as possible within the range of the hardware's processing performance, and various display specifications can be flexibly responded to. (Implementation method 2)

[0115] The method of the first embodiment is a method of reducing the processing load by actually monitoring the preparation time Tp required for the preparation of the video data or image data and completing the preparation of the video data or image data within the control cycle Tc, in other words, the processing cycle. In the second embodiment, a method of reducing the processing load by predicting the preparation time Tp and completing the preparation of the video data or image data within the control cycle Tc, in other words, the processing cycle is described. The following description uses video data.

[0116] <About image data preparation>

[0117] Fig.13A It is shown in Figure 3A as well as Figure 3B FIG. 1 is a schematic diagram showing an example of the relationship between image data written into a frame buffer and display contents displayed on an image display unit in the HUD device shown. Fig. 13B is shown with Fig.13A A diagram of an example of different relationships. Figure 6 As described above, in the preparation process of the image data (step S12 ), the image data based on the determined display content is written to the frame buffer 18 (steps S122 , S123 ).

[0118] exist Fig.13A In the example, the six images VDa1 to VDa3, VDb, VDc, and VDe are written as image data to the storage area in the frame buffer 18 corresponding to the position to be displayed. Figure 5 As described in , images VDa1 to VDa3 are warning images superimposed on detected objects. Image VDb indicates the direction of travel of the vehicle, image VDc indicates navigation information, and image VDe indicates the speed information of the vehicle. In addition, for example, images VDa1 to VDa3 and VDb are 3D graphics, and images VDc and VDe are 2D graphics.

[0119] Here, in Fig.13A In the example shown, the size of the frame buffer 18 is equal to the size of the image display unit 11. In this case, on the image display unit 11, each image VD written in the frame buffer 18, in other words, each image object VD is displayed at the original position. Fig. 13B In the example shown, the size of the frame buffer 18 is larger than the size of the video display unit 11. In this case, a part of each video VD written in the frame buffer 18 is displayed on the video display unit 11.

[0120] exist Fig. 13B In order to cope with the case where the display moves up and down due to pitch correction, for example, the image VD is also arranged in the area outside the display area of ​​the image display unit 11 in the frame buffer 18. In this case, the size of the processing load required for the preparation process is determined according to how many images VD are arranged in the frame buffer 18 as a whole, including the area outside the display area of ​​the image display unit 11.

[0121] Fig.14 It is shown in Figure 6FIG. 1 is a diagram showing an example of items that affect the preparation time of image data. As items that affect the preparation time, the number of image VDs and the display form of each image VD can be cited. As for the number of image VDs, the larger the number, the greater the processing load becomes, and even the preparation time Tp becomes longer. In the display form of each image VD, as items, for example, size, display position, design category, distortion correction, etc. are included.

[0122] Regarding the size, the larger the size, the longer the preparation time Tp becomes. Regarding the display position, generally, the closer the display position is to the front, in other words, the closer it is, the larger the size becomes, so the preparation time Tp becomes longer. The design category includes, for example, categories such as 2D / 3D graphics, categories such as with / without layers, etc. For example, the more complex the design, such as when using 3D graphics with a large number of polygons and layers, the longer the preparation time Tp becomes. In addition, regarding the preparation time Tp associated with distortion correction, as in Figure 6 It is described that it becomes shorter when distortion correction is performed using hardware, and becomes longer when distortion correction is performed using software.

[0123] Fig.15 1 is a flowchart showing an example of a process when displaying an image in the HUD device of the second embodiment. Figure 3A or Figure 3B The structure shown is implemented. Fig.15 In the figure, it is shown that Figure 6 The same process as in the case of Figure 6 Similarly, the control unit 20 determines display content based on the information related to transportation obtained by the communication unit 16, i.e., the information acquisition unit, prepares image data based on the determined display content, and displays an image based on the prepared image data on the image display unit 11.

[0124] However, in Fig.15 In, with Figure 6 The control unit 20 determines the display content after appropriately adjusting the display content. Fig.15 In the process of preparing the image data (step S12A), the contents of the image data preparation process (step S12A) are the same as those of the Figure 6 The situation is slightly different. That is, in Fig.15 In step S12A shown in FIG. 1 , the control unit 20 first determines the display content by performing a display content adjustment process (step S41A), and then creates image data based on the adjusted display content (step S42A). Figure 6 Similarly to the case of , the control unit 20 writes the created video data into the frame buffer 18 (step S123A), and performs distortion correction using hardware processing or software processing (step S124A).

[0125] Fig.16 It is shown Fig.15 The flowchart is a flowchart of an example of the detailed processing contents of the adjustment process of the display content (step S41A) in FIG. 2. As will be described in detail later, in the control unit 20, as a premise, the predicted time information that specifies the relationship between the display content, such as the difference in the display content, and the predicted time required for the preparation of the image data is stored in the memory 17 in advance. Under this premise, first, the control unit 20 performs the adjustment process according to the Fig.15 The control unit 20 temporarily determines the display content before determining the display content based on the information related to the transportation tool obtained in step S11 (step S411). Next, the control unit 20 predicts the preparation time Tp required for preparing the image data based on the temporarily determined display content (also referred to as temporary display content) based on the predicted time information (step S412).

[0126] Next, the control unit 20 determines whether the preparation time Tp predicted in step S412 is shorter than the predetermined control cycle Tc, in other words, the processing cycle (step S413). In the case where the preparation time Tp is longer than the control cycle Tc (step S413: "No" case), the control unit 20 returns to step S412 after changing the temporarily determined display content and repeats the same process (step S414). Thus, the control unit 20 changes the temporarily determined display content, that is, the temporary display content, so that the preparation time Tp is shorter than the control cycle Tc. On the other hand, in the case where the preparation time Tp is shorter than the control cycle Tc (step S413: "Yes" case), the control unit 20 determines the temporarily determined display content, that is, the current temporary display content, as the final display content (step S415).

[0127] <Prediction method of preparation time>

[0128] Fig.17A It is shown Fig.16 FIG. 4 is a diagram showing a specific example of the method for predicting the preparation time (step S412 ). Fig. 17B is shown with Fig.17A Figures of different specific examples. Fig.17A as well as Fig. 17B As shown, the control unit 20 pre-stores predicted time information 400 that specifies the relationship between the difference in display content and the predicted time required for preparing image data. Specifically, the predicted time information 400 includes a basic time (0.8 [msec] in this example) and Fig.14 The coefficients corresponding to the items included in each image display format are shown.

[0129] That is, the prediction time information 400 includes a size coefficient C1, a display position coefficient C2, and a design category coefficient C3. The design category coefficient C3 includes a polygon number coefficient C31 and a layer coefficient C32. The size coefficient C1 is a coefficient proportional to the size of the image VD. For example, for Fig.13A The size coefficient C1 is fixedly determined for each type of image VD such as the image VDa1 indicating a warning, the image VDb indicating a traveling direction, and the image VDc indicating navigation information.

[0130] The display position coefficient C2 is determined to be larger as the display position of the image VD is closer. In this example, the display position is divided into three stages: near, middle, and far. The display position coefficient C2 is determined to be 1.5 when the display position is near, 1.0 when it is middle, and 0.5 when it is near. The polygon number coefficient C31 is determined to be larger as the number of polygons when drawing the image VD increases. In this example, the number of polygons is divided into two stages: more than the reference value and less than the reference value, that is, more and standard. The polygon number coefficient C31 is determined to be 1.2 when the number of polygons is more, and 1.0 when it is standard.

[0131] The hierarchy coefficient C32 is a coefficient corresponding to whether or not there is a hierarchy. In this example, the hierarchy coefficient C32 is determined to be 1.1 when there is a hierarchy, and is determined to be 1.0 when there is no hierarchy. Fig.15 The basic time and the values ​​of each coefficient are determined based on the total time of the image data creation process in step S42A, that is, the time required for the rendering process, the time required for the writing process to the frame buffer 18 in step S123A, and the time required for the distortion correction in step S124A. As a specific determination method, for example, a method based on simulation and a method based on actual measured values ​​can be cited.

[0132] The control unit 20 uses the predicted time information 400 to multiply the basic time by each coefficient to predict the preparation time Tp. Specifically, the control unit 20 calculates the predicted time Tr required for the preparation of the image data by "basic time × C1 × C2 × C31 × C32" for each image VD. In addition, the calculation method of the predicted time Tr is not limited to this, and it can also be calculated by "basic time × C1 × C2 × C31" depending on the situation. In addition, when there are other parameters that affect the predicted time Tr, the predicted time Tr can also be calculated by considering the parameters.

[0133] Furthermore, when calculating the predicted time Tr, it is also possible not to calculate it based on the basic time and the coefficient each time, but to store information indicating how much time is required for preparation when displaying which image with what size, at what position and with what design in advance as the predicted time information. In this case, it is necessary to store the time required for preparation in advance for each image VD, but it is not necessary to calculate the predicted time Tr each time. The control unit 20 predicts the preparation time Tp by summing up the predicted time Tr for each image VD calculated by the above method.

[0134] As a specific example, suppose that Fig.16 In step S411 shown in FIG. Fig.17A The three images VD1 to VD3 shown in the figure temporarily determine the display content. Image VD1 is an image with a size coefficient C1 of 5, and is displayed in layers using 3D graphics at a near display position. Image VD2 is an image with a size coefficient C1 of 2, and is displayed in a middle display position using 2D graphics. Image VD3 is an image with a size coefficient C1 of 10, and is displayed in layers using 2D graphics at a far display position.

[0135] In this case, the control unit 20 calculates the predicted time Tr required for preparing the image data of the image VD1 by 7.9 [msec] = 0.8 × C1 (= 5) × C2 (= 1.5) × C31 (= 1.2) × C32 (= 1.1). Similarly, the control unit 20 calculates the predicted time Tr required for preparing the image data of the image VD2 as 1.6 [msec], and the predicted time Tr required for preparing the image data of the image VD3 as 4.4 [msec].

[0136] Furthermore, the control unit 20 calculates the predicted time Tr of each of the images VD1 to VD3 and predicts the preparation time Tp to be 13.9 [msec] = 7.9 + 1.6 + 4.4. For example, when the control cycle Tc is 16.6 ms, the predicted preparation time Tp is shorter than the control cycle Tc. Therefore, the control unit 20 Fig.16 In step S415 shown, three images VD1 to VD3 having the original display format are determined as display contents.

[0137] In addition, as another specific example, it is assumed that Fig. 17B As shown, except Fig.17A In addition to the three images VD1 to VD3 shown, two images VD4 and VD5 are displayed to temporarily determine the display content. Image VD4 is an image with a size coefficient C1 of 8 and is displayed using 2D graphics at a middle display position. Image VD5 is an image with a size coefficient C1 of 2 and is displayed using 3D graphics at a close display position.

[0138] In this case, the control unit 20 calculates the predicted time Tr required for the preparation of the image data of the image VD4 by 6.4 [msec] = 0.8 × C1 (= 8) × C2 (= 1.0) × C31 (= 1.0) × C32 (= 1.0). Similarly, the control unit 20 calculates the predicted time Tr required for the preparation of the image data of the image VD5 as 2.9 [msec]. Furthermore, the control unit 20 adds up the calculated predicted time Tr for each of the images VD1 to VD5 and predicts the preparation time Tp to be 23.2 [msec]. The predicted preparation time Tp is longer than the control cycle Tc. Therefore, the control unit 20 Fig.16 In step S414 shown, the display content, that is, the temporarily determined display content, is changed.

[0139] <How to change displayed content>

[0140] Fig.18 Yes Description Fig.16 FIG. 2 is a diagram showing an example of a method for changing the display content in step S414. The control unit 20 changes the number of images VD or changes the display form of each image VD, that is, changes at least one of the size, display position, and design type, so that the preparation time Tp is shorter than the control cycle Tc. At this time, the control unit 20, for example, Fig.18 As shown, priority information 410 for determining the priority of each item to be changed is stored in advance in the memory 17, etc. Then, the control unit 20 adds items to be changed based on the priority information 410 and changes the display content until the preparation time Tp becomes shorter than the control cycle Tc.

[0141] exist Fig.18 In the priority information 410 shown, the number of images VD is set to a lower priority than the display format of each image VD. Fig.18 In the example shown, from high to low, they are design category, size, display position, and distortion correction. The control unit 20 first changes the design category of each image VD according to the priority information 410. Specifically, the control unit 20 changes 3D graphics to 2D graphics, and changes layered graphics to non-layered graphics.

[0142] Therefore, in Fig.16In the case where the newly predicted preparation time Tp is shorter than the control cycle Tc (steps S412 and S413), the control unit 20 determines the display content using the changed image VD (step S415). On the other hand, in the case where the newly predicted preparation time Tp is still longer than the control cycle Tc (steps S412 and S413), the control unit 20 reduces the size of each image VD in addition to the design category of each image VD. Specifically, for example, the control unit 20 determines a lower limit value of the size for each type of the image VD in advance, and sequentially reduces the size of each image VD until the lower limit value is reached.

[0143] As a specific example, in Fig. 17B it is assumed that the lower limit value of the size of the image VD3, specifically, the lower limit value of the size coefficient C1, is set to 8. In this case, the control unit 20 first changes the size of the image VD3 to a size where the size coefficient C1 is 9, and if Tp > Tc still holds after that, changes it to a size where the size coefficient C1 is 8. In addition, at this time, depending on the type of the image VD, an image VD for which changing the size coefficient C1 is not permitted, that is, for which the lower limit value is not set, may also be set.

[0144] Subsequently, in the same manner, the control unit 20 adds the display position and the distortion correction to the items to be changed in order according to the priority information 410 until Tp < Tc is achieved. When the control unit 20 changes the display position, it reduces the size by moving the display position inward. At this time, depending on the type of the image VD, an image VD for which changing the display position is not permitted may also be set. In addition, when the control unit 20 changes the item of the distortion correction, for example, it applies a pre-determined simplified distortion correction or does not perform the distortion correction itself.

[0145] Even if the display form of each image VD is changed in this way and Tp > Tc still holds, as a last resort, the control unit 20 reduces the number of image VDs. At this time, for example, the control unit 20 determines which image VD to sequentially cut from according to the suppression table described in Fig. 11C As described above, for the vehicle information 4 that contributes more to safe driving, that is, the type of the image VD, a higher priority is set to avoid being a deletion target.

[0146] Fig.19 is a schematic diagram showing an example of the display content of the HUD device obtained as a result of changing the display content in Fig.18 In the upper part of Fig.19 it shows, as the display content before the change, the content related to Fig.11AThe same display content as in the case of . That is, 7 images VDa1~VDa3, VDb~VDe are displayed here. Images VDa1 and VDa2 are displayed in a manner overlapping with objects OB1 and OB2 (here, people), respectively. Image VDa3 is displayed in a manner overlapping with object OB3 (here, a vehicle). Images VDb, VDc, VDd, and VDe respectively indicate the direction of travel, navigation information, road signs, and speed information. It is assumed that 3D graphics are used and there are layers in images VDa1~VDa3.

[0147] exist Fig.19 The lower part of the figure shows the display content after the change. In the display content after the change, compared with the display content before the change, the images VDa1 to VDa3 are replaced by images VDa1x to VDa3x respectively. In the images VDa1x to VDa3x, 2D graphics are used and there is no layering. Furthermore, the size of the images VDa1x to VDa3x is slightly smaller than that of the images VDa1 to VDa3.

[0148] Furthermore, in Fig.19 In the example shown, even when the images VDa1x to VDa3x are used, the preparation time Tp is still longer than the control cycle Tc, so the image VDd representing the speed information is deleted as shown in the portion of the symbol 405. That is, in this example, it is assumed that the image VDd representing the speed information contributes less to safe driving than the other images VD, and the image VDd is deleted. By using such a change method, from the perspective of safe driving, the display content can be changed in a manner that does not make the driver or other users feel a great sense of disobedience while maintaining the display content before the change as much as possible.

[0149] Fig. 20 is used Fig.15 as well as Fig.16 The process shown in the figure is used to display the timing diagram of the image. Fig. 20 In, with Figure 7 Similarly, the actions performed in the 1st to 4th processing cycles or control cycles Tc[1] to Tc[4] are shown. The control cycle Tc is, for example, 16.6 ms. Fig. 20 As shown, assuming that the display content is not changed, for example, the preparation time Tp[2a] in the control cycle Tc[2] may be longer than the control cycle Tc[2]. Figure 7 As described in , frame loss occurs.

[0150] On the other hand, when changing the display content, for example, the preparation time Tp[2b] in the control cycle Tc[2] can be made shorter than the control cycle Tc[2]. As a result, frame loss can be prevented. Fig.15 as well as Fig.16 The time required for the adjustment process of the display content shown (step S41A) is shown. Therefore, the time required for this step S41A becomes the overhead time within the preparation time Tp, but is usually sufficiently small compared with the time required for steps S42, S123, and S124, and is considered to be at a negligible level.

[0151] In the above-mentioned first embodiment, since the actual preparation time Tp is monitored and control, for example, feedback control, is performed based on the monitoring result, there is a relationship that the instantaneous generation of the Fig. 20 On the other hand, in the method of the second embodiment, the preparation time Tp is predicted to perform control, for example, feedforward control, so ideally, there will be no such situation. Fig. 20 The situation of the preparation time Tp[2a] is shown.

[0152] <Main Effects of Embodiment 2>

[0153] As described above, by using the method of embodiment 2, the same effects as those described in embodiment 1 can be obtained. That is, frame loss can be suppressed or prevented, and the minimum display quality can be ensured. Moreover, the required display specifications can be met as much as possible within the range of the hardware processing performance, and various display specifications can be flexibly responded to. Furthermore, by predicting the preparation time Tp, it is less likely to cause excessive changes or suppression of the display content compared to the method of embodiment 1, so the display quality can be further improved.

[0154] In addition, the method of embodiment 1 has a lot of variable factors in display specifications such as initial products, and it is difficult to produce such Fig.17A as well as Fig. 17B On the other hand, the method of the second embodiment is particularly useful when there are few variable elements in the display specifications, such as a product that is mature to a certain extent. In addition, in the method of the second embodiment, it is necessary to prepare the predicted time information 400, so from the perspective of flexibility or versatility when dealing with display specifications that vary greatly, the method of the first embodiment may be more useful. (Implementation method 3)

[0155] <Details of the control unit>

[0156] Fig.21A is a schematic diagram showing an example of main processing contents of the control unit in the HUD device according to the third embodiment. Fig. 21B yes Fig.21A Supplementary Figure. Fig.22A is shown with Fig.21A A different example diagram, Fig. 22B yes Fig.22A The HUD device 1 of the third embodiment is as follows: Figure 3A or Figure 3B The control unit 20 is implemented as shown in the structure. Fig.21A Image VDa3a and Fig.22A As shown in the image VDa3b in FIG. 3 , the warning image is overlapped on the object OB3 (here, the vehicle).

[0157] At this time, the control unit 20 determines the color or shape of the images VDa3a and VDa3b as warning images according to the distance from the object OB3. Fig.21A as well as Fig. 21B In the example of FIG. 4 , the distance between the vehicle 420 equipped with the HUD device 1 and the object OB3 in front of the vehicle 420 is 50 m. In this case, the control unit 20 determines the color of the image VDa3a to be green, for example. Fig.22A as well as Fig. 22B In the example of , the distance between the vehicle 420 and the object OB3 is 10 m. In this case, the control unit 20 determines the color of the image VDa3b to be red, for example.

[0158] Fig.23 FIG. 2 is a diagram showing a configuration example of a display setting table stored in the control unit in the HUD device of the third embodiment. Fig.23 The display setting table 415 shown is pre-stored in the memory 17 or the like. The display setting table 415 specifies the correspondence between the distance to the object OB3 ahead and the color or shape (here, color) of the warning image. In this example, the color of the warning image is determined to be red when the distance to the object OB3 is less than 12m, green when the distance is greater than 32m, and yellow when the distance is in the range of 17m to 27m. In addition, the colors in the range of 12m to 17m and the range of 27m to 32m, which are intermittent distances, are appropriately determined to be intermediate colors.

[0159] The display setting table 415 is set, for example, so that the closer the distance to the object OB3 is, the more the color or shape reminds the user, for example, the driver, to pay attention. This can contribute to safe driving. However, the implementation method of the color or shape depends on the user's subjectivity. Therefore, it can also be configured so that the color or shape in the display setting table 415 can be arbitrarily selected from multiple options through the initial setting performed by the user. Furthermore, it can also be configured so that the range of distance can be arbitrarily set in the display setting table 415.

[0160] Fig.24AFIG. 1 is a flowchart showing an example of the processing contents of the control unit in the HUD device according to the third embodiment. Fig.24A In the embodiment 2, the control unit 20 executes Fig.16 In addition, before this process, the communication unit 16, i.e., the information acquisition unit, Fig.15 In step S11 shown in FIG. 1 , the distance between the vehicle 420 and the object OB3 existing in front of the vehicle 420 is acquired as one of the information related to the transportation means.

[0161] The control unit 20 Fig.24A The processing of steps S411 to S415 shown in the figure determines the display content in such a way that the preparation time Tp is shorter than the control cycle Tc, in other words, the processing cycle. After that, the control unit 20 determines whether the display content determined in step S415 includes a warning image superimposed on the object OB3 (step S416A). In the case where the determined display content includes a warning image (step S416A: "Yes" case), the control unit 20 refers to the distance information from the object OB3 obtained by the communication unit 16 (step S417), and updates the color or shape of the warning image according to the display setting table 415 (step S418A). In addition, in the case where multiple warning images are included, this update processing is performed for each warning image.

[0162] In this way, Fig.24A In the flow shown, when the determined display content includes a warning image, the control unit 20 updates the color or shape of the warning image (step S418A) without re-estimating the preparation time Tp, that is, without going through the process of step S412 and the like. Therefore, it is preferable to predetermine the color or shape of the warning image for each distance so that the preparation time Tp remains the same as before the update even when the color or shape of the warning image is updated.

[0163] Fig. 24B is shown with Fig.24A A flowchart showing an example of different processing contents. Fig. 24B In the process shown, Fig.24A The process shown is different and is not Fig.16 After the processing of steps S411 to S415 described in the above, the processing of steps S411 to S415 is embedded in the Fig.24A The same processing is performed as steps S416A, S417, and S418A described in .

[0164] exist Fig. 24BIn the embodiment, after the control unit 20 temporarily determines the display content (step S411), it determines whether the temporarily determined display content includes a warning image superimposed on the object OB3 (step S416B). If the temporarily determined display content includes a warning image (step S416B: "Yes"), the control unit 20 refers to the distance information to the object OB3 (step S417), and determines the color or shape of the warning image according to the display setting table 415 (step S418B).

[0165] Then, the control unit 20 estimates the preparation time Tp required for preparing the image data based on the provisionally determined display content (step S412). If the provisionally determined display content does not include a warning image (step S416B: No), the control unit 20 directly transfers to step S412. After step S412, the control unit 20 and Fig.16 Similarly, through the processing of steps S413 to S415, the provisionally determined display content is changed so that the preparation time Tp is shorter than the control cycle Tc.

[0166] However, with Fig.16 In a different situation, the control unit 20 changes the temporarily determined display content and transfers to step S416B (step S414). Therefore, even if the color or shape of the warning image is changed by the processing of step S414, it is possible to accurately restore it by the processing of steps S417 and S418B, and predict the preparation time Tp (step S412).

[0167] In use Fig. 24B In the process shown in FIG. 1 , the preparation time Tp can be predicted based on the change in color or shape reflected in the warning image, so it is similar to using Fig.24A Compared with the process shown in the figure, the prediction accuracy is sometimes improved. Fig. 24B In the process shown in FIG. 1 , each time a loop from step S414 to step S416B is generated, steps S416B, S417, and S418B need to be processed, so the processing overhead may increase. Therefore, from this point of view, using Fig.24A The process shown is more beneficial.

[0168] In addition, for example, based on Fig.24A The processing of displaying the setting table 415 described in steps S416A, S417, and S418A in the embodiment is not limited to the method of the second embodiment, but can also be applied to the method of the first embodiment. In this case, the control unit 20, for example, Figure 6 When the display content is determined based on the internal state in step S121 shown, at its final stage, the processing described in steps S416A, S417, and S418A may be executed.

[0169] <Main Effects of Implementation Method 3>

[0170] As described above, when the method of Embodiment 3 is used, in addition to the various effects described in Embodiments 1 and 2, by changing the color or shape of the warning image according to the distance to the object, it is possible to further contribute to safe driving.

[0171] In addition, when using the methods of each embodiment, the user 6 can visually recognize various information required for driving, such as warning information when detecting oncoming vehicles and pedestrians, in addition to navigation information such as destination and speed, as an image across the windshield 3, and can visually recognize an image that ensures a minimum display quality even if the display specifications change. As a result, it is possible to provide a HUD device 1 that reduces the movement of the user's 6 viewpoint and contributes to supporting safe driving. As a result, traffic accidents can be prevented. Furthermore, it is possible to contribute to "3. Guaranteeing health and welfare for all" of the sustainable development goals (SDGs: Sustainable Development Goals) advocated by the United Nations.

[0172] The invention completed by the inventor is specifically described above based on the embodiments, but the present invention is not limited to the embodiments, and various changes can be made within the scope of the gist thereof. For example, the above embodiments are examples described in detail to explain the present invention in an easy-to-understand manner, and may not have all the structures described. In addition, a part of the structure of a certain embodiment can be replaced with the structure of other embodiments, and the structure of other embodiments can also be added to the structure of a certain embodiment. In addition, other structures can be added, deleted, or replaced with respect to a part of the structure of each embodiment.

[0173] (Explanation of symbols)

[0174] 1: head-up display (HUD) device; 2: vehicle; 4: vehicle information; 5: display area; 6: user; 11: image display unit; 16: communication unit (information acquisition unit); 20: control unit; 400: prediction time information; 410: priority information; 415: display setting table; C1~C3, C31, C32: coefficients; M1: reflector (image light projection unit); ST0: normal state; ST1: recovery transfer state; ST2: inhibition transfer state; ST3: inhibition state; Tc: control cycle; Tp: preparation time; Tr: prediction time; Tth1: first threshold time; Tth2: second threshold time; TthD: threshold duration; VD: image.

Claims

1. A head-up display device, mounted on a vehicle, wherein: The head-up display device comprises: An information acquisition unit, which acquires information related to the transportation means; An image display unit that displays an image and emits image light of the displayed image; an image light projection unit configured to project the image light emitted from the image display unit onto a display area so that the projected image light is visually recognized as a virtual image; as well as a control unit that prepares image data based on the information related to the transportation equipment acquired by the information acquisition unit, and displays an image based on the prepared image data on the image display unit, The control unit performs control to reduce a processing load required for preparing the image data so that preparation of the image data is completed within a predetermined processing cycle.

2. The head-up display device according to claim 1, wherein: The control unit monitors a preparation time required for preparing the image data, and starts control to reduce the processing load when the preparation time satisfies a predetermined condition.

3. The head-up display device according to claim 2, wherein: The control unit starts control to reduce the processing load when the preparation time is longer than the processing cycle.

4. The head-up display device according to claim 2, wherein: The control unit uses a first threshold time shorter than the processing cycle, and starts control to reduce the processing load when the preparation time is longer than the first threshold time for a plurality of consecutive times.

5. The head-up display device according to claim 2, wherein: After starting control to reduce the processing load, the control unit increases the amount of reduction in the processing load stepwise for each processing cycle during a period of suppressing the transition.

6. The head-up display device according to claim 5, wherein: The control unit reduces the amount of reduction in the processing load stepwise for each processing cycle during a recovery transition period after the transition suppression period has elapsed.

7. The head-up display device according to claim 6, wherein: The control unit performs control in the recovery transition period after the suppression transition period has elapsed, via a suppression period for preparing the image data with the reduced processing load.

8. The head-up display device according to claim 7, wherein: During the suppression period, the control unit uses a second threshold time shorter than the processing cycle and a threshold duration period that is a multiple of the processing cycle or more, and starts control during the recovery transfer period when the preparation time is shorter than the second threshold time for more than the threshold duration period.

9. The head-up display device according to claim 5, wherein: The control unit generates the video data for notifying a user that the transition is suppressed during the transition suppression period.

10. The head-up display device according to claim 1, wherein: The control unit reduces the processing load by reducing the amount of the image data to be prepared.

11. A head-up display device, mounted on a vehicle, wherein: The head-up display device comprises: An information acquisition unit, which acquires information related to the transportation means; an image display unit that displays an image and emits image light of the displayed image; and an image light projection unit that projects the image light emitted from the image display unit onto a display area so that the projected image light is visually recognized as a virtual image, Image data is prepared based on the information related to the transportation facility acquired by the information acquisition unit, and when a preparation condition of the image data does not satisfy a predetermined condition in a predetermined processing cycle, a part of the content of the image data to be prepared is changed.

12. A method for processing image data for a head-up display device mounted on a vehicle, wherein: obtaining information related to the transportation means, preparing image data based on the acquired information related to the transportation means, and displaying an image based on the prepared image data on an image display unit of the head-up display device, When preparing the image data, control is performed to reduce a processing load required for preparing the image data so that preparation of the image data is completed within a predetermined processing cycle.

13. The method for processing image data according to claim 12, wherein: The preparation time required for preparing the image data is monitored, and when the preparation time satisfies a predetermined condition, control to reduce the processing load is started.

14. The method for processing image data according to claim 13, wherein: When the preparation time is longer than the processing cycle, control for reducing the processing load is started.

15. The method for processing image data according to claim 13, wherein: Using a first threshold time shorter than the processing cycle, control to reduce the processing load is started when the preparation time is longer than the first threshold time for a plurality of consecutive times.

16. The method for processing image data according to claim 13, wherein: After the control for reducing the processing load is started, the amount of reduction in the processing load is increased stepwise for each processing cycle during a period of suppressing the transition.

17. The method for processing image data according to claim 16, wherein: After the transition suppression period has elapsed, the amount of reduction in the processing load is reduced stepwise for each of the processing cycles during the transition recovery period.

18. A head-up display device, mounted on a vehicle, wherein: The head-up display device comprises: An information acquisition unit, which acquires information related to the transportation means; An image display unit that displays an image and emits image light of the displayed image; an image light projection unit configured to project the image light emitted from the image display unit onto a display area so that the projected image light is visually recognized as a virtual image; as well as a control unit that determines display content based on the information related to the transportation tool acquired by the information acquisition unit, prepares image data based on the determined display content, and causes the image display unit to display an image based on the prepared image data, The control unit changes the provisionally determined display content when a preparation time required for preparation of the video data of the provisionally determined display content before determining the display content is longer than a predetermined processing cycle.

19. The head-up display device according to claim 18, wherein: Before determining the display content, the control unit predicts the preparation time required for preparing the image data based on the temporarily determined display content based on predicted time information that specifies the relationship between the display content and the predicted time required for preparing the image data, and, if the preparation time is longer than the processing cycle, changes the temporarily determined display content in a manner shorter than the processing cycle.

20. The head-up display device according to claim 19, wherein: The items for changing the temporarily determined display content include the number of images, or the display format of each image, The items for changing the display form of each of the images include at least one of size, display position, and design type.

21. The head-up display device according to claim 20, wherein: The control unit stores priority information that specifies the priority of each of the items in advance, and changes the provisionally determined display content while adding the items to be changed according to the priority information until the preparation time becomes shorter than the processing cycle.

22. The head-up display device according to claim 21, wherein: The items for changing the temporarily determined display content include the number of images and the display format of each image. In the priority information, the number of the images is set to a lower priority than the display form of each image.

23. The head-up display device according to claim 20, wherein: The predicted time information includes a basic time and a coefficient corresponding to the item included in the display form of each image. The control unit predicts the preparation time by multiplying the basic time by the coefficient.

24. The head-up display device according to claim 18, wherein: The information acquisition unit acquires a distance between the vehicle on which the head-up display device is mounted and an object existing in front of the vehicle. When the control unit superimposes the warning image on the object, the control unit determines a color or a shape of the warning image according to a distance from the object.

25. The head-up display device according to claim 24, wherein: After determining the display content so that the preparation time becomes shorter than the processing cycle, the control unit updates the color or shape of the warning image without re-estimating the preparation time when the determined display content includes the warning image.

26. A method for processing image data for a head-up display device mounted on a vehicle, wherein: obtaining information related to the transportation means, determining display contents based on the acquired information related to the transportation means, preparing image data based on the determined display contents, and causing an image based on the prepared image data to be displayed on an image display unit of the head-up display device, When the preparation time required for the preparation of the video data of the provisionally determined display content before the display content is determined is longer than a predetermined processing cycle, the provisionally determined display content is changed.

27. The method for processing image data according to claim 26, wherein: Before determining the display content, the preparation time required for preparing the image data based on the temporarily determined display content is predicted based on predicted time information that specifies the relationship between the display content and the predicted time required for preparing the image data. If the preparation time is longer than a predetermined processing cycle, the temporarily determined display content is changed in a manner shorter than the processing cycle.

28. The method for processing image data according to claim 27, wherein: The items for changing the temporarily determined display content include the number of images, or the display format of each image, The items for changing the display form of each of the images include at least one of size, display position, and design type.

29. The method for processing image data according to claim 28, wherein: Priority information defining the priority of each of the items is stored in advance, and the items to be changed are added according to the priority information, and the provisionally determined display content is changed until the preparation time becomes shorter than the processing cycle.

30. The method for processing image data according to claim 29, wherein: The items for changing the temporarily determined display content include the number of images and the display format of each image. In the priority information, the number of the images is set to a lower priority than the display form of each image.

31. The method for processing image data according to claim 28, wherein: The predicted time information includes a basic time and a coefficient corresponding to the item included in the display form of each image. The preparation time is predicted by multiplying the basic time by the coefficient.

32. The method for processing image data according to claim 26, wherein: obtaining a distance between the vehicle equipped with the head-up display device and an object existing in front of the vehicle, When the warning image is superimposed on the object, the color or shape of the warning image is determined according to the distance from the object.

33. The method for processing image data according to claim 32, wherein: After the display content is determined so that the preparation time becomes shorter than the processing cycle, if the warning image is included in the determined display content, the color or shape of the warning image is updated without re-estimating the preparation time.

Images