Display control device

The display control device for HUDs addresses the lack of effective hazard notification by sequentially displaying non-superimposed and superimposed content, enhancing user awareness and safety by providing clear advance warnings and reducing visual complexity.

WO2024232428A1PCT designated stage expired Publication Date: 2024-11-14NIPPON SEIKI CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/017380
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-10
Filing Date
2024-05-10
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Existing head-up display (HUD) technologies do not effectively notify users of dangerous situations in a way that is easy to understand, particularly during driving, which poses a safety concern as they do not provide adequate advance warning or clear indication of potential hazards.

Method used

A display control device that sequentially displays non-superimposed and superimposed content on a HUD, with non-superimposed content providing advance notification of potential dangers, allowing users to prepare for hazardous situations by distributing information and reducing visual burden through controlled timing and positioning of display elements.

Benefits of technology

Enables users to understand and prepare for dangerous situations more effectively by providing clear, advance warnings and reducing the visual and mental burden associated with processing multiple information sources during driving.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024017380_14112024_PF_FP_ABST
    Figure JP2024017380_14112024_PF_FP_ABST
Patent Text Reader

Abstract

The present invention makes it possible to, for example, prepare for danger by notifying a user of a dangerous situation in advance and in an easily-understandable format. Provided is a display control device 80 which controls a display device (display unit 90) for displaying content so as to be superimposed on the actual view and which has: a display information generation unit 83 that generates first content (superimposed content) to be superimposed and displayed on a specific superimposition target present in the actual view and second content (non-superimposed content) to be superimposed and displayed on the actual view without being superimposed on the superimposition target; and a display control unit 84 that displays the first content and the second content sequentially on the display device.
Need to check novelty before this filing date? Find Prior Art

Description

Display Control Device

[0001] The present invention relates to a display control device suitable for use in a head-up display device or the like mounted on a vehicle.

[0002] 2. Description of the Related Art A head-up display (HUD) device is known as a display device mounted on a vehicle, which displays a virtual image by emitting display light onto a projection target member such as a windshield.

[0003] For example, Patent Document 1 describes a display control device used in a vehicle that controls display by a HUD, and includes a display generation unit that generates superimposed content that is displayed superimposed on a specific superimposed object in the foreground of the vehicle, and non-superimposed content that is displayed superimposed on the foreground without being superimposed on the superimposed object, and a target position grasping unit that grasps the distance to a specific position related to the superimposed object or the time required to reach the specific position, in which, prior to displaying the superimposed content, the display generation unit displays non-superimposed content that has substantially the same meaning as the superimposed content, and, when the distance or time grasped by the target position grasping unit falls below a threshold, superimposes and displays the superimposed content on the superimposed object.

[0004] JP 2022-58537 A (paragraph

[0006] )

[0005] According to the technology described in the aforementioned Patent Document 1, the superimposed display on the HUD is changed from non-superimposed content having substantially the same meaning to superimposed content, which makes it less likely for the user to misunderstand the change even if information presentation is started early using a HUD with a limited angle of view, thereby enabling highly convenient information presentation.

[0006] However, the inventors recognized that the technology described in Patent Document 1 is insufficient, for example, in terms of notifying a user of a dangerous driving situation in a manner that is easy for the user to understand. Therefore, there is a safety issue of notifying a user of a dangerous situation in advance in a manner that is easy for the user to understand, so that the user can prepare for the danger. Patent Document 1 does not mention this issue at all, nor does it suggest any countermeasures.

[0007] The present invention has been made to solve the above-mentioned problems, and aims to provide a display control device that can, for example, notify a user of dangerous situations in advance in an easily understandable manner, allowing the user to prepare for danger.

[0008] Other objects of the present invention will become apparent to those skilled in the art by referring to the following exemplary aspects and best modes, as well as the accompanying drawings.

[0009] In order to facilitate an understanding of the outline of the present invention, the following examples are given of embodiments according to the present invention.

[0010] In a first aspect, a display control device controls a display device that displays content superimposed on a real scene, and includes a display information generation unit that generates first content that is superimposed on a specific superimposed object present in the real scene, and second content that is superimposed on the real scene without being superimposed on the superimposed object, and a display control unit that sequentially displays the first content and the second content on the display device.

[0011] Here, the "superimposed object" refers to a specific object in the foreground, such as a road surface, a vehicle ahead, a pedestrian, or a road sign. The "first content" refers to a highly conspicuous AR display object (virtual image) using augmented reality (AR), and is superimposed content whose display position is associated with a specific superimposed object in the foreground. The "second content" refers to an AR display object superimposed on the foreground, excluding the first content, and is non-superimposed content that, unlike the first content, is superimposed on the foreground in a manner that does not superimpose on the specific superimposed object and has substantially the same meaning as the first content.

[0012] According to a first aspect, the display control unit sequentially displays on the display device a first content that is superimposed on a specific superimposition object present in the real scene and a second content that is superimposed on the real scene without being superimposed on the superimposition object. In other words, the amount of information provided to the user is distributed between the first content and the second content, and the first content and the second content are displayed in stages. This makes it possible, for example, to notify the user of a dangerous driving situation in an easy-to-understand manner with minimal burden, thereby enabling the user to prepare for the danger.

[0013] In a second aspect dependent on the first aspect, the display control unit may display the second content and then display the first content.

[0014] According to the second aspect, the first content is displayed after the second content is displayed. In this manner, by providing advance warning using non-superimposed content that has substantially the same meaning as the first content, rather than the highly conspicuous first content, it is possible to transmit necessary information to the user while ensuring forward visibility. Therefore, for example, by providing advance warning of a dangerous situation in an easily understandable manner to the user, it is possible to prepare for danger.

[0015] In a third aspect that is dependent on the first or second aspect, the display control unit may display the first content after displaying the second content, even if the first content can be superimposed on the superimposition object.

[0016] According to the third aspect, if the highly eye-catching first content is displayed first, the user will become less vigilant towards others, so by displaying the second content first, the user can accept information while diverting their attention to some extent.

[0017] In a fourth aspect dependent on the first to third aspects, the display control unit may display the first content further away from the user than the second content.

[0018] According to the fourth aspect, by shifting the gaze from the second content displayed close to the user to the first content displayed far away, the number of times the gaze must move back and forth can be reduced, making it easier for the user to understand and allowing for smoother viewing.

[0019] In a fifth aspect which is dependent on the first to fourth aspects, the display device may have a first display and a second display, and the display control unit may display the first content on the first display and the second content on the second display.

[0020] Here, the first display is, for example, a HUD, and the second display is, for example, a WSD (Windshield Display) that projects an image onto a black ceramic portion (black portion of the windshield) of the vehicle. According to the fifth aspect, by displaying the first content and the second content on separate displays, a larger content display area can be secured and the display can be prevented from becoming complicated.

[0021] In a sixth aspect dependent on the first to fifth aspects, the display control unit may display the first content in a first aspect and the second content in a second aspect.

[0022] According to the sixth aspect, the display control unit can provide an advance warning using the second content (non-superimposed content) in the form of text, icons, or the like, thereby allowing the user to understand, for example, the nature of the danger in advance.

[0023] In a seventh aspect which is dependent on the first to sixth aspects, the display control unit may include a relevance determination unit which determines the relevance between the first content and the second content, and when the first content displayed corresponding to a first superimposition object among the specific superimposition objects, and the second content displayed corresponding to a second superimposition object which is the specific superimposition object, and the relevance indicating the relevance is less than a predetermined value, are displayed simultaneously, the display control unit may perform control to delay the timing of starting to display the second content and / or advance the timing of starting to display the first content.

[0024] According to the seventh aspect, when a first content in a certain superimposed object and a second content in another superimposed object are displayed simultaneously, the display control unit can delay the timing at which the second content starts to be displayed, or control the timing at which the first content starts to be displayed to be earlier, or perform both, thereby preventing the user from misunderstanding that the content is related.

[0025] Here, the relevance between the first content and the second content can be determined, for example, by determining that the degree of relevance is high if the superimposed content and non-superimposed content are created for a single object monitored by a perimeter monitoring sensor. Alternatively, for example, a label can be assigned to each piece of content, and if the number of matches between the superimposed content and non-superimposed content is equal to or greater than a predetermined threshold, the degree of relevance can be determined. Alternatively, a correlation coefficient can be calculated.

[0026] Furthermore, the control of the display start timing can be realized, for example, by providing a range (timing settable range) for the display start timing setting and setting the display start timing to be earlier or later than the standard display start timing, and then controlling the rise of the blanking period (during the display period for both horizontal and vertical synchronization signals) according to that setting.

[0027] In an eighth aspect which is dependent on the first to sixth aspects, the display control unit may include a relevance determination unit which determines the relevance between the first content and the second content, and when the relevance indicating the relevance is equal to or greater than a predetermined value, the display control unit may display the first content and the second content by arranging them on a straight line connecting the superimposed object and the user's eyepoint when looking at the superimposed object from a bird's eye view.

[0028] When a user views the first content or the second content from the real scene ahead of the vehicle, the user needs to shift their eye focus (point of gaze) from the real scene in real space to the virtual image. According to the eighth aspect, when the degree of association is equal to or greater than a predetermined value, the first content and the second content are displayed on a straight line connecting the user's eyepoint when viewing the superimposed object from above (the user sees the first content and the second content in a line). This makes the user's line of sight move more naturally, making it easier to visually understand the association between the content. In other words, when the user's point of gaze switches from the real scene ahead to the virtual image, the user's line of sight is reduced as much as possible (the need for eye focus adjustment is reduced as much as possible).

[0029] In a ninth aspect dependent on the first to eighth aspects, the vehicle may have a visibility determination unit that determines the visibility of the forward field of view of the vehicle, and the display control unit may control the timing of starting to display the first content according to a visibility indicating the visibility.

[0030] For example, in a situation where a user's visibility is reduced due to a change in weather such as rain or snow, the superimposed object cannot be seen until the last moment. However, according to the ninth aspect, in such a situation, the display control unit controls the timing of the start of displaying the first content to be advanced (in other words, the timing of the information notification to be advanced), thereby reducing the mental burden on the user and enabling the user to prepare in advance for impending danger.

[0031] Here, visibility can be determined by, for example, generating feature values ​​for each part of a specific subject, comparing them with feature values ​​for the same subject under predefined conditions of high visibility, and calculating a score based on the comparison result. The visibility of a subject is highest when all parts are clearly visible, and the more obscured the subject is due to weather changes such as rain or snow, the smaller the clearly visible area becomes. Therefore, the degree to which each part of the subject is visible can be estimated from the score for each part. Visibility can be evaluated by calculating the ratio of the sum of the areas of high-scoring parts to the total area of ​​the subject as a quantity representing the degree to which the subject is visible. Furthermore, the timing of display start can be controlled by, for example, providing a range for the display start timing setting (a settable range of timing), setting an earlier or later display start timing depending on the visibility, and controlling the rising edge of the blanking period (during the display period for both horizontal and vertical synchronization signals) according to the setting.

[0032] In a tenth aspect dependent on the first to ninth aspects, the display control unit may control a timing of starting display of the first content in accordance with a display distance of the second content.

[0033] According to the tenth aspect, the display control unit determines the timing for starting display of the first content depending on the display distance of the second content, thereby making it possible to set the timing for starting display so that the eye movement from the second content to the first content does not become too large, thereby reducing the visual burden on the user (the burden of adjusting the focus of the eyes).

[0034] Here, the "display distance of the second content" refers to the distance from the user's eyepoint to the second content, and the display distance is controlled by at least one of changing the optical path length from a display unit having a backlit LCD panel or the like to the projection target (for example, the length of the optical path from a first point where the optical axis of the optical system intersects with the display surface of the display unit to a second point where it intersects with the projection target), or changing the display position of the second content on the display surface.

[0035] In an eleventh aspect that is dependent on the first to tenth aspects, the display control unit may control a display distance interval between the first content and the second content according to a display distance of the first content.

[0036] According to the eleventh aspect, the display control unit controls the display distance interval between the first content and the second content according to the display distance of the first content, thereby setting the display distance interval so that the eye movement from the second content to the first content is not too large, thereby reducing the visual burden on the user.

[0037] In a twelfth aspect which is dependent on the first to eleventh aspects, the display control unit may have a distance determination unit which determines the distance between the vehicle position and the superimposed object, and the display control unit may control the display distance interval between the first content and the second content according to the distance determined by the distance determination unit.

[0038] According to the twelfth aspect, the display control unit controls the display distance interval between the first content and the second content according to the distance between the vehicle position and the superimposed object, thereby setting the display distance interval so that the eye movement from the second content to the first content does not become too large, thereby reducing the visual burden on the user.

[0039] In a thirteenth aspect which is dependent on the first to twelfth aspects, the display control unit may include a relevance determination unit which determines the relevance between the first content and the second content, and the display control unit may adjust the display distance of the second content according to the relevance degree indicating the relevance, thereby controlling the display distance interval.

[0040] According to the thirteenth aspect, for example, assuming that the display distance of the first content is d1 and the display distance of the second content is d2, if the relevance between the first content and the second content is high, the value of d2 is increased to narrow the display distance interval, and if the relevance between the first content and the second content is low, the value of d2 is decreased to widen the display distance interval, thereby adjusting the display distance of the second content. This makes it possible to visually separate the first content and the second content that are less relevant to each other, and to realize a notification that does not mislead the user.

[0041] In a fourteenth aspect that is dependent on the first to thirteenth aspects, the display control unit may perform control such that, after the second content is displayed, the second content is not displayed until the first content is displayed.

[0042] According to the fourteenth aspect, the display control unit hides the second content after the second content is displayed until the first content is displayed, thereby reducing the amount of content occupying the display field of view and also reducing the visual burden on the user.

[0043] In a fifteenth aspect dependent on the first to fourteenth aspects, the display control unit may perform control to hide the second content after the second content and the first content are displayed.

[0044] According to the fifteenth aspect, the display control unit can reduce the amount of content occupying the display field of view by displaying the second content and the first content and then hiding the second content, thereby reducing the visual burden on the user.

[0045] Those skilled in the art will easily understand that the exemplified embodiments according to the present invention can be further modified without departing from the spirit of the present invention.

[0046] FIG. 1( a) shows an outline of an HUD mounted on a vehicle to which a display control device according to an embodiment of the present invention is applied, and FIG. 1( b) is a cross-sectional view of a main part showing its internal configuration. FIG. 2 is a block diagram showing the configuration of a display control system using a display control device according to an embodiment of the present invention. FIG. 3 is a flowchart showing the operation of a display control system to which a display control device according to an embodiment of the present invention is applied. FIG. 4 is a diagram showing a display example (1) of content generated by a display control device according to an embodiment of the present invention. FIG. 5 is a diagram showing a display example (2) of content generated by a display control device according to an embodiment of the present invention. FIG. 6 is a diagram showing a display example (3) of content generated by a display control device according to an embodiment of the present invention. FIG. 7 is a diagram showing a display example (4) of content generated by a display control device according to an embodiment of the present invention. FIG. 8 is a diagram showing a display example (5) of content generated by a display control device according to an embodiment of the present invention. FIG. 9 is a diagram showing a display example (6) of content generated by a display control device according to an embodiment of the present invention. FIG. 10 is a diagram showing a display example (7) of content generated by a display control device according to an embodiment of the present invention.

[0047] The best mode described below is used to facilitate understanding of the present invention, and therefore, those skilled in the art should be aware that the present invention is not unduly limited by the mode described below (hereinafter referred to as the present mode).

[0048] Hereinafter, as this embodiment, an example in which the present invention is applied to a HUD mounted on a vehicle (automobile) will be described with reference to the accompanying drawings.

[0049] (Configuration of the embodiment) FIG. 1A shows an outline of a HUD to which a display control device according to the embodiment is applied, which is mounted on a vehicle, and FIG. 1B shows a cross-sectional view of a main part showing the internal configuration of the HUD.

[0050] 1A, a projection display by an HUD involves reflecting display light L projected by an HUD 12, which is a display device provided inside an instrument panel 11 of a vehicle 10, onto a windshield 13 of the vehicle 10 in the direction of a user 14 of the vehicle 10 (a driver seated in the driver's seat of the vehicle 10), thereby displaying a virtual image V. In other words, the HUD 12 emits (projects) display light (image) L emitted from a liquid crystal display 20 (described later) onto the windshield 13, which is a projection target, and allows the user to view display light (virtual image) V obtained in response to this emission. This allows the user 14 to observe the virtual image V superimposed on the scenery ahead of the vehicle 10.

[0051] In this case, the HUD 12 projects display light L such that the virtual image V is tilted at an angle of 45 degrees or less (0 to 45 degrees) relative to the road surface on which the vehicle 10 is traveling, and the upper part of the virtual image V appears to be tilted farther from the user than the lower part. This allows the user 14 to view the virtual image V displayed mainly on the road surface ahead of the vehicle 10, and obtain various information indicated by the content of the image.

[0052] Incidentally, an angle of view (AV) is set for the HUD 12. If the imaginary range in space where the HUD 12 can form a virtual image V is defined as an imaging plane (IP), the angle of view (AV) is a field of view (FOV) defined based on a virtual line connecting the user's eye point (EP) and the outer edge of the imaging plane (IP). The angle of view (AV) is the angular range in which the user can view the virtual image V as viewed from the eye point (EP). In the HUD 12, the horizontal angle of view in the horizontal direction (e.g., approximately 10 to 12 degrees) is larger than the vertical angle of view in the vertical direction (e.g., approximately 4 to 5 degrees). The range of the angle of view (AV) is the forward range that overlaps with the imaging plane (IP) when viewed from the eye point (EP).

[0053] A display device (here, HUD 12) to which the display control device of this embodiment is applied displays first content (superimposed content) superimposed on a specific superimposed object present in the real scene, and second content (non-superimposed content) superimposed on the real scene without superimposing on the superimposed object. Specifically, HUD 12 displays superimposed content 211, 221 to 223, 231 (see FIGS. 4 to 10) and non-superimposed content 211 to 217 (see FIGS. 4 to 10) as a virtual image V.

[0054] Here, the superimposed content is a highly conspicuous AR display object using augmented reality (AR), and its display position is associated with a specific superimposed object in the foreground, such as the road surface, a vehicle ahead, a pedestrian, or a road sign. The superimposed content is displayed superimposed on the specific superimposed object, and, for example, a dangerous area on the road may be indicated by a frame so that the user can understand the extent of the dangerous area (see 221 in Figures 4 to 10 described later), or the inside of the frame may be painted in color, or a three-dimensional representation including the height direction may be used. The superimposed content can move as seen by the user, following the superimposed object. In other words, the positional relationship between the user's eyepoint (EP), the superimposed object in the foreground, and the superimposed content is continuously maintained.

[0055] On the other hand, non-superimposed content is a non-AR display object that is a display object superimposed on the foreground, excluding the superimposed content. Unlike the superimposed content, the non-superimposed content is displayed superimposed on the foreground without being superimposed on a specific superimposed object. The display position of the non-superimposed content is not associated with the specific superimposed object. The non-superimposed content is displayed as if fixed relative to the vehicle configuration, such as the windshield 13, without following the specific superimposed object. Note that due to the positional relationship between the vehicle 10 and the superimposed object, even non-superimposed content may be temporarily superimposed on the superimposed object.

[0056] As shown in FIG. 1B, the HUD 12 is mainly composed of a liquid crystal display 20, a first reflector 30, a second reflector 40, and a housing 50.

[0057] The liquid crystal display 20 is mainly composed of a light source 21 made of a light emitting diode mounted on a wiring board, and a TFT (Thin Film Transistor Liquid Crystal) liquid crystal display element 22 located on the emission side (directly above) of the light source 21 so as to transmit illumination light from the light source 21 and form display light L. This means that the light source 21 is provided behind the liquid crystal display element 22, and the liquid crystal display element 22 displays a predetermined image using light emitted from the light source 21.

[0058] The liquid crystal display 20 outputs display light L consisting of light in the visible wavelength range, and may be, for example, a light source 21 that emits white light. The light emitted from the light source 21 passes through the liquid crystal display element 22, thereby outputting display light L containing a desired image. The liquid crystal display element 22 forms a desired image based on display image data (drive signals) generated by a drawing calculation performed by a display control device 80 (see FIG. 2 ), which will be described later. The display control device 80 generates image data to be displayed based on vehicle information input from the vehicle 10 via communication, and displays the image data on the liquid crystal display element 22.

[0059] The first reflector 30 includes a cold mirror 31. The cold mirror 31 reflects light in the visible wavelength range (450 to 750 nm), which includes the emission wavelength range of the liquid crystal display 20, with a high reflectance of, for example, 80% or more, and reflects light outside the visible wavelength range with a low reflectance. In this case, the cold mirror 31 is designed to reflect light outside the visible wavelength range, particularly light in the infrared wavelength range (heat rays from external light such as sunlight), with a low reflectance of, for example, 15% or less. Light that is not reflected by the reflective layer of the cold mirror 31 is configured to pass through the cold mirror 31. The cold mirror 31 prevents the liquid crystal display element 22 from becoming too hot due to heating from sunlight or the like.

[0060] The second reflector 40 includes a concave mirror 41 that reflects the display light L from the cold mirror 31 (that is, the liquid crystal display element 22 ), and a mirror holder 42 that holds the concave mirror 41 .

[0061] The concave mirror 41 has a second reflective layer 41 a formed by vapor deposition on a resin substrate made of polycarbonate having a concave surface. The concave mirror 41 is provided at an angle so that the second reflective layer 41 a faces the cold mirror 31 and the light-transmitting cover and can be seen from the light-transmitting cover.

[0062] The concave mirror 41 magnifies the display light L from the cold mirror 31 and reflects (projects) it toward the translucent cover (windshield 13 of the vehicle 10). This means that the concave mirror 41 magnifies the display light L reflected by the cold mirror 31 and projects the magnified display light L through the translucent cover onto the windshield 13. The concave mirror 41 is adhered to a mirror holder 42 with a double-sided adhesive member. The mirror holder 42 is made of a synthetic resin such as ABS (Acrylonitrile Butadiene Styrene) and is fixed to the housing 50.

[0063] The housing 50 is made of, for example, a black light-blocking synthetic resin material and is formed in an approximately box-like shape, and its internal space, the space portion 51, holds and accommodates the projection unit consisting of the LCD display 20, the first reflector 30, and the second reflector 40, and is provided with an opening window portion 52 that opens to the upper part (windshield 13 side) of the concave mirror 41 in the second reflector 40.

[0064] The housing 50 is also provided with a light-transmitting cover 53, which serves as an emission section, to cover the open window section 52. This light-transmitting cover 53 is made of a light-transmitting synthetic resin material (e.g., acrylic resin), is formed in a curved shape (curved surface shape), and functions as a light-transmitting member through which the display light L reflected by the concave mirror 41 passes (passes through). That is, the display light L reflected by the cold mirror 31 and the concave mirror 41, i.e., the display light L output by the projection section, is projected onto the windshield 13 through the light-transmitting cover 53 formed on the housing 50, thereby displaying the virtual image V.

[0065] The HUD 12 projects and outputs onto the lower side of the windshield 13. Therefore, the user 14 can see the virtual image V superimposed on the scenery ahead from the lower area of ​​the windshield 13. Therefore, the user 14 mainly sees the virtual image V displayed on the road surface near the vehicle 10. Furthermore, due to the configuration of the projection unit of the HUD 12, the virtual image V is outputted so that the upper part of the output image appears to be tilted so that it is farther from the vehicle 10 than the lower part.

[0066] FIG. 2 is a block diagram showing the configuration of a display control system 200 using the display control device 80 of this embodiment.

[0067] As shown in FIG. 2, the display control system 200 includes a perimeter monitoring sensor 60, a navigation unit 70, a display control device 80 of this embodiment, and a display unit 90.

[0068] The perimeter monitoring sensor 60 is a sensor necessary for recognizing the surrounding driving environment including the area ahead of the vehicle 10, and includes at least a camera 61, a LiDAR (Light Detection And Ranging) 62, and a millimeter-wave radar 63. Detection information detected by the perimeter monitoring sensor 60 is transferred to the navigation unit 70 and the display control device 80 of this embodiment via a CAN (Control Area Network) 100, which is an in-vehicle LAN.

[0069] The camera 61 captures at least the field of view (actual view) ahead of the vehicle 10, and the LiDAR uses near-infrared light, visible light, and ultraviolet light to, for example, irradiate an obstacle or the like ahead of the vehicle 10 captured by the camera 61 with light, capture the reflected light with an optical sensor, and determine the distance to the obstacle based on the time difference. The millimeter-wave radar 63 uses a frequency band of 30 GHz to 300 GHz (EHF: Extra High Frequency) and, like the LiDAR 62, determines the distance to the obstacle by remote sensing.

[0070] The navigation unit 70 uses a global navigation satellite system (GNSS) such as the global positioning system (GPS) and a gyro sensor to obtain map information via a recording medium or wireless communication with the outside of the vehicle, and provides guidance to facilities around the vehicle 10, route guidance, etc. The navigation unit 70 transfers a signal prompting display output based on this guidance information to the display unit 90 via the CAN 100 and the display control device 80 at an appropriate timing.

[0071] The navigation unit 70 has a map database (map DB 71). The map DB 71 acquires and stores the latest map information, for example, by communicating with an external center (not shown) via a V2X (Vehicle to X) type communication system (not shown). The map information stored in the map DB 71 is mapping data that has been digitized to represent the driving environment of the vehicle 10. The mapping data is preferably digital data of a particularly high-precision dynamic map. A "dynamic map" is a digital map that combines a huge amount of dynamic information that changes every moment, such as traffic regulations, construction information, accidents, congestion, pedestrians, and traffic lights, with static information such as high-precision three-dimensional position information (road surface information, slope information, three-dimensional structures).

[0072] Although not shown, a driving assistance ECU (electronic control unit) that assists the user in driving is also connected to CAN 100. The driving assistance ECU has at least one of a driving assistance function that assists the user in driving operations and an automatic driving function that can perform driving operations on behalf of the user, and is capable of recognizing the driving environment ahead of vehicle 10 based on detection information acquired from perimeter monitoring sensor 60, controlling vehicle 10 in accordance with the analysis results of the recognized detection information, and transferring the analysis results of the detection information to control device 80 via CAN 100.

[0073] Display control device 80 generates first content (superimposed content) that is displayed superimposed on specific superimposed objects (road surface, preceding vehicle, pedestrian, road sign, etc.) that exist in the real scene, and second content (non-superimposed content) that is displayed superimposed on the real scene without being superimposed on the superimposed objects, and sequentially displays the superimposed content and the non-superimposed content on display unit 90. To this end, display control device 80 includes a control unit 81, a storage unit 82, a display information generation unit 83, and a display control unit 84.

[0074] The control unit 81 determines the distance between the vehicle position and the superimposed object based on the superimposed object captured by the periphery monitoring sensor 60, as well as determines the relevance between the superimposed content and the non-superimposed content, and also determines the visibility of the forward field of view of the vehicle 10. To this end, the control unit 81 includes a main control unit 810, a distance determination unit 811, a relevance determination unit 812, and a visibility determination unit 813.

[0075] The main control unit 810 controls the interface with the perimeter monitoring sensor 60 and the navigation unit 70, and, for example, recognizes information about superimposed objects such as the road surface, vehicles ahead, pedestrians, and road signs based on information monitored by the perimeter monitoring sensor 60.In addition, the main control unit 810 acquires static information such as high-precision three-dimensional position information (road surface information, diagonal line information, three-dimensional structures) from the map DB 71 held by the navigation unit 70, as well as dynamic information such as ever-changing weather, and transfers this information to the distance determination unit 811, the relevance determination unit 812, and the visibility determination unit 813.

[0076] A distance determination unit 811 determines the distance between the vehicle position and the superimposed object, a relevance determination unit 812 determines the relevance between the superimposed content and the non-superimposed content, and a visibility determination unit 813 determines the visibility in the forward field of view of the vehicle 10. The distance information determined by the distance determination unit 811, the relevance information determined by the relevance determination unit 812, and the visibility information determined by the visibility determination unit 813 are all output (written) to the memory unit 82 via an internal bus 85 consisting of a plurality of lines for addresses, data, and control.

[0077] The memory unit 82 is allocated a memory area for storing the entire content, including superimposed content and non-superimposed content, for content generation and display, and a memory area for VRAM (Video RAM), and is composed of, for example, DRAM (Dynamic Random Access Memory) or static RAM.

[0078] The display information generation unit 83 generates superimposed content that is superimposed on a specific superimposed object present in the real scene, and non-superimposed content that is superimposed on the real scene without being superimposed on the superimposed object, associates these contents with their display attributes (display position, display color, and display start timing setting information described below), and transfers them to the memory unit 82 via the internal bus 85 (where they are written to a specified area of ​​the control unit 81).

[0079] The display control unit 84 reads the superimposed content and non-superimposed content from the storage unit 82 and displays them sequentially on the display unit 90. Here, the non-superimposed content is displayed first, followed by the superimposed content. Specifically, the display control unit 84 reads the superimposed content and non-superimposed content written in a predetermined area of ​​the storage unit 82, draws them in a VRAM allocated to a separate area in synchronization with the refresh rate according to the display attribute information (display position, display color, etc.) stored corresponding to those contents, and displays them on the display unit 90 by sequentially reading them out of the VRAM in synchronization with the display start timing.

[0080] Note that even in a situation where the superimposed content can be superimposed on the superimposition target, the display control unit 84 displays the non-superimposed content and then displays the superimposed content on the display unit 90. Furthermore, the display control unit 84 displays the superimposed content farther away from the user than the non-superimposed content.

[0081] Here, the display unit 90 used as the display device may have, for example, a display A (first display) 91 and a display B (second display), and the display control unit 84 may display the superimposed content on the display A (91) and the non-displayed content on the display B (92). Here, the display A (91) may be, for example, the HUD 12 shown in Figures 1(a) and 1(b), and the display B (92) may be, for example, a WSD that projects an image onto the black ceramic part (black part of the windshield) of the vehicle.

[0082] In addition, the display control unit 84 may display the superimposed content in a first manner (e.g., a frame line 221 surrounding the superimposed object shown in Figure 4 described later) and display the non-superimposed content in a second manner (e.g., text 211 or icon 212 shown in Figure 4 described later).

[0083] Furthermore, when a first superimposed content (e.g., 221 in Figure 9 described later) displayed corresponding to a first superimposed object and a second superimposed content (e.g., 223 in Figure 9 described later) displayed corresponding to a second superimposed object, which is a specific superimposed object, are simultaneously displayed among specific superimposed objects whose relevance indicating the relevance between the superimposed content and the non-superimposed content output by the relevance determination unit 812 is less than a predetermined value, the display control unit 84 may delay the timing of starting to display the second superimposed content (223), or may control to advance the timing of starting to display the first superimposed content (221), or may perform both.

[0084] In addition, when the degree of relevance indicating the relevance between the superimposed content and the non-superimposed content output by the relevance determination unit 812 is equal to or greater than a predetermined value, the display control unit 84 may control the display of the superimposed content and the non-superimposed content by arranging them on a straight line connecting the superimposed object and the user's eyepoint when looking at the superimposed object from above.

[0085] The relevance between the superimposed content and the non-superimposed content can be determined as being high, for example, if the superimposed content and the non-superimposed content are created for a single object captured by the perimeter monitoring sensor 60. Alternatively, for example, a label can be assigned to each piece of content, and if the number of matching labels between the superimposed content and the non-superimposed content is equal to or greater than a predetermined threshold, the relevance can be determined as being high. Another possible method is to calculate a correlation coefficient, which will be described later.

[0086] The display control unit 84 may also control the timing of starting display of the superimposed content according to the visibility, which indicates the visibility and is output from the visibility determination unit 813, which determines the visibility of the forward field of view of the vehicle 10. Here, the visibility determination can be achieved, for example, by generating feature amounts for each part constituting a specific subject, comparing them with feature amounts for the same subject under a pre-set high-visibility condition, and calculating a score according to the comparison result. The visibility of a subject is highest when all parts are clearly visible, and the more obscured the subject is due to weather or other factors, the smaller the clearly visible area becomes. Therefore, the degree to which each part constituting the subject is visible can be estimated from the score for each part. Visibility can be evaluated by calculating the ratio of the sum of the areas of the parts with high scores to the total area of ​​the subject as a quantity representing the degree to which the subject is visible.

[0087] Furthermore, the control of the display start timing can be realized, for example, by providing a range (timing settable range) for the display start timing setting and setting the display start timing to be earlier or later than the standard display start timing, and then controlling the rise of the blanking period (during the display period for both horizontal and vertical synchronization signals) according to that setting.

[0088] Furthermore, display control unit 84 may control the timing of starting to display the superimposed content according to the display distance of the non-superimposed content. Here, the display distance of the non-superimposed content is the distance from the user's eyepoint EP ( FIG. 1A ) to the non-superimposed content, and the display distance is controlled by at least one of changing the optical path length from display unit 90, such as a backlit LCD panel, to windshield 13 ( FIG. 1A ), which is the projection target (specifically, the length of the optical path from a first point where the optical axis of the optical system intersects with the display surface of display unit 90, to a second point where the optical axis intersects with windshield 13, which is the projection target), and changing the display position of the non-superimposed content on the display surface.

[0089] Furthermore, display control unit 84 may control the display distance interval between the superimposed content and the non-superimposed content in accordance with the display distance of the superimposed content, or may control the display distance interval between the superimposed content and the non-superimposed content in accordance with the distance between the vehicle position and the superimposed object determined by distance determination unit 811. In this case, the display distance interval is controlled by adjusting the display distance of the non-superimposed content in accordance with the degree of relevance indicating the relevance between the superimposed content and the non-superimposed content determined by relevance determination unit 812.

[0090] In addition, the display control unit 84 may perform control to hide the non-superimposed content after the non-superimposed content is displayed until the superimposed content is displayed, or may perform control to hide the superimposed content after the non-superimposed content and the superimposed content are displayed.

[0091] (Operation of the embodiment) Fig. 3 is a flowchart showing the operation of the display control system 200 to which the display control device 80 of this embodiment is applied. Also, Figs. 4 to 10 show display examples (1) to (7) of content generated by the display control device 80 of this embodiment. Note that, unless otherwise specified, the content referred to here is content that is displayed by combining superimposed content and non-superimposed content.

[0092] The operation of the display control device 80 of this embodiment shown in FIG. 1 will be described in detail below with reference to the flowchart shown in FIG. 3 and content display examples (1) to (7) shown in FIGS.

[0093] 3 , the display control system 200 first captures a superimposition object, which is a specific object present in the foreground, such as the road surface, a vehicle ahead, a pedestrian, or a road sign (step S101). The superimposition object is captured in real time by information exchange via the CAN 100 between the surrounding monitoring sensor 60, which includes a camera 61, a LiDAR 62, and a millimeter-wave radar 63, and the navigation unit 70, which has map data and a dynamic map as a map DB 71, and is transferred to the display control device 80 (main control unit 810) via the CAN 100.

[0094] The term "dynamic map" as used here refers to a dynamic map in a broad sense, and includes not only dynamic map platforms used for autonomous driving but also commonly used maps, such as those used in map applications, that include real-time dynamic information such as weather information and road closures due to construction. The superimposed objects primarily refer to objects that may put the user at risk, including road obstacles and dangerous road areas. Examples of dangerous road areas include areas where road conditions have changed from normal, such as areas where the road surface is frozen due to low temperatures, areas with severe unevenness, and areas where construction work has suddenly occurred.

[0095] Next, the distance determination unit 811 of the display control device 80 calculates the remaining distance from the vehicle's position to the superimposition object (step S102). The remaining distance can be calculated by calculating the distance from the superimposition object captured in step S101 to the vehicle's position (measured by a GPS or the like provided in the navigation unit 70). Here, the remaining distance includes a straight-line distance and a route distance. The distance determination unit 811 may also calculate a predicted arrival time from the route distance and the vehicle's speed (measured by a vehicle speed sensor or the like mounted on the vehicle 10 as a behavior sensor), and determine the timing to start displaying the content from this predicted arrival time.

[0096] Control of the display start timing can be achieved, for example, by providing a range (timing settable range) for the display start timing setting, setting the display start timing to be earlier or later than the standard display start timing, and controlling the rise of the blanking period (during the display period for both horizontal and vertical synchronization signals) according to that setting.

[0097] Next, the distance determination unit 811 determines whether the remaining distance is less than a first threshold (step S103). The first threshold here is, for example, 60 meters from the superimposed object. This value is double the time to signal a right or left turn or a lane change, as stated in Article 21 of the Enforcement Order of the Road Traffic Act. Regarding the driver's cognitive reaction time, for example, the cognitive reaction time for a recognized object on the road has been shown to be a maximum of 1.5 seconds for young people and a maximum of 1.8 seconds for elderly people. Since a vehicle traveling at 60 km / h requires approximately 3.6 seconds to reach a point 60 meters ahead, it is considered appropriate to set the first threshold at 60 meters.

[0098] If the remaining distance is less than the first threshold (step S103 "YES"), the display information generation unit 83 generates non-overlapping content (step S104). The non-overlapping content generated by the display information generation unit 83 is written to a predetermined area of ​​the storage unit 82 together with its display attributes (display position, display start timing, display color, etc.) (step S105). The display control unit 84 then reads the non-overlapping content written to the predetermined area of ​​the storage unit 82, renders the non-overlapping content in a VRAM allocated to a separate area in synchronization with the refresh rate according to the display attributes stored in association with the non-overlapping content, and sequentially reads the non-overlapping content from the VRAM area in synchronization with the display start timing, thereby displaying it on the display unit 90, for example, as one of display examples (1) to (7) shown in Figures 4 to 10 (step S106).

[0099] Display examples (1) to (7) of non-superimposed content will be described using FIGS. 4 to 10 . An example of non-superimposed content is non-superimposed content 211 ( FIG. 4 ), which indicates danger in text format. This format notifies users of danger without mentioning its details. However, non-superimposed content 213 ( FIG. 6 ) also includes a format that notifies users of danger but also mentions the details of the danger. Alternatively, an icon may be displayed to intuitively indicate danger to the user, as in non-superimposed content 212 ( FIGS. 4 and 5 ). Similar to text, icons may also be displayed to notify users of danger without mentioning its details, as in non-superimposed content 214 ( FIG. 6 ). Furthermore, non-superimposed content is not limited to being displayed singly; multiple pieces of non-superimposed content can be displayed in combination to more intuitively convey danger to the user.

[0100] Other examples include a small-world format, such as non-overlapping content 215 (FIG. 7), which displays a bird's-eye view of road conditions. This displays the superimposed content to be displayed and the vehicle's position in advance, superimposed on a map display showing the planned driving route. Another possible format is non-overlapping content 216 (FIG. 8), which displays the remaining distance to the recognized superimposed object using a gauge (elevator icon).

[0101] On the other hand, if the remaining distance determination process in step S103 determines that the remaining distance is not less than the first threshold (is greater than or equal to the first threshold) (step S103 "NO"), the remaining distance calculation process in step S102 is repeatedly executed.

[0102] Next, the distance determination unit 811 calculates the remaining distance from the vehicle position to the superimposed object (step S107), similar to the remaining distance calculation process in step S102. Then, it is determined whether the remaining distance is less than a second threshold (step S108). Here, the second threshold is, for example, 30 m before the superimposed object. This is the value of the timing for signaling a right or left turn or lane change, as described in Article 21 of the Enforcement Order of the Road Traffic Act.

[0103] Here, if we calculate in the same way as with the first threshold, it takes approximately 1.8 seconds for a vehicle traveling at 60 km / h to reach a point 30 m away. This coincides with the maximum cognitive reaction time of 1.8 seconds for the elderly to recognize an object on the road that needs to be recognized. In reality, it is thought that the display of non-overlapping content allows users to detect danger in advance and slow down when driving, so this is considered to be a sensible threshold.

[0104] If the remaining distance is less than the second threshold in the remaining distance determination process of step S108 ("YES" in step S108), the display information generation unit 83 generates superimposed content (step S109). The non-superimposed content generated by the display information generation unit 83 is written to a predetermined area of ​​the storage unit 82 together with its display attributes (display position, display start timing, display color, etc.) (step S110). The display control unit 84 then reads the superimposed content written to the predetermined area of ​​the storage unit 82, renders the superimposed content in a VRAM allocated to a separate area in synchronization with the refresh rate according to the display attributes stored in the storage unit 82 corresponding to the superimposed content, and sequentially reads the rendered content from the VRAM in synchronization with the display start timing, thereby displaying it on the display unit 90, for example, as one of display examples (1) to (7) shown in Figures 4 to 10 (step S111).

[0105] On the other hand, if the remaining distance determination process in step S108 determines that the remaining distance is not less than the second threshold (is greater than or equal to the second threshold) (step S108 "NO"), the remaining distance calculation process in step S107 is repeatedly executed.

[0106] Here, display examples (1) to (7) of superimposed content will be described using Figures 4 to 10. An example of superimposed content is superimposed content 221 (Figures 4, 7 to 10) showing a dangerous area on a road. The superimposed content is superimposed content that conveys the range of the superimposed object to the user, and is not limited to representations that only present a frame line (Figures 4, 7 to 10). It may also be representations that paint the inside of the frame with color or a three-dimensional representation that includes the height direction. Also included is a display form that does not show the dangerous area, but depicts a route that can avoid the dangerous area, as shown in superimposed content 222 (Figure 5).

[0107] Alternatively, instead of indicating a danger zone as in superimposed content 221 (FIGS. 4 and 7 to 10), an expression that allows for an intuitive understanding of the nature of the danger may be used, such as superimposed content 223 (FIG. 9). Note that superimposed content 223 shown in FIG. 9 is an example of an intuitive expression indicating the danger of frozen road surfaces. Furthermore, it is also possible to present an animation rather than just a still image, which allows for a more intuitive understanding.

[0108] 4 to 10 are display examples in which superimposed content and non-superimposed content are combined, with the superimposed content being displayed farther away from the user than the non-superimposed content. This allows the user to look at something close and then move their gaze to a distant location, reducing the number of round-trip gaze movements and enabling smoother viewing.

[0109] Here, even in a situation where the superimposed content can be superimposed on the superimposed object, the non-superimposed content can be displayed first, followed by the superimposed content. For example, if another vehicle cuts in between the vehicle 10 and the superimposed object while the perimeter monitoring sensor 60 is capturing the superimposed object, even if the superimposed content is displayed, it will be difficult to determine its correspondence with the superimposed object, and the effects of the present invention cannot be guaranteed. Therefore, by displaying the non-superimposed content first, it is possible to provide advance information in a form that is easy for the user to understand.

[0110] Furthermore, because superimposed content generally has the characteristic of being displayed so as to visually overlap the object being superimposed, it is highly likely to be displayed above display area 201 (FIGS. 4 to 10). Therefore, non-superimposed content is preferably displayed below display area 201 to avoid cluttering display area 201. For example, if content showing a speedometer that displays the speed of the vehicle has a character height of 0.54° and is drawn at the bottom edge of display area 201, the area at the bottom edge of display area 201 that is 0.54° in the vertical direction may be used as the area for displaying non-superimposed content.

[0111] Alternatively, the relevance determination unit 812 may determine the relevance between the superimposed content and the non-superimposed content, and the display control unit 84 may control the display timing and display position of the superimposed content and the non-superimposed content according to the relevance indicating the relevance. For example, in a situation shown in FIG. 9 where non-superimposed content 211 and superimposed content 221 that notify information about a superimposed object a (first superimposed object) and non-superimposed content 213 and superimposed content 223 that notify information about a superimposed object b (second superimposed object) are displayed, the relevance is determined to be low because the superimposed object a and the superimposed object b are different. In FIG. 9 , when there is a possibility that the superimposed content 221 and the non-superimposed content 213 will be displayed simultaneously, the display control unit 84 controls to delay the display start timing of the non-superimposed content 213 or to advance the display start timing of the superimposed content 221.

[0112] As described above, the degree of relevance is determined to be high if the superimposed content and non-superimposed content are generated for one target captured by the perimeter monitoring sensor 60. Alternatively, for example, a label may be assigned to each piece of content, and if the number of matching labels between the superimposed content and non-superimposed content is equal to or greater than a predetermined threshold, the degree of relevance may be determined to be high.

[0113] Alternatively, a method of calculating a correlation coefficient can be considered. In this case, for example, the number of times each piece of content is displayed within a certain interval is calculated, and this is performed for multiple intervals. Then, a correlation coefficient is calculated for all possible combinations of each piece of superimposed content and each piece of non-superimposed content. Combinations for which the absolute value of the correlation coefficient is equal to or greater than a predetermined threshold are determined to have a high degree of association. The following formula is used to derive the correlation coefficient, where r represents the correlation coefficient, x represents the number of times the superimposed content is displayed, and y represents the number of times the non-superimposed content is displayed.

[0114]

[0115] When the degree of association described above is equal to or greater than a predetermined threshold, the display control unit 84 displays the superimposed content and the non-superimposed content by arranging them on a straight line connecting the user's eyepoint EP ( FIG. 1A ) when looking down on the superimposed object (the first content and the second content are viewed in a line by the user), which makes the movement of the user's line of sight natural and makes the association of the content visually easy to understand. In other words, when the user's point of gaze switches from the real view in front to the virtual image, the user's line of sight is reduced as much as possible (the need for eye focus adjustment is reduced as much as possible).

[0116] Furthermore, the timing to start displaying the superimposed content may be determined according to the display distance of the non-superimposed content. For example, if the display distance of the non-superimposed content is short, the timing to start displaying the superimposed content is delayed. If the non-superimposed content is short and the superimposed content is displayed immediately, the user's line of sight will move too much from the non-superimposed content to the superimposed content, which is expected to increase the user's visibility load from the perspective of convergence accommodation and focus accommodation. Therefore, if the display distance of the non-superimposed content is short, the visibility load can be reduced by delaying the timing to start displaying the superimposed content.

[0117] For example, if the non-superimposed content is displayed at a display distance less than a predetermined threshold, the superimposed content may be displayed with a delay of a predetermined number of seconds. Alternatively, the second threshold in the "remaining distance determination process" in step S108 may be reduced to delay the timing at which the superimposed content starts to be displayed.

[0118] Furthermore, the timing to start displaying the superimposed content may be determined based on the visibility (visibility) of the forward field of view determined by the visibility determination unit 813. For example, a situation may arise in which the user's visibility of the outside world is reduced due to a change in weather such as rain or snow. In such a situation, the user may not be able to see the superimposed object until the last moment, and if the information notification by the superimposed content is made at the last moment, this may upset the user. Therefore, by having the display control unit 84 perform control to advance the timing to start displaying the superimposed content, it becomes possible to prepare for danger in advance while reducing the mental burden on the user.

[0119] In addition, a situation may be considered in which a vehicle ahead is traveling in a way that blocks (obstructs) the user's field of vision, making it impossible to see the superimposed object. In this situation, as described above, since there is a possibility that the superimposed object will not be visible until the last moment, the display control unit 84 can execute processing to advance the timing of starting to display the superimposed content.

[0120] Furthermore, the display distance interval between the superimposed content and the non-superimposed content may be determined according to the display distance of the superimposed content. For example, after the superimposed object is captured by the perimeter monitoring sensor 60 and the display distance of the superimposed content is determined, the display distance of the non-superimposed content is determined, and the display distance interval is determined. Here, assuming that the display distance of the superimposed content is d1 and the display distance of the non-superimposed content is d2, the display control unit 84 may perform control such that if the relevance determined by the relevance determination unit 812 is high, the value of the display distance d2 of the non-superimposed content is increased to narrow the display distance interval, and conversely, if the information relevance is low, the value of the display distance d2 of the non-superimposed content is decreased to widen the display distance interval. This is to visually separate content with low relevance and to provide a notification that does not mislead the user.

[0121] Note that when the display unit 90 displaying the superimposed content is display device A (91), the non-superimposed content may be displayed using a separate display device B (92) instead of display device A (91). A display example (10) in this case is shown in FIG. 10 . A possible combination is that display device A (91) is a HUD, and display device B (92) is a WSD that enables an image to be projected onto the black center of the vehicle. In this case, as shown in FIG. 10 , the superimposed content is displayed in display area 201, and the non-superimposed content is displayed in display area 202. By displaying the superimposed content and the non-superimposed content in separate display areas, it is possible to achieve a display format that ensures the effects of the present invention while reducing the clutter of content within a single display area.

[0122] Effect of the Embodiment As described above, according to the display control device 80 of this embodiment, the display control unit 84 sequentially displays on the display device superimposed content (first content) that is displayed superimposed on a specific superimposed object present in the real scene, and non-heavy content (second content) that is displayed superimposed on the real scene without being superimposed on the superimposed object. In this way, the amount of information provided to the user is distributed between the superimposed content and the non-superimposed content, and the superimposed content and the non-superimposed content are displayed in stages. This makes it possible, for example, to notify the user of a dangerous driving situation in an easy-to-understand manner with minimal burden, thereby enabling the user to prepare for the danger.

[0123] Furthermore, according to the display control device 80 of this embodiment, the display control unit 84 displays the superimposed content after displaying the non-superimposed content. In this way, advance warning using non-superimposed content that has substantially the same meaning as the superimposed content, rather than highly conspicuous superimposed content, can convey necessary information to the user while ensuring forward visibility. Therefore, for example, by warning the user of a dangerous situation in advance in an easily understandable manner, the user can be prepared for danger.

[0124] Furthermore, according to the display control device 80 of this embodiment, since displaying highly eye-catching superimposed content first can lead to a lack of vigilance toward other things, by displaying non-superimposed content first, the user can absorb information while somewhat diverting their attention. Also, by shifting the user's gaze from non-superimposed content displayed close to the user to superimposed content displayed farther away, the number of round-trip gaze movements can be reduced, enabling a smoother viewing experience that is easy for the user to understand.

[0125] Furthermore, according to the display control device 80 of this embodiment, the display device (display unit 90) has a first display (display unit A (91)) and a second display (display unit B (92)), and the display control unit 84 displays superimposed content on display unit A (91) and non-superimposed content on display unit B (92). For example, by configuring display unit A (91) as a HUD and display unit B (92) as a WSD that projects an image onto the black window of the vehicle (the black part of the windshield), the content display area can be made larger and the display can be prevented from becoming complicated.

[0126] Furthermore, according to the display control device 80 of this embodiment, the display control unit 84 can provide advance warning using non-superimposed content such as text or icons, thereby allowing the user to understand the nature of the danger in advance, for example.

[0127] Furthermore, according to the display control device 80 of this embodiment, when superimposed content for one superimposed object and non-superimposed content for another superimposed object are displayed simultaneously, the display control unit 84 delays the timing at which the non-superimposed content starts to be displayed, or controls the timing at which the superimposed content starts to be displayed to be earlier, or performs both, thereby preventing the user from misunderstanding that the content is related.

[0128] Furthermore, according to the display control device 80 of this embodiment, when a user views superimposed content or non-superimposed content from the real scene in front of the vehicle, the user needs to direct the focus of their eyes (point of gaze) from the real scene in real space to the virtual image. However, when the degree of association is equal to or greater than a predetermined value, the display control unit 84 displays the superimposed content and non-superimposed content on a straight line connecting the superimposed content and the user's eye point EP when looking down on the superimposed object (the user sees the superimposed content and non-superimposed content in a line). This makes the user's line of sight move more naturally, and makes the association between the content more visually understandable. In other words, when the user's point of gaze switches from the real scene in front of the vehicle to the virtual image, the user's gaze is reduced as much as possible (the need to adjust the focus of the eyes is minimized as much as possible).

[0129] Furthermore, according to the display control device 80 of this embodiment, in situations where the user's visibility is reduced due to changes in weather such as rain or snow, for example, the superimposed object cannot be seen until the last moment, but under such circumstances, the display control unit 84 performs control to advance the timing of the start of displaying the superimposed content (in other words, to advance the timing of the information notification), thereby reducing the user's mental burden and enabling the user to prepare in advance for impending danger. Furthermore, by determining the timing of the start of displaying the superimposed content according to the display distance of the non-superimposed content, the display start timing can be set so that the user's line of sight does not move too much from the non-superimposed content to the superimposed content, thereby reducing the user's visual burden (the burden of adjusting the user's eye focus).

[0130] Furthermore, according to the display control device 80 of this embodiment, the display control unit 84 controls the display distance interval between the superimposed content and the non-superimposed content in accordance with the display distance of the superimposed content, thereby enabling a display distance interval that does not require excessive line-of-sight movement from the non-superimposed content to the superimposed content, thereby reducing the visual burden on the user (the burden of adjusting the focus of the user's eyes). Furthermore, the display control unit 84 controls the display distance interval between the superimposed content and the non-superimposed content in accordance with the distance between the vehicle position and the superimposed object, thereby enabling a display distance interval that does not require excessive line-of-sight movement from the non-superimposed content to the superimposed content, thereby reducing the visual burden on the user (the burden of adjusting the focus of the user's eyes).

[0131] Furthermore, according to the display control device 80 of this embodiment, for example, if the display distance of the superimposed content is d1 and the display distance of the non-superimposed content is d2, the display control unit 84 adjusts the display distance of the non-superimposed content by increasing the value of d2 to narrow the display distance interval if the degree of association between the superimposed content and the non-superimposed content is high, and decreasing the value of d2 to widen the display distance interval if the degree of association is low. This makes it possible to visually separate superimposed content and non-superimposed content that are less related, and to provide a notification that does not mislead the user.

[0132] Furthermore, according to the display control device 80 of this embodiment, the display control unit 84 hides the non-superimposed content after the non-superimposed content is displayed until the superimposed content is displayed, thereby reducing the amount of content occupying the display field of view and also reducing the visual burden on the user. Furthermore, the display control unit 84 hides the non-superimposed content after the non-superimposed content and the superimposed content are displayed, thereby reducing the amount of content occupying the display field of view and also reducing the visual burden on the user.

[0133] (Modification) In the display control device 80 of the present embodiment, the distance determination unit 811, the relevance determination unit 812, and the visibility determination unit 813 have been described as being included in the control unit 81 within the display control device 80, but at least one of these may be replaced by a configuration included in the navigation unit 70 or a driving assistance ECU (electronic control unit) not shown. In this case, the load on the display control device 80 is reduced.

[0134] Furthermore, the functions of the control unit 81 described above are programs for causing a computer to function, and can be realized by programs for causing a computer to function as each of the functional blocks 810 to 813. In this case, the control unit 81 includes a computer having at least one control device (e.g., a processor), at least one storage device (e.g., RAM), and a peripheral control LSI (e.g., a display information generation unit 83, a display control unit 84) as hardware for executing the above-mentioned programs. By executing the above-mentioned programs using this control device, storage device, and peripheral control LSI, the control of each functional block described in the display control device 80 of this embodiment can be realized.

[0135] Furthermore, the above-mentioned program may be recorded on one or more computer-readable recording media, rather than being temporarily stored. The recording media may or may not be included in the control unit 81. In the latter case, the above-mentioned program may be supplied to the control unit 18 via any wired or wireless transmission medium.

[0136] Furthermore, at least some of the functions of each of the above-described control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit functioning as each of the above-described functional blocks is formed is also included in the scope of the present invention. Alternatively, the functions of each functional block can be realized by, for example, a quantum computer. Furthermore, each process described in this embodiment may be executed by AI (Artificial Intelligence). In this case, the AI ​​may operate in the control unit 81 of this embodiment, or may operate in another device (e.g., a cloud server, etc.).

[0137] The present invention is not limited to the above-described exemplary embodiments, and those skilled in the art will be able to easily modify the above-described exemplary embodiments to the extent that they fall within the scope of the claims.

[0138] 10...vehicle, 12...HUD, 13...windshield, 20...liquid crystal display, 60...periphery monitoring sensor, 61...camera, 62...LiDAR, 63...millimeter wave radar, 70...navigation unit, 71...map DB, 80...display control device, 81...control unit, 82...memory unit, 83...display information generation unit, 84...display control unit, 85...internal bus, 90...display unit, 91...display unit A, 92...display unit B, 100...CAN, 810...main control unit, 811...distance determination unit, 812...relevance determination unit, 813...visibility determination unit, 200...display control system

Claims

1. A display control device that controls a display device that displays content superimposed on a real scene, the display control device having: a display information generation unit that generates a first content to be superimposed on a specific superimposition object present in the real scene, and a second content to be superimposed on the real scene without being superimposed on the superimposition object; and a display control unit that sequentially displays the first content and the second content on the display device.

2. The display control device according to claim 1, wherein the display control unit displays the second content and then displays the first content.

3. A display control device as described in claim 1, wherein the display control unit displays the first content after displaying the second content even if the first content can be superimposed on the superimposition object.

4. The display control device according to claim 1, wherein the display control unit displays the first content farther away from the user than the second content.

5. A display control device as described in claim 1, wherein the display device has a first display and a second display, and the display control unit displays the first content on the first display and the second content on the second display.

6. The display control device according to claim 1, wherein the display control unit displays the first content in a first manner and displays the second content in a second manner.

7. A display control device as described in claim 1, further comprising a relevance determination unit that determines the relevance between the first content and the second content, wherein the display control unit performs control to delay the timing of starting to display the second content and / or advance the timing of starting to display the first content when the first content displayed corresponding to a first superimposed object among the specific superimposed objects, and the second content displayed corresponding to a second superimposed object which is the specific superimposed object, are displayed simultaneously, the degree of relevance indicating the relevance being less than a predetermined value.

8. A display control device as described in claim 1, further comprising a relevance determination unit that determines the relevance between the first content and the second content, and wherein the display control unit, when the relevance level indicating the relevance is equal to or greater than a predetermined value, displays the first content and the second content by arranging them on a straight line connecting the superimposed object and the user's eyepoint when looking down on the superimposed object.

9. A display control device as described in claim 1, further comprising a visibility determination unit that determines the visibility of the forward field of view of the vehicle, and the display control unit controls the timing of starting to display the first content according to a visibility indicating the visibility.

10. The display control device according to claim 1, wherein the display control unit controls the timing of starting display of the first content in accordance with the display distance of the second content.

11. A display control device according to claim 10, wherein the display control unit controls a display distance interval between the first content and the second content in accordance with a display distance of the first content.

12. A display control device as described in claim 11, further comprising a distance determination unit that determines the distance between the vehicle position and the superimposed object, and the display control unit controls the display distance interval between the first content and the second content in accordance with the distance determined by the distance determination unit.

13. A display control device as described in claim 11 or 12, further comprising a relevance determination unit that determines the relevance between the first content and the second content, and the display control unit adjusts the display distance of the second content according to a relevance degree indicating the relevance, thereby controlling the display distance interval.

14. The display control device according to claim 1, wherein the display control unit performs control such that, after the second content is displayed, the second content is not displayed until the first content is displayed.

15. The display control device according to claim 1, wherein the display control unit performs control to hide the second content after the second content and the first content are displayed.

Citation Information

Patent Citations

  • Display control device and display control program

    JP2019206256A

  • Display device

    JP2020163942A

  • Display control device and display control program

    JP2021037916A

  • Display control device

    JP2021091334A

  • Display control device and display control program

    JP2021127032A