Display control device, display device, and display control method

Abstract

To perform display control which assists recovery of attention or concentration of a driver at an initial period or a transition period of shifting from a normal driving state to a careless driving state.SOLUTION: A display control device 80 mounted on a vehicle 1 has a control unit 82 which controls image display of a display device 111. The control unit 82 has: a driver state determination unit 84 which determines a state of a driver 5 during driving; a cautious event detection unit 83 which detects a cautious event which is an event that the driver should pay attention or be cautious; a potential risk event determination unit 85 which determines whether or not there is a potential risk event which is an event presumed to have a possibility of leading to future risk; and an awakening support processing unit 94 which performs awakening support processing in which display states of images AR1, AR2 are changed to provide visual stimulation different from stimulation felt during normal driving to the driver and thereby support recovery of attention or concentration of the driver when it is determined that there is the potential risk event.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a display control device, a display device, a display control method, and the like, which are mounted on a vehicle such as an automobile. [Background technology]

[0002] Patent document 1 discloses a driving assistance device that includes an estimation means for estimating the driver's state, a determination means for determining the type to which the driver belongs based on driving operation characteristics, and a presentation means for presenting driving assistance information according to the estimated driver's state and the type to which the driver belongs.

[0003] In

[0060] of Patent Document 1, an example of driving assistance is given in which the driver is alerted on a long downhill slope or around a sharp curve. Specifically, drivers are classified into "Type A drivers who can apply the brakes properly" and "Type B drivers who take time to assess the situation outside and therefore cannot apply the brakes properly," and the driver's state is classified into "normal state" and "distracted state."

[0004] As described in

[0062] , when the driver is absent-minded, the driving support device issues an alarm to the type A driver, prompts the driver to check the speedometer by voice, and displays an exclamation mark on the HUD (Head-Up Display) to call the driver's attention. On the other hand, for Type B drivers, a voice prompt is given to encourage them to apply the brakes, a mark instructing them to slow down is displayed on the HUD, and warning marks etc. are displayed in stages according to the deceleration situation. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-97485 A Summary of the Invention [Problem to be solved by the invention]

[0006] In the above-mentioned Patent Document 1, after determining that the driver is driving carelessly, a warning mark is displayed in dangerous driving situations.

[0007] However, during actual driving of a vehicle, there are driving situations in which the driving assistance device of Patent Document 1 cannot provide an assistance display.

[0008] For example, it is assumed that a vehicle is traveling on a straight road with good visibility without any particular problems, there is no imminent danger around the vehicle, and a stable driving condition is ensured. In such driving situations, the scenery seen by the driver is monotonous and stable driving conditions are maintained, so that the driver may gradually become complacent over time and his / her attention and concentration may decrease. This state of the driver can be said to be a pre-distracted driving state, in other words, an early stage of the transition from normal driving to distracted driving, or a transitional state. In this case, it is unreasonable to determine that "the driver is driving distractedly." Furthermore, in such driving situations, if a warning mark were to be displayed as in Patent Document 1, the driver might perceive it as an excessive warning, which could cause the driver to feel uneasy or uncomfortable. Therefore, it is difficult to provide any driving assistance display at a stage prior to distracted driving.

[0009] Prior art including Patent Document 1 does not anticipate providing effective driving assistance to restore the driver's attention and concentration in the process in which the driver becomes careless due to visual monotony or the like and his or her attention and concentration decline over time, in other words, in the stage prior to mindless driving, thereby preventing mindless driving in advance. The above problems have been identified by the present inventors.

[0010] One object of the present invention is to realize display control that assists the driver in recovering attention or concentration at the beginning of or during the transition period from a normal driving state to a distracted driving state.

[0011] Other objects of the present invention will become apparent to those skilled in the art by referring to the following exemplary aspects and best modes, as well as the accompanying drawings. [Means for solving the problem]

[0012] In order to facilitate an understanding of the outline of the present invention, embodiments according to the present invention will be exemplified below.

[0013] In a first aspect, the display control device is a display control device that is mounted on a vehicle and has a control unit that controls the image display of a display device that allows a driver of the vehicle to view an image, and the control unit has a driver's state determination unit that determines the state of the driver based on information about the driver acquired while driving, a caution event detection unit that detects a caution event that is an event that the driver should pay attention to or be aware of in order to achieve safe driving based on information about the situation around the vehicle while driving, a possible danger event determination unit that determines the presence or absence of a possible danger event that is an event that may be estimated to have the possibility of leading to future danger based on the result of the determination of the driver's state, or based on the result of the determination of the driver's state and the detection result of the caution event, and an awakening support processing unit that performs an awakening support process when it is determined that the possible danger event exists, by changing the display state of an image to provide the driver with a visual stimulus that is different from that during normal driving, thereby assisting the driver in recovering his or her attention or concentration.

[0014] In the first aspect, the concept of a "potentially dangerous event (an event that can be assumed to have the potential to lead to future danger)" is introduced in order to determine the state prior to distracted driving, and when it is determined that a "potentially dangerous event" has occurred, an "awakening assistance process" is implemented to provide the driver with visual stimuli that differ from those encountered during normal driving, thereby assisting the driver in regaining attention or concentration. The presence or absence of a "possible danger event" can be determined based on the result of determining the driver's state while driving, or based on the result of determining the driver's state and the detection result of a caution-requiring event. For example, a level determination is performed on the risk possibility level, and if it is equal to or greater than a predetermined level, it can be determined that a risk possibility event has occurred (in other words, a risk possibility event has occurred). According to this aspect, it is possible to realize appropriate display control (in other words, one that does not cause the driver any discomfort or discomfort) that assists the driver in recovering attention or concentration during the initial period of transition from a normal driving state to a distracted driving state, or during the transitional period of the transition (when the driver still has enough attention to look ahead).

[0015] In a second aspect that is dependent on the first aspect, the image includes a superimposed image that is displayed so as to overlap an object as seen by the driver, and a non-superimposed image that is not intended to be superimposed on an object, and when the awakening assistance processing unit changes the display state of the image, it may change the display state of at least one of the superimposed image and the non-superimposed image, or it may change the display state of the non-superimposed image.

[0016] In a second aspect, when the displayed images (including virtual images displayed by a HUD device or the like) include a superimposed image that is superimposed on an object (for example, an image of a navigation arrow superimposed on the road surface, etc.) and a non-superimposed display that is not intended to be superimposed on an object (for example, information indicating the state of the vehicle such as a vehicle speed display, or information regarding vehicle operating restrictions such as a speed limit display, or a display of additional information such as a time display), a change in the display state can be performed on at least one of the non-superimposed image and the superimposed image.

[0017] For example, a driver whose attention or concentration has somewhat decreased due to a prolonged monotonous driving situation will visually recognize the fluctuation of the superimposed and non-superimposed images, and will be exposed to visual stimuli different from those experienced during normal driving, becoming aware that he or she is becoming distracted (in other words, being made aware), and will be able to increase his or her own attention or concentration through self-control. In this way, driver awakening support at the stage before absent-minded driving, which was previously impossible, is realized.

[0018] However, the change in the display state may be performed for non-overlapping display. Generally, the display position and display mode of the non-superimposed image do not change during normal driving. Therefore, intentionally changing the display state of the non-superimposed image has a large impact on the driver's vision, effectively providing visual stimulation, and it can be said that effective awakening assistance can be easily achieved. In addition, since the non-overlapping image is not superimposed on the object (real scene), there is no need to consider the relationship with the object (real scene) when changing the display state, which has the advantage that the change process is not complicated and there is less possibility of misleading the driver (or, more broadly, the viewer).

[0019] In a third aspect dependent on the first or second aspect, the awakening support processing unit, when changing the display state of the image, may change the display mode of the image so as to improve visibility.

[0020] In the third aspect, when changing the display state, the display aspect of the image is changed so as to improve visibility. As a specific example, it may be envisaged that the visibility of an image may be improved by changing the design of the image, such as its shape, pattern, colour, etc., changing its size, or making the image blink. This can increase the driver's attention-grabbing ability of the image, and can assist the driver in regaining his or her attention or concentration.

[0021] In a fourth aspect which is dependent on any one of the first to third aspects, the awakening assistance processing unit may perform a rocking process to rock the image when changing the display state of the image, or may perform a rocking process to rock the image in the vertical direction when the height direction of the vehicle is the vertical direction.

[0022] In the fourth aspect, when changing the display mode of the image, the image is swayed, so that the driver can be given a visual stimulus different from that during normal driving. Therefore, the effect of helping the driver recover his / her attention or concentration and encouraging awakening is obtained. The shaking direction may be the "vertical direction." When a vehicle is traveling on a road, the image may be seen shaking up and down due to the pitching effect. Therefore, it is considered that the driver is accustomed to the "vertical shaking of the image" to some extent. Therefore, when an image is given an unusual vertical shaking as an awakening process, the advantage is that the driver is less likely to feel discomfort (excessive discomfort) compared to when the image is given a lateral shaking, which is less frequently experienced during normal driving.

[0023] In a fifth aspect dependent on any one of the first to fourth aspects, when the height direction of the vehicle is the up-down direction, the display control device has a position correction unit that corrects fluctuations in the up-down display position of the image due to pitching, and when changing the display state of the image, the awakening assistance processing unit may cause the image to shake in the up-down direction by suppressing the degree of position correction in the position correction unit or disabling the position correction.

[0024] In the fifth aspect, when changing the display mode of an image, the image is shaken up and down, and the shaking is achieved by suppressing the amount of correction of a function (pitching correction function) that corrects the fluctuation in the display position of the image in the up and down direction due to pitching, or by disabling the position correction. By intentionally weakening the pitching correction function, the fluctuation in the vertical position of the image caused by pitching cannot be sufficiently corrected, which can cause the image to shake vertically. Display control devices generally have a pitching correction function, and therefore by utilizing this existing correction function to intentionally cause the image to shake up and down, it is no longer necessary to add special means for shaking, simplifying the device configuration. In addition, it is possible to realize natural swaying according to the driving environment, and it is also easy to prevent the driver from feeling excessive discomfort.

[0025] In a sixth aspect which is dependent on any one of the first to fifth aspects, the awakening support processing unit may further enhance the effect of supporting the driver's awakening by displaying in a different state when the level of the danger possibility of the danger potential event is high than when the level is low.

[0026] In the sixth aspect, the degree of change in the display state of the image is varied depending on the level of danger possibility of the danger potential event, and when the level of danger possibility is high, the effect of supporting the driver's awakening is enhanced more than when the level of danger possibility is low. For example, when a process for changing the display mode of an image is performed, it may be assumed that at least a part of the design of the image is changed to improve visibility (or eye-catchingness). Examples of changes to the design of the image may include changing from cool colors to warm colors, temporarily increasing the size of the image, or blinking the image.

[0027] When performing processing to shake an image, the effect of supporting wakefulness can be enhanced by, for example, shortening the period of the shaking or increasing the amplitude of the shaking. According to this aspect, it is possible to provide appropriate awakening assistance according to the level of danger possibility.

[0028] In a seventh aspect, a display device includes a display unit that displays an image, and a display control device according to any one of the first to sixth aspects.

[0029] According to the seventh aspect, a highly functional and user-friendly display device can be realized that is capable of display control to assist the driver in recovering attention or concentration during the initial period of transition from a normal driving state to a distracted driving state, or during the transition period of the transition.

[0030] In an eighth aspect dependent on the seventh aspect, the display device may be a projection type display device that displays an image displayed on the display unit as a virtual image on a virtual image display surface by projecting display light of the image displayed on the display unit onto a projection target.

[0031] According to the eighth aspect, a projection type display device with high functionality and improved user-friendliness can be realized. Typically, a head-up display (HUD) device or a windshield display (WSD) device can be used as the projection type display device. The term "projection type display device" is to be interpreted in a broad sense. The projection target may be a windshield of a vehicle. The projection target may also be the glasses part of a glasses-type head-mounted display. In this case, the head-mounted display may be understood as a type of head-up display in the broad sense. Therefore, it can be included in the present invention.

[0032] In a ninth aspect, a display control method is a display control method for controlling image display of a display device that allows a driver of a vehicle to view an image, and includes the steps of: determining the state of the driver based on information about the driver acquired while driving; detecting a caution event, which is an event that the driver should pay attention to or be aware of in order to achieve safe driving, based on information about the situation around the vehicle while driving; determining the presence or absence of a possible danger event, which is an event that may be presumed to have the possibility of leading to future danger, based on the result of determining the driver's state, or based on the result of determining the driver's state and the result of detecting the caution event; and when it is determined that the possible danger event exists, performing an awakening support process by changing the display state of an image to provide the driver with a visual stimulus different from that during normal driving, thereby supporting the driver in recovering his or her attention or concentration.

[0033] According to the ninth aspect, a display control method can be realized that can assist the driver in recovering attention or concentration at the beginning of the transition from a normal driving state to a distracted driving state, or during the transition period of the transition.

[0034] Those skilled in the art will easily understand that the exemplified embodiments according to the present invention can be further modified without departing from the spirit of the present invention. [Brief description of the drawings]

[0035] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a head-up display (HUD) device capable of implementing an awakening assistance process to which the present invention is applied. [Diagram 2] FIG. 2 is a diagram showing an example of the configuration of a projection-type display device (including a HUD device and a WSD device) installed in a dashboard. [Diagram 3] FIG. 3 is a diagram showing an example of a possible danger event and an example of a suitable time for awakening assistance. [Figure 4] 4A and 4B are diagrams showing an example of a change in the display position of an image in the up-down direction due to pitching. [Diagram 5] FIG. 5 is a diagram showing an example in which shaking is caused in an image by suppressing or disabling a function for correcting fluctuations in the display position of an image in the up and down direction caused by pitching (pitching correction function). [Figure 6] 6A and 6B are diagrams showing an example in which the display mode of an image is changed. [Figure 7] FIG. 7 is a diagram showing a display example in which a shaking process is performed to shake an image up and down while the vehicle is traveling. [Figure 8] FIG. 8 is a diagram showing a display example in which the image display mode is changed while the vehicle is traveling. [Figure 9] FIG. 9 is a flowchart showing an example of a main procedure of display control. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] The best mode described below is used to easily understand the present invention. Therefore, those skilled in the art should be aware that the present invention is not unduly limited by the mode described below.

[0037] (First embodiment) Please refer to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of a head-up display (HUD) device capable of implementing an awakening assistance process to which the present invention is applied. The height direction of the vehicle 1 is defined as the up-down direction (Y direction), the width direction of the vehicle 1 is defined as the left-right direction (X direction), and the front-rear direction is defined as the Z direction.

[0038] As shown in A-1 of Fig. 1, the vehicle 1 is traveling at a speed of 50 km / h on a straight road with a speed limit of 50 km / h. Visibility is good, there is not much traffic, and the driving conditions are stable.

[0039] 1, a leading vehicle 8 is braking at a considerable distance ahead of the vehicle (host vehicle) 1. This leading vehicle 8 can be determined as a cautionary object α.

[0040] Furthermore, a projection-type display device 111 is installed inside the dashboard 41. In the example of FIG.

[0041] In a broad sense, the WSD device is a type of HUD device. However, while the HUD device projects (projects) the display light of an image onto a projection target (for example, the windshield 2) via an optical system, the WSD device projects (projects) the display light of an image emitted from a display unit (such as a liquid crystal display device) directly onto a projection target without going through an optical system.

[0042] In the example of Figure 1, the WSD device 110 displays a speed limit display SL and a vehicle speed display SP as a non-overlapping image AR1 that is not intended to be superimposed on an object, at a specified position (here, a predetermined display position) on the lower side of the windshield 2.

[0043] In addition, the HUD device 100 displays an image (virtual image) AR2 of a navigation arrow figure (here, a three-dimensional arrow figure controlled by parallax display) as a superimposed image so as to be superimposed on the road surface 9 as the target object.

[0044] In the example of Fig. 1, the vehicle 1 is a right-hand drive vehicle in which the steering wheel 4 is located on the right side. The user of the projection-type display device 111 equipped with the HUD device 100 and the WSD device 110 is a driver 5 who drives the vehicle 1. In a broad sense, the "driver" can also be called a "passenger" who is in the vehicle, or a "viewer" who visually recognizes an image.

[0045] Also, as shown in A-1 of FIG. 1, the vehicle 1 is provided with a driver imaging camera (driver monitoring camera) 226 that captures the eyes, face, etc. of the driver 5, and a surrounding imaging camera 227 that captures the actual scene around the vehicle 1.

[0046] The driver imaging camera 226 constitutes a driver monitoring system (DMS) together with the driver's state determination unit 84 shown in A-2 of Fig. 1. A specific example of the determination of the driver's state will be described later.

[0047] As shown in A-2 of FIG. 1, the display system 115 has a radar unit 71 that can be used to measure the distance between an object ahead and the vehicle 1, an image processing unit 72 (including a distance measuring unit 73 and an object type / size detection unit 74), various sensors (such as an acceleration sensor) 75, an ECU (Electronic Control Unit) 77 that can collect various vehicle information, etc., a HUD device 100, and a driver imaging camera 226 and a surrounding imaging camera 227 shown in A-1 of FIG. 1.

[0048] The HUD device 100 includes a display control device (processor) 80 and a projection unit 90. The display control device (processor) 80 includes a control unit 82. The control unit 82 has a caution event detection unit 83, a driver's state determination unit 84, a potentially dangerous event determination unit 85, a position correction unit 93 corresponding to pitching (in the following description, this may be referred to as a ``pitching correction unit''), and an awakening support processing unit 94. The awakening support processing unit 94 has an image display state determination unit 86 and a display state change unit 87. In a preferred embodiment, the display state changing unit 87 includes at least one of a display mode changing unit 95 that changes the display mode of the image, a display shaking unit 96 that executes a shaking process to shake the image, and a position correction suppression unit 97 that suppresses or disables the position correction function in the position correction unit (pitching correction unit) 93 corresponding to the above-mentioned pitching.

[0049] The projection unit 90 has a display unit (a display panel such as a liquid crystal display device, a screen for displaying an image, etc.) 151 and a display optical system 92 that projects (projects) display light of the image onto the windshield 2. Note that when the WSD device 110 is used, the display optical system 92 is not provided.

[0050] Thus, the display control device 80 has a control unit 82 that is mounted on the vehicle 1 and controls image display of the display device 111 that allows the driver (passenger, viewer) 5 of the vehicle 1 to view the images AR1 and AR2.

[0051] The control unit 82 includes a driver's state determination unit 84 which determines the state of the driver 5 based on information about the driver 5 acquired while driving; an attention-requiring event detection unit 83 which detects an attention-requiring event which is an event that the driver 5 should pay attention to or be aware of in order to achieve safe driving based on information about the situation around the vehicle 1 while driving; a possible danger event determination unit 85 which determines the presence or absence of a possible danger event which is an event that can be estimated to have the possibility of leading to future danger based on the determination result of the driver 5's state or based on the determination result of the driver 5's state and the detection result of the attention-requiring event; and an awakening support processing unit 94 which, when it is determined that there is a possible danger event (for example, a possible danger event whose possible danger level is determined to be equal to or above a predetermined standard), performs an awakening support processing unit 94 which changes the display state of an image (which may include at least one of a non-superimposed image AR1 and a superimposed image AR2, but in a preferred example, the superimposed image AR2) to provide the driver 5 with a visual stimulus different from that during normal driving, thereby performing an awakening support processing to assist the driver 5 in recovering his / her attention or concentration.

[0052] More specifically, the driver's state determination unit 84 incorporates, for example, a careless driving determination algorithm capable of detecting a careless driving state by dividing it into a plurality of levels. This careless driving determination algorithm performs multifaceted determination by, for example, an AI using deep learning based on images of the eyes and face of the driver 5 captured by the driver imaging camera (driver monitoring camera) 226 and input in time series. For example, the level of distracted driving can be judged (detected) into multiple levels by analyzing changes in the driver's 5 eye position, frequency of eye movement, frequency of blinking, changes in facial direction and facial expression, etc. from multiple angles. In addition, by using not only the imaging information from the driver imaging camera 226 but also biological information such as pulse rate, the accuracy of determining the driver's condition can be further improved. The driver's state may be determined, for example, when the vehicle 1 is traveling at a speed of about 5 km / h or higher.

[0053] In addition, the "warning event" that is the target of detection by the warning event detection unit 83 is an event that the driver 5 should pay attention to or be aware of in order to achieve safe driving, and specifically, an event that can be estimated to result in a dangerous situation in the near future if the driver 5 overlooks it due to, for example, a decrease in attention or concentration. For example, as shown in A-1 of Fig. 1, a phenomenon in which a leading vehicle 8 suddenly brakes at a considerable distance can be a caution event. Other caution events can also be when a leading vehicle traveling at a certain distance ahead exhibits unusual behavior, or when a person or animal that may enter the road is detected at a certain distance away.

[0054] Furthermore, in an embodiment of the present invention, the concept of a "potentially dangerous event (an event that can be assumed to have the potential to lead to future danger)" is introduced in order to determine the state prior to distracted driving, and when it is determined that a "potentially dangerous event" has occurred, an "awakening assistance process" is implemented to provide the driver with visual stimuli that are different from those encountered during normal driving, thereby assisting the driver in regaining their attention or concentration.

[0055] The presence or absence of a "possible danger event" can be determined based on the result of determining the driver's state while driving, or based on the result of determining the driver's state and the detection result of a caution-requiring event. For example, a level determination is performed on the risk possibility level, and if it is equal to or greater than a predetermined level, it can be determined that a risk possibility event has occurred (in other words, a risk possibility event has occurred).

[0056] For example, when the driver's state determination unit 84 determines the driver's state by dividing it into three stages, namely, "initial stage of transition to a mindless driving state (first state)," "transitional period of transition to a mindless driving state (second state)," and "already transitioned to a mindless driving state (third state)," the possible danger event determination unit 85 can determine the occurrence of the first and second states (occurring events) as a possible danger event, and can also determine an event in which the occurrence of the first state (occurring event) and the occurrence of a cautionary event (occurring event) occur at the same time as a possible danger event.

[0057] Here, the possible danger event will be described in more detail with reference to Fig. 3. Fig. 3 is a diagram showing an example of a possible danger event and an example of a suitable time for awakening assistance. In Fig. 3, the same reference numerals are used to designate parts common to Fig. 1.

[0058] In FIG. 3, the possible danger events are labeled DEV. At time t1 in FIG. 3, the driver 5 has normal attention and concentration (state J0, which is a normal driving state). At time t2, as a result of the continuation of the monotonous driving situation, the driver 5 enters a state J1 in which his / her attention and concentration are somewhat reduced. In this state, the driver 5 moves his / her gaze less and pays less attention to the surroundings, blinks less, and changes in the direction and expression of his / her face become somewhat less frequent. This state J1 corresponds to the above-mentioned "initial stage of transition to absentminded driving state (first state)." At time t3, the driver 5 enters state J2, where his / her attention and concentration are lower than in state J1. In this state, for example, the driver 5 tries to recover his / her attention and concentration by exerting self-control, but they immediately return to their original state, a characteristic behavior that can be observed. This state J2 corresponds to the above-mentioned "transitional period to absentminded driving state (second state)". However, in this second state, the driver still has enough attention to gaze ahead. At time t4, the driver 5 enters state J3, which is a state in which the attention and concentration of the driver 5 are further decreased. This state J3 corresponds to the above-mentioned "shift to absentminded driving state (third state)."

[0059] In addition, in Figure 3, the occurrence of a possible danger event can be confirmed even if state J5 occurs (in the case of an isolated occurrence) in which an object requiring attention is present (a state in which an object requiring attention has been detected), for example, between times t2 and t3. Furthermore, if state J1 or state J2 and state J5 occur at the same time, the possible danger event determination unit 85 shown in FIG. 1 sets the possible danger level of the possible danger event higher.

[0060] Let us return to A-2 in FIG. 1 to continue the explanation. When the possible danger event determination unit 85 determines that a possible danger event has occurred, the awakening support processing unit 94 executes an awakening support process for supporting the awakening of the driver 5.

[0061] First, the image display state determining unit 86 determines how to change the display state of the currently displayed image.

[0062] Possible modes of changing the display state of an image include "changing the display mode (changing the shape, pattern, color, etc. of the image to improve visibility, or changing the image to a blinking state, etc.)" including the design of the image, a "first rocking mode" that forcibly rocks the image in a predetermined direction (the direction is arbitrary), and a "second rocking mode" that causes the image to rock naturally by reducing or disabling the function of position correction corresponding to vehicle vibrations such as pitching. Note that "rocking" can also be rephrased as "vibration."

[0063] In the example of A-1 in FIG. 1, for example, the "second oscillation pattern" is applied, and as a result, both the non-superimposed image (including the speed limit display SL and the vehicle speed display SP) AR1 and the superimposed image (arrow graphic for navigation) AR2 are oscillating vertically.

[0064] For example, when a driver 5 experiences a slight decline in attention or concentration due to a prolonged period of monotonous driving, the driver 5 visually recognizes the shaking of the above-mentioned images AR1 and AR2, and is thus exposed to visual stimuli different from those experienced during normal driving. The driver then becomes aware that he or she is becoming distracted, and is able to increase his or her attention and concentration by using self-control. In this way, the driver 5 can be assisted in awakening before the driver starts to drive aimlessly, which was previously impossible.

[0065] Thus, according to the example of FIG. 1, display control can be realized that assists the driver in recovering attention or concentration in the early stage or during the transition period of the transition from a normal driving state to a distracted driving state.

[0066] In the example of FIG. 1, when the displayed image (including a virtual image displayed by a HUD device or the like) includes a superimposed image (for example, an image of a navigation arrow superimposed on the road surface 9) that is superimposed on an object, and a non-superimposed display that is not intended to be superimposed on an object (for example, information indicating the state of a vehicle such as a vehicle speed display, or information regarding vehicle operation restrictions such as a speed limit display, or a display of additional information such as a time display), the change in the display state is This may be performed on at least one of the non-overlapping image AR1 and the overlapping image AR2, or may be performed on the non-overlapping image AR1 (preferably only the non-overlapping image AR1).

[0067] As described above, for example, by causing the non-overlapping image AR1 and the overlaid image AR2 to shake vertically, a driver 5 whose attention or concentration has somewhat decreased due to a prolonged monotonous driving situation can visually recognize the shaking of the above-mentioned images AR1 and AR2, and thus be exposed to visual stimuli different from those experienced during normal driving, making the driver aware that he or she is becoming distracted, and able to increase his or her attention or concentration through self-control.

[0068] However, in a preferred example, the change in the display state may be performed only on the non-overlapping image AR1. Generally, the display position and display mode of the non-superimposed image do not change during normal driving. Therefore, intentionally changing the display state of the non-superimposed image has a large impact on the driver's vision, effectively providing visual stimulation and making it easy to realize effective awakening assistance. In addition, since the non-overlapping image is not superimposed on the object (real scene), there is no need to consider the relationship with the object (real scene) when changing the display state, which has the advantage that the change process does not become complicated and also reduces the possibility of misleading the driver (or, more broadly, the viewer). A specific example of changing the display state will be described later.

[0069] Next, reference will be made to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of a display device installed in a dashboard. In Fig. 2, the same reference numerals are used to designate parts common to Fig. 1.

[0070] In FIG. 2, the display device 111 includes a HUD device 100 and a WSD (windshield display) device 110.

[0071] The HUD device 100 is housed in the dashboard 41. The HUD device 100 has a housing 103, a display unit (light source unit) 151 that emits display light for an image, and a concave mirror (curved mirror) 170 that constitutes an optical system. The concave mirror 170 projects an image (display light for the image) onto the windshield 2 as a projection target member.

[0072] A part of the display light of the HUD device 100 is reflected by the windshield 2 and enters the eye (viewpoint) 3 of the driver (passenger) 5, and as a result, a virtual image V1 is displayed on the virtual image display surface PS1. In FIG. 1 described above, an image (virtual image) AR2 of an arrow for navigation is displayed as a superimposed image on the virtual image display surface PS1.

[0073] Furthermore, by not making the curvature of the concave mirror 170 uniform, but rather appropriately setting areas of high curvature, medium curvature, low curvature, etc., the virtual image display surface can be made an inclined surface (for example, a curved inclined surface), making it possible to display a sense of depth.

[0074] The WSD device 110 is a projection display device with a short projection distance that projects the display light of an image displayed on the display surface of the display 152 directly onto the projection target (windshield) 2 without passing through an optical system. A virtual image V2 is displayed on the virtual image display surface PS2 by the WSD device 110. In FIG. 1 described above, a non-superimposed image AR1 including a speed limit display SL and a vehicle speed display SP is displayed on this virtual image display surface PS2.

[0075] Second embodiment Next, reference is made to Fig. 4. Fig. 4(A) and (B) are diagrams showing an example of fluctuation in the display position of an image in the up and down direction due to pitching.

[0076] 4A, the vehicle 1 is traveling on a flat road surface 9, but due to sudden acceleration, for example, the front end of the body of the vehicle 1 is lifted off the road surface 9, and the body of the vehicle 1 is in a backward tilted posture. The pitching angle is θa.

[0077] 4B, for example, due to sudden braking, the rear end of the body of the vehicle 1 is lifted off the road surface 9, and the body of the vehicle 1 is in a forward tilted position. The pitch angle is θb.

[0078] When pitching occurs, a misalignment (mismatch) occurs between the XYZ Cartesian coordinate system (first coordinate system) set in real space based on the road surface 9 and the local X1Y1Z1 Cartesian coordinate system (second coordinate system) that fluctuates depending on the posture of the body of the vehicle 1. This misalignment causes a fluctuation in the display position of the image. Pitching correction is an image correction that suppresses this fluctuation (basically, a coordinate conversion correction that converts the second coordinate system into the first coordinate system).

[0079] As shown by the solid lines in the upper part of Figures 4(A) and (B), before pitching occurs, a warning mark (virtual image) 70 is displayed superimposed on the object ahead (e.g., the vehicle ahead) 60, but when pitching occurs, as shown by the dashed lines, the position of the warning mark (virtual image) moves in the vertical direction (the height direction (Y direction) of the vehicle 1 in the XYZ Cartesian coordinate system). In Fig. 4(A), the amount of displacement is indicated as ΔD1. In Fig. 4(B), the amount of displacement is indicated as ΔD2. In addition, in the drawings, the reference symbol 70' is attached to the attention drawing mark (virtual image) after the positional movement has occurred.

[0080] The position correction unit (pitching correction unit) 93 corresponding to pitching shown in FIG. 1 has a function of correcting such a position shift of an image (virtual image). By using this pitching correction function, it is possible to generate a natural shaking in the image (virtual image) with a simple configuration. This point will be described below with reference to FIG. 5.

[0081] FIG. 5 is a diagram showing an example in which shaking is caused in an image by suppressing or disabling a function for correcting fluctuations in the display position of an image in the up and down direction caused by pitching (pitching correction function).

[0082] Between times t0 and t1, the vehicle 1 is in a traveling state. In A-1 of Fig. 5, in response to pitching of the vehicle 1, an angular velocity detection signal output from an angular velocity sensor (one of various sensors indicated by reference numeral 75 in Fig. 1) provided on the vehicle 1 has a vibration waveform. As shown in A-2 of FIG. 5, the attitude change detection signal output from a sensor (one of the various sensors indicated by reference numeral 75 in FIG. 1) that detects the attitude change of the vehicle 1 also has a vibration waveform. Here, as shown in A-3 of FIG. 5, pitching correction is performed to generate a display correction value. This display correction value can be obtained by inverting the polarity of the vibration waveform of A-2 in FIG. As shown in A-4 of FIG. 5, by performing position correction (pitching correction) using the display correction value, the deviation of the display position of the image (virtual image) can be sufficiently suppressed. Therefore, as shown in A-5 of FIG. 5, the display position of the non-overlapping image AR1 is maintained at approximately the same position even after pitching occurs. The above is the content of normal pitching correction.

[0083] In this embodiment, the pitching correction function in the position correction unit (pitching correction unit) 93 that corresponds to pitching is suppressed or disabled, so that a natural, unforced vertical shaking can be caused in the image (virtual image).

[0084] In the previous explanation, the process moved from A-2 in FIG. 5 to A-3 in FIG. 5, but here, the process moves from A-2 in FIG. 5 to A-6 in FIG. In A-6 of FIG. 5, the amplitude of the display correction value is approximately half that of A-3 of FIG. 5, and therefore the pitching correction function is suppressed. Therefore, the display position of the non-overlapping image AR1 is affected by pitching and wobbles in the vertical direction, as shown by A-7 and A-8 in Fig. 5. However, the amplitude of the wobbles is relatively small.

[0085] Similarly, assume a transition from A-2 in FIG. 5 to A-9 in FIG. In A-9 of Figure 5, the pitching correction function is disabled and no pitching correction is performed. Therefore, as shown by A-10 and A-11 in FIG. 5, the display position of the non-overlapping image AR1 is affected by pitching and swings relatively significantly in the vertical direction.

[0086] (Third embodiment) Please refer to Fig. 6. Fig. 6(A) and (B) are diagrams showing an example of changing the display mode of an image.

[0087] In FIG. 6(A), changes to the display mode (e.g., changing the color of at least a part of the image or blinking the image to improve visibility) can also be implemented for at least one of the speed limit display SL and the vehicle speed display SP.

[0088] However, since the speed limit display SL and the vehicle speed display SP are used to present specific information to the driver 5 during normal driving, it cannot be said that any change in their design, etc., could give the driver 5 the misunderstanding that there has been a change in the information being presented.

[0089] Therefore, in the example of Figure 6 (A), a straight line LN is added to the upper side of the speed limit display SL and the vehicle speed display SP in the non-overlapping image AR3, and the brightness of this straight line LN is changed, for example, periodically to provide a visual stimulus to the driver 5.

[0090] Since the straight line LN is not a symbol that provides specific information to the driver 5 during normal driving, the driver 5 will not be misled even if the luminance of the straight line LN is changed periodically.

[0091] In FIG. 6(B), in the non-overlapping image AR4, a circular attention recovery indicator (a kind of simple display) WR that is not used during normal driving is provided above the speed limit display SL and the vehicle speed display SP.

[0092] When a possible danger event is detected, the indicator WR can be made to blink periodically, for example, to provide assistance in restoring attention and concentration. The above examples are merely examples, and the present invention is not limited to these examples.

[0093] (Fourth embodiment) Please refer to Fig. 7. Fig. 7 is a diagram showing a display example in which a shaking process is performed to shake an image up and down while the vehicle is traveling.

[0094] In A-1 of FIG. 7, it is assumed that the driver 5 is in a state J2 (a state in which attention and concentration are somewhat decreased) shown in A-2 of FIG. When it is detected that the driver 5 is in state J2, it is determined that a possible danger event has occurred, and an awakening assistance process is carried out.

[0095] Here, in A-1 of Fig. 7, there is no preceding vehicle or the like, and there is no object requiring attention. Therefore, in A-1 of Fig. 7, the non-superimposed display is swung up and down (the amplitude of the swung is small) as shown in A-8 of Fig. 5, thereby helping the driver 5 to recover his / her attention and concentration.

[0096] In A-3 of FIG. 7, the driver 5 is also in state J2 (a state in which attention and concentration are somewhat decreased) shown in A-2 of FIG. However, in A-3 in Fig. 7, the preceding vehicle 8 is braking, and this preceding vehicle 8 may be recognized as a cautionary object α. Therefore, A-3 in Fig. 7 may be said to be a state with a higher danger possibility level than A-1 in Fig. 7.

[0097] Therefore, in A-3 of FIG. 7, the non-superimposed display is caused to shake up and down as shown in A-11 of FIG. 5 previously (the amplitude of the shaking is large and the image's ability to assist in awakening is enhanced), thereby helping the driver 5 to regain their attention and concentration.

[0098] Fifth embodiment FIG. 8 is a diagram showing a display example in which the image display mode is changed while the vehicle is traveling.

[0099] The driving conditions at A-1 and A-3 in FIG. 8 are the same as the driving conditions at A-1 and A-3 in FIG.

[0100] In the example of FIG. 8, in the non-overlapping image AR5, the straight line LN previously shown in FIG. 6(A) and the indicator WR for restoring attention shown in FIG. 6(B) are used in combination.

[0101] In A-1 of FIG. 8, the straight line LN remains unchanged, while the indicator WR for restoring attention flashes, thereby carrying out the wakefulness support process.

[0102] In A-3 of FIG. 8, in addition to the blinking of the attention recovery indicator WR, a process of periodically changing the luminance of the straight line LN is performed, thereby enhancing the awakening assistance capability of the image in the awakening assistance process.

[0103] Sixth embodiment Please refer to Fig. 9. Fig. 9 is a flowchart showing an example of a main procedure of display control.

[0104] In step S1, the driver's state is determined (driver state determination process) by a driver monitoring system (DMS) or the like, and the presence or absence of a caution-requiring object is detected (caution-requiring object detection process).

[0105] In step S2, a process for determining the occurrence of a possible danger event is carried out (a process for predicting the possibility of danger).

[0106] In step S3, it is determined whether or not there is a possible danger event. If the answer is No, the process returns to step S1, and if the answer is Yes, the process proceeds to step S4.

[0107] In step S4, a process of determining the display state (including the display mode) of the image is carried out.

[0108] In step S5, the display state of the image is changed (display state change process).

[0109] In step S6, it is determined whether the possible danger event has disappeared. If the result is N, the determination in step S6 is continued, and if the result is Y, the process proceeds to step S7. In step S7, a process of returning the display state to the original state is performed.

[0110] In step S6, a level lower than the danger possibility level that is the standard for determining the presence or absence of a danger possibility event in step S3 is set, and it is determined that the danger possibility event has disappeared when the level falls below this lower level. This allows the display state to be restored to its original state only when the danger possibility level has sufficiently decreased, thereby preventing a situation in which the process of changing the display state and the process of restoring the changed display state to its original state are repeated in a short period of time.

[0111] In step S8, it is determined whether or not driving has ended. If the answer is Y, the display control ends, and if the answer is N, the process returns to step S1.

[0112] As described above, according to the present invention, it is possible to realize display control that assists a driver in recovering attention or concentration during the initial period of transition from a normal driving state to a distracted driving state, or during the transition period of the transition.

[0113] The HUD device or display system also includes those used as simulators (for example, aircraft simulators, simulators as game devices, etc.).

[0114] In the above-described embodiment, the projection type display device has been mainly described as an example of a HUD device, but the present invention is not limited to this, and for example, an in-car projector or the like can also be used.

[0115] The present invention is not limited to the above-described exemplary embodiments, and those skilled in the art could easily modify the above-described exemplary embodiments to the extent that they fall within the scope of the claims. [Explanation of symbols]

[0116] 1···vehicle (own vehicle), 2···projected part (windshield, etc.), 3···eye (viewpoint), 4···steering wheel, 5··driver (occupant, viewer), 8···vehicle ahead, 9··road surface, 41···dashboard, 71···radar unit, 72···image processing unit, 73···distance measuring unit, 74···object type / size detection unit, 75···various sensors (acceleration sensor, etc.), 77 ECU, 80 display control device, 82 control unit, 83 caution event detection unit, 84 driver state determination unit, 85 possible danger event determination unit, 86 image display state determination unit, 87 display state change unit, 90 projection unit, 92 display optical system, 93 position correction unit corresponding to pitching (pitching correction unit), 94 awakening support processing unit, 95 96...display mode changing unit, 97...position correction suppressing unit, 100...HUD device, 103...housing, 110...WSD device, 111...display device (projection type display device), 151...display unit (light source unit), 152...display device, 170...concave mirror (curved mirror), 226...driver imaging camera (driver monitoring camera), 227...surrounding imaging camera, PS 1, PS2...virtual image display surface, V1, V2...image (virtual image), AR1, AR3 to AR5...non-superimposed image, AR2...superimposed image (arrow shape as superimposed image, three-dimensional arrow shape, arrow shape for navigation), α...object requiring caution, SL...speed limit display, SP...vehicle speed display, LN...straight line (straight line symbol), WR...indicator (indicator for restoring attention).

Claims

1. A display control device having a control unit mounted on a vehicle and controlling image display of a display device that allows a driver of the vehicle to view an image, The control unit is A driver state determination unit that determines a state of the driver based on information about the driver acquired during driving; an attention-requiring event detection unit that detects an attention-requiring event that is an event that the driver should be aware of or pay attention to in order to achieve safe driving, based on information about a situation around the vehicle while driving; a risk possibility event determination unit that determines the presence or absence of a risk possibility event, which is an event that can be estimated to have the possibility of leading to a future danger, based on the result of the determination of the driver's state, or based on the result of the determination of the driver's state and the detection result of the caution event; an awakening support processing unit that performs an awakening support process for supporting the recovery of the driver's attention or concentration by changing a display state of an image when it is determined that the risk possibility event has occurred, and providing the driver with a visual stimulus different from that during normal driving, the visual stimulus being different from that during normal driving; having Display control device.

2. The images include a superimposed image that is displayed so as to be superimposed on an object as viewed by the driver, and a non-superimposed image that is not intended to be superimposed on an object, When changing the display state of the image, the awakening support processing unit: changing a display state of at least one of the superimposed image and the non-superimposed image; Or, changing a display state of the non-overlapping image; The display control device according to claim 1 .

3. When changing the display state of the image, the awakening support processing unit: Changing the display mode of the image so as to improve visibility. The display control device according to claim 1 .

4. When changing the display state of the image, the awakening support processing unit: performing a shaking process for shaking the image; Or, When the height direction of the vehicle is the up-down direction, a swinging process is performed to swing the image in the up-down direction. The display control device according to claim 1 .

5. When a height direction of the vehicle is defined as a vertical direction, the display control device has a position correction unit that corrects a fluctuation in a display position of the image in a vertical direction caused by pitching, When changing the display state of the image, the awakening support processing unit: suppressing a degree of position correction in the position correction unit, or disabling the position correction, thereby causing vertical shaking in the image; The display control device according to claim 1 .

6. The awakening assistance processing unit, when the level of the danger possibility of the danger possibility event is high, displays the event in a different state from when the level of the danger possibility event is low, thereby enhancing the effect of assisting the driver in awakening. The display control device according to claim 1 .

7. A display unit for displaying an image; A display control device according to any one of claims 1 to 6, A display device having the above configuration.

8. The display device is a projection type display device that projects display light of an image displayed on the display unit onto a projection target, thereby displaying the image as a virtual image on a virtual image display surface. The display device according to claim 7.

9. A display control method for controlling an image display on a display device that allows a driver of a vehicle to view an image, comprising: determining a state of the driver based on information about the driver acquired during driving; Detecting an attention-requiring event, which is an event that the driver should be aware of or pay attention to in order to achieve safe driving, based on information about a situation around the vehicle while driving; A step of determining whether or not there is a risk potential event, which is an event that may be estimated to have the possibility of leading to a future danger, based on the result of the determination of the driver's state, or based on the result of the determination of the driver's state and the result of the detection of the caution event; a step of performing an awakening support process for supporting the recovery of the driver's attention or concentration by changing a display state of an image when it is determined that the risk-prone event has occurred, and providing the driver with a visual stimulus different from that during normal driving; Including, Display control method.

Images