Wakefulness maintaining device for driver

Abstract

To improve a wakefulness maintaining effect with respect to a driver.SOLUTION: A wakefulness maintaining device includes: a head-up display 10 capable of displaying a specified image I, which is an image below the perception threshold of the driver's visual perception, in front of the driver; speakers 9R, 9L, and 9U installed in a cabin of a vehicle and capable of outputting a specified sound image that is below the perception threshold of the driver's auditory perception; and an ECU 50 electrically connected to the head-up display 10 and the speakers 9R, 9L, and 9U. The ECU 50 controls the head-up display 10 and the speakers 9R, 9L, and 9U so that the specified image I and the specified sound image are output in a time-correlated manner.SELECTED DRAWING: Figure 5C

Description

[Technical Field]

[0001] The technology disclosed herein belongs to the technical field of a driver wakefulness maintaining device. [Background technology]

[0002] Patent document 1 discloses a device that maintains the driver's state of alertness by displaying a specific image, which is lower than the driver's visual perception threshold, in the driver's field of view when it is determined that the driver's state of alertness is declining. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2022-144989 Summary of the Invention [Problem to be solved by the invention]

[0004] Meanwhile, studies are being conducted to more effectively maintain the driver's wakefulness by providing the driver with not only visual stimuli but also auditory and tactile stimuli.

[0005] As a result of extensive research, the inventors of the present invention have found that it is difficult to improve the wakefulness-maintaining effect on the driver by simply adding other stimuli.

[0006] The technology disclosed herein has been made in view of the above points, and its purpose is to improve the effect of maintaining wakefulness in the driver. [Means for solving the problem]

[0007] In order to solve the above problem, a first aspect of the technology disclosed herein is directed to a device for maintaining wakefulness of a vehicle driver, and comprises a display unit capable of displaying a specific image in front of the driver, the specific image being an image lower than the driver's visual perception threshold, an audio unit capable of outputting a specific sound image in the passenger compartment of the vehicle, the specific sound image being a sound image lower than the driver's auditory perception threshold, and a control unit electrically connected to the display unit and the audio unit, and the control unit controls the display unit and the audio unit so that the specific image and the specific sound image are output in a time-correlated manner.

[0008] The inventors of the present application have found that even when an image is displayed that is lower than the perception threshold that the driver can perceive through his / her eyes, the driver's brain recognizes the specific image as a stimulus. Similarly, they have found that even when a sound image is output that is lower than the perception threshold that the driver can perceive through his / her eyes, the driver's brain recognizes the specific sound image as a stimulus. When the brain recognizes a stimulus, the coordination of brain activity increases, and projections from the cerebrum to the hypothalamus become stronger. The stronger projections to the hypothalamus improve the effect on the sympathetic nervous system. This increases sympathetic nervous activity, and the driver becomes more alert.

[0009] Further research by the inventors of the present invention has revealed that by temporally correlating visual and auditory stimuli, the brain stimulation is stronger than when there is no correlation between the visual and auditory stimuli. The stronger the brain stimulation, the more likely the driver is to be alert. Therefore, the effect of maintaining alertness in the driver can be improved.

[0010] A second aspect of the technology disclosed herein is the first aspect, in which the control unit displays the specific image on the display unit by changing the transparency of the specific image over time, and outputs the specific sound image to the acoustic unit by synchronizing the change in amplitude of the specific sound image with the change in transparency of the specific image over time.

[0011] In the second aspect, by synchronizing the change in the amplitude of the specific sound image with the change in the transparency of the specific image in time series, it is possible to enhance the temporal correlation between the visual and auditory stimuli, thereby improving the effect of maintaining the driver's wakefulness.

[0012] A third aspect of the technology disclosed herein is the first aspect, in which the control unit controls at least one of the display unit and the audio unit so that the specific image and the specific sound image are output in a spatially correlated manner.

[0013] In the third aspect, the specific image and the specific sound image are correlated not only temporally but also spatially, thereby improving the effect of keeping the driver awake.

[0014] A fourth aspect of the technology disclosed herein is the third aspect, in which the control unit controls the audio unit so that the positioning of the specific sound image output by the audio unit matches the display area of ​​the specific image displayed by the display unit.

[0015] In the fourth aspect, the spatial correlation between the visual and auditory stimuli can be increased, thereby improving the effect of keeping the driver awake.

[0016] A fifth aspect of the technology disclosed herein is the fourth aspect, wherein the acoustic unit is a plurality of speakers, and the plurality of speakers are arranged on the right, left, and upper sides of the driver's seat in the vehicle cabin.

[0017] In the fifth aspect, the localization of the specific sound image can be easily adapted to the display area of ​​the specific image, thereby increasing the spatial correlation between the visual and auditory stimuli and improving the effect of maintaining wakefulness in the driver.

[0018] A sixth aspect of the technology disclosed herein is any one of the first to fifth aspects, wherein the specific image is an image with a transparency of 94±2%, and the specific sound image is a sound image with a frequency of 40 kHz or higher.

[0019] In the sixth aspect, by setting the transparency of the specific image to 94±2%, it is possible to appropriately stimulate the driver's brain without causing annoyance to the driver. Also, since the human audible range is generally 20Hz to 20kHz, a sound image with a frequency of 40kHz or higher is difficult for the driver to perceive. Therefore, it is possible to improve the effect of maintaining the driver's wakefulness without causing annoyance to the driver. [Effects of the Invention]

[0020] As described above, the technology disclosed herein can improve the effect of keeping the driver awake. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 is a schematic diagram showing the interior of a vehicle equipped with an alertness maintaining device according to this embodiment. [Figure 2] FIG. 2 is a block diagram showing a control system of the vehicle. [Figure 3] FIG. 3 is a diagram showing a drive simulation used in the experiment. [Figure 4] FIG. 4 is a diagram illustrating the positions of images shown in a driving simulation and the positions of sound image localization. [Figure 5A] FIG. 5A is a diagram illustrating an example of a time-series change in transparency of a specific image. [Figure 5B] FIG. 5B is an explanatory diagram showing the process of generating a specific sound image. [Figure 5C] FIG. 5A is a comparison diagram of a time series change in transparency of a specific image and a change in amplitude of a specific sound image. [Figure 6] FIG. 6 is a graph comparing the amount of sympathetic nerve activity in the case of visual stimulation alone with the case of visual and auditory stimulation. [Figure 7] FIG. 7 is a graph comparing the amount of sympathetic nerve activity depending on whether or not there is a temporal coincidence between visual and auditory stimuli and whether or not there is a spatial coincidence between them. [Figure 8]FIG. 8 is a block diagram illustrating an input system and an output system in the wakefulness maintenance control of the ECU. [Figure 9] FIG. 9 is a diagram for explaining the analysis by the eyelid opening / closing degree analysis unit. [Figure 10] FIG. 10 is a flowchart showing the processing operations in the wakefulness maintenance control of the ECU. DETAILED DESCRIPTION OF THE INVENTION

[0022] Exemplary embodiments will now be described in detail with reference to the drawings.

[0023] (1) Vehicle configuration The interior of a vehicle equipped with an alertness maintenance device according to this embodiment is shown. The vehicle is a four-wheeled automobile, and two of the four wheels (e.g., the front wheels), which are positioned symmetrically on the left and right, are driven by a drive unit (not shown). This allows the vehicle to move (travel).

[0024] As shown in FIG. 1, the vehicle is a right-hand drive vehicle, with a steering wheel 8 located on the right side. Inside the vehicle cabin, a windshield glass 1 is located in front of the driver's seat. When viewed from inside the vehicle cabin, the windshield glass 1 is defined by a number of vehicle components. Specifically, the windshield glass 1 is defined by left and right front pillar trims 2, a roof trim 3, and an instrument panel 4.

[0025] The left and right front pillar trims 2 respectively constitute outer boundaries in the vehicle width direction of the front window glass 1. Each front pillar trim 2 is disposed along each front pillar.

[0026] The roof trim 3 forms the upper boundary of the front windshield glass 1. The roof trim 3 covers the interior side of the vehicle's roof panel. A rearview mirror 5 is attached to the center of the front windshield glass 1 in the vehicle width direction and slightly below the roof trim 3.

[0027] The instrument panel 4 forms the lower boundary of the front windshield 1. A meter box and a center display 7 are provided on the instrument panel 4. A driver monitor camera 101 (see FIG. 2) that captures images of the interior of the vehicle, particularly the face of the driver, is provided near the center display 7. By locating the driver monitor camera 101 near the center display 7, it is possible to capture images of the driver's eyes without being obstructed by the driver's bangs or eyelashes.

[0028] The vehicle also has a head-up display 10 (hereinafter referred to as HUD 10) as a display device that displays an image in a field of view in front of the driver. Specifically, the HUD 10 displays a virtual image on the windshield 1. The HUD 10 projects an image onto the windshield 1 from below, thereby displaying a virtual image of the image in a predetermined area on the windshield 1, including the field of view of the driver. In the first embodiment, the display range of the HUD 10 is set to the entire windshield 1. The windshield 1 and the HUD 10 are an example of a display unit that is located in front of the driver and displays a virtual image.

[0029] The vehicle also has side mirrors 6 located on the outside of the left and right front pillars in the vehicle width direction. Each side mirror 6 is positioned so that the driver sitting in the driver's seat can see through the window of the side door.

[0030] The vehicle is equipped with multiple (here, three) speakers 9R, 9L, and 9U. The speakers include a right speaker 9R, a left speaker 9L, and an upper speaker 9U. The right speaker 9R is located, for example, inside the right side door. The left speaker 9L is located, for example, inside the left side door. The upper speaker 9U is located, for example, inside the roof. The volume and sound direction of each speaker 9R, 9L, and 9U can be adjusted independently. The positioning of the sound image output from the speakers 9R, 9L, and 9U in the vehicle cabin can be adjusted by adjusting the phase, volume, sound direction, etc. of the sound of the speakers 9R, 9L, and 9U. The speakers 9R, 9L, and 9U are an example of an acoustic unit.

[0031] (2) Control system In this embodiment, the wakefulness maintaining device controls the HUD 10 and each of the speakers 9R, 9L, and 9U according to the driver's state. The wakefulness maintaining device includes an ECU 50 (Electrical Control Unit) that controls the HUD 10. The ECU 50 is a controller based on a well-known microcomputer and includes a central processing unit (CPU) 50a, a memory 50b, and an I / F circuit 50c. The CPU 50a executes programs. The memory 50b is configured, for example, by a random access memory (RAM) or a read-only memory (ROM) and stores programs and data. The I / F circuit 50c inputs and outputs electrical signals. The ECU 50 is an example of a control unit.

[0032] As shown in Figure 2, the ECU 50 generates a control signal to the HUD 10 based on information input from a plurality of sensors and external terminals. The sensors that input information to the ECU 50 include: a plurality of front cameras 100 for capturing images of the area in front of the vehicle; A driver monitor camera 101 that captures the driver's upper body including the face, a vehicle speed sensor 102 for detecting the traveling speed of the vehicle; an acceleration sensor 103 for detecting acceleration in the longitudinal direction of the vehicle; a yaw rate sensor 104 for detecting the rotational angular velocity (yaw rate) of the vehicle around the vertical axis; a steering angle sensor 105 for detecting the steering angle of the driver's steering wheel 8; a steering torque sensor 106 for detecting the steering torque of the driver's steering wheel 8; an accelerator opening sensor 107 for detecting the amount of operation of the accelerator pedal; a brake sensor 108 for detecting the amount of operation of the brake pedal; a GPS sensor 109 that detects the position of the vehicle using a Global Positioning System (GPS); Includes.

[0033] The driver monitor camera 101 is a camera equipped with an LED that emits near-infrared light, and can capture images by illuminating the driver's face with this LED without being affected by external light.

[0034] As will be described in more detail later, the ECU 50 estimates the driver's level of alertness based on information from the driver monitor camera 101, vehicle speed sensor 102, acceleration sensor 103, yaw rate sensor 104, steering angle sensor 105, steering torque sensor 106, accelerator opening sensor 107, brake sensor 108, and GPS sensor 109, and controls the HUD 10 and each of the speakers 9R, 9L, and 9U based on the estimated level of alertness.

[0035] The ECU 50 outputs control signals to a steering control device 61, a brake control device 62, and an engine control device 63 based on information from the sensors 100 to .

[0036] The steering control device 61 is a device for controlling an EPAS (Electronic Power Assist Steering) (not shown). Specifically, it is a device for controlling the EPAS so that a desired assist torque is output. The ECU 50 generates a control signal for the steering control device 61 based mainly on information from a vehicle speed sensor 102, a steering angle sensor 105, and a steering torque sensor 106.

[0037] The brake control device 62 is a device for controlling a brake device (not shown) so as to obtain a desired braking force. The ECU 50 generates a control signal for the brake control device 62 based mainly on information from the vehicle speed sensor 102 and the brake sensor 108.

[0038] The engine control device 63 is a device for controlling the engine (not shown) so as to obtain a desired driving force. Specifically, the engine control device 63 controls the fuel injection amount, fuel injection timing, ignition timing of the spark plug, etc. The ECU 50 generates a control signal for the engine control device 63 mainly based on information from the vehicle speed sensor 102 and the accelerator position sensor 107.

[0039] Furthermore, as will be described in detail later, when an abnormality occurs with the driver, the ECU 50 sends control signals to a steering control device 61, a brake control device 62, and an engine control device 63 to bring the vehicle to an emergency stop.

[0040] (3) Relationship between visual and auditory stimuli and driver alertness The present inventors have studied ways to stimulate the driver's brain through the driver's vision and hearing in order to maintain the driver's level of alertness as high as possible. In particular, the present inventors have studied ways to maintain the driver's level of alertness while not causing the driver to feel annoyed. Note that a state in which the driver's level of alertness is low refers to, for example, a state in which the driver feels drowsy or is absent-minded and has reduced attention.

[0041] The inventors of the present application conducted an experiment on a plurality of test subjects using a driving simulator 80 as shown in Fig. 3. In the experiment, while the test subjects were driving, an image serving as noise (hereinafter referred to as a specific image I) was displayed on the screen of the driving simulator 80, and a sound image serving as noise (hereinafter referred to as a specific sound image) was output from the speaker 81. In the following description, the display of the specific image I may be referred to as a visual stimulus, and the output of the specific sound image may be referred to as an auditory stimulus.

[0042] As shown in Figure 4, a specific image I was displayed on the driving simulator 80 at a position overlapping with the road in the visual field of the subject. In Figure 4, the display position is shown hatched to make it easier to understand, but the specific image I actually displayed is an image with high transparency that is lower than the visual perception threshold of the subject. The visual perception threshold refers to the threshold at which a subject can visually recognize an image. More specifically, an "image above the perception threshold" refers to an image that the subject can visually recognize, and an "image below the perception threshold" refers to an image that the subject cannot visually recognize.

[0043] A sound image (hereinafter referred to as a specific sound image) lower than the visual perception threshold of the subject was output from the speaker 81. The auditory perception threshold refers to the threshold at which a subject can recognize a sound image through their hearing. More specifically, a "sound image above the perception threshold" refers to a sound image that the subject can recognize through their hearing, and a "sound image below the perception threshold" refers to a sound image that the subject cannot recognize through their hearing.

[0044] The speaker 81 is a portable speaker. The sound image localization of the speaker 81 can be changed by changing the position of the speaker 81 itself. For example, if the speaker 81 is placed between the screen of the driving simulator 80 and the driver's seat, the sound image can be localized at a first position SP1 shown in FIG. 4. If the speaker 81 is placed next to the driver's seat, the sound image can be localized at a second position SP2 shown in FIG. 4. When the sound image is localized at the first position SP1, the sound image is located within the field of view of the subject, and the display area of ​​the specific image I and the sound image localization are matched. On the other hand, when the sound image is localized at the second position SP2, the sound image is located outside the field of view of the subject, and the display area of ​​the specific image I and the sound image localization are not matched.

[0045] The inventors of the present application conducted measurements to clarify the visual perception threshold of the subjects. Specifically, (i) An image with transparency between 80.0% and 99.5% is displayed while increasing the transparency in increments of 0.5% from 80.0%, and participants are asked to report when the image becomes invisible. (ii) Images with transparency between 80.0% and 99.5% are displayed while decreasing the transparency in increments of 0.5% from 99.5%, and participants are asked to declare when they can see the image. These two measurements were taken and their average value was calculated. The image to be displayed was a monochrome image with a brightness of 50%. Note that the transparency here is such that 0% is completely opaque and 100% is completely transparent and nothing can be seen through.

[0046] As a result of these measurements, it was found that subjects could not perceive a transparency of approximately 92%, i.e., the perception threshold was between 91% and 92% transparency. For this reason, specific image I was made an image with a transparency of 94% ± 2%, so that 92% transparency was the lower limit.

[0047] The inventors of the present application also set the specific image I to an image with a brightness of 50%±2%. Brightness is a parameter expressed from 0% to 100%, with the image appearing whiter as it approaches 100% and blacker as it approaches 0%. Therefore, when the brightness of an image is high, the driver can easily perceive the image in dark surroundings, such as inside a tunnel or at night. Conversely, when the brightness is low, the driver can easily perceive the image in bright surroundings. Therefore, by setting the brightness of the specific image to 50±2%, the specific image I can be blended as much as possible into the scenery outside the vehicle (particularly the scenery ahead of the vehicle). This makes it difficult for the driver to perceive it visually.

[0048] The inventors of the present application repeatedly performed control to display, as visual stimuli, images randomly selected from a plurality of specific images I with a transparency of 94%±2% and a brightness of 50%±2% on the screen of the driving simulator 80. The images to be displayed were different images, each with a transparency within the range of 94%±2% and a brightness of 50%±2%. FIG. 5A shows an example of the change in transparency of the displayed specific image I. In the control, the inventors displayed the specific image I so that the change in transparency was a 1 / f fluctuation. The frequency at which the specific image I was displayed was, for example, 30 Hz.

[0049] The inventors of the present application set the auditory perception threshold of the test subject with reference to the human audible range. The human audible range is generally 20 Hz to 20 kHz. Taking into consideration individual differences in the audible range, the inventors of the present application set a sound image with a frequency of 40 kHz or more, which is more than twice the highest frequency of the audible range, as the specific sound image. The frequency of the specific sound image is not particularly limited as long as it is 40 kHz or more, but is, for example, 48 kHz.

[0050] FIG. 5B shows the process of generating a specific sound image. The upper diagram of FIG. 5B is the sound image before processing, and the lower diagram of FIG. 5B is the sound image after processing. The sound image shown in the lower diagram of FIG. 5B corresponds to the specific sound image. The inventors changed the amplitude of the sound image shown in the upper diagram of FIG. 5B (AM modulation) so that the change in amplitude of the sound image becomes a 1 / f fluctuation. Specifically, as shown in FIG. 5C, the inventors changed the amplitude of the sound image so that the change in transparency of specific image I and the change in amplitude of the sound image are temporally correlated. In particular, the inventors changed the amplitude of the sound image so that the change in transparency of specific image I and the change in amplitude of the sound image are synchronized in time series to create a specific sound image. Note that FIG. 5C shows only the peak side of the waveform of the specific sound image to make it easier to compare the change in transparency of specific image I and the change in amplitude of the specific sound image.

[0051] Figure 6 shows a comparison of the average sympathetic nerve activity when only visual stimulation is applied and when auditory stimulation is applied in addition to visual stimulation. "Visual stimulation only" refers to the case where no specific sound image is output from speaker 81. The higher the level of arousal, the greater the sympathetic nerve activity, and the lower the level of arousal (the more relaxed one is), the smaller the level of sympathetic nerve activity.

[0052] Figure 6 shows that sympathetic nerve activity increases when auditory stimulation is added compared to when only visual stimulation is used. In other words, the level of arousal is higher when visual stimulation and auditory stimulation are combined compared to when only visual stimulation is used. This is thought to be because the stochastic resonance effect is enhanced by combining visual stimulation and auditory stimulation.

[0053] Figure 7 shows the results of an investigation into the stochastic resonance effect between visual and auditory stimuli. The stochastic resonance effect is believed to be enhanced when there is temporal and spatial correlation between visual and auditory stimuli. The inventors of the present application investigated the temporal and spatial correlation between visual and auditory stimuli and the stochastic resonance effect. The temporal correlation was evaluated based on whether or not there was a match between the change in transparency of specific image I and the change in amplitude of the specific sound image. Specifically, as shown in Figure 5C, a state in which the change in transparency of specific image I and the change in amplitude of the specific sound image were synchronized was defined as a state in which there was temporal correspondence between the visual and auditory stimuli. A state in which there was no temporal correspondence between the visual and auditory stimuli was defined as a state in which the sound image was output without AM modulation. Spatial correspondence was evaluated based on the positional relationship between specific image I and sound image localization. Specifically, as shown in Figure 4, a state in which the sound image localization was located within the visual field (located at the first position SP1) was defined as a state in which there was spatial correspondence between the visual and auditory stimuli. The state in which there was no spatial correspondence between the visual and auditory stimuli was defined as the state in which the sound image was located outside the visual field (located at the second position SP2).

[0054] The inventors of the present application have examined the amount of sympathetic nerve activity by dividing it into four patterns depending on whether or not there is a temporal and spatial correspondence between visual and auditory stimuli. The four patterns are: Pattern 1: Temporally inconsistent and spatially inconsistent Pattern 2: Temporally inconsistent and spatially consistent Pattern 3: Temporally coincident but spatially inconsistent Pattern 4: Temporally and spatially consistent FIG. 7 shows the ratio of sympathetic nerve activity in Pattern 1 to the other patterns. As shown in FIG. 7, in Pattern 2, the sympathetic nerve activity is only slightly increased compared to Pattern 1. On the other hand, in Pattern 3, the sympathetic nerve activity is increased compared to Patterns 1 and 2. Furthermore, in Pattern 4, the sympathetic nerve activity is significantly increased compared to Patterns 1, 2, and 3. In particular, the sympathetic nerve activity is significantly increased compared to Patterns 2 and 3 combined. This shows that while a certain degree of stochastic resonance effect occurs even with only temporal coincidence between visual and auditory stimuli, the addition of spatial coincidence between visual and auditory stimuli significantly enhances the stochastic resonance effect. Therefore, it can be said that the stochastic resonance effect between visual and auditory stimuli provides a strong stimulus to the brain, resulting in a wakefulness maintenance effect.

[0055] (4) The process of maintaining wakefulness Fig. 8 shows the process of wakefulness maintenance control. As shown in Fig. 8, the EUC 50 receives as inputs the steering torque detected by the steering torque sensor 106, the longitudinal acceleration detected by the acceleration sensor 103, the angular velocity detected by the yaw rate sensor 104, the vehicle speed input by the vehicle speed sensor 102, the driver image acquired by the driver monitor camera 101, and the forward image acquired by the front camera 100.

[0056] The ECU 50 has an operation analysis unit 51, a continuous driving time calculation unit 52, an eyelid opening / closing degree analysis unit 53, a roadway analysis unit 54, and a wakefulness reduction determination unit 55. The ECU 50 also has an image output control unit 56, an audio image output control unit 57, and an abnormality vehicle control unit 58. The operation analysis unit 51, the continuous driving time calculation unit 52, the eyelid opening / closing degree analysis unit 53, the roadway analysis unit 54, the wakefulness reduction determination unit 55, the image output control unit 56, the audio image output control unit 57, and the abnormality vehicle control unit 58 are functional units that are executed by the CPU 50a reading programs stored in the memory 50b.

[0057] The operation analysis unit 51 analyzes the magnitude of acceleration / deceleration by the driver, the amount of operation of the steering wheel 8, the operation timing, and the like.

[0058] The continuous driving time calculation unit 52 calculates the continuous driving time by the same driver. The continuous driving time calculation unit 52 calculates the time from when the vehicle speed becomes greater than 0 until it becomes 0 again as the continuous driving time.

[0059] The eyelid open / closed degree analysis unit 53 detects the proportion of time the eyelids are open / closed based on the driver image. For example, as shown in Fig. 9, the eyelid open / closed degree analysis unit 53 analyzes the proportion of time the eyelids are open / closed.

[0060] The roadway analysis unit 54 analyzes the shape of the roadway on which the vehicle is traveling and the shape of the roadway ahead of the vehicle where the vehicle is scheduled to travel. In particular, the roadway analysis unit 54 analyzes whether or not there are curves, intersections, crosswalks, etc. ahead of the vehicle.

[0061] The alertness reduction determination unit 55 determines the decline in the level of alertness of the driver. The alertness reduction determination unit 55 determines the decline in the level of alertness of the driver based on the results of the operation analysis unit 51, the continuous driving time calculation unit 52, the eyelid opening / closing degree analysis unit 53, and the driving road analysis unit 54. The alertness reduction determination unit 55 evaluates the decline in the level of alertness of the driver, for example, on the following five levels: Level 1: Calm state (fast and frequent eye movements, stable blinking cycle), Level 2: Slightly decreased alertness (lips open, slow eye movement), Level 3: Decreased alertness (slow and frequent blinking, sitting upright, hands on face), Level 4: Significantly reduced alertness (conscious blinking, frequent yawning and deep breathing, slow blinking); Level 5: Very decreased alertness (eyelids closed, head tilted back and forth).

[0062] The arousal decline determination unit 55 determines the driver's level of alertness based on information such as the presence or absence of sudden acceleration, sudden deceleration, or sharp turns by the driver and the number of such occurrences, delays in acceleration and turning operations, continuous driving time, the proportion of eyelid opening and closing times, and eye gaze movements. The arousal decline determination unit 55 determines this driving-related information in association with the shape of the road to determine whether a sudden change in acceleration or deceleration or a sudden increase in angular velocity is due to a decrease in the driver's level of alertness or due to the shape of the road, and to determine delays in acceleration and turning operations. The arousal decline determination unit 55 determines the driver's eye gaze movement in association with the road to determine whether the driver's eye gaze movement is due to inattentive driving caused by a decrease in alertness or a movement to check for safety.

[0063] When the driver's level of alertness is decreasing but remains relatively high (for example, below level 3), the alertness decrease determination unit 55 outputs a control signal to the image output control unit 56 and the sound image output control unit 57 to maintain the driver's level of alertness at a high level.

[0064] The image output control unit 56 repeatedly controls the HUD 10 to display an image randomly selected from a plurality of pre-stored specific images I at a predetermined timing. That is, the image output control unit 56 randomly selects an image from the plurality of specific images I at a certain timing and displays the selected image on the HUD 10, and then randomly selects another image from the plurality of specific images I at the next timing and displays the selected image on the HUD 10. The specific image I displayed by the HUD 10 is an image below the driver's perception threshold, as used in the aforementioned experiment, i.e., a monochromatic image with a transparency of 94%±2% and a brightness of 50%±2%. The plurality of pre-stored specific images I each have different transparency and brightness within the ranges of 94%±2% for transparency and 50%±2% for brightness. The image output control unit 56 displays the image so that the change in transparency of the specific image I exhibits 1 / f fluctuation. The predetermined timing is, for example, a timing at which the specific image I can be displayed 30 times per second, that is, a timing at which the frequency at which the specific image I is displayed becomes 30 Hz, which is approximately 30 milliseconds.

[0065] The image output control unit 56 displays the specific image I in an area of ​​the front windshield 1 that overlaps with the road ahead of the vehicle when viewed from inside the vehicle compartment. This is because the driver's line of sight is generally directed toward the road ahead of the vehicle. The image output control unit 56 may control the HUD 10 to display the image in front of the driver's line of sight, based on the driver's line of sight calculated by the driver monitor camera 101.

[0066] The sound image output control unit 57 generates a specific sound image whose amplitude changes with 1 / f fluctuation so that changes in the amplitude of the pre-stored sound image are synchronized in time series with changes in the transparency of the image displayed by the image output control unit 56. The sound image output control unit 57 outputs the specific sound image from each of the speakers 9R, 9L, and 9U in synchronization with the display of the specific image I by the image output control unit 56. This causes the specific image I and the specific sound image to be output in a time-correlated state, particularly in a time-matched state.

[0067] The sound image output control unit 57 adjusts the output of each speaker 9R, 9L, 9U so that the sound image localization matches the display area of ​​the specific image I displayed by the HUD 10. "A state in which the sound image localization matches the display area of ​​the specific image I" means a state in which the sound image localization is located within the field of view of the driver viewing the specific image I. The sound image localization can be adjusted by adjusting the phase, volume, and output direction of the sound image output from each speaker 9R, 9L, 9U. This causes the specific image I and the specific sound image to be output in a spatially correlated state, particularly in a spatially coincident state.

[0068] When the driver's level of alertness decreases and becomes low (for example, when the level becomes higher than level 3), the alertness decrease determination unit 55 determines that an abnormality has occurred in the driver, and sends a control signal to an abnormality vehicle control unit 58. The abnormality vehicle control unit 58 sends control signals to a steering control device 61, a brake control device 62, and an engine control unit 63 to control the vehicle when the driver is abnormal. The abnormality vehicle control unit 58 sends control signals to the steering control device 61, the brake control device 62, and the engine control unit 63, for example, to stop the vehicle on the shoulder of the road or in a nearby parking lot.

[0069] FIG. 10 is a flowchart of the wakefulness maintenance control by the ECU 50.

[0070] In step S1, the ECU 50 acquires information from various sensors.

[0071] Next, in step S2, the ECU 50 determines the wakefulness level of the driver.

[0072] Next, in step S3, the ECU 50 determines whether the driver's level of alertness has decreased. If the result is YES, which indicates that the driver's level of alertness has decreased, the ECU 50 proceeds to step S4. On the other hand, if the result is NO, which indicates that the driver's level of alertness has not decreased, the ECU 50 returns.

[0073] In step S4, the ECU 50 determines whether the driver's level of alertness is equal to or lower than a predetermined level. If the determination is YES, meaning that the driver's level of alertness is equal to or lower than the predetermined level, the ECU 50 proceeds to step S5. On the other hand, if the determination is NO, meaning that the driver's level of alertness is higher than the predetermined level, the ECU 50 proceeds to step S6. The predetermined level is, for example, the aforementioned level 3.

[0074] In step S5, ECU 50 outputs the specific image I and the specific sound image in a timely and spatially synchronized manner to maintain a high level of driver alertness. After step S5, the process returns. Note that "when the visual stimulus and the auditory stimulus are in timely synchronization" does not only refer to a case where the transparency change of the specific image I and the amplitude change of the specific sound image are perfectly synchronized over the entire time period, but also includes a case where the transparency change of the specific image I and the amplitude change of the specific sound image are slightly out of sync on the time axis, as long as they are within a range that can be considered substantially synchronized. Furthermore, "when the visual stimulus and the auditory stimulus are in spatial synchronization" also includes a case where the display area of ​​the specific image I and the localization of the specific sound image do not match, as shown in FIG. 4, as long as the localization of the specific sound image is within the driver's visual field.

[0075] On the other hand, in step S6, the ECU 50 shifts to control for driver abnormality and controls the vehicle to stop the vehicle on the roadside or in a nearby parking lot. After step S6, the wakefulness maintenance control ends.

[0076] (5) Effects of the embodiment In this embodiment, the vehicle includes a HUD 10 capable of displaying a specific image I in front of the driver, which is an image lower than the driver's visual perception threshold; speakers 9R, 9L, and 9U in the vehicle cabin capable of outputting a specific sound image lower than the driver's auditory perception threshold; and an ECU 50 electrically connected to the HUD 10 and the speakers 9R, 9L, and 9U. The ECU 50 controls the HUD 10 and the speakers 9R, 9L, and 9U so that the specific image I and the specific sound image are output in a time-correlated manner. The stochastic resonance effect between the visual and auditory stimuli increases the driver's sympathetic nervous activity more than with visual stimulation alone, making the driver more likely to be alert. This improves the driver's alertness maintenance effect.

[0077] In this embodiment, the ECU 50 displays the specific image I on the HUD 10 by changing the transparency of the specific image I over time, and outputs the specific sound image to the speakers 9R, 9L, and 9U by synchronizing the change in amplitude of the specific sound image with the change in transparency of the specific image I over time. This allows the change in amplitude of the specific sound image to be synchronized with the change in transparency of the specific image over time, thereby increasing the temporal correlation between the visual and auditory stimuli. As a result, the wakefulness of the driver can be maintained more effectively.

[0078] In this embodiment, the ECU 50 controls the speakers 9R, 9L, and 9U so that the specific image I and the specific sound image are output in a spatially correlated manner. This results in the specific image I and the specific sound image being correlated not only temporally but also spatially, thereby improving the stochastic resonance effect. As a result, the amount of sympathetic nerve activity of the driver can be further improved, and the wakefulness maintenance effect on the driver can be improved.

[0079] In this embodiment, the ECU 50 controls the speakers 9R, 9L, and 9U so that the position of the specific sound image output by the speakers 9R, 9L, and 9U matches the display area of ​​the specific image I by the HUD 10. In particular, the ECU 50 controls the speakers 9R, 9L, and 9U so that the position of the specific sound image output by the speakers 9R, 9L, and 9U is located within the field of view of the driver. This increases the spatial correlation between the visual stimulus and the auditory stimulus, thereby improving the effect of maintaining wakefulness for the driver.

[0080] In this embodiment, the speakers 9R, 9L, and 9U include a right speaker 9R located on the right side of the driver's seat in the vehicle cabin, a left speaker 9L located on the left side of the driver's seat in the vehicle cabin, and an upper speaker 9U located above the driver's seat in the vehicle cabin. While the right speaker 9R and the left speaker 9L alone can adjust the sound image localization in the left-right direction, it is difficult to adjust the sound image localization in the up-down direction. By including the upper speaker 9U as in this embodiment, the localization of a specific sound image can be appropriately adjusted in the up-down direction as well. This makes it easier to match the localization of a specific sound image with the display area of ​​a specific image I. As a result, the spatial correlation between visual and auditory stimuli can be increased, improving the effect of maintaining driver alertness.

[0081] In this embodiment, the specific image I is an image with a transparency of 94±2%, and the specific sound image is a sound image with a frequency of 40 kHz or higher. By setting the transparency of the specific image I to 94±2%, it is possible to appropriately stimulate the driver's brain without causing annoyance to the driver. Furthermore, since the human audible range is generally 20 Hz to 20 kHz, a sound image with a frequency of 40 kHz or higher, which is more than twice the maximum frequency of the audible range, is unlikely to be perceived by the driver. Therefore, the effect of maintaining the driver's wakefulness can be improved without causing annoyance to the driver.

[0082] Other Embodiments The technology disclosed herein is not limited to the above-described embodiments, and can be substituted within the scope of the claims.

[0083] In the above-described embodiment, the ECU 50 outputs the specific image I and the specific sound image in temporal and spatial synchronization, but may output them in temporal synchronization only.

[0084] In the above-described embodiment, the ECU 50 displays the specific image I so that the change in transparency of the specific image I is subject to 1 / f fluctuation, but the change in transparency of the specific image I does not have to be subject to 1 / f fluctuation. Also, the ECU 50 does not have to subject the change in amplitude of the specific sound image to 1 / f fluctuation, as long as the change in amplitude of the specific sound image is synchronized with the change in transparency of the specific image I.

[0085] In the above-described embodiment, the image output control unit 56 displays the specific image I in the driver's line of sight, particularly in an area of ​​the windshield 1 that overlaps with the road ahead of the vehicle when viewed from inside the vehicle compartment. However, the specific image I is not limited to this, and does not necessarily have to be an area that overlaps with the road ahead of the vehicle as long as it overlaps with the driver's field of view on the windshield 1 when viewed from inside the vehicle compartment. When the display area of ​​the specific image I is changed, the position of the specific sound image is also changed accordingly.

[0086] In the above embodiment, the speaker (upper speaker 9U) is arranged in the upper part of the vehicle interior, but instead of the upper speaker 9U, a speaker may be arranged in the lower part of the vehicle interior.

[0087] In the above-described embodiment, the right speaker 9R and the left speaker 9L are built into the side doors, but they may also be disposed in the instrument panel 4.

[0088] The above-described embodiments are merely examples and should not be construed as limiting the scope of the present disclosure. The scope of the present disclosure is defined by the claims, and all modifications and variations that fall within the scope of the claims equivalents are within the scope of the present disclosure. [Industrial Applicability]

[0089] The technology disclosed herein is useful for a driver wakefulness maintenance device. [Explanation of symbols]

[0090] 9R Right speaker (sound section) 9L Left speaker (sound section) 9U Upper speaker (sound section) 10 Head-up display (display) 50 ECU (control unit) I Specific Image

Claims

1. A device for maintaining wakefulness of a vehicle driver, comprising: a display unit capable of displaying a specific image in front of the driver, the specific image being an image lower than the driver's visual perception threshold; an acoustic unit capable of outputting a specific sound image, which is a sound image lower than the hearing perception threshold of the driver, into a cabin of the vehicle; a control unit electrically connected to the display unit and the audio unit, The control unit controls the display unit and the audio unit so that the specific image and the specific sound image are output in a time-correlated manner.

2. 2. The driver wakefulness maintaining device according to claim 1, The control unit displaying the specific image on the display unit while changing the transparency of the specific image in a time series manner; outputting the specific sound image from the acoustic unit while synchronizing the change in amplitude of the specific sound image with the change in transparency of the specific image in time series; A device that keeps drivers awake.

3. 2. The driver wakefulness maintaining device according to claim 1, The control unit controls at least one of the display unit and the audio unit so that the specific image and the specific sound image are output in a spatially correlated manner.

4. 4. The driver wakefulness maintaining device according to claim 3, The control unit controls the audio unit so that the position of the specific sound image output by the audio unit matches the display area of ​​the specific image displayed by the display unit.

5. 5. The driver wakefulness maintaining device according to claim 4, the acoustic unit is a plurality of speakers, The device for maintaining driver wakefulness, wherein the plurality of speakers are arranged on the left, right, and above the driver's seat in the vehicle compartment.

6. The driver wakefulness maintaining device according to any one of claims 1 to 5, the specific image has a brightness of 50±2% and a transparency of 94±2%, The specific sound image is a sound image having a frequency of 40 kHz or more.

Images