Driver alertness support system
Patent Information
- Application Number
- CN202512004861.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-03-26
- Filing Date
- 2025-12-29
- Publication Date
- 2026-09-29
AI Technical Summary
[0007]根据本发明的各方式,能够配合驾驶员来适当调整清醒支援的刺激。
Smart Images

Figure CN122830701A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a driver alertness support system. Background Technology
[0002] Previously, U.S. Patent Application Publication No. 2024 / 0190456 was known as a technical document related to driver alertness support systems. This publication discloses a vehicle vision system that uses an in-vehicle camera to capture images of the driver's head area and issues warnings via visual, auditory, and tactile means when driver inattention is detected.
[0003] Patent Document 1: U.S. Patent No. 2024 / 0190456 Summary of the Invention
[0004] However, if a driver who is inattentive due to lack of sleep or other reasons is given an overly strong warning, the driver may overreact. Therefore, it is preferable to adjust the warning according to the driver's condition.
[0005] One aspect of the present invention is a driver alertness support system that supports the driver's alertness when the driver of a vehicle is in an unconscious state. The driver alertness support system includes: a driver state determination unit that determines whether the driver is in an unconscious state based on images from a driver monitoring camera in the vehicle; an alertness support unit that, when the driver is determined to be in an unconscious state, uses the vehicle's equipment to perform alertness support on the driver; and a reaction determination unit that determines whether the driver who has received alertness support has overreacted based on images from the driver monitoring camera or driver input to the vehicle's driving equipment. If the alertness support unit determines that the driver has overreacted, it reduces the output of the alertness support or changes the type of alertness support to reduce the stimulation of the alertness support on the driver.
[0006] Invention Effects
[0007] According to various embodiments of the present invention, it is possible to appropriately adjust the stimuli for alertness support in accordance with the driver. Attached Figure Description
[0008] Figure 1 This is a block diagram illustrating a driver alertness support system according to one embodiment.
[0009] Figure 2 This is a flowchart illustrating an example of the driver alertness support system's alertness support processing.
[0010] Explanation of reference numerals in the attached figures
[0011] 1-Driver monitoring camera, 2-Camera, 3-Radar sensor, 4-Steering sensor, 5-Accelerator pedal sensor, 6-Brake pedal sensor, 7-HMI, 8-Lighting controller, 9-Steering vibration actuator, 10-Seat vibration actuator, 20-Sober support ECU, 21-Driver status determination unit, 22-Sober support unit, 23-Response determination unit, 24-Event determination unit, 100-Driver sober support system. Detailed Implementation
[0012] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.
[0013] Figure 1 This is a block diagram illustrating a driver alertness support system 100 according to one embodiment. Figure 1 The driver wakefulness support system 100 shown is a system that supports the driver to stay awake by means of sound, vibration, etc., when the driver of the vehicle is in a sleep state or other non-awake state.
[0014] like Figure 1 As shown, the driver alertness support system 100 according to this embodiment includes an electronic control unit (ECU) 20. The alertness support ECU 20 is an electronic control unit having a central processing unit (CPU) and a storage unit. The storage unit may be composed of, for example, read-only memory (ROM), random access memory (RAM), or electrically erasable programmable read-only memory (EEPROM). In the alertness support ECU 20, various functions are implemented, for example, by the CPU executing programs stored in the storage unit. The alertness support ECU 20 may be composed of multiple electronic units or may be part of another ECU.
[0015] The wake-up support ECU20 includes a driver monitoring camera 1, a camera 2, a radar sensor 3, a steering sensor 4, an accelerator pedal sensor 5, a brake pedal sensor 6, a human machine interface (HMI) 7, a lighting controller 8, a steering vibration actuator 9, and a seat vibration actuator 10.
[0016] The driver monitoring camera 1 is a camera used to capture images of the driver. The driver monitoring camera 1 can be configured as a digital camera with an imaging element such as a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor image sensor (CMOS image sensor). The driver monitoring camera 1 is, for example, mounted on the cover of the steering column in front of the driver, and captures images of the driver's head, including their face, at a specified frame rate. The driver monitoring camera 1 sends the captured image of the driver to the wake-up support ECU 20.
[0017] Camera 2 is an imaging device (external sensor) that captures images of the vehicle's external environment. For example, the camera may be positioned behind the vehicle's windshield to capture images of the front of the vehicle. The camera may include not only a front camera capturing images of the front of the vehicle, but also multiple cameras capturing images of the rear or sides. The camera sends images related to the vehicle's external environment to the wake-up support ECU 20.
[0018] Radar sensor 3 is a detection device (external sensor) that uses radio waves (e.g., millimeter waves) or light to detect objects around the vehicle. For example, radar sensor 3 includes millimeter-wave radar or Light Detection and Ranging (LiDAR). Radar sensor 3 sends information about the detected objects to the wake support ECU 20.
[0019] Steering sensor 4 is a sensor that detects the rotation angle or rotation speed of the steering wheel. Steering sensor 4 may be constructed using a rotation angle sensor such as an encoder or potentiometer mounted on the steering column. Steering sensor 4 detects the rotation angle or rotation speed of the steering wheel when the driver operates it and sends this data to the wake-up support ECU 20.
[0020] Accelerator pedal sensor 5 is a sensor that detects the amount or speed at which the accelerator pedal is depressed. Accelerator pedal sensor 5 may be constructed using, for example, a potentiometer or Hall effect sensor mounted on the accelerator pedal. Accelerator pedal sensor 5 detects the amount or speed at which the driver depresses the accelerator pedal and sends this data to the wake-up support ECU 20.
[0021] Brake pedal sensor 6 is a sensor that detects the amount or speed at which the brake pedal is depressed. Brake pedal sensor 6 may be constructed using, for example, a potentiometer or pressure sensor mounted on the brake pedal. Brake pedal sensor 6 detects the amount or speed at which the driver depresses the brake pedal and sends this data to the wake-up support ECU 20.
[0022] The HMI7 is an interface for information input and output between the wake-up support ECU20 and the driver. The HMI7 may include displays and speakers located within the vehicle interior. The HMI7 outputs images from the displays and audio from the speakers in response to control signals from the wake-up support ECU20. The displays may be multi-information displays (MIDs) or head-up displays (HUDs). The HMI7 may include various indicator lights.
[0023] The lighting controller 8 is a device for controlling the lighting inside the vehicle interior. For example, the lighting controller 8 can control LED lights or ambient lights installed on the ceiling inside the vehicle interior. The lighting controller 8 adjusts the brightness or color of the lighting based on control signals from the wakefulness support ECU 20. For example, if the driver is determined to be in a drowsy state, the lighting controller 8 can promote the driver's wakefulness by brightening the interior lighting. Furthermore, by using light of a specific color (e.g., blue or white), the driver's attention can be drawn, promoting wakefulness.
[0024] The steering vibration actuator 9 is a device for vibrating the steering wheel. The steering vibration actuator 9 is constructed, for example, using a motor or vibrator installed inside the steering wheel. The steering vibration actuator 9 vibrates the steering wheel according to a control signal from the wakefulness support ECU 20. By vibrating the steering wheel, the steering vibration actuator 9 can apply tactile stimulation to the driver, promoting wakefulness.
[0025] The seat vibration actuator 10 is a device for vibrating the driver's seat. The seat vibration actuator 10 is constructed, for example, using a motor or vibrator installed inside the seat. The seat vibration actuator 10 vibrates the seat according to a control signal from the wakefulness support ECU 20. By vibrating the seat, the seat vibration actuator 10 can apply tactile stimulation to the driver, promoting wakefulness.
[0026] Next, the functional structure of the sobriety support ECU20 will be explained. For example... Figure 1 As shown, the sobriety support ECU 20 includes a driver status determination unit 21, a sobriety support unit 22, a reaction determination unit 23, and an event determination unit 24.
[0027] The driver state determination unit 21 determines whether the driver is in an unconscious state based on image data of the driver's face or head transmitted from the driver monitoring camera 1. An unconscious state refers to a state in which the driver is not focused on driving due to drowsiness or fatigue. An unconscious state includes a state in which the driver is asleep or confused due to drowsiness or fatigue. An unconscious state can also include a state in which the driver's attention is not focused due to operating a smartphone, talking to other people in the vehicle, being attracted by the scenery outside, etc.
[0028] The driver state determination unit 21 uses, for example, image processing algorithms or machine learning models to analyze the driver's eye opening and closing state, gaze direction, facial orientation, and facial expressions. The driver state determination unit 21 uses, for example, facial recognition technology or eye opening and closing detection algorithms to detect the driver's eye opening and closing state or gaze direction. Furthermore, the driver state determination unit 21 detects signs of driver inattention or drowsiness by analyzing changes in facial orientation or facial expressions. Thus, it is possible to detect when the driver is drowsy or asleep, or in a state of inactivity. Various known methods can be used to determine the driver's state of inactivity.
[0029] If the driver's status determination unit 21 determines that the driver is in an unconscious state, the wakefulness support unit 22 performs wakefulness support for the driver. The wakefulness support unit 22 promotes the driver's wakefulness by providing appropriate wakefulness support corresponding to the driver's status.
[0030] Awake support includes at least one of the following: a buzzer or voice alarm output from the speaker of the HMI7; vibration of the steering wheel based on the steering vibration actuator 9; and vibration of the driver's seat based on the seat vibration actuator 10. Awake support may include an image display on the HMI7's display or the illumination of indicator lights, and may also include an increase in brightness or a change in color of the interior lighting based on the lighting controller 8.
[0031] The wakefulness support unit 22 selects an appropriate wakefulness support method corresponding to the driver's unconscious state. For example, when the driver is in a mild unconscious state, the wakefulness support unit 22 can select visual / auditory stimuli such as voice alarms or lighting adjustments, and when the driver is in a severe unconscious state, it can select tactile stimuli using the steering vibration actuator 9 or the seat vibration actuator 10.
[0032] The reaction determination unit 23 determines whether the driver who has received wakefulness support is awake based on the video image captured by the driver monitoring camera 1. Awake state refers, for example, a state where the driver is conscious and focused on driving the vehicle. The reaction determination unit 23 uses, for example, facial recognition technology or expression analysis algorithms to analyze changes in the driver's posture or facial expressions, eye movements, etc., thereby determining whether the driver who has received wakefulness support is awake.
[0033] The reaction determination unit 23 determines whether the driver who received alertness support overreacted based on the video images from the driver monitoring camera 1. For example, the reaction determination unit 23 analyzes changes in the driver's facial expressions or eye movements using facial recognition technology or expression analysis algorithms based on image data of the driver's face or head transmitted from the driver monitoring camera 1. If the driver is surprised by alertness support, such as jumping up, or exhibits behavior that shows displeasure, the reaction determination unit 23 determines that the driver overreacted.
[0034] Furthermore, the reaction determination unit 23 can determine whether the driver who has received sobering support has overreacted based on the driver's input to the vehicle's driving equipment. Driving equipment refers to the steering wheel, accelerator pedal, brake pedal, etc. Driving equipment may include the manual brake, gear shift lever, direction indicator lever, windshield wiper lever, etc.
[0035] The reaction determination unit 23 analyzes the driver's input signals sent from the steering sensor 4, accelerator pedal sensor 5, and brake pedal sensor 6. For example, if the reaction determination unit 23 detects abnormal input signals such as a sudden steering operation, sudden accelerator pedal operation, or sudden brake pedal operation performed by a driver receiving sobriety support, it determines that the driver has overreacted. A sudden steering operation can be defined as a steering operation where the steering amount exceeds a threshold within a certain time period (e.g., within 1 second). The same applies to sudden accelerator pedal operations and sudden brake pedal operations. Even in cases where the operation is not sudden, if a driver receiving sobriety support depresses the accelerator or brake pedal fully, the reaction determination unit 23 can determine that the driver has overreacted.
[0036] Furthermore, the reaction determination unit 23 can determine that the driver has overreacted when the driver suddenly pulls the manual brake, suddenly pulls the gear shift lever to activate the engine brake, or suddenly operates the direction indicator lever or the wiper lever to return to its original position.
[0037] The reaction determination unit 23 can determine the driver's overreaction based on the vehicle's driving environment. For example, if a driver who has received alertness support performs a steering operation while driving on a straight road, the reaction determination unit 23 can use a threshold value smaller than that for driving on a curve to determine that the driver has performed a sudden steering operation. If there is no vehicle or obstacle in front of the vehicle, and the driver who has received alertness support performs a brake pedal operation, the reaction determination unit 23 can use a threshold value smaller than that for cases where there is a vehicle or obstacle in front of the vehicle to determine that the driver has performed a sudden brake pedal operation.
[0038] The reaction determination unit 23 can use machine learning or deep learning methods to make various determinations. For example, the reaction determination unit 23 can use machine learning algorithms such as support vector machines or random forests to extract feature quantities from the camera images of the driver monitoring camera 1 and determine whether the driver is awake or overreacting.
[0039] The event determination unit 24 determines whether a pre-set driver reaction event has occurred around the vehicle based on at least one of the video image from the camera 2 and the detection result from the radar sensor 3. The event determination unit 24 uses an object detection algorithm or a dynamic object detection algorithm and an image analysis algorithm to detect driver reaction events occurring around the vehicle.
[0040] A driver reaction event refers to an event outside the vehicle, presumably an event that could trigger a reaction deemed excessive by the driver. Driver reaction events include at least one of the following: other vehicles cutting in front of the vehicle, approaching other vehicles, pedestrians approaching the vehicle, objects falling from vehicles ahead, a red light appearing at a traffic light ahead of the vehicle, or the sudden appearance of a pedestrian or animal in front of the vehicle. Driver reaction events may also include the detection of collision sounds or loud noises / screams outside the vehicle. In this case, the event determination unit 24 detects sound from an external microphone or a microphone inside the vehicle used for detecting external sounds.
[0041] The event determination unit 24 can determine whether a driver reaction event has occurred based on the operating information of the Advanced Driver Assistance System (ADAS). For example, if an ADAS-based forward collision warning or automatic braking is activated, the event determination unit 24 can determine that a driver reaction event has occurred.
[0042] The alertness support unit 22 adjusts the driver's alertness support based on the judgment results of the reaction judgment unit 23 and the event judgment unit 24. The alertness support unit 22 adjusts the volume of the voice alarm, the brightness of the image display, and the vibration intensity of the vibration actuator. The alertness support unit 22 can adjust the brightness or color of the interior lighting. Furthermore, the alertness support unit 22 can change the types of stimuli provided to the driver, such as image display, sound output, voice output, and vibration. Multiple types of stimuli can be combined.
[0043] If the reaction determination unit 23 determines that the driver who received sobriety support is not in a sobriety state, the sobriety support unit 22 increases the output of sobriety support or changes the type of sobriety support to increase the stimulation of sobriety support for the driver.
[0044] The increase in the output of the wakefulness support includes, for example, an increase in the volume of the warning sound or voice alarm based on the HMI7, an increase in the vibration intensity based on the steering vibration actuator 9 or the seat vibration actuator 10, an increase in the brightness of the image display of the HMI7, switching the illumination of the indicator light to flashing, an increase in the brightness of the interior lighting based on the lighting controller 8, or a switch of the lighting color to a stimulating color.
[0045] Changes to the type of alertness support that amplify stimulation may include adding vibration of the steering wheel or seat to alertness support with only voice alarms, or vibrating not only one of the steering wheel and the driver's seat, but both, or changing the voice alarm to a buzzer. Through these adjustments to alertness support, the driver can more easily transition from an unconscious state to an alert state.
[0046] On the other hand, if the reaction determination unit 23 determines that the driver who received sobering support has overreacted and the event determination unit 24 does not determine that a driver reaction event has occurred, the sobering support unit 22 reduces the output of sobering support or changes the type of sobering support to reduce the stimulation of sobering support.
[0047] The reduction in the output of the wakefulness support includes, for example, at least one of the following: reducing the volume of the warning sound or voice alarm based on HMI7; reducing the vibration intensity based on steering vibration actuator 9 or seat vibration actuator 10; reducing the brightness of the image display on HMI7; switching the flashing of the indicator light to on; reducing the brightness of the interior lighting based on lighting controller 8; or switching the color of the lighting to a calm color.
[0048] Changes in the type of wakefulness support with reduced stimulation can include stopping voice alarms and only vibrating the steering wheel, or vibrating only one of the steering wheel and the driver's seat instead of both. Changes in the type of wakefulness support can also include stopping control of the interior lighting and only providing other stimuli. Through these adjustments to wakefulness support, it is possible to suppress the driver's overreaction to wakefulness support.
[0049] The sobriety support unit 22 can adjust the sobriety support appropriately using machine learning or deep learning methods. The sobriety support unit 22 can select appropriate stimuli for sobriety support based on the driver's past state, current sobriety support, and current reaction results.
[0050] Furthermore, even if the reaction determination unit 23 determines that the driver has overreacted when the event determination unit 24 determines that the driver who received alertness support has overreacted, the alertness support unit 22 will not adjust the alertness support. This is because it is assumed that the driver reacted to the driver reaction event rather than the alertness support. For example, in the case of a forward collision warning or automatic braking activation, it is assumed that the driver reacted to these events.
[0051] The sobriety support unit 22 can perform individual driver authentication. The sobriety support unit 22 can associate and store the driver's personal profile with sobriety support adjustment parameters. The sobriety support unit 22 optimizes sobriety support for each driver.
[0052] Next, the control method of the driver alertness support system 100 according to this embodiment will be described with reference to the accompanying drawings. Figure 2 The flowchart shown illustrates an example of the wake-up support processing of a driver wake-up support system based on the wake-up support ECU 20. According to the flowchart, how the wake-up support ECU 20 monitors the driver's state and provides appropriate wake-up support is explained in detail. The wake-up support processing is performed, for example, at regular intervals while the vehicle is in motion.
[0053] In step S10, the wakefulness support ECU 20 first determines whether the driver is in a state of inattention. Inattention refers to a state where the driver feels drowsy or is not concentrating. The wakefulness support ECU 20 analyzes image data from the driver monitoring camera 1, for example, based on the driver's eye opening / closing status, gaze direction, and facial orientation to determine inattention. If the driver is not determined to be in an inattention state (step S10: No), the wakefulness support ECU 20 terminates the process. On the other hand, if the driver is determined to be in an inattention state (step S10: Yes), the wakefulness support ECU 20 proceeds to step S11 to perform wakefulness support.
[0054] In step S11, the wakefulness support ECU 20 performs wakefulness support for the driver. Wakefulness support includes at least one of the following: warning sound, voice alarm, lighting adjustment, steering wheel vibration, seat vibration, etc. The wakefulness support ECU 20 selects and executes an appropriate wakefulness support method corresponding to the degree of the driver's incoordination.
[0055] Next, the sobriety support ECU 20 proceeds to step S12 to determine whether the driver who has received sobriety support is in a sobriety state. The sobriety support ECU 20 analyzes image data from the driver monitoring camera 1 or the driver's operation input to determine whether the driver is in a sobriety state. If the driver is not determined to be in a sobriety state (step S12: No), the sobriety support ECU 20 proceeds to step S13. On the other hand, if the driver is determined to be in a sobriety state (step S12: Yes), the sobriety support ECU 20 proceeds to step S14.
[0056] In step S13, the wakefulness support ECU 20 adjusts the wakefulness support for future sessions to increase the stimulation. Specifically, compared to the current wakefulness support, the stimulation to the driver is enhanced by increasing the volume of the voice alarm or increasing the vibration intensity of the vibration actuator. The type of wakefulness support for future sessions can also be changed to increase the stimulation. This also includes adding new types of wakefulness support. Then, the wakefulness support ECU 20 ends the current processing.
[0057] In step S14, the alertness support ECU 20 determines whether the driver has overreacted. Overreacting refers to the driver's excessive surprise or unpleasant reaction to alertness support. The alertness support ECU 20 analyzes image data from the driver monitoring camera 1 or the driver's input to determine whether the driver has overreacted. If the driver has not overreacted (step S14: No), the alertness support ECU 20 terminates the process. On the other hand, if the driver has overreacted (step S14: Yes), the alertness support ECU 20 proceeds to step S15.
[0058] In step S15, the alertness support ECU 20 determines whether a driver reaction event has occurred. Driver reaction events include loud external noises such as collisions or loud noises, the sudden appearance of an animal in front of the vehicle, other vehicles cutting in front of the vehicle, objects falling from vehicles in front, red lights, etc. The alertness support ECU 20 analyzes data from the vehicle's external sensors to determine whether a driver reaction event has occurred. If a driver reaction event has occurred (step S15: No), the alertness support ECU 20 terminates the process. Conversely, if no driver reaction event has occurred (step S15: Yes), the alertness support ECU 20 proceeds to step S16.
[0059] In step S16, the wakefulness support ECU 20 adjusts the wakefulness support for future sessions to reduce the stimulation. Specifically, compared to the current wakefulness support, the stimulation to the driver is reduced by lowering the volume of the voice alarm or decreasing the vibration intensity of the vibration actuator. The type of wakefulness support for future sessions can also be changed to reduce the stimulation. This also includes reducing the number of wakefulness support types. Then, the wakefulness support ECU 20 ends the current process.
[0060] The driver alertness support system 100 described above can reduce the stimulation to the driver by reducing the output of alertness support stimuli or changing the type of stimuli when the driver overreacts to alertness support. It can suppress repeated excessive surprise or discomfort to the driver and can adjust the alertness support stimuli appropriately in accordance with the driver.
[0061] Furthermore, according to the driver wakefulness support system 100, when the driver is not awake, even if wakefulness support is received, the stimulation to the driver can be enhanced and effective wakefulness support can be provided by increasing the output or changing the type of wakefulness support.
[0062] Furthermore, according to the driver alertness support system 100, even if a driver who has received alertness support overreacts, when a driver reaction event is determined to have occurred, by reducing the output or changing the type of alertness support, the possibility of incorrectly adjusting the alertness support in relation to the driver's reaction caused by a driver reaction event outside the vehicle can be reduced.
[0063] The embodiments of the present invention have been described above, but the present invention is not limited to the above embodiments. The present invention can be implemented in various ways, for example, by making various changes and improvements to the above embodiments based on the knowledge of those skilled in the art.
[0064] The driver alertness support system 100 does not necessarily need to adjust the alertness support to increase the stimulus when the driver receiving alertness support is not in an alert state. That is, in Figure 2 In the flowchart shown, step S13 can be deleted.
[0065] Furthermore, the driver alertness support system 100 does not necessarily need to determine the occurrence of a driver reaction event. The driver alertness support system 100 can adjust the mode of alertness support to reduce the stimulus when it is determined that the driver who has received alertness support has overreacted. That is, in Figure 2 In the flowchart shown, step S15 can be deleted.
[0066] The Driver Awareness Support System 100 can determine the driver's state based on ecological signals from a wearable device worn by the driver. The wearable device can be various types, such as a ring, watch, or glasses. The Driver Awareness Support System 100 can also add temperature control functionality to the seat or steering wheel to provide alertness support based on temperature changes.
Claims
1. A driver wakefulness support system that supports the driver's wakefulness when the driver of a vehicle is in an unconscious state, the driver wakefulness support system being characterized by comprising: The driver status determination unit determines whether the driver is in an unconscious state based on the video images captured by the driver monitoring camera of the vehicle. A wakefulness support unit, which, upon determining that the driver is unconscious, uses the vehicle's equipment to provide wakefulness support to the driver; and The reaction determination unit determines, based on the video image from the driver monitoring camera or the driver's input to the vehicle's driving equipment, whether the driver, having received the sobriety support, has overreacted. If the sobriety support unit determines that the driver has overreacted, it reduces the output of the sobriety support or changes the type of sobriety support to reduce the stimulation of the driver's sobriety support compared to the case where the driver has not overreacted.
2. The driver alertness support system according to claim 1, characterized in that, The reaction determination unit determines whether the driver who received the wake-up support is not conscious based on the video image from the driver monitoring camera. If the sobriety support unit determines that the driver who has received the sobriety support is not sobriety, it increases the output of the sobriety support or changes the type of sobriety support to increase the stimulation of the sobriety support on the driver compared to the case where the driver who has received the sobriety support is sobriety.
3. The driver alertness support system according to claim 1, characterized in that, It also has: The event determination unit determines, based on the detection results of the vehicle's external sensors, whether a pre-set driver reaction event has occurred around the vehicle. Even if the driver is determined to have overreacted, the sobering support unit will not reduce the output of sobering support or change the type of sobering support when the driver reaction event is determined to have occurred.
Citation Information
Patent Citations
Vehicular driver monitoring system
US20240190456A1