Emergency driving assistance system based on emotion and gaze

Abstract

The invention relates to a vehicle emergency driving assistance system. A control circuit of the vehicle emergency driving assistance system receives a constraint signal, a characteristic signal and a motion signal. The feature signal is in a first state indicating that the driver exhibits a non-neutral emotional state or a non-neutral gaze state or a second state indicating that the driver exhibits a neutral emotional state and a neutral gaze state. The constraint signal indicates a motion constraint of the vehicle on a section of the road. The motion signal indicates vehicle motion. The control circuit determines whether the vehicle violates the constraint based on the constraint signal and the motion signal. When the characteristic signal is in the first state and the vehicle violates the constraint, the control circuit releases full control of the vehicle subsystem by the driver.

Description

Technical Field

[0001] The present invention relates to automotive electronic equipment, and in particular to electronic equipment for assisting a driver in driving a vehicle. Background Art

[0002] Different sections of road often have different speed limits. Drivers may not be aware when the speed limit changes. As a result, drivers may inadvertently drive faster than the permitted speed.

[0003] It may be very useful to warn the driver when his driving style does not comply with the speed limit. Known methods of providing such warnings depend on the speed of the vehicle and the speed limit of the location of the vehicle.

[0004] However, in some cases, the driver may ignore such warnings. This often happens when the driver is in an emotional state. When driving in this state, especially when exceeding the permitted speed, the risk of an accident increases. Summary of the invention

[0005] In one aspect, the present invention describes a method that includes causing an emergency driver assistance system to emerge from an inactive state, releasing a vehicle driver from control of a vehicle subsystem of a vehicle. The emergency driver assistance system emerges from the inactive state when it is determined that the vehicle is operating in a manner that does not comply with constraints on vehicle movement on a section of a road and when it is determined that the vehicle driver exhibits characteristics of a non-neutral emotional state or a non-neutral gaze state.

[0006] In some practices, the emergency driving assistance system selects a vehicle subsystem and outputs a control signal to control the subsystem to at least a certain extent.

[0007] In the above method practices, the constraint is a speed limit; the subsystem is a speed control subsystem. In these practices, the control signal inhibits the driver from driving above a certain speed but allows the driver to drive below the speed.

[0008] In the practice of the above method, determining that the vehicle driver exhibits non-neutral characteristics includes making a determination based at least in part on a signal from a camera directed toward the driver, at least in part on a signal from a microphone directed toward the driver, or at least in part on a camera signal and a microphone signal.

[0009] In still other practices, determining the constraint is based at least in part on an external camera viewing the sign; such determination is based at least in part on a GPS signal.

[0010] In the practice of the present invention, the non-neutral feature is a non-neutral gaze state; the feature is a non-neutral emotion; and the existing feature includes a non-neutral emotion and a non-neutral gaze state.

[0011] In some implementations, it is detected that the non-neutral feature is no longer present, thereby resuming control of the vehicle subsystem.

[0012] On the other hand, the present invention relates to an emergency driving assistance system for a vehicle. The above-mentioned emergency driving assistance system includes a control circuit, which receives a constraint signal indicating the movement constraint of the vehicle on a certain section of the road, a motion signal indicating the movement of the vehicle, and a signal indicating whether the driver exhibits a non-neutral emotion or a non-neutral gaze state. The control circuit is configured to output a control signal to the vehicle subsystem when the following conditions are determined: (1) the driver exhibits non-neutral characteristics, i.e., a non-neutral gaze state or a non-neutral emotion; (2) the vehicle's operating mode violates the constraint. The control signal prevents the driver from fully controlling the vehicle subsystem.

[0013] In the embodiment of the present invention, the non-neutral feature includes a non-neutral emotion; the non-neutral feature includes a non-neutral gaze state.

[0014] In yet other embodiments, the emergency driving assistance system is configured to resume control of the vehicle subsystem when it is determined that the non-neutral characteristic no longer exists.

[0015] As used herein, "gaze state" and "emotion state" are time-varying variables observed over a finite interval. Thus, "gaze state" can be classified as "neutral" or "non-neutral" based on how gaze direction changes over time. Similarly, "emotion state" can be classified as "neutral" or "non-neutral" based on how driver emotions change over time.

[0016] For example, if a gaze state indicates that the driver is looking in the same direction for a long period of time, regardless of the direction, it can be inferred that a driver assistance system may be needed based solely on the driver's long period of inactivity. On this basis, this gaze or "gaze state" can be classified as "non-neutral". BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other features of the present invention will be clearly understood in conjunction with the accompanying drawings and specific embodiments, in which:

[0019] Figure 1 The vehicle is shown to have an emergency driving assistance system to provide assistance to the driver;

[0020] Figure 2 Shows Figure 1 the road on which the vehicle is to be driven;

[0021] Figure 3 Shows Figure 1 Specific implementation details of the emergency driving assistance system are shown. DETAILED DESCRIPTION

[0022] Figure 1The vehicle 10 is shown having a passenger cabin 12 with passengers 14 and a driver 16 seated in seats 18. Each seat 18 has an associated microphone 20, speaker 22, and interior camera 24. The interior cameras 24 are directed toward various portions of the cabin 12. The vehicle 10 also includes exterior cameras 26. The exterior cameras 26 are directed toward the vehicle environment.

[0023] The driver 16 faces an instrument panel 28 that has a speedometer 30 integrated therein. In addition, the driver 16 holds a steering wheel 32 for controlling the direction of the vehicle. A haptic actuator 34 is coupled to one or more structures that, when vibrated or otherwise moved, attracts the driver's attention. In the illustrated embodiment, the haptic actuator 34 is coupled to the steering wheel 32.

[0024] The vehicle 10 also includes a mood analyzer 36 that receives inputs from a particular interior camera 24 and microphone 20 that are best suited to observe the driver 16. Based on features present in these inputs, the mood analyzer 36 outputs a mood signal 38. The mood signal 38 indicates whether the driver's emotional state is neutral or non-neutral. In certain embodiments, the mood signal 38 indicates the extent to which the driver's emotional state deviates from a neutral state.

[0025] In some embodiments, the emotion analyzer 36 also receives input from the interior camera 24 facing the passengers 14 and the microphone facing the passengers 14. In these embodiments, the emotion signal 38 also includes information indicating whether any one or more of the passengers 14 are in a non-neutral emotional state. This information can be used to assess the driver's emotional state.

[0026] During driving, the driver 16 spends most of the time looking forward through the vehicle windshield.

[0027] The driver 16 regularly checks the instrument panel 28, especially the speedometer 30, and is able to observe any visual warnings issued through the instrument panel 28. Many warnings are issued in this way, including, for example, the blue high beam on indicator light, the low fuel indicator light, the parking brake on indicator light, the seat belt unfastened indicator light, and various other indicators, the meaning of which is often unclear and requires consulting the vehicle manual. Such a driver is considered to be in a "neutral" gaze state.

[0028] When people drive, their eyes naturally stray away from the dashboard 28. After all, it is unsafe to stare at the dashboard 28 while driving without looking at the road ahead. Therefore, the driver may not look at the dashboard 28 for a long time. During this period, the driver cannot observe the warnings issued through the dashboard 28. Therefore, it is beneficial to identify the functional relationship between the driver's gaze state and time, so that the driver can identify that the driver has not looked at the dashboard for a long time, so as to issue warnings that may be missed in a recognizable manner. If the driver 16's gaze is not looking at the dashboard 28 for a long time, it is considered to be a "non-neutral" gaze.

[0029] To facilitate the ability to observe the direction of the driver's gaze, and thus determine the driver's gaze state, the vehicle 10 also includes a gaze detector 37 that receives inputs from a particular interior camera 24 and microphone 20 that are best suited to observe the driver 16. Based on features present in these inputs, the gaze detector 37 outputs a gaze signal 39. In an embodiment of the present invention, the gaze signal 39 is dependent on head movement, eye movement, or a combination of head movement and eye movement.

[0030] Based on the gaze signal 39, the driver's gaze state can be determined based on the gaze direction that changes over time. With such information, it can be inferred that the driver 16 may not notice the warning signal because the instrument panel 28 is not in the driver's field of view or the instrument panel 28 is located at the edge of the driver's field of view, where the eye resolution is the worst.

[0031] Reference Figure 2 , a typical road 40 traveled by the vehicle 10 includes a first road segment 42 and a second road segment 44. A first constraint 46 imposes restrictions on the movement of the vehicle within the first road segment 42. Similarly, a second constraint 48 imposes restrictions on the movement of the vehicle within the second road segment 44.

[0032] Typical constraints 46, 48 are maximum speeds. However, some roads 40 also specify minimum speeds for vehicles 10 traveling on the road. Therefore, other examples of constraints 46, 48 include minimum speeds and speed bands between the minimum speed and the maximum speed.

[0033] Still other examples of constraints 46 , 48 include vehicle direction constraints. For example, a vehicle 10 traveling the wrong way on a one-way street may be traveling below the speed limit and still fail to comply with constraints 46 , 48 .

[0034] Still other examples include vehicle direction changes. For example, some roads 40 (particularly hilly areas) have "no passing" zones. In this case, the constraints 46, 48 allow the vehicle to change direction as needed to follow the road 40, but do not allow the change of direction to result in a lane change, such as crossing lanes or leaving the lane in which the vehicle 10 is currently traveling.

[0035] Preferably, the driver 16 operates the vehicle 10 in a manner that complies with the various constraints 46, 48. Figure 1 , the emergency driving assistance system 50 assists the driver 16 in operating within the constraints 46, 48 corresponding to the specific road sections 42, 44. This is achieved by controlling the vehicle subsystem 45, thereby inhibiting the driver's ability to operate the vehicle in a manner that violates the constraints 46, 48. In some embodiments, the vehicle subsystem 45 is a speed control subsystem.

[0036] The driver assistance system 50 relies on information about applicable constraints 46, 48. In some embodiments, the driver assistance system 50 uses input from the external camera 26 to identify the sign and then extracts information about the relevant constraints 46, 48. In other embodiments, the GPS 52 maintains a library of constraints 46, 48 for a particular location and uses the vehicle location to select the relevant constraints 46, 48 and then provides them to the driver assistance system 50.

[0037] The driver assistance system 50 also relies on a motion sensor 54 which provides it with a motion signal 56. The motion signal 56 carries information about the vehicle's motion. In certain embodiments, the motion signal 56 provides information about the vehicle's speed.

[0038] The driver assistance system 50 is also configured to identify situations indicative of a driver state. Specifically, the driver assistance system 50 receives both an emotion signal 38 from the emotion analyzer 36 and a gaze signal 39 from the gaze detector 37. The emotion analyzer 36 categorizes the driver 16 as being in a neutral emotion state or a non-neutral emotion state. The gaze detector 37 categorizes the driver 16 as being in a neutral gaze state or a non-neutral gaze state.

[0039] In order to classify the driver 16 as being in one emotional state or another emotional state, the emotion analyzer 36 observes various features of the driver 16. Each feature is assigned a numerical value and becomes an element of a measured feature vector. Then, the emotion analyzer 36 compares the measured feature vector with a standard feature vector. The element values ​​of the standard feature vector have been determined to be associated with the driver 16 in a neutral state through processes such as machine learning. An applicable comparison method is to set a threshold value, and if the inner product of the measured feature vector and the standard feature vector exceeds the threshold value, the driver 16 is classified as being in a neutral state, otherwise the driver 16 is classified as being in a non-neutral state.

[0040] Examples of features available in the measured feature vector and the standard feature vector include: the extent to which the driver's facial expression deviates from a neutral expression, the extent to which the driver's grip on the steering wheel deviates from a neutral force, the extent to which the driver's voice amplitude deviates from a neutral voice, the extent to which the driver 16 fails to comply with applicable driving constraints over a period of time, and the driver's 16 previous history, including any traffic violation history.

[0041] The process of classifying the driver's gaze state by the gaze detector 37 is performed in a similar manner. In this case, the feature is the gaze direction that changes over time. The resulting time series is then compared with the time series associated with the "neutral gaze state".

[0042] In some embodiments, a "neutral gaze state" occurs if the gaze duty cycle is higher than a neutral gaze duty cycle. As used herein, a "gaze duty cycle" is calculated by dividing the time spent looking at the speedometer by the time spent not looking at the speedometer. If the gaze duty cycle is between 0 and 1 / 10, it indicates that the driver is looking at the speedometer for a short time. As a result, the gaze state is a non-neutral gaze state. If the gaze duty cycle is between 9 / 10 and 1, it indicates that the driver is staring at the speedometer for too long. The gaze state is again a non-neutral gaze state.

[0043] In some embodiments, a "neutral emotional state" is defined relative to a set of feature vectors provided by the training data. Each N-dimensional feature vector defines a point in the corresponding N-dimensional vector space. For the resulting point distribution, there is a closed surface "S", the score of points within the range of S is α, and the score of points outside the range of S is 1-α, where α is a real number specified by the designer based on the expected error rate. If the driver's own N-dimensional feature vector is within the range of S, the driver is said to be in a "neutral emotional state". In some embodiments, α = 0.8. In other embodiments, α = 0.9. In an embodiment of the present invention, the feature that can allow a higher error rate is α = 0.7 or lower.

[0044] In some embodiments, the driver assistance system 50 relies on observing patterns of driver noncompliant behavior. In these embodiments, the driver assistance system 50 observes a series of rapid lane changes or turns, indicating that the driver may be obstructed in some way. When combined with the inferences made by the emotion analyzer 36 based on one or more internal cameras 24 and microphones 20, the driver assistance system 50 can infer that the observed patterns are due to the driver being drunk, a distracted passenger 14, or road rage. This in turn informs whether to control the vehicle subsystem 45.

[0045] Refer to the following Figure 3 , the driving assistance system 50 has a control circuit 58, which receives the emotion signal 38, the motion signal 56 and the gaze signal 39. In addition, the control circuit 58 also receives a constraint signal 60. The constraint signal 60 indicates the constraints for the section of road that the vehicle 10 is traveling. Based on these inputs, the control circuit 58 outputs a control signal 64, which releases the driver from control of a particular vehicle subsystem 45 and takes over control of the subsystem.

[0046] In one example, the emotion signal 38 indicates a non-neutral emotion state, and the motion signal 56 and the constraint signal 60 together indicate that the driver is speeding. In this case, the driver assistance system 50 removes the driver 16 from full control of the vehicle speed control subsystem and replaces it with a speed governor that limits the maximum speed. Nevertheless, the driver 16 can still change speed according to the maximum speed constraint. Therefore, the driver assistance system 50 only restrains the driver's control to the extent necessary to ensure compliance with the relevant constraints.

[0047] In another example, the gaze signal 39 indicates a non-neutral gaze state, and the motion signal 56 and the constraint signal 60 together indicate that the driver is speeding. In this case, the driver assistance system 50 performs the same procedure, that is, the driver 16 is relieved of full control of the vehicle speed control subsystem and replaced with a speed governor that limits the maximum speed. Nevertheless, the driver 16 can still change speed according to the maximum speed constraint. Therefore, the driver assistance system 50 only restrains the driver's control to the extent necessary to ensure compliance with the relevant constraints.

[0048] In the illustrated embodiment, there are two methods of acquiring the constraint signal 60. The first method includes receiving the constraints 46, 48 directly from the GPS 52. The second method uses a feature extractor 66 to receive the signal from the external camera 26 and extract the relevant constraints 46, 48 from the signal.

[0049] The signal from GPS 52 and the signal from feature extractor 66 are both provided to a multiplexer 68. The user may select which method to use by setting a stored user preference 70. In other embodiments, only one method or the other is available, in which case multiplexer 68 is not required.

[0050] The user preferences 70 also specify other operational details of the driver assistance system 50. For example, the driver assistance system 50 can be completely disabled or the driver assistance system 50 can be prohibited from using the emotion analyzer 36 by setting the user preferences 70 appropriately. Other settings include the length of time that the driver assistance system can tolerate illegal driving before taking over the vehicle subsystem 45. This is a useful feature to prevent juddering. Such juddering may occur when the average driving speed of the vehicle 10 is equal to the speed limit but slightly different from the average speed.

[0051] The foregoing describes the novel technical solutions and preferred embodiments of the present invention.

Claims

1. A method comprising: causing the emergency driver assistance system to emerge from an inactive state and releasing the driver of the vehicle from full control of the vehicle subsystems of the vehicle, Wherein, emerging from the inactive state includes: determining that the vehicle is operating in a manner that does not comply with constraints imposed on vehicle movement on the road segment; and The driver is determined to exhibit a characteristic selected from the group consisting of a non-neutral gaze state and a non-neutral emotional state.

2. The method according to claim 1, further comprising: A vehicle subsystem is selected and a control signal is outputted, wherein the control signal releases the driver from control of the vehicle subsystem.

3. The method according to claim 1, wherein: The constraint is a speed limit, and the vehicle subsystem is a speed control subsystem.

4. The method according to claim 1, wherein: Outputting a control signal includes outputting a control signal that inhibits the driver from driving above a certain speed but allows the driver to drive below the certain speed.

5. The method according to claim 1, wherein: Determining that a driver of the vehicle exhibits the characteristic includes making the determination based at least in part on a signal from a camera directed toward the driver.

6. The method according to claim 1, wherein: Determining that a driver of the vehicle exhibits the characteristic includes making the determination based at least in part on a signal from a microphone directed toward the driver.

7. The method according to claim 1, further comprising: The restriction is determined based at least in part on an exterior camera viewing the sign.

8. The method according to claim 1, further comprising: The constraints are determined based at least in part on GPS signals.

9. The method according to claim 1, wherein: The characteristic is the non-neutral emotional state.

10. The method according to claim 1, wherein: The feature is the non-neutral gaze state.

11. The method according to claim 1, wherein: The emergency driving assistance system performs the following steps: determining that the driver no longer exhibits the characteristic, restoring the driver's control over the vehicle subsystem, and transitioning to its inactive state.

12. A device comprising: An emergency driving assistance system for a vehicle, the emergency driving assistance system comprising a control circuit, The control circuit receives a constraint signal, a motion signal and a characteristic signal. The characteristic signal is in one of a first state and a second state, the first state indicates that the driver exhibits at least one of a non-neutral emotional state and a non-neutral gaze state, and the second state indicates that the driver exhibits a neutral emotional state and a neutral gaze state, wherein the constraint signal indicates a motion constraint of the vehicle on a section of the road, wherein the motion signal indicates the motion of the vehicle on the section of the road, wherein the control circuit is configured to determine whether the vehicle is operating in a manner that violates the motion constraint of the vehicle on the section of the road based on the constraint signal and the motion signal, and Wherein, the control circuit is configured to output a control signal to the vehicle subsystem when the characteristic signal is in the first state and the vehicle is operating in a manner that violates the motion constraint of the vehicle on the section of the road, and the control signal prevents the driver from fully controlling the vehicle subsystem.

Images