Multi-level warning based on driver emotion
By combining camera and microphone signals to determine the driver's emotions, combined with GPS and external camera signals, a vehicle warning system is built, which solves the problem of driver's unintentional speeding, realizes personalized warnings, and improves compliance with speed limits.
Patent Information
- Application Number
- CN202410616020.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-18
- Filing Date
- 2024-05-17
- Publication Date
- 2024-11-22
AI Technical Summary
Drivers may unintentionally speed because they are not aware of changes in speed limits, and the prior art relies on speed limits in vehicle speed and position to provide effective warnings.
By determining the emotional state of the vehicle, selecting the appropriate warning message highlighting level, using a combination of camera and microphone signals, combining GPS and external camera signals, a warning system for the vehicle is constructed, which receives motion, constraints and emotional signals. Select the level of the warning signal and output it.
Effectively provide personalized warnings, which improves compliance with speed limits and reduces unintentional speeding behavior based on the driver's emotional state and vehicle movement.
Smart Images

Figure CN118991810A_ABST
Abstract
Description
Background Art
[0001] Different sections of road often have different speed limits. Drivers may not be aware when the speed limit changes. As a result, drivers may inadvertently drive at a higher speed than permitted.
[0002] It is useful to provide a warning to a driver who is driving in a manner that does not comply with the speed limit. Known methods of providing such a warning depend on the speed of the vehicle and the speed limit at the location of the vehicle. Summary of the invention
[0003] In one aspect, the present invention features a method comprising determining that a vehicle is operating in a manner inconsistent with constraints imposed on the movement of the vehicle on a road segment, determining that a driver of the vehicle is exhibiting a non-neutral emotion, selecting a prominence level for a warning message based on having determined that the driver is exhibiting a non-neutral emotion, and outputting the warning message at that level.
[0004] The practice of this method includes the constraint as a speed limit.
[0005] Other practices of this approach include selecting levels based on the extent to which the non-neutral sentiment deviates from the neutral sentiment.
[0006] In practice of the method, determining that the driver of the vehicle exhibits a non-neutral emotion includes making a determination based at least in part on a signal from a camera directed at the driver, at least in part on a signal from a microphone directed at the driver, or in part on a signal from the camera and a signal from the microphone.
[0007] Other practices include determining constraints based at least in part on observation of landmarks by external cameras and determining constraints based at least in part on GPS signals.
[0008] In another aspect, the invention features a warning system for a vehicle. The warning system includes an upgrader and a level selector. The level selector receives a constraint signal, a motion signal, and an emotion signal, the constraint signal representing a constraint on the movement of the vehicle on the road segment, the motion signal representing the movement of the vehicle, and the emotion signal representing an emotional state of a driver of the vehicle. The level selector is configured to select a prominence level of the warning signal based at least in part on the emotion signal. The upgrader receives the level from the level selector and outputs the warning signal based on the level.
[0009] As used herein, "sentiment" and "emotion" are considered to have the same meaning.
[0010] In some embodiments, the device further comprises a camera, a GPS, a user preference, a motion sensor, and a sentiment analyzer, wherein the GPS maintains a library of constraints based on the vehicle's position and provides a first constraint signal based on the vehicle's position; the user preference controls the operation of the warning system; the motion sensor provides a signal to the warning system indicating the speed of the vehicle to the level selector; and the sentiment analyzer provides a signal to the level selector indicating whether the driver is in a neutral state or a non-neutral state. In such embodiments, the warning system further comprises a feature extractor and a multiplexer, wherein the feature extractor extracts constraint information from one or more images provided by the camera; the multiplexer receives a second constraint signal from the feature extractor and a first constraint signal from the GPS and selects which of the first constraint signal and the second constraint signal is provided to the level selector.
[0011] Embodiments also include a mood analyzer that generates a mood signal for the level selector. Such a mood analyzer receives input from a camera positioned to acquire an image of the driver and a microphone positioned to listen to the driver. In embodiments that include a mood analyzer, the mood analyzer generates a signal that represents the degree to which the driver deviates from a neutral mood.
[0012] Other embodiments include user preferences that enable the driver to control and / or configure various aspects of the system, such as selectively disabling the sentiment analyzer, the warning system, or both. Embodiments also include user preferences that configure the warning system to output a warning signal only when the time spent in driving violation exceeds a threshold.
[0013] Other embodiments include external devices such as external cameras or GPS. In those embodiments featuring external cameras, the warning system includes a feature extractor that receives an image from the external camera and determines a restraint signal based on the image. In those embodiments including a GPS, the GPS maintains a library of restraints for a particular location, selects a restraint from the library using the location of the vehicle, and provides a restraint signal that communicates the restraint to the warning system.
[0014] Other embodiments include a multiplexer that selects the constraint signal from a plurality of constraint signal sources.
[0015] Embodiments include the level selector being further configured to select the highlight level based on the constraints, and the level selector being further configured to select the highlight level based on an observed pattern of driver noncompliance with the one or more constraints, the pattern comprising two or more instances of noncompliance with the one or more constraints.
[0016] These and other features of the present invention will be apparent from the following detailed description and accompanying drawings, in which: BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1shows a vehicle having a warning system embedded therein,
[0018] Figure 2 Shows Figure 1 the road that the vehicle is to cross, and
[0019] Figure 3 Shows Figure 1 Details of a particular embodiment of the warning system are shown. DETAILED DESCRIPTION
[0020] Figure 1 A vehicle 10 is shown having a passenger compartment 12 with passengers 14 and a driver 16 seated on seats 18 in the passenger compartment 12. Each seat 18 has an associated microphone 20, speaker 22, and interior camera 24. The interior camera 24 is directed toward various portions of the passenger compartment 12. The vehicle 10 also includes exterior cameras 26. The exterior cameras 26 are directed toward the environment of the vehicle.
[0021] The driver 16 faces an instrument panel 28 on which a speedometer 30 is integrated. In addition, the driver 16 has a steering wheel 32 for controlling the direction of the vehicle. A haptic actuator 34 is coupled to one or more structures that, when vibrated or otherwise moved, will attract the driver's attention. In the illustrated embodiment, the haptic actuator 34 is coupled to the steering wheel 32.
[0022] The vehicle 10 also includes a mood analyzer 36 that receives inputs from a particular interior camera 24 and microphone 20 that are best suited to observe the driver 16. Based on the features present in these inputs, the mood analyzer 36 outputs a mood signal 38. The mood signal 38 indicates whether the driver's mood is in a neutral state or a non-neutral state. In some embodiments, the mood signal 38 indicates the degree to which the driver's mood deviates from a neutral state.
[0023] In some embodiments, the emotion analyzer 36 also receives input from the interior camera 24 facing the passengers 14. In such embodiments, the emotion signal 38 also includes information indicating whether any one or more of the passengers 14 is in a non-neutral emotional state. Such information is useful for assessing the driver's potential level of distraction.
[0024] Reference Figure 2 , a typical road 40 along which the vehicle 10 travels includes a first segment 42 and a second segment 44. A first constraint 46 imposes restrictions on the movement of the vehicle within the first segment 42. Similarly, a second constraint 48 imposes restrictions on the movement of the vehicle within the second segment 44.
[0025] Typical constraints 46, 48 are maximum speeds. However, some roads 40 also specify a minimum speed for vehicles 10 operating thereon. Thus, other examples of constraints 46, 48 include a minimum speed and a speed range between the minimum speed and the maximum speed.
[0026] Other examples of constraints 46 , 48 include constraints on the direction of vehicle travel. For example, a vehicle 10 traveling in the wrong direction on a one-way street may be traveling below the speed limit and still not comply with constraints 46 , 48 .
[0027] Other examples also include changes in vehicle direction. For example, some roads 40, especially in hilly areas, have "no-go" areas. In this case, constraints 46, 48 will allow the vehicle's direction to be changed as needed to follow the road 40, but will not allow changes that would result in a lane change. Another example of constraints 46, 48 includes determining when a vehicle crosses or leaves its lane.
[0028] Preferably, the driver 16 operates the vehicle 10 in a manner that complies with the various constraints 46, 48. Figure 1 , the warning system 50 assists the user in operating within the constraints 46, 48 corresponding to a particular segment 42, 44. It does this by providing appropriate warning signals 64.
[0029] The warning system 50 relies on information about applicable constraints 46, 48. In some embodiments, the warning system 50 uses input from the external camera 26 to identify a sign and then extracts information from the sign about the relevant constraints 46, 48. In other embodiments, the GPS 52 maintains a library of constraints 46, 48 for a particular location and uses the location of the vehicle to select the relevant constraints 46, 48 and then provide them to the warning system 50.
[0030] The warning system 50 also relies on a motion sensor 54, which provides a motion signal 56 to the warning system 50. The motion signal 56 carries information about the motion of the vehicle. In some embodiments, the motion signal 56 provides information about the speed of the vehicle.
[0031] The warning system 50 is also configured to identify situations that require different levels of warning. Specifically, the warning system 50 receives the emotion signal 38 from the emotion analyzer 36 and selects an escalated warning level when it is identified that the driver 16 is in a non-neutral state.
[0032] Now refer to Figure 3The warning system 50 has a level selector 58 that receives the emotion signal 38 and the motion signal 56. In addition, the level selector 58 receives a constraint signal 60. The constraint signal 60 indicates the constraints on the road segment that the vehicle 10 is traveling on. Based on these three inputs, the level selector 58 causes the upgrader 62 to output a warning signal 64 of different levels.
[0033] In some embodiments, the degree of the upgrade depends at least in part on the degree to which the driver does not comply with the constraints. In such embodiments, driving at a speed just slightly above the speed limit will trigger only a small upgrade, while driving at a speed significantly above the speed limit will trigger a larger upgrade.
[0034] Likewise, the nature of the driver's noncompliance is also related to the level of escalation. For example, driving in the wrong direction is much more dangerous than driving at a speed limit exceeding the speed limit. In this case, it is useful for the escalator 62 to immediately escalate the warning signal 64 to its most prominent level.
[0035] In some embodiments, the warning system 50 relies on an observed pattern of non-compliant behavior by the driver. In such embodiments, the warning system 50 observes a series of rapid lane changes or sharp turns, which indicates that the driver may also be impaired in some way. When combined with the inferences made by the emotion analyzer 36 based on one or more interior cameras 24 and microphones 20, the warning system 50 is able to infer whether the observed pattern stems from the driver's intoxication, the distraction of the passenger 14, or from road rage. This in turn will inform the level of escalation.
[0036] For example, a driver 16 whose judgment is affected by road rage will generally not respond well to a loud notification from the warning system 50. Therefore, upon inferring that road rage is a possible cause of the driver's noncompliance based in part on the emotional signal 38, the warning system 50 switches to a calm voice speaking in a cadence appropriate to the situation.
[0037] In some embodiments, the warning matrix stores information representing the properties of each level of the warning signal 64. Each row of the warning matrix maps to a particular warning level. For each row, the columns specify the properties of the warning signal 64 at that level. The level selector 58 sets a pointer to the selected particular row, which is a function of a three-element vector that includes, as its elements, information from the constraint signal 60, the motion signal 56, and the emotion signal 38.
[0038] Different rows are characterized by different levels of prominence. The movement of the pointer to a more prominent warning signal 64 is referred to as "upgrading." The movement of the pointer to a less prominent warning signal 64 is referred to as "downgrading." As used herein, "prominence" is a measure of the probability that a driver with average driving skill will notice the warning signal 64.
[0039] In some cases, the level selector 58 determines that the movement of the vehicle complies with the constraints 46, 48. In this case, the level selector 58 provides a pointer to the upgrader 62, which causes the upgrader 62 to output a null warning signal 64.
[0040] The null warning signal 64 is a signal where no signal is observed by the driver 16. Therefore, this is indistinguishable from the absence of a warning signal 64 at all. When the pointer points to the null warning signal 64, only escalation is possible.
[0041] In other cases, the level selector 58 determines that the vehicle's motion does not comply with the constraints 46, 48. In this case, the emotional signal 38 comes into play.
[0042] If the emotion signal 38 indicates that the driver's emotion is neutral, the level selector 58 causes the escalator 62 to escalate the warning signal 64 to a first prominence level. An example of a warning signal 64 at this first level could be a light on the instrument panel 28, similar to the light that typically appears when the vehicle 10 is low on fuel.
[0043] Conversely, if the emotion signal 38 indicates that the driver's emotion deviates from neutral, the level selector 58 causes the escalator 62 to escalate the warning signal 64 to a second prominence level that is more prominent than the first level.
[0044] One example of a warning signal 64 at this second level is a light on the instrument panel combined with an audible alert, such as a spoken word that informs the driver 16 and possibly all passengers 14 in the passenger compartment 12 that the driver 16 is currently operating the vehicle 10 in an illegal manner. Other examples include the warning signal 64 including a tactile signal. An example of such a tactile signal is the tactile actuator 34 vibrating the steering wheel 32.
[0045] In the illustrated embodiment, there are two methods for obtaining the constraint signal 60. The first method involves receiving the constraints 46, 48 directly from the GPS 52. The second method is to use a feature extractor 66 to receive the signal from the external camera 26 and extract the relevant constraints 46, 48 from the signal.
[0046] The signal from GPS 52 and the signal from feature extractor 66 are both provided to a multiplexer 68. The selection of which one is made using a stored user preference 70 set by the user. In other embodiments, only one or the other method is available, in which case multiplexer 68 is not required.
[0047] The user preferences 70 also specify other operational details of the warning system 50. For example, by appropriately setting the user preferences 70, the warning system 50 can be completely disabled or the mood analyzer 36 can be disabled to implement the warning system 50. Other settings include the time allowed for illegal driving before the warning signal 64 changes. This is a useful function to prevent juddering. Such juddering occurs when the vehicle 10 is traveling at an average speed that is equal to the speed limit but slightly different from the speed limit.
Claims
1. A method comprising: Determining that a vehicle (10) is operating in a manner inconsistent with constraints imposed on the movement of the vehicle on a road segment, determining that a driver (16) of the vehicle is exhibiting a non-neutral emotion, selecting a prominence level for a warning message based on determining that the driver is exhibiting a non-neutral emotion, and outputting the warning message at that level.
2. The method according to claim 1, wherein The constraint is a speed limit.
3. The method according to claim 1, wherein The level is selected based on the extent to which the non-neutral emotion deviates from the neutral emotion.
4. The method according to claim 1, wherein Determining that the driver of the vehicle exhibits a non-neutral emotion includes making the determination based at least in part on a signal from a camera directed toward the driver.
5. The method according to claim 1, wherein Determining that the driver of the vehicle exhibits a non-neutral emotion includes making the determination based at least in part on a signal from a microphone directed toward the driver. The method of claim 1 , further comprising determining the constraint based at least in part on observation of a landmark by an external camera. The method of claim 1 , further comprising determining the constraint based at least in part on a GPS signal.
8. An apparatus comprising a warning system (50) for a vehicle, the warning system comprising an upgrader (62) and a level selector (58), wherein: The level selector receives a constraint signal (60), a motion signal (56), and an emotion signal (38), wherein the constraint signal indicates a constraint on the movement of a vehicle on a road segment, wherein the motion signal indicates the movement of the vehicle, wherein the emotion signal indicates whether a driver of the vehicle has a non-neutral emotion, wherein the level selector is configured to select a prominence level of a warning signal (64) based at least in part on the emotion signal, and wherein the escalator receives the level from the level selector and outputs a warning signal based on the level.
9. The apparatus of claim 8 further comprising a camera, a GPS, a user preference, a motion sensor, and a sentiment analyzer, wherein the GPS maintains a library of constraints based on vehicle position and provides a first constraint signal based on the vehicle position; the user preference controls the operation of the warning system; the motion sensor provides a signal indicating the speed of the vehicle to the level selector to the warning system; and the sentiment analyzer provides a signal indicating whether the driver is in a neutral state or a non-neutral state to the level selector; wherein The warning system also includes a feature extractor and a multiplexer, wherein the feature extractor extracts constraint information from one or more images provided by the camera; the multiplexer receives a second constraint signal from the feature extractor and a first constraint signal from the GPS and selects which of the first constraint signal and the second constraint signal to provide to the level selector.
10. The apparatus of claim 8 further comprising a sentiment analyzer that generates the sentiment signal for the level selector, wherein the sentiment analyzer receives input from a camera positioned to capture an image of the driver and a microphone positioned to listen to the driver. 11 . The apparatus of claim 8 , further comprising a sentiment analyzer and a user preference for enabling the sentiment analyzer to be selectively disabled by the driver. 12 . The apparatus of claim 8 , further comprising a user preference enabling the driver to selectively disable the warning system.
13. The apparatus of claim 8, further comprising a user preference that configures the warning system to output the warning signal only when the time spent in the driving violation exceeds a threshold.
14. The apparatus of claim 8, further comprising an external camera, wherein The warning system includes a feature extractor that receives an image from the external camera and determines the restraint signal based on the image.
15. The apparatus of claim 8 further comprising a GPS that maintains a library of constraints for specific locations, selects a constraint from the library using the vehicle location, and provides the constraint signal communicating the constraint to the warning system.
16. The apparatus of claim 8, further comprising user preferences that control operation of the warning system.
17. The device according to claim 8, wherein The warning system also includes a multiplexer that selects the restraint signal from a plurality of restraint signal sources. 18 . The apparatus of claim 8 , further comprising a mood analyzer that generates a signal representing the extent to which the driver deviates from a neutral mood.
19. The device according to claim 8, wherein The level selector is further configured to select the highlight level based on the constraint.
20. The device according to claim 8, wherein The level selector is further configured to select the highlight level based on an observed pattern of non-compliance with the constraints by the driver, the pattern including two or more instances of non-compliance with the constraints.
21. The device according to claim 8, wherein The level selector is further configured to select the highlight level based on an observed pattern of non-compliance with a plurality of constraints by the driver, the pattern comprising two or more instances of non-compliance with the plurality of constraints.
