Methods and systems for determining driver emotions in conjunction with driving environment

Abstract

A method for determining driver emotions in conjunction with the driving environment includes: determining the driver's basic emotions; acquiring driving environment information; determining the driver's emotions based on the driving environment information and the basic emotions; and providing a service that displays the driver's emotions or a service based on the driver's emotions.

Description

Technical Field

[0001] This invention relates to a method and system for determining a driver's emotions in conjunction with the driving environment. Background Technology

[0002] Global lifestyle paradigms have led to the emergence of specialized technologies that act as human assistants to improve quality of life. In particular, vehicle technology is needed to ensure safe and convenient driving environments in various vehicles essential to modern life, but to date, innovative vehicle technologies are still needed to provide new concepts for emotion-based, emotion-based, and safe driving functions.

[0003] Traditional driver emotion classification systems use information about the driver's emotional state and other information to provide services tailored to the driver. They use information about the driving environment and the driver's emotions to identify the driver's skill level and adjust the settings of the vehicle control devices based on the identification technology.

[0004] Instead, a technology is needed for more accurate identification of driver emotional states, which uses information about the driver's emotions in a primary category and information about the vehicle's driving environment when emotions change to classify driver emotions in detail. Summary of the Invention

[0005] Therefore, the present invention aims to provide a method and system for determining driver emotions in conjunction with the driving environment, for classifying driver emotions in detail using driver emotion information and driving environment information of the vehicle when emotions change.

[0006] The technical problems solved by the implementation methods are not limited to the above-mentioned technical problems, and other technical problems not described herein will become apparent to those skilled in the art from the following description.

[0007] To achieve these objectives and other advantages, and according to the purposes of the invention, as embodied and broadly described herein, a method for determining driver emotions in conjunction with the driving environment includes the following steps: determining the driver's basic emotions; acquiring driving environment information; determining the driver's emotions based on the driving environment information and the basic emotions; and providing a service that displays the driver's emotions or a service based on the driver's emotions.

[0008] In some implementations, driving environment information may include at least one of the following: vehicle speed, acceleration, global positioning system (GPS) information, passenger riding information, voice speech information, or information device operation information.

[0009] In some implementations, the step of determining the driver's emotion may include: simultaneously receiving real-time generated basic emotion and driving environment information, and identifying whether a driver emotion is present; and matching the basic emotion with the driving environment information according to a preset matching table, and determining whether a preset driver emotion is present.

[0010] In some implementations, the method may further include determining the positive and negative levels of basic emotions.

[0011] In some implementations, the step of determining the driver's emotion may include: when the underlying emotion is a positive emotion, determining that the underlying emotion is the driver's emotion while driving; and when the underlying emotion is a negative emotion, classifying the underlying emotion into subdivided negative emotions based on negative emotion and driving environment information.

[0012] In some implementations, the method may further include: detecting driving environment events; and activating a basic emotion recognition system when the detected driving environment event indicates an emotion-related event.

[0013] In some implementations, the method may further include determining whether a basic emotion occurs within a threshold time period after activating the basic emotion recognition system.

[0014] In some implementations, the method may further include: disabling the basic emotion recognition system when no basic emotion is detected within a threshold time; and determining the driver's emotion based on driving environment information and the basic emotion when a basic emotion is detected within the threshold time.

[0015] In some implementations, the step of activating the basic emotion recognition system when a driving environment event is detected may include: classifying the driving environment event into events that cause emotional changes and events that require time to cause emotional changes, and changing the timing of activating the emotion recognition system. Attached Figure Description

[0016] The accompanying drawings are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this application. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. In the drawings:

[0017] Figure 1 This is a block diagram of a system for determining driver emotions in conjunction with the driving environment, according to an embodiment of the present invention.

[0018] Figure 2 This is a diagram illustrating the results of driver emotion classification and extraction according to an embodiment of the present invention;

[0019] Figure 3This is a diagram illustrating vehicle service and vehicle control corresponding to driver emotions according to a first embodiment of the present invention;

[0020] Figure 4 This is a diagram illustrating a driver-emotion-based service according to a first embodiment of the present invention;

[0021] Figure 5 This is a diagram illustrating a method for determining a driver's emotions in conjunction with the driving environment, according to a first embodiment of the present invention;

[0022] Figure 6 This is a diagram illustrating a method for determining driver emotions in conjunction with the driving environment according to a second embodiment of the present invention; and

[0023] Figure 7 This is a diagram illustrating a method for determining a driver's emotions in conjunction with the driving environment, according to a second embodiment of the present invention. Detailed Implementation

[0024] Reference will now be made in detail to exemplary embodiments of the invention, examples of which are illustrated in the accompanying drawings. For ease of description, the suffixes “module” and “unit” are used herein, and are therefore used interchangeably without any distinguishing meaning or function.

[0025] In the description of exemplary embodiments, it should be understood that when an element is referred to as being “on” or “below” and “before” or “after” another element, the element may be directly on the other element, or there may be an intermediate element present.

[0026] It should be understood that although the terms first, second, A, B, (a), (b), etc., may be used herein to describe various elements of the invention, these terms are only used to distinguish one element from another, and the basic order or sequence of the corresponding elements is not limited by these terms. It should be understood that when an element is referred to as “connected to,” “linked to,” or “accessed to” another element, that element may be “connected to,” “linked to,” or “accessed to” the other element via further elements, or that element may be directly connected to or directly accessed to the other element.

[0027] Unless otherwise stated, the terms “comprising,” “including,” and “having” described herein should not be construed as excluding other elements but rather including such other elements. Unless otherwise mentioned, all terms including technical or scientific terms have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. General terms, such as those defined in dictionaries, should be interpreted in accordance with their meanings in the context of the invention. These terms should not be interpreted in an ideal or excessive manner unless defined differently in this invention.

[0028] Figure 1 This is a block diagram of a system for determining a driver's emotions in conjunction with the driving environment, according to an embodiment of the present invention.

[0029] Reference Figure 1 A system for determining a driver’s emotions in conjunction with the driving environment may include: a camera 10, a basic emotion recognition system 20, a driving environment information acquirer 30, and a driver emotion classification system 40.

[0030] The camera 10, which includes an image sensor, can be installed in the vehicle and can acquire images of the driver's face.

[0031] The basic emotion recognition system 20 may include a preprocessing module, a feature extraction module, and an emotion classification module.

[0032] The preprocessing module can preprocess the raw data acquired by the camera 10 to correct image quality and remove noise.

[0033] The feature extraction module can detect feature points such as emotions or motion from the input image.

[0034] Feature extraction can be performed using the following methods: a holistic approach that detects features by modeling or representing the intensity of pixel values ​​across the entire facial image; a geometric approximation approach that detects features by searching the geometric arrangement and location of features in the face; and an active appearance model (AAM) formed by combining the holistic approach and the geometric approximation approach.

[0035] The emotion classification module can classify the state of a given image based on the classification of patterns in detected features.

[0036] The emotion classification module can classify emotions using methods such as Bayesian networks with predefined conditional probabilities, K-nearest neighbors algorithm, and artificial neural networks.

[0037] The basic emotion recognition system 20 can repeatedly classify patterns in images classified through post-processing in multiple stages and can output the final selected result as the determination result.

[0038] The driver emotion classification system 40 can use driving information and the results of basic emotion identification to classify the driver's basic emotions. The following will refer to... Figures 2 to 4 Detailed description of the driver emotion classification system 40.

[0039] The driving environment information acquisition device 30 may include: a sensor configured to determine vehicle speed; a sensor configured to determine vehicle acceleration (e.g., an inertial sensor); a global positioning system (GPS) configured to acquire vehicle GPS information; a sensor configured to acquire passenger occupancy information; a microphone with voice recognition configured to acquire voice speech information; or a user interface configured to receive control information input by the driver or passenger. In this regard, the driving environment information may include at least one of vehicle speed, acceleration, GPS information, passenger occupancy information, voice speech information, or information device control information.

[0040] Figure 2 This is a diagram illustrating the results of driver emotion classification and extraction according to an embodiment of the present invention.

[0041] Reference Figure 2 The first to third experimental data 210, 220, and 230 can be experimental data illustrating the experimental results obtained by extracting driver emotions during actual driving. The system for determining driver emotions in conjunction with the driving environment can detect basic emotions using camera 10 and basic emotion recognition system 20 during a preset vehicle driving time, and can derive driver emotions using the detected basic emotions and driving environment information.

[0042] Referring to the first experimental data 210, when the basic emotion of surprise is detected multiple times, the driver emotion classification system 40 can subdivide the emotion of surprise into three types of driver emotions based on driving environment information, and can determine the driver's emotion accordingly. Therefore, when the driving environment information corresponds to the state of a narrow street, the driver emotion classification system 40 can determine the driver's emotion as focus.

[0043] Referring to the second experimental data 220, when the basic emotion of joy is detected, the driver emotion classification system 40 can further subdivide the emotion of joy based on driving environment information and determine the driver's emotion. Therefore, when the driving environment information corresponds to listening to the radio, the driver emotion classification system 40 can determine the driver's emotion as excitement.

[0044] Referring to the third experimental data 230, when basic contempt is detected, the driver emotion classification system 40 can further subdivide the contemptuous emotion based on driving environment information and determine the driver's emotion. Therefore, when the driving environment information corresponds to repeated operation of the air conditioning device, the driver emotion classification system 40 can determine the driver's emotion as dissatisfaction.

[0045] Figure 3 This diagram illustrates vehicle services and vehicle control corresponding to driver emotions according to a first embodiment of the present invention.

[0046] Reference Figure 3 The driver emotion classification system 40 can pre-configure specific driving environment events that match specific basic emotions. That is, the driver emotion classification system 40 can pre-define detailed emotions that form the intersection between specific basic emotions and specific driving environments based on the configuration.

[0047] When the underlying emotion is contempt, and if the driving environment information corresponds to traffic congestion, the driver emotion classification system 40 can identify the driver's emotion as anxiety. Therefore, the system used to determine the driver's emotion in conjunction with the driving environment can guide the driver to a bypass.

[0048] However, when the underlying emotion is anger, if the driving environment information corresponds to traffic congestion, the driver emotion classification system 40 may not identify the driver's emotion as anxiety. Therefore, the system used to determine the driver's emotion in conjunction with the driving environment may not perform vehicle control and service provision.

[0049] When the underlying emotion is surprise, if the driving environment information corresponds to traffic congestion, the driver emotion classification system 40 may not classify the driver's emotion as anxiety. Therefore, the system used to determine the driver's emotion in conjunction with the driving environment may not perform vehicle control and service provision.

[0050] When the underlying emotion is joy, if the driving environment information corresponds to traffic congestion, the driver emotion classification system 40 may not classify the driver's emotion as anxiety. Therefore, the system used to determine the driver's emotion in conjunction with the driving environment may not perform vehicle control and service provision.

[0051] When the underlying emotion is contempt, if the driving environment information does not correspond to traffic congestion, the driver emotion classification system 40 may not classify the driver emotion as anxiety. Therefore, the system used to determine the driver's emotion in conjunction with the driving environment may not perform vehicle control and service provision.

[0052] Based on the configured matching table, the driver emotion classification system 40 can determine that a specific "driver emotion" only occurs when both the basic emotion and the driving environment event are present. In this case, the driver emotion classification system 40 can simultaneously receive the real-time "basic emotion" and "driving environment information" and can identify whether a driver emotion is present.

[0053] Figure 4 This is a diagram illustrating a driver-emotion-based service according to a first embodiment of the present invention.

[0054] Reference Figure 4The driver emotion classification system 40 can classify the driver's emotions 330 based on the basic emotion information 310 from the basic emotion recognition system 20 and the driving environment information 320 obtained in real time from the driving environment information acquirer 30.

[0055] Drivers frequently experience the basic emotion of surprise while driving, so it is necessary to design a driver emotion recognition system to subdivide the emotion of surprise. It is also important to identify the driver's negative emotions for the sake of driver safety, so it is necessary to design a system to subdivide negative emotions (surprise, contempt, anger, or disgust).

[0056] When the underlying emotion is joy, if the driving environment information corresponds to conversation with passengers, the driver emotion classification system 40 can determine the driver's emotion as agitated. Therefore, the system used to determine the driver's emotion in conjunction with the driving environment can perform actions such as reducing volume and eliminating noise based on service and vehicle control.

[0057] When the underlying emotion is joy, if the driving environment information corresponds to the listened-to media, the driver emotion classification system 40 can determine the driver's emotion as excitement. Therefore, the system used to determine the driver's emotion in conjunction with the driving environment can recommend subsequent content based on services and vehicle control.

[0058] When the underlying emotion is surprise, if the driving environment information indicates driving on a narrow street, the driver emotion classification system 40 can determine the driver's emotion as focused. Therefore, the system used to determine the driver's emotion in conjunction with the driving environment can provide an AVM image based on service and vehicle control.

[0059] When the underlying emotion is surprise, if driving environment information indicates approaching the destination and searching for a parking space, the driver emotion classification system 40 can determine the driver's emotion as confusion. Therefore, the system used to determine the driver's emotion in conjunction with the driving environment can provide information about nearby parking lots based on service and vehicle control.

[0060] When the underlying emotion is contempt, and if driving environment information indicates traffic congestion, the driver emotion classification system 40 can identify the driver's emotion as anxiety. Therefore, the system used to determine the driver's emotion in conjunction with the driving environment can guide the driver to a bypass based on service and vehicle control.

[0061] When the underlying emotion is anger, if driving environment information indicates aggressive driving by other vehicles, the driver emotion classification system 40 can determine that the driver's emotion is threatened. Therefore, the system used to determine the driver's emotion in conjunction with the driving environment can store images in a black box based on service and vehicle control.

[0062] When the underlying emotion is aversion, and if driving environment information indicates prolonged driving and cruising, the driver emotion classification system 40 can determine the driver's emotion as boredom. Therefore, the system used to determine the driver's emotion in conjunction with the driving environment can recommend music and guide the driver to a rest area based on service and vehicle control information.

[0063] Figure 5 This is a diagram illustrating a method for determining a driver's emotions in conjunction with the driving environment, according to a first embodiment of the present invention.

[0064] Reference Figure 5 The system for determining the driver's emotions in conjunction with the driving environment can identify basic emotions based on images of the driver input through camera 10 (S510).

[0065] After S510 is executed, the driver emotion classification system 40 can determine the positive and negative levels of basic emotions (S520).

[0066] After operation S520, when the basic emotion is determined to be a positive emotion (S530), the driver emotion classification system 40 can output the determined basic emotion as the driver's emotion while driving (S540). Therefore, when the basic emotion is a positive emotion, the driver emotion classification system 40 can apply weights to the basic emotion and classify the driver's emotion.

[0067] After S540, the system for determining the driver's mood in conjunction with the driving environment can provide services for displaying the driver's mood or services based on the driver's mood (S570).

[0068] After operation S520, when the basic emotion is determined to be negative (S550), the driver emotion classification system 40 can output the driver's emotion based on driving environment information (S560). Therefore, when the basic emotion is negative, the driver emotion classification system 40 can apply weights to the driving environment and classify the driver's emotion. In this case, detailed emotions matching the driving environment that causes negative emotions can be pre-configured. For example, when the driving environment indicates traffic congestion, the driver's emotion can be matched with anxiety; when the driving environment indicates sudden stopping, the driver's emotion can be matched with embarrassment; and when the driving environment indicates cruising, the driver's emotion can be matched with boredom.

[0069] After S560 is executed, the system used to determine the driver's mood in conjunction with the driving environment can provide a service that displays the driver's mood or a service based on the driver's mood (S570).

[0070] Figure 6 This is a diagram illustrating a method for determining a driver's emotions in conjunction with the driving environment, according to a second embodiment of the present invention.

[0071] Reference Figure 6 The system used to determine the driver's emotions in conjunction with the driving environment can detect driving environment events (S610). Therefore, information about emotions can be obtained not throughout the entire driving time, but in situations that are predictable to the user.

[0072] After S610 is executed, the system used to determine the driver's emotions in conjunction with the driving environment can generate emotion-related events (S620).

[0073] After operating S620, the basic emotion recognition system 20 can be activated. Basic emotions can be recognized through the activated basic emotion recognition system 20 (S630). That is, the emotion recognition system can be activated only when a driving environment event occurs.

[0074] After S630, the basic emotion recognition system 20 can determine whether the emotion has changed within a preset threshold time (S640).

[0075] If there is no change in emotion within a preset threshold time during operation S640, the basic emotion recognition system 20 may not be activated (S645).

[0076] When the driver's emotions change within a preset threshold time during operation S640, the driver's emotion classification system 40 can identify the driver's emotions based on driving environment information and basic emotions identified according to driving environment events (S650).

[0077] After S650 is operated, the system for determining the driver's mood in combination with the driving environment can provide services for displaying the driver's mood or services based on the driver's mood (S660).

[0078] Figure 7 This is a diagram illustrating a method for determining a driver's emotions in conjunction with the driving environment, according to a second embodiment of the present invention.

[0079] Reference Figure 7 The timing of activating the emotion recognition system can be adjusted based on the type of driving environment event.

[0080] As shown in the figure, when a surprise-related event (e.g., sudden stop) occurs at the first moment 710 when the emotion recognition system is activated, the system used to determine the driver's emotion in conjunction with the driving environment can immediately activate the emotion recognition system when the emotion-related event occurs.

[0081] As shown in the figure, when a joy-related event (e.g., listening to media) occurs at the second time 720 when the emotion recognition system is activated, the system used to determine the driver's emotion in conjunction with the driving environment can activate the emotion recognition system after a preset time interval from the time of occurrence of the emotion-related event.

[0082] Therefore, a system for determining a driver's emotions in conjunction with the driving environment can distinguish between events that cause immediate emotional changes and events that require time to cause emotional changes, and can activate the emotion recognition system at the time of the emotional change.

[0083] The method and system for determining driver emotions in conjunction with the driving environment according to the present invention can advantageously subdivide the emotions frequently experienced by the driver and can apply an emotion classification assigned to the driver.

[0084] The technology used to activate emotion recognition systems in conjunction with the driving environment can advantageously avoid wasting vehicle computing power and can protect driver privacy.

[0085] Those skilled in the art will understand that the effects achievable by the present invention are not limited to those specifically described above, and that other advantages of the present invention will become more apparent from the detailed description.

[0086] The aforementioned method according to the embodiments can be implemented as computer-readable code stored in a computer-readable recording medium (including in a network device). Computer-readable recording media include all types of recording media that store data readable by a processor. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage devices, etc., and implementations as carrier waves (e.g., transmission via the Internet). Furthermore, the computer-readable recording medium can be distributed to computer systems connected via a network, stored and executed as code readable in a distributed manner. Additionally, those skilled in the art to which this invention pertains can readily understand the functional programs, code, and code segments used to implement this invention. The basic emotion recognition system 20 and the driver emotion classification system 40, individually or together, or the system used to determine the driver's emotion, can be implemented as a computer, processor, or microprocessor. When the computer, processor, or microprocessor reads and executes the computer-readable code stored in the computer-readable recording medium, the computer, processor, or microprocessor can be configured to perform the aforementioned operations.

[0087] It will be apparent to those skilled in the art that various modifications and variations can be made to the invention without departing from the spirit and scope of the embodiments. Therefore, the invention is intended to cover modifications and variations of the embodiments, provided that such modifications and variations fall within the scope of the appended claims and their equivalents.

Claims

1. A method for determining a driver's emotions in conjunction with the driving environment, the method comprising the following steps: Determine the driver's basic emotions; Obtain driving environment information; The driver's emotions are determined based on the driving environment information and the basic emotions. as well as Provide services that display the driver's emotions or services based on the driver's emotions; The steps for determining the driver's emotional state include: Simultaneously receive the real-time generated basic emotion and the driving environment information, and identify whether the driver's emotion is present; and The basic emotions are matched with the driving environment information according to a preset matching table, and it is determined whether a preset driver emotion has appeared.

2. The method according to claim 1, wherein, The driving environment information includes at least one of the following: vehicle speed, acceleration, GPS information, passenger riding information, voice information, or information device operation information.

3. The method according to claim 1, further comprising: Determine the positive and negative levels of the basic emotions.

4. The method according to claim 3, wherein, The steps to determine a driver's emotional state include: When the basic emotion is a positive emotion, it is determined that the basic emotion is the driver's emotion while driving; and When the basic emotion is a negative emotion, the basic emotion is classified into subdivided negative emotions based on the negative emotion and the driving environment information.

5. The method according to claim 1, further comprising: Detect driving environment events; and When a detected driving environment event indicates an emotion-related event, the basic emotion recognition system is activated.

6. The method according to claim 5, further comprising: After activating the basic emotion recognition system, it is determined whether the basic emotion occurs within a threshold time period.

7. The method according to claim 6, further comprising: If the basic emotion does not appear within the threshold time, the basic emotion recognition system is deactivated. as well as When the basic emotion occurs within the threshold time, the driver's emotion is determined based on the driving environment information and the basic emotion.

8. The method according to claim 5, wherein, When the driving environment event is detected, the step of activating the basic emotion recognition system includes: classifying the driving environment event into events that cause emotional changes and events that require time to cause emotional changes, and changing the time of activating the basic emotion recognition system.

9. A non-transitory computer-readable recording medium having a program recorded thereon for performing the method according to any one of claims 1-8.

10. A system for determining a driver's mood in conjunction with the driving environment, the system comprising: Camera; A basic emotion recognition system is configured to classify the driver's basic emotions using facial recognition information from the camera. Driving environment information acquirer, configured to acquire driving environment information; as well as A driver emotion classification system is configured to: classify the driver's basic emotions using the driving environment information and the results of identifying the basic emotions, and classify the driver's emotions accordingly. The system configuration for determining the driver's emotions includes providing a service that displays the driver's emotions or a service based on the driver's emotions. The driver emotion classification system simultaneously receives the real-time generated basic emotions and driving environment information, identifies whether the driver emotion is present, matches the basic emotions and driving environment information according to a preset matching table, and determines whether a preset driver emotion is present.

11. The system according to claim 10, wherein, The driving environment information includes at least one of the following: vehicle speed, acceleration, GPS information, passenger riding information, voice information, or information device operation information.

12. The system according to claim 10, wherein, The driver emotion classification system determines the positive and negative levels of the basic emotions.

13. The system according to claim 12, wherein, When the basic emotion is a positive emotion, the driver emotion classification system determines that the basic emotion is the driver's emotion while driving; and when the basic emotion is a negative emotion, the driver emotion classification system classifies the basic emotion into subdivided negative emotions based on the negative emotion and the driving environment information.

14. The system according to claim 10, wherein, The basic emotion recognition system is activated when a driving environment event is detected, and when the detected driving environment event indicates an emotion-related event.

15. The system according to claim 14, wherein, After the basic emotion recognition system is activated, the basic emotion recognition system determines whether the basic emotion occurs within a threshold time period.

16. The system according to claim 15, wherein, When the basic emotion does not occur within the threshold time, the basic emotion recognition system is deactivated; and when the basic emotion occurs within the threshold time, the driver emotion classification system determines the driver's emotion based on the driving environment information and the basic emotion.

17. The system according to claim 14, wherein, When the driving environment event is detected, if the basic emotion recognition system is activated, the system for determining the driver's emotion will classify the driving environment event into events that cause emotional changes and events that require time to cause emotional changes, and change the time when the emotion recognition system is activated.

Images