Techniques for separating driving emotions from media-induced emotions in driver monitoring systems

Abstract

To provide techniques for separating driving emotion from media induced emotion in a driver monitoring system.SOLUTION: One or more embodiments include an emotion analysis system for computing and analyzing an emotional state of a user. The emotion analysis system acquires, via at least one sensor, sensor data associated with the user. The emotion analysis system determines, based on the sensor data, an emotional state associated with the user. The emotion analysis system determines a first component of the emotional state that corresponds to media content being accessed by the user. The emotion analysis system applies a first function to the emotional state to remove the first component from the emotional state.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] Field of the Disclosed Embodiments FIELD Embodiments of the present disclosure relate generally to psychophysiological sensing systems, and more particularly to techniques for separating driving emotions from media-induced emotions in driver monitoring systems. [Background technology]

[0002] Computer-based recognition of human emotional states is increasingly being adopted in various application fields. In one particular example, a driver monitoring system (DMS) may detect a driver's emotional state to assess how the driver is reacting to various driving conditions associated with the vehicle, such as weather, traffic, and / or road conditions. Bad weather, heavy traffic, and poor road conditions may cause the driver to be in a distressed, angry, and / or agitated emotional state. In contrast, good weather, light traffic, and well-maintained roads may cause the driver to be in a calm, relaxed, and / or comfortable emotional state. Depending on the assessment of the driver's emotional state, the DMS may take certain actions, such as presenting suggestions and / or alerts to the driver via text, voice, or other forms such as indicators, lights, haptic outputs, and / or the like.

[0003] One potential drawback of the above techniques is that a driver's emotional state may be attributable to multiple factors in addition to current driving conditions. For example, a driver's emotional state may be influenced by media content the driver is listening to, such as music, news, talk radio, audiobooks, and / or the like. Furthermore, a driver's emotional state may be influenced by in-person conversations between the driver and a passenger, or by phone conversations between the driver and one or more other people. Aggressive music or unpleasant conversations may cause a driver to be in a distressed, angry, and / or agitated emotional state, even when driving conditions are favorable. In contrast, meditative music or pleasant conversations may cause a driver to be calm, relaxed, and / or in a pleasant emotional state, even under stressful driving conditions. Such meditative music or pleasant conversations may cause a driver to be calm and relaxed, even under driving conditions that require the driver to be alert and active. As a result, assessments of driving conditions based on computer recognition of a driver's emotional state may be inaccurate if the emotional state is influenced by factors other than the current driving conditions. Therefore, a DMS that relies on such inaccurate assessments of driving conditions may present erroneous information to the driver.

[0004] As the foregoing indicates, improved techniques for determining user reactions to driving conditions would be useful. Summary of the Invention [Means for solving the problem]

[0005] Various embodiments of the present disclosure provide a computer-implemented method for calculating and analyzing a user's emotional state. The method includes acquiring sensor data associated with the user via at least one sensor. The method further includes determining an emotional state associated with the user based on the sensor data. The method further includes determining a first component of the emotional state corresponding to media content being accessed by the user. The method further includes applying a first function to the emotional state to remove the first component from the emotional state.

[0006] Other embodiments include, but are not limited to, systems implementing one or more aspects of the disclosed technology and one or more computer-readable media containing instructions for carrying out one or more aspects of the disclosed technology.

[0007] At least one technical advantage of the disclosed technology over the prior art is that it can process data related to a driver's emotional state to more accurately separate the contribution of driving conditions to the driver's emotional state from the driver's overall emotional state by removing the contribution of media content and / or other factors to the driver's emotional state. As a result, the DMS can generate more appropriate responsive actions for the driver depending on the contribution of driving conditions to the driver's emotional state. Another technical advantage of the disclosed technology is that a central server system that aggregates emotional state data from multiple drivers can use the more accurate assessment of the contribution of driving conditions to the driver's emotional state to generate a more accurate assessment of the overall favorability or unfavorability of driving conditions in a particular area. These technical advantages represent one or more technical improvements over prior art approaches.

[0008] So that the recited features of one or more embodiments can be understood in detail in the manner set forth above, a more particular description of one or more embodiments briefly summarized above can be made by reference to certain specific embodiments, some of which are illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings depict only typical embodiments and therefore should not be construed as limiting the scope in any way, since other embodiments are encompassed within the scope of the various embodiments. For example, the present application provides the following: (Item 1) 1. A computer-implemented method for calculating and analyzing a user's emotional state, comprising: acquiring, via at least one sensor, sensor data associated with a user; determining an emotional state associated with the user based on the sensor data; determining a first component of the emotional state corresponding to media content being accessed by the user; applying a first function to the emotional state to remove the first component from the emotional state; The computer-implemented method includes: (Item 2) determining a second component of the emotional state based on factors associated with the user; applying a second function to the emotional state to remove the second component from the emotional state; and The computer-implemented method of the preceding item further comprises: (Item 3) 10. The computer-implemented method of claim 1, wherein the factor is associated with a direct conversation between the user and another person. (Item 4) 10. The computer-implemented method of claim 1, wherein the factor is associated with a telephone conversation between the user and another person. (Item 5) The media content includes music content, and determining the first component of the emotional state includes: analyzing said musical content to determine said first component; 2. The computer-implemented method of claim 1, comprising: (Item 6) The media content includes music content, and determining the first component of the emotional state includes: Retrieving the first component from a database 2. The computer-implemented method of claim 1, comprising: (Item 7) The media content includes vocal content, and determining the first component of the emotional state includes: analyzing the vocal content to determine the first component based on at least one of a tone of voice or a text segment of the vocal content; 2. The computer-implemented method of claim 1, comprising: (Item 8) the emotional state includes an emotional valence value and an emotional arousal value, and applying the first function to the emotional state comprises: applying the first function to the emotional valence value to remove a first component of the emotional valence value that corresponds to the media content being accessed by the user; applying the first function to the emotional arousal value to remove a first component of the emotional arousal value that corresponds to the media content being accessed by the user; 2. The computer-implemented method of claim 1, comprising: (Item 9) The emotional state further includes an emotional dominance value, and applying the first function to the emotional state includes: applying the first function to the emotional importance value to remove a first component of the emotional importance value that corresponds to the media content being accessed by the user; 2. The computer-implemented method of claim 1, comprising: (Item 10) the emotional state includes a first emotion and a second emotion, and applying the first function to the emotional state includes: applying the first function to the first emotion to remove a first component of the first emotion that corresponds to the media content being accessed by the user; applying the first function to the second emotion to remove a second component of the second emotion corresponding to the media content accessed by the user; 2. The computer-implemented method of claim 1, comprising: (Item 11) When executed by one or more processors, acquiring, via at least one sensor, sensor data associated with a user; determining an emotional state associated with the user based on the sensor data; determining a first component of the emotional state corresponding to media content being accessed by the user; applying a first function to the emotional state to remove the first component from the emotional state; one or more computer-readable storage media comprising instructions for causing the one or more processors to calculate and analyze the emotional state of the user by executing: (Item 12) The above command, determining a second component of the emotional state based on factors associated with the user; applying a second function to the emotional state to remove the second component from the emotional state; One or more computer-readable storage media according to the preceding item, further causing the one or more processors to execute the following: (Item 13) 10. The one or more computer-readable storage media of any one of the preceding items, wherein the factor is associated with a direct conversation between the user and another person. (Item 14) 10. The one or more computer-readable storage media of any one of the preceding items, wherein the factor is associated with a telephone conversation between the user and another person. (Item 15) The media content includes music content, and determining the first component of the emotional state includes: analyzing said musical content to determine said first component; One or more computer-readable storage media according to any one of the preceding items, including: (Item 16) The media content includes music content, and determining the first component of the emotional state includes: Retrieving the first component from a database One or more computer-readable storage media according to any one of the preceding items, including: (Item 17) The media content includes vocal content, and determining the first component of the emotional state includes: analyzing the vocal content to determine the first component based on at least one of a tone of voice or a text segment of the vocal content; One or more computer-readable storage media according to any one of the preceding items, including: (Item 18) the emotional state includes an emotional valence value and an emotional arousal value, and applying the first function to the emotional state comprises: applying the first function to the emotional valence value to remove a first component of the emotional valence value that corresponds to the media content being accessed by the user; applying the first function to the emotional arousal value to remove a first component of the emotional arousal value that corresponds to the media content being accessed by the user; One or more computer-readable storage media according to any one of the preceding items, including: (Item 19) The emotional state further includes an emotional dominance value, and applying the first function to the emotional state includes: applying the first function to the emotional importance value to remove a first component of the emotional importance value that corresponds to the media content being accessed by the user; One or more computer-readable storage media according to any one of the preceding items, including: (Item 20) a memory containing instructions; a processor coupled to the memory that, when executing the instructions, acquiring, via at least one sensor, sensor data associated with the user; determining an emotional state associated with the user based on the sensor data; determining a first component of the emotional state corresponding to media content being accessed by the user; applying a first function to the emotional state to remove the first component from the emotional state; The above processor and a first endpoint device, (Summary) One or more embodiments include a sentiment analysis system for calculating and analyzing a user's emotional state. The sentiment analysis system acquires sensor data associated with the user via at least one sensor. The sentiment analysis system determines an emotional state associated with the user based on the sensor data. The sentiment analysis system determines a first component of the emotional state that corresponds to media content the user is accessing. The sentiment analysis system applies a first function to the emotional state to remove the first component from the emotional state. [Brief explanation of the drawings]

[0009] [Figure 1] 1 illustrates a system configured to implement one or more aspects of the present disclosure. [Figure 2] 2 is a more detailed diagram of the sentiment analysis system of FIG. 1 in accordance with various embodiments. [Figure 3A] 2A-2C are conceptual diagrams illustrating various configurations of the system of FIG. 1, according to various embodiments. [Figure 3B] 2A-2C are conceptual diagrams illustrating various configurations of the system of FIG. 1, according to various embodiments. [Figure 3C] 2A-2C are conceptual diagrams illustrating various configurations of the system of FIG. 1, according to various embodiments. [Figure 4A] 2 illustrates an exemplary arrangement of sensors associated with the system of FIG. 1, according to various embodiments. [Figure 4B] 2 illustrates an exemplary arrangement of sensors associated with the system of FIG. 1, according to various embodiments. [Figure 5A] 1 illustrates an exemplary model for mapping emotional states along various dimensions, according to various embodiments. [Figure 5B] 1 illustrates an exemplary model for mapping emotional states along various dimensions, according to various embodiments. [Figure 6] 1 is a flowchart of method steps for calculating and analyzing a user's emotional state, according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0010] In the following description, numerous specific details are set forth to provide a thorough understanding of certain illustrative embodiments. However, it will be apparent to one skilled in the art that other embodiments may be practiced without one or more of these specific details or with additional specific details.

[0011] As further described herein, existing sentiment analysis systems are configured to determine a user's overall emotional state. For example, a sentiment analysis system associated with a driver monitoring system (DMS) may be configured to determine the overall emotional state of a vehicle driver. The driver's overall emotional state may be a composite of various factors, including, but not limited to, driving conditions, media content the driver is accessing, in-person conversations between the driver and passengers, phone conversations between the driver and a remote person, and / or the like. Certain applications associated with DMS benefit from accurately assessing the driver's emotional state due to driving conditions, such as weather, traffic conditions, and / or road conditions. In this regard, the sentiment analysis system of the present disclosure determines one or more components of the driver's emotional state due to factors other than the driving conditions and removes those components from the driver's overall emotional state. As a result, the remaining emotional state data represents the driver's emotional state due to factors other than the removed components. In this regard, the remaining emotional state data more accurately represents the driver's emotional state due to the driving conditions.

[0012] In one embodiment, the sentiment analysis system determines the overall emotional state of the driver. Then, the sentiment analysis system determines the component of the driver's emotional state that is due to listening to or consuming the media content. The sentiment analysis system removes the component of the driver's emotional state that is due to listening to the media content from the driver's overall emotional state. Additionally or alternatively, the sentiment analysis system determines the component of the driver's emotional state that is due to one or more additional factors, such as a face-to-face conversation between the driver and a passenger, a phone conversation between the driver and another person, and / or the like. Then, the sentiment analysis system removes the component of the driver's emotional state that is due to each of these other factors. The remaining emotional state data more accurately represents the driver's emotional state due to the driving conditions.

[0013] System Overview 1 illustrates a system 100 configured to implement one or more aspects of the present disclosure. As shown, the system 100 includes, but is not limited to, a remote server system 102, a telemetrics and radio system 104, a driver monitoring system 106, and a sentiment analysis system 108. The remote server system 102 and the telemetrics and radio system 104 communicate with each other via a communications network 110. The communications network 110 may be any suitable environment that enables communication between remote or local computer systems and computing devices, including, but not limited to, Bluetooth® communications channels, wireless and wired LANs (local area networks), WANs (wide area networks), cellular networks, satellite networks, high-altitude balloon networks (and other atmospheric satellite networks), peer-to-peer type networks, vehicle-to-everything (V2X) networks, and / or the like. The remote server system 102 and the telemetrics and radio system 104 communicate over the communications network 110 via communications links 112 and 114, respectively. Additionally, the telemetrics and radio system 104 communicates with the driver monitoring system 106 and the sentiment analysis system 108 via communication links 120 and 118, respectively. The driver monitoring system 106 communicates with the sentiment analysis system 108 via communication link 116. The communication links 116, 118, and 120 may use any one or more technically feasible communication media and protocols in any combination.

[0014] The remote server system 102 includes, but is not limited to, a computing device, which may be a standalone server, a cluster or "farm" of servers, one or more network appliances, or any other device suitable for implementing one or more aspects of the present disclosure. Illustratively, the remote server system 102 communicates over a communications network 110 via a communications link 112.

[0015] During operation, the remote server system 102 receives emotional state data from one or more sentiment analysis systems 108. In some embodiments, the remote server system 102, in conjunction with the sentiment analysis system 108, performs one or more of the techniques described herein. Additionally, the remote server system 102 aggregates and analyzes emotional state data from multiple users in a given geographic region. Based on the aggregated emotional state data, the remote server system 102 can assess whether driving conditions in the given geographic region are favorable or unfavorable. The central server system can route other drivers away from areas where driving conditions are rated as less favorable and toward areas where driving conditions are rated as more favorable. In this manner, computer-based recognition of human emotional states can improve the experience not only of individual drivers but also of groups of drivers as a whole.

[0016] The telemetrics and radio system 104 includes a computing device, which may be, but is not limited to, a standalone server, a cluster or "farm" of servers, one or more network appliances, or any other device suitable for implementing one or more aspects of the present disclosure. Illustratively, the telemetrics and radio system 104 communicates over a communications network 110 via communications link 114. Additionally, the telemetrics and radio system 104 communicates with a driver monitoring system 106 and a sentiment analysis system 108 via communications links 120 and 118, respectively.

[0017] During operation, the telemetrics and wireless system 104 receives measurement data from the driver monitoring system 106 and / or the sentiment analysis system 108. The measurement data may include information related to various components of the system 100, including, but not limited to, sensor data, equipment, camera images, video, and / or the like. The measurement data may further include processed data, where the driver monitoring system 106 and / or the sentiment analysis system 108 analyzes certain measurement data, such as sensor data, equipment, camera images, video, and / or the like, and generates processed data therefrom. Such processed data may include, but is not limited to, emotional state data. The telemetrics and wireless system 104 then transmits the measurement data from the driver monitoring system 106 and / or the sentiment analysis system 108 to the remote server system 102 via the communication network 110.

[0018] The driver monitoring system 106 includes a computing device, which may be, but is not limited to, a standalone server, a cluster or "farm" of servers, one or more network appliances, or any other device suitable for implementing one or more aspects of the present disclosure. Illustratively, the driver monitoring system 106 communicates with the telemetrics and radio system 104 and the sentiment analysis system 108 via communication links 120 and 116, respectively.

[0019] During operation, the driver monitoring system 106 monitors the vehicle's driver to determine certain characteristics, such as the driver's alertness state. The driver monitoring system 106 receives measurement data via various devices, including, but not limited to, cameras, microphones, infrared sensors, ultrasonic sensors, radar sensors, thermal imaging sensors, heart rate and respiration monitors, vehicle instrument sensors, and / or the like. By analyzing the measurement data, the driver monitoring system 106 determines the driver's overall physiological state, which may include the driver's alertness level. If the driver monitoring system 106 determines that the driver is not sufficiently alert, the driver monitoring system 106 can initiate certain response actions, including, but not limited to, flashing interior lights, sounding an alarm, applying the brakes to safely slow or stop the vehicle, and / or the like. Additionally, the driver monitoring system 106 transmits the measurement data received via the various devices to the sentiment analysis system 108 for further analysis, as further described herein.

[0020] Sentiment analysis system 108 includes, but is not limited to, a computing device, which may be a standalone server, a cluster or "farm" of servers, one or more network appliances, or any other device suitable for implementing one or more aspects of the present disclosure. Illustratively, sentiment analysis system 108 communicates with telemetrics and radio system 104 and driver monitoring system 106 via communication links 118 and 116, respectively.

[0021] During operation, the sentiment analysis system 108 receives measurement data from the driver monitoring system 106. The measurement data is received via various devices associated with the driver monitoring system 106. The sentiment analysis system 108 analyzes the measurement data to generate processed data related to the emotional state of the driver or other user, as described further herein. The sentiment analysis system 108 stores one or both of the measurement data and the processed data in a data store. In some embodiments, the sentiment analysis system 108 can transmit the processed data to the driver monitoring system 106. The driver monitoring system 106 can then perform one or more response actions based on the processed data. In some embodiments, the sentiment analysis system 108 can transmit one or both of the measurement data and the processed data to the telemetrics and radio system 104. The telemetrics and radio system 104 can then transmit the measurement data and / or the processed data to the remote server system 102 via the communication network 110.

[0022] It will be understood that the systems shown herein are exemplary and that variations and modifications are possible. In one embodiment, the remote server system 102, the telemetrics and radio system 104, the driver monitoring system 106, and the sentiment analysis system 108 are shown communicating via particular networked communication links. However, the remote server system 102, the telemetrics and radio system 104, the driver monitoring system 106, and the sentiment analysis system 108 can communicate with each other via any technically feasible networks and communication links in any technically feasible combination within the scope of this disclosure.

[0023] In another example, the remote server system 102, the telemetrics and radio system 104, the driver monitoring system 106, and the sentiment analysis system 108 are shown as separate systems included in the system 100 of FIG. 1 . However, the techniques performed by the remote server system 102, the telemetrics and radio system 104, the driver monitoring system 106, and the sentiment analysis system 108 may be performed by one or more application programs or modules executing on any technically feasible processor(s) included in one or more computing devices in any technically feasible combination. Such computing devices may include, but are not limited to, a head unit and auxiliary units deployed in a vehicle. In yet another example, the remote server system 102, the telemetrics and radio system 104, the driver monitoring system 106, and the sentiment analysis system 108 are shown and described in the context of a vehicle-based computing system that receives and processes emotional state data of a driver and / or one or more passengers. However, the techniques described herein may be deployed on any technically feasible system that receives monitoring of a user's emotional state, including, but not limited to, a smartphone, a laptop computer, a tablet computer, a deskside computer, and / or the like. In yet another embodiment, system 100 may include any technically feasible number of remote server systems 102, telemetrics and radio systems 104, driver monitoring systems 106, and emotion analysis systems 108 in any technically feasible combination.

[0024] Operation of the emotional state analysis system Figure 2 is a more detailed diagram of the sentiment analysis system 108 of Figure 1, in accordance with various embodiments. As shown, the sentiment analysis system 108 includes, but is not limited to, a processor 202, storage 204, an input / output (I / O) device interface 206, a network interface 208, an interconnect 210, and a system memory 212.

[0025] The processor 202 reads and executes programming instructions stored in the system memory 212. Similarly, the processor 202 stores and reads application data residing in the system memory 212. The interconnect 210 facilitates the transmission of programming instructions, application data, and the like between the processor 202, the I / O device interface 206, the storage 204, the network interface 208, and the system memory 212. The I / O device interface 206 is configured to receive input data from a user I / O device 222. Examples of the user I / O device 222 may include one or more buttons, a keyboard, a mouse or other pointing device, and / or the like. The I / O device interface 206 may also include an audio output unit configured to generate an electrical audio output signal, and the user I / O device 222 may further include a speaker configured to generate an acoustic output in response to the electrical audio output signal. Another example of the user I / O device 122 is a display device, which generally represents any technically feasible means for generating images for display. For example, the display device may be a liquid crystal display (LCD) display, an organic light-emitting diode (OLED) display, or a digital light processing (DLP) display. The display device may be a television including a broadcast or cable tuner for receiving digital or analog television signals. The display device may be included in a VR / AR headset or a head-up display (HUD) assembly. Furthermore, the display device may project images onto one or more surfaces, such as a wall, a projection screen, or a vehicle windshield. Additionally or alternatively, the display device may project images directly into a user's eyes (e.g., via retinal projection).

[0026] Processor 202 is included to represent a single central processing unit (CPU), multiple CPUs, a single CPU with multiple processing cores, a digital signal processor (DSP), a field programmable gate array (FPGA), a graphics processing unit (GPU), a tensor processing unit, and / or the like. And, system memory 212 is included to generally represent random access memory. Storage 204 may be a disk drive storage device. Although shown as a single unit, storage 204 may be a combination of fixed and / or removable storage devices, such as fixed disk drives, floppy disk drives, tape drives, removable memory cards, optical storage, network-attached storage (NAS), or a storage area network (SAN). Processor 202 communicates with other computing devices and systems via network interface 208, which is configured to send and receive data over a communications network.

[0027] The system memory 212 includes, but is not limited to, a driver emotion recognition module 232, a media emotion recognition module 234, a speech emotion recognition module 236, an emotion normalizer module 238, and a data store 242. The driver emotion recognition module 232, the media emotion recognition module 234, and the emotion normalizer module 238, when executed by the processor 202, perform one or more operations associated with the emotion analysis system 108 of FIG. 1, as described further herein. When performing operations associated with the emotion analysis system 108, the driver emotion recognition module 232, and the media emotion recognition module 234, the emotion normalizer module 238 may store data in and read data from the data store 242.

[0028] During operation, the driver emotion recognition module 232 determines and classifies the driver's emotional state based on various sensor data received, such as facial features and other visual cues, tone of voice and other audio cues, physiological signals, steering wheel jerks or other movements, frequent braking, and / or the like. The driver emotion recognition module 232 can classify the emotional state based on a two-dimensional or three-dimensional model of the emotional state, as further described herein. Emotional states are often qualitatively described in descriptive terms such as joy, sadness, anger, and enjoyment. Such descriptive terms can be difficult to analyze algorithmically. As a result, the driver emotion recognition module 232 determines numerical values ​​along various dimensions to represent the driver's emotional state. In the two-dimensional model, the driver emotion recognition module 232 determines values ​​for two numerical representations of the emotional state. For example, the driver emotion recognition module 232 can determine a first numerical value for emotional arousal and a second numerical value for emotional valence. The driver's emotional state in the two-dimensional model can be expressed by the following Equation 1: emotions ドライバー =f(driver 誘意性 ,driver 喚起性 ) formula 1

[0029] In the three-dimensional model, the driver emotion recognition module 232 determines values ​​for three numerical representations of the emotional state. For example, the driver emotion recognition module 232 may determine a first numerical value for emotional arousal, a second numerical value for emotional valence, and a third numerical value for emotional dominance. Emotional dominance is also referred to herein as emotional stance. The driver's emotional state in the three-dimensional model may be expressed by the following Equation 2: emotions ドライバー =f(driver 誘意性 ,driver 喚起性 ,driver 優位性 ) Equation 2

[0030] Whether a two-dimensional or three-dimensional model is used, the driver emotion recognition module 232 determines the driver's emotional state from all relevant factors. The following description assumes a three-dimensional model for clarity. However, each of the techniques disclosed herein can use two-dimensional models, three-dimensional models, or higher-dimensional models in any technically feasible combination.

[0031] During operation, the media emotion recognition module 234 determines and classifies the emotional content of media being accessed by the driver and / or other users due to certain factors. The emotional content of media being accessed by the driver and / or other users may influence, at least in part, the emotional state of the driver and / or other users. The media emotion recognition module 234 may determine and classify the emotional content of media via any technically feasible technique, including, but not limited to, algorithmic and machine learning techniques. The media emotion recognition module 234 may classify the emotional content based on a two-dimensional model, a three-dimensional model, or a higher-dimensional model of the emotional state, as further described herein.

[0032] More specifically, the media emotion recognition module 234 determines the emotional content of the media content currently playing in the vehicle. The media content can be in any format, including, but not limited to, music, news programs, talk radio, and / or the like. For example, if the driver is listening to an entertaining talk radio program, the media emotion recognition module 234 can determine that the driver is in a happy, joyful, and / or excited emotional state due to the media content. If the driver is listening to loud, aggressive music, the media emotion recognition module 234 can determine that the driver is in an aggressive and / or stressful emotional state due to the media content. The media emotion recognition module 234 can classify the media content via any one or more technically feasible techniques. In one embodiment, the media emotion recognition module 234 can acoustically analyze the music content to determine a typical emotional state expressed or evoked by the media content. In another embodiment, the media emotion recognition module 234 can retrieve song lyrics from a music lyrics database. Additionally or alternatively, the media emotion recognition module 234 can perform speech-to-text conversion on music lyrics, news programs, or talk radio programs. The media emotion recognition module 234 can analyze the resulting text and determine an emotional state associated with the text. Additionally or alternatively, the media emotion recognition module 234 can analyze the tone of voice of sung or spoken words and associate an emotional state based on whether the tone of voice is aggressive, calm, and / or the like.

[0033] In yet another embodiment, the media emotion recognition module 234 may classify the media content based on previously defined classifications in the form of descriptions, labels, tags, metadata, and / or the like. The media emotion recognition module 234 may map or otherwise correlate such descriptions, labels, tags, or metadata to specific emotional states. The media emotion recognition module 234 may categorize the media content into specific types of media content and generate appropriate labels for the media content by using heuristics. For example, the media emotion recognition module 234 may classify media content categorized as comedy as inducing a happy emotional state. Similarly, the media emotion recognition module 234 may classify media content categorized as heavy metal as inducing an aggressive emotional state. The media emotion recognition module 234 may convert these categories or labels into a two-dimensional model (including emotional arousal and emotional valence), a three-dimensional model (including emotional arousal, emotional valence, and emotional dominance), or a higher-dimensional model. The emotional state represented by the media content in the three-dimensional model attributed to the media content may be expressed as Equation 3 below: emotions メディア =f(media 誘意性 ,media 喚起性 ,media 優位性 ) Equation 3

[0034] During operation, the emotion normalizer module 238 continuously generates driver emotional state data for driving conditions by removing components of the driver's emotional state due to listening to media content from the overall emotional state. The emotion normalizer module 238 receives the driver's overall emotional state from the driver emotion recognition module 232. The emotion normalizer module 238 receives the driver's emotional state due to listening to media content from the media emotion recognition module 234. In response, the emotion normalizer module 238 removes components representing the driver's emotional state due to listening to media content from the driver's overall emotional state. The emotion normalizer module 238 removes the components via any technically feasible technique, including, but not limited to, an addition / subtraction technique, a logical AND technique, a logical OR technique, a Bayesian model technique, and / or the like. The remaining emotional state data represents the driver's emotional state due to factors other than listening to media content. As a result, the remaining emotional state data more accurately represents the driver's emotional state due to driving conditions. The driver's emotional state due to driving conditions can be expressed by the following Equation 4: emotions 運転から =f(emotion ドライバー, emotions メディア ) Equation 4

[0035] The driver's emotional state due to driving conditions can be fully represented using a three-dimensional model according to Equation 5 below. emotions 運転から =f{(driver 誘意性 ,media 誘意性 ),(driver 喚起性 ,media 喚起性 ),(driver 優位性 ,media 優位性 )} Expression 5 emotions ドライバー and emotions メディア If there is a subtractive relationship between the two, the driver's emotional state depending on the driving conditions can be equivalently expressed by the following Equations 6 and 7. emotions 運転から = feelings ドライバー -Emotions メディア formula 6 emotions 運転から =f((driver 誘意性 -media 誘意性 ),(driver 喚起性 -media 喚起性 ),(driver 優位性 -media 優位性 )) Equation 7

[0036] In some embodiments, the conversation emotion recognition module 236, in conjunction with the emotion normalizer module 238, may remove one or more additional components of the emotional state from the driver's overall emotional state to further refine the resulting emotional state. In one example, the conversation emotion recognition module 236 may analyze a face-to-face conversation between the driver and a passenger or a phone conversation between the driver and another person. The conversation emotion recognition module 236 may perform speech-to-text conversion on words spoken by the driver. The conversation emotion recognition module 236 may analyze the resulting text and determine an emotional state associated with the text. Additionally or alternatively, the conversation emotion recognition module 236 may analyze the tone of voice of the spoken words to associate an emotional state based on whether the tone of voice is aggressive or calm, and / or the like. Similarly, the conversation emotion recognition module 236 may analyze the text and tone of voice of a passenger or other person participating in the conversation. In such a case, the conversation emotion recognition module 236 may apply a higher weight to the emotional state derived from the driver's speech than to the emotional state derived from the passenger's or other person's speech. The emotion normalizer module 238 can then remove this additional component from the driver's emotional state. The driver's emotional state due to driving conditions can then be expressed by the following Equation 8: emotions 運転から = f(emotion ドライバー, emotions メディア ,feelings 会話から ) Equation 8

[0037] emotions ドライバー , emotions メディア , and emotions会話から If there is a subtractive relationship between the driver's emotional state and the driving conditions, the driver's emotional state can be equivalently expressed by the following Equation 9. emotions 運転から = feelings ドライバー -Emotions メディア -Emotions 会話から formula 9

[0038] It will be understood that the systems illustrated herein are exemplary and that variations and modifications are possible. In particular, sentiment analysis system 108 may not include speech emotion recognition module 236. In such cases, sentiment analysis system 108 may not perform the functionality described in conjunction with speech emotion recognition module 236. Additionally or alternatively, one or more of the functionality described in conjunction with speech emotion recognition module 236 may be performed by one or more other modules, such as driver emotion recognition module 232, media emotion recognition module 234, emotion normalizer module 238, and / or the like.

[0039] In some embodiments, the emotional states are analyzed on an emotion-by-emotion basis rather than based on a two-dimensional or three-dimensional model. In such embodiments, the media emotion recognition module 234 and / or the speech emotion recognition module 236 can analyze happiness, anger, and / or other related emotions one at a time. Then, for each emotion, the emotion normalizer module 238 can remove components attributable to the media content, the driver's speech, and / or other components. The resulting emotional states of happiness and anger due to driving conditions can then be expressed as Equations 10 and 11 below. happiness 運転から =f(happiness ドライバー、 happiness メディア ) Equation 10 anger 運転から =f(anger ドライバー, anger メディア ) Equation 11

[0040] If the elements of Equations 10 and 11 are in a subtractive relationship, the emotional states of happiness and anger due to driving conditions can be equivalently expressed by the following Equations 12 and 13. happiness 運転から = happiness ドライバー -happiness メディア Formula 12 anger 運転から =Anger ドライバー -anger メディア Formula 13

[0041] In some embodiments, the media emotion recognition module 234 and / or the speech emotion recognition module 236 can generate emotional state data customized for a particular driver. As an example, aggressive, heavy metal music may generally cause a driver's emotional state to be more distressed and aggressive. However, a driver with a strong affinity for heavy metal music may experience a more calm and / or pleasant emotional state when listening to such music. Similarly, calm, meditative music may generally cause a driver's emotional state to be more calm and / or pleasant. However, a driver who strongly dislikes meditative music may experience a more agitated and / or stressful emotional state when listening to such music.

[0042] In some embodiments, the media emotion recognition module 234 and / or the speech emotion recognition module 236 can track changes in emotional state due to media content, speech, and / or other components over time. For example, a driver may be listening to hard rock music and then switch to easy listening music. As a result, the media emotion recognition module 234 may determine that the driver's emotional state is becoming more calm and less stressed as a result of the change from hard rock music to easy listening music, rather than a change in driving conditions. Similarly, a driver may be listening to easy listening music and then switch to hard rock music. As a result, the media emotion recognition module 234 may determine that the driver's emotional state is becoming more aroused and stressed as a result of the change from easy listening music to hard rock music, rather than a change in driving conditions.

[0043] The various types of sensor data and associated processing are now described in more detail. The sensor data are categorized as emotion sensing, physiological sensing, behavioral sensing, acoustic sensing, and pupillometry-based cognitive workload sensing.

[0044] Emotion sensing involves the detection and classification of emotions and emotional states. Emotion sensing includes the detection of unobtrusive, known emotions such as happiness, satisfaction, anger, and frustration. Emotion sensing involves the calculation of parameterized metrics related to emotional states, such as emotional arousal levels and emotional valence levels. Emotion sensing is based on data received from various types of sensors.

[0045] Sensor data can be received from psychophysiological sensors measuring various biological and physiological signals associated with the user, including, but not limited to, sweating, heart rate, breathing rate, blood flow, blood oxygen levels, galvanic skin response, body temperature, sounds made by the user, user behavior, and / or the like. Such sensor data represents various types of signals related to emotion detection. Additionally, image data can be received from cameras and other image sensors configured to capture still and video images, including, but not limited to, color images, black and white images, thermal images, infrared images, and / or the like. Such cameras and image sensors capture the user's facial expressions or other images of the user's body position and / or distortion, which may be indicative of emotion. In some embodiments, images can be received from an array of cameras or image sensors to simultaneously capture multiple perspectives of the user's body and head. Furthermore, in some embodiments, images can be received from depth cameras or image sensors to sense body posture and body position.

[0046] Physiological sensing involves detection systems that capture various physiological signals that correlate with emotional states. Signals received from such sensors correlate with specific emotional states and are therefore relevant for emotional classification. For example, galvanic skin response (GSR) may indicate the intensity of an emotional state. Physiological sensors include, but are not limited to, GSR sensors that measure changes in the electrical resistance of the skin caused by emotional stress, imagers that detect blood oxygen levels, thermal sensors that detect blood flow, optical sensors that detect blood flow, EEG systems that detect brain surface potentials, EOG sensors (electro-oculography sensors that measure eye movement by monitoring the electrical potential between the front and back of the human eye), EMG sensors (electromyography sensors that measure electrical activity in response to muscle stimulation via nerves), ECG sensors (electrocardiography sensors that measure electrical activity of the heart), high-frequency wireless sensors such as GHz-band radios that measure heart rate and respiration rate, neural systems that detect neural correlates of emotion, and / or the like.

[0047] Acoustic sensing includes analyzing the words spoken by a user as well as analyzing how the user speaks certain phrases that indicate emotional feelings. Acoustic sensing also includes non-speech human sounds made by a user, including, but not limited to, whistling, humming, laughing, or shouting, which may indicate the user's emotional state. In one example, natural language processing methods, emotion analysis, and / or speech analysis can measure emotion through the semantic meaning of language. In another example, tone of voice analysis can detect emotion from actual speech signals. Either method can be used alone or in combination. Typical acoustic sensor data includes, but is not limited to, microphones, microphone arrays, and / or other audio sensing technologies.

[0048] Behavioral sensing includes detecting user activity in and around the vehicle. Some of the sensors described further herein can be used to detect movement in and around the vehicle. Application and service usage data can also indicate user behavior and, through a classification system, infer emotions. In one example, mobile usage data can indicate patterns of application usage by a user that correlate with a particular emotional state. If an application is categorized as a gaming application or a social application, execution of such an application can correlate with joy, happiness, and / or related social emotions. Behavioral sensors further include, but are not limited to, cameras, image sensors, auditory sensors, depth cameras, pressure sensors, and / or the like. These sensors record a user's body position, movement, and other behavior in and around the vehicle. Such body position, movement, and / or behavior data can be correlated with emotions such as boredom, fatigue, and alertness. Behavioral sensors further include, but are not limited to, touch sensors, acoustic sensors, registering button presses, or other user interface interactions and / or the like that determine how a user is behaving in the vehicle. Such sensor data can indicate which systems a user is accessing and where the user places their hands at any given time.

[0049] Pupilometry-based cognitive workload sensing measures subtle variations in a user's pupil diameter. These subtle variations are scientifically related to the cognitive workload a user is experiencing from time to time. Other related technologies can be used to measure cognitive workload. Sensors that enable cognitive workload measurement include, but are not limited to, cameras and image sensors that image a user's pupils to measure changes in pupil diameter and / or eye movement. Such cameras and image sensors include, but are not limited to, infrared cameras, thermal sensors, high-resolution color or black-and-white cameras, camera arrays that capture multiple perspectives of a user's body and head, and / or the like. Physiological sensors include, but are not limited to, galvanic skin response sensors, heart rate sensors, skin temperature sensors, and / or the like that measure cognitive workload at relatively low resolution. In some embodiments, EEG and other neural interfaces can detect multiple cognitive workload levels. Related methods for measuring cognitive workload from EEG and other neural data include spectral entropy, weighted average frequency, bandwidth, spectral edge frequency, and / or the like. In some embodiments, speech analysis can be used for cognitive workload sensing. In particular, the frequency and amplitude of the spectral centroids can successfully classify different cognitive workload levels using some parameters suitable for the filter length and number of filters.

[0050] In some embodiments, the driver emotion recognition module 232, the media emotion recognition module 234, and / or the emotion normalizer module 238 can divide the sensor data and processed data into several levels, each associated with a different degree of abstraction. A first data level can include raw sensor data, including but not limited to data from cameras, microphones, infrared sensors, and vehicle instrument sensors. A second data level can include, but is not limited to, biological data associated with the user, including but not limited to, heart rate, body temperature, sweat, head position, face position, pupil diameter data, and gaze direction.

[0051] A third data level may include processed data representing various higher states of the user, including, but not limited to, emotional states. Emotional state data indicates how the user is feeling. Emotional state data can be divided into emotional arousal data and emotional valence data. Emotional arousal data represents the degree of emotional state experienced by the user. Emotional valence data indicates whether the emotional state is associated with a positive emotion, such as happiness or satisfaction, or a negative emotion, such as anger or frustration.

[0052] In one particular example, a user is driving on the highway, heading home on a beautiful day in a very calm and scenic environment. The user is listening to heavy metal music on the vehicle's media player. The driver emotion recognition module 232 detects that the user is in an agitated and aggressive state. The media emotion recognition module 234 detects the agitated and aggressive state due to the driver listening to heavy metal music. The emotion normalizer module 238 removes the agitated and aggressive component of the emotional state due to the heavy metal music content from the driver's overall emotional state. The emotion normalizer module 238 determines that the resulting emotional state is calmer and less stressed. As a result, the emotion normalizer module 238 does not perform any responsive action.

[0053] In another particular example, a user is driving in a crowded urban area during heavy rain and sleet. The user is listening to a tape book of meditation instructions. The driver emotion recognition module 232 detects that the user's emotional state is very relaxed, almost sleepy. The media emotion recognition module 234 analyzes the media content the user is listening to and determines that much of the user's calmness is due to the calm media content. The emotion normalizer module 238 removes the calm component of the emotional state due to the meditation-related media content from the driver's overall emotional state. The emotion normalizer module 238 determines that the resulting emotional state due to the driving conditions is indicative of stressful driving conditions. As a result, the emotion normalizer module 238 can perform one or more responsive actions.

[0054] In yet another particular example, after the emotion normalizer module 238 determines the user's emotional state due to driving conditions, the emotion normalizer module 238 transmits this emotional state data to the telemetrics and radio system 104. The telemetrics and radio system 104 then transmits this emotional state data to the remote server system 102. The remote server system 102 receives the emotional state data from various emotion analysis systems 108 within a particular geographic region. The remote server system 102 generates a heat map of the geographic region showing the happiness, stress, and / or other emotional states of each driver within the region. The remote server system 102 can aggregate the emotional states of each driver within the geographic region over time and / or across groups of drivers. The remote server system 102 then transmits the heat map data to one or more emotion analysis systems 108. Drivers of particular vehicles receiving the heat map data can then select routes through areas where driving conditions result in more drivers having happy emotional states and / or fewer drivers having stressed emotional states.

[0055] 3A-3C are conceptual diagrams illustrating various configurations of the system of FIG. 1, according to various embodiments.

[0056] 3A , head unit 310 includes a core head unit module 332, a driver monitoring module 334, a telemetrics and wireless module 336, and a sentiment analysis module 338. Head unit 310 includes a computing device with sufficient processing and memory resources associated with core head unit module 332, driver monitoring module 334, telemetrics and wireless module 336, and sentiment analysis module 338. Core head unit module 332 performs various functions related to the operation of the vehicle, including, but not limited to, entertainment and media functions, navigation, and vehicle monitoring. Vehicle monitoring includes monitoring and display functions related to tire pressure, oil level, coolant temperature, vehicle maintenance, etc.

[0057] The driver monitoring module 334 performs various functions associated with the driver monitoring system 106 of Figure 1. These functions include, but are not limited to, monitoring the driver of the vehicle to determine the driver's alertness state and / or transmitting measurement data received via various devices to the sentiment analysis module 338 for further analysis.

[0058] The telemetrics and wireless module 336 performs various functions associated with the telemetrics and wireless system 104 of Figure 1. These functions include, but are not limited to, receiving measurement data from the driver monitoring system 106 and / or the sentiment analysis system 108, transmitting measurement data to the remote server system 102, receiving data from the remote server 102, forwarding data received from the remote server 102 to the driver monitoring system 106 and / or the sentiment analysis system 108, and / or the like.

[0059] The sentiment analysis module 338 performs various functions associated with the sentiment analysis system 108 of FIG. 1. These functions include, but are not limited to, receiving measurement data received from the driver monitoring module 334 via various devices, analyzing the measurement data to generate processed data regarding the emotional state of the driver or other user, and / or storing the measurement data, the processed data, or both. In some embodiments, the sentiment analysis module 338 can transmit the measurement data, the processed data, or both to the telemetrics and wireless module 336. The telemetrics and wireless module 336 can then transmit the measurement data and / or the processed data to the remote server system 102 via the communications network 110.

[0060] In some embodiments, the head unit may not have sufficient processor and memory resources to execute all of the core head unit module 332, the driver monitoring module 334, the telemetrics and wireless module 336, and the sentiment analysis module 338. As a result, one or more of these modules may execute on a computing device associated with one or more auxiliary units. Such auxiliary units may include an internal computing device along with local and / or remote connections to one or more communication channels. One exemplary auxiliary unit may be a “dongle” inserted into a port on the vehicle, such as an on-board diagnostics 2 (OBD2) port, where the dongle is a small device that can connect to and communicate with another device, such as the head unit. Another exemplary auxiliary unit may be a unit embedded in the vehicle's dash panel, under the driver's or passenger's seat, or elsewhere in the vehicle. Yet another exemplary auxiliary unit may be a smartphone or other mobile device running an application that communicates with another device, such as the head unit, via one or more wired or wireless communication channels. Such auxiliary units may include a computing device that, when executing instructions, is capable of performing any one or more of the techniques described herein. Additionally, such auxiliary units may include wired and / or wireless network interfaces for communicating with one or more local and / or remote devices.

[0061] 3B , head unit 312 includes a core head unit module 332, a driver monitoring module 334, and a telemetrics and radio module 336. Head unit 312 includes a computing device with sufficient processing and memory resources associated with core head unit module 332, driver monitoring module 334, and telemetrics and radio module 336. Head unit 312 communicates with auxiliary unit 322, which includes a sentiment analysis module 338. Auxiliary unit 322 includes a computing device with sufficient processing and memory resources associated with sentiment analysis module 338.

[0062] In some embodiments, the legacy head unit may include only functionality related to the core head unit module 332. In such embodiments, the remaining functionality may be performed on a computing device associated with one or more auxiliary units.

[0063] 3C , head unit 314 includes a core head unit module 332. Head unit 312 includes a computing device with sufficient processing and memory resources associated with core head unit module 332. Head unit 312 communicates with auxiliary unit 322, which includes a driver monitoring module 334, a telemetrics and wireless module 336, and a sentiment analysis module 338. Auxiliary unit 322 includes a computing device with sufficient processing and memory resources associated with driver monitoring module 334, a telemetrics and wireless module 336, and a sentiment analysis module 338.

[0064] Although particular configurations are shown in FIGS. 3A-3C, the functionality associated with the core head unit module 332, the driver monitoring module 334, the telemetrics and wireless module 336, and the sentiment analysis module 338 may be executed on one or more computing devices in any technically feasible combination and configuration within the scope of this disclosure.

[0065] 4A-4B illustrate exemplary arrangements of sensors associated with the system of FIG. 1 , according to various embodiments. As shown in FIG. 4A , a steering wheel 410 is equipped with a psychophysiological sensor 430 and a camera 432. The psychophysiological sensor 430 may be configured to measure technically feasible psychophysiological data through contact with the user's hands, including, but not limited to, heart rate, body temperature, and sweat data. The camera 432 can capture still or video images. The captured images may include technically feasible image data, including, but not limited to, color, black-and-white, thermal, and infrared images. The psychophysiological sensor 430 and the camera 432 can transmit the psychophysiological data and images to one or both of the driver monitoring system 106 and the emotion analysis system 108. Also shown in FIG. 4B is a head unit 420 equipped with a camera 434. The camera 434 can capture still or video images. The captured images may include any technically feasible image data, including, but not limited to, color images, black and white images, thermal images, and infrared images. Camera 434 may transmit images to one or both of driver monitoring system 106 and emotion analysis system 108. It will be understood that the systems illustrated herein are exemplary, and that variations and modifications are possible. In particular, the various sensors, including psychophysiological sensor 430 and cameras 432 and 434, may be located on the surface of the vehicle dashboard, integrated into the vehicle's instrument cluster, concealed within the vehicle's display unit, under the vehicle's rearview mirror, and / or similarly located in any technically feasible location.

[0066] 5A-5B illustrate exemplary models for mapping emotional states along various dimensions, according to various embodiments. As shown in FIG. 5A, a two-dimensional model 500 maps emotional states along two dimensions: an emotional valence dimension 510 and an emotional arousal dimension 512.

[0067] The emotional valence dimension 510 is a measure of how pleasant or unpleasant a user feels. For example, anger and sadness represent unpleasant emotions. Therefore, anger and sadness are located within the negative valence region of the two-dimensional model 500. On the other hand, joy and / or pleasure represent positive emotions. Therefore, joy and pleasure are located within the positive valence region of the two-dimensional model 500.

[0068] The emotional arousal dimension 512 measures how energized or sleepy the user feels, rather than the intensity of the emotion. In that mood, sadness and happiness represent low arousal emotions and are therefore placed in the low arousal region of the two-dimensional model 500. Anger and joy represent high arousal emotions and are therefore placed in the high arousal region of the two-dimensional model 500.

[0069] As shown in Figure 5B, three-dimensional model 550 maps emotional states along three dimensions, also referred to herein as emotional stance dimensions: emotional valence dimension 560, emotional arousal dimension 562, and emotional dominance dimension 564. Emotional valence dimension 560 is a measure of how comfortable or uncomfortable a user feels and is analogous to emotional valence dimension 510 of Figure 5A. Emotional arousal dimension 562 measures how energized or drowsy a user feels and is analogous to emotional arousal dimension 512 of Figure 5A.

[0070] The affective dominance dimension 564 represents the user's affective state with respect to dominance, control, or stance. A closed stance represents an affective state of dominance or control. An open stance represents an affective state of submission or being controlled.

[0071] In that mood, disgust and anger represent closed stance or dominant emotions and are therefore placed in the closed stance region of three-dimensional model 550. Acceptance and fear represent open stance or submissive emotions and are therefore placed in the open stance region of three-dimensional model 550.

[0072] When measuring the overall emotional state of the driver, the driver emotion recognition module 232 can use the two-dimensional model 500 of Figure 5A or the three-dimensional model 550 of Figure 5B. Similarly, when measuring specific components of the driver's emotional state due to media content or other factors, the media emotion recognition module 234 can use the two-dimensional model 500 of Figure 5A or the three-dimensional model 550 of Figure 5B.

[0073] 6 is a flowchart of method steps for calculating and analyzing a user's emotional state, according to various embodiments. Although the method steps are described in conjunction with the systems of FIGS. 1-5, one skilled in the art will understand that any system configured to perform the method steps in any order is within the scope of the present disclosure.

[0074] As shown, method 600 begins at step 602, where the driver emotion recognition module 232 executing on the emotion analysis system 108 obtains sensor data related to the user's emotional state. The sensor data may be raw sensor data, including, but not limited to, data from a camera, microphone, infrared sensor, vehicle instrument sensor, and / or the like. Additionally or alternatively, the sensor data may include, but is not limited to, biological data associated with the user, including, but not limited to, heart rate, body temperature, sweat, head position, face position, pupil diameter data, gaze direction, and / or the like.

[0075] In step 604, the driver emotion recognition module 232 determines the user's overall emotional state based on the sensor data. The driver emotion recognition module 232 may determine the user's overall emotional state based on a two-dimensional model. In such a case, the driver emotion recognition module 232 may determine a first numerical value for emotional arousal and a second numerical value for emotional valence. Alternatively, the driver emotion recognition module 232 may determine the user's overall emotional state based on a three-dimensional model. In such a case, the driver emotion recognition module 232 may determine a first numerical value for emotional arousal, a second numerical value for emotional valence, and a third numerical value for emotional dominance. Alternatively, the driver emotion recognition module 232 may determine the user's overall emotional state based on a higher dimensional model. Alternatively, the driver emotion recognition module 232 may determine the user's overall emotional state on an emotion-by-emotion basis. In such a case, the driver emotion recognition module 232 may determine the overall emotional state for each of a set of specified emotions.

[0076] In step 606, the media emotion recognition module 234 running on the emotion analysis system 108 determines the user's emotional state as a result of listening to the media content. Generally, the media emotion recognition module 234 uses the same two-dimensional, three-dimensional, higher-dimensional, or emotion-specific analysis used by the driver emotion recognition module 232.

[0077] In step 608, the emotion normalizer module 238 running on the emotion analysis system 108 removes the user's emotional state due to listening to the media content from the user's overall emotional state. As a result, the remaining emotional state data represents the driver's emotional state due to factors other than listening to the media content. In that respect, the remaining emotional state data more accurately represents the driver's emotional state due to driving conditions.

[0078] In step 610, the speech emotion recognition module 236 executing on the emotion analysis system 108 determines the user's emotional state based on other factors. Such other factors may include, but are not limited to, in-person conversations between the driver and passengers, phone conversations between the driver and other people, and / or the like. Generally, the speech emotion recognition module 236 uses the same two-dimensional, three-dimensional, higher-dimensional, or emotion-specific analysis used by the driver emotion recognition module 232 and / or the media emotion recognition module 234.

[0079] In step 612, the emotion normalizer module 238 removes the user's emotional state due to these other factors. As a result, the remaining emotional state data represents the driver's emotional state due to factors other than listening to media content, the driver's speech, and / or the like. In that respect, the remaining emotional state data even more accurately represents the driver's emotional state due to driving conditions.

[0080] In step 614, the emotion normalizer module 238 performs one or more response actions based on the user's emotional state after removing components related to listening to media content, driver speech, and / or the like. As a result, the user's remaining emotional state after such removal is due entirely or primarily to driving conditions, although some additional residual factors may continue to be present in the emotional state. These response actions may include, but are not limited to, presenting suggestions and / or alerts to the driver in text format, presenting suggestions and / or alerts to the driver in audio format, and / or the like.

[0081] In step 616, the emotion normalizer module 238 transmits the sensor data and / or emotional state data to the remote server 102 via the telemetrics and wireless system 104. In response, the remote server system 102 aggregates and analyzes the emotional state data received from the emotion recognition systems 108 associated with multiple users in a given geographic region. Based on the aggregated emotional state data, the remote server system 102 can evaluate whether driving conditions in the given geographic region are favorable or unfavorable. The central server system can route other drivers away from areas where driving conditions are rated as less favorable and toward areas where driving conditions are rated as more favorable. In this way, computer-based recognition of human emotional states can improve the experience of not only individual drivers but also groups of drivers as a whole. Method 600 then ends.

[0082] In summary, the sentiment analysis system evaluates the user's emotional state due to various input conditions. More specifically, the sentiment analysis system analyzes sensor data to determine the driver's overall emotional state. The sentiment analysis system determines the driver's emotional state resulting from listening to specific media content. The sentiment analysis system applies a function to remove components of the emotional state resulting from listening to specific media content from the overall emotional state. Optionally, the sentiment analysis system determines the driver's emotional state resulting from additional factors, such as a direct conversation between the driver and a passenger or a phone conversation between the driver and another person, and / or the like. The sentiment analysis system applies one or more additional functions to remove components of the emotional state resulting from these additional secondary effects from the overall emotional state. The resulting emotional state more accurately reflects the driver's emotional state due to the current driving conditions. The sentiment analysis system performs one or more response actions based on the user's emotional state due to the driving conditions. The response actions may include presenting suggestions and / or alerts to the driver in text, audio, or other forms, such as indicators, lights, haptic outputs, and / or the like. Additionally, the sentiment analysis system may transmit the sensor data and / or emotional state data to a remote server system that aggregates and analyzes the sensor data and / or emotional state data received from the multiple sentiment analysis systems to assess the overall emotional state of the multiple drivers due to driving conditions.

[0083] At least one technical advantage of the disclosed technology over the prior art is that it can process data related to a driver's emotional state to more accurately separate the contribution of driving conditions to the driver's emotional state from the driver's overall emotional state by removing the contribution of media content and / or other factors to the driver's emotional state. As a result, the DMS can generate more appropriate responsive actions for the driver depending on the contribution of driving conditions to the driver's emotional state. Another technical advantage of the disclosed technology is that a central server system that aggregates emotional state data from multiple drivers can use the more accurate assessment of the contribution of driving conditions to the driver's emotional state to generate a more accurate assessment of the overall favorability or unfavorability of driving conditions in a particular area. These technical advantages represent one or more technical improvements over prior art approaches.

[0084] Clause 1. In some embodiments, a computer-implemented method for calculating and analyzing an emotional state of a user includes obtaining sensor data associated with the user via at least one sensor; determining an emotional state associated with the user based on the sensor data; determining a first component of the emotional state that corresponds to media content being accessed by the user; and applying a first function to the emotional state to remove the first component from the emotional state.

[0085] Clause 2. The computer-implemented method of clause 1, further comprising determining a second component of the emotional state based on factors associated with the user, and applying a second function to the emotional state to remove the second component from the emotional state.

[0086] Clause 3. The computer-implemented method of clause 1 or clause 2, wherein the factor is associated with a direct conversation between the user and another person.

[0087] Clause 4. The computer-implemented method of any of clauses 1 to 3, wherein the factor is associated with a telephone conversation between the user and another person.

[0088] Clause 5. The computer-implemented method of any of clauses 1 to 4, wherein the media content includes musical content and determining the first component of the emotional state includes analyzing the musical content to determine the first component.

[0089] Clause 6. The computer-implemented method of any of clauses 1 to 5, wherein the media content includes music content and determining the first component of the emotional state includes retrieving the first component from a database.

[0090] Clause 7. The computer-implemented method of any of clauses 1 to 6, wherein the media content includes vocal content and determining the first component of the emotional state includes analyzing the vocal content to determine the first component based on at least one of a vocal tone or a text segment of the vocal content.

[0091] Clause 8. The computer-implemented method of any of clauses 1 to 7, wherein the emotional state includes an emotional valence value and an emotional arousal value, and applying the first function to the emotional state includes applying the first function to the emotional valence value to remove a first component of the emotional valence value that corresponds to the media content being accessed by the user, and applying the first function to the emotional arousal value to remove a first component of the emotional arousal value that corresponds to the media content being accessed by the user.

[0092] Clause 9. The computer-implemented method of any of clauses 1 to 8, wherein the emotional state further includes an emotional prominence value, and wherein applying the first function to the emotional state includes applying the first function to the emotional prominence value to remove a first component of the emotional prominence value that corresponds to the media content being accessed by the user.

[0093] Clause 10. A computer-implemented method described in any of clauses 1 to 9, wherein the emotional state includes a first emotion and a second emotion, and applying the first function to the emotional state includes applying the first function to the first emotion to remove a first component of the first emotion that corresponds to the media content being accessed by the user, and applying the first function to the second emotion to remove a second component of the second emotion that corresponds to the media content being accessed by the user.

[0094] Clause 11. In some embodiments, one or more computer-readable storage media include instructions that, when executed by one or more processors, cause the one or more processors to calculate and analyze an emotional state of the user by performing the steps of acquiring sensor data associated with a user via at least one sensor; determining an emotional state associated with the user based on the sensor data; determining a first component of the emotional state that corresponds to media content being accessed by the user; and applying a first function to the emotional state to remove the first component from the emotional state.

[0095] Clause 12. The one or more computer-readable storage media of Clause 11, wherein the instructions further cause the one or more processors to perform the steps of determining a second component of the emotional state based on factors associated with the user, and applying a second function to the emotional state to remove the second component from the emotional state.

[0096] Clause 13. One or more computer-readable storage media according to clause 11 or clause 12, wherein the factor is associated with a direct conversation between the user and another person.

[0097] Clause 14. One or more computer-readable storage media according to any of clauses 11 to 13, wherein the factor is associated with a telephone conversation between the user and another person.

[0098] Clause 15. One or more computer-readable storage media described in any of clauses 11 to 14, wherein the media content includes musical content and determining the first component of the emotional state includes analyzing the musical content to determine the first component.

[0099] Clause 16. One or more computer-readable storage media described in any of clauses 11 to 15, wherein the media content includes music content and determining the first component of the emotional state includes retrieving the first component from a database.

[0100] Clause 17. One or more computer-readable storage media described in any of clauses 11 to 16, wherein the media content includes vocal content and determining the first component of the emotional state includes analyzing the vocal content to determine the first component based on at least one of a vocal tone or a text segment of the vocal content.

[0101] Clause 18. One or more computer-readable storage media described in any of Clauses 11 to 17, wherein the emotional state includes an emotional valence value and an emotional arousal value, and applying the first function to the emotional state includes applying the first function to the emotional valence value to remove a first component of the emotional valence value that corresponds to the media content being accessed by the user, and applying the first function to the emotional arousal value to remove a first component of the emotional arousal value that corresponds to the media content being accessed by the user.

[0102] Clause 19. One or more computer-readable storage media described in any of clauses 11 to 18, wherein the emotional state further includes an emotional prominence value, and applying the first function to the emotional state includes applying the first function to the emotional prominence value to remove a first component of the emotional prominence value that corresponds to the media content being accessed by the user.

[0103] Clause 20. In some embodiments, a first endpoint device includes a memory including instructions; and a processor coupled to the memory that, when executed, acquires sensor data associated with a user via at least one sensor; determines an emotional state associated with the user based on the sensor data; determines a first component of the emotional state that corresponds to media content being accessed by the user; and applies a first function to the emotional state to remove the first component from the emotional state.

[0104] Any and all combinations, in any manner, of any claim elements recited in any claim and / or any elements described in this application are within the intended scope of the disclosure and protection.

[0105] The description of various embodiments is presented for purposes of illustration and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

[0106] Aspects of the present embodiments may be embodied as a system, a method, or a computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which may all generally be referred to herein as a "module" or a "system." Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code embodied thereon.

[0107] Any combination of one or more computer-readable medium(s) may be utilized. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media would include an electrical connection having one or more communication lines, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an electrically erasable PROM (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that contains or can store a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0108] Aspects of the present disclosure are described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine that, when executed by the processor of the computer or other programmable data processing apparatus, implements the functions / acts specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor may be, but is not limited to, a general-purpose processor, a special-purpose processor, an application-specific processor, or a field-programmable processor.

[0109] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code, including one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially simultaneously, or the blocks may be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions.

[0110] While the forgoing is directed to embodiments of the present disclosure, other and further embodiments of the present disclosure may be devised without departing from the basic scope thereof, which scope is determined by the appended claims.

Claims

1. 1. A method of operation of a system for calculating and analyzing a user's emotional state, the system comprising at least one sensor and one or more processors, the method comprising: the one or more processors acquiring, via the at least one sensor, sensor data associated with a user; determining, by the one or more processors, an emotional state associated with the user based on the sensor data; the one or more processors determining a first component of the emotional state based on media content accessed by the user; the one or more processors applying a first function to the emotional state to remove the first component from the emotional state; A method of operation comprising:

2. The one or more processors determining a second component of the emotional state based on factors associated with the user; and the one or more processors applying a second function to the emotional state to remove the second component from the emotional state; The method of claim 1 further comprising:

3. The method of claim 2 , wherein the factor is associated with a direct conversation between the user and another person.

4. 3. The method of claim 2, wherein the factor is associated with a telephone conversation between the user and another person.

5. The media content includes music content, and determining the first component of the emotional state includes: the one or more processors analyzing the musical content to determine the first component. The method of claim 1 , comprising:

6. The media content includes music content, and determining the first component of the emotional state includes: the one or more processors reading the first component from a database. The method of claim 1 , comprising:

7. The media content includes vocal content, and determining the first component of the emotional state includes: the one or more processors analyzing the vocal content to determine the first component based on at least one of a vocal tone or a text segment of the vocal content. The method of claim 1 , comprising:

8. the emotional state includes an emotional valence value and an emotional arousal value, and applying the first function to the emotional state comprises: applying, by the one or more processors, the first function to the emotional valence value to remove a first component of the emotional valence value that corresponds to the media content being accessed by the user; the one or more processors applying the first function to the emotional arousal value to remove a first component of the emotional arousal value that corresponds to the media content being accessed by the user; The method of claim 1 , comprising:

9. The emotional state further comprises an emotional dominance value, and applying the first function to the emotional state comprises: the one or more processors applying the first function to the emotional prominence value to remove a first component of the emotional prominence value that corresponds to the media content being accessed by the user. The method of claim 1 , comprising:

10. the emotional state includes a first emotion and a second emotion, and applying the first function to the emotional state includes: applying the first function to the first emotion to remove a first component of the first emotion that corresponds to the media content being accessed by the user; applying the one or more processors to the second emotion to remove a second component of the second emotion corresponding to the media content being accessed by the user; The method of claim 1 , comprising:

11. When executed by one or more processors, acquiring, via at least one sensor, sensor data associated with a user; determining an emotional state associated with the user based on the sensor data; determining a first component of the emotional state based on media content accessed by the user; applying a first function to the emotional state to remove the first component from the emotional state; one or more computer-readable storage media comprising instructions for causing the one or more processors to calculate and analyze the user's emotional state by executing:

12. The instruction: determining a second component of the emotional state based on factors associated with the user; applying a second function to the emotional state to remove the second component from the emotional state; The one or more computer-readable storage media of claim 11 , further causing the one or more processors to execute:

13. The one or more computer-readable storage media of claim 12 , wherein the factor is associated with a direct conversation between the user and another person.

14. The one or more computer-readable storage media of claim 12 , wherein the factor is associated with a telephone conversation between the user and another person.

15. The media content includes music content, and determining the first component of the emotional state includes: analyzing the musical content to determine the first component; 12. One or more computer-readable storage media according to claim 11, comprising:

16. The media content includes music content, and determining the first component of the emotional state includes: Retrieving the first component from a database.

12. One or more computer-readable storage media according to claim 11, comprising:

17. The media content includes vocal content, and determining the first component of the emotional state includes: analyzing the vocal content to determine the first component based on at least one of a tone of voice or a text segment of the vocal content; 12. One or more computer-readable storage media according to claim 11, comprising:

18. the emotional state includes an emotional valence value and an emotional arousal value, and applying the first function to the emotional state comprises: applying the first function to the emotional valence value to remove a first component of the emotional valence value that corresponds to the media content being accessed by the user; applying the first function to the arousal value to remove a first component of the arousal value that corresponds to the media content being accessed by the user; 12. One or more computer-readable storage media according to claim 11, comprising:

19. The emotional state further comprises an emotional dominance value, and applying the first function to the emotional state comprises: applying the first function to the emotional salience value to remove a first component of the emotional salience value that corresponds to the media content being accessed by the user; 12. One or more computer-readable storage media according to claim 11, comprising:

20. a memory containing instructions; a processor coupled to the memory that, when executing the instructions, acquiring, via at least one sensor, sensor data associated with a user; determining an emotional state associated with the user based on the sensor data; determining a first component of the emotional state based on media content accessed by the user; applying a first function to the emotional state to remove the first component from the emotional state; The processor and a first endpoint device including:

Images