Techniques to separate driving emotions from media-induced emotions in a driver monitoring system

Abstract

One or more embodiments include an emotion analysis system for computing and analyzing an emotional state of a user. The emotion analysis system acquires sensor data associated with a user via at least one sensor. The emotion analysis system determines an emotional state associated with the user based on the sensor data. The emotion analysis system determines a first component in the emotional state that corresponds to media content being accessed by the user. The emotion analysis system applies a first function to the emotional state to remove the first component from the emotional state.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure generally relate to psychophysiological sensing systems, and more particularly to techniques for separating driving emotions from media-induced emotions in a driver monitoring system. BACKGROUND

[0002] Computer-based recognition of human emotional states is increasingly being applied in various applications. In one particular example, a driver monitoring system (DMS) can detect an emotional state of a driver in order to assess how the driver is reacting to various driving conditions associated with the vehicle, such as weather conditions, traffic conditions, and / or road conditions. Inclement weather, heavy traffic, and poorly maintained roads can cause the driver to fall into a state of distress, anger, and / or agitation. In contrast, good weather, light traffic, and well-maintained roads can cause the driver to be in a state of calm, relaxation, and / or enjoyment. In response to the assessment of the emotional state of the driver, the DMS can take certain actions, such as presenting suggestions and / or alerts to the driver in text, or audio, or other forms, such as indicators, lights, haptic outputs, etc.

[0003] One potential drawback of the above-described techniques is that the emotional state of the driver can be caused by a variety of factors other than the current driving conditions. For example, the emotional state of the driver can be influenced by the media content that the driver is listening to, such as music, news, talk radio, an audiobook, etc. Further, the emotional state of the driver can be influenced by a face-to-face conversation between the driver and a passenger or a phone conversation between the driver and one or more other people. Even in good driving conditions, aggressive music or an unpleasant conversation can cause the driver to be in a state of distress, anger, and / or agitation. In contrast, even in stressful driving conditions, meditative music or an enjoyable conversation can cause the driver to be in a state of calm, relaxation, and / or enjoyment. Even in driving conditions where the driver should be alert and proactive, such meditative music or an enjoyable conversation can cause the driver to be in a state of calm and relaxation. Thus, in situations where the emotional state is influenced by factors other than the current driving conditions, the assessment of the driving conditions based on computer-based recognition of the emotional state of the driver can be incorrect. Thus, a DMS that relies on such incorrect assessment of the driving conditions can provide incorrect information to the driver.

[0004] As the foregoing indicates, improved techniques for determining a user’s reaction to driving conditions would be useful. SUMMARY

[0005] Various embodiments of the present disclosure describe a computer-implemented method for calculating and analyzing an emotional state of a user. The method includes acquiring sensor data associated with the user via at least one sensor. The method further includes determining an emotional state associated with the user based on the sensor data. The method further includes determining a first component of the emotional state that corresponds to media content being accessed by the user. The method further includes applying a first function to the emotional state to remove the first component from the emotional state.

[0006] Other embodiments include, but are not limited to, a system that implements one or more aspects of the disclosed technology and one or more computer-readable media including instructions for performing one or more aspects of the disclosed technology.

[0007] At least one technical advantage of the disclosed technology over the prior art is that data associated with the driver's emotional state can be processed to more accurately separate the contribution to the driver's emotional state caused by the driving condition from the driver's overall emotional state by removing the contribution to the driver's emotional state caused by media content and / or other factors. As a result, the DMS can generate a more appropriate response action for the driver in response to the contribution of the driving condition to the driver's emotional state. Another technical advantage of the disclosed technology is a central server system that aggregates emotional state data from multiple drivers and can use a more accurate assessment of the contribution of the driving condition to the driver's emotional state to generate a more accurate assessment of the overall favorable or unfavorable driving conditions in a specific area. These technical advantages represent one or more technical improvements over prior art methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order that the manner in which the above-cited features of one or more embodiments may be understood in detail, a more particular description of one or more embodiments briefly summarized above may be given by reference to certain specific embodiments, some of which are illustrated in the accompanying drawings. It should be noted, however, that the drawings illustrate only typical embodiments and, therefore, are not to be considered limiting of the scope in any way, as the scope of the present disclosure also encompasses other embodiments.

[0009] Figure 1 A system configured to implement one or more aspects of the present disclosure is shown;

[0010] Figure 2 According to various implementation plans Figure 1 A more detailed diagram of the sentiment analysis system;

[0011] Figure 3A-3C is shown according to various embodiments Figure 1 conceptual diagrams of various configurations of the system;

[0012] Figure 4A-4B Shown according to various embodiments Figure 1 Example arrangements of sensors associated with a system of;

[0013] Figure 5A-5B shows example models for mapping emotional states along various dimensions, according to various embodiments; and

[0014] Figure 6 is a flow chart of method steps for calculating and analyzing a user's emotional state, according to various embodiments. DETAILED DESCRIPTION

[0015] In the following description, numerous specific details are set forth to provide a more thorough understanding of certain embodiments. However, it will be apparent to one skilled in the art that other embodiments may be practiced without one or more of these specific details or with additional specific details.

[0016] As further described herein, existing emotion analysis systems are configured to determine the overall emotional state of a user. For example, an emotion analysis system associated with a driver monitoring system (DMS) may be configured to determine the overall emotional state of a vehicle driver. The driver's overall emotional state may be a combination of multiple factors, including but not limited to driving conditions, media content being accessed by the driver, face-to-face conversations between the driver and passengers, telephone conversations between the driver and remotely located people, and the like. Certain applications associated with the DMS benefit from an accurate assessment of the driver's emotional state due to driving conditions (such as weather conditions, traffic conditions, and / or road conditions). In this regard, the emotion analysis system of the present disclosure determines one or more components of the driver's emotional state due to factors other than the driving conditions and removes those components from the driver's overall emotional state. Therefore, the remaining emotional state data represents the driver's emotional state due to factors other than the removed components. In this regard, the remaining emotional state data more accurately represents the driver's emotional state due to the driving conditions.

[0017] In one example, a mood analysis system determines the driver's overall emotional state. The mood analysis system then determines the component of the driver's emotional state that is caused by listening to or otherwise consuming media content. The mood analysis system removes the component of the driver's emotional state that is caused by listening to media content from the driver's overall emotional state. Additionally or alternatively, the mood analysis system determines the component of the driver's emotional state that is caused by one or more additional factors, such as face-to-face conversations between the driver and passengers, telephone conversations between the driver and other persons, etc. The mood analysis system then removes the component of the driver's emotional state that is caused by each of these other factors. The remaining emotional state data more accurately represents the driver's emotional state that is caused by the driving situation.

[0018] System Overview

[0019] Figure 1 A system 100 configured to implement one or more aspects of the present disclosure is shown. As shown, the system 100 includes, but is not limited to, a remote server system 102, a telemetry and air system 104, a driver monitoring system 106, and a sentiment analysis system 108. The remote server system 102 and the telemetry and air system 104 communicate with each other via a communication network 110. The communication network 110 can be any suitable environment that enables communication between remote or local computer systems and computing devices, including but not limited to Bluetooth communication channels, wireless and wired LANs (local area networks), WANs (wide area networks), cellular networks, satellite networks, high-altitude balloon networks (and other atmospheric satellite networks), peer-to-peer networks, vehicle-to-everything (V2X) networks, and the like. The remote server system 102 and the telemetry and air system 104 communicate through the communication network 110 via communication links 112 and 114, respectively. Further, the telemetry and air system 104 communicates with the driver monitoring system 106 and the sentiment analysis system 108 via communication links 120 and 118, respectively. Driver monitoring system 106 communicates with emotion analysis system 108 via communication link 116. Communication links 116, 118, and 120 may employ any one or more technically feasible communication media and protocols in any combination.

[0020] The remote server system 102 includes, but is not limited to, a computing device that can be a stand-alone server, a cluster or "farm" of servers, one or more network appliances, or any other device suitable for implementing one or more aspects of the present disclosure. Illustratively, the remote server system 102 communicates over a communication network 110 via a communication link 112.

[0021] In operation, the remote server system 102 receives emotional state data from one or more emotion analysis systems 108. In some embodiments, the remote server system 102 performs one or more of the techniques described herein in conjunction with the emotion analysis system 108. Further, the remote server system 102 aggregates and analyzes emotional state data from multiple users in a given geographic area. Based on the aggregated emotional state data, the remote server system 102 can assess whether driving conditions in the given geographic area are favorable or unfavorable. The central server system can direct other drivers to drive away from areas assessed to have less favorable driving conditions and toward areas assessed to have more favorable driving conditions. In this way, computer-based human emotional state recognition can improve the experience of individual drivers and a group of drivers as a whole.

[0022] The telemetry and air system 104 includes, but is not limited to, a computing device, which may be a standalone server, a cluster or "farm" of servers, one or more network appliances, or any other device suitable for implementing one or more aspects of the present disclosure. Illustratively, the telemetry and air system 104 communicates via a communication link 114 over the communication network 110. Further, the telemetry and air system 104 communicates with the driver monitoring system 106 and the emotion analysis system 108 via communication links 120 and 118, respectively.

[0023] In operation, the telemetry and air system 104 receives measurement data from the driver monitoring system 106 and / or the emotion analysis system 108. The measurement data may include information related to various components of the system 100, including but not limited to sensor data, instrumentation, camera images, video, etc. The measurement data may further include processed data, wherein the driver monitoring system 106 and / or the emotion analysis system 108 analyzes certain measurement data, such as sensor data, instrumentation, camera images, video, etc., and generates processed data therefrom. Such processed data may include but not limited to emotional state data. The telemetry and air system 104 then transmits the measurement data from the driver monitoring system 106 and / or the emotion analysis system 108 to the remote server system 102 via the communication network 110.

[0024] The driver monitoring system 106 includes, but is not limited to, a computing device, which may be a standalone server, a cluster or "farm" of servers, one or more network appliances, or any other device suitable for implementing one or more aspects of the present disclosure. Illustratively, the driver monitoring system 106 communicates with the telemetry and air system 104 and the emotion analysis system 108 via communication links 120 and 116, respectively.

[0025] In operation, the driver monitoring system 106 monitors the driver of the vehicle to determine certain characteristics, such as the driver's state of alertness. The driver monitoring system 106 receives measurement data via various devices, including but not limited to cameras, microphones, infrared sensors, ultrasonic sensors, radar sensors, thermal imaging sensors, heart rate and breathing monitors, vehicle instrument sensors, etc. By analyzing the measurement data, the driver monitoring system 106 determines the driver's overall physiological state, which may include the driver's level of alertness. If the driver monitoring system 106 determines that the driver is not sufficiently alert, the driver monitoring system 106 may initiate certain response actions, including but not limited to flashing interior lights, sounding an alarm, applying brakes to safely slow down or stop the vehicle, etc. Further, the driver monitoring system 106 transmits the measurement data received via the various devices to the emotion analysis system 108 for additional analysis, as further described herein.

[0026] The emotion analysis system 108 includes, but is not limited to, a computing device, which may be a standalone server, a cluster or "farm" of servers, one or more network appliances, or any other device suitable for implementing one or more aspects of the present disclosure. Illustratively, the emotion analysis system 108 communicates with the telemetry and air system 104 and the driver monitoring system 106 via communication links 118 and 116, respectively.

[0027] In operation, the emotion analysis system 108 receives measurement data from the driver monitoring system 106. The measurement data is received via various devices associated with the driver monitoring system 106. The emotion analysis system 108 analyzes the measurement data to generate processed data related to the emotional state of the driver or other user, as further described herein. The emotion analysis system 108 stores one or both of the measurement data and the processed data in a data storage area. In some embodiments, the emotion analysis system 108 may transmit the processed data to the driver monitoring system 106. The driver monitoring system 106 may then perform one or more response actions based on the processed data. In some embodiments, the emotion analysis system 108 may transmit one or both of the measurement data and the processed data to the telemetry and air system 104. The telemetry and air system 104 may then transmit the measurement data and / or the processed data to the remote server system 102 via the communication network 110.

[0028] It should be understood that the systems shown herein are illustrative and that variations and modifications are possible. In one example, the remote server system 102, the telemetry and air system 104, the driver monitoring system 106, and the emotion analysis system 108 are shown as communicating via certain networking and communication links. However, within the scope of the present disclosure, the remote server system 102, the telemetry and air system 104, the driver monitoring system 106, and the emotion analysis system 108 may communicate with each other in any technically feasible combination via any technically feasible networking and communication links.

[0029] In another example, the remote server system 102, the telemetry and air system 104, the driver monitoring system 106, and the emotion analysis system 108 are shown as included in Figure 1 108. However, the techniques performed by the remote server system 102, telemetry and air system 104, driver monitoring system 106, and emotion analysis system 108 may be performed by one or more applications or modules executed on one or more technically feasible processors included in one or more computing devices, in any technically feasible combination. Such computing devices may include, but are not limited to, head units and auxiliary units deployed in vehicles. In another example, the remote server system 102, telemetry and air system 104, driver monitoring system 106, and emotion analysis system 108 are shown and described in the context of a vehicle-based computing system that receives and processes emotional state data of a driver and / or one or more passengers. However, the techniques described herein may be deployed in any technically feasible system that receives and monitors the emotional state of a user, including, but not limited to, smartphones, laptops, tablet computers, desktop computers, and the like. In another example, the system 100 may include any technically feasible number of remote server systems 102, telemetry and air system 104, driver monitoring system 106, and emotion analysis system 108 in any technically feasible combination.

[0030] Operation of the emotional state analysis system

[0031] Figure 2 According to various implementation plans Figure 1 1 is a more detailed diagram of the sentiment analysis system 108. As shown, the sentiment analysis system 108 includes, but is not limited to, a processor 202, a storage device 204, an input / output (I / O) device interface 206, a network interface 208, an interconnect 210, and a system memory 212.

[0032] The processor 202 retrieves and executes programmed instructions stored in the system memory 212. Similarly, the processor 202 stores and retrieves application data that resides in the system memory 212. The interconnect 210 facilitates transfers, such as transfers of programmed instructions and application data, between the processor 202, the I / O device interface 206, the storage 204, the network interface 208, and the system memory 212. The I / O device interface 206 is configured to receive input data from a user I / O device 222. Examples of the user I / O device 222 can include one or more buttons, a keyboard, a mouse or other pointing device, and the like. The I / O device interface 206 can also include an audio output unit configured to generate electrical audio output signals, and the user I / O device 222 can further include a speaker configured to generate sound output in response to the electrical audio output signals. Another example of the user I / O device 222 is a display device, which generally represents any technologically feasible means for generating images for display. For example, the display device can be a liquid crystal display (LCD) display, an organic light-emitting diode (OLED) display, or a digital light processing (DLP) display. The display device can be a TV that includes a broadcast or cable tuner for receiving digital or analog television signals. The display device can be included in a VR / AR headset or heads-up display (HUD) assembly. Further, the display device can project images onto one or more surfaces, such as a wall, a projection screen, or a windshield of a vehicle. Additionally or alternatively, the display device can project images directly onto a user’s eyes (e.g., via retinal projection).

[0033] The processor 202 is included to represent a single central processing unit (CPU), multiple CPUs, a single CPU having multiple processing cores, a digital signal processor (DSP), a field programmable gate array (FPGA), a graphics processing unit (GPU), a tensor processing unit, and the like. And the system memory 212 is generally included to represent a random access memory. The storage 204 can be a disk drive storage device. Although shown as a single unit, the storage 204 can be a combination of fixed and / or removable storage devices, such as a fixed disk drive, a floppy disk drive, a magnetic tape drive, a removable memory card or optical storage, a network attached storage (NAS), or a storage area network (SAN). The processor 202 communicates with other computing devices and systems via the network interface 208, which is configured to transmit and receive data via a communication network.

[0034] The system memory 212 includes, but is not limited to, a driver emotion recognition module 232, a media emotion recognition module 234, a conversation emotion recognition module 236, an emotion normalizer module 238, and a data storage area 242. When executed by the processor 202, the driver emotion recognition module 232, the media emotion recognition module 234, and the emotion normalizer module 238 perform the same operations as described above. Figure 1 When performing operations associated with the emotion analysis system 108 , the driver emotion recognition module 232 , the media emotion recognition module 234 , and the emotion normalizer module 238 may store data in and retrieve data from the data store 242 .

[0035] In operation, the driver emotion recognition module 232 determines and classifies the driver's emotional state based on various sensor data, which receives facial features and other visual cues, voice tones and other audio cues, physiological signals, steering wheel shaking or other movements, frequent braking, etc. The driver emotion recognition module 232 can classify emotional states based on a two-dimensional model or a three-dimensional model of the emotional state, as further described herein. Emotional states are typically described qualitatively using descriptive terms such as happiness, sadness, anger, and pleasure. Such descriptive terms may be difficult to analyze algorithmically. Therefore, the driver emotion recognition module 232 determines numerical values ​​along various dimensions to represent the driver's emotional state. In the two-dimensional model, the driver emotion recognition module 232 determines the values ​​of two digital representations of the emotional state. For example, the driver emotion recognition module 232 can determine a first numerical value for emotional arousal and a second numerical value for emotional valence. In the two-dimensional model, the driver's emotional state can be expressed in the following formula 1:

[0036] mood 驾驶员 =f(driver 效价 , driver 唤醒 ) Formula 1

[0037] In the three-dimensional model, the driver emotion recognition module 232 determines three numerical values ​​representing the emotional state. For example, the driver emotion recognition module 232 may determine a first numerical value for emotional arousal, a second numerical value for emotional valence, and a third numerical value for emotional dominance. Emotional dominance is also referred to herein as emotional stance. In the three-dimensional model, the driver's emotional state can be expressed using the following formula 2:

[0038] mood 驾驶员 =f(driver 效价 , driver 唤醒 , driver 主导 ) Formula 2

[0039] Regardless of whether a two-dimensional model or a three-dimensional model is used, the driver emotion recognition module 232 determines the driver's emotional state due to all relevant factors. In the following discussion, a three-dimensional model is assumed for clarity. However, each of the techniques disclosed herein may employ a two-dimensional model, a three-dimensional model, or a higher-dimensional model in any technically feasible combination.

[0040] In operation, the media emotion recognition module 234 determines and classifies the emotional content of the media being accessed by the driver and / or other users due to certain factors. The emotional content of the media being accessed by the driver and / or other users may affect at least a portion of the driver's and / or other users' emotional state. The media emotion recognition module 234 may determine and classify the emotional content of the media using any technically feasible technique, including but not limited to algorithmic techniques and machine learning techniques. The media emotion recognition module 234 may classify the emotional content based on a two-dimensional model, a three-dimensional model, or a higher-dimensional model of the emotional state, as further described herein.

[0041] More specifically, the media emotion recognition module 234 determines the emotional content of the media content currently playing in the vehicle. The media content can be in any format, including but not limited to music, news programs, talk radio, and the like. For example, if the driver is listening to an entertaining talk radio program, the media emotion recognition module 234 may determine that the driver is experiencing a happy, joyful, and / or excited emotional state due to the media content. If the driver is listening to loud and aggressive music, the media emotion recognition module 234 may determine that the driver is experiencing an aggressive and / or stressed emotional state due to the media content. The media emotion recognition module 234 may classify the media content using any one or more technically feasible techniques. In one example, the media emotion recognition module 234 may acoustically analyze the music content to determine the typical emotional state represented or evoked by the media content. In another example, the media emotion recognition module 234 may retrieve song lyrics from a music lyrics database. Additionally or alternatively, the media emotion recognition module 234 may perform speech-to-text conversion on the music lyrics, news program, or talk radio program. The media emotion recognition module 234 may analyze the resulting text and determine the emotional state associated with the text. Additionally or alternatively, the media emotion recognition module 234 may analyze the vocal intonation of the sung or spoken words to associate an emotional state based on whether the vocal intonation is aggressive, soothing, or the like.

[0042] In yet another example, the media emotion identification module 234 can classify the media content based on previously defined categories in the form of descriptions, tags, labels, metadata, etc. The media emotion identification module 234 maps or otherwise correlates such descriptions, tags, labels, or metadata to particular emotional states. The media emotion identification module 234 can classify the media content as certain types of media content and generate appropriate labels for the media content using a heuristic approach. For example, the media emotion identification module 234 can classify media content categorized as comedy as inducing a happy emotional state. Similarly, the media emotion identification module 234 can classify media content categorized as heavy metal as inducing an aggressive emotional state. The media emotion identification module 234 converts these categories or labels to a two-dimensional model (including emotional arousal and emotional valence), a three-dimensional model (including emotional arousal, emotional valence, and emotional dominance), or a higher dimensional model. In the three-dimensional model, the emotional state resulting from the media content represented by the media content can be expressed with Equation 3 below:

[0043] Emotional State 媒体 = f(Media 效价 , Media 唤醒 , Media 主导 ) Equation 3

[0044] In operation, the emotional normalizer module 238 continuously generates driver emotional state data resulting from driving conditions by removing the component of the emotional state resulting from listening to media content from the overall emotional state. The emotional normalizer module 238 receives the overall emotional state of the driver from the driver emotion identification module 232. The emotional normalizer module 238 receives the emotional state of the driver resulting from listening to media content from the media emotion identification module 234. In response, the emotional normalizer module 238 removes the component representing the emotional state of the driver resulting from listening to media content from the overall emotional state of the driver. The emotional normalizer module 238 removes the component via any technically feasible technique, including but not limited to additive / subtractive techniques, conjunctive techniques, disjunctive techniques, Bayesian model techniques, etc. The remaining emotional state data represents the emotional state of the driver resulting from factors other than listening to media content. Thus, the remaining emotional state data more accurately represents the emotional state of the driver resulting from driving conditions. The emotional state of the driver resulting from driving conditions can be expressed with Equation 4 below:

[0045] Emotional State 因驾驶引起 = f(Emotional 驾驶员 , Emotional 媒体 ) Equation 4

[0046] The emotional state of the driver resulting from driving conditions can be fully expressed with Equation 5 below using the three-dimensional model:

[0047] Emotional State因驾驶引起 =f{(driver 效价 ,media 效价 ),(driver 唤醒 ,media 唤醒 ),(driver 主导 ,media 主导 )}Formula 5 If the emotion 驾驶员 and emotions 媒体 With a subtraction relationship, the driver's emotional state caused by driving conditions can be equivalently expressed by the following formulas 6 and 7

[0048] mood 因驾驶引起 =Emotion 驾驶员 -mood 媒体 Formula 6 mood 因驾驶引起 =f((driver 效价 -media 效价 ),(driver 唤醒 -media 唤醒 ),(driver 主导 -media 主导 ))Formula 7

[0049] In some embodiments, the conversation emotion recognition module 236 may be combined with the emotion normalizer module 238 to remove one or more additional components of the driver's overall emotional state to further refine the resulting emotional state. In one example, the conversation emotion recognition module 236 may analyze a face-to-face conversation between the driver and a passenger, or a telephone conversation between the driver and another person. The conversation emotion recognition module 236 may perform speech-to-text conversion on the driver's spoken words. The conversation emotion recognition module 236 may analyze the resulting text and determine an emotional state associated with the text. Additionally or alternatively, the conversation emotion recognition module 236 may analyze the vocal intonation of the spoken words to associate an emotional state based on whether the vocal intonation is aggressive, soothing, or the like. Similarly, the conversation emotion recognition module 236 may analyze the text and vocal intonation of a passenger or other person participating in the conversation. In such cases, the conversation emotion recognition module 236 may assign a higher weight to the emotional state derived from the driver's speech than to the emotional state derived from the passenger's or other person's speech. The emotion normalizer module 238 may then remove this additional component from the driver's emotional state. Then, the driver's emotional state caused by the driving situation can be expressed by the following formula 8:

[0050] mood 因驾驶引起 =f(emotion 驾驶员 ,mood 媒体 ,mood 因交谈引起 ) Formula 8

[0051] If the emotion 驾驶员,mood 媒体 and emotions 因交谈引起 With a subtraction relationship, the driver's emotional state caused by driving conditions can be equivalently expressed by the following formula 9:

[0052] mood 因驾驶引起 =Emotion 驾驶员 -mood 媒体 -mood 因交谈引起 Formula 9

[0053] It should be understood that the system shown herein is illustrative, and variations and modifications are possible. Specifically, the emotion analysis system 108 may not include the conversation emotion recognition module 236. In such a case, the emotion analysis system 108 may not perform the functions described in conjunction with the conversation emotion recognition module 236. Additionally or alternatively, one or more of the functions described in conjunction with the conversation emotion recognition module 236 may be performed by one or more other modules, such as the driver emotion recognition module 232, the media emotion recognition module 234, the emotion normalizer module 238, and the like.

[0054] In some embodiments, emotional states are analyzed on an emotion-by-emotion basis rather than based on a two-dimensional or three-dimensional model. In such embodiments, the media emotion recognition module 234 and / or the conversation emotion recognition module 236 may analyze happiness, anger, and / or any other relevant emotions one at a time. The emotion normalizer module 238 may then remove components due to media content, driver conversation, and / or other components on an emotion-by-emotion basis. The resulting emotional states of happiness and anger due to the driving situation may then be expressed using the following equations 10 and 11:

[0055] happiness 因驾驶引起 =f(happiness 驾驶员 ,happiness 媒体 ) Formula 10

[0056] anger 因驾驶引起 =f(anger 驾驶员 ,anger 媒体 ) Formula 11

[0057] If the elements of Formulas 10 and 11 have a subtractive relationship, the emotional states of happiness and anger caused by the driving situation can be equivalently expressed by the following Formulas 12 and 13:

[0058] happiness 因驾驶引起 =Happiness 驾驶员 -happiness 媒体 Formula 12

[0059] anger 因驾驶引起 =Anger 驾驶员 -anger 媒体 Formula 13

[0060] In some embodiments, the media emotion recognition module 234 and / or the conversation emotion recognition module 236 can generate emotional state data customized for a specific driver. As an example, aggressive heavy metal music typically causes the driver's emotional state to become more distressed and aggressive. However, a driver with a strong affinity for heavy metal music may experience a calmer and / or more pleasant emotional state when listening to such music. Similarly, calm, meditative music typically causes the driver's emotional state to become calmer and / or more pleasant. However, a driver who strongly dislikes meditative music may experience a more agitated and / or more stressed emotional state when listening to such music.

[0061] In some embodiments, the media emotion recognition module 234 and / or the conversation emotion recognition module 236 can track changes in emotional state due to media content, conversation, and / or other components over time. For example, a driver may be listening to hard rock music and then switch to light music. Therefore, the media emotion recognition module 234 can determine that the driver's emotional state has become calmer and less stressed due to the change from hard rock music to light music, rather than a change in driving conditions. Similarly, the driver may be listening to light music and then switch to hard rock music. Therefore, the media emotion recognition module 234 can determine that the driver's emotional state has become more agitated and more stressed due to the change from light music to hard rock music, rather than a change in driving conditions.

[0062] The various types of sensor data and associated processing are now described in more detail.The sensor data is categorized into emotion sensing, physiological sensing, behavioral sensing, acoustic sensing, and pupillometry-based cognitive workload sensing.

[0063] Emotion sensing involves detecting and classifying emotions and emotional states. Emotion sensing involves detecting discreet and known emotions, such as happiness, contentment, anger, and frustration. Emotion sensing involves calculating parameterized metrics related to emotional states, such as emotional arousal and emotional valence. Emotion sensing is based on data received from various types of sensors.

[0064] Sensor data can be received from psychophysiological sensors that measure various biological and physiological signals associated with a user, including but not limited to perspiration, heart rate, respiration rate, blood flow, blood oxygen levels, galvanic skin response, temperature, sounds emitted by the user, behavior of the user, and the like. Such sensor data represents various types of signals that are relevant to emotion detection. Further, image data can be received from cameras and other imaging sensors configured to capture still and motion images, including but not limited to color images, black and white images, thermal images, infrared images, and the like. Such cameras and imaging sensors capture facial expressions of the user or other images of the user’s body position and / or contortion that can be indicative of an emotion. In some embodiments, images can be received from an array of cameras or imaging sensors in order to capture multiple perspectives of the user’s body and head simultaneously. Further, in some embodiments, images can be received from depth cameras or imaging sensors in order to sense body posture and body positioning.

[0065] Physiological sensing includes detection systems that capture various physiological signals related to emotional states. Signals received from such sensors are related to certain emotional states and, thus, to emotion classification. For example, galvanic skin response can be indicative of the intensity of an emotional state. Physiological sensors can include, but are not limited to, galvanic skin response sensors to measure changes in electrical resistance of the skin caused by emotional stress, imagers to detect blood oxygen levels, thermal sensors to detect blood flow, optical sensors to detect blood flow, EEG systems to detect brain surface potentials, EOG sensors (electrooculogram sensors that measure eye movement by monitoring the electrical potential between the front and back of the human eye), EMG sensors (electromyography sensors that measure electrical activity in response to neural stimulation of muscles), ECG sensors (electrocardiogram sensors that measure electrical activity of the heart), high frequency radio sensors (such as GHz band radios to measure heart rate and respiration rate), nervous systems to detect neural correlates of emotion, and the like.

[0066] Acoustic sensing includes analyzing the words spoken by the user and how the user speaks a given phrase, which are indicative of emotional sentiment. Acoustic sensing further includes non- speech human sounds emitted by the user, including but not limited to whistling, humming, laughing, or screaming, which can be indicative of the emotional state of the user. In one example, natural language processing methods, sentiment analysis, and / or speech analysis can measure emotion via semantic meaning of language. In another example, voice tonality analysis can detect emotion from actual speech signals. Both methods can be used individually or in combination. Typical acoustic sensor data includes, but is not limited to, microphones, microphone arrays, and / or other audio sensing technology.

[0067] Behavior sensing includes detecting activities of users within and around the vehicle. Some of the sensors further described herein can be used to detect motion within and around the vehicle. Application and service usage data can also indicate behavior of the user and infer emotion through a classification system. In one example, mobile usage data can indicate patterns of application usage by the user that are related to specific emotional states. If the application is classified as a gaming application or a social application, execution of such an application can be related to happiness, well-being, and / or related social emotions. Behavior sensors can further include, but are not limited to, cameras, imaging sensors, auditory sensors, depth cameras, pressure sensors, etc. These sensors record body position, motion, and other behavior of the user in and around the vehicle. Such body position, motion, and / or behavior data can be related to emotions such as boredom, fatigue, and alertness. Behavior sensors can further include, but are not limited to, touch sensors, acoustic sensors, recording of button presses, or other user interface interactions that determine how the user behaves in the vehicle, etc. Such sensor data can indicate which systems the user is accessing at any given time and where the user has his or her hands.

[0068] Cognitive workload sensing based on pupil measurements measures small fluctuations in the diameter of the user's pupils. Such small fluctuations have been scientifically linked to the cognitive workload that the user is experiencing from time to time. Other related techniques can be employed to measure cognitive workload. Sensors that are capable of measuring cognitive workload include, but are not limited to, cameras and imaging sensors that image the user's pupils in order to measure changes in pupil diameter and / or eye movement. Such cameras and imaging sensors include, but are not limited to, infrared cameras, thermal sensors, high resolution color or black and white cameras, camera arrays that capture multiple perspectives of the user's body and head, etc. Physiological sensors include, but are not limited to, galvanic skin response sensors, heart rate sensors, skin temperature sensors, etc. that measure cognitive workload at relatively low resolution. In some embodiments, EEG and other neural interfaces can detect multiple levels of cognitive workload. Methods related to measuring cognitive workload from EEG and other neural data include spectral entropy, weighted average frequency, bandwidth, spectral edge frequency, etc. In some embodiments, speech analysis can be used for cognitive workload sensing. In particular, spectral centroid frequency and amplitude, with some parameter fitting to filter length and number of filters, can successfully classify various levels of cognitive workload.

[0069] In some embodiments, the driver emotion recognition module 232, the media emotion recognition module 234, and / or the emotion normalizer module 238 can partition the sensor data and processed data into several levels, where each level is associated with a different degree of abstraction. A first level of data can include raw sensor data, including but not limited to data from cameras, microphones, infrared sensors, and vehicle instrument sensors. A second level of data can include, but is not limited to, biometric data associated with the user, including but not limited to heart rate, temperature, perspiration, head position, face position, pupil diameter data, and eye gaze direction.

[0070] A third level of data can include processed data that represents various higher states of the user, including but not limited to emotional states. Emotional state data indicates the feelings of the user. The emotional state data can be separated into emotional arousal data and emotional valence data. Emotional arousal data represents the degree of emotional state experienced by the user. Emotional valence data indicates whether the emotional state is associated with positive emotions, such as happiness and satisfaction, or negative emotions, such as anger and frustration.

[0071] In one particular example, the user is driving home on a highway, the weather is nice, and the environment is very quiet and beautiful. The user is listening to heavy metal music on the vehicle's media player. The driver emotion recognition module 232 detects that the user is in an agitated, aggressive state. The media emotion recognition module 234 detects the agitated, aggressive state due to the heavy metal music being listened to by the driver. The emotion normalizer module 238 removes the agitated, aggressive component of the emotional state due to the heavy metal music content from the overall emotional state of the driver. The emotion normalizer module 238 determines that the resulting emotional state is more calm and stress-free. Accordingly, the emotion normalizer module 238 does not perform any responsive action.

[0072] In another particular example, the user is driving in a busy city during a rainstorm and sleet. The user is listening to a guided meditation book on a tape. The driver emotion recognition module 232 detects that the user's emotional state is extremely relaxed and almost drowsy. The media emotion recognition module 234 analyzes the media content being listened to by the user and determines that the user's calmness is mostly due to the calming media content. The emotion normalizer module 238 removes the calm component of the emotional state due to the meditation-related media content from the overall emotional state of the driver. The emotion normalizer module 238 determines that the resulting emotional state due to the driving conditions indicates stressful driving conditions. Accordingly, the emotion normalizer module 238 can perform one or more responsive actions.

[0073] In yet another specific example, after the emotion normalizer module 238 determines the user's emotional state due to the driving conditions, the emotion normalizer module 238 transmits the emotional state data to the telemetry and air system 104. The telemetry and air system 104 then transmits the emotional state data to the remote server system 102. The remote server system 102 receives the emotional state data from various emotion analysis systems 108 in a particular geographic area. The remote server system 102 generates a heat map of the geographic area that displays the happiness, stress, and / or other emotional states of each driver in the area. The remote server system 102 may aggregate the emotional state of each driver in the geographic area over time and / or across a group of drivers. The remote server system 102 then transmits the heat map data to one or more emotion analysis systems 108. The driver of a particular vehicle that has received the heat map data may then select a route through the area where the driving conditions result in more drivers experiencing a happy emotional state and / or fewer drivers experiencing a stressed emotional state.

[0074] Figure 3A-3C is shown according to various embodiments Figure 1 Conceptual diagram of various configurations of the system.

[0075] like Figure 3A As shown, the head unit 310 includes a core head unit module 332, a driver monitoring module 334, a telemetry and air module 336, and a sentiment analysis module 338. The head unit 310 includes a computing device with sufficient processing and memory resources associated with the core head unit module 332, the driver monitoring module 334, the telemetry and air module 336, and the sentiment analysis module 338. The core head unit module 332 performs various functions associated with the operation of the vehicle, including, but not limited to, entertainment and media functions, navigation, and vehicle monitoring. Vehicle monitoring includes monitoring and display functions related to tire pressure, oil level, coolant temperature, vehicle maintenance, etc.

[0076] The driver monitoring module 334 performs the Figure 1 Various functions associated with the driver monitoring system 106 of the vehicle. These functions include, but are not limited to, monitoring the vehicle driver to determine the driver's alertness state, and / or transmitting measurement data received via various devices to the emotion analysis module 338 for additional analysis.

[0077] Telemetry and air module 336 performs the Figure 1Various functions associated with the telemetry and airborne systems 104. These functions include, but are not limited to: receiving measurement data from the driver monitoring system 106 and / or the emotion analysis system 108; transmitting the measurement data to the remote server system 102; receiving data from the remote server 102; forwarding data received from the remote server 102 to the driver monitoring system 106 and / or the emotion analysis system 108, etc.

[0078] Sentiment analysis module 338 performs Figure 1 Various functions associated with the emotion analysis system 108 of the embodiment of the present invention may be performed. These functions include, but are not limited to: receiving measurement data received from the driver monitoring module 334 via various devices; analyzing the measurement data to generate processed data related to the emotional state of the driver or other user; and / or storing one or both of the measurement data and the processed data. In some embodiments, the emotion analysis module 338 may transmit one or both of the measurement data and the processed data to the telemetry and air module 336. The telemetry and air module 336 may then transmit the measurement data and / or the processed data to the remote server system 102 via the communication network 110.

[0079] In some embodiments, the head unit may not have sufficient processor and memory resources to execute all of the core head unit module 332, the driver monitoring module 334, the telemetry and air module 336, and the sentiment analysis module 338. Therefore, one or more of these modules may be executed on a computing device associated with one or more auxiliary units. Such auxiliary units may include internal computing devices and local and / or remote connections to one or more communication channels. One exemplary auxiliary unit may be a "dongle" that plugs into a port of the vehicle, such as the on-board diagnostics 2 (OBD2) port, where a dongle is a small device that can connect to and communicate with another device, such as a head unit. Another exemplary auxiliary unit may be a unit embedded in the dashboard of the vehicle, under the driver's seat or passenger seat of the vehicle, or elsewhere in the vehicle. Yet another exemplary auxiliary unit may be a smartphone or other mobile device that executes an application that communicates with another device, such as a head unit, via one or more wired or wireless communication channels. Any such auxiliary unit may include a computing device that, when executing instructions, can perform any one or more of the techniques described herein. Further, any such auxiliary unit may include a wired and / or wireless network interface to communicate with one or more local and / or remote devices.

[0080] like Figure 3BAs shown, the head unit 312 includes a core head unit module 332, a driver monitoring module 334, and a telemetry and air module 336. The head unit 312 includes a computing device with sufficient processing and memory resources associated with the core head unit module 332, the driver monitoring module 334, and the telemetry and air module 336. The head unit 312 communicates with an auxiliary unit 322, which includes a sentiment analysis module 338. The auxiliary unit 322 includes a computing device with sufficient processing and memory resources associated with the sentiment analysis module 338.

[0081] In some embodiments, a conventional head unit may include only the functionality associated with the core head unit module 332. In such embodiments, the remaining functionality may be performed on a computing device associated with one or more auxiliary units.

[0082] like Figure 3C As shown, the head unit 314 includes a core head unit module 332. The head unit 312 includes a computing device with sufficient processing and memory resources associated with the core head unit module 332. The head unit 312 communicates with an auxiliary unit 322, which includes a driver monitoring module 334, a telemetry and air module 336, and a sentiment analysis module 338. The auxiliary unit 322 includes a computing device with sufficient processing and memory resources associated with the driver monitoring module 334, the telemetry and air module 336, and the sentiment analysis module 338.

[0083] Despite Figure 3A-3C A specific configuration is shown in FIG, but within the scope of the present disclosure, the functionality associated with the core head unit module 332, the driver monitoring module 334, the telemetry and air module 336, and the emotion analysis module 338 may be executed on any one or more computing devices in any technically feasible combination and configuration.

[0084] Figure 4A-4B Shown according to various embodiments Figure 1 Example arrangement of sensors associated with the system. Figure 4A As shown, the steering wheel 410 is equipped with a psychophysiological sensor 430 and a camera 432. The psychophysiological sensor 430 can be configured to measure any technically feasible psychophysiological data via contact with the user's hand, including but not limited to heart rate, temperature, and perspiration data. The camera 432 can capture still or moving images. The captured images can include any technically feasible image data, including but not limited to color images, black and white images, thermal images, and infrared images. The psychophysiological sensor 430 and the camera 432 can transmit the psychophysiological data and images to one or both of the driver monitoring system 106 and the emotion analysis system 108. Similarly, Figure 4BAs shown, the head unit 420 is equipped with a camera 434. The camera 434 can capture still or moving images. The captured images may include any technically feasible image data, including but not limited to color images, black and white images, thermal images, and infrared images. The camera 434 may transmit the images to one or both of the driver monitoring system 106 and the emotion analysis system 108. It should be understood that the system shown herein is illustrative and that variations and modifications are possible. Specifically, the various sensors (including the psychophysiological sensor 430 and the cameras 432 and 434) may be placed in any technically feasible position, such as on the surface of the vehicle dashboard, integrated into the vehicle's instrument cluster, hidden in the vehicle's display unit, placed under the vehicle's rearview mirror, etc.

[0085] Figure 5A-5B Example models for mapping emotional states along various dimensions are shown, according to various embodiments. Figure 5A As shown, the two-dimensional model 500 maps emotional states along two dimensions, an emotional valence dimension 510 and an emotional arousal dimension 512 .

[0086] The emotional valence dimension 510 measures the degree of pleasure or displeasure a user experiences. For example, anger and sadness represent unpleasant emotions. Therefore, anger and sadness are placed in the negative valence region of the two-dimensional model 500. On the other hand, happiness and / or pleasure represent positive emotions. Therefore, happiness and pleasure are placed in the positive valence region of the two-dimensional model 500.

[0087] The emotional arousal dimension 512 measures how energetic or sleepy the user feels, rather than the intensity of the emotion. In this case, sadness and joy represent low arousal and are therefore placed in the low arousal region of the two-dimensional model 500. Angry and happiness represent high arousal and are therefore placed in the high arousal region of the two-dimensional model 500.

[0088] like Figure 5B As shown, the three-dimensional model 550 maps emotional states along three dimensions: emotional valence dimension 560, emotional arousal dimension 562, and emotional dominance dimension 564, which is also referred to as the emotional stance dimension in this article. The emotional valence dimension 560 measures the degree of pleasure or displeasure felt by the user and is similar to Figure 5A The emotional valence dimension 510. The emotional arousal dimension 562 measures how energetic or sleepy the user feels and is similar to Figure 5A The emotional arousal dimension512.

[0089] The emotional dominance dimension 564 represents the user's emotional state relative to dominance, control, or stance. A closed stance represents a dominant or controlling emotional state. An open stance represents a submissive or controlled emotional state.

[0090] In this case, aversion and anger represent closed positions or dominant feelings, and are therefore placed in the closed position area of the three-dimensional model 550. Acceptance and fear represent open positions or submissive feelings, and are therefore placed in the open position area of the three-dimensional model 550.

[0091] When measuring the overall emotional state of the driver, the driver emotion recognition module 232 can employ the two-dimensional model 500 of Figure 5A or the three-dimensional model 550 of Figure 5B Similarly, when measuring specific components of the driver’s emotional state caused by media content or other factors, the media emotion recognition module 234 can employ the two-dimensional model 500 of Figure 5A or the three-dimensional model 550 of Figure 5B

[0092] Figure 6 is a flowchart of method steps for computing and analyzing a user’s emotional state according to various embodiments. Although the method steps are described in conjunction with Figure 1 the system of FIG. 5, one skilled in the art will understand that any system configured to perform the method steps in any order is within the scope of the present disclosure.

[0093] As shown, the method 600 begins at step 602, where the driver emotion recognition module 232 executing on the emotion analysis system 108 acquires sensor data associated with a user’s emotional state. The sensor data can be raw sensor data, including but not limited to data from cameras, microphones, infrared sensors, vehicle instrument sensors, etc. Additionally or alternatively, the sensor data can include, but is not limited to, biometric data associated with the user, including but not limited to heart rate, temperature, perspiration, head position, facial position, pupil diameter data, eye gaze direction, etc.

[0094] ​At step 604, the driver emotion recognition module 232 determines the overall emotional state of the user based on the sensor data. The driver emotion recognition module 232 may determine the overall emotional state of the user based on a two-dimensional model. In such cases, the driver emotion recognition module 232 may determine a first numerical value for emotional arousal and a second numerical value for emotional valence. Alternatively, the driver emotion recognition module 232 may determine the overall emotional state of the user based on a three-dimensional model. In such cases, the driver emotion recognition module 232 may determine a first numerical value for emotional arousal, a second numerical value for emotional valence, and a third numerical value for emotional dominance. Alternatively, the driver emotion recognition module 232 may determine the overall emotional state of the user based on a higher dimensional model. Alternatively, the driver emotion recognition module 232 may determine the overall emotional state of the user on an emotion-by-emotion basis. In such cases, the driver emotion recognition module 232 may determine the overall emotional state of each of a set of specified emotions.

[0095] At step 606, the media emotion recognition module 234 executed on the emotion analysis system 108 determines the user's emotional state caused by listening to the media content. Generally speaking, the media emotion recognition module 234 uses the same two-dimensional model, three-dimensional model, higher-dimensional model, or emotion-by-emotion analysis as the driver emotion recognition module 232.

[0096] At step 608, the emotion normalizer module 238 executed on the emotion analysis system 108 removes the driver's emotional state due to listening to the media content from the user's overall emotional state. Therefore, the remaining emotional state data represents the driver's emotional state due to factors other than listening to the media content. In this regard, the remaining emotional state data more accurately represents the driver's emotional state due to the driving situation.

[0097] At step 610 , the conversation emotion recognition module 236 executed on the emotion analysis system 108 determines the user's emotional state due to other factors. Such other factors may include, but are not limited to, face-to-face conversations between the driver and passengers, telephone conversations between the driver and other persons, etc. Generally speaking, the conversation emotion recognition module 236 employs the same two-dimensional model, three-dimensional model, higher-dimensional model, or emotion-by-emotion analysis employed by the driver emotion recognition module 232 and / or the media emotion recognition module 234 .

[0098] At step 612, the emotion normalizer module 238 removes the driver's emotional state due to these other factors. Thus, the remaining emotional state data represents the driver's emotional state due to factors other than listening to media content, driver conversation, etc. In this regard, the remaining emotional state data even more accurately represents the driver's emotional state due to the driving situation.

[0099] At step 614, the emotion normalizer module 238 performs one or more responsive actions based on the user's emotional state after removing components related to listening to media content, driver conversation, etc. Thus, after this removal, the user's remaining emotional state is entirely or primarily due to the driving situation, although some additional residual factors may continue to exist in the emotional state. These responsive actions may include, but are not limited to, presenting suggestions and / or warnings to the driver in text form, presenting suggestions and / or warnings to the driver in audio form, etc.

[0100] At step 616, the emotion normalizer module 238 transmits the sensor data and / or emotional state data to the remote server 102 via the telemetry and air system 104. In response, the remote server system 102 aggregates and analyzes the emotional state data associated with multiple users in a given geographic area received from the emotion recognition system 108. Based on the aggregated emotional state data, the remote server system 102 can assess whether the driving conditions in the given geographic area are favorable or unfavorable. The central server system can direct other drivers to drive away from areas assessed to have less favorable driving conditions and toward areas assessed to have more favorable driving conditions. In this way, computer-based human emotional state recognition can improve the experience of individual drivers and a group of drivers as a whole. The method 600 then terminates.

[0101] In summary, the emotion analysis system assesses a user's emotional state induced by various input conditions. More specifically, the emotion analysis system analyzes sensor data to determine the driver's overall emotional state. The emotion analysis system determines the driver's emotional state due to listening to specific media content. The emotion analysis system applies a function to remove the emotional state component caused by listening to the specific media content from the overall emotional state. Optionally, the emotion analysis system determines the driver's emotional state due to additional factors, such as face-to-face conversations between the driver and passengers or telephone conversations between the driver and other persons. The emotion analysis system applies one or more additional functions to remove the emotional state components caused by these additional secondary effects from the overall emotional state. The resulting emotional state more accurately reflects the driver's emotional state induced by the current driving situation. The emotion analysis system performs one or more response actions based on the user's emotional state induced by the driving situation. The response actions may include presenting suggestions and / or alerts to the driver in the form of text, audio, or other forms such as indicators, lights, tactile output, etc. Furthermore, the emotion analysis system may transmit the sensor data and / or emotional state data to a remote server system. The remote server system aggregates and analyzes the sensor data and / or emotional state data received from the plurality of emotion analysis systems to assess the overall emotional state of the plurality of drivers due to driving conditions.

[0102] At least one technical advantage of the disclosed technology over the prior art is that data associated with the driver's emotional state can be processed to more accurately separate the contribution to the driver's emotional state caused by the driving condition from the driver's overall emotional state by removing the contribution to the driver's emotional state caused by media content and / or other factors. As a result, the DMS can generate a more appropriate response action for the driver in response to the contribution of the driving condition to the driver's emotional state. Another technical advantage of the disclosed technology is a central server system that aggregates emotional state data from multiple drivers and can use a more accurate assessment of the contribution of the driving condition to the driver's emotional state to generate a more accurate assessment of the overall favorable or unfavorable driving conditions in a specific area. These technical advantages represent one or more technical improvements over prior art methods.

[0103] 1. In some embodiments, a computer-implemented method for calculating and analyzing a user's emotional state includes: acquiring sensor data associated with the user via at least one sensor; determining an emotional state associated with the user based on the sensor data; determining a first component in the emotional state corresponding to media content being accessed by the user; and applying a first function to the emotional state to remove the first component from the emotional state.

[0104] 2. The computer-implemented method of clause 1 , further comprising: determining a second component of the emotional state based on factors associated with the user; and applying a second function to the emotional state to remove the second component from the emotional state.

[0105] 3. The computer-implemented method of clause 1 or clause 2, wherein the factor is associated with a face-to-face conversation between the user and another person.

[0106] 4. The computer-implemented method of any of clauses 1-3, wherein the factor is associated with a telephone conversation between the user and another person.

[0107] 5. The computer-implemented method of any of clauses 1-4, wherein the media content comprises musical content, and wherein determining the first component of the emotional state comprises analyzing the musical content to determine the first component.

[0108] 6. The computer-implemented method of any of clauses 1-5, wherein the media content comprises music content, and wherein determining the first component of the emotional state comprises retrieving the first component from a database.

[0109] 7. A computer-implemented method according to any one of clauses 1-6, wherein the media content includes sound content, and wherein determining the first component of the emotional state includes: analyzing the sound content to determine the first component based on at least one of a sound tone or a text segment in the sound content.

[0110] 8. A computer-implemented method according to any one of clauses 1-7, wherein the emotional state includes an emotional valence value and an emotional arousal value, and wherein applying the first function to the emotional state includes: applying the first function to the emotional valence value to remove a first component of the emotional valence value corresponding to the media content being accessed by the user; and applying the first function to the emotional arousal value to remove a first component of the emotional arousal value corresponding to the media content being accessed by the user.

[0111] 9. A computer-implemented method according to any one of clauses 1-8, wherein the emotional state also includes an emotion dominant value, and wherein applying the first function to the emotional state includes: applying the first function to the emotion dominant value to remove a first component in the emotion dominant value corresponding to the media content being accessed by the user.

[0112] 10. A computer-implemented method according to any one of clauses 1-9, wherein the emotional state includes a first emotion and a second emotion, and wherein applying the first function to the emotional state includes: applying the first function to the first emotion to remove a first component of the first emotion corresponding to the media content being accessed by the user; and applying the first function to the second emotion to remove a second component of the second emotion corresponding to the media content being accessed by the user.

[0113] 11. In some embodiments, one or more computer-readable storage media include instructions that, when executed by one or more processors, cause the one or more processors to calculate and analyze the user's emotional state by performing the following steps: acquiring sensor data associated with the user via at least one sensor; determining an emotional state associated with the user based on the sensor data; determining a first component in the emotional state that corresponds to the media content being accessed by the user; and applying a first function to the emotional state to remove the first component from the emotional state.

[0114] 12. One or more computer-readable storage media according to claim 11, wherein the instructions further cause the one or more processors to perform the following steps: determining a second component of the emotional state based on factors associated with the user; and applying a second function to the emotional state to remove the second component from the emotional state.

[0115] 13. One or more computer-readable storage media as recited in clause 11 or clause 12, wherein the factor is associated with a face-to-face conversation between the user and another person.

[0116] 14. The one or more computer-readable storage media of any of clauses 11-13, wherein the factor is associated with a telephone conversation between the user and another person.

[0117] 15. One or more computer-readable storage media as recited in any of clauses 11-14, wherein the media content comprises music content, and wherein determining the first component of the emotional state comprises analyzing the music content to determine the first component.

[0118] 16. One or more computer-readable storage media as recited in any one of clauses 11-15, wherein the media content comprises music content, and wherein determining the first component of the emotional state comprises retrieving the first component from a database.

[0119] 17. One or more computer-readable storage media according to any one of clauses 11-16, wherein the media content includes sound content, and wherein determining the first component of the emotional state includes: analyzing the sound content to determine the first component based on at least one of a sound tone or a text segment in the sound content.

[0120] 18. One or more computer-readable storage media according to any one of clauses 11-17, wherein the emotional state includes an emotional valence value and an emotional arousal value, and wherein applying the first function to the emotional state includes: applying the first function to the emotional valence value to remove a first component of the emotional valence value corresponding to the media content being accessed by the user; and applying the first function to the emotional arousal value to remove a first component of the emotional arousal value corresponding to the media content being accessed by the user.

[0121] 19. One or more computer-readable storage media according to any one of clauses 11-18, wherein the emotional state also includes an emotion dominant value, and wherein applying the first function to the emotional state includes: applying the first function to the emotion dominant value to remove a first component in the emotion dominant value corresponding to the media content being accessed by the user.

[0122] 20. In some embodiments, a first endpoint device includes: a memory containing instructions; and a processor, which is coupled to the memory and, when executing the instructions: obtains sensor data associated with a user via at least one sensor; determines an emotional state associated with the user based on the sensor data; determines a first component in the emotional state that corresponds to media content being accessed by the user; and applies a first function to the emotional state to remove the first component from the emotional state.

[0123] Any and all combinations of any claim elements recited in any claim and / or any elements described in this application, in any manner, are within the intended scope of this disclosure and protection.

[0124] The description of the various embodiments has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0125] Aspects of the present embodiment may be embodied as a system, method, or computer program product. Thus, aspects of the present disclosure may take the form of a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.), or an implementation combining software and hardware aspects, which may be collectively referred to herein as a "module" or "system." Additionally, aspects of the present disclosure may take the form of a computer program product implemented in one or more computer-readable media having computer-readable program code implemented thereon.

[0126] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media would include the following: an electrical connection having one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program used by or in conjunction with an instruction execution system, apparatus, or device.

[0127] Aspects of the present disclosure are described above with reference to the flowchart illustrations and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present disclosure. It should be understood that each frame in the flowchart illustration and / or block diagram and the frame combination of the flowchart illustration and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that the instruction executed via the processor of the computer or other programmable data processing device can implement the function / action specified in one or more frames of the flowchart and / or block diagram. Such processors can be, but are not limited to, general-purpose processors, special-purpose processors, application-specific processors or field programmable gate arrays.

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, segment, or portion of a code that includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions mentioned in the box may not appear in the order mentioned in the accompanying drawings. For example, depending on the functions involved, the two boxes shown in succession may be executed substantially simultaneously, or the boxes may sometimes be executed in the opposite order. It should also be noted that each box in the block diagram and / or flowchart illustration and the combination of boxes in the block diagram and / or flowchart illustration can be implemented by a dedicated hardware-based system or a combination of dedicated hardware and computer instructions that performs the specified function or action.

[0129] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the present disclosure may be devised without departing from the basic scope thereof, which is determined by the claims that follow.

Claims

1. A computer-implemented method for calculating and analyzing a user's emotional state, the method comprising: acquiring sensor data associated with the user via at least one sensor; determining an initial emotional state associated with the user based on the sensor data; determining the emotional content of media being listened to by the user; determining a first component of the emotional content attributable to the media in the initial emotional state of the user, wherein the first component is an emotional contribution to the initial emotional state; as well as An updated emotional state associated with the user is determined by applying a first function to the emotional state to remove the first component from the emotional state using an addition technique, a subtraction technique, a conjunction technique, a separation technique, or a Bayesian model technique.

2. The computer-implemented method of claim 1 , further comprising: determining a second component of the initial emotional state based on factors associated with the user; as well as A second function is applied to the initial emotional state to remove the second component from the initial emotional state.

3. The computer-implemented method of claim 2, wherein the factor is associated with a face-to-face conversation between the user and another person.

4. The computer-implemented method of claim 2, wherein the factor is associated with a telephone conversation between the user and another person.

5. The computer-implemented method of claim 1 , wherein the media content comprises music content, and wherein determining the first component of the initial emotional state comprises: The music content is analyzed to determine the first component.

6. The computer-implemented method of claim 1 , wherein the media content comprises music content, and wherein determining the first component of the initial emotional state comprises: The first component is retrieved from a database.

7. The computer-implemented method of claim 1 , wherein the media content comprises audio content, and wherein determining the first component of the initial emotional state comprises: The voice content is analyzed to determine the first component based on at least one of a voice tone or a text segment in the voice content.

8. The computer-implemented method of claim 1 , wherein the initial emotional state comprises an emotional valence value and an emotional arousal value, and wherein applying the first function to the initial emotional state comprises: applying the first function to the emotional potency value to remove a first component of the emotional potency value corresponding to the media content being accessed by the user; as well as The first function is applied to the emotional arousal value to remove a first component of the emotional arousal value corresponding to the media content being accessed by the user.

9. The computer-implemented method of claim 1 , wherein the initial emotional state further comprises an emotion dominance value, and wherein determining an updated emotional state associated with the user by applying the first function to the initial emotional state comprises: The first function is applied to the emotion-dominant value to remove a first component of the emotion-dominant value corresponding to the media content being accessed by the user.

10. The computer-implemented method of claim 1 , wherein the initial emotional state comprises a first emotion and a second emotion, and wherein applying the first function to the initial emotional state to determine an updated emotional state associated with the user comprises: applying the first function to the first emotion to remove a first component of the first emotion corresponding to the media content being accessed by the user; as well as The first function is applied to the second emotion to remove a second component of the second emotion corresponding to the media content being accessed by the user.

11. One or more computer-readable storage media comprising instructions that, when executed by one or more processors, cause the one or more processors to calculate and analyze a user's emotional state by performing the following steps: acquiring sensor data associated with the user via at least one sensor; determining an initial emotional state associated with the user based on the sensor data; determining the emotional content of media being listened to by the user; determining a first component of the emotional content of the initial emotional state of the user that is attributable to the media that the user is listening to, wherein the first component is an emotional contribution to the initial emotional state; as well as An updated emotional state associated with the user is determined by applying a first function to the emotional state to remove the first component from the emotional state using an addition technique, a subtraction technique, a conjunction technique, a separation technique, or a Bayesian model technique.

12. The one or more computer-readable storage media of claim 11, wherein the instructions further cause the one or more processors to perform the following steps: determining a second component of the initial emotional state based on factors associated with the user; and A second function is applied to the initial emotional state to remove the second component from the initial emotional state.

13. The one or more computer-readable storage media of claim 12, wherein the factor is associated with a face-to-face conversation between the user and another person.

14. The one or more computer-readable storage media of claim 12, wherein the factor is associated with a telephone conversation between the user and another person.

15. The one or more computer-readable storage media of claim 11, wherein the media content comprises music content, and wherein determining the first component of the initial emotional state comprises: The music content is analyzed to determine the first component.

16. The one or more computer-readable storage media of claim 11, wherein the media content comprises music content, and wherein determining the first component of the initial emotional state comprises: The first component is retrieved from a database.

17. The one or more computer-readable storage media of claim 11, wherein the media content comprises audio content, and wherein determining the first component of the initial emotional state comprises: The voice content is analyzed to determine the first component based on at least one of a voice tone or a text segment in the voice content.

18. The one or more computer-readable storage media of claim 11, wherein the initial emotional state comprises an emotional valence value and an emotional arousal value, and wherein applying the first function to the initial emotional state to determine an updated emotional state associated with the user comprises: applying the first function to the emotional potency value to remove a first component of the emotional potency value corresponding to the media content being accessed by the user; as well as The first function is applied to the emotional arousal value to remove a first component of the emotional arousal value corresponding to the media content being accessed by the user.

19. The one or more computer-readable media of claim 11, wherein the emotional state further comprises an emotional dominance value, and wherein applying the first function to the initial emotional state to determine an updated emotional state associated with the user comprises: The first function is applied to the emotion-dominant value to remove a first component of the emotion-dominant value corresponding to the media content being accessed by the user.

20. A first endpoint device comprising: a memory comprising instructions; and a processor coupled to the memory and, when executing the instructions: acquiring sensor data associated with the user via at least one sensor; determining an initial emotional state associated with the user based on the sensor data; determining the emotional content of media being listened to by the user; determining a first component of the emotional content in the initial emotional state that is attributable to the media being listened to by the user, wherein the first component is an emotional contribution to the initial emotional state; as well as An updated emotional state associated with the user is determined by applying a first function to the initial emotional state to remove the first component from the initial emotional state using an addition technique, a subtraction technique, a conjunction technique, a separation technique, or a Bayesian model technique.

Images