Visual Emotion Tagging and Heat Mapping

Abstract

A system and method for visually tagging communications in a communication session includes: monitoring a current communication session associated with a user device that is in communication with one or more other user devices; the current communication session includes a plurality of messages including a set of messages associated with the user device; identifying and classifying an emotional state for each of the messages in the set based on analyzing the set of messages in light of contextual information relating to a user of the user device; detecting the classified emotional state of at least one message as triggering a tagging rule associated with the user of the user device; and modifying a presentation of the set of messages within the current communication session based on the tagging rule; the modified presentation including one or more tags indicating the classified emotional state of the at least one message.

Description

[Technical Field]

[0001] The present invention relates generally to visually mapping emotions, and more particularly to visual tagging and heat mapping of emotions. [Background technology]

[0002] Currently available digital content titles may evoke a variety of emotional responses in each user who interacts with various digital elements in the associated digital environment of such title. Furthermore, engagement with some digital content titles may involve interactions with other users within the digital (e.g., virtual) environment or within an associated communication session. Such other users may exhibit emotions of their own, and therefore, interactions with other users may themselves evoke or otherwise affect the type and degree of emotional state of the user.

[0003] To some extent, users may be unaware of the types of emotions they feel or display within an interactive (e.g., gameplay) session and associated communication session(s). For example, some users may feel frustrated if their current gameplay session is not going well. In other situations, users may feel stressed when other users yell or otherwise express anger. In these or similar cases, prolonged exposure to the negative emotions of others may deteriorate a user's mental health or sensory processing state or cause other distress. Another example may include a player who may wish to avoid being exposed to profanity, graphic content, bullying language, gender-misleading language, or other offensive, hostile, or violent language (e.g., harassment, threats). Younger players (e.g., children), in particular, may have parents, guardians, or other supervisors (e.g., relatives, childcare professionals) who want to limit the player's exposure to such negative emotions and associated language.

[0004] In certain situations, users may be unaware of the types of emotions others are feeling or displaying. Such a lack of awareness may be exacerbated if users are unable to see or read others' facial expressions or other conventional indicators of emotion. For example, during gameplay sessions, many players use avatars to interact with other players while avoiding revealing their real-world appearance (including facial expressions and gestures). Furthermore, during chat-based communication sessions, tone of voice may not always translate into simple text. Thus, emotional cues may be missed in such contexts.

[0005] Therefore, there is a need in the art for improved systems and methods for visual emotional tagging and heat mapping. Summary of the Invention

[0006] Embodiments of the present invention include systems and methods for emotional visual tagging and heat mapping. A current communication session associated with a user device communicating with one or more other user devices may be monitored. The current communication session may include a plurality of messages, including a set of messages associated with the user device. Based on analyzing the set of messages in light of contextual information regarding a user of the user device, an emotional state may be identified and classified for each of the messages in the set. The classified emotional state of at least one message may be detected as triggering a tagging rule associated with the user of the user device. A presentation of the set of messages may be modified within the current communication session based on the tagging rule. The modified presentation may include one or more tags indicating the classified emotional state of the at least one message. [Brief explanation of the drawings]

[0007] [Figure 1] 1 illustrates a network environment in which a system for emotional visual tagging and heat mapping may be implemented.

[0008] [Figure 2] 1 illustrates an exemplary unified data system (UDS) that can be used to provide data to a system for visual tagging and heat mapping of emotions.

[0009] [Figure 3] 1 is a flowchart illustrating an exemplary method for emotional visual tagging and heat mapping.

[0010] [Figure 4A] 1 shows an example chat stream visually tagged based on emotion.

[0011] [Figure 4B] 1 illustrates an exemplary implementation of emotion-based filtration of emojis within a communication session.

[0012] [Figure 4C] 1 illustrates an exemplary heatmap of emotions associated with players of an interactive content title.

[0013] [Figure 4D] 1 illustrates an exemplary heatmap of emotions associated with multiple players across multiple interactive activities.

[0014] [Figure 5] FIG. 1 is a block diagram of an exemplary electronic entertainment system that may be used with embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] Embodiments of the present invention include systems and methods for emotional visual tagging and heat mapping. A current communication session associated with a user device communicating with one or more other user devices may be monitored. The current communication session may include a plurality of messages, including a set of messages associated with the user device. Based on analyzing the set of messages in light of contextual information regarding a user of the user device, an emotional state may be identified and classified for each of the messages in the set. The classified emotional state of at least one message may be detected as triggering a tagging rule associated with the user of the user device. A presentation of the set of messages may be modified within the current communication session based on the tagging rule. The modified presentation may include one or more tags indicating the classified emotional state of the at least one message.

[0016] FIG. 1 illustrates a network environment in which a system for emotional visual tagging and heat mapping may be implemented. The network environment 100 may include one or more content source servers 110 that provide digital content for distribution (e.g., games, other applications, and services), one or more content provider server application program interfaces (APIs) 120, a content delivery network server 130, a sentiment analysis server 140, and one or more user devices 150. The devices in the network environment 100 communicate with each other using one or more communication networks, which may include dedicated local networks (e.g., intranets) and / or may be part of a larger wide area network. The communication networks may be local area networks (LANs) that may be communicatively coupled to wide area networks (WANs) such as the Internet. The Internet is a broad network of interconnected computers and servers that enables the transmission and exchange of Internet Protocol (IP) data between connected users via network service providers. Examples of network service providers include the public switched telephone network, cable service providers, digital subscriber line (DSL) service providers, or satellite service providers. The one or more communication networks enable communication between the various components of the network environment 100.

[0017] The servers described herein may include any type of server known in the art, including standard hardware computing components such as network and media interfaces, non-transitory computer-readable media (memory), and a processor for executing instructions or accessing information that may be stored in the memory. The functionality of multiple servers may be integrated into a single server. Any of the foregoing servers (or integrated servers) may exhibit characteristics of particular client-side, cache, or proxy servers. These characteristics may depend on the particular network placement of the server or the particular configuration of the server.

[0018] Content source server 110 may maintain and provide a variety of digital content and services that are available for distribution over a communications network. Content source server 110 may be associated with any content provider that makes its content available for access over a communications network. Thus, content source server 110 may host a variety of different content titles that may be further associated with object data (e.g., activity information, zone information, character information, player information, other game media information, etc.) regarding digital or virtual objects that are displayed in a digital or virtual environment during an interactive session.

[0019] Such content may include not only digital video and games, but also other types of digital applications and services. Such applications and services may include any of a variety of different digital content and functionality that may be provided to user device 150, including providing and supporting chat and other communication channels. Chat and communication services may include voice-based, text-based, and video-based messaging. Thus, user device 150 may participate in one or more communication sessions and simultaneously in a gameplay session, and the gameplay and communication sessions may be hosted on one or more content source servers 110.

[0020] Content from content source servers 110 may be provided via content provider server API 120, which enables various types of content source servers 110 to communicate with other servers (e.g., user devices 150) in network environment 100. Content provider server API 120 may be specific to the particular operating language, system, platform, protocol, etc. of the content source server 110 providing the content, as well as the user devices 150 and other devices in network environment 100. In network environment 100 including multiple different types of content source servers 110, there may likewise be a corresponding number of content provider server APIs 120, which enable various formats, conversions, and other cross-device and cross-platform communication processes for providing content and other services to different user devices 150, each of which may process such content using different operating systems, protocols, etc. Accordingly, applications and services may be made available in different formats to be compatible with a variety of different user devices 150. In a network environment 100 that includes multiple different types of content source servers 110, content delivery network servers 130, sentiment analysis servers 140, user devices 150, and databases 160, there may likewise be a corresponding number of APIs managed by the content provider server API 120.

[0021] The content provider server API 120 may further facilitate each of the user devices 150's access to content hosted by or services provided by the content source server 110, either directly or via the content delivery network server 130. Additional information, such as metadata about the accessed content or service, may also be provided to the user device 150 by the content provider server API 120. As described below, the additional information (e.g., object data, metadata) may be available to provide details about the content or service provided to the user device 150. In some embodiments, the services provided by the content source server 110 to the user device 150 via the content provider server API 120 may include supporting services associated with other content or services, such as chat services, ratings, and profiles associated with particular games, teams, communities, etc. In such cases, the content source servers 110 may also communicate with each other via the content provider server API 120.

[0022] The content delivery network servers 130 may include servers that provide resources, files, etc. related to content from the content source servers 110, including various content and service configurations with the user devices 150. The content delivery network servers 130 may also be invoked by user devices 150 requesting access to particular content or services. The content delivery network servers 130 may include universe management servers, game servers, streaming media servers, servers hosting downloadable content, and other content delivery servers known in the art.

[0023] The sentiment analysis server 140 may include any data server known in the art that is capable of communicating with the different content source servers 110, the content provider server API 120, the content delivery network server 130, the user devices 150, and the database 160. Such a sentiment analysis server 140 may be implemented in one or more cloud servers that execute instructions associated with interactive content (e.g., games, activities, videos, podcasts, user-generated content (“UGC”), publisher content, etc.). The sentiment analysis server 140 may further execute instructions to, for example, monitor a current communication session associated with a specified user device communicating with one or more other user devices; classify an emotional state associated with each of the messages associated with the specified user device based on analyzing the set of messages taking into account contextual information about a user of the user device; detect that the classified emotional state of at least one message triggers a tagging rule associated with the user of the user device; and modify the presentation of the set of messages in the current communication session to include one or more tags indicating the classified emotional state of the at least one message based on the tagging rule.

[0024] The user device 150 may include multiple different types of computing devices. The user device 150 may be a server that provides internal services (e.g., to other servers) in the network environment 100. In such a case, the user device 150 may correspond to one of the content servers 110 described herein. Alternatively, the user device 150 may be a computing device that may include any number of different game consoles, mobile devices, laptops, and desktops. Such user devices 150 may also be configured to access data from other storage media, such as, but not limited to, memory cards or disk drives, which may be appropriate for downloaded services. Such user devices 150 may include standard hardware computing components, such as, but not limited to, a network interface, a media interface, non-transitory computer-readable storage (memory), and a processor that executes instructions that may be stored in the memory. These user devices 150 may also run a variety of different operating systems (e.g., iOS, Android), applications, or computing languages ​​(e.g., C++, JavaScript). Exemplary client devices 150 are described in detail herein with respect to Figure 5. Each of the user devices 150 may be associated with a participant (e.g., a player) or other type of user (e.g., a spectator) associated with a collection of digital content streams.

[0025] Although depicted separately, database 160 may be stored on the same server, a different server, or any of the servers and devices depicted in network environment 100, on any of user devices 150. Such database 160 may store or link to various voice-emotion learning models, language-emotion models, facial expression-emotion models, or other emotion models, each of which may be specific to a different emotion, user, or user group. Database 160 may also store other resources used to evaluate the emotional state exhibited during a communication session. Additionally, database 160 may store emotion profiles and tagging rules that may be specific to, and even customizable for, particular emotions, users, user groups or teams, user categories, game titles, game genres, language categories, etc. One or more user emotion profiles and associated emotion learning models may also be stored in database 160 for each user. In addition to gameplay data about the user (e.g., the user's progress in an activity and / or media content title, user ID, user game character, etc.), the user emotion profile may include an emotion database specific to the user.

[0026] Exemplary learning models related to identifying emotions may include various detectable indicators, including textual indicators (e.g., vocabulary or word choice, profanity, keywords, terms, phrases, etc.), spoken language indicators (e.g., changes in vocal intonation, pitch or frequency, speaking rate or tempo, rhythm, volume, etc.), visual indicators (e.g., facial expression, skin tone, such as flushed or pale compared to a calm state), physiological or behavioral indicators (e.g., changes in biometrics such as heart rate, signs of stress, increased input errors, etc.), in-game indicators (e.g., errors in game play, character behavior, etc.), and other indicators related to mood, sentiment, emotion, etc., as known in the art. Learning models can be further refined for different players, who may exhibit different indicators for particular emotions. Additionally, a particular player may be associated with multiple different learning models corresponding to different emotions. Further refinement can be based on user input from the user or from other users that may provide feedback regarding the emotional state of a particular user. Thus, the learned model can be continually updated with different indicators or levels of indicators for a given emotion for a given user.

[0027] Database 160 may further store one or more tagging rules that correlate visual tags (e.g., emojis) with different emotional states and levels. A tagging rule may, for example, specify trigger conditions based on a particular emotion associated with performing a tagging action. The trigger conditions associated with a particular emotion may include one or more threshold levels of emotion indicators that trigger the associated tagging rule. For example, a tagging rule may specify that a high level of anger indicators in a message (e.g., loud volume, yelling, vulgar language, hate speech, racist language, sexist language, homophobic language, or gender-misleading language) may trigger tagging with an anger-related tag. A tagging rule may also specify a combination of various types of indicators described herein, such as indicators related to facial expressions, voice changes, and in-game states and behaviors.

[0028] Thus, when tagging rules are applied to a communication session, different messages that trigger different tagging rules may be tagged in real time or near real time. Because different messages may be associated with different users, each message may be evaluated based not only on emotional indicators (defined and refined by the learning model) but also on contextual information about the particular user that originated the message. Such contextual information may include real-time or historical data about the user, their physical status, their in-game status, etc. Such contextual information may be obtained from a variety of sources, including cameras, sensors, content source server 110, UDS system 200, etc.

[0029] Tags may be updated when the context of the conversation indicates that the originally assigned tag does not indicate the user's emotional state. For example, a yelled or shouted message (whether verbal or typed in all caps) may initially be tagged as angry, but subsequent conversation may reveal that the associated user is simply excited, surprised, hard of hearing, in a noisy real-world environment, or some other situation. Additionally, tags may be updated based on input or feedback from the designated user or other users in the communication session.

[0030] Tags assigned according to tagging rules may further serve as the basis for filtering messages, search results, options, and controlling behavior. For example, a user may want to filter messages associated with undesirable emotions when interacting with other users in a communication session to protect their mental and emotional health and avoid stress associated with such emotions. Such users may also specify or customize the presentation of communication sessions, interactive sessions, and device controls according to their emotional state. Such customization may include lowering game difficulty, reducing levels of aggressive gameplay, increasing thresholds for various emotional triggers, self-censoring vulgar language, disabling or modifying their own voice, triggering calming effects (e.g., music, videos, temperature control devices such as fans, etc.), etc.

[0031] Various types of tags can be used to provide visual displays indicating the emotional state of each participating user within a communication session. Such visual displays may indicate various emotions and levels of such emotions (e.g., low, medium, high). Various emotions may be visualized, for example, in an emotion dashboard that includes various types of measurement or metric presentations. Such visual displays may be triggered based on custom alert or notification settings related to thresholds associated with particular emotion indicators. Furthermore, the emotion analysis server 140 may track data related to emotion indicators for multiple users over time and across multiple sessions. Such data may be used to generate and provide various visual presentations of users' emotion data, in addition to being used to develop and refine emotion-related learning models.

[0032] A user can request to view an emotion dashboard (associated with their own emotion data or the emotion data of others) in which the emotion data can be further subdivided by game title, game session, interactions with different specific users or user groups, time period, or other parameters tracked and made accessible to the emotion analysis server 140 (including data tracked by the UDS system 200). Exemplary presentations of emotion data are shown in Figures 4A-4D, which are described in further detail herein.

[0033] In some implementations, filters can be set not only by the user themselves, but also by the user's parent, guardian, supervisor, etc., who may want the user to avoid exposure to certain types of emotions and their indicators, including vulgar language, graphic content, bullying language, gender-misleading language, or other offensive, hostile, or violent language (e.g., harassment, threatening). Thus, assigned tags can be used as the basis for filtering specific messages or collections of messages from presentation in a communication session. For example, audio-based messages detected as being associated with undesirable emotions and tagged accordingly can be filtered from the version of the communication session being presented to the user by muting, modulating, auto-tuning, or replacing them with a sound generator. In some cases, a user may choose to censor themselves and filter their messages from being presented to other users. For example, a user may specify that a particular audience (e.g., younger players) cannot view, hear, or access certain types of content generated by the user under a particular emotional state identified by an assigned tag.

[0034] During the course of a communication session, each participating user can not only tag their own or other users' messages to indicate emotion, but also tag themselves or other users. Tagging may be based on textual communication, transcripts of spoken words, recorded in-game behavior (e.g., user-generated content captured by UDS system 200), selection from an automatically generated menu of various indicators detected as present in the current communication session, free-form descriptions, or other selection mechanisms known in the art. The tagged emotion indicators can then be added to a learning model to update tagging rules and evaluate incoming communications within the current communication session.

[0035] FIG. 2 illustrates an exemplary unified data system (UDS) that can be used to provide data to a system for emotional visual tagging and heat mapping. Based on the data provided by the UDS 200, the sentiment analysis server 140 recognizes current session conditions, e.g., the in-game objects, entities, activities, and events in which the user and associated character(s) are engaged, which, along with the current gameplay and in-game activity, can support emotion detection and emotion-based response analysis and adjustment by the sentiment analysis server 140. Each user interaction can be associated with metadata, such as the type of in-game interaction, its location within the in-game environment, its point in time within the in-game timeline, and other players, objects, and entities involved. Thus, metadata can be tracked for any of a variety of user interactions that may occur during a game session, including associated activities, entities, settings, outcomes, actions, effects, locations, character statistics, and the like. Such data can be further aggregated, applied to a data model, and subjected to analysis. Such a UDS data model can be used to assign contextual information to pieces of information in a uniform manner across multiple games.

[0036] For example, various content titles may depict one or more objects with which a user can interact (e.g., engage in in-game activities) and / or UGC (e.g., screenshots, videos, play-by-play commentary, mashups, etc.) created by peers, the publisher of the media content title, and / or third-party publishers. Such UGC may include metadata for searching such UGC. Such UGC may also include information about the media and / or peers. Such peer information may be derived from data collected during peer interactions with objects in interactive content titles (e.g., video games, interactive books, etc.) and may be “bound” to and stored with the UGC. Such binding enhances UGC because the UGC may deep link (e.g., launch directly) to the objects, provide information about the objects and / or peers in the UGC, and / or enable users to interact with the UGC.

[0037] 2, an exemplary console 228 (e.g., user device 130) and exemplary servers 218 (e.g., streaming server 220, activity feed server 224, user-generated content (UGC) server 232, and object server 226) are shown. In one example, the console 228 may be implemented on either the platform server 120, a cloud server, or a server 218. In an illustrative example, a content recorder 202 may be implemented on the platform server 120, a cloud server, or any of the servers 218. Such a content recorder 202 receives content (e.g., media) from an interactive content title 230 and records it on a content ring buffer 208. Such a ring buffer 208 may store multiple content segments (e.g., v1, v2, and v3), a start time for each segment (e.g., V1_START_TS, V2_START_TS, V3_START_TS), and an end time for each segment (e.g., V1_END_TS, V2_END_TS, V3_END_TS). Such segments may be stored by the console 228 as media files 212 (e.g., MP4, WebM, etc.). Such media files 212 may be uploaded to a streaming server 220 for storage and subsequent streaming or use, although the media files 212 may be stored on any server, cloud server, any console 228, or any user device 130. Such start and end times for each segment may be stored by the console 228 as a content timestamp file 214. Such a content timestamp file 214 may also include a stream ID that matches the stream ID of the media file 212, thereby associating the content timestamp file 214 with the media file 212.Such content timestamp files 214 may be uploaded and stored on the activity feed server 224 and / or UGC server 232, but the content timestamp files 214 may be stored on any server, cloud server, any console 228, or any user device 130.

[0038] At the same time that the content recorder 202 receives and records content from the interactive content title 230, the object library 204 receives data from the interactive content title 230, and the object recorder 206 tracks the data to determine the start and end times of the object. The object library 204 and the object recorder 206 may be implemented on either the platform server 120, the cloud server, or the server 218. When the object recorder 206 detects the start of an object, it receives object data (e.g., data if the object is an activity, user interaction with the activity, activity ID, activity start time, activity end time, activity result, activity type, etc.) from the object library 204 and records this activity data (e.g., ActivityID1, START_TS; ActivityID2, START_TS; ActivityID3, START_TS) in the object ring buffer 210. Such activity data recorded on the object ring buffer 210 may be stored in the object file 216. Such object files 216 may also include activity start time, activity end time, activity ID, activity result, activity type (e.g., competition, quest, task, etc.), and user or peer data related to the activity. For example, object files 216 may store data regarding items used during the activity. Such object files 216 may be stored on an object server 226, although object files 216 may be stored on any server, cloud server, any console 228, or any user device 130.

[0039] Such object data (e.g., object files 216) can be associated with content data (e.g., media files 212 and / or content timestamp files 214). In one example, the UGC server 232 stores the content timestamp files 214 with the object files 216 and associates the content timestamp files 214 with the object files 216 based on a match between the stream ID of the content timestamp files 214 and the corresponding activity ID of the object files 216. In another example, the object server 226 can store the object files 216 and receive queries for the object files 216 from the UGC server 232. Such queries can be performed by searching for an activity ID of the object files 216 that matches the stream ID of the content timestamp files 214 transmitted with the query. In yet another example, queries of stored content timestamp files 214 can be performed by matching the start and end times of the content timestamp files 214 with the start and end times of the corresponding object files 216 transmitted with the query. Such an object file 216 may also be associated with a matching content timestamp file 214 by the UGC server 232, although this association may be performed by any server, cloud server, any console 228, or any user device 130. In another example, the object file 216 and the content timestamp file 214 may be associated by the console 228 during the creation of each of the files 216, 214.

[0040] In an exemplary embodiment, the media files 212 and activity files 216 may provide information about current session conditions to the sentiment analysis server 140. Accordingly, the sentiment analysis server 140 may use such media files 212 and activity files 216 to determine whether the context of a particular message satisfies any of the trigger conditions of tagging rules associated with the monitored communication session. For example, the media files 212 and activity files 216 may include references to specific characters, locations, and objects displayed and interacting in the current gameplay session. Thus, based on such files 212 and 216, the sentiment analysis server 140 may identify that in-game behavior may be indicative of emotion rather than competitive gameplay. For example, anomalous behavior, such as attacking a teammate while yelling profanities, may be identified as an indicator of anger rather than a sincere effort to compete within the context of the game session. Thus, evaluating whether current in-game data provides a context that satisfies a trigger condition may include evaluating the user's or other users' specific in-game status. In such cases, the sentiment analysis server 140 can use the media file 212 and the activity file 216 to evaluate and determine whether such in-game status satisfies or contributes to a trigger condition.

[0041] 3 is a flowchart illustrating an exemplary method for emotional visual tagging and heat mapping. The method 300 of FIG. 3 may be embodied as executable instructions in a non-transitory computer-readable storage medium, including, but not limited to, non-volatile memory such as a CD, DVD, or hard drive. The instructions in the storage medium may be executed by a processor (or multiple processors) to cause various hardware components of a computing device hosting or otherwise accessing the storage medium to accomplish the method. The steps (and their order) specified in FIG. 3 are exemplary and may include various alternative, equivalent, or derivative forms thereof, including, but not limited to, their execution order.

[0042] In step 310, emotional profiles may be stored in memory (e.g., database 160) for available tags. Different tags may be associated with different emotional profiles, further customized for each user. Each emotional profile may be associated with one or more learning models for identifying emotional states, as well as one or more emotion-based tagging rules, each of which may specify one or more trigger emotional conditions associated with one or more tagging actions. The trigger emotional conditions may relate to any emotional condition and context related to the user, other users, messages in a particular communication session, and related content session(s). Meanwhile, tags may include any type of visual indicator, including any combination of emojis, colors, text, icons, etc.

[0043] At step 320, the current communication session may be monitored by the sentiment analysis server 140 based on the emotional profile. Because each possible emotion may be associated with a particular emotional profile, the sentiment analysis server 140 may retrieve the associated emotional profile from the database 160 and use the associated tagging rule to monitor the communication session in real time (or near real time). Furthermore, if the associated tagging rule may specify a trigger condition that includes other sessions or related sessions (e.g., gameplay sessions), the sentiment analysis server 140 may also retrieve the media files 212 and activity files 216 associated with the user who specified such trigger condition. The media files 216 and activity files 216 may be used by the sentiment analysis server 140 in conjunction with the conditions of the communication session to determine whether the trigger condition is met.

[0044] At step 330, one or more emotional states associated with different messages in the communication session may be identified and classified. Such identification and classification may be based on a set of indicators associated with one or more messages in the communication session that may be detected as triggering a tagging rule. As described above, a set of indicators may trigger a tagging rule if the trigger conditions specified by the tagging rule are met by the indicators. Because different users may exhibit different indicators for a given emotion, a set of indicators that triggers a tagging rule for one user may not trigger the tagging rule for another user. In a multi-user communication session, in which all users may be monitored, various emotion-based learning models and trigger conditions may be used to evaluate and characterize each user's emotional state(s).

[0045] In step 340, messages in a communication session that trigger a condition of a tagging rule can be tagged to visually indicate the user's corresponding emotional state in real time or near real time. Such tagging may include adding a visual tag to a presentation of the communication session. If the communication session includes a text-based chat stream, the tag may be displayed next to the associated chat message. If the communication session includes a video-based chat stream, the tag may be displayed next to a transcript of the message or the user who originated the message.

[0046] In step 350, the sentiment analysis server 140 can generate an analytic map of emotions for a given communication session or across multiple sessions. Such analytic maps can include emotional timelines, emotional journey summaries, heat maps, and other graphical map displays. In some implementations, the analytic map can further include data about other users and their emotional states in the same session or across multiple sessions. Any session parameters (e.g., a particular game title, a particular in-game activity, competitors, peers) can also be used to generate the heat map. As a result, various types of emotional heat maps can be generated that illustrate the type and range of emotions exhibited by a user or group of users across one or more parameters.

[0047] 4A illustrates an exemplary chat stream visually tagged based on emotion. As shown, a collection of emotion-based tags 410 can be added to a display of a chat stream 420. The illustrated chat stream 420 includes messages from two different users, one of whom, associated with a circled C icon, exhibits little change in emotional state. However, the other user, associated with an @ icon, exhibits multiple different emotional states as the chat stream 420 progresses. The emotion-based tags 410 indicate that the first message in the chat stream 420 originating from the @ user was tagged with a pensive or thinking emoji, but was subsequently tagged with annoyed, raised eyebrow, steaming breath, and angry flushed emoji.

[0048] The addition of emotion-based tags 410 may be added automatically in real time or near real time as each message is sent to the chat stream 420. The emotion-based tags may be made visible to the user, other users, or selected groups of users, depending on each user's privacy or preference settings. As a result, the display may be dynamically updated to add emotion-based tags 410 that indicate the originating user's emotional state at the time the message was sent. Thus, throughout the course of a communication session, the emotional states of participating users may be tracked and visually tagged with emotion-based tags. In addition to informing participating users of their own emotional state and the emotional states of their peers, the emotion-based tags may be further tracked and used to perform further actions, including emotion-based filtering and mapping.

[0049] FIG. 4B illustrates an exemplary implementation of emotion-based filtration of emojis within the communication session of FIG. 4A. As shown, the emotion-based tag 410 associated with the last message in the chat stream 420 indicates a flush of anger. The originating user (e.g., the @user) can select the last message to provide feedback regarding the accuracy of the emotion-based tag or to enter additional message input. In such an example, a filtered menu 430 of alternative but related emotion-based tags may be generated based on the last emotion-based tag 410. Such a filtered menu 430 may also be generated for another user to provide feedback regarding the perceived accuracy of the tag assignment. The originating user or another user can also select another tag that they believe is more accurate or reflective of the emotional state associated with the message.

[0050] 4C illustrates an exemplary emotional journey 440 associated with the communication session of FIG. 4A. As illustrated, emotional journey 440 illustrates changes in emotional state associated with user C during the communication session. Because emotional journey 440 may be generated to illustrate the emotional state of user C, emotions and messages associated with other users may not be displayed. However, specific messages related to the emotional changes may be included to illustrate what was happening during the communication session at the same time the emotional changes occurred.

[0051] FIG. 4D shows an exemplary heatmap 450 of emotions associated with multiple players across multiple interactive activities. As shown, heatmap 450 can map the intensity of different emotional states exhibited by peers A-D during gameplay in association with activities A-D (e.g., in different game titles, or different levels, locations, quests, etc. within a given game title). Such heatmap 450 can indicate, for example, comparable levels of excitement or engagement, thereby further providing useful information to developers, tournament organizers, matchmaking systems, and users themselves regarding which types of activities to recommend engagement with. While heatmap 450 focuses on mapping by activity, different heatmaps 450 can be generated for different types of session parameters. For example, heatmap 450 may be generated for emotions exhibited with respect to different peers, game titles, in-game events, in-game objects, and other identifiable conditions of the session.

[0052] Figure 5 is a block diagram of an exemplary electronic entertainment system that may be used with embodiments of the present invention. The entertainment system 500 of Figure 5 includes a main memory 505, a central processing unit (CPU) 510, a vector unit 515, a graphics processing unit 520, an input / output (I / O) processor 525, an I / O processor memory 530, a controller interface 535, a memory card 540, a universal serial bus (USB) interface 545, and an IEEE interface 550. The entertainment system 500 further includes an operating system read-only memory (OS ROM) 555, an audio processing unit 560, an optical disc control unit 570, and a hard disk drive 565, which are connected to the I / O processor 525 via a bus 575.

[0053] Entertainment system 500 may be an electronic game console. Alternatively, entertainment system 500 may be implemented as a general-purpose computer, a set-top box, a handheld gaming device, a tablet computing device, a mobile computing device, or a mobile phone. Entertainment systems may include more or fewer operating components depending on their particular form factor, purpose, or design.

[0054] The CPU 510, vector unit 515, graphics processing unit 520, and I / O processor 525 of FIG. 5 communicate via a system bus 585. Additionally, the CPU 510 of FIG. 5 may communicate with main memory 505 via a dedicated bus 580, and the vector unit 515 and graphics processing unit 520 may communicate via a dedicated bus 590. The CPU 510 of FIG. 5 executes programs stored in the OS ROM 555 and the main memory 505. The main memory 505 of FIG. 5 may contain pre-stored programs and programs transferred via the I / O processor 525 from a CD-ROM, DVD-ROM, or other optical disk (not shown) using the optical disk control unit 570. The I / O processor 525 of FIG. 5 may also enable the introduction of content transferred over wireless or other communication networks (e.g., 4G, LTE, 3G, etc.). The I / O processor 525 of FIG. 5 primarily controls the exchange of data between various devices of the entertainment system 500, including the CPU 510, the vector unit 515, the graphics processing unit 520, and the controller interface 535.

[0055] The graphics processing unit 520 of Figure 5 executes graphics instructions received from the CPU 510 and the vector unit 515 to generate images for display on a display device (not shown). For example, the vector unit 515 of Figure 5 may convert an object from three-dimensional coordinates to two-dimensional coordinates and send the two-dimensional coordinates to the graphics processing unit 520. Additionally, the audio processing unit 560 executes instructions to generate audio signals, which are output to an audio device such as a speaker (not shown). Other devices may be connected to the entertainment system 500 via the USB interface 545 and the IEEE 1394 interface 550, such as a wireless transceiver; these interfaces may also be embedded in the system 500 or as part of some other component, such as a processor.

[0056] 5 provides instructions to CPU 510 via controller interface 535. For example, the user may instruct CPU 510 to store certain game information on memory card 540 or other non-transitory computer-readable storage medium, or may instruct a character in the game to perform some particular action.

[0057] The present invention may be implemented in an application that may be operable by a variety of end-user devices. For example, the end-user device may be a personal computer, a home entertainment system (e.g., Sony PlayStation2® or Sony PlayStation3® or Sony PlayStation4®), a portable gaming device (e.g., Sony PSP® or Sony Vita®), or a home entertainment system from a different, but lesser manufacturer. It is fully contemplated that the methods described herein will operate on a variety of devices. The present invention may also be implemented in a cross-title neutral manner, such that embodiments of the inventive system may be utilized across a variety of titles from a variety of publishers.

[0058] The present invention may be implemented in applications that may be operable using a variety of devices. A non-transitory computer-readable storage medium refers to any medium or media that participates in providing instructions to a central processing unit (CPU) for execution. Such media can take many forms, including, but not limited to, non-volatile media such as optical disks, magnetic disks, and volatile media such as dynamic memory. Common forms of non-transitory computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic media, CD-ROM disks, digital video disks (DVDs), any other optical media, RAM, PROM, EPROM, FLASHEPROM, and any other memory chip or cartridge.

[0059] Various forms of transmission media may be involved in carrying one or more sequences of one or more instructions to a CPU for execution. A bus carries data to system RAM, from which the CPU retrieves and executes the instructions. The instructions received by the system RAM may optionally be stored on a fixed disk either before or after execution by the CPU. Various forms of storage may also be implemented, as well as network interfaces and network topologies necessary to implement the same.

[0060] The foregoing detailed description of the technology has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the technology to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. The described embodiments were selected to best explain the principles of the technology, its practical application, and to enable those skilled in the art to utilize the technology in various embodiments and with various modifications suitable for the particular uses contemplated. It is intended that the scope of the technology be defined by the claims.

Claims

1. 1. A method for visually tagging communications in a communication session, comprising: monitoring a current communication session associated with a user device communicating with one or more other user devices over a communication network, the current communication session being associated with a current virtual environment including characters controlled by the user device, the current communication session indicating behavior of one or more characters within the current virtual environment while a set of messages is being communicated from the user device; classifying an emotional state associated with one or more of the messages in the set based on behavior of the character shown in the current virtual environment while the set of messages is being communicated from the user device; and detecting a categorized emotional state of at least one of the messages in the set triggering a tagging rule customized for the user of the user device; modifying one or more versions of a visual presentation of messages in the current communication session based on the customized tagging rules to include one or more visual tags indicating the classified emotional state of the at least one message, wherein the version of the visual presentation provided to the user device is different from the version of the visual presentation provided to one or more of the other user devices in the current communication session; A method comprising:

2. 10. The method of claim 1, further comprising: storing a user emotion profile in a memory of the user, the user emotion profile including the tagging rule among a plurality of tagging rules.

3. The method of claim 2 , wherein each of the tagging rules specifies an emotional state of the user and one or more corresponding visual tags.

4. The method of claim 1 , wherein the visual presentation is modified in real time to include the visual tag as the at least one message is entered into the user device.

5. detecting activity associated with the set of messages; filtering available visual tags based on the classified emotional states of the set of messages; generating a menu displaying the filtered set of visual tags; The method of claim 1 further comprising:

6. Tracking a plurality of emotional states associated with the user; and generating a display based on the tracked emotional states. The method of claim 1 further comprising:

7. The method of claim 6 , wherein the display is a timeline that presents the tracked emotional states according to the sequence in which the tracked emotional states were detected during one or more communication sessions.

8. tracking a plurality of emotional states associated with a plurality of other users, wherein the display is a heat map visually presenting the tracked emotional states associated with the user and the other users across one or more parameters; The method of claim 6 further comprising:

9. 1. A system for visually tagging communications in a communication session, comprising: a communications interface for communicating over a communications network, the communications interface monitoring a current communications session associated with a user device communicating with one or more other user devices, the current communications session including a visual presentation of a plurality of messages communicated from the user device and the other user devices, the current communications session being associated with a current virtual environment including a character controlled by the user device, and showing behavior of one or more characters within the current virtual environment while a set of messages is being communicated from the user device; A processor that executes instructions stored in a memory, the processor executing the instructions to: classifying an emotional state associated with one or more of the messages in the set based on behavior of the character shown in the current virtual environment while the set of messages is being communicated from the user device; and detecting a categorized emotional state of at least one of the messages in the set triggering a tagging rule customized for the user of the user device; modifying one or more versions of the visual presentation of messages in the current communication session based on the customized tagging rules to include one or more visual tags indicating the classified emotional state of the at least one message, wherein the version of the visual presentation provided to the user device is different from the version of the visual presentation provided to one or more of the other user devices in the current communication session; the processor performing A system comprising:

10. a memory that stores a user emotion profile of the user, the user emotion profile including the tagging rule among a plurality of tagging rules; The system of claim 9 further comprising:

11. The system of claim 10 , wherein each of the tagging rules specifies an emotional state of the user and one or more corresponding visual tags.

12. The system of claim 9 , wherein the visual presentation is modified in real time to include the visual tag as the at least one message is entered into the user device.

13. The processor: detecting activity associated with the set of messages; filtering available visual tags based on the classified emotional states of the set of messages; generating a menu displaying the filtered set of visual tags; The system of claim 9 , further executing instructions to:

14. 10. The system of claim 9, further comprising a memory that tracks a plurality of emotional states associated with the user, and wherein the processor further executes instructions that generate a display based on the tracked emotional states.

15. 15. The system of claim 14, wherein the display is a timeline that presents the tracked emotional states according to the sequence in which the tracked emotional states were detected during one or more communication sessions.

16. 15. The system of claim 14, wherein the memory further tracks a plurality of emotional states associated with a plurality of other users, and wherein the display is a heat map visually presenting the tracked emotional states associated with the user and the other users across one or more parameters.

17. 1. A non-transitory computer-readable storage medium having embodied thereon a program executable by a processor to perform a method for visually tagging communications in a communication session, the method comprising: monitoring a current communication session associated with a user device communicating with one or more other user devices over a communication network, the current communication session being associated with a current virtual environment including characters controlled by the user device, the current communication session indicating behavior of one or more characters within the current virtual environment while a set of messages is being communicated from the user device; classifying an emotional state associated with one or more of the messages in the set based on behavior of the character shown in the current virtual environment while the set of messages is being communicated from the user device; and detecting a categorized emotional state of at least one of the messages in the set triggering a tagging rule customized for the user of the user device; modifying one or more versions of a visual presentation of messages in the current communication session based on the customized tagging rules to include one or more visual tags indicating the classified emotional state of the at least one message, wherein the version of the visual presentation provided to the user device is different from the version of the visual presentation provided to one or more of the other user devices in the current communication session; 1. A non-transitory computer-readable storage medium comprising:

18. The method of claim 1 , wherein classifying the emotional state is further based on an emotion-based learning model specific to the user of the user device.

19. The method of claim 1 , wherein classifying the emotional state is further based on one or more in-game errors in the current virtual environment.

Images