Contextual scene enhancement

Abstract

Certain aspects of the present disclosure include systems and techniques for generating content indicative of a feeling associated with audio. An example method generally includes monitoring audio to play during display of an associated portion of an interactive content stream, the interactive content stream being provided to at least one viewing device over a communication network during an interactive session; and analyzing the audio via a machine learning component to determine a feeling associated with at least a portion of the audio. The method may also include determining an effect indicative of the feeling, wherein the effect is associated with one or more output devices associated with the at least one viewing device; and outputting an indication of the effect to an associated output device, wherein the effect is configured to be output in real time with the audio while displaying the associated portion of the interactive content stream.

Description

background 1. Technical Field

[0002] The present disclosure generally relates to content generation. For example, aspects of the present disclosure include systems and techniques for generating content that indicates feelings associated with audio.

[0003] 2. Related technical description

[0004] Interactive content (such as video games) has undergone many changes over the years. As computing power increases, developers of interactive content and video games use enhanced computing power to provide more immersive, more complex, and more nuanced user experiences in one or more rendered virtual environments. Excellent visual effects, immersive audio, and integrated haptics can be combined to create immersive experiences, atmospheres, and narratives in games or other interactive content, and to evoke certain emotional responses. For example, audio can be used not only to convey the voice that tells the story, but also to convey the feelings associated with the voice or environment. Therefore, the audio associated with interactive content is an integral part of the experience and is used to immerse the audience and tell a fascinating story. As used herein, feelings may refer to the nuances of unspoken meanings in a narrative or scene, including emotions, subtext, atmosphere, the increase and relief of tension, and other expected effects on the user.

[0005] Some people (e.g., those who are hearing-impaired, neurodivergent, or have speech disorders) may not be able to fully experience certain intended sensations in a virtual reality (VR) environment, particularly when in-game events may occur quickly, other players may be conversing, or other virtual or real-world distractions may prevent full focus and comprehension. Referring to the example above, while an audio recording displayed in subtitle form may convey the speech in a particular scene, other feelings associated with the original scene may not be conveyed by subtitles alone. For example, emotions such as anger or sadness may be expressed in the tone and context of speech, background music and soundtracks, facial expressions and body language of characters, etc., but may not be conveyed through a text recording that is slightly separated in time from real-time events. As another example, background sounds that may depict a particular emotion (such as a gloomy environment) may not be depicted in subtitles alone.

[0006] Therefore, there is a need in the art for improved systems and methods for contextual scene augmentation. Summary of the invention

[0007] Certain aspects of the present disclosure relate to a method for content display. The method generally includes: monitoring audio to be played during display of an associated portion of an interactive content stream, the interactive content stream being provided to at least one viewing device over a communication network during an interactive session; analyzing the audio via a machine learning component to determine a sensation associated with at least a portion of the audio; determining an effect indicative of the sensation, wherein the effect is associated with one or more output devices associated with the at least one viewing device; and outputting an indication of the effect to the associated output device, wherein the effect is configured to be output in real time with the audio while the associated portion of the interactive content stream is displayed.

[0008] Certain aspects of the present disclosure relate to a device for content display. The device generally includes a memory and one or more processors coupled to the memory, the one or more processors being configured to: monitor audio to be played during display of an associated portion of an interactive content stream, the interactive content stream being provided to at least one viewing device over a communication network during an interactive session; analyze the audio via a machine learning component to determine a sensation associated with at least a portion of the audio; determine an effect indicative of the sensation, wherein the effect is associated with one or more output devices associated with at least one viewing device; and output an indication of the effect to the associated output device, wherein the effect is configured to be output in real time with the audio while the associated portion of the interactive content stream is displayed.

[0009] Certain aspects of the present disclosure relate to a non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: monitor audio to be played during display of an associated portion of an interactive content stream that is provided to at least one viewing device over a communications network during an interactive session; analyze the audio via a machine learning component to determine a sensation associated with at least a portion of the audio; determine an effect indicative of the sensation, wherein the effect is associated with one or more output devices associated with the at least one viewing device; and output an indication of the effect to the associated output device, wherein the effect is configured to be output in real time with the audio while the associated portion of the interactive content stream is displayed. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A network environment is shown in which a system for enhancing content display may be implemented.

[0011] Figure 2 An exemplary unified data system (UDS) that may be used to provide data to a system for content display is shown.

[0012] Figure 3An example content generation system including an audio analysis subsystem in accordance with certain aspects of the present disclosure is shown.

[0013] Figure 4 An example content generation system having a crowd-sourced verification component according to certain aspects of the present disclosure is shown.

[0014] Figure 5 An example content generation system of an audio analysis system of sign language having a transcript generated is shown in accordance with certain aspects of the present disclosure.

[0015] Figure 6 is a flow diagram illustrating example operations for content display in accordance with certain aspects of the present disclosure.

[0016] Figure 7 is a block diagram of an exemplary electronic entertainment system that may be used in aspects of the present disclosure.

[0017] Figure 8 is a block diagram of an exemplary computing device that may be used to implement embodiments of the present invention. DETAILED DESCRIPTION

[0018] Certain aspects of the present disclosure relate to a content generation system that analyzes audio to determine a sensation associated with the audio, determines an effect indicative of the sensation, and outputs the effect to a user device to achieve contextual scene enhancement. The effect indicative of the sensation can be a visual effect displayed with the audio, such as a symbol, color, font, or image. In some aspects, the effect can be a haptic effect that can be output to a haptic device, such as a haptic vest or haptics integrated into headphones or a controller, as described in more detail herein.

[0019] In some aspects, the content generation system can be implemented using a machine learning component that is trained to analyze audio and indicate feelings associated with the audio. For example, the machine learning component can be trained to detect anger or sadness in speech and provide an indication of the feeling for generating an effect to be displayed or output to a haptic device.

[0020] Figure 1An exemplary network environment 100 in which a system for providing sensory effects can be implemented is shown. The network environment 100 may include: one or more content source servers 110, which provide digital content (e.g., games, other applications and services) for distribution; one or more content provider server application program interfaces (APIs) 120; a content distribution network server 130; an audio analysis server 140; and one or more user devices 150. The servers described herein may include any type of server as known in the art, including standard hardware computing components, such as network and media interfaces, non-transitory computer-readable storage devices (memory), and a processor for executing instructions or accessing information that may be stored in the memory. The functionality of multiple servers may be integrated into a single server. Any of the aforementioned servers (or integrated servers) may have certain client, cache, or proxy server features. These features may depend on the specific network arrangement of the server or certain configurations of the server.

[0021] The content source server 110 may maintain and provide a variety of digital content available for distribution. The content source server 110 may be associated with any content provider that makes its content available for access through a communication network. Such content may include not only digital videos and games, but also other types of digital applications and services. Such applications and services may include any of a variety of different digital content and functionality that may be provided to the user device 150.

[0022] Content from the content source servers 110 may be provided via a content provider server API 120 that allows various types of content source servers 110 to communicate with other servers (e.g., user devices 150) in the network environment 100. The content provider server API 120 may be specific to the content source server 110 providing the content and the specific language, operating system, protocol, etc. of the user device 150. In a network environment 100 that includes a variety of different types of content source servers 110, there may likewise be a corresponding number of content provider server APIs 120 that allow various formatting, conversion, and other cross-device and cross-platform communication processes for providing content and other services to different user devices 150 that may use different operating systems, protocols, etc. to handle such content. Thus, applications and services in different formats may be made available for compatibility with a variety of different user devices 150.

[0023] The content provider server API 120 may also facilitate each of the user devices 150 to access content hosted or services provided by the content source server 110, either directly or via the content distribution network server 130. Additional information about the accessed content or services, such as metadata, may also be provided to the user devices 150 by the content provider server API 120. As described below, the additional information (i.e., metadata) may be used to provide details about the content or services provided to the user devices 150. In some embodiments, the services provided to the user devices 150 from the content source servers 110 via the content provider server API 120 may include supporting services associated with other content or services, such as chat services, ratings, and profiles associated with a particular game, team, community, etc. In such cases, the content source servers 110 may also communicate with each other via the content provider server API 120.

[0024] The content distribution network server 130 may include a server that provides resources, files, etc. (including various content and service configurations) related to content from the content source server 110 to the user device 150. The content distribution network server 130 may also be called by the user device 150 requesting access to specific content or services. The content distribution network server 130 may include a global management server, a game server, a streaming media server, a server that hosts downloadable content, and other content distribution servers known in the art.

[0025] The audio analysis server 140 may include any data server known in the art that is capable of analyzing audio to determine the expected feeling associated with the audio and generating an output effect indicating the expected feeling. In an exemplary implementation, the audio analysis server 140 monitors the audio-visual content of the incoming stream portion before playing the stream portion (e.g., from a buffer) during an interactive session established with at least one viewing device (e.g., user device 150) via a communication network. The audio analysis server 140 can analyze the audio to determine the expected feeling associated with at least a portion of the audio. One or more machine learning models can be used to analyze the audio (e.g., associated with content title, content type, content developer, user parameters). Such a machine learning model can track patterns of in-game indicators associated with certain feelings. For example, a certain content title may belong to the horror genre, and the audio analysis server 140 can identify the genre based on certain audio-visual cues, which particularly indicate sudden fright, monsters or creatures from time / rhythm, and cues from the player himself (e.g., screaming, panting, biological features indicating accelerated heartbeat). Therefore, the audio analysis server 140 can evaluate new content titles to determine which machine learning models can be applied to situational scene enhancement.

[0026] Once an applicable machine learning model is identified, the audio analysis server 140 can use such model to determine the desired feeling in a particular upcoming scene, determine which type of effect or enhancement is associated with the desired feeling, and generate instructions for the determined effect to an associated output device. Thus, the effect can be output in real time (or near real time) along with the associated audio, while a portion of the interactive content stream is visually displayed.

[0027] The user device 150 may include a variety of different types of computing devices. The user device 150 may be a server that provides internal services (e.g., provided to other servers) in the network environment 100. In such cases, the user device 150 may correspond to one of the content servers 110 described herein. Alternatively, the user device 150 may be a client device that may include any number of different game consoles, mobile devices, laptop computers, and desktop computers. Such user devices 150 may also be configured to access data from other storage media, such as, but not limited to, appropriate memory cards or disk drives in the case of downloaded services. Such user devices 150 may include standard hardware computing components, such as, but not limited to, network and media interfaces, non-temporary computer-readable storage devices (memory), and processors for executing instructions that may be stored in memory. These user devices 150 may also be run using a variety of different operating systems (e.g., iOS, Android), applications, or computing languages ​​(e.g., C++, JavaScript). Each user device 150 may be associated with a group of participants or other types of viewers of a digital content stream.

[0028] Figure 2 An exemplary universal or unified data system (UDS) 200 that can be used to provide data to a system for sensory effect generation is shown. Based on the data provided by the UDS, the audio analysis server 140 can be informed of which in-game objects, entities, activities, and events the user has participated in, and thus support and coordinate sensory analysis and effect generation. Each user interaction can be associated with metadata for the type of in-game interaction, the location within the in-game environment, and the time point within the in-game timeline, as well as other players, objects, entities, etc. involved. Therefore, metadata can be tracked for any of the various user interactions that may occur during a game session, including associated activities, entities, settings, results, actions, effects, locations, and character statistics. Such data can also be aggregated, applied to a data model, and analyzed. This UDS data model can be used to assign context information to each information portion across games in a unified manner.

[0029] like Figure 2, an exemplary console 228 (e.g., user device 150) and exemplary servers 218 (e.g., streaming server 220, activity feed server 224, user generated content (UGC) server 232, and object server 226) are shown. In one example, console 228 may be implemented on a platform server, a cloud server, or on any of server 218. In an exemplary example, content recorder 202 may be implemented on a platform server, a cloud server, or on any of server 218. Such content recorder 202 receives content (e.g., media) from interactive content title 230 and records the content on content ring buffer 208. Such ring buffer 208 may store multiple content segments (e.g., v1, v2, and v3), the start time of each segment (e.g., V1_START_TS, V2_START_TS, V3_START_TS), and the end time of each segment (e.g., V1_END_TS, V2_END_TS, V3_END_TS). Such segments may be stored by the console 228 as a media file 212 (e.g., MP4, WebM, etc.). Such media files 212 may be uploaded to the streaming server 220 for storage and subsequent streaming or use, but the media files 212 may be stored on any server, cloud server, any console 228, or any user device 150. Such start time and end time of each segment may be stored by the console 228 as a content timestamp file 214. Such content timestamp files 214 may also include a streaming ID that matches the streaming ID of the media file 212, thereby associating the content timestamp file 214 to the media file 212. Such content timestamp files 214 may be uploaded and stored to the activity feed server 224 and / or the UGC server 232, but the content timestamp files 214 may be stored on any server, cloud server, any console 228, or any user device 150.

[0030] Concurrently with the content recorder 202 receiving and recording content from the interactive content title 230, the object library 204 receives data from the interactive content title 230, and the object recorder 206 tracks the data to determine when an object starts and ends. The object library 204 and the object recorder 206 may be implemented on a platform server, a cloud server, or on any of the servers 218. When the object recorder 206 detects the start of an object, the object recorder 206 receives object data (e.g., whether the object is an activity, user interaction with the activity, activity ID, activity start time, activity end time, activity results, activity type, etc.) from the object library 204 and records the activity data on the object ring buffer 210 (e.g., ActivityID1, START_TS; ActivityID2, START_TS; ActivityID3, START_TS). Such activity data recorded on the object ring buffer 210 may be stored in the object file 216. Such object files 216 may also include the activity start time, activity end time, activity ID, activity results, activity type (e.g., competition, quest, mission, etc.), user or companion data related to the activity. For example, object files 216 may store data about items used during the activity. Such object files 216 may be stored on object server 226, but object files 216 may be stored on any server, cloud server, any console 228, or any user device 150.

[0031] Such object data (e.g., object files 216) may be associated with content data (e.g., media files 212 and / or content timestamp files 214). In one example, the UGC server 232 stores the content timestamp file 214 and associates the content timestamp file with the object file 216 based on a match between the streaming ID of the content timestamp file 214 and the corresponding activity ID of the object file 216. In another example, the object server 226 may store the object file 216 and may receive a query for the object file 216 from the UGC server 232. Such a query may be performed by searching for the activity ID of the object file 216 that matches the streaming ID of the content timestamp file 214 transmitted with the query. In yet another example, the query may be performed by matching the start time and end time of the content timestamp file 214 with the start time and end time of the corresponding object file 216 transmitted with the query for the stored content timestamp file 214. Such object files 216 may also be associated with matching content timestamp files 214 by the UGC server 232, but the association may also be performed by any server, cloud server, any console 228, or any user device 150. In another example, the object files 216 and content timestamp files 214 may be associated by the console 228 during creation of each file 216, 214.

[0032] In an exemplary embodiment, the media files 212 and activity files 216 may provide information about the current session conditions to the video analysis server 140. Therefore, the audio analysis server 140 may use such media files 212 and activity files 216 to determine whether the current session (in-game) conditions correspond to one or more machine learning models used for situational scene enhancement and for evaluating the expected feeling and selecting the enhancement effect by the selected machine learning model. For example, the media files 212 and activity files 216 may include indicators of user reactions and references to specific characters, locations, and objects displayed and interacted in the current gameplay session. Therefore, based on such files 212 and 216, the audio analysis server 140 may use such indicators and references to determine which machine learning model may be most relevant to evaluating the current scene, and feed the indicators and references into the determined machine learning model, so as to better distinguish the expected feeling of the scene being set. In addition, some content titles may include characters with backgrounds, which can be used as context to distinguish the expected feeling of the current scene. Thus, the associated scene enhancements may include clipped media from the media files 212 and activity files 216 (e.g., in an overlay window on the primary or secondary screen) or link to the media files 212 and activity files 216. In addition, when the analyzed audio-visual content may include certain in-game states of the user or other users, the audio analysis server 140 may use the media files 212 and activity files 216 to evaluate and use such in-game states to determine the intended feel and associated enhancement effects.

[0033] Figure 3 300 includes an audio analysis subsystem 301 according to certain aspects of the present disclosure. As shown, the audio analysis subsystem 301 can receive and monitor audio from an interactive content source 302. The interactive content source 302 can correspond to an audio source related to a video. Figure 1 The interactive content source 302 can communicate and manage interactive content for an experience or game to at least one user device 312. In some aspects, the interactive content source 302 can receive an instruction to pause or replay the interactive content from the user device 312. For example, the content display can provide a link (e.g., on a portion of the display showing the interactive content) that allows the user to pause and replay the interactive content, such as a cutscene in a game.

[0034] The audio analysis subsystem 301 may include a machine learning component 304 to analyze the audio from the interactive content source 302. In some aspects, the machine learning component may be trained to generate a transcript of the speech in the audio. As shown, the transcript may be sent to a user device (e.g., user device 312) via a communication network 316.

[0035] In some aspects, the machine learning component 304 can be trained to detect feelings associated with the audio. For example, audio data labeled with feelings can be used to train the machine learning component 304 using supervised learning. Once trained, the machine learning component 304 can be used for feeling detection. The feeling can include an emotion associated with the audio or the emotion of the speech in the audio. Once the feeling in the audio is detected, an indication of the feeling can be provided to the effect determination component 306. Then, the effect determination component 306 can determine an effect to depict (e.g., indicate) the feeling.

[0036] In some cases, the effect may be a visual effect or a tactile effect. For example, in some aspects, the effect determination component 306 may generate an enhanced transcript with a custom font, size, or color to indicate a feeling. In some aspects, the sound effects in the audio may be converted into a tactile sensation to be provided to the tactile device. For example, the user device 312 may be coupled to the tactile device 308. The tactile device may be part of a headset (e.g., a virtual reality headset), a wearable tactile device (such as a tactile vest), or in a controller. The tactile effect may cause the tactile device 308 to vibrate, indicating a specific feeling associated with the audio. For example, a specific vibration pattern may indicate suspense that would otherwise be expressed in the audio. In some aspects, the tactile device 308 may be built into the display device 310, such as built into a tablet device or a virtual reality (VR) headset.

[0037] In some cases, sound effects in the audio can be converted into symbols (e.g., emoticons) for display on one or more display devices 310, 314 of the user device 312. For example, a symbol showing anger can be displayed along with subtitles showing a speech transcript, indicating that the speech has an angry tone. The audio analysis subsystem 301 can use emotion detection of the conversation audio and emotion-based subtitle tagging to visually indicate the detected emotion (e.g., anger, shouting, whispering). In this way, the audio analysis subsystem 301 can intuitively map emotions, music, or sound effects to visual or tactile effects for users who are deaf or hard of hearing.

[0038] In some cases, the audio analysis subsystem 301 can use sentiment analysis to provide environmental display / effects on certain screen portions or auxiliary devices (e.g., phones, televisions, tablets, other smart devices, or peripheral devices), such as providing a rain effect or a three-dimensional (3D) model of clouds to extend and indicate the atmosphere of the virtual environment. For example, the user device 312 can be coupled to two display devices 310, 314. Although the interactive content can be displayed on the display device 310, the visual effect can be displayed on the display device 314. For example, the display device 314 can display a rain effect to indicate the sound of rain in the audio. In other aspects, the content can be displayed on a first portion of the display, and the effect can be displayed on a second portion of the display. In some aspects, user preferences or device settings can be used to determine which effects are displayed on each display device and / or which portions of the screen are used to display each effect. In some cases, the interactive content can be displayed on more than one display device, such as display devices 310, 314. For example, the display 310 can display the interactive content without obstruction, and the display device 314 can display the interactive content and superimpose the displayed effects on the interactive content. In another example, the interactive content may be displayed and scaled to fit a portion of the screen on display device 314, while the effects are displayed on other portions of the screen.

[0039] Figure 4 A content generation system 300 with a crowdsourcing verification component 402 according to certain aspects of the present disclosure is shown. As shown, interactive content can be provided to multiple user devices, including user device 1 to user device n, where n is any integer greater than 1. Each of the user devices can experience interactive content, which can include subtitles and effects indicating feelings associated with audio, as described herein. In certain aspects of the present disclosure, an interface can be provided to users of the user devices, allowing users to provide feedback on transcripts or effects. For example, one or more of the user devices can be used to identify improvements (e.g., errors) in transcripts or effects, and the improvements are indicated to the crowdsourcing verification component 402 via the communication network 316. The crowdsourcing verification component 402 can determine whether a specific improvement has been identified by a threshold number of user devices. For example, if more than 100 users identify a specific improvement, the crowdsourcing verification component 402 may consider the improvement to be valid. Crowd-sourced verification component 402 may send indications of improvements to audio analysis subsystem 301 , which may re-analyze the associated portion of the audio to generate an updated transcript or effect for presentation to one or more user devices during a subsequent interactive session.

[0040] In some aspects, crowdsourced verification component 402 can coordinate incentives for users to identify improvements to transcripts or effects. For example, for transcripts or effects within a game, an in-game incentive can be provided to a user who identifies a particular improvement (e.g., a bug) that resulted in an updated transcript or effect. Crowdsourced verification component 402 can determine a reward to be provided to one or more user accounts associated with the user device for submitting a badge, and store an indication of the determined reward in association with the user account.

[0041] Figure 5 A content generation system 300 with an audio analysis subsystem 301 for generating a transcript of a sign language according to certain aspects of the present disclosure is shown. As shown, a machine learning component 304 can generate a transcript of speech in the audio. The transcript can be provided to an expression determination component 504. The expression determination component 504 can generate a visual expression of the transcript, such as a sign language, which can be output via a network for display to a user device 312, as shown. In some aspects, the expression determination component can be implemented using machine learning to read the transcript and convert the transcript into an expression (e.g., a sign language).

[0042] In some aspects, the audio analysis subsystem 301 and / or the crowd-sourced verification component 402 may be implemented as part of a processor (e.g., processor 810) and / or as instructions in a storage device (e.g., storage device 830). The audio analysis subsystem 301 and / or the crowd-sourced verification component 402 may be implemented in hardware, software, or a combination of hardware and software. In some aspects, the audio analysis subsystem 301 and / or the crowd-sourced verification component 402 may be implemented by the same hardware, software, or a combination of hardware and software (e.g., by the same processor). In some aspects, the audio analysis subsystem 301 and / or the crowd-sourced verification component 402 may be implemented by separate hardware, software, or a combination of hardware and software.

[0043] Figure 6 is a flow diagram illustrating example operations 600 for content display. Operations 600 may be performed using a content generation system, such as content generation system 300.

[0044] At block 602, a content generation system may monitor (e.g., audio analysis subsystem 301) video to be played during display of an associated portion of an interactive content stream provided to at least one viewing device (e.g., display device 310) over a communication network (e.g., network 316) during an interactive session. The interactive content stream may be associated with a game title. In some cases, the interactive content stream includes a virtual reality environment.

[0045] At block 604, the content generation system (e.g., via the machine learning component 304 of the audio analysis subsystem 301) analyzes the audio to determine a feeling associated with at least a portion of the audio. Determining the feeling may include identifying an emotion (e.g., anger or sadness) associated with speech in the audio.

[0046] At box 606, the content generation system determines (e.g., effect determination component 306) an effect indicating a sensation. For example, the effect can be associated with one or more output devices (e.g., user device 312) associated with at least one viewing device. In some aspects, the effect associated with the audio can be a visual effect. The visual effect can include at least one of a specified font, size, or color of a displayed transcript of the speech in the audio. The visual effect can be a symbol displayed with the audio. The visual effect can be a sign language visual effect displayed with the audio. The visual effect can be an animation, such as rain falling, glass breaking, a dog barking, or an explosion. The effect can be a tactile effect that occurs with the playing audio.

[0047] At block 608, the content generation system outputs an indication of the effect to an associated output device. The effect may be configured to be output in real time with the audio while the associated portion of the interactive content stream is displayed. In some aspects, the content generation system may generate a display of a transcript corresponding to speech within the audio as the audio is received during the interactive session. The output of the effect may be coordinated with the display of the transcript.

[0048] In some aspects, the content generation system can be generated from one or more user devices (e.g., Figure 4 The user devices 1 to n described receive (e.g., crowdsourcing verification component 402) a flag associated with an effect that updates audio or a portion of a displayed transcript of the audio. The content generation system may reanalyze the corresponding portion of the audio (e.g., via the machine learning component 304 of the audio analysis subsystem 301) in response to receiving a threshold number of flags, generate an updated effect or transcript based on the reanalyzed corresponding portion of the audio (e.g., via the effect determination component 306), and output the updated effect or transcript in a subsequent session associated with the interactive content stream. In some cases, the content generation system determines (e.g., via the crowdsourcing verification component 402) a reward for submitting the flag to one or more user accounts associated with the user device, and stores an indication of the determined reward in association with the user account (e.g., storage device 830).

[0049] In some cases, the interactive content stream is displayed on a first display device (e.g., display device 310) associated with the viewing device, and the effect is output on a second display device associated with the viewing device, where the second display device is separate from the first display device.

[0050] Figure 7 is a block diagram of an exemplary electronic entertainment system 700 . Figure 7 The entertainment system 700 includes a main memory 705, a central processing unit (CPU) 710, a vector unit 715, a graphics processing unit 720, an input / output (I / O) processor 725, an I / O processor memory 730, a controller interface 735, a memory card 740, a universal serial bus (USB) interface 745, and an IEEE interface 750. The entertainment system 700 also includes an operating system read-only memory (OS ROM) 755 connected to the I / O processor 725 via a bus 775, a sound processing unit 760, an optical disk control unit 770, and a hard disk drive 765.

[0051] The entertainment system 700 may be an electronic game console. Alternatively, the entertainment system 700 may be implemented as a general purpose computer, a set-top box, a handheld gaming device, a tablet computing device, or a mobile computing device or a phone. The entertainment system may contain more or fewer operating components depending on the particular form factor, purpose, or design.

[0052] Figure 7 The CPU 710, vector unit 715, graphics processing unit 720 and I / O processor 725 communicate via the system bus 785. In addition, Figure 7 The CPU 710 communicates with the main memory 705 via a dedicated bus 780 , while the vector unit 715 and the graphics processing unit 720 may communicate via a dedicated bus 790 . Figure 7 The CPU 710 executes programs stored in the OS ROM 755 and the main memory 705 . Figure 7 The main memory 705 may contain pre-stored programs and programs transferred from a CD-ROM, DVD-ROM or other optical disk (not shown) via the I / O processor 725 using the optical disk control unit 770 . Figure 7 The I / O processor 725 may also allow for the introduction of content transmitted via wireless or other communication networks (e.g., 4$, LTE, 3G, etc.). Figure 7 The I / O processor 725 mainly controls data exchange between various devices of the entertainment system 700 (including the CPU 710, the vector unit 715, the graphics processing unit 720 and the controller interface 735).

[0053] Figure 7 The graphics processing unit 720 executes graphics instructions received from the CPU 710 and the vector unit 715 to generate images for display on a display device (not shown). For example, Figure 7The vector unit 715 can transform the object from three-dimensional coordinates to two-dimensional coordinates and send the two-dimensional coordinates to the graphics processing unit 720. In addition, the sound processing unit 760 executes instructions to generate sound signals that are output to an audio device (not shown) such as a speaker. Other devices can be connected to the entertainment system 700 via the USB interface 745 and the IEEE 1394 interface 750 such as a wireless transceiver, which can also be embedded in the system 700 or as part of some other component such as a processor.

[0054] Figure 7 The user of the entertainment system 700 provides instructions to the CPU 710 via the controller interface 735. For example, the user may instruct the CPU 710 to store certain game information on the memory card 740 or other non-transitory computer-readable storage medium, or instruct a character in the game to perform a certain specified action.

[0055] Figure 8 The architecture of a computing system 800 is shown, wherein the components of the system 800 are in electrical communication with each other using a connection 808, such as a bus. The exemplary system 800 includes a processing unit (CPU or processor) 810 and a system connection 808, which couples various system components including system memory 815, such as a read-only memory (ROM) 820 and a random access memory (RAM) 828, to the processor 810. The system 800 may include a cache of high-speed memory directly connected to the processor 810, close to the processor 810, or integrated as part of the processor 810. The system 800 may copy data from the memory 815 and / or the storage device 830 to the cache 812 for quick access by the processor 810. In this way, the cache can provide a performance boost, thereby avoiding delays in the processor 810 while waiting for data. These and other modules can control or be configured to control the processor 810 to perform various actions. Other system memories 815 are also available. The memory 815 may include a variety of different types of memory with different performance characteristics. The processor 810 may include any general purpose processor and hardware or software services configured to control the processor 810 (such as service 1 832, service 2 834, and service 3 836 stored in the storage device 830), as well as a dedicated processor in which software instructions are incorporated into the actual processor design. The processor 810 may be a completely independent computing system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0056] To enable interaction between the client and the computing system 800, the input device 845 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, etc. The output device 835 can also be one or more of a variety of output mechanisms known to those skilled in the art. In some cases, a multi-mode system can enable the client to provide multiple types of input to communicate with the computing system 800. The communication interface 840 can generally control and manage client input and system output. There is no limitation to operating on any particular hardware arrangement, and therefore the basic features here can be easily replaced by improved hardware or firmware arrangements developed.

[0057] The storage device 830 is a non-volatile memory and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette tape, a random access memory (RAM) 825, a read-only memory (ROM) 820, and hybrids thereof.

[0058] The present invention may be implemented in an application program that can be operated by a variety of end-user devices. For example, an end-user device may be a personal computer, a home entertainment system (e.g., a Sony or Sony or Sony ), portable gaming devices (e.g., Sony or Sony ) or home entertainment systems of different manufacturers. The inventive methods described herein are fully intended to be operable on a variety of devices. The present invention may also be implemented in a cross-title neutral context, where embodiments of the present system may be utilized across a variety of titles from a variety of publishers.

[0059] The present invention may be implemented in applications that may be operated using a variety of devices. Non-transitory computer-readable storage media refers to any one or more media that participate in providing instructions to a central processing unit (CPU) for execution. Such media may take many forms, including but not limited to non-volatile and volatile media, such as optical or magnetic disks and dynamic memory, respectively. Common forms of non-transitory computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROM disks, digital video disks (DVDs), any other optical media, RAM, PROM, EPROM, FLASHEPROM, and any other memory chip or cassette.

[0060] Various forms of transmission media may be involved in transmitting one or more sequences of one or more instructions to the CPU for execution. The bus transmits the data to the system RAM, from which the CPU retrieves and executes the instructions. The instructions received by the system RAM may optionally be stored on a fixed disk before or after execution by the CPU. Various forms of storage devices and the necessary network interfaces and network topologies may likewise be implemented to achieve the same purpose.

[0061] The foregoing detailed description of the technology has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the technology to the precise form disclosed. In view of the above teachings, many modifications and variations are possible. The described embodiments are selected in order to best explain the principles of the technology, its practical application, and to enable others skilled in the art to utilize the technology in various embodiments and make various modifications suitable for the intended specific use. It is intended that the scope of the technology be defined by the claims.

Claims

1. A method for enhancing content display, the method comprising: monitoring audio to be played during display of an associated portion of an interactive content stream provided to at least one viewing device over a communications network during an interactive session; analyzing the audio via a machine learning component to identify a feeling associated with the audio; determining an effect indicative of the identified feeling, wherein the effect is associated with an instruction for one or more output devices associated with the at least one viewing device; as well as The instructions associated with the effect are provided to the viewing device, wherein the instructions are executable by the viewing device to output the effect via the output device in coordination with the audio and in real time while displaying the associated portion of the interactive content stream.

2. The method of claim 1, wherein identifying the feeling comprises identifying an emotion associated with speech in the audio.

3. The method of claim 1, wherein the effect associated with the audio comprises a visual effect.

4. The method of claim 3, wherein the visual effect comprises at least one of a specified font, size, or color of a displayed transcript of speech in the audio.

5. The method of claim 3, wherein the visual effect comprises a symbol displayed with the audio.

6. The method of claim 3, wherein the visual effect comprises a visual effect of sign language displayed along with the audio.

7. The method of claim 1, wherein the effect comprises a haptic effect actuated in coordination with the outputted audio.

8. The method of claim 1, further comprising: receiving, from one or more user devices, one or more indicia regarding an update to the effect associated with the audio or a portion of the displayed transcript; reanalyzing a corresponding portion of the audio in response to receiving a threshold number of the markers; generating instructions for an updated effect or transcript based on reanalyzing the corresponding portion of the audio; as well as The instructions regarding the updated effect or transcript are provided to an output device in a subsequent session associated with the interactive content stream.

9. The method of claim 8, further comprising: determining a reward to be offered to one or more user accounts associated with the user device for submitting the logo; as well as An indication of the determined reward is stored in association with the user account.

10. The method of claim 1, wherein the interactive content stream is displayed on a first display device associated with the viewing device, and wherein the output device comprises a second display device associated with the viewing device and separate from the first display device.

11. The method of claim 1, wherein the interactive content stream is associated with a gaming title.

12. The method of claim 1, wherein the interactive content stream comprises a virtual reality environment.

13. The method of claim 1 further comprising generating a display of a transcript corresponding to speech within the audio when the audio is received during the interactive session, wherein the outputting of the effect is coordinated with the display of the transcript.

14. A device for enhancing content display, the device comprising: a communication interface for communicating over a communication network, wherein the communication interface monitors audio to be played during display of an associated portion of an interactive content stream provided over the communication network to at least one viewing device during an interactive session; as well as one or more processors that execute instructions stored in the memory, wherein the processors execute the instructions to: analyzing the audio via a machine learning component to determine a feeling associated with the audio; determining an effect indicative of the feeling, wherein the effect is associated with an instruction for one or more output devices associated with the at least one viewing device; as well as The instructions associated with the effect are provided to the viewing device, wherein the instructions are executable by the viewing device to output the effect via the output device in coordination with the audio and in real time while displaying the associated portion of the interactive content stream.

15. The apparatus of claim 14, wherein the processor identifies the feeling by identifying an emotion associated with speech in the audio.

16. The apparatus of claim 14, wherein the effect associated with the audio comprises a visual effect.

17. The apparatus of claim 16, wherein the visual effect comprises at least one of a specified font, size, or color of a displayed transcript of speech in the audio.

18. The device of claim 16, wherein the visual effect comprises a symbol displayed with the audio.

19. The device of claim 16, wherein the visual effect comprises a visual effect of sign language displayed with the audio.

20. A non-transitory computer-readable storage medium having instructions stored thereon, the instructions being executable by one or more processors to perform a method for enhancing content display, the method comprising: monitoring audio to be played during display of an associated portion of an interactive content stream provided to at least one viewing device over a communications network during an interactive session; analyzing the audio via a machine learning component to identify a feeling associated with the audio; determining an effect indicative of the identified feeling, wherein the effect is associated with an instruction for one or more output devices associated with the at least one viewing device; as well as The instructions associated with the effect are provided to the viewing device, wherein the instructions are executable by the viewing device to output the effect via the output device in coordination with the audio and in real time while displaying the associated portion of the interactive content stream.

Images