Simulating crowd noise at live events with sentiment analysis of distributed inputs

Abstract

A method and system are provided for generating crowd noise associated with a media event presented using a cloud service. The method includes receiving audio data captured from a viewer of the media event. The method includes processing the audio data to identify speech of the viewer. In one embodiment, features of the speech are classified to build a reaction model to identify reaction states of the viewer. The method includes generating a soundscape for the crowd noise, the soundscape blending audio of general crowd noise associated with the media event with audio corresponding to one or more of the reaction states of the viewer. In one embodiment, the soundscape is output to a speaker associated with the presentation of the media event to the viewer.

Description

[Technical field]

[0001] The present disclosure relates generally to generating crowd noise for viewers watching a media event, and more particularly, to methods and systems for generating crowd noise associated with a media event presented using a cloud service. [Background technology]

[0002] 2. Description of Related Art The video game industry has undergone many changes over the years. In particular, media events such as E-sports have seen significant growth in terms of the number of live events, viewership, and revenue. Recently, however, E-sports events and other media events (e.g., sporting events, concerts, music festivals, etc.) have been negatively affected due to the COVID-19 pandemic. To minimize the spread of COVID-19, many jurisdictions have limited or restricted public gatherings such as E-sports events and other live media events. Nowadays, media events are open with a limited number of in-person participants, and online viewers can watch the media event remotely from their homes in the safety and comfort of their own homes. To this end, developers have been exploring ways to develop advanced manipulations that would improve crowd noise so that the crowd noise of a media event sounds more realistic and authentic to viewers.

[0003] A growing trend in the video game industry is to develop unique ways to enhance the experience of online viewers watching media content from remote locations. Due to capacity limitations and limitations on the number of in-person participants that can attend live shows of media events, generic crowd audio noise is artificially generated and incorporated into the media content to simulate the sounds of a live audience attending the media event. Unfortunately, many remote viewers may find that generic crowd noise sounds unrealistic, lifeless, and boring, and detracts from the cheering sounds from a live audience. As a result, the current process of using artificially simulated generic crowd noise to represent the sound of crowd noise at a media event may sound too unrealistic and may result in viewers losing interest in the media event.

[0004] It is in this context that embodiments of the present disclosure arise. Summary of the Invention

[0005] An embodiment of the present disclosure for this includes methods, systems, and devices related to generating crowd noise associated with a media event performed by a cloud service. In some embodiments, a method is disclosed for enabling the use of a viewer's verbal expressions and their corresponding reactions to generate a soundscape for the crowd noise, where the soundscape blends audio of a general crowd noise associated with the media event with audio corresponding to one or more reaction states of the viewer. For example, a viewer may be remotely watching gameplay in which players are competing in an E-sports event (e.g., a media event) held in an empty stadium where the event is held with no live participants (or a limited number of participants) physically present. Since the number of participants physically present in the stadium is limited, a method disclosed herein outlines a method for generating a soundscape for the crowd noise such that the crowd noise sounds realistic as if a large number of spectators were in attendance and watching the players compete in the event, instead of using only general crowd noise to recreate the sound of a live crowd cheering on their favorite team and players.

[0006] Thus, as the remote viewers react and cheer on their favorite teams and players during the event, the viewers' utterances are captured and processed to build a reaction model. In some embodiments, the reaction model can be used to identify a reaction state of the viewers that can be used to generate a soundscape for crowd noise. In this manner, as the viewers watch the media event, the soundscape is output to the viewers' speakers, thereby enabling the viewers to receive a soundscape that includes an accurate representation of a live crowd reacting to what is happening at the media event.

[0007] In one embodiment, a method is provided for generating crowd noise associated with a media event presented using a cloud service. The method includes receiving audio data captured from a viewer of the media event. The method includes processing the audio data to identify speech of the viewer. In one embodiment, features of the speech are classified to build a reaction model to identify reaction states of the viewer. The method includes generating a soundscape for the crowd noise, the soundscape blending audio of general crowd noise associated with the media event with audio corresponding to one or more of the reaction states of the viewer. In one embodiment, the soundscape is output to a speaker associated with the presentation of the media event to the viewer.

[0008] In another embodiment, a method is provided for generating crowd noise associated with a media event presented to a plurality of audience members using a cloud service. The method includes receiving audio data captured from a plurality of audience members of the media event. The method includes processing the audio data to identify speech of the plurality of audience members. In one embodiment, features of the speech are classified to build a reaction model to identify reaction states of the plurality of audience members. The method includes generating a soundscape for the crowd noise. In one embodiment, the soundscape blends audio of general crowd noise associated with the media event with audio corresponding to one or more of the reaction states of the plurality of audience members.

[0009] Other aspects and advantages of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the disclosure.

[0010] The present disclosure may be better understood by reference to the following description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0011] [Figure 1A]1 illustrates an embodiment of a system configured to generate crowd noise associated with a media event and output the crowd noise to a plurality of viewers watching the media event in accordance with an embodiment of the present disclosure.

[0012] [Figure 1B] According to one embodiment of the present disclosure, an embodiment is illustrated in which a cloud service receives audio data captured from viewers of a media event and processes the audio data to generate a soundscape for crowd noise associated with the media event.

[0013] [Figure 2A] 2A-2C are exemplary diagrams illustrating various audio signal waveforms associated with the vocal output of a viewer 102 of a media event, according to one embodiment of the present disclosure.

[0014] [Figure 2B] 2 is an exemplary diagram illustrating an audio signal waveform corresponding to a viewer's vocal output while watching a media event, according to one embodiment of the present disclosure.

[0015] [Diagram 3] 1 illustrates an embodiment of an audio data machine learning processor that receives a viewer's utterances for processing to build a response model used to identify a viewer response state of a viewer according to one implementation of the present disclosure.

[0016] [Figure 4] 1 illustrates an embodiment of a crowd simulator that receives audience reaction states for processing to generate a soundscape output for crowd noise associated with a media event, according to one implementation of the present disclosure.

[0017] [Diagram 5] 1 illustrates an embodiment of a cloud service configured to process viewer utterances to build a response model for identifying a viewer response state of a viewer according to one embodiment of the present disclosure.

[0018] [Figure 6] 1A-1C are exemplary diagrams illustrating various audio signal waveforms associated with an output soundscape according to one embodiment of the present disclosure.

[0019] [Figure 7] FIG. 2 is an exemplary diagram illustrating a viewer-customized soundscape output based on viewer preferences according to one embodiment of the present disclosure.

[0020] [Figure 8] 1 illustrates a method for generating crowd noise associated with a media event presented using a cloud service, according to one embodiment of the present disclosure.

[0021] [Figure 9] 1 illustrates components of an exemplary device that can be used to implement aspects of various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] The following embodiments of the present disclosure provide methods, systems, and devices for generating customized crowd noise associated with a media event presented using a cloud service. In one embodiment, the media event may be a live or recorded event, such as an e-sports event, a sporting event, a concert, a music festival, a theatrical performance, a comedy show, and the like. For example, while watching a media event including gameplay of players competing in a live e-sports event from a remote location, the viewer may comment, cheer, and verbally react to what is happening in the gameplay. The voice output and speech (e.g., spoken words, utterances, tone of voice, etc.) generated by the viewer may be captured, processed, and used to generate a custom-generated crowd noise soundscape associated with the media event. In one embodiment, generating a crowd noise soundscape for the viewer may enhance the viewer's viewing experience, for example, by providing a more realistic crowd noise that is custom-generated using vocal input from one or more viewers. In some embodiments, the soundscape for crowd noise may provide the viewer with a simulated experience of watching the media event live and in person with other viewers of the media event.

[0023] For example, while watching a sporting event involving an American football game, a viewer watching the football game from home can comment and verbally cheer on their favorite player and team. The viewer's utterances are continuously captured and processed while watching the event to build a reaction model that can be used to identify the viewer's reaction state. In one embodiment, the viewer's reaction state can be fused with a general crowd noise associated with the football game to generate a soundscape for the crowd noise. In some embodiments, the soundscape can be output to a speaker associated with the viewer while watching the football game. In general, the methods described herein provide a method for generating crowd noise associated with a media event such that the crowd noise accurately reflects the sound of a live audience watching the media event in person. Similarly, the viewing experience of a viewer watching a media event remotely can be improved, such that the viewer is more likely to want to continue watching the media event and other content related to the media event.

[0024] As used herein, the term "soundscape" should be understood broadly to refer to sounds or combinations of sounds that form or originate from an immersive environment. For clarity, in the general broad sense, references to "soundscape" should be interpreted to include the blending of typical crowd noise sounds occurring in a venue at a live or recorded media event, such as fans cheering, booing, clapping, gesturing, yelling, etc., with additional simulated noises that correspond to the captured voices, speech, and / or emotions of a viewer or particular group of viewers.

[0025] In one embodiment, the soundscape is generated in a customized manner, whereby the general crowd noise can be combined or fused with additional simulated noises corresponding to the captured speech, emotions, and reactions of one or more viewers. In one embodiment, the added simulated noises are not the live sounds from the captured voices, speech, or reactions, but instead are generated to best represent or correspond to the intensity and / or emotions detected in the captured voices from the viewing. In one embodiment, these additional simulated noises can be accessed from a noise database. The noise database may have hundreds or thousands of sounds associated with a particular type of event, and the system selects a combination of those sounds or files from the database (e.g., generating a composite blend of sounds from the database) to generate the added simulated noises (which are then fused with the general crowd noise). For the viewer, the added simulated noises delivered in the soundscape are influenced by the viewer's captured voices and emotions, but also by the captured voices and emotions of others (e.g., the voices, speech, and emotions of their friends who are watching the event together online). As an example, the soundscape may resemble the actual live sounds a user would experience in a stadium, and the sounds and emotions the user hears may be generated in part by people not just the viewer, but around the viewer or in different parts of the stadium, although additional simulated noises may also be influenced by others watching the event remotely, for example friends or non-friends of the viewer.

[0026] As an example, in one embodiment, a method is disclosed that enables generating crowd noise associated with a presented media event using a cloud service. The method includes receiving audio data captured from a viewer of the media event. In one embodiment, the method may further include processing the audio data to identify the viewer's speech. In one example, the speech features are classified to build a reaction model to identify the viewer's reaction state. In another embodiment, the method may include generating a soundscape for the crowd noise. In one example, the soundscape blends audio of a general crowd noise associated with the media event with audio corresponding to one or more of the viewer's reaction states. The audio blended with the general crowd noise may be accessed from a database and would represent the types of sounds, voices, speech, and emotions detected from the viewer. In another embodiment, the soundscape is output to a speaker associated with the presentation of the media event to the viewer. However, it will be apparent to one of ordinary skill in the art that the present disclosure may be practiced without some or all of the specific details currently described. In other examples, well known process operations have not been described in detail so as not to unnecessarily obscure the present disclosure.

[0027] According to an embodiment, a system is disclosed for generating crowd noise associated with a media event presented to a viewer using a cloud service. For example, multiple viewers may be connected to watch a media event, such as a live E-sports event. In one embodiment, the system includes a connection to a network. In some embodiments, multiple viewers may be connected over the network to watch players compete at the live E-sports event. In some embodiments, multiple viewers may be connected over the network to a cloud service, which is configured to run games and enable connection to multiple viewers when hosting the live E-sports event or other media event. The cloud service may be configured to receive, process, and act on data from multiple devices controlled by the viewers.

[0028] In some embodiments, when multiple viewers are watching the live Esports event, the cloud service is configured to receive and process audio data from the multiple viewers to generate a soundscape for crowd noise associated with the live Esports event. In some embodiments, the soundscape is output to speakers associated with the presentation of the live Esports event to provide the viewers with a simulated crowd noise that would occur at the venue of the live Esports event if the venue were filled with fans. In one embodiment, the cloud service may include an audio data machine learning processor configured to process the viewer audio data and identify speech to build a response model. In some embodiments, the response model can be used to identify a viewer response state that can be used to generate a soundscape for crowd noise associated with the media event.

[0029] With the above summary in mind, the following provides several illustrative figures to facilitate understanding of example embodiments.

[0030] FIG. 1A illustrates an embodiment of a system configured to generate crowd noise associated with a media event and output the crowd noise to a plurality of viewers watching the media event. In one embodiment, FIG. 1 illustrates a plurality of viewers 102a-102n, a network 105, and a cloud service 116. As illustrated in FIG. 1, each viewer 102 is shown watching the media event on a viewer's display screen 108. In one embodiment, the media event can be displayed on the viewer's mobile device or any other device, such as a personal computer, laptop, tablet computer, monitor and console / PC setup, television and console setup, peripherals, tablets, thin clients, set-top boxes, network devices / appliances, etc. In some embodiments, the plurality of viewers 102a-102n may be optionally distributed across different geographic locations 101a-101n. For example, viewers 102a-102b may be watching a media event from Japan, while viewers 102c-102n may be distributed across different regions of the world.

[0031] In some embodiments, the media event presented to the audience may be an e-sports event, a video game, a movie, a sporting event, a concert, a music festival, a theatrical performance, a comedy show, and the like. In one embodiment, the media event is a live event or a recording of an event. In one example, the media event may be viewed live in person, remotely from any geographic location, or as a group with other viewers from any remote geographic location. In some embodiments, the media event is provided by a television network hosting the media event, such as, for example, ESPN™, NBC™, CBS™, ABC™, Fox™, MLB™ Network, NBA TV, NFL Network, and the like. In some embodiments, the media event, as provided by the television network, may include general crowd noise associated with the media event. In some embodiments, the general crowd noise may vary and may depend on the particular type of media event selected by the television network. For example, the general crowd noise may include pre-recorded crowd noise, crowd chatter, or general sounds of a crowd reacting in response to a particular action at the media event. Although generic crowd noise provides a better viewing experience compared to having no crowd noise and only silence, generic crowd noise may be too repetitive and unrealistic, resulting in a viewer not engaging with the media content. For example, using generic crowd noise over game action of a team scoring a game winning point in a championship game may make the event appear less realistic and may not be a true representation of what would be heard if there was a live audience attending the championship game.

[0032] In some embodiments, the cloud service 116 is configured to present the media event to a plurality of viewers 102a-102n. In one example, the cloud service 116 may be a media entertainment service provider, such as the PlayStation Network, that can be used to watch a television broadcast of the media event provided by a television network. In one embodiment, the cloud service 116 is connected to a plurality of viewers 102a-102n via the network 105. In some embodiments, the cloud service 116 is configured to maintain and run media events or video games selected by the viewers 102. In one embodiment, the cloud service 116 is configured to receive input from the viewers 102 watching the media event. For example, in one embodiment, as the viewers 102 watch the media event, the viewers verbally express and react to what is happening in the media event. In one embodiment, the viewers' verbal expressions and reactions (e.g., speech) are captured by a microphone and processed by the cloud service 116. In other embodiments, the cloud service 116 is configured to receive inputs such as video recordings of the viewer's facial expressions, text messages provided by the viewer via a keyboard or device, or phrases and chants that are selectable by the viewer via a menu. For example, a menu may be provided to the viewer's device. The menu may include various encouraging phrases or words that the viewer can select from, such as, for example, defense, go, almost there, etc. In one embodiment, the selected phrases are received as input by the cloud service 116 and can be used to generate the crowd noise soundscape.

[0033] In one embodiment, the cloud service 116 is configured to capture and receive audio data from viewers 102 of a media event. The audio data, including the captured speech of the viewers, can be processed by the cloud service 116 to generate a soundscape for crowd noise associated with the media event. In one embodiment, the generated soundscape may include a blend of audio of general crowd noise associated with the media event and audio corresponding to one or more reaction states of the viewers watching the media event. In some embodiments, the cloud service 116 is configured to output the generated soundscape and transmit it to the viewers watching the media event. In one embodiment, the soundscape is output to speakers associated with the presentation of the media event. In this manner, the viewing experience of the viewers 102 is enhanced because the audio associated with the media event includes audio corresponding to the reaction states of the viewers, rather than just the general crowd noise provided by the television network.

[0034] FIG. 1B illustrates an embodiment in which the cloud service 116 receives audio data captured from a viewer 102 of a media event and processes the audio data to generate a soundscape output 124 for crowd noise associated with the media event. In an embodiment, the viewer 102 may be connected to the cloud service 116 via a network. In some embodiments, the viewer 102 may be watching the media event from any geographic location. In one example, as shown in FIG. 2, a viewer 102a is shown watching a media event on a display screen 108 including players 110a-110n competing in a live soccer match. As the viewer 102 watches the soccer match, the microphones 104a-104n are configured to capture vocal output 106 (e.g., audio data) generated by the viewer 102 or sounds from the environment in which the viewer 102 is located. In some embodiments, the microphones 104 may be integrated with the viewer's device, such as a television, controller, mobile phone, personal computer, laptop, smart speaker, or any other device that may be present in the viewer's environment.

[0035] For example, when a viewer 102 is watching a soccer game, the viewer 102 may comment, cheer, yell, shout, and react to what is happening at the soccer game. Utterances (e.g., spoken words, remarks, tones of voice, etc.) made by the viewer 102 while watching the soccer game may be captured by the microphone 104 and processed by the cloud service 116 to generate a soundscape of crowd noise associated with the soccer game. As further shown in FIG. 1B , the soundscape is output to one or more speakers 112 associated with the presentation of the media event to the viewer 102. In other embodiments, the speakers 112 may be integrated into the device presenting the media event or may be part of a surround sound speaker system configured to deliver the soundscape to the viewer. In another embodiment, the camera 114 may be used to capture the viewer's facial expressions as the viewer watches the media event. In one embodiment, the viewer's facial expressions may be analyzed and processed to determine the viewer's mood and emotions while watching the media event.

[0036] In some embodiments, in addition to displaying the media event on the display screen 108, the cloud service 116 is configured to generate a sound level meter (not shown) for display on the display screen 108. In some embodiments, the sound level meter can be used to motivate the viewer and encourage the viewer to make more noise and be more verbal. For example, the sound level meter can be displayed on the display screen 108 along with the media event. The sound level meter can provide the viewer with an indication of how much noise is being captured from all of the viewers watching the media event. If the system determines that more audio data is needed for processing, the sound level meter can provide instructions to the viewer to encourage the viewer to make more noise and be more verbal, for example, cheer louder, yell, scream, etc.

[0037] 1B , the cloud service 116 is configured to receive viewer utterances (e.g., voice data) as the viewer watches the media event. In one embodiment, the cloud service 116 includes a voice data processor 118, a voice data machine learning processor 120, and a cloud simulator 122 configured to receive, process, and generate a soundscape output 124 for output to the speakers 112 of the viewer 102.

[0038] In one embodiment, the audio data processor 118 is configured to receive the viewer's audio data for processing to identify viewer utterances. As discussed above, viewer utterances may be any combination of spoken words, utterances, and tones of voice expressed by the viewer. In other embodiments, the audio data processor 118 is configured to identify an acoustic intensity associated with each viewer utterance. In some embodiments, each utterance may have a corresponding acoustic intensity level, emotion, mood, or any other speech characteristic associated with the utterance. The acoustic intensity level is associated with a human perception of loudness. For example, a viewer is watching a media event including a championship game soccer match. When the team the viewer is rooting for scores the game winning goal, the viewer verbally expresses "Yeah! We won!", which is processed by the audio data processor 118 to identify an acoustic intensity level associated with the user's verbal expression. In some embodiments, the acoustic intensity level associated with the viewer's utterance may be based on the context of what is happening in the media event and the meaning of the words expressed by the viewer.

[0039] After the audio data processor 118 processes the audio data to identify the speech of the viewer 102, the audio data machine learning processor 120 is configured to process output from the audio data processor 118. In one embodiment, the audio data machine learning processor may include a feature extraction operation configured to identify features associated with the speech and a classifier operation configured to classify the features using one or more classifiers. In some embodiments, the audio data machine learning processor 120 includes a response model, the response model configured to receive the classified features. In one embodiment, the response model can be used to identify a response state of the viewer.

[0040] In some embodiments, the crowd simulator 122 is configured to generate a soundscape for crowd noise associated with the media event. In one embodiment, using a reaction model to identify the reaction states of the audience, the crowd simulator 122 is configured to blend audio of general noise associated with the media event with audio corresponding to one or more reaction states of the audience. After generating the soundscape output 124, the soundscape output 124 can be transmitted to the audience 102 while watching the media event. In one embodiment, the soundscape output 124 is delivered to one or more speakers 112 associated with the presentation of the media event to the audience 102. In this manner, the soundscape includes both general crowd noise and audio corresponding to the reaction states of the audience, which may enhance the viewing experience of the audience.

[0041] In some embodiments, the operation of the audio data processor 118 and the audio data machine learning processor 120 may be local to the viewer's mobile device or any other device, such as a personal computer, laptop, tablet computer, television, etc. In one embodiment, the viewer's audio data is processed locally on the viewer's device to identify the viewer's reaction state, thereby minimizing latency and thereby preventing delays in the viewer receiving the soundscape output 124. In other embodiments, processing the audio data and identifying the viewer's reaction state locally on the viewer's device may help promote data privacy, since the viewer's audio data is processed locally on the device and is not transmitted over a communication channel. Also, in some embodiments, this may reduce costs associated with transmitting audio data over a network, since only the viewer's reaction state is transmitted over the network to the cloud service 116.

[0042] After the voice data machine learning processor 120 operation identifies the viewer's reaction state, the viewer reaction state is received by the cloud service 116 for processing by the cloud simulator 122. For example, a viewer 102 watching an NFL football game yells out "You idiot!" in response to the quarterback letting the ball roll. The vocal output (e.g., You idiot!) is captured by the microphone 104 and processed locally at the viewer's device. The local device may include an embedded voice data processor 118 operation and a voice data machine learning processor 120 configured to identify a reaction state corresponding to the vocal output (e.g., You idiot!). Once the reaction state is identified and a corresponding score is generated for the reaction state, such as, for example, an emotional state, anger, score 7, the reaction state and the corresponding score are received by the cloud service 116 for further processing by the cloud simulator 122.

[0043] 2A is an exemplary diagram illustrating various audio signal waveforms associated with the vocal output (e.g., audio data) of viewers 102 of a media event. As shown in the figure, each vocal output associated with a viewer is represented by audio signal waveforms 204a-204n over a time period, e.g., t1-tn. While viewing the media event, each viewer 102 may verbally comment, cheer, and react to various actions occurring at the media event. In some embodiments, each audio signal waveform may have a different amplitude, frequency, and magnitude.

[0044] For example, audio signal waveform 204b is associated with the vocal output of viewer 102. Audio signal waveform 204b may show a constant waveform (e.g., minimal change over time), which may indicate that viewer 102b is not making any verbal expressions, or that viewer 102b is quietly whispering. In another example, audio signal waveform 204a associated with the vocal output of viewer 102a shows that audio signal waveform 204a varies over time. The audio signal waveform associated with the viewer's vocal output may include multiple utterances with pauses during which the viewer is not making any verbal expressions. For example, in time period t1-t2, viewer 102a may say the phrase "Defense, defense." In time period t3-t4, viewer 102a may say the phrase "Block him." In time period t5-tn, viewer 102a may say the phrase "Yeah, that's good." Conversely, during periods t2-t3 and t4-t5, viewer 102a may be silent and his / her microphone may only capture the viewer's background noise. Thus, each vocal output of viewer 102 is received and examined by cloud service 116 to identify periods of viewer speech and silence for processing to build a response model.

[0045] FIG. 2B is an exemplary diagram illustrating an audio signal waveform 204a corresponding to the vocal output (e.g., voice data) of the viewer 102a while watching a media event. In one embodiment, the vocal output of the viewer 102a is received and processed by the voice data processor 118 of the cloud service 116. In some embodiments, the voice data processor 118 is configured to identify the viewer's speech. For example, as shown in FIG. 2B, utterances 202a-202n are identified by the voice data processor 118 over time periods t0-tn. As discussed above, the speech may be spoken words, utterances, tones of voice, etc., made by the viewer 102 while watching the media event. As shown, utterance 202a occurred during time period t1-t2, utterance 202b occurred during time period t3-t2, and utterance 202n occurred during time period t5-tn. During periods where no speech is identified, such as from t2 to t3 and from t4 to t5, the viewer 102a is silent and does not verbally respond to the media events.

[0046] In some embodiments, each utterance 202a-202n may be divided into different time slices. For example, utterance 202a may be divided into 40 separate time slices. In one embodiment, each of the different separate time slices may have a different reaction state. For example, while watching a media event of an American football game, utterance 202a may be associated with a verbal response "Yes! No!". The verbal response of the viewer 102 may respond to a game action during the football game where a player the viewer 102 is rooting for intercepts the ball but immediately drops it. Thus, the viewer's verbal response and utterances, such as, for example, Yes! and No!, may include a mixture of different emotional reaction states. Thus, utterance 202a may include both verbal responses, such as, for example, Yes! and No!, and utterance 202a may have different reaction states. In one example, the verbal response "Yes!" may correspond to a reaction state that includes an emotional type such as excitement, happiness, surprise, etc. On the other hand, the verbal response "No!" may correspond to reaction states that include emotions such as kinds, anger, sadness, disgust, and fear.

[0047] 3 illustrates an embodiment of the audio data machine learning processor 120 that receives the utterance 202 of the viewer 102 for processing to build a reaction model 306 that is used to identify a viewer reaction state 308 of the viewer 102. As illustrated in FIG. 3, after the utterance 202 of the viewer 102 is identified by the system, the audio data machine learning processor 120 may receive the utterance as an input. In one embodiment, the audio data machine learning processor 120 may include a speech feature extraction 302 operation configured to extract and identify features from the utterance 202. After the features are identified by the speech feature extraction 302 operation, the speech classifier 304 operation is configured to classify the extracted features associated with the viewer's utterance. In some embodiments, the features are labeled using a classification algorithm for further refinement by the reaction model 306.

[0048] In some embodiments, the response model 306 can be configured to receive classified features from the speech classifier 304 operation as input. Using the classified features as input, the response model 306 can be used to identify a response state of the viewer 102 that can be used to generate a soundscape for crowd noise. In some embodiments, the response state of the viewer 102 can include various affective characteristics and emotion types corresponding to the viewer's utterances, such as joy, sadness, fear, anger, surprise, disgust, contempt, panic, etc. For example, a viewer may be watching a media event including a soccer match where the team the viewer is rooting for is behind by one point with 90 seconds remaining in the game. When the viewer verbalizes the phrase "Go team," the response model 306 can be used to identify a response state corresponding to the manifested phrase, including an emotion type of "fear" because the viewer's team is on the verge of losing the game. Thus, in one embodiment, the reaction model 306 may take into account the context of the media event (e.g., which team the viewer is rooting for, the viewer's favorite player, the game action, points scored, etc.) when identifying the reaction state of the viewer 102.

[0049] In some embodiments, the reaction model 306 may be based initially on a global model that may be trained using global features of other viewers similar to the viewer 102. Over time, based on the utterances 202 of the viewer 102, the reaction model 306 is trained to understand the reaction state of the viewer. Thus, over time, the reaction model 306 builds and becomes more specific to the viewer 102. As the reaction model 306 receives more data sets, the reaction model 306 improves and the predicted viewer reaction state 308 becomes more accurate and more valid and applicable to the viewer 102.

[0050] In one embodiment, the response model 306 is configured to generate a score for the utterances 202 of the viewer 102 using a machine learning model. In some embodiments, each utterance 202a-202n may be segmented into different time slices and include an emotional profile with various emotional states. For example, a segment of the viewer utterance 202 may have an emotional profile that includes various emotional states such as happiness, sadness, anger, and disappointment. For a particular segment, the response model 306 may provide a score for each emotional state, which may range from 0 to 10. A value of "10" for an emotional state may indicate that the corresponding emotion has an intensity that is the maximum value. Conversely, a value of "0" for an emotional state may indicate that the corresponding emotion has an intensity that is not significant. For example, a segment of the viewer utterance 202 may correspond to the viewer verbalizing the word "Yeah!" when the viewer's favorite player hits a game-winning home run in a baseball game. Because the viewer's favorite player hit a game-winning home run, the response model 306 may assign an emotional state corresponding to "happy" a value of "10." Conversely, for an emotional state corresponding to "sad," the response model 306 may assign a value of "0" because the viewer does not show any indication of being sad. Thus, each utterance 202 and each segment of utterance may be given a score that can be used to generate a soundscape.

[0051] In some embodiments, the viewer reaction states 308 may include one or more emotional states associated with the viewer's utterances while reacting to a media event. In one embodiment, the one or more emotional states may be scored by the reaction model 306. In some embodiments, the reaction model 306 may provide a score for each emotional state, which may range from 0 to 10. A value of "10" for an emotional state may indicate that the corresponding emotion has an intensity that is at a maximum value. Conversely, a value of "0" for an emotional state may indicate that the corresponding emotion has an intensity that is not significant. For example, while watching a media event, during a certain period of time, the viewer 102 may have reaction states that include emotional states such as anger, excitement, and sadness, with corresponding intensity values ​​of "1," "8," and "2," respectively. In some embodiments, the viewer reaction states 308 may be used to generate a soundscape for crowd noise associated with the media event the viewer is watching. In one embodiment, the score associated with the emotional state may be used to select a corresponding sound from a noise database.

[0052] FIG. 4 illustrates an embodiment of a crowd simulator 122 receiving viewer reaction states 308a-n for processing to generate a soundscape output 124 for crowd noise associated with a media event. As illustrated in FIG. 4, the system includes a crowd simulator 122 configured to receive viewer reaction states 308a-n identified by an audio data machine learning processor 120. In one embodiment, as multiple viewers 102 watch a media event, the vocal output 106 (e.g., audio data) generated by each viewer 102 is captured and processed to identify a viewer reaction state 308a-n corresponding to the viewer's respective vocal output. As discussed above, the viewer reaction state of each viewer may include various affective characteristics and affective states corresponding to the viewer's utterances, such as joy, sadness, fear, anger, disgust, contempt, panic, etc. As each viewer watches a media event, each viewer may have a different reaction state because each viewer may have a different perspective and opinion on the content being viewed.

[0053] 4, in one embodiment, the crowd simulator 122 includes a reaction synthesis 402 operation configured to process and synthesize the viewer reaction states of each viewer. After the reaction synthesis 402 operation synthesizes the various reaction states of each user, a reaction component mixer 404 operation is configured to generate the soundscape output 124. Using the soundscape output 124 generated by the reaction component mixer 404, the system may provide each of the viewers 102a-102n with a soundscape for crowd noises associated with the media event the viewer is watching. In some embodiments, the soundscape may be customized for each viewer based on the viewer's preferences.

[0054] In one embodiment, once the viewer reaction states 308 are identified by the audio data machine learning processor 120, the reaction synthesis 402 operation is configured to receive as input the viewer reaction states 308a-308b associated with each viewer. Because each viewer may verbally manifest various phrases of different reaction states, in one embodiment, the reaction synthesis 402 operation is configured to combine the various viewer reaction states 308a-308b associated with each viewer for further processing by the reaction component mixer 404 operation.

[0055] In one embodiment, the reactive component mixer 404 is configured to generate the soundscape output 124 for the crowd noise. In some embodiments, the reactive component mixer 404 is configured to fuse audio for the general crowd noise 406 with audio corresponding to one or more viewer reaction states 308a-308n of the viewer 102 to generate the soundscape output 124 for the crowd noise. The audio to be fused with the general crowd noise 406 may be accessed from a noise database 408 and may represent the types of sounds, voices, speech, and emotions detected from the viewer. In some embodiments, the audio for the general crowd noise 406 may be a library including pre-recorded artificial crowd noises and sound effects that simulate the sounds of a crowd during a media event, such as a sporting event. For example, the general crowd noise 406 may include various audio files including sounds of crowds clapping, cheering, chanting, cheering, yelling, laughing, moaning, etc. In some embodiments, the general crowd noise 406 may be included with a corresponding media event and generated by a television network hosting the media event. For example, NBCTM may be broadcasting an NBA basketball game. The media event (e.g., an NBA basketball game) may include general crowd noise generated by the NBCTM to simulate the sound of a live crowd at a basketball game.

[0056] In some embodiments, the sounds to be fused with the general crowd noise 406 can be accessed from a noise database 408. The sounds to be fused with the general crowd noise 406 can include simulated noises that correspond to the captured voices, speech, reactions, and / or emotions of a viewer or a particular group of viewers. In one embodiment, the noise database 408 can include pre-recorded sound files that correspond to the viewer response states 308a-308n of a viewer. In other embodiments, the noise database 408 can have hundreds or thousands of sounds associated with a particular type of event, and the system selects a combination of those sounds or sound files from the database to generate the sound corresponding to the viewer response states 308a-308n (which is then fused with the general crowd noise 406).

[0057] Using the output of the reaction synthesis 402 operation, which includes the audience response states 308a-308n, the reaction component mixer 404 is configured to identify sounds from the noise database 408 and correlate the sounds with the corresponding audience response states to build an overall audience response soundscape. For example, in one embodiment, the sounds corresponding to the audience response states 308a-308n of the audience are not the actual speech of the audience 102, but instead, the sounds corresponding to the audience response states 308a-308n of the audience are sounds that resemble, match, mimic, or approximate the actual speech of the audience 102. In other embodiments, the sounds in the noise database 408 may be tagged with a corresponding emotional state. In one embodiment, the emotional score may range from 1 to 10 and may indicate the intensity associated with the sound. In one embodiment, the emotional score of the sounds in the noise database 408 may be used to select the appropriate sounds corresponding to the audience response states.

[0058] For example, while watching a basketball game, a viewer may manifest a vulgar term, such as the phrase "trash," in response to a player's failed attempt to score a field goal. The viewer response state corresponding to the phrase "trash" may indicate that the user is "disappointed," and the corresponding score may have a value of "7" because the player could have taken the lead in the basketball game. Thus, instead of using the phrase "trash" to generate the soundscape output 124, the reaction component mixer 404 may use the noise database 408 to select sounds that correspond to the viewer's "disappointed" viewer response state, such as shit, stupid, nasty, trash, etc. In one embodiment, the viewer response has a score value of "7," so when selecting sounds that correspond to the viewer response state, the reaction component mixer 404 may select sounds from the noise database 408 that have a score value of about "7." In other embodiments, the reaction component mixer 404 is configured to use the viewer's actual speech to blend with sounds of general crowd noise to generate the soundscape output 410 for the crowd noise.

[0059] In some embodiments, the reaction component mixer 404 is configured to use the aggregated audience reaction states 308a-308n from the reaction synthesis 402, the general crowd noise 406 audio, and the audio from the noise database 408 as inputs to statistically distribute and build accurate soundscapes for crowd noise for each particular time segment of the media event. For example, a total of 100,000 viewers may be watching a media event of an NFL football game where 65% of the viewers are fans of the home team and 35% of the viewers are fans of the away team. When the home team scores a touchdown, based on the audio data captured from multiple viewers, the system may determine that 50% of the viewers are reacting with an emotional state of "excited", 15% of the viewers are reacting with an emotional state of "relief", 25% of the viewers are reacting with an emotional state of "disappointment", and 10% of the viewers are reacting with an emotional state of "anger". The various viewer reaction states of the audience 102 can be used by the reaction component mixer 404 to select and blend corresponding sounds from the general crowd noise 406 and corresponding sounds from the noise database 408 to construct a soundscape for the audience reaction. Thus, the generated soundscape output 124 takes into account the distribution of emotional states of the audience watching the media content resulting in a realistic and accurate representation of the crowd noise. In this way, when the soundscape output 124 is provided to the audience 102 of a media event, the soundscape provides the audience with a realistic experience of numerous spectators participating in a stadium reacting to what is happening at the media event.

[0060] In some embodiments, the soundscape output 124 may include three-dimensional (3D) audio effects to make sound sources appear anywhere in the three-dimensional space of the stadium venue where the media event is occurring. For example, the soundscape output 124 provided to the viewer may be customized to make it appear as if the viewer is sitting in a particular portion of the stadium venue, such as a location closer to the field or toward the upper deck of the stadium venue. In another example, the soundscape output 124 provided to the viewer may be customized to make it appear as if the viewer is sitting in a portion of the stadium venue near other fans cheering for the same team. In another embodiment, the soundscape output 124 may include crowd noises of fans cheering for the home team distributed to the viewer's left speaker and crowd noises of fans cheering for the away team distributed to the viewer's right speaker.

[0061] In some embodiments, the crowd simulator 122 is configured to augment the media event with viewer reaction states 308a-308n. In one embodiment, the media event may be augmented with avatars representing viewers 102 of the media event. For example, a viewer watching a sporting event rooting for the home team may have a viewer reaction state including an emotional state of "anger" because the referee made an incorrect call against the home team. The crowd simulator 122 may generate avatars to represent the viewer expressing the "anger" emotion, such as, for example, clenched fists, grinding teeth, and aggressive postures. Conversely, a viewer watching a sporting event rooting for the away team may have an emotional state of "happy" because the referee made a call in favor of the away team. The crowd simulator 122 may generate avatars to represent the viewer expressing the "happy" emotion, such as, for example, laughing, cheering, and high-fiving other fans.

[0062] FIG. 5 illustrates an embodiment of a cloud service 116 configured to process an utterance 202 of a viewer 102 to build a reaction model 306 for identifying a viewer reaction state 308 of the viewer 102. As illustrated in FIG. 5, the utterance feature extraction 302 operation is configured to extract and identify features from the utterance 202 to generate a reaction feature matrix 504. In one embodiment, the viewer utterance 202 may be divided and segmented into different time slices 502a-502n. For example, the utterance 202a may be divided into 40 separate time slices. In one embodiment, each of the various separate time slices 502a-502n may have different reaction states that may occur during a particular period of time. In some embodiments, the reaction feature matrix 504 may include multiple emotion profiles 506a-506n corresponding to the time slices 502a-502n of the utterance 202. In one embodiment, each emotion profile 506 may include various emotion states. For example, as shown in FIG. 5, emotion profile 506a corresponds to time slice 502a that includes emotional states such as happy, sad, angry, disgusted, surprised, excited, etc.

[0063] After features are identified by the speech feature extraction 302 operation and the response feature matrix 504 is generated, the speech classifier 304 operation is configured to classify the extracted features associated with the viewer's speech. In some embodiments, the features are labeled using a classification algorithm for further refinement by the response model 306.

[0064] In some embodiments, the reaction model 306 can be configured to receive as input the classified features from the speech classifier 304 operation. Using this input, the reaction model 306 can be used to identify reaction states 308 of the audience 102 that can be used to generate a soundscape for audience crowd noise. As described above, the audience reaction states 308 can include various emotional characteristics corresponding to the audience's speech, such as happiness, sadness, fear, anger, surprise, disgust, contempt, panic, etc. Over a period of time, the audience reaction states can change and depend on the context of the media event. In one example, as shown in FIG. 5, in the period t2-t3, the emotional characteristics corresponding to the audience's speech can include an emotional state of "excited". In another example, in the period t2-t3, the emotional characteristics corresponding to the audience's speech can include a combination of different emotional states, such as "excited" and "angry". In yet another example, the emotional characteristics corresponding to the audience's speech can include a mixture of different emotions, such as "angry", "happy", and "sad". Thus, the viewer reaction state 308 may have one or more emotional states because aspects of the media event may be continually changing, which may cause the viewer to have different emotional responses.

[0065] In some embodiments, the response model 306 may be configured to receive as an input a profile associated with the viewer 102. The viewer profile may include various attributes associated with the viewer, such as the viewer's favorite team, players, interests, preferences, likes, dislikes, age, gender, etc. In one embodiment, the response model 306 is configured to use the viewer's profile and utterances to identify a viewer response state 308 associated with the viewer's utterances. Other inputs that are not direct inputs or lack of input / feedback may also be interpreted as inputs to the response model 306 to identify the viewer response state 308.

[0066] In other embodiments, the cloud service 116 is configured to process face capture data captured by the viewer's camera 114. In one embodiment, the face capture data can be processed by the cloud service 116 to determine emotions associated with the viewer's facial expressions when verbalizing and reacting to a media event. These emotions may include, without limitation, fear, sadness, happiness, anger, etc. In one embodiment, the face capture data can be processed by a feature processing operation to identify features associated with the viewer's facial expressions. Once the features are identified, a classifier operation is configured to classify the features that can be used as inputs to construct a reaction model 306 for identifying a viewer reaction state 308.

[0067] 6 is an exemplary diagram illustrating various audio signal waveforms associated with an output soundscape 410. As shown, the output soundscape 124 includes audio of generic crowd noise fused with audio corresponding to a viewer response state, such as crowd noise 410 and a noise database 408.

[0068] 7 is an exemplary diagram illustrating a viewer-customized soundscape output based on viewer preferences. As shown, table 702 includes a viewer identification 704 and a customized soundscape output 708 for each viewer of a media event at a particular point in time 706. In one embodiment, the customized soundscape output 708 may be a combination of general crowd noise 406 and audio corresponding to a viewer reaction state 120. In some embodiments, the table may include viewer personalization settings 710 that may enable a viewer to further customize the output soundscape based on the viewer's personal preferences.

[0069] As shown in Figure 7, each viewer 102 can customize how they want their corresponding soundscape output to be distributed. For example, as shown, for viewer 1, at time tn, the customized soundscape output 708 includes 25% general crowd noise, 20% happiness, 10% anger, 10% sadness, 10% stress, and 25% excitement. Viewer 1's viewer personalization 710 also indicates that the viewer is rooting for the home team and that the viewer has selected a sound setting value that corresponds to the feature that corresponds to a value of "1". In one embodiment, the sound setting feature may vary and include multiple different types of customizable features that are selectable by the viewer.

[0070] In some embodiments, the soundscape output can be customized based on a viewer's selection specifying whether they are a fan of the home team or an away team. For example, selecting the home team may adjust the soundscape output such that it emphasizes crowd noises associated with the home team rather than the away team. In one embodiment, this may simulate the sound of a viewer sitting in a part of the venue near other fans of the home team.

[0071] In one embodiment, the voice settings features may include adjusting sound effect characteristics of the voices, such as pitch, rate, timbre, loudness, etc. For example, if a viewer prefers to emphasize the sounds of women and children in the audience, the viewer may make a selection to adjust the pitch of the voices to emphasize the speech of women and children.

[0072] In some embodiments, the soundscape output can be customized to emphasize or de-emphasize crowd sounds reacting to particular players, teams, or game actions at a media event. For example, a viewer can choose to cheer for a particular player participating in a sporting event, while de-emphasizing negative reactions associated with the particular player, and to emphasize crowd sounds indicative of support for the particular player.

[0073] In other embodiments, the soundscape output can be customized to emphasize crowd noises consistent with the same reactions, preferences, and interests of the viewer. In one embodiment, the soundscape output can be customized so that the viewer hears only crowd noises indicative of support for the team and player that the viewer is rooting for and cheering for. For example, if a viewer verbalizes the words "Nice pass!" in response to an action at a media event, the soundscape output can include sounds of a crowd reacting positively and actively to the same game action.

[0074] In other embodiments, the soundscape output can be customized to highlight viewer reactions of friends or other individual viewers of the media event. For example, if a friend of the viewer is watching a media event, the soundscape output can be customized to highlight the friend's reaction while de-emphasizing the reactions of other viewers of the media event. In one embodiment, the soundscape output can be customized to include the actual speech of the friend of the viewer or other particular viewer watching the media event. For example, if a friend of the viewer verbals the phrase "Yay!", this verbal rendition can be incorporated into the soundscape such that the viewer can hear the friend verbalize the phrase "Yay!".

[0075] In other embodiments, the magnitude of the soundscape output can be customized to represent a particular number of participants watching the media event live in person. In one example, if only 1000 participants are watching the media event live in person, the magnitude of the soundscape output can be adjusted to simulate the sound of a crowd of 100,000 participants watching the media event live in person.

[0076] FIG. 8 illustrates a method for generating crowd noise associated with a media event presented using a cloud service 116. In one embodiment, the method includes an operation 802 configured to receive audio data captured from viewers 102 of the media event. For example, multiple viewers 102 may be watching a media event, such as an e-sports event, from remote locations. While watching gameplay of players competing in the e-sports event, the multiple viewers 102 may verbally react to the gameplay, which may include viewers cheering, yelling, yelling, talking, gesturing, laughing, crying, screaming, or making other utterances in response to the game action at the e-sports event. In other embodiments, the operation 802 may capture voice output from multiple viewers 102 simultaneously and may distinguish the vocal output of each viewer. In other embodiments, the operation 802 is configured to capture viewer comments provided by the viewer via selection from a menu or via typing comments and text using the viewer's device. In other embodiments, the operation 802 is configured to receive face capture data captured by a camera while the viewer is watching the media event.

[0077] 8 next flows to operation 804 where operations are configured to process the audio data to identify the viewer's speech. In some embodiments, operation 804 may include a speech feature extraction 302 operation configured to extract and identify features from the user's speech 202. In other embodiments, operation 804 may include a speech classifier 304 operation configured to classify the extracted features associated with the viewer's speech. In some embodiments, operation 804 is configured to use the classified features to construct a reaction model 306 for identifying a reaction state of the viewer 102.

[0078] The method flows to operation 806 configured to generate a soundscape output 124 for crowd noise associated with the media event. In some embodiments, operation 806 is configured to blend audio of general crowd noise associated with the media event with audio corresponding to a viewer's reaction state to generate the soundscape output 124. In some embodiments, after generating the soundscape output 124, operation 806 is configured to transmit the soundscape output 124 to a viewer 102 of the media event. In one embodiment, the soundscape output 124 is output to a speaker associated with the presentation of the media event to the viewer.

[0079] FIG. 9 illustrates components of an exemplary device 900 that can be used to implement aspects of various embodiments of the present disclosure. The block diagram illustrates a device 900 that can incorporate or be a personal computer, video game console, personal digital assistant, server, or other digital device suitable for implementing embodiments of the present disclosure. The device 900 includes a central processing unit (CPU) 902 for executing software applications and optionally an operating system. The CPU 902 may be comprised of one or more homogeneous or heterogeneous processing cores. For example, the CPU 902 is one or more general-purpose microprocessors having one or more processing cores. Further embodiments can implement processing operations such as interpreting queries, identifying contextually relevant resources, and immediately implementing and rendering the contextually relevant resources within a video game using one or more CPUs having a microprocessor architecture that is particularly adapted for highly parallel and computationally intensive applications. The device 900 may be localized to the player playing the game segments (e.g., a game console), or remote from the player (e.g., a back-end server processor), or one of many servers that use virtualization in a game cloud system for remote streaming of game play to clients.

[0080] Memory 904 stores applications and data used by CPU 902. Storage 906 provides non-volatile storage and other computer-readable media for applications and data, and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices, as well as signal transmission and storage media. User input devices 908 communicate user input from one or more users to device 900, examples of which may include a keyboard, mouse, joystick, touchpad, touch screen, still or video recorder / camera, gesture-recognizing tracking device, and / or microphone. Network interface 914 enables device 900 to communicate with other computer systems over an electronic communications network, which may include wired or wireless communications over local area networks and wide area networks such as the Internet. The audio processor 912 is adapted to generate analog or digital audio output from instructions and / or data provided by the CPU 902, memory 904, and / or storage 906. The components of the device 900, including the CPU 902, memory 904, data storage 906, user input device 908, network interface 910, and audio processor 912, are connected via one or more data buses 922.

[0081] A graphics subsystem 920 is further coupled to a data bus 922 and the components of the device 900. The graphics subsystem 920 includes a graphics processing unit (GPU) 916 and a graphics memory 918. The graphics memory 918 includes a display memory (e.g., a frame buffer) used to store pixel data for each pixel of an output image. The graphics memory 918 may be integrated in the same device as the CPU 908, coupled as a separate device to the GPU 916, and / or implemented within the memory 904. The pixel data may be provided to the graphics memory 918 directly from the CPU 902. Alternatively, the CPU 902 provides data and / or instructions to the GPU 916 that define a desired output image, based on which the GPU 916 generates pixel data for one or more output images. The data and / or instructions that define the desired output image may be stored in the memory 904 and / or the graphics memory 918. In one embodiment, GPU 916 includes 3D rendering capabilities that generate pixel data for output images from instructions and data that define scene geometry, lighting, shading, texturing, motion, and / or camera parameters. GPU 916 may further include one or more programmable execution units capable of executing shader programs.

[0082] Graphics subsystem 914 periodically outputs pixel data of an image from graphics memory 918 for display on display device 910. Display device 910 may be any device, including CRT, LCD, plasma, and OLED displays, capable of displaying visual information in response to signals from device 900. Device 900 may provide analog or digital signals to display device 910, for example.

[0083] It should be noted that access services distributed over a wide geography, such as providing access to games in the current embodiment, often use cloud computing. Cloud computing is a computing modality in which dynamically scalable, often virtualized resources are provided as a service over the Internet. Users do not need to be experts in the technical infrastructure of the "cloud" that supports them. Cloud computing can be categorized into different services such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services often provide common applications, such as video games, online, accessed through a web browser, but the software and data are stored on servers in the cloud. The term cloud is used as a metaphor for the Internet based on the way the Internet is depicted in computer network diagrams, and is an abstract concept that hides a complex infrastructure.

[0084] A game server may be used in some embodiments to perform the operation of a persistent information platform for video game players. Most video games played over the Internet operate through a connection to a game server. Typically, the game uses a dedicated server application that collects data from the player and distributes the collected data to other players. In other embodiments, the video game may be executed by a distributed game engine. In these embodiments, the distributed game engine may run on multiple processing entities (PEs), such that each PE executes a functional segment of the given game engine on which the video game is executed. Each processing entity is viewed by the game engine as simply a computational node. A game engine typically performs a functionally diverse set of operations to execute a video game application along with additional services experienced by the user. For example, the game engine implements game logic and performs game calculations, physics, geometry transformations, rendering, lighting, shading, audio, and additional in-game or game-related services. The additional services may include, for example, messaging, social utilities, voice communication, game play playback functions, help functions, and the like. The game engine may run on an operating system virtualized by a hypervisor on a particular server, but in other embodiments the game engine itself may be distributed across multiple processing entities, each of which may reside on a different server unit in a data center.

[0085] According to this embodiment, for execution, each processing entity may be a server unit, a virtual machine, or a container, depending on the needs of each game engine segment. For example, if a game engine segment is responsible for camera transformation, since it will be performing a lot of relatively simple mathematical operations (e.g., matrix transformations), that particular game engine segment may be provisioned with a virtual machine associated with a graphics processing unit (GPU). Other game engine segments, requiring fewer but more complex operations, may be provisioned with processing entities associated with one or more higher-powered central processing units (CPUs).

[0086] By distributing the game engine, the game engine has elastic computational characteristics that are not bound by the capabilities of a physical server unit. Instead, the game engine is provisioned with more or fewer computation nodes as needed to meet the demands of the video game. From the perspective of the video game and the video game player, a game engine that is distributed across multiple computation nodes is indistinguishable from a non-distributed game engine running on a single processing entity, since a game engine manager or supervisor distributes the workload and seamlessly integrates the results to provide the video game output component to the end user.

[0087] Users access remote services through a client device that includes at least a CPU, a display, and I / O. The client device may be a PC, a cell phone, a netbook, a PDA, etc. In one embodiment, a network running on the game server recognizes the type of device used by the client and adjusts the communication method employed. In another case, the client device accesses the application on the game server over the Internet using a standard communication method such as HTML.

[0088] It should be understood that a given video game or game application may be developed for a particular platform and a particular associated controller device. However, when making such games available through a game cloud system as presented herein, users may access the video game with different controller devices. For example, a game may be developed for a game console and its associated controller, but a user may be accessing a cloud-based version of the game from a personal computer utilizing a keyboard and mouse. In such a scenario, the input parameter configuration may define a mapping from inputs that can be generated by the user's available controller device (in this case, a keyboard and mouse) to inputs that are acceptable for execution of the video game.

[0089] In another example, a user may access the cloud gaming system via a tablet computing device, a touchscreen smartphone, or other touchscreen driven device. In this case, the client device and the controller device are integrated together in the same device, and input is provided by detected touchscreen input / gestures. For such devices, the input parameter configuration may define a particular touchscreen input that corresponds to game input for the video game. For example, a button, a directional pad, or other type of input element may be displayed or overlaid during the execution of the video game to indicate locations on the touchscreen that the user can touch to generate game input. Gestures such as swipes in a particular orientation, or particular touch motions may also be detected as game input. In one embodiment, to familiarize the user with control operations on the touchscreen, a tutorial may be provided to the user showing how to input gameplay via the touchscreen, for example, before starting gameplay of the video game.

[0090] In some embodiments, the client device serves as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection and transmits inputs from the controller device to the client device. The client device may then process these inputs and then transmit the input data to a cloud gaming server over a network (e.g., a network accessed via a local network device such as a router). However, in other embodiments, the controller itself may be a networked device that has the ability to communicate inputs directly to the cloud gaming server over a network, without the need for such inputs to be communicated through the client device first. For example, the controller may connect to a local network device (such as the router mentioned above) and transmit data to and receive data from the cloud gaming server. Thus, input latency may be reduced by allowing the controller to transmit inputs directly over the network to the cloud gaming server, bypassing the client device, while the client device may still be required to receive video output from the cloud-based video game and render it on a local display.

[0091] In one embodiment, the networked controller and client devices can be configured to transmit certain types of inputs directly from the controller to the cloud gaming server and other types of inputs via the client device. For example, inputs whose detection does not rely on any additional hardware or processing, apart from the controller itself, can be transmitted directly from the controller to the cloud gaming server over the network, bypassing the client device. Such inputs can include button inputs, joystick inputs, embedded motion detection inputs (e.g., accelerometers, magnetometers, gyroscopes), and the like. However, inputs that utilize additional hardware or require processing by the client device can be transmitted by the client device to the cloud gaming server. These can include video or audio captured from the gaming environment that can be processed by the client device prior to transmission to the cloud gaming server. In addition, inputs from the controller's motion detection hardware can be processed by the client device in conjunction with the captured video to detect the position and movement of the controller, which is then communicated by the client device to the cloud gaming server. It should be understood that the controller device according to various embodiments can also receive data (e.g., feedback data) from the client device or directly from the cloud gaming server.

[0092] It should be understood that the various embodiments defined herein may be combined or assembled into specific implementations that use various features disclosed herein. Thus, the examples provided are only some of the possible examples and are not intended to limit the various implementations that may be defined by combining various elements. In some examples, an implementation may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0093] Embodiments of the present disclosure may be practiced with a variety of computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0094] Although the operations of the method have been described in a particular order, it should be understood that other housekeeping operations may be performed between operations, or operations may be coordinated to occur at slightly different times, or operations may be distributed within the system to allow processing operations to occur at various intervals relative to processing, so long as the processing of telemetry and game state data to generate modified game state is performed in the desired manner.

[0095] One or more embodiments may also be fabricated as computer readable code on a computer readable medium. The computer readable medium may be any data storage device that can store data. The data may then be read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), read only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical and non-optical data storage devices. The computer readable medium may include computer readable tangible media distributed across network connected computer systems such that the computer readable code is stored and executed in a distributed fashion.

[0096] In one embodiment, the video game is executed locally on a game console, personal computer, or server. In some cases, the video game is executed by one or more servers in a data center. When the video game is executed, some instances of the video game may be simulations of the video game. For example, the video game may be executed by an environment or server that generates a simulation of the video game. A simulation, in some embodiments, is an instance of the video game. In other embodiments, the simulation may be generated by an emulator. In either case, when the video game is represented as a simulation, the simulation may be executed to render interactive content that can be interactively streamed, executed, and / or controlled by user input.

[0097] Although the foregoing embodiments have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Thus, the present embodiments should be considered as illustrative and not restrictive, and the present embodiments should not be limited to the details set forth herein, but may be modified within the scope of the appended claims and their equivalents.

Claims

1. 1. A method for generating crowd noise associated with a media event presented using a cloud service, comprising: receiving captured audio data from a viewer of the media event; processing the audio data to identify speech of the viewer, wherein features of the speech are classified to construct a response model for identifying a response state of the viewer; generating a soundscape for the crowd noise, the soundscape blending audio of general crowd noise associated with the media event with audio corresponding to one or more of the reaction states of the audience; Including, the soundscape is output to a speaker associated with the presentation of the media event to the viewer. method.

2. presenting the media event to a plurality of additional viewers; and receiving captured audio data from said plurality of additional viewers; processing the audio data of the additional viewers to identify utterances of the additional viewers, the response models being used to identify response states of the additional viewers; augmenting the generated soundscape for the crowd noise to blend additional sounds corresponding to the reaction states of the additional audience members; The method of claim 1 further comprising:

3. The method of claim 2 , wherein the viewer and each of the additional viewers receive the augmented soundscape as output to a respective speaker associated with the presentation of the media event.

4. The method of claim 1 , wherein the sounds corresponding to one or more of the reaction states of the viewer are customizable based on received preferences of the viewer.

5. Processing additional reaction states of other viewers of the media event; and identifying a sound corresponding to the additional reaction state; and augmenting the generated soundscape to further include blending the audio to correspond to the additional reaction states of the other audience members; The method of claim 1 further comprising:

6. The method of claim 5 , wherein the soundscape is presented to the viewer and / or one of the other viewers as output to speakers when viewing the media event.

7. The method of claim 1 , wherein the media event is a live event or an event that is viewed by the viewer and other viewers as a group.

8. The method of claim 1 , wherein the reaction state of the viewer includes one or more emotion types associated with the viewer's utterances.

9. The method of claim 8 , wherein each of the emotion types is scored by the response model, the score corresponding to an intensity associated with the corresponding utterance of the viewer.

10. The method of claim 9 , wherein the scores are used to select the sounds that correspond to one or more of the reaction states of the audience.

11. The method of claim 1 , wherein the audio corresponding to one or more of the reaction states of the viewer is not the speech of the viewer.

12. The method of claim 1 , wherein the sounds corresponding to one or more of the reaction states of the viewer are sounds that approximate the speech of the viewer.

13. 2. The method of claim 1, wherein the sounds corresponding to one or more of the reaction states are accessed from a database of pre-recorded sound files, the pre-recorded sound files being tagged with an emotion score and used to select the sounds corresponding to one or more of the reaction states of the viewer.

14. 2. The method of claim 1, wherein the reaction model implements a machine learning engine configured to identify features of the utterance to classify attributes of the viewer, the attributes of the viewer being used to identify the reaction state of the viewer.

15. 1. A method for generating crowd noise associated with a media event presented to a plurality of audiences using a cloud service, comprising: receiving captured audio data from the plurality of viewers of the media event; processing the audio data to identify speech of the plurality of viewers, wherein features of the speech are classified to construct a response model for identifying response states of the plurality of viewers; generating a soundscape for the crowd noise, the soundscape blending audio of general crowd noise associated with the media event with audio corresponding to one or more of the reaction states of the plurality of audience members; A method comprising:

16. The method of claim 15 , wherein the soundscape is output to speakers associated with the presentation of the media event to the multiple audience members.

17. The method of claim 15 , wherein the soundscape is customizable based on received preferences of the multiple viewers.

18. The method of claim 15 , wherein the media event is a live or recorded event being viewed by the multiple viewers as a group or separately in different geographic locations.

19. The method of claim 15 , wherein the reaction states of the plurality of viewers include one or more emotion types associated with the utterances of the plurality of viewers.

20. 20. The method of claim 19, wherein each of the emotion types is scored by the response model, the score corresponding to an intensity associated with the corresponding utterance for the plurality of viewers.

Images