Expanding game content for previously developed game applications

Abstract

The present disclosure relates to systems, methods, and computer-readable media for identifying various events occurring within a game session and implementing one or more in-game enhancements based on the identified events. For example, a game service (e.g., a cloud game server) can leverage content analysis and event recognition services on a cloud computing system to detect one or more in-game events based on game content (e.g., video content, audio content, controller input) delivered to a client system. The systems described herein can cause the client system to implement in-game enhancements in response to the detected events. The detection of events and implementation of various enhancements can be performed without significant redevelopment of the game application, even if the game application was previously shipped.

Description

[Background technology]

[0001] In recent years, the use of remote computing resources to generate, process, and deliver multimedia content has grown exponentially. For example, cloud computing systems now generate and deliver gaming content to a variety of local devices with various computing and display capabilities. As video games and other content become higher quality and network bandwidth increases, cloud-based gaming has become an increasingly popular platform for delivering gaming content to individuals.

[0002] For example, while much popular gaming content was previously available through a select few game consoles that were specifically developed to generate and deliver gaming content from a particular set of game titles, cloud-based gaming paves the way for gaming content from multiple publishers to be delivered through the same client device. Additionally, cloud gaming often allows different versions of a game title that were previously only playable on different systems to be accessible through a single client system. While cloud-based gaming has grown in popularity due to increased accessibility and availability to a larger collection of game titles, there are many challenges associated with maintaining or increasing engagement with both new and existing titles in gaming applications.

[0003] For example, while cloud-based gaming provides immediate access to many previously shipped game titles, updating these games from previous versions is often a difficult and costly process. For example, developers of previously shipped games have often moved on to developing new games and are no longer compiling updates to previously shipped game applications. Often, older games have been archived or lost and the source is lost, and implementing an update would require redeveloping significant portions of the application software. Furthermore, even when game development is possible and developers are available to engage in it, developing patches and other fixes to game applications and deploying those developments to the user base often requires significant time and cost for the publishers and providers of those game titles. Summary of the Invention

[0004] These and other issues exist with respect to developing and redeveloping gaming applications. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 illustrates an example implementation of a game enhancement system for generating and providing enhancements to game content, according to one or more embodiments. [Diagram 2] FIG. 1 illustrates an example implementation of a game enhancement system in which an event recognizer is trained to detect in-game events, according to one or more embodiments. [Diagram 3] FIG. 1 illustrates an example implementation of a game enhancement system in which enhancements to a game session are provided without redeveloping the source of the game application. [Figure 4]4A and 4B are diagrams illustrating example implementations in which the presentation of a game session is enhanced, according to one or more embodiments. [Diagram 5] FIG. 1 illustrates another example embodiment in which the presentation of a game session is enhanced, according to one or more embodiments. [Figure 6] FIG. 1 illustrates an exemplary sequence of operations illustrating the expansion of a game session in accordance with one or more embodiments. [Figure 7] FIG. 1 illustrates certain components that may be included within a computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] The present disclosure generally relates to a game enhancement system that provides various enhancements to a game session without time-consuming and costly development of game application software. For example, as described in further detail below, a game service (e.g., a cloud game service) can leverage the content analysis and event recognition services of an event recognizer system implemented in a cloud computing system to detect various game events and provide enhancements to the game based on the detected events. This can be done without analyzing back-end processing signals and without modifications to the game application itself. Thus, the systems and methods described herein can efficiently facilitate the enhancement of various game applications without requiring costly and time-consuming development of software for the game application.

[0007] As an illustrative example, in one or more embodiments, the game enhancement system identifies game content for a game session. The game content may include video content and other types of content associated with the game session (e.g., audio content and controller input). Additionally, the game enhancement system may apply one or more event recognizers to the game content to identify various events that a detection model (e.g., a machine learning model) of the event recognizer is trained to detect. The game enhancement system may identify one or more in-game enhancements associated with the detected events, and may cause the in-game enhancements to be implemented on the client system in conjunction with the presentation of the game session on the client system based on the detection of one or more instances of the events.

[0008] The present disclosure includes many practical applications that provide advantages and / or solve problems associated with enhancing game content in conjunction with the delivery of game content to client systems. Some exemplary advantages are discussed in further detail below. It should be understood that these examples are provided by way of example, and that many additional advantages and practical applications are realized by the features and functionality described herein in connection with the game enhancement system and related components.

[0009] As a first example, the systems described herein facilitate the enhancement of a gaming session in a variety of ways without requiring costly and time-consuming software development of the underlying software code for the gaming application. As discussed in more detail below, the event recognizer may include one or more detection models trained to detect signals indicative of a particular event based on an analysis of the gaming content (e.g., video content). Because the recognizer is trained to detect various events based on the gaming content (e.g., rather than the underlying code or back-end processing data), the systems described herein can be trained to detect events and provide associated enhancements without requiring access to the software, source code, or processing signals of the gaming application. The systems described herein further provide a flexible environment for detecting various events in various gaming applications associated with any number of publishers. For example, by training a model of the event recognizer to evaluate video content, the one or more event recognizers can be trained to recognize events across multiple applications, including applications of different titles and / or applications associated with different publishers. For example, a first recognizer may be trained to detect instances of an event in a game session for a first game application, and a second recognizer (or simply an additional model of the first recognizer) is trained to identify instances of the event (or a similar type of event) in a game session for a second game application. In this manner, similar enhancements based on similar events can be applied to multiple titles and across multiple publishers without redeveloping separate updates (e.g., software patches or new application versions) for each different title and / or publisher.

[0010] Further advantages are realized in connection with one or more embodiments described herein by leveraging the computing power and resources of the cloud. For example, by implementing both the gaming service and the recognition event service on a common cloud computing system, game processing tasks and event recognition tasks can be provided without interfering with the game session from the end user's perspective. Furthermore, since many of the systems and services described herein are implemented on the cloud, any number of recognizers can be tasked with detecting any number of events related to a particular game session. This provides an advantage over traditional game consoles, where attempts to implement these services on a local game console would most likely interfere with the performance of the console in delivering game content to end users. Thus, as discussed in more detail below, the game enhancement system can provide any number of enhancements without compromising the audiovisual requirements of the game console involved in delivering the game content of the game session to client systems.

[0011] As set forth in the preceding discussion, this disclosure utilizes various terms to describe the features and advantages of the systems, methods, and computer-readable media described herein. The meaning of some of these terms will now be described in more detail.

[0012] As used herein, a "cloud computing system" refers to a network of connected computing devices that provide various services to computing devices (e.g., customer devices). For example, a cloud computing system may include a collection of physical server devices (e.g., server nodes and game consoles) organized into a hierarchical structure including clusters, computing zones, virtual local area networks (VLANs), racks, fault domains, and the like. In one or more embodiments, the server devices may be implemented in a data center (e.g., a regional data center) that includes any number of server devices connected via the data center's internal network to provide any number of cloud computing services to clients of the cloud computing system. In one or more embodiments described herein, the cloud computing system includes game consoles and other server nodes that cooperate to perform the features and functions described herein in connection with detecting instances of events in a game session and having client systems implement enhancements to the game session.

[0013] As used herein, a "client system" represents one or more devices through which digital content (e.g., gaming content) is provided to a user. For example, a client system may include a local streaming device, a display device, one or more controllers for providing input commands, and any other device configured to provide content to a user. In one or more embodiments, a client system represents a combination of multiple devices (e.g., a local game console, a television, and a game controller). In one or more implementations, a client system represents a single or integrated computing device, such as a laptop computer or a mobile device.

[0014] As used herein, "game content" refers to any content related to the manner in which a user can experience a game session. For example, in one or more embodiments described herein, game content refers to video content displayed via a display device having a display screen and / or a graphical user interface displayed thereon. Additionally, game content may refer to audio content presented via a speaker or other audio display device. In one or more embodiments, game content may also refer to haptic and / or visual feedback provided via a non-display device, such as vibration or vibrations of a game controller, or lights that can light up and / or change color (e.g., keyboard lights or controller lights). In one or more embodiments, game content refers to user input or user commands that can affect the game content presented via a display device. Indeed, game content may refer to any game content and associated data that is delivered or otherwise presented via a client system device and experienced or otherwise consumed by a user.

[0015] As used herein, a "game session" refers to an instance of a gaming application running on one or more devices. For example, a game session may refer to some period of time associated with launching or joining a game application or game service and logging off or otherwise leaving the game application or game service. By way of illustration, a game session may include an instance where a user starts a video game, plays the video game for some period of time, logs off, closes the application, or simply leaves the video game for some period of time (e.g., an inactivity period threshold). In another implementation, a game session may include an instance where a user logs on to a gaming service or platform, plays one or more video games hosted or accessible through the gaming service or platform, and logs off, closes the service, or leaves the game service for some period of time.

[0016] In one or more embodiments described herein, a game enhancement system may cause a game system to implement one or more enhancements in conjunction with a game session. As used herein, an "enhancement" or "in-game enhancement" may represent additional content or augmentation to game content that may be provided via a client system. For example, an enhancement may represent a graphical icon displayed in conjunction with video content from the game content. In another example, an enhancement may represent audio content or haptic feedback presented via a gaming device and / or a controller device. In one or more embodiments, an enhancement may also represent the removal of a portion of the game content, such as lowering the volume of audio content or removing some graphical icon or text from the display. Indeed, an in-game enhancement may represent any modification of the game content as it is delivered to a user via a client system.

[0017] As used herein, a "recognizer" or "event recognizer" may refer to a component of a game enhancement system having one or more models (e.g., machine learning models) trained to detect objects, content, or any detectable events in game content provided to a client system. For example, a recognizer may include one or more models that are trained to detect various objects, combinations of objects, a series of inputs, or a combination of any number of signals that indicate an instance of an event occurring within a game session. By way of example and not limitation, if a character in a game session jumps, a recognizer may be trained to detect a series of movements or objects that indicate the character's jump and generate an output that indicates the instance of the jump and related information within the game session. Other examples and use cases are discussed in connection with one or more embodiments described herein.

[0018] As used herein, an "event" or "in-game event" refers to a set of signals associated with a given in-game event. For example, as discussed in more detail below, an in-game event may refer to a set of signals related to a particular occurrence or occurrence within a game session. As indicated above, an in-game event may refer to a character jumping within a game session. Other in-game events may include occurrences such as the death of a character, the start of a new level, the end of a level, pausing a game, pausing a game for more than a threshold period of time, unpausing a game, winning a race, losing a race, finishing a race, finishing a race within a predetermined time, and the like.

[0019] Indeed, an in-game event may be defined to include any number of occurrences (or multiple predefined events) defined by a user, administrator, developer, or other entity. An in-game event may represent a combination of one or more signals based on the output of various models that make up one or more recognizers. Although an event may be specific to a particular game application, in one or more embodiments described herein, an event may also include instances of an event that occur across different game applications, across different generations of similar game applications (e.g., different versions or editions of a common game title), and even across game applications associated with entirely different genres and publishers.

[0020] Further details regarding a system for detecting in-game events and providing enhancements to a game session in response to the detected in-game events will now be described. For example, FIG. 1 illustrates an example environment 100 in which a game enhancement system 108 is implemented according to one or more embodiments. As illustrated in FIG. 1, the environment 100 includes a server device 102, which may include any number of server nodes of a cloud computing system. As further illustrated, the environment 100 may include one or more third-party server devices 104 and client systems 106 (e.g., client game systems).

[0021] As shown in FIG. 1, the server device 102 may include a game enhancement system 108 implemented thereon, which may include a game service 110, an event recognizer system 112, and an event analysis system 114. Each of these components 110-114 and associated features and functions may be implemented across multiple server devices of a cloud computing system. Alternatively, the features and functions discussed in connection with each component 110-114 of the game enhancement system 108 may be executed on the same server device. As further shown in FIG. 1, the event recognizer system 112 may include a recognizer training system 116 (or simply a "training system"), an event recognizer 118, and a detection model 120. Each of these components is discussed in further detail below.

[0022] As described above, the environment 100 may include one or more third-party server devices 104 on which one or more publisher systems 122 are implemented. The publisher systems 122 may include provider or publisher-specific services hosted by various publishers related to one or more game applications. In one or more embodiments, the publisher systems 122 may maintain original copies of game applications and user information associated with users of the client systems 106. Although FIG. 1 illustrates an example implementation in which the publisher systems 122 are hosted on one or more third-party server devices 104, it should be understood that one or more features of the publisher systems 122 may be implemented on the game services 110 or other components 112-114 of the game enhancement system 108 and in the same cloud computing system as the server devices 102 shown. For example, features of the publisher systems 122 may be implemented in the game enhancement system 108 if the same entity providing the game services 110 also provides certain game content or if a third-party publisher provides full access to the game applications hosted by the game services 110.

[0023] 1, client system 106 may include one or more display devices 124 and one or more input devices 126. For example, as described above, client system 106 may include a local computing device (e.g., a game console) coupled to various hardware for providing game content for a game session to a user of client system 106. In one or more embodiments, this may include separate display device 124 and input device 126 (e.g., a desktop computer and keyboard / mouse, a television and game controller coupled to a local console). In one or more embodiments, display device 124 and input device 126 are part of a single or integrated computing system (e.g., a laptop computer or a mobile device).

[0024] In fact, it should be understood that the client system 106 may represent various types of computing devices coupled to or including the display device 124 and / or the input device 126. For example, the client system 106 may include a mobile device, such as a mobile phone, a smart phone, a personal digital assistant (PDA), a tablet, or a laptop. Additionally or alternatively, the client system 106 may include one or more non-mobile devices, such as a desktop computer, a server device, or other non-portable devices. In one or more implementations, the client system 106 represents a dedicated gaming device (e.g., a handheld gaming device) or a video game console that communicates with a display device. In one or more implementations, the client system 106 may include a graphical user interface integrated therein (e.g., a mobile device screen). Additionally or alternatively, the client system 106 may be communicatively coupled (e.g., wired or wireless) to a display device having a graphical user interface for providing a display of visual game content. It should also be understood that the server device 102 and the third-party server device 104 may represent various computing devices. Each of the devices and systems 102-106 of environment 100 may include the features and functionality described below in conjunction with FIG.

[0025] 1, the server device 102, the third party server device 104, and the client system 106 can communicate with each other directly or indirectly via a network 128. The network 128 may include one or more networks and may use one or more communication platforms or technologies suitable for transmitting data. The network 128 may represent any data link that enables the transport of electronic data between the devices and / or modules of the environment 100. The network 128 may represent a wired network, a wireless network, or a combination of wired and wireless networks. In one or more embodiments, the network 128 includes the Internet.

[0026] As discussed in further detail below, gaming service 110 may provide one or more services related to hosting and delivering gaming content to client systems 106. For example, in one or more embodiments, gaming service 110 represents a cloud-based gaming service in which one or more physical game consoles on a cloud computing system can host game sessions and deliver game content to client systems 106. In this example, the game consoles may run or host game sessions and process input received from client systems 106 (e.g., from input devices 126). In one or more embodiments, gaming service 110 may stream or otherwise deliver game content to client systems 106 over network 128.

[0027] In addition to delivering game content to client systems 106, game services 110 may provide copies or feeds of game content to event recognizer system 112. Game services 110 may also provide indications of input commands received in association with the game content. As discussed in further detail below, event recognizer system 112 may apply one or more event recognizers 118 to the game content to identify instances of one or more events occurring within a game session. As shown in FIG. 1, event recognizer 118 may utilize any number of detection models 120 trained (e.g., by recognizer training system 116) to detect various signals indicative of one or more events occurring within a game session.

[0028] Upon identifying one or more instances of an event, the event recognizer system 112 can provide the event information to the event analysis system 114. The event analysis system 114 can associate the event with one or more in-game augmentations. Based on this association, the event analysis system 114 can determine to augment the game content being provided to the client system 106. As discussed in further detail below, the event analysis system 114 can augment the game session in various ways. For example, in one or more embodiments, the event analysis system 114 can provide augmentation instructions or commands to the gaming service 110 that enable the gaming service 110 to provide the augmented content to the client system 106 over the network 128. In one or more embodiments, the event analysis system 114 provides the augmentation instructions to one or more components of the client system 106 to enable the client system 106 to locally execute the augmentation instructions and present one or more local augmentations.

[0029] 2 illustrates an example implementation of the game enhancement system 108 that can train event recognizers to detect various events according to one or more embodiments. FIG 2 illustrates an example workflow 200 in which an example implementation of the recognizer training system 116 can train one or more recognizers to detect various events based on game content associated with a given game session.

[0030] As discussed below, the recognizer training system 116 may be trained based on a variety of training data. In particular, in the workflow 200 shown in FIG. 2, the recognizer training system 116 may receive a collection of video frames 202 that include digital images of video content for one or more gaming sessions. In one or more embodiments, the video frames 202 represent previously recorded and stored video content associated with one or more gaming sessions in which various events are occurring.

[0031] In one or more embodiments, the recognizer training system 116 receives additional game content and training data associated with the video frames 202. For example, in one or more embodiments, the recognizer training system 116 receives audio content associated with or otherwise presented in conjunction with the video frames 202. In one or more implementations, the recognizer training system 116 further receives input data including an indication of any input commands received in conjunction with the corresponding video frames 202 and associated audio data. For example, the video frames may be associated with corresponding timestamps, and the timestamps can be used to associate the video frames 202 with the corresponding audio content, input commands, and any other game content associated with the corresponding game session.

[0032] In addition to the video frames 202 and any additional game content, the recognizer training system 116 may further receive training input 204 associated with the corresponding video frames 202. In particular, the recognizer training system 116 may receive training input 204 indicating ground truth associated with one or more events determined to have occurred within the corresponding game content. In one or more embodiments, the training input 204 may represent human input or other supervised training input that provides highly accurate predictions of ground truth or truth data corresponding to particular events or types of events that occurred within the video frames 202.

[0033] As an example, if the recognizer training system 116 is being used to train or otherwise generate an event recognizer to recognize a pause event related to pausing a game, the recognizer training system 116 may receive training input 204 indicating whether the game is paused for a given video frame. In one or more embodiments, this may be based on tags provided by a user based on the user's observations of the video frame 202 and other game content. As a simple example, if a video frame includes a display of a pause menu or other visible icon indicating that the game session is paused, the corresponding training input 204 indicates that the game is paused for the given video frame. Conversely, if a video frame does not include a display of a pause menu or other icon indicating a pause, the corresponding training input 204 indicates that the game is not paused.

[0034] The recognizer training system 116 can utilize these input video frames 202 and associated training inputs 204 to train an event detection model to mimic the training inputs and predict whether a game session will be paused based on correlations between the training inputs 204 and various features or characteristics of the tagged video frames 202. In this manner, one or more pause detection models can be trained to recognize when a game is paused based on the display of a pause icon or any combination of observed signals (e.g., a pause in music, a lack of movement, a received pause input), thereby enabling the models and associated event recognizers to accurately predict when a game session will be paused.

[0035] The detection models (e.g., detection models 212a-b) may represent various models and algorithms that may be used to predict various signals and ultimately determine whether an instance of an event corresponding to the signal occurred within a game session. In one or more embodiments, the detection models may represent machine learning models or other deep learning models. As used herein, a "machine learning model" may represent one or more computer algorithms or models (e.g., classification models or regression models) that may be tuned (e.g., trained) based on training inputs to approximate an unknown function. For example, the machine learning model may represent a neural network (e.g., a convolutional neural network or a recurrent neural network) or other architecture that learns and approximates a complex function and generates an output based on one or more inputs provided to the machine learning model. As discussed in connection with various examples herein, the various machine learning models may generate various output signals that are combined or considered by the recognizer to determine whether an event occurred with respect to a corresponding portion of the game content (e.g., within a particular video frame or sequence of video frames).

[0036] In training the various models and recognizers, in one or more implementations, the recognizer training system 116 may receive region of interest information 206 associated with locations in the game content where various events are likely to occur. As shown in FIG. 2, the recognizer training system 116 may receive region of interest information 206 that includes a segmentation mask that indicates an area of ​​the video frame 202 that should be considered in determining whether an instance of an event occurs in the corresponding video frame. In this manner, the recognizer training system 116 can downplay or simply not consider other portions of the video frame in training the models to detect various signals and events. This restriction of training to regions of interest in the video frame 202 allows the recognizer training system 116 to more efficiently train various machine learning models in less time and using less training data.

[0037] In addition to more efficiently training various event detection models and recognizers, providing the region of interest information 206 can provide useful information for training models related to detecting similar types of events between different gaming applications. As an example, if an event represents an occurrence in which an avatar's life bar falls below some threshold level or percentage, the region of interest may indicate a first portion of a video frame in which the life bar appears for a first gaming application. Similarly, the region of interest may indicate a second portion of a video frame in which the life bar appears (e.g., a different region within the video frame) for a second gaming application. This may involve training two different models for each gaming application to detect similar events, but the models may nevertheless be trained using relatively fewer video frames and associated training inputs than if entire portions of the video frames were considered in training the corresponding event detection model.

[0038] After training the various models, the recognizer training system 116 may provide the event recognizers and associated detection models to a recognizer database 208. As shown in FIG. 2, the recognizer database 208 may include multiple event recognizers and associated detection models (e.g., one or more models). For example, the recognizer database 208 may include a first event recognizer 210a with an associated detection model 212a. As further shown, the recognizer database 208 may include a second event recognizer 210b and a set of associated detection models 212b.

[0039] As shown in FIG. 2, various event recognizers may include one or more models based on numerous considerations and details of the unique event that the recognizer is trained to detect. For example, in one or more embodiments, recognition of a simple event such as the appearance of a black screen may include a single model trained to detect the occurrence of a threshold number of black video frames. Alternatively, recognition of a more complex event such as an event entitled "barely completed the level" may include a first model trained to detect when a life bar falls below a threshold level in addition to a second one or more models trained to detect when a level has been completed. In one or more embodiments, the features of these models may be combined to detect the in-game event to which a particular event recognizer corresponds (e.g., completed a level but the avatar's life is below a threshold).

[0040] In addition to an event recognizer having one or more models, in one or more embodiments, one or more models may be utilized by different recognizers. For example, in one or more embodiments, when tasked with training an event recognizer, the recognizer training system 116 may identify previously trained or commonly used models that relate to many different events and incorporate the features of the models into the event recognizer. In this way, as the recognizer database 208 becomes more robust over time, the recognizer training system 116 can leverage the features of previously trained models or simply fine-tune existing models to fit the given features of one or more event recognizers.

[0041] 3 illustrates an exemplary implementation of the game enhancement system 108, in which a previously trained event recognizer is used to detect various in-game events. For example, FIG. 3 illustrates an exemplary workflow 300 illustrating the flow of data and game content between the game service 110 on a cloud computing system, the event recognizer system 112, and the event analysis system 114. In particular, as described above and discussed in further detail below, the components 110-114 of the game enhancement system 108 can cooperate to identify instances of various in-game events occurring within a game session and provide one or more enhancements without requiring redeployment or redevelopment of the associated game application.

[0042] 3, the gaming service 110 can provide first game content 302a (e.g., original game content) to the client system 106. In one or more embodiments, the first game content 302a represents an original feed of game content that is streamed or otherwise delivered to the client system 106. In one or more embodiments, the gaming service 110 includes a cloud-based gaming console that generates and processes the game content (e.g., multimedia content) and delivers the first game content 302a to be displayed on a display device of the client system 106.

[0043] As shown in FIG. 3, the client system 106 may provide input commands 304 associated with the first game content 302a. In particular, if the client system 106 includes one or more controller devices (e.g., keyboard, mouse, game controller, touch screen interface) capable of detecting user input, the client system 106 may provide the input commands 304 including various inputs as they are received at the input devices of the client system 106. In one or more embodiments, the gaming service 110 processes the input commands, implements the commands within the game session, and continues to provide the first game content 302a based on the input commands. While this processing is taking place on the gaming service 110, from the user's perspective, the delivery and processing of the input commands may be very similar to a conventional system in which a local game console hosts and provides game content and processes local input commands.

[0044] Concurrent with delivery of the first game content 302a to the client system 106, or at some point thereafter, the second game content 302b (e.g., a copy of the original game content) may be provided to an event recognizer of the event recognizer system 112. In the example shown in FIG. 3, a copy of the game content 302b is provided to each of a plurality of event recognizers 306a-c, each of which is trained to detect in-game events. For example, the first event recognizer 306a may include one or more machine learning models trained to detect instances of a first type of event, the second event recognizer 306b may include one or more machine learning models trained to detect instances of a second type of event, and the third event recognizer 306c may include one or more machine learning models trained to detect instances of a third type of event.

[0045] 3 illustrates an example that includes three event recognizers 306a-c that are individually trained to detect different types of events, other implementations may include fewer or additional event recognizers trained to detect any number of events. For example, in one or more embodiments, a copy of the game content 302b may be provided to each of the multiple event recognizers simultaneously and processed in parallel.

[0046] As discussed above, a copy of the game content 302b may be provided to the event recognizers 306a-c at the same time that the original game content 302a is delivered to the client system 106. Alternatively, in one or more embodiments, a copy of the game content 302b is provided to the event recognizers 306a-c after the fact for purposes of telemetry tracking and possibly additional training of the event recognizers 306a-c.

[0047] Additionally, in one or more embodiments, one or more event recognizers may be used to detect events in real-time as original game content is delivered to client system 106, while one or more additional event recognizers may be used to detect events after the fact for other enhancement-related purposes. For example, in one or more embodiments, an event recognizer may detect events based on a history of events observed over an extended period of time.

[0048] Thus, tracking and maintaining the current number of event instances may be less important for a particular event type, and the event recognizer may be more efficiently utilized during periods of non-peak computing demand on the cloud computing system, rather than using computing resources during periods of peak demand on the cloud computing system. In this and other examples, the event recognizer system 112 may run one or more event recognizers as part of a batch processing job.

[0049] Another example where real-time recognition may not be as necessary is when watching a replay of a game session. Indeed, as watching replays of game sessions has become increasingly popular, one or more event recognizers may be applied to the video and other game content after the fact (i.e., after the original game content is delivered in real time to the client system 106) to identify various in-game events for the purpose of enhancing the replay of the game session. This may include icons indicating particular in-game events or any other events according to the embodiments described herein.

[0050] In any implementation involving delayed or real-time analysis of the copy of the game content 302b, the recognizer training system 116 may generate an output signal 308 from the event recognizers 306a-306c indicating whether one or more instances of an event have been detected with respect to a corresponding portion of the game content (e.g., a video frame or range of video frames). As shown in FIG. 3, the output signal 308 may be provided to the event analysis system 114 for further processing.

[0051] In particular, as described above, in one or more embodiments, the event analysis system 114 can receive an indicated instance of an event at a corresponding portion or timestamp of the game content and determine an associated enhancement to apply to the event. In one or more embodiments, the event analysis system 114 maintains mapping data that enables the event analysis system 114 to correlate any number of output signals 308 indicating the occurrence of an event with a corresponding enhancement implemented on the client system 106.

[0052] In addition to generally providing an output signal 308 indicative of an instance of an event, the event analysis system 114 may receive information related to an individual or hardware associated with a particular client system 106 that is receiving (or has received) the original game content 302a. In one or more embodiments, the output signal 308 includes a JavaScript Object Notation (JSON) object that includes an indication of the corresponding event and a gamertag or user identifier that may be used to identify the particular game session and / or associated client system 106 to which the in-game enhancement applies. Other types of data objects having various file formats may likewise be used to convey information related to the detected event and the corresponding game session.

[0053] Upon associating the detected event instance with a corresponding enhancement, the event analysis system 114 may identify an enhancement instruction or command that includes specific data executable by one or more of the gaming service 110 and / or the client system 106. In particular, the event analysis system 114 may identify an enhancement instruction that enables a specific enhancement to be implemented in conjunction with the presentation of the game session at the client system 106. This may include specific graphics and associated commands for displaying the graphics in conjunction with the presentation of the original game content 302a. This may also include instructions for presenting haptic, audio, or other game content that can be consumed by a user of the client system 106. One or more examples of specific types of enhancements to a game session are discussed below in conjunction with FIGS. 4A-5.

[0054] As described above, the event analysis system 114 may implement the enhancements to the game session on the client system 106 in a variety of ways. In a first example, the event analysis system 114 may provide the enhancement instructions 310a to the game service 110 for incorporation into the presentation of the game content 302a. In this example, a game console at the game service 110 may receive the enhancement instructions 310a and distribute or otherwise provide the enhanced game content 312 to the client system 106 for presentation via one or more devices of the client system 106. In one or more embodiments, the enhanced game content 312 includes one or more displayable icons that are presented in conjunction with video frames of the original game content 302a. In one or more embodiments, the enhanced game content 312 includes additional sounds or sound modifications from the original game content 302a delivered to the client system 106. In one or more embodiments, the enhanced game content 312 may include executable instructions by a controller device to vibrate or pulsate, while the original game content 302 may not include similar commands.

[0055] As a second example, in addition to or as an alternative to providing the augmentation instructions 310a to the gaming service 110 for further processing, in one or more embodiments the event analysis system 114 provides the augmentation instructions 310b directly to the client system 106. These augmentation instructions may not necessarily be the same and may not necessarily be related to augmentation instructions for enhancing video content provided by the gaming service 110, but the augmentation instructions 310b may include instructions related to providing haptic, audio, and / or visual feedback to an input device, such as a keyboard or game controller, that can receive and execute a variety of instructions.

[0056] Where each of the components 110-114 of the game enhancement system 108 and the client system 106 may be located remotely from one another, there may be short periods during which latency may cause one or more problems in delivering and presenting the enhancements in conjunction with the game content 302a being delivered to the client system 106. Some events and associated enhancements may be less affected by some latency, while other events and associated enhancements may disrupt the game experience if the delay is too long. For example, if the enhancement represents a mini-map that is displayed on a portion of the screen, latency may not be much of an issue, and the enhancement may be presented regardless of the amount of latency detected by the client system 106.

[0057] Alternatively, if the augmentations represent real-time controller vibrations based on detected events or display of information to assist the user in providing specific input commands to overcome in-game challenges in a timely manner (or various events and associated augmentations), the presentation of these augmentations may interrupt the game session or cause confusion to the end user due to observed delays. In one or more embodiments, the game augmentation system 108 may prevent instances of augmentation delays by passing timing information (e.g., timestamps) along with a copy of the game content 302b and continuing to pass the timing information along with the output signal 308 (e.g., in a JSON object) to the event analysis system 114. The event analysis system 114 may further include this timing information when providing the augmentation instructions 310a-b to the game service 110 and the client system 106.

[0058] This timing information can be used to determine whether to discard the augmentations or to implement the augmentations within the presentation of the game session. For example, when the augmentation instructions 310a are provided to the game service 110, the game service 110 can determine whether the provision of the augmented game content 312 is delayed beyond a threshold period (e.g., 200 milliseconds) and simply continue to provide the original game content 302a without the associated augmentations. In one or more embodiments, each event may be associated with a threshold delay time that is acceptable in determining whether to provide the augmented game content 312 to the client system 106.

[0059] In an example where the augmentation instructions 310b are provided directly to the client system 106, the client system 106 may receive the timing information and locally determine whether to provide the augmentation to the game content 302a being provided from the gaming service 110. In one or more embodiments, the augmentation instructions 310b may similarly include both a timestamp associated with the live game content 302a and a designated threshold period during which the augmentation may be delayed without interrupting the game experience (e.g., excessively delayed vibration of a game controller). In this example, the gaming service 110 may locally determine whether the threshold period has elapsed since the timestamp. If the threshold period has elapsed, the client system 106 may decide not to implement the augmentation. If the threshold period has not elapsed, the augmentation may be presented in conjunction with the presentation of the original game content 302a (e.g., within an acceptable delay defined by the augmentation instructions 310b).

[0060] Further details will now be discussed in connection with example implementations in which various extensions are provided in combination with game content provided to a client device. For example, FIG. 4A illustrates an example environment showing a game service 110 providing original game content 402a to a client system. As shown in FIG. 4A, the client system may include a client device 404 that displays video content from the received game content 402a via a graphical user interface 406 on the client device 404. As further shown, the client system may provide input commands 403 to the game service 110 for execution on one or more remote game consoles that are generating and distributing the original game content 402a.

[0061] As further shown in FIG. 4A, the gaming service 110 may provide copies of the gaming content 402b to any number of event recognizers trained to detect various events in the gaming content delivered to the client system. In this example, the related gaming content may relate to a racing application including video content showing various information. For example, as shown in FIG. 4A, a dashboard icon may appear in a lower left portion of the graphical user interface 406 showing various information that may be detected from one or more event recognizers. For example, an event recognizer may determine a revolutions per minute (RPM) metric associated with a displayed gauge. Another event recognizer may track or otherwise determine a current speed of a car in the racing application. Another event recognizer may track a displayed ranking of a racer based on the appearance of text displayed via the graphical user interface 406 indicating the current ranking of the car in the racing application.

[0062] According to one or more embodiments described above, any number of event recognizers may utilize one or more event detection models (e.g., machine learning models) to detect or otherwise identify a variety of events, which may often be based on a combination of multiple observed signals. As an illustrative example, the event recognizer may be trained to determine a user's current location on a map of a virtual environment based on a history of tracked movements and turns made by the user (and other users) over time. This may be based on any number of machine learning models trained to track the movement of a car (e.g., within a virtual environment of a game session for a racing application). The same or other recognizers may be applied to the historical data to generate a map representative of a given racing environment. This information may be used by the event recognizer to determine the current location of the car in the racing environment.

[0063] As another example, a recognizer may be trained to determine whether a car in a virtual environment is traveling at a speed above a predetermined threshold. In this example, the event recognizer may determine that the car is speeding above a 100 miles per hour (MPH) threshold based on detected text displayed within a dashboard icon at the bottom left of the screen. In accordance with one or more embodiments described above, the recognizer may make a decision about a speed-based event based on selective evaluation of the bottom left portion of the screen based on a segmentation mask or other indicator of an area of ​​interest associated with a known location of the dashboard icon.

[0064] As described above, the recognizer may be trained to detect multiple related events associated with the corresponding extension. For example, in the example shown in FIG. 4A, the recognizer may be trained to determine that a car is traveling above a threshold speed while approaching a curve. A first model or models of the recognizer may detect that the car is traveling above the threshold speed, and a second model or models of the recognizer may detect that the car is approaching a curve (e.g., based on a known position of the car in a racing environment or based on a detected curve shown on a display).

[0065] 4B illustrates one or more enhancements presented over a game session, according to one or more examples described above. For example, as shown in FIG. 4B, the gaming service 110 may receive an enhancement instruction 408 from the event analysis system 114 indicating one or more enhancements or augmentations related to a detected event to be made to the gaming content 402a being provided to the client device 404. Upon receiving the enhancement instruction 408, the gaming service 110 may provide one or more enhancements 410 to be presented in combination with the gaming content 402a being streamed to the client device 404.

[0066] 4B, the expanded presentation includes a first expansion 412a in which a minimap is displayed in the lower right corner of the graphical user interface 406. While the minimap may be generated based on any number of signals or combinations of signals, including a combination of signals observed in real time and previously collected and analyzed signals, the minimap may include a map representing the race track and an icon indicating the current position of the car in the virtual environment relative to the race track.

[0067] As another example, the augmented presentation may include a second augmentation 412b in which a "Slow down!" warning is displayed over a portion of the graphical user interface 406. According to one or more embodiments described herein, this augmentation may be generated based on a recognizer trained to detect instances of cars traveling faster than a threshold speed and approaching an upcoming curve in the game. Additionally, as with other implementations described herein, the augmented instructions may be associated with a particular combination of detected signals and / or events that correspond to a recognizer trained to detect a particular event.

[0068] 4A-4B illustrate features and functionality that may be applicable to many use cases. As an example, a racing application may represent a game application previously released for an older generation gaming system. In one or more embodiments, a particular recognizer and corresponding extensions may be trained or otherwise identified based on newer features that have been added to current or newer versions of a similar game title. For example, if a newer edition of a game has a minimap, a user of the older title may desire to include a similar minimap while enjoying the particular features of the older title. However, because developing software patches or adding features to older titles (or even simply redeveloping the entire game application) may be difficult or even impossible without incurring significant development costs, the event detection and extension features described herein may provide a low-cost, non-invasive solution for adding various features to older game titles. Additionally, while FIGS. 4A-4B illustrate an example in which an extension may be added to a racing application, similar features and functionality associated with extensions to older titles of previously released game applications may be applied to game applications of virtually any genre.

[0069] As another example, in accordance with one or more embodiments described above, the features and functions described in connection with the embodiments herein may be applied across multiple publishers and even across multiple genres of games. As an example, if an event is "winning a race," one or more recognizers related to winning a race may be applied to the various races occurring within a game application. For example, winning a race may be applied to a car racing game, such as the example described in connection with FIG. 4A. However, other games have variations of racing games, such as role-playing games (RPGs) with various side quests, or first-person games in which multiple players compete to complete a task within a limited time period.

[0070] As another example, one or more embodiments may include detecting events that may be relevant to providing enhancements in future game sessions or in relation to game sessions of other users of the game service 110. For example, in one or more embodiments, the game enhancement system 108 may detect various events that occur during the initial execution of a game application for a user. In subsequent or future game sessions, the game enhancement system 108 may cause one or more enhancements to be presented based on the detected events of the previous game session. For example, when a player approaches a path decision or fork in the road, the game enhancement system 108 may cause a pop-up icon or other visual enhancement to be presented that indicates that the player took a particular path in a previous game session. As another example, the game enhancement system 108 may cause a pop-up icon to be generated that indicates what other players of the game service 110 did, such as what the user's friends or popular gamers (e.g., well-known streamers or other content creators) did when faced with a similar decision in the past in a game.

[0071] In these examples, the systems described herein may train recognizers that utilize multiple different machine learning models trained to detect race winning events across different types of gaming applications. For example, a first model or models may be trained to detect a win in a car race based on text displayed on a screen in combination with one or more observable signals. Conversely, a second model or models may be trained to detect a first-person win based on text displayed in different locations or based on other observable features from the gaming content presented via the client system (e.g., a display of a reward associated with winning a race or a detected sound associated with winning a race). Each of these models may be incorporated within a single recognizer. Alternatively, multiple recognizers may be trained to detect different instances of a common event.

[0072] While Figures 4A-4B illustrate an example in which an event is detected based on visual content and the augmentation relates to visual augmentation presented via a graphical user interface of a client device, Figure 5 illustrates another example in which the augmentation may relate to non-visual augmentation. For example, Figure 5 illustrates an example of an exemplary game service providing original game content 502 to a client system 504 that includes a display device 506 on which a graphical user interface 508 is presented and an input device 510 (e.g., a keyboard) communicatively coupled to the display device 506. As illustrated in Figure 5, the client system 504 can provide input commands 503 for execution by the game service 110, and the input commands 503 can be incorporated within the game content 502 delivered to the client system 504.

[0073] 3, the expansion instructions may be received at the gaming service 110 and / or the client system 504. In this example, the game expansion instructions may be received at the client system 504 and implemented by hardware of the client system 504 with or without additional expansion input from the gaming service 110.

[0074] In this example, the recognizer may detect an event associated with the display of a "x" press indicator 512 displayed via the graphical user interface 508 of the display device 506. In accordance with the example described above, the recognizer may be trained to selectively evaluate particular portions of the game content 502 (e.g., based on a segmentation map or other indication of an area of ​​interest) to detect particular text corresponding to the "x" press event. In response to detecting this event, the client system 504 may be instructed to perform any number of enhancements corresponding to the detected event.

[0075] 5, the x key 514 on the input device 510 may light up to indicate which x key the user should press based on the content displayed on the display device 506. This visual feedback, although not displayed, can greatly enhance the gaming experience by providing a more enjoyable gaming experience and by providing additional instructions to the user of the gaming application. This can be provided in conjunction with a game tutorial or simply as an additional aid enabled by the system features and functionality described herein.

[0076] While one or more examples described herein may be particularly relevant to enhancements executed on a server device of the game enhancement system 108 or, alternatively, on hardware of the client system 106, it should be understood that certain examples described in conjunction with enhancement instructions provided to and executed by the respective systems may also be implemented by one or both of the game enhancement system 108 and / or the client system 106. For example, while the examples discussed above in conjunction with FIGS. 4A-4B were described in conjunction with enhancement instructions provided to and executed on the gaming service 110, similar enhancements may be executed in whole or in part on the client system 106. As another example, while FIG. 5 was described in conjunction with enhancement instructions provided to and executed on the client system 106, similar enhancements may be received by and executed or otherwise implemented on the gaming service 110 and provided to the client system 106 in conjunction with the game content.

[0077] Referring now to FIG. 6, this figure illustrates an exemplary flow chart including a series of operations for detecting an in-game event and implementing an enhancement to a game session based on the detected in-game event. Although FIG. 6 illustrates operations according to one or more embodiments, alternative embodiments may omit, add, rearrange, and / or modify any of the operations illustrated in FIG. 6. Additionally, one or more of the individual features and functions described in connection with an individual operation or series of operations in connection with FIG. 6 may be equally applicable to other embodiments and examples described herein. The operations of FIG. 6 may be implemented as part of a method. Alternatively, a non-transitory computer-readable medium may include instructions that, when executed by one or more processors, cause a computing device to perform the operations of FIG. 6. In yet another embodiment, a system may perform the operations of FIG. 6.

[0078] As discussed above, Figure 6 illustrates a series of operations 600 related to detecting an in-game event and implementing an enhancement to a game session based on the detected in-game event. As shown in Figure 6, the series of operations 600 includes an operation 610 of identifying game content for the game session being provided to a client system. For example, in one or more implementations, the operation 610 includes identifying game content for the game session, where the game content includes video content provided to the client system by a game server on a cloud computing system.

[0079] As further illustrated, the set of operations 600 includes an operation 620 of applying an event recognizer to the game content, the event recognizer being trained to detect an instance of the event based on output from the one or more detection models. For example, in one or more implementations, the operation 620 includes applying a first event recognizer to the game content to detect an instance of the event, the first event recognizer including a first one or more detection models trained to output one or more signals associated with an occurrence of a given instance of the event based on the game content for the corresponding game session. In one or more embodiments, the first one or more detection models include machine learning models that are applied to video frames of the game content at the same time the game content is provided to the client system.

[0080] As further shown, the set of operations 600 includes an operation 630 that identifies an in-game enhancement associated with the event. As further shown, the set of operations 600 may include an operation 640 that causes a client system to implement the in-game enhancement in conjunction with the presentation of the game session. For example, in one or more embodiments, the operation 640 includes causing a client system to implement the in-game enhancement in conjunction with the presentation of the game session at the client system based on a first event recognizer that detects an instance of the event.

[0081] In one or more embodiments, the game content includes video game content streamed to the client system by a game server. Further, in one or more embodiments, the game content includes one or more of audio content and controller input. In one or more embodiments, the one or more detection models are trained to output one or more signals associated with the occurrence of a given instance of an event based on a combination of two or more of the video game content, the audio content, and the controller input.

[0082] In one or more embodiments, the first event recognizer includes a first detection model or models trained to output one or more signals associated with the occurrence of a given instance of an event based on game content from a first game application. The first event recognizer may further include a second detection model or models trained to output one or more signals associated with the occurrence of a given instance of an event based on game content from a second game application.

[0083] In one or more embodiments, the first game application is a previously released version of the second game application. Further, in one or more embodiments, the first game application and the second game application are associated with different publishers. In one or more embodiments, the first game application and the second game application are associated with different genres of game applications.

[0084] In one or more embodiments, the set of operations 600 includes applying a second event recognizer to the game content to detect an instance of the second event, the second event recognizer including a second one or more detection models trained to output one or more signals associated with an occurrence of a given instance of the second event based on the game content for the corresponding game session. The set of operations 600 may further include identifying a second in-game enhancement associated with the second event and causing the client system to implement the second in-game enhancement in conjunction with presenting the game session on the client system and in conjunction with the in-game enhancement associated with the instance of the first event based on the second event recognizer detecting the instance of the second event.

[0085] In one or more embodiments, the in-game augmentations include graphical user interface (GUI) augmentations associated with displaying one or more graphical icons on a display of the game session in a GUI of the client system. In one or more embodiments, the in-game augmentations include input device augmentations associated with haptic or visual feedback presented via an input device of the client system.

[0086] In one or more embodiments, the operations of causing the client system to implement the in-game augmentations include receiving augmented video content from a game server, the augmented video content including both video content and one or more augmentations to the video content presented via a graphical user interface (GUI) of the client system. In one or more embodiments, the operations of causing the client system to implement the in-game augmentations include receiving one or more augmentation commands from a data analytics service on a cloud computing system and executing the one or more augmentation commands locally on hardware of the client system.

[0087] 7 illustrates certain components that may be included within a computer system 700. One or more computer systems 700 can be used to implement the various devices, components, and systems described herein.

[0088] Computer system 700 includes a processor 701. Processor 701 may be a general purpose single or multi-chip microprocessor (e.g., an Advanced RISC (Reduced Instruction Set Computer) Machine (ARM)), a special purpose microprocessor (e.g., a Digital Signal Processor (DSP)), a microcontroller, a programmable gate array, or the like. Processor 701 is sometimes referred to as a central processing unit (CPU). Although only one processor 701 is shown in computer system 700 of Figure 7, in alternative configurations a combination of processors (e.g., an ARM and a DSP) could be used.

[0089] The computer system 700 also includes a memory 703 in electronic communication with the processor 701. The memory 703 may be any electronic component capable of storing electronic information. For example, the memory 703 may be embodied as a random access memory (RAM), a read-only memory (ROM), a magnetic disk storage medium, an optical storage medium, a flash memory device in RAM, an on-board memory included in the processor, an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) memory, registers, and the like, and combinations thereof.

[0090] The instructions 705 and data 707 may be stored in memory 703. The instructions 705 may be executable by the processor 701 to implement some or all of the functionality disclosed herein. Execution of the instructions 705 may include use of the data 707 stored in the memory 703. Any of the various examples of modules and components described herein may be implemented, in part or in whole, as instructions 705 stored in the memory 703 and executed by the processor 701. Any of the various examples of data described herein may be among the data 707 stored in the memory 703 and used during execution of the instructions 705 by the processor 701.

[0091] The computer system 700 may also include one or more communication interfaces 709 for communicating with other electronic devices. The communication interfaces 709 may be based on wired communication technologies, wireless communication technologies, or both. Some examples of communication interfaces 709 include a Universal Serial Bus (USB), an Ethernet adapter, a wireless adapter that operates according to the Institute of Electrical and Electronics Engineers (IEEE) 802.11 wireless communication protocol, a Bluetooth® wireless communication adapter, and an infrared (IR) communication port.

[0092] The computer system 700 may also include one or more input devices 711 and one or more output devices 713. Some examples of input devices 711 include a keyboard, a mouse, a microphone, a remote control device, buttons, a joystick, a trackball, a touch pad, and a light pen. Some examples of output devices 713 include speakers and a printer. One particular type of output device typically included in a computer system 700 is a display device 715. The display device 715 used in the embodiments disclosed herein may utilize any suitable image projection technology, such as liquid crystal display (LCD), light emitting diode (LED), gas plasma, electroluminescence, and the like. A display controller 717 may also be provided for converting data 707 stored in memory 703 into text, graphics, and / or video (as appropriate) shown on the display device 715.

[0093] The various components of the computer system 700 may be coupled together by one or more buses, which may include a power bus, a control signal bus, a status signal bus, a data bus, etc. For purposes of clarity, the various buses are illustrated in FIG. 7 as a bus system 719.

[0094] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a particular manner. Any features described as modules, components, etc. may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a non-transitory processor-readable storage medium that includes instructions that, when executed by at least one processor, perform one or more of the methods described herein. The instructions may be organized into routines, programs, objects, components, data structures, etc., that may perform particular tasks and / or particular data types, and may be combined or distributed as desired in various embodiments.

[0095] The method steps and / or operations described herein may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or operations is required for the proper operation of the method being described, the order and / or use of specific steps and / or operations may be modified without departing from the scope of the claims.

[0096] The term "determining" encompasses a variety of operations, and thus "determining" can include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or another data structure), ascertaining, and the like. "Determining" can also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. "Determining" can also include resolving, selecting, choosing, setting, and the like.

[0097] The terms "comprising," "including," and "having" are intended to be inclusive and mean that there may be additional elements other than the listed elements. Moreover, it should be understood that references to "one embodiment" or "embodiments" in this disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. For example, any element or feature described in connection with one embodiment herein may be combined, where compatible, with any element or feature of any other embodiment described herein.

[0098] The present disclosure may be embodied in other specific forms without departing from its spirit or characteristics. The described embodiments are illustrative and should not be considered limiting. The scope of the present disclosure is therefore indicated by the appended claims, rather than the foregoing description. All changes that come within the meaning and scope of the claims are embraced within their scope.

Claims

1. 1. A method implemented by one or more server devices on a cloud computing system, comprising: obtaining a video portion of game content of a game session, the video portion being obtained from the game content as the game content is provided to a client system from a game server on the cloud computing system; applying a first event recognizer to the video portion of the game content to detect instances of an event, the first event recognizer including one or more detection models trained to output one or more signals associated with an occurrence of a given instance of the event based on the video portion of the game content for a corresponding game session, the first event recognizer recognizing the event based on content of one or more video frames of the video portion; identifying an in-game enhancement associated with the event; implementing the in-game enhancements at the client system in conjunction with presenting the game session at the client system based on the first event recognizer detecting the instance of the event; The method includes:

2. 2. The method of claim 1, wherein the game content includes video game content streamed to the client system by the game server, and the video game content corresponding to the video portion of the game content is obtained when the game content is provided to the client system.

3. 3. The method of claim 2, wherein the game content further includes one or more of audio content and a controller input, and the one or more detection models are trained to output the one or more signals associated with an occurrence of the given instance of the event based on a combination of two or more of the video game content, the audio content, and the controller input.

4. 2. The method of claim 1 , wherein the first event recognizer comprises: a first one or more detection models trained to output at least one signal associated with an occurrence of the given instance of the event based on game content originating from a first game application; a second one or more detection models trained to output at least one signal associated with the occurrence of the given instance of the event based on game content originating from a second game application; A method comprising:

5. 5. The method of claim 4, the first game application is a previously released version of the second game application; the first game application and the second game application are associated with different publishers; or The first game application and the second game application are associated with game applications of different genres. method.

6. 2. The method of claim 1 , applying a second event recognizer to the video portion of the game content to detect instances of a second event, the second event recognizer including second one or more detection models trained to output one or more signals associated with an occurrence of a given instance of the second event based on the video portion of the game content for a corresponding game session; identifying a second in-game enhancement associated with the second event; implementing the second in-game enhancement at the client system in conjunction with the presentation of the game session at the client system and in conjunction with the in-game enhancement associated with the instance of the event based on the second event recognizer detecting the instance of the second event; The method further comprising:

7. 2. The method of claim 1, wherein the in-game enhancements include a graphical user interface (GUI) enhancement associated with displaying one or more graphic icons on a display of the game session on a graphical user interface (GUI) of the client system.

8. 2. The method of claim 1, wherein the in-game augmentations include input device augmentations associated with haptic or visual feedback presented via an input device of the client system.

9. 2. The method of claim 1 , wherein implementing the in-game augmentations at the client system includes receiving augmented video content from the game server that includes both the video portion of game content and one or more augmentations to the game content that are presented via a graphical user interface (GUI) of the client system.

10. 2. The method of claim 1, wherein the step of implementing the in-game enhancements at the client system comprises: receiving one or more extension commands from a data analysis service on the cloud computing system; executing the one or more extended commands locally on the hardware of the client system; A method comprising:

11. 2. The method of claim 1, wherein the one or more detection models include machine learning models that are applied to video frames of the gaming content at the same time the gaming content is provided to the client system.

12. 1. A system implemented by one or more server devices on a cloud computing system, the system comprising: one or more processors; memory in electronic communication with the one or more processors; instructions stored in said memory; the instructions include, by the one or more processors: obtaining a video portion of game content of a game session, the video portion being obtained from the game content as the game content is provided to a client system from a game server on the cloud computing system; applying a first event recognizer to a video portion of the game content to detect instances of an event, the first event recognizer including one or more detection models trained to output one or more signals associated with an occurrence of a given instance of the event based on the video portion of the game content for a corresponding game session, the first event recognizer recognizing the event based on content of one or more video frames of the video portion; identifying an in-game enhancement associated with the event; implementing the in-game enhancements at the client system in conjunction with presenting the game session at the client system based on the first event recognizer detecting the instance of the event; The system is operable to:

13. 13. The system of claim 12, wherein the game content includes video game content streamed to the client system by the game server, and the video game content corresponding to the video portion of the game content is obtained when the game content is provided to the client system.

14. 14. The system of claim 13, wherein the game content includes one or more of audio content and controller input, and the one or more detection models are trained to output the one or more signals associated with the occurrence of the given instance of the event based on a combination of two or more of the video game content, the audio content, and the controller input.

15. 13. The system of claim 12, wherein the in-game enhancement comprises: a graphical user interface (GUI) extension associated with displaying one or more graphical icons for a display of the game session on a GUI of the client system; an input device extension associated with haptic or visual feedback presented via an input device of the client system; A system comprising:

16. 13. The system of claim 12, wherein implementing the in-game augmentations at the client system includes receiving augmented video content from the game server that includes both the video portion of game content and one or more augmentations to the game content that are presented via a graphical user interface (GUI) of the client system.

17. 13. The system of claim 12, wherein the step of implementing the in-game enhancements at the client system comprises: receiving one or more extension commands from a data analysis service on the cloud computing system; executing the one or more extended commands locally on the hardware of the client system; Including, the system.

18. In the computing system, obtaining a video portion of game content of a game session, the video portion being obtained from the game content as the game content is provided to a client system from a game server on a cloud computing system; applying a first event recognizer to the video portion of the game content to detect instances of an event, the first event recognizer including one or more detection models trained to output one or more signals associated with an occurrence of a given instance of the event based on the video portion of the game content for a corresponding game session, the first event recognizer recognizing the event based on content of one or more video frames of the video portion; identifying an in-game enhancement associated with the event; implementing the in-game enhancements at the client system in conjunction with presenting the game session at the client system based on the first event recognizer detecting the instance of the event; A program that performs the following.

19. 20. The program of claim 18, wherein the in-game expansion comprises: a graphical user interface (GUI) extension associated with displaying one or more graphical icons for a display of the game session on a GUI of the client system; an input device extension associated with haptic or visual feedback presented via an input device of the client system; A program comprising one or more of:

20. 20. The program according to claim 18, wherein the step of implementing the in-game extension in the client system comprises: receiving augmented video content from the game server, the augmented video content including both the video portion of game content and one or more enhancements to the game content presented via a graphical user interface (GUI) of the client system; receiving one or more extension commands from a data analysis service on the cloud computing system and executing the one or more extension commands locally on the hardware of the client system; A program comprising one or more of:

Images