Implementing graphical overlays based on the current game scene for streaming games

Abstract

A system is configured to implement a graphical overlay in a streaming game based on a current game state. Game data generated by a video game is received, including a game video in the form of a video stream containing game video frames. The game video is displayed on a display screen of a computing device to represent the video game to a user playing the video game at the computing device. At least one feature of the video game is identified in at least the game data. A user interface (UI) control configuration associated with the identified at least one feature is selected from a plurality of UI control configurations for the video game, and a graphical overlay corresponding to the selected UI control configuration is implemented on the video game in the display screen.

Description

[0001] background

[0002] Game streaming is a technology that transfers video game data from a first user device (e.g., an "originating" or "source" device) running a video game application to a second user device (e.g., a "playing" or "client" device) to enable the user to play the video game using the second user device. For example, video game data can be streamed from a desktop computer to a smartphone, tablet, or other user device with a touch screen (e.g., a "touch device") to enable the user to play the video game using the touch screen. Such video game streaming can enable a user to play a video game on a device for which the video game was not originally designed. For example, game streaming can enable a device designed to run on a video game console (e.g., Microsoft Sony Nintendo Games played using a game controller on a mobile device (such as ), can interact with and thus be played by a user on the mobile device.

[0003] In order to allow users to play video games that are not designed for touch screen devices on touch screen devices, graphical overlays can be presented on touch screen devices. Graphical overlays provide touch screen controls that users interact with to play games, thereby providing a mapping of user interactions with the touch controls of the touch device to physical controls of the source device for the video game. For example, a video game can be designed for a video game console that a user uses a handheld game controller to interact with. The video game can be streamed to a touch device (such as a smart phone). In this way, graphical overlays can be used to present touch controls for user interaction on a touch device. Touch screen controls (e.g., graphical buttons, etc.) are mapped to game controller controls (physical buttons, rods, etc.). When the graphical overlay on the touch screen presents all game controller controls as touch controls, the touch screen may become cluttered, and touch controls may be presented in a position that blocks the user from viewing the video game.

[0004] Overview

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0006] Provided are methods, systems, and computer program products for implementing a graphical overlay in a streaming game. For example, a computing device may include a display screen and a video game streaming client that receives game data streamed from a source device. The video game streaming client includes a game engine that is configured to receive game data and display a game video on a display screen to represent the video game to a user playing the video game at the computing device. The video game streaming client further includes a control configuration determiner that includes a feature identifier and a control configuration selector. The feature identifier is configured to identify at least one feature of the video game in the game data. The control configuration selector is configured to select a UI control configuration associated with at least one identified feature from a plurality of user interface (UI) control configurations for the video game. The game engine implements a graphical overlay corresponding to the selected UI control configuration on the video game in the display screen.

[0007] Further features and advantages of the embodiments and the structure and operation of the embodiments are described in detail below with reference to the accompanying drawings. It should be noted that the methods and systems are not limited to the specific embodiments described herein. These embodiments are presented herein for illustrative purposes only. Based on the teachings contained herein, additional embodiments will be apparent to those skilled in the relevant art. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings, which are incorporated herein and constitute a part of the specification, illustrate various embodiments of the present application and, together with the description, further serve to explain the principles of the various embodiments and enable those skilled in the relevant art to make and use the embodiments.

[0010] Figure 1 A block diagram of a system for implementing a graphics overlay for streaming video games is shown according to an example embodiment.

[0011] Figure 2 A flow chart for training a machine learning model for video game feature recognition is shown according to an example embodiment.

[0012] Figure 3 A flow chart for selecting and implementing a graphical overlay on a display screen based on a current game state is shown according to an example embodiment.

[0013] Figure 4 According to an example embodiment Figure 1 A block diagram of a user device of a system that is configured to select and implement a graphical overlay based on a current video game state.

[0014] Figure 5 According to an example embodiment Figure 1A block diagram of a system of a user device configured to identify a feature used to select a graphical overlay in a video game.

[0015] Figure 6 A flow chart for identifying objects in a video game is shown according to an example embodiment.

[0016] Figure 7 According to an example embodiment Figure 1 A block diagram of a system for a user device that is configured to enable tuning of user interface controls of a video game graphics overlay.

[0017] Figure 8 A flow chart for tuning controls associated with a touch screen graphical overlay is shown according to an example embodiment.

[0018] Fig. 9 is a block diagram of an example processor-based computer system that may be used to implement various embodiments.

[0019] Features and advantages of the embodiments described herein will become more apparent from the detailed description set forth below when taken in conjunction with the accompanying drawings in which the same reference numerals identify corresponding elements. In the accompanying drawings, the same reference numerals generally indicate identical, functionally similar, and / or structurally similar elements. The drawing in which an element first appears is indicated by the leftmost digit(s) in the corresponding reference numeral.

[0020] Detailed Description

[0021] introduction

[0022] This specification and the accompanying drawings disclose one or more embodiments that incorporate the features of the disclosed embodiments. The scope of the embodiments is not limited to the aspects disclosed herein. The disclosed embodiments are merely illustrative of the intended scope, and modified versions of the disclosed embodiments are also contemplated. The embodiments are defined by the appended claims.

[0023] References in the specification to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, whether or not explicitly described, it is considered to be within the knowledge of those skilled in the art to implement the feature, structure, or characteristic in conjunction with other embodiments.

[0024] In addition, it should be understood that the spatial descriptions used herein (e.g., "above," "below," "up," "left," "right," "lower," "top," "bottom," "vertical," "horizontal," etc.) are for illustrative purposes only and that actual implementations of the structures described herein can be spatially arranged in any orientation or manner.

[0025] In the discussion, unless otherwise stated, adjectives such as "substantially" and "approximately" that modify conditions or relationship characteristics of one or more features of an embodiment of the present disclosure should be understood to mean that the condition or characteristic is limited to a tolerance of the operation of the embodiment that is acceptable for the intended application of the embodiment.

[0026] A number of exemplary embodiments are described below. It should be noted that any section / subsection titles provided herein are not intended to be limiting. Various embodiments are described in this document, and any type of embodiment may be included under any section / subsection. In addition, the various embodiments disclosed in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0027] Example Embodiments

[0028] In order to allow users to play video games that are not designed for touch screen devices on touch screen devices, graphical overlays can be presented on touch screen devices. The graphical overlay provides touch screen controls that users interact with to play video games, thereby mapping user interactions with the touch controls of the touch device to physical controls of the source device for the video game. For example, a video game can be designed for a video game console that users interact with using a handheld game controller. The video game can be streamed to a touch device (such as a smart phone). The graphical overlay can be presented on the touch device to map game controller controls (physical buttons, rods, etc.) to touch screen controls (e.g., graphical buttons, etc.) for user touch interaction.

[0029] Such graphic overlays are conventionally configured in several ways. For example, a two-digit configuration can be used when a user applies both thumbs to controls on a touch screen. Touch input is mapped to gestures that use only these two fingers. However, video games designed to be used with a game controller, mouse, and / or keyboard may require the user to manipulate many physical controls concurrently. Accordingly, a two-digit configuration on a touch screen may make some video games unplayable because the user cannot provide all the input signals required for the video game.

[0030] In another conventional implementation, a one-size-fits-all graphical overlay may be presented that includes controls for every control of a game controller on the touch screen. However, because game controllers typically include many different types of controls at various locations, such a control layout may present some controls that the user does not need to use and occupy most, if not all, of the entire display screen, which may make some video games unplayable because the user cannot see the actual video game behind the control layout. Accordingly, a "one-size-fits-all" configuration is generally impractical from a user experience perspective.

[0031] In an attempt to address these issues, the control layout of a graphical overlay may be customized for a particular game scene. However, many video games switch control layouts based on what is happening in the video game (i.e., a "game state" or "game scene"). For example, different sets of controls may be used to operate a video game's menu, to make a character in the game walk around, drive a car, fly, etc. Accordingly, it is often impractical to use a set of controls customized for one game scene, because changes in the game scene result in controls that do not exactly match the desired controls.

[0032] Various embodiments overcome these and other problems associated with graphics overlays in video game streaming. In one embodiment, machine learning can be used to determine the current game scene and cause a switch from a previous graphics overlay to a more appropriate graphics overlay designed for the determined current game scene. In an alternative embodiment, a specific identified pixel arrangement and / or sound of a video game can identify the current game scene and cause a switch from a previous graphics overlay to a more appropriate graphics overlay for the determined current game scene.

[0033] In various embodiments, the game data generated by the video game at the first computing device may include a game video in the form of a video stream of video frames. The game data may be streamed to a second computing device. The game video is displayed on a display screen of the second computing device to represent the video game to the user playing the video game. During the game, game-related data (e.g., game video frames, game audio data, input event streams, hardware usage metrics during the video game, other metadata generated during the video game, such as log file content, API (application programming interface) access, etc.) may be analyzed to identify one or more game-related features corresponding to the current game scene (which may have just changed from a previous game scene, or may be in the process of changing). In response to the determination, a UI control configuration associated with the identified feature (s) is selected from a plurality of user interface (UI) control configurations for the video game. A graphic overlay corresponding to the selected UI control configuration is implemented on the video game in the display screen. The graphic overlay is configured for the specific current game scene, so that the graphic overlay has a specific selected control set, control position, etc. configured to make the gameplay more efficient and pleasing to the user in the game scene. The user is allowed to interact with the graphical overlay to play the video game within the current game scene.

[0034] Thus, embodiments enable providing appropriate controls for specific gaming scenarios based on identified game-related features. In this way, the user experience for a video game streamed to a touch device closely approximates the user experience for a video game played on the source device for which the video game was designed.

[0035] In one embodiment, a trained machine learning model may be generated and used to enable an appropriate graphical overlay to be selected for a game scene. In one embodiment, a trained machine learning model may receive game-related data to identify game features, which are used to generate a confidence score indicating the confidence level that the feature(s) have actually been identified and that the game has been switched to the associated game scene.

[0036] Such a trained learning model can be generated in various ways. For example, to generate such a model, a video game can be trained in a machine learning (ML) application such as TensorFlow. TM ) to generate training game data including a training video stream. Training instructions corresponding to the video game can be input into the ML application during training gameplay. In one embodiment, the training instructions are manually input (e.g., manually input by a game developer). Alternatively, the training instructions can be provided automatically, such as by a computer. A trained machine learning model is generated based on the training instructions and the generated training game data.

[0037] In some instances (such as with a racing game), the graphical overlay may require additional modifications to accurately simulate the original input device (such as a game controller). Embodiments enable control of the tuning of the implemented graphical overlay based on user input. For example, a game input response curve associated with a physical game controller input device can be linearized. The linearized game input response curve can be tuned for the touch control of the graphical overlay so that the tuned input response curve is associated with the touch control. The tuned input response curve determines how the game input responds to the user input during game play. In this way, the user experience can be improved for the touch control.

[0038] Embodiments for generating and utilizing a graphical overlay for a streamed video game may be implemented in various ways. For example, Figure 1 1 is a block diagram of a system 100 for implementing a graphics overlay for streaming video games according to an example embodiment. Figure 1 As shown, system 100 includes a first computing device 102 (training device) included in a training phase 126, and a second computing device 136 (game source / originating device) and a third computing device 104 (game playing / client device) included in a live game playing phase 128. Computing device 102 includes a video game application 106 and a machine learning (ML) application 108. Computing device 136 includes a streaming service 138 that streams a source video game application 140. Computing device 104 includes a display screen 112 and a video game streaming client 114. Video game streaming client 114 includes a video game event recognition machine language (VGERML) model 110, a control configuration determiner 118, a game engine 120, and a control configuration library 122. The following is further described in detail. Figure 1 .

[0039] Computing device 102 and computing device 136 may each include any type of mobile or fixed computing device, such as a desktop computer, a server, a video game console, etc. Computing device 104 may be any type of mobile computing device (e.g., devices, personal digital assistants (PDAs), laptops, notebooks, tablet computers (such as the Apple iPad TM ), netbooks, etc.), mobile phones (e.g., cellular phones, smartphones (such as Microsoft Phone, Apple iPhone, Realization Android TM operating system, etc.), wearable computing devices (e.g., head-mounted devices including smart glasses (such as Glass TM 、Oculus VR Ltd.'s Oculus etc.), stationary computing devices (such as desktop computers or PCs (personal computers)), game consoles / systems (e.g., Microsoft Sony Nintendo or etc.

[0040] The training phase 126 is used to generate a machine learning model that is used during the live game play phase 128 to identify features in the game-related data that are used to select graphical overlays for display. Figure 1 As shown, the video game application 106 is a video game program (e.g., implemented with program code executed by one or more processors). The video game application 106 can be any type of video game, including cause-and-effect games, serious games, educational games, shooting games, driving games, etc. During execution, the video game application 106 generates training game data 148, which includes data representing the video game during game play. The training game data 148 is game data of the video game generated during training and is presented to the user playing the game. For example, the training game data 148 may include video data for display on a display screen, audio data for playback by one or more speakers, and / or other types of data generated during the training phase. During the training phase 126, the video game application 106 also receives user input data 134 from one or more user interface devices used by the user (such as a game controller, a touch screen, a keyboard, etc.). The user input data 134 indicates the actions taken by the user when playing the video game during the training phase (e.g., pressing a button, moving a stick to the right, etc.). The user input data 134 is processed by the video game application 134 to determine how the video game is played and is thus used in the generation of training game data 148 that is presented to the user.

[0041] The ML application 108 is configured to receive and process the training game data 148 and the training instructions 146 to generate a video game event recognition machine language (VGERML) model 110. For example, in one embodiment, the ML application 108 can implement a supervised machine learning algorithm based on the actual game data of the training game data 148 and the training input provided in the form of the training instructions 146 to generate the VGERML model 110. The VGERML model 110 is a machine learning model generated to identify features in game-related data during the live game play session 128. The identified features are used to select a graphical overlay for display and interaction by a user of a game consumption device different from the source device (computing device 102). As mentioned above, the training instructions 146 can be manually input or input by a computer. In various embodiments, the training instructions 146 can indicate the location of objects displayed in the game video frames of the training game data, can indicate the timing of sounds in the audio frames of the training game data, and / or can indicate other game-related aspects.

[0042] For example, objects such as weapons, tools, characters, vehicles, parts thereof (e.g., eyes, headlights, taillights, license plates, etc.), and other objects may be displayed by the video game during game play. Objects such as these or other objects determined to be important to a particular game scene may be flagged (or automatically flagged) by the game developer user during training gameplay for model training, such as by indicating their location in the game video (e.g., by the user indicating the location of the object by a point, by drawing a box around the object, etc.). Additionally or alternatively, the training indications 146 may include indications of the timing of sounds in the audio of the training game data. For example, a user may indicate the timing of the sound of a car engine starting, the sound of a weapon being used (e.g., a chainsaw running, a gun being fired), the sound of a particular character speaking, and the like that are considered relevant to a particular game scene in the video game.

[0043] In further embodiments, the training instructions 146 may include a stream of input events. For example, a user performing model training may indicate specific input events associated with a specific game scene, such as one or more input events corresponding to a user selecting a car from a list or screen, one or more input events corresponding to a user selecting a specific weapon, and the like. In still other embodiments, the training instructions 146 may include instructions for specific hardware usage. For example, the indicated hardware usage may include instructions for: certain processor utilization levels, memory usage levels, disk accesses, etc. corresponding to specific game activities, as well as log file contents, API accesses, etc., specific objects rendered to the screen (such as cars, trucks, ships, helicopters, characters, etc.) associated with specific game scenes.

[0044] The ML application 108 generates a VGERML model 110 based on the training game data 148 and the training instructions 146. The ML application 108 can use any suitable technique to generate the VGERML 110, including supervised ML model generation algorithms (such as supervised vector machines (SVM)), linear regression, logistic regression, naive Bayes, linear discriminant analysis, decision trees, k-nearest neighbor algorithms, neural networks, etc. In one embodiment, the generated VGERML model 110 can provide a confidence level indicating whether a feature is identified in the game data. If the confidence level is sufficient (e.g., greater than 50%), a graphical overlay corresponding to the game scene in which the feature (or features) is identified can be selected.

[0045] In particular, upon the ML application 108 generating the VGERML model 110, the training phase 126 is complete and the play live game phase 128 may begin. To enable the play live game phase 128, the VGERML model 110 is included in a video game streaming client 114 of the computing device 104. The video game streaming client 114 is a client-based application used by a game player at the computing device 104 to play a streamed instance of a video game during the play live game phase 128. The play live game phase 128 is described in further detail below.

[0046] As mentioned above, computing device 104 can be used by a user to play video games that are not designed to be played on computing device 104. Video games can be streamed from a computing device, such as computing device 136. For example, a source video game application 140 can be executed on computing device 136 (e.g., a desktop computer, a game console, etc.). Source video game application 140 is an instance of a video game application 106 that can be executed by computing device 136 to enable a game player to play a video game. In addition, computing device 136 includes a streaming service 138 configured to stream game data 144 of source video game application 140 to another device, such as computing device 104. In particular, streaming service 138 is configured to transmit game data 144 via a wireless and / or wired network, which can include one or more network cables, a local area network (LAN) (such as a wireless LAN (WLAN) or "Wi-Fi"), and / or a wide area network (WAN) (such as the Internet).

[0047] In one embodiment, a streaming service 138 at computing device 136 and a video game streaming client 114 at computing device 104 work together to present a video game of a source video game application 140 executing at computing device 136 to a user at computing device 104. In particular, streaming service 138 streams game data 144 (e.g., video and audio data) generated by execution of source video game application 140 to video game streaming client 114, which presents the video game to the user at computing device 140. In response, video game streaming client 114 streams user input data 142 received from a user interacting with the video game at computing device 104 to streaming service 138 to be provided as user input events to source video game application 140. In this way, while source video game application 140 is executing at a first computing device (computing device 136), the user is able to play the video game at a second computing device (computing device 140), even though the video game is not designed to be played on the second computing device. Examples of server-client streaming services into which embodiments may be incorporated include those provided by Microsoft Those server-client streaming services provided by the platform, and those provided by Valve Platform, etc.

[0048] In further example details, such as Figure 1 As shown, the game engine 120 of the video game streaming client 114 receives game data 144 from the source video game application 140 streamed by the streaming service 138. In one embodiment, the video game streaming client 114 is configured to present the video game to the user of the computing device 104, including presenting the video and audio of the video game at the computing device 104, and receiving user input events provided by the user at the computing device 104. For example, in various embodiments, the game engine 120 may display video of the video data in the game data 144 on the display screen 112, broadcast the audio of the audio data in the game data 144 through the speakers of the computing device 104 (if present), and receive input signals from input controls presented to the user of the computing device 104. The user input data 142 is received by the game engine 120 from one or more user interfaces at the computing device 104 (including the display screen 112, which may be a touch screen). The user input data 142 indicates actions taken by the user when playing the video game during the live game play phase 128 (e.g., touching displayed controls on the display screen 112). As shown in FIG. Figure 1As shown, the game engine 120 may transmit input signals from the input controls to the source video game application 140 as user input data 142. The user input data 142 is processed by the source video game application 140 to determine subsequent video game execution and is thus used in generating game data 144 presented to the user.

[0049] In one embodiment, game data 144 includes game video in the form of a video stream containing game video frames. Display screen 112 is configured to display the game video to represent the video game to a user playing the video game at the computing device. For example, and as Figure 1 As shown, display screen 112 receives and displays video game video 116 at computing device 104. In addition, display screen 112 may display a graphics overlay 124 on video game video 116. Graphics overlay 124 includes an arrangement of one or more graphical controls (referred to herein as a user interface (UI) control configuration) that a game player may interact with on display screen 112 during game play.

[0050] The control configuration determiner 118 is configured to select a UI control configuration based on the current game scene of the video game. To help identify the game scene, the control configuration determiner 118 is configured to identify one or more features in the game data 144. As mentioned above, the feature may be present in game video frames, game audio data, input event streams provided to the video game, hardware usage of the computing device, log files, APIs, etc.

[0051] For example, and if Figure 1 As shown, the control configuration determiner 118 can access the VGERML 110 to identify the feature 130. Figure 1As shown, the VGERML model 110 receives game data 144 as input, and may additionally receive input events 150 from a display screen 150 and hardware / machine usage data 152 from a task manager or other hardware manager of the computing device 104. The input events 150 include an indication of a stream of user input events (UI control interactions) provided by a user at the display screen 112. The input events 150 are also received by the game engine 120 to be transmitted by the source video game application 140 in user input data 142, as described elsewhere herein. The hardware usage data 152 includes an indication of the hardware of the computing device 104 during the play of the video game, including processor utilization levels, memory usage levels, disk accesses, log file contents / accesses, API accesses, etc. corresponding to specific game activities. Based on the game data 144, and optionally based on the input events 150 and / or the hardware usage data 152, the VGERML model 110 identifies features and associated confidence scores 130. The features and associated confidence scores 130 may be used by the configuration determiner 118 to determine a game scenario.

[0052] In one embodiment, if the confidence score has a predetermined relationship with a threshold, the feature is determined to be present. For example, the confidence score can have any suitable range, including 0.0 (low confidence) to 1.0 (high confidence). The confidence score can be compared to a predetermined threshold (such as 0.5, 0.7, etc.). If the confidence score is greater than the threshold, the control configuration determiner 118 indicates the feature as being identified, and is configured to interface with the control configuration library 122 to select a UI control configuration indicating the identified feature. For example, and if Figure 1 As shown, control configuration determiner 118 may select a UI control configuration from control configuration library 122. Control configuration library 122 includes a plurality of control configurations with associated graphical overlays that may be rendered during play of a video game. Figure 1 In the example of , in response to the identified characteristics, control configuration determiner 118 selects control configuration 132 that corresponds to graphical overlay 124 .

[0053] Alternatively, the control configuration determiner 118 can access the feature-to-scene graph to identify the game scene based on the identified features. In particular, a special search can be performed to find one or more features in the feature-to-scene graph. If one or more features (e.g., a specific sound, one or more specific pixels with a specific attribute (such as a specific color)) are found in the game data, the identified (all) features are directly mapped to the corresponding current game scene by the feature-to-scene graph. For example, in the example video game, when the character is maimed, one or more specific screen pixels in the game data 144 can be changed to black. These black pixels can be features of a specific game scene mapped to the feature-to-scene graph. Game data 144 can be applied to the feature-to-scene graph, and these pixels can cause corresponding changes in the game scene indicated in the figure when blackened (e.g., by selecting / mapping to the corresponding UI control configuration).

[0054] Once a UI control configuration is selected, the control configuration determiner 118 is configured to implement the associated graphical overlay 124 as an overlay to the video game. Figure 1 As shown, control configuration determiner 118 selects control configuration 132 and provides an indication of the selected control configuration 132 to game engine 120. In response, game engine 120 is configured to display graphical overlay 124 of control configuration 132 as an overlay on video game video 116 in display screen 112.

[0055] For example, if the tail lights of a car are identified in the video data as features with a high confidence value (e.g., 0.8), the identification of the tail lights can identify the presence of the car and thus can indicate to the game player that the car has been selected to be driven in the video game. As such, a control configuration having a graphical overlay for driving the car can be selected from the control configuration library 122 based on the identified tail light features. The graphical overlay can include one or more graphical controls for steering the car, accelerating the car, braking the car, etc. Upon identification of the tail lights, the graphical overlay can be displayed on the display screen 112, and the game player can interact with the graphical overlay to drive the car.

[0056] In various embodiments, system 100 may operate in various ways to perform its functions. For example, Figure 2 A flowchart 200 for generating a trained machine learning model for video game event recognition is shown according to an example embodiment. In one embodiment, the flowchart 200 may be executed by the computing device 102. For the purpose of explanation, continue to refer to Figure 1 To describe Figure 2200. Note that flowchart 200 relates to identifying displayed objects in game video data, but in other embodiments, sounds may be identified in game audio data, input events may be identified in user input data, and machine usage information may be identified in machine usage data in a similar manner. Any such data may be used as training data for generating a machine learning model.

[0057] Figure 2 Flowchart 200 of FIG. 1 begins with step 202. In step 202, a video game is executed to generate training game data including a training video stream. For example, referring to FIG. Figure 1 , the video game application 106 may be executed in the computing device 102 to generate training game data 148. As described above, the training game data 148 is game data of the video game generated during training and presented to the user playing the game. During the training phase 126, the video game application 106 also receives user input data 134 from one or more user interface devices used by the user (such as a game controller, a touch screen, a keyboard, etc.). The user input data 134 is received by the video game application 106 during execution and is used to generate other instances of training game data 148 presented to the user.

[0058] For example, a game player may interact with the controls to fire an in-game weapon. The game player's interaction with the controls is received in user input data 134. The video game application 106 executes the video game to incorporate the weapon firing and any effects thereof, which may be output in training game data 148.

[0059] In step 204, training instructions for objects displayed in game video frames of a training video stream are received. Figure 1 , the ML application 108 receives training indications of objects displayed in the game video frames of the training game data. For example, the ML application 108 may receive an indication of a car's taillights, an indication of a car's license plate, and / or other indication of the rear of a car as a training indication. Such training indications are provided as features associated with the rear end of a car to train the machine learning model to recognize the rear end of a car, which may be displayed on a display screen of a video game when a user's character is about to enter the car to drive it. In such a situation, the user may desire the computing device to automatically display a graphical overlay including controls for driving the car to the display screen.

[0060] Such training instructions may be provided in any suitable manner. For example, a game developer may indicate the screen location / area of ​​an object and an identifier of the object as a training instruction. For example, a data pair may be provided as a training instruction. A game developer may draw a rectangle around the taillights of a car displayed on a display screen in a video frame as the location of an example of the taillights of the car, and may identify the rectangle as including the taillights and / or tail end of the car. As described above, the tail end of the car may be a desired feature to be identified in the game video as indicating that the game player has selected the car to drive in the video game.

[0061] In step 206, the training video stream and the training instructions are applied to the machine learning algorithm to generate a trained machine learning model. Figure 1 , the ML application 108 applies the training video stream and the training instructions to generate a VGERML model 110. Following the example above, the VGERML model 110 may be a trained model for a car-related video game, and thus may be trained to be able to determine that a character is about to drive a car, etc. (e.g., identify weapons, vehicles, characters, etc.). The machine learning algorithm may receive many various training instructions associated with identifying the rear end of a car for learning how to recognize the rear end of a car during play of the video game.

[0062] It should be noted that the VGERML model 110 can be generated in various forms. According to one embodiment, the ML application 108 can generate the VGERML model 110 according to a suitable supervised machine learning algorithm mentioned elsewhere herein or otherwise known. For example, the ML application 108 can implement a gradient boosting tree algorithm or other decision tree algorithm to generate and / or train a VGERML model 110 in the form of a decision tree. The decision tree can be traversed with input data (video data, audio data, input events, machine usage data, etc.) to identify features. Alternatively, the application 108 can implement an artificial neural network learning algorithm to generate the VGERML model 110 as a neural network of interconnected groups of artificial neurons. Input data can be presented to the neural network to identify features.

[0063] As mentioned above, the VGERML model 110 is included in the video game streaming client 114 for selecting and implementing graphical overlays based on the current game scene. Any number of such configured video game streaming clients 114 can be implemented in corresponding user devices to implement various embodiments in these user devices. The video game streaming client 114 can operate in various ways to perform this function. For example, Figure 3 300 is shown for selecting and implementing a graphical overlay on a display screen based on a current game state according to an example embodiment. For purposes of illustration, continue to refer to Figure 1 And refer to Figure 4Flowchart 300 will be described. Figure 4 A method for selecting and implementing a graphics overlay based on a current video game state according to an example embodiment is shown. Figure 1 The relevant part of the computing device 104.

[0064] Figure 4 The video game streaming client 114 and the display screen 112 are as described above. Figure 1 Flowchart 300 may be executed by video game streaming client 114. Figure 4 As shown, the video game streaming client 114 includes a control configuration determiner 118, a game engine 120, and a storage 404. The game engine 120 includes an optional game video modifier 410. The control configuration determiner 118 includes a feature identifier 406 (which includes the VGERML model 110) and a control configuration selector 408. The storage includes Figure 1 The control configuration library 122. Based on the following discussion of the flowchart 300, other structural and operational embodiments will be apparent to those skilled in the relevant art.

[0065] Flowchart 300 begins with step 302. In step 302, game data generated by a video game is received, the game data including a game video in the form of a video stream containing game video frames. Figure 1 , the game engine 120 receives game data 144 including video game video 116 generated by the video game. Figure 1 As shown, game data 144 may be streamed from source video game application 140 to video game streaming client 114 via streaming service 138 .

[0066] In step 304, the game video is displayed on a display screen to represent the video game to a user playing the video game at the computing device. Figure 1 , video game video 116 is extracted from game data 144 and provided to display screen 112 by game engine 120. Display screen 112 displays video game video 116 to a user playing the video game at computing device 104.

[0067] In step 306, at least one feature of the video game is identified in at least the game data. Figure 3, feature identifier 406 can be configured to identify one or more features of a video game in the game data. Feature identifier 406 identifies features of a video game by using VGERML model 110. As noted above, game data 144 may include a stream of game video frames and game audio data. Further game-related data may include user input events 150 and hardware usage data 152. In various embodiments, VGERML model 110 may receive any one or more of the following to identify features of a video game: video data and / or audio data of game data 144, user input events 150, and / or hardware usage data 152. As described above, VGERML 110 may generate an indication of a feature and an associated confidence value indicating the likelihood that the indicated feature exists. The identified feature may be any video game feature, including a visual object or a sound. In addition, multiple features may be identified simultaneously, or a sequence of features may be identified.

[0068] For example, the VGERML model 110 may generate the features 130 to indicate a first feature indicating that the music changes to a specific track with a confidence value of 0.9, and a second feature indicating that the screen dims with a confidence value of 0.95.

[0069] Alternatively, and as described above, Figure 4 The feature identifier 406 can be configured to directly identify one or more features of the video game in the game data, such as a specific pixel arrangement (e.g., a specific pixel with a specific attribute (such as color)) and / or a sound included in the feature-to-scene graph. For example, a specific color and / or a specific sound of one or more specific pixels can be mapped to a specific game scene by the feature-to-scene graph, resulting in a change in the game scene (e.g., from a game character walking to a character riding in a car). In such a case, instead of using the VGERML model 110, the feature-to-scene graph can be used by the control configuration determiner 118. The feature-to-scene graph maps specific game data items (e.g., attributes of specific pixels, sounds, etc.) to corresponding UI control configurations for associated game scenes. The feature-to-scene graph can be constructed manually (e.g., by a game developer), or can be configured in other ways in various embodiments.

[0070] In step 308, a user interface (UI) control configuration associated with the identified at least one feature is selected from a plurality of UI control configurations for the video game, each UI control configuration defining a corresponding graphical overlay for the video game that is configured to be interacted with in a corresponding live game scene of the video game. Figure 4, the control configuration selector 408 may select a UI control configuration 132 from the control configuration library 122 based on the feature 130. For example, the UI control configuration 132 may be associated with the feature 130 in the library 122, such that when the feature 130 is identified (with an acceptably high confidence), the UI control configuration 132 is selected. In one embodiment, the UI control configuration 132 defines / includes a graphical overlay 124. The graphical overlay 124 is configured for gameplay in a particular scene of a video game.

[0071] It should be noted that in some embodiments, a single identified feature can be used by the control configuration to select a UI control configuration. For example, if the taillights are the identified feature, a UI control configuration for controlling a car can be selected. In other embodiments, multiple identified features can be used by the control configuration to select a UI control configuration. For example, it may be necessary to identify multiple car-related features (e.g., taillights, license plate, rear windshield) in order to select a UI control configuration for controlling a car.

[0072] In step 310, a graphical overlay corresponding to the selected UI control configuration is implemented on the video game in the display screen. Figure 3 , the game engine 120 may access (via the control configuration selector 408) the graphical overlay 124 associated with the selected UI control configuration 132. The game engine 120 displays the graphical overlay 124 as an overlay to the video game video 116 in the display screen 112.

[0073] Accordingly, while the graphical overlay 124 is displayed in the display screen 112, the game player may interact with the displayed controls of the graphical overlay 124. The graphical overlay 124 corresponds to the current game scene, which corresponds to the presence of one or more features identified in the game-related data as described above.

[0074] It is noted that, in one embodiment, a video game modifier 410 may be present to modify the video data stream received from the video game and displayed as the video game video 116. For example, in one embodiment, the video game modifier 410 is configured to render an image of a frame (or frames) to be displayed as the video game video 116. The video game modifier 410 may composite the image into the received video frame to create a modified video frame in the video game video 116. The video game modifier 410 may alternatively generate an entirely new video frame that is inserted into the video stream for display as the video game video 116, and / or replaces a video frame therein.

[0075] As described above, an object displayed during play of a video game may indicate a change in the game scene and thus may be a trigger for changing the game overlay. Figure 6Flowchart 600 for identifying objects in a video game according to an example embodiment is shown. Flowchart 600 may be in the form of flowchart 300 ( Figure 3 ) is performed during step 306. For the purpose of explanation, continue to refer to Figure 4 And refer to Figure 5 600. Note that flowchart 600 is described with respect to identifying displayed objects in game video data, but in other embodiments, sounds may be identified as features in game audio data. Additionally, input events in user input data and machine usage data may be used to identify features. Embodiments are directed to any such data in any combination for identifying features of a video game.

[0076] Figure 5 A method for identifying features for selecting a graphical overlay in a video game is shown according to an example embodiment. Figure 1 600 is a block diagram of relevant portions of computing device 104. Flowchart 600 may be performed by feature identifier 306. Other structural and operational embodiments will be apparent to those skilled in the relevant art(s) based on the following discussion of flowchart 600. Flowchart 600 is described as follows.

[0077] Flowchart 600 begins with step 602. In step 602, a predetermined screen area of ​​a display screen is analyzed to find an image of an object. Figure 5 , the VGERML model 110 of the feature identifier 302 may analyze all display regions (including the predetermined screen region 518 (shown on the display screen 112)) in ( Figure 1 The VGERML model 110 may be configured to focus on the predetermined screen area 518 because it is where objects of interest are known to appear during game play in association with changes in the game scene. For example, the predetermined screen area 518 may have been used to train the VGERML model 110 (e.g., Figure 2 200) is indicated as the position of the object.

[0078] For example, Figure 5 As shown, the predetermined screen area 518 may be a predetermined location where the taillights of a car appear when the user controls their in-game character to select and enter a car to drive. The taillights identified in the predetermined screen area 518 are highly relevant to the game of entering a car driving scene. Thus, the VGERML model 110 may identify the taillights of the car in area 518.

[0079] In step 604, a confidence score associated with the object is determined, which indicates the probability that the image of the object is contained in the predetermined screen area. Figure 5, the VGERML model 110 determines a confidence score associated with the object, which indicates the probability that the image of the object is contained in a predetermined screen area.

[0080] For example, Figure 5 As shown, feature identifier 306 outputs feature 508 generated by VGERML model 110. Feature 508 may indicate that a car taillight is identified and an associated generated confidence score. Control configuration selector 408 receives feature 508, and if the confidence score is high enough (e.g., greater than a predetermined threshold), control configuration selector 408 selects a UI control configuration associated with the identified feature from library 122. Figure 5 In the example of FIG. 4 , the control configuration selector 408 selects the car UI control configuration 502 associated with the identified car taillights.

[0081] The control configuration selector 408 provides the car UI overlay 510 associated with the car UI control configuration 502 to the game engine 120. The game engine 120 is configured to display the car UI overlay 510 on the video game video 116 in the display screen 112. In this way, the user can interact with the controls of the car UI overlay 510 to drive the car through his in-game character. The car UI overlay 510 is customized for driving a car in a video game, rather than a generic UI overlay, thereby improving the user's gaming experience.

[0082] As mentioned above, in order to accommodate certain video games, various embodiments enable tuning of the input responses provided by the controls of the graphical overlay. Game developers can create dead zones and custom response curves for UI controls during the process of targeting a specific target input device. For example, the target input device can be a keyboard, a mouse, or a game controller. Accordingly, when game streaming is implemented, the input response of the control may be incorrect because the new input control (e.g., a touch control) at the client computing device is not the originally intended input control (which can be a stick on a game controller). Thus, as described below, the system provides tuning of the input response of the UI controls in the client device.

[0083] For example, a two-dimensional control (such as a thumb stick) has a two-dimensional tuning curve (turning curve) and may have geometric shapes such as radial, elliptical, or square. If a video game is tuned for a game controller, the horizontal axis on the thumb stick may pivot from -1 to +1 to turn the car left or right. Game developers may program the area in the middle pivot area of ​​the thumb stick (e.g., -.25 to +.25) as a dead zone, where no turning occurs when the thumb stick is positioned there, which prevents the user from feeling too anxious about turning. In addition, at the end of the response curve, a more extreme response may be programmed to achieve a sharp turn when the user makes an extreme move with the thumb stick. However, when used on a different device (such as a touch screen), the user will have a bad experience because the touch control will not necessarily perform well with the thumb stick response curve.

[0084] Thus, in one embodiment, control tuning can be used to flatten the game tuning, with a uniform response of the control being 1 to 1 for in-game responses. After flattening, the control can be tuned for client controls (such as a touch screen). Such tuning can include, for example, inserting one or more dead zones with corresponding ranges, clipping regions of the curve, changing the curvature of the response, etc.

[0085] Accordingly, in various embodiments, the control configuration determiner 118 may be configured to enable control tuning. For example, the control configuration determiner 118 may be configured to enable control tuning based on the control configuration. Figure 8 to operate. Figure 8 A flowchart 800 for tuning controls associated with a touch screen graphical overlay is shown according to an example embodiment. For purposes of illustration, continue to refer to Figure 4 and Figure 5 And refer to Figure 7 Flowchart 800 is described. Figure 7 An example embodiment of a user interface control for tuning a video game graphics overlay is shown. Figure 1 104 is a block diagram of relevant portions of the computing device 104.

[0086] Flowchart 800 may be executed by game input response tuner 720. Other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the following discussion of flowchart 800. Flowchart 800 is described as follows.

[0087] Flowchart 800 begins with step 802. In step 802, a game input response curve associated with a physical game controller input device is linearized. Figure 7, the game input response tuner 720 linearizes the control input response curve 702 associated with the control of the automobile UI control configuration 706. For example, the control input response curve 702 may have originally been associated with a stick or button of a game controller, and thus may not be linear as described above. The game input response tuner 720 is configured to flatten the control input response curve 702 to have a 1:1 (unbent) input response.

[0088] In step 804, the linearized game input response curve is tuned for the touch input of the graphics overlay. Figure 7 , the game input response tuner 720 enables a game developer or other user to input tuning information for a control input response curve 702 of a graphical overlay. In various embodiments, the tuning information may be in the form of touch input adjustments at a display screen 112 that displays a car graphical overlay 710 associated with a car UI control configuration 706. The game developer may be enabled to adjust the tuning of the control input response curve 702 in any manner, including by interacting with one or more tuning controls displayed at the display screen 112 by the game input response tuner 720. The game input response tuner 720 enables the control input response curve 702 to be tuned from the flattened version (of step 802) to have any desired curvature and to have one or more dead zones if desired.

[0089] In step 806, the tuned game input response curve is associated with the touch input of the graphical overlay. Figure 7 As shown, the game input response tuner 720 associates the control input response curve 702 with the associated control of the car graphic overlay 710 defined in the car UI control configuration 702 .

[0090] Example Computer System Implementation

[0091] Computing device 102, computing device 104, video game application 106, ML application 108, VGERML model 110, display screen 112, video game streaming client 114, video game video 116, control configuration determiner 118, game engine 120, control configuration library 122, game input response tuner 720, storage 404, feature identifier 406, control configuration selector 408, game video modifier 410, flowchart 200, flowchart 300, flowchart 600 and flowchart 800 may be implemented in hardware or hardware in combination with one or both of software and / or firmware. For example, computing device 102, computing device 104, video game application 106, ML application 108, VGERML model 110, display screen 112, video game streaming client 114, video game video 116, control configuration determiner 118, game engine 120, control configuration library 122, game input response tuner 720, storage 404, feature identifier 406, control configuration selector 408, game video modifier 410, flowchart 200, flowchart 300, flowchart 600, and flowchart 800 may be implemented as computer program code / instructions configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, computing device 102, computing device 104, video game application 106, ML application 108, VGERML model 110, display screen 112, video game streaming client 114, video game video 116, control configuration determiner 118, game engine 120, control configuration library 122, game input response tuner 720, storage 404, feature identifier 406, control configuration selector 408, game video modifier 410, flowchart 200, flowchart 300, flowchart 600 and flowchart 800 may be implemented as hardware logic / circuitry.

[0092] For example, in one embodiment, one or more of any combination of computing device 102, computing device 104, video game application 106, ML application 108, VGERML model 110, display screen 112, video game streaming client 114, video game video 116, control configuration determiner 118, game engine 120, control configuration library 122, game input response tuner 720, storage 404, feature identifier 406, control configuration selector 408, game video modifier 410, flowchart 200, flowchart 300, flowchart 600, and flowchart 800 may be implemented together in a SoC. The SoC may include an integrated circuit chip that includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more of one or more communication interfaces, and / or other circuits, and may optionally execute received program code and / or include embedded firmware to perform functions.

[0093] Fig. 9 An exemplary implementation of a computing device 900 in which embodiments may be implemented is depicted. For example, computing device 102 and computing device 104 may each be implemented in one or more computing devices similar to computing device 900 in a fixed or mobile computer embodiment, including one or more features and / or alternative features of computing device 900. The description of computing device 900 provided herein is for illustration only and is not intended to be limiting. Embodiments may also be implemented in other types of computer systems known to those skilled in the relevant art.

[0094] like Fig. 9 As shown, the computing device 900 includes one or more processors (referred to as processor circuits 902), a system memory 904, and a bus 906 that couples various system components including the system memory 904 to the processor circuit 902. The processor circuit 902 is an electronic and / or optical circuit implemented as a central processing unit (CPU), a microcontroller, a microprocessor, and / or other physical hardware processor circuits using one or more physical hardware electronic circuit device elements and / or integrated circuit devices (semiconductor material chips or dies). The processor circuit 902 can execute program codes stored in a computer-readable medium, such as operating systems 930, application programs 932, program codes of other programs 934, etc. The bus 906 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or a local bus using any of various bus architectures. The system memory 904 includes a read-only memory (ROM) 908 and a random access memory (RAM) 910. The basic input / output system 912 (BIOS) is stored in the ROM 908.

[0095] The computing device 900 also has one or more of the following drives: a hard disk drive 914 for reading and writing to a hard disk, a disk drive 916 for reading and writing to a removable disk 918, and an optical drive 920 for reading and writing to a removable optical disk 922 (such as a CD-ROM, DVD or other optical media). The hard disk drive 914, the disk drive 916, and the optical drive 920 are connected to the bus 906 via a hard disk drive interface 924, a disk drive interface 926, and an optical drive interface 928, respectively. These drives and their associated computer-readable media provide non-volatile storage of computer-readable instructions, data structures, program modules and other data for the computer. Although a hard disk, a removable disk and a removable optical disk are described, other types of hardware-based computer-readable storage media such as flash memory cards, digital video disks, RAM, ROM and other hardware storage media may also be used to store data.

[0096] Several program modules may be stored on a hard disk, disk, optical disk, ROM or RAM. These programs include an operating system 930, one or more application programs 932, other programs 934, and program data 936. Application programs 932 or other programs 934 may include, for example, computer program logic (e.g., computer program code or instructions) for implementing the following: computing device 102, computing device 104, video game application 106, ML application 108, VGERML model 110, display screen 112, video game streaming client 114, video game video 116, control configuration determiner 118, game engine 120, control configuration library 122, game input response tuner 720, storage 404, feature identifier 406, control configuration selector 408, game video modifier 410, flowchart 200, flowchart 300, flowchart 600, and flowchart 800, and / or other embodiments described herein.

[0097] A user may enter commands and information into the computing device 900 through input devices such as a keyboard 938 and a pointing device 940. Other input devices (not shown) may include a microphone, a joystick, a game controller, a satellite dish, a scanner, a touch screen and / or touch pad, a voice recognition system for receiving voice input, a gesture recognition system for receiving gesture input, and the like. These and other input devices are typically connected to the processor circuit 902 through a serial port interface 942 coupled to the bus 906, but may be connected through other interfaces such as a parallel port, a game port, or a universal serial bus (USB).

[0098] A display screen 944 is also connected to the bus 906 via an interface, such as a video adapter 946. The display screen 944 may be external to or incorporated into the computing device 900. The display screen 944 may display information, as well as serve as a user interface for receiving user commands and / or other information (e.g., by touch, finger gestures, a virtual keyboard, etc.). In addition to the display screen 944, the computing device 900 may also include other peripheral output devices (not shown), such as speakers and printers.

[0099] The computing device 900 is connected to a network 948 (e.g., the Internet) through an adapter or network interface 950, a modem 952, or other means for establishing communications over the network. The modem 952, which may be internal or external, may be connected to the bus 906 via a serial port interface 942, such as a serial bus. Fig. 9 As shown, or another interface type including a parallel interface may be used to connect to bus 906.

[0100] As used herein, the terms "computer program medium", "computer readable medium" and "computer readable storage medium" are used to refer to physical hardware media, such as a hard disk associated with hard disk drive 914, a removable disk 918, a removable optical disk 922, other physical hardware media (such as RAM, ROM), flash memory cards, digital video disks, zip disks, MEMs, nano-based storage devices, and other types of physical / tangible hardware storage media. Such computer readable storage media are distinguished from and do not overlap with communication media (excluding communication media). Communication media carry computer readable instructions, data structures, program modules or other data in modulated data signals such as carrier waves. The term "modulated data signal" means a signal in which one or more characteristics are set or changed in a manner that information is encoded in the signal. As an example and not limitation, communication media include wireless media (such as acoustic, RF, infrared and other wireless media) and wired media. Each embodiment also relates to such communication media that are separate and do not overlap with the embodiments for computer readable storage media.

[0101] As noted above, computer programs and modules (including application programs 932 and other programs 934) may be stored on a hard disk, disk, optical disk, ROM, RAM, or other hardware storage medium. Such computer programs may also be received via a network interface 950, a serial port interface 942, or any other interface type. These computer programs, when executed or loaded by an application, enable the computer 900 to implement the features of the various embodiments discussed herein. Accordingly, such computer programs represent controllers of the computer system 900.

[0102] The embodiments also relate to computer program products including computer codes or instructions stored on any computer readable medium. Such computer program products include hard drives, optical drives, memory device packages, portable memory sticks, memory cards, and other types of physical storage hardware.

[0103] Additional Example Embodiments

[0104] A method is described herein in a computing device having a display screen. The method includes: receiving game data generated by a video game, the game data including a game video in the form of a video stream containing game video frames; displaying the game video on the display screen to represent the video game to a user playing the video game at the computing device; identifying at least one feature of the video game in the game data; selecting a user interface (UI) control configuration associated with the identified at least one feature from a plurality of UI control configurations for the video game, each UI control configuration defining a corresponding graphical overlay for the video game, the corresponding graphical overlay being configured to be interacted with in a corresponding live game scene of the video game; and implementing a graphical overlay corresponding to the selected UI control configuration on the video game in the display screen.

[0105] In one embodiment of the foregoing method, the identifying includes identifying a characteristic of the video game in at least one of: a game video frame of game data, game audio data of the game data, an input event stream provided to the video game, or hardware usage of a computing device.

[0106] In another embodiment of the foregoing method, at least one feature includes an object, and the identification further includes: analyzing a predetermined screen area of ​​the display screen to find an image of the object; and determining a confidence score associated with the object, the confidence score indicating a probability that the image of the object is contained in the predetermined screen area.

[0107] In yet another embodiment of the foregoing method, the analyzing includes applying a portion of a game video frame of a video stream containing an image of the object to a trained machine learning model to generate a confidence score.

[0108] In another embodiment of the foregoing method, the method further includes: executing a video game to generate training game data including a training video stream; receiving training instructions for objects displayed in game video frames of the training game data including the training video stream; and applying the training game data including the training video stream and the training instructions to a machine learning algorithm to generate a trained machine learning model.

[0109] In another embodiment of the foregoing method, the display screen is a touch screen, and the method further includes: tuning controls associated with a selected UI control configuration corresponding to a graphical overlay for the touch screen, the tuning including: linearizing a game input response curve associated with a physical game controller input device, tuning the linearized game input response curve for touch input of the graphical overlay, and associating the tuned game input response curve with the touch input of the graphical overlay.

[0110] In yet another embodiment of the foregoing method, the game data includes a game video frame including images rendered by a computing device and composited into the game video frame.

[0111] A system in a computing device is described herein. The system includes: a video game streaming client, including: a game engine, which is configured to: receive game data generated by a video game, the game data including a game video in the form of a video stream containing game video frames; and display the game video on a display screen to represent the video game to a user playing the video game at the computing device; and a control configuration determiner, including: a feature identifier, which is configured to: identify at least one feature of the video game in at least the game data; and a control configuration selector, which is configured to: select a user interface (UI) control configuration associated with the identified at least one feature from a plurality of UI control configurations for the video game, each UI control configuration defining a corresponding graphic overlay to the video game, the corresponding graphic overlay being configured to interact with it in a corresponding live game scene of the video game; and implement a graphic overlay corresponding to the selected UI control configuration on the video game in the display screen.

[0112] In one embodiment of the foregoing system, the feature identifier is further configured to: identify features of the video game in at least one of: game video frames of game data, game audio data of the game data, an input event stream provided to the video game, or hardware usage of a computing device.

[0113] In another embodiment of the foregoing system, the at least one feature includes an object, and the feature identifier is further configured to: analyze a predetermined screen area of ​​the display screen to find an image of the object; and determine a confidence score associated with the object, the confidence score indicating a probability that the image of the object is contained in the predetermined screen area.

[0114] In yet another embodiment of the foregoing system, the feature identifier is configured to apply a portion of a game video frame of a video stream containing an image of the object to a trained machine learning model to generate a confidence score.

[0115] In another embodiment of the foregoing system, the display screen is a touch screen, and the video game streaming client further includes: a game input response tuner configured to tune controls associated with a selected UI control configuration corresponding to a graphical overlay for the touch screen, the game input response tuner configured to: linearize a game input response curve associated with a physical game controller input device, tune the linearized game input response curve for touch input to the graphical overlay, and associate the tuned game input response curve with touch input to the graphical overlay.

[0116] In yet another embodiment of the foregoing system, the game data includes game video frames, and the game engine includes a game video modifier configured to: render the image; and composite the image into the game video frame.

[0117] A computer-readable medium having computer program logic recorded thereon, which, when executed by at least one processor, causes the at least one processor to perform a method, the method comprising: receiving game data generated by a video game, the game data comprising a game video in the form of a video stream containing game video frames; displaying the game video on a display screen to represent the video game to a user playing the video game at a computing device; identifying at least one feature of the video game in at least the game data; selecting a user interface (UI) control configuration associated with the at least one identified feature from a plurality of UI control configurations for the video game, each UI control configuration defining a corresponding graphical overlay for the video game, the corresponding graphical overlay being configured to be interacted with in a corresponding live game scene of the video game; and implementing a graphical overlay corresponding to the selected UI control configuration on the video game in the display screen.

[0118] In one embodiment of the foregoing computer-readable medium, the identifying includes identifying a characteristic of the video game in at least one of: a game video frame of game data, game audio data of the game data, an input event stream provided to the video game, or hardware usage of a computing device.

[0119] In another embodiment of the foregoing computer-readable medium, at least one feature includes an object, and the identification further includes: analyzing a predetermined screen area of ​​a display screen to find an image of the object; and receiving a confidence score associated with the object, the confidence score indicating a probability that the image of the object is contained in the predetermined screen area.

[0120] In another embodiment of the foregoing computer-readable medium, the analyzing includes applying a portion of a game video frame of a video stream containing an image of the object to a trained machine learning model to generate a confidence score.

[0121] In another embodiment of the foregoing computer-readable medium, the method further includes: executing a video game to generate a training video stream; receiving training instructions for objects displayed in game video frames of the training video stream; and applying the training video stream and the training instructions to a machine learning algorithm to generate a trained machine learning model.

[0122] In another embodiment of the aforementioned computer-readable medium, the display screen is a touch screen, and the method further includes: tuning controls associated with a selected UI control configuration corresponding to a graphical overlay for the touch screen, the tuning including: linearizing a game input response curve associated with a physical game controller input device, tuning the linearized game input response curve for touch input of the graphical overlay, and associating the tuned game input response curve with the touch input of the graphical overlay.

[0123] In another embodiment of the foregoing computer-readable medium, the game data includes a game video frame including images rendered by a computing device and composited into the game video frame.

[0124] Conclusion

[0125] Although various embodiments of the present disclosure have been described above, it should be understood that they are presented as examples only and not as limitations. Those skilled in the relevant art will understand that various changes can be made in form and detail without departing from the spirit and scope of the application as defined in the appended claims. Accordingly, the breadth and scope of the present invention should not be limited by any of the exemplary embodiments described above, but should only be limited according to the following claims and their equivalents.

Claims

1. A method in a computing device having a display screen, comprising: receiving game data generated by a video game, the game data comprising game video in the form of a video stream including game video frames; displaying the game video on the display screen to represent the video game to a user playing the video game at the computing device; identifying at least one feature of the video game in at least the game data; selecting a user interface (UI) control configuration associated with the identified at least one feature from a plurality of UI control configurations for the video game, each of the UI control configurations defining a corresponding graphical overlay for the video game that is configured to be interacted with in a corresponding live game scene of the video game, wherein each graphical overlay is configured to correspond to a live game scene of the video game and each graphical overlay is configured to be interactable by a user to enable the user to play the video game in the corresponding live game scene; The graphical overlay corresponding to the selected UI control configuration is implemented on the video game in the display screen.

2. The method of claim 1, wherein: The identification includes: Identify a feature of the video game in at least one of the following: a game video frame of the game data, game audio data of the game data, a stream of input events provided to said video game, or The hardware usage of the computing device.

3. The method of claim 1, wherein: The at least one feature comprises an object, and the identifying further comprises: analyzing a predetermined screen area of ​​the display screen to find an image of the object; and A confidence score associated with the object is determined, the confidence score indicating a probability that an image of the object is contained within the predetermined screen area.

4. The method of claim 3, wherein: The analysis included: A portion of a game video frame of the video stream that includes an image of the object is applied to the trained machine learning model to generate the confidence score.

5. The method of claim 4, further comprising: executing the video game to generate training game data including a training video stream; receiving a training instruction for an object displayed in a game video frame of the training video stream; as well as Applying the training video stream and the training instructions to a machine learning algorithm to generate the trained machine learning model.

6. The method of claim 1, wherein: The display screen is a touch screen, and the method further comprises: Tuning controls associated with a selected UI control configuration corresponding to the graphical overlay for the touch screen, the tuning comprising: Linearizes the game input response curve associated with a physical game controller input device, tuning a linearized game input response curve for touch input of the graphical overlay, and A tuned game input response curve is associated with a touch input of the graphical overlay.

7. The method of claim 1, wherein: The game data includes game video frames including images rendered by the computing device and composited into the game video frames.

8. A system implemented in a computing device, comprising: A video game streaming client, the video game streaming client comprising: A game engine, wherein the game engine is configured to: receiving game data generated by a video game, the game data comprising game video in the form of a video stream including game video frames; and displaying the game video on a display screen to represent the video game to a user playing the video game at the computing device; and A control configuration determiner, the control configuration determiner comprising: A feature identifier, the feature identifier being configured to: identifying at least one feature of the video game in at least the game data; and A control configuration selector, wherein the control configuration selector is configured to: selecting a user interface (UI) control configuration associated with the identified at least one feature from a plurality of UI control configurations for the video game, each of the UI control configurations defining a corresponding graphical overlay for the video game that is configured to be interacted with in a corresponding live game scene of the video game, wherein each graphical overlay is configured to correspond to a live game scene of the video game and each graphical overlay is configured to be interactable by a user to enable the user to play the video game in the corresponding live game scene; and The graphical overlay corresponding to the selected UI control configuration is implemented on the video game in the display screen.

9. The system of claim 8, wherein: The feature identifier is further configured to: Identify a feature of the video game in at least one of the following: a game video frame of the game data, game audio data of the game data, a stream of input events provided to said video game, or The hardware usage of the computing device.

10. The system of claim 8, wherein: The at least one feature comprises an object, and the feature identifier is further configured to: analyzing a predetermined screen area of ​​the display screen to find an image of the object; and A confidence score associated with the object is determined, the confidence score indicating a probability that an image of the object is contained within the predetermined screen area.

11. The system of claim 10, wherein: The feature identifier is configured to: A portion of a game video frame of the video stream that includes an image of the object is applied to the trained machine learning model to generate the confidence score.

12. The system of claim 8, wherein: The display screen is a touch screen, and the video game streaming client further comprises: a game input response tuner configured to tune controls associated with a selected UI control configuration corresponding to the graphical overlay for the touch screen, including being configured to: linearize a game input response curve associated with a physical game controller input device, tune the linearized game input response curve for touch input to the graphical overlay, and A tuned game input response curve is associated with a touch input of the graphical overlay.

13. The system of claim 8, wherein: The game data includes game video frames, and the game engine includes a game video modifier, the game video modifier being configured to: Rendering images; as well as The image is composited into the game video frame.

14. A computer readable medium having program code recorded thereon, the program code, when executed by at least one processor, causing the at least one processor to perform a method comprising: receiving game data generated by a video game, the game data comprising game video in the form of a video stream including game video frames; displaying the game video on a display screen to represent the video game to a user playing the video game at a computing device; identifying at least one feature of the video game in at least the game data; selecting a user interface (UI) control configuration associated with the identified at least one feature from a plurality of UI control configurations for the video game, each of the UI control configurations defining a corresponding graphical overlay for the video game, the corresponding graphical overlay being configured to be interacted with in a corresponding live game scene of the video game, wherein each graphical overlay is configured to correspond to a live game scene of the video game and each graphical overlay is configured to be interactable by a user to enable the user to play the video game in the corresponding live game scene; as well as The graphical overlay corresponding to the selected UI control configuration is implemented on the video game in the display screen.

15. The computer-readable medium of claim 14, wherein: The identification includes: Identify a feature of the video game in at least one of the following: a game video frame of the game data, game audio data of the game data, a stream of input events provided to said video game, or The hardware usage of the computing device.

16. The computer readable medium of claim 14, wherein: The at least one feature comprises an object, and the identifying further comprises: analyzing a predetermined screen area of ​​the display screen to find an image of the object; and A confidence score associated with the object is received, the confidence score indicating a probability that an image of the object is contained within the predetermined screen area.

17. The computer-readable medium of claim 16, wherein: The analysis included: A portion of a game video frame of the video stream that includes an image of the object is applied to the trained machine learning model to generate the confidence score.

18. The computer readable medium of claim 17, wherein: The method further comprises: executing the video game to generate training game data including a training video stream; receiving a training indication of an object displayed in a game video frame of the training video stream; and Applying the training video stream and the training instructions to a machine learning algorithm to generate the trained machine learning model.

19. The computer-readable medium of claim 14, wherein: The display screen is a touch screen, and the method further comprises: Tuning controls associated with a selected UI control configuration corresponding to the graphical overlay for the touch screen, the tuning comprising: Linearizes the game input response curve associated with a physical game controller input device, A linearized game input response curve is tuned for touch input to the graphical overlay and the tuned game input response curve is associated with the touch input to the graphical overlay.

20. The computer readable medium of claim 14, wherein: The game data includes game video frames including images rendered by the computing device and composited into the game video frames.

21. A computer system comprising means for executing the method according to any one of claims 1 to 7.

Images