Performance prediction for virtualized game application

Abstract

To provide a method for providing predicted performance of a game application when executed in a virtualized environment of a computing device.SOLUTION: A computing system receives an indication of device characteristics of a computing device to determine a predicted performance score for a game application; uses a game performance model trained using machine learning; determines, on the basis of the device characteristics and application characteristics of the game application, a predicted performance of the game application when executed in a virtualized environment of the computing device; and transmits an indication of the predicted performance to the computing device.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] Computing devices may differ in many respects, such as central processing unit (CPU) performance, graphics processing unit (GPU) performance, display size and / or resolution, memory performance, etc. Due to such differences in device configurations, gaming applications executing on various computing devices may perform differently across different computing devices. For example, a gaming application may output image data at different frame rates while running on different computing devices. Summary of the Invention

[0002] In general, the techniques of this disclosure are directed to determining a predicted performance of a gaming application when executed on a computing device. The predicted performance is indicative of how well the gaming application is likely to perform when executed on the computing device. A computing system may determine a predicted performance of the gaming application when executed in a virtualized environment of the computing device based at least in part on one or more device characteristics of the computing device and one or more application characteristics of the gaming application using a gaming performance model trained using machine learning to determine a predicted performance score for the gaming application, and may transmit to the computing device an indication of the predicted performance of the gaming application when executed in the virtualized environment of the computing device.

[0003] The computing device may determine whether to recommend and / or recommend the gaming application for download and installation on the computing device based on the predicted performance of the gaming application when executed in the virtualized environment of the computing device. For example, if the predicted performance of the gaming application when executed in the virtualized environment of the computing device indicates that the gaming application may not perform satisfactorily when executed in the virtualized environment of the computing device, the computing device may refrain from recommending or recommending the gaming application for download and installation on the computing device.

[0004] The computing device may also adjust one or more graphical parameters of the game application based on the predicted performance of the game application when executed in the virtualized environment of the computing device. For example, if the predicted performance of the game application when executed in the virtualized environment of the computing device indicates that the game application may not perform adequately when executed in the virtualized environment of the computing device, the computing device may reduce the graphical quality of image data rendered and output by the game application to improve the performance of the game application when executed in the virtualized environment of the computing device.

[0005] In some aspects, the techniques described herein relate to a method that includes receiving, by one or more processors of a computing system, an indication of one or more device characteristics of the computing device from the computing device; determining, by the one or more processors, a predicted performance of the gaming application when executed in a virtualized environment of the computing device based at least in part on the one or more device characteristics of the computing device and one or more application characteristics of the gaming application using a game performance model trained using machine learning to determine a predicted performance score for the gaming application; and transmitting, by the one or more processors, an indication of the predicted performance of the gaming application when executed in the virtualized environment of the computing device to the computing device.

[0006] In some aspects, the techniques described herein relate to a computing system that includes a memory; a network interface; and one or more processors operatively connected to the memory and the network interface, the one or more processors configured to: receive from the computing device via the network interface an indication of one or more device characteristics of the computing device; determine a predicted performance score for the gaming application based at least in part on the one or more device characteristics of the computing device and one or more application characteristics of the gaming application using a game performance model trained using machine learning to determine a predicted performance score for the gaming application when executing in a virtualized environment of the computing device; and transmit to the computing device via the network interface an indication of the predicted performance of the gaming application when executing in the virtualized environment of the computing device.

[0007] In some aspects, the techniques described herein relate to a computer-readable storage medium storing instructions that, when executed, cause one or more processors of a computing system to: receive from the computing device an indication of one or more device characteristics of the computing device; determine a predicted performance of the gaming application when executing in a virtualized environment of the computing device based at least in part on the one or more device characteristics of the computing device and one or more application characteristics of the gaming application using a game performance model trained using machine learning to determine a predicted performance score for the gaming application; and transmit to the computing device an indication of the predicted performance of the gaming application when executing in the virtualized environment of the computing device.

[0008] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the disclosure will be apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example environment in which an example computing system is configured to determine a predicted performance of a gaming application when executed on an example computing device, in accordance with one or more aspects of the present disclosure. [Diagram 2] FIG. 1 is a block diagram illustrating an exemplary computing system in accordance with one or more aspects of the present disclosure. [Figure 3A] FIG. 1 is a conceptual diagram illustrating aspects of an example machine learning model in accordance with an embodiment of the present disclosure. [Figure 3B] FIG. 1 is a conceptual diagram illustrating aspects of an example machine learning model in accordance with an embodiment of the present disclosure. [Figure 3C] FIG. 1 is a conceptual diagram illustrating aspects of an example machine learning model in accordance with an embodiment of the present disclosure. [Figure 3D] FIG. 1 is a conceptual diagram illustrating aspects of an example machine learning model in accordance with an embodiment of the present disclosure. [Figure 3E] FIG. 1 is a conceptual diagram illustrating aspects of an example machine learning model according to an embodiment of the present disclosure. [Figure 4] 1 is a flowchart illustrating an example of an operational mode of a computing system for determining a predicted performance of a gaming application when executed on a computing device in accordance with one or more techniques of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] 1 is a conceptual diagram illustrating an example environment in which an example computing system is configured to determine a predicted performance of a gaming application when executed on an example computing device, in accordance with one or more aspects of the present disclosure. In the example of FIG. 1, environment 100 may include computing device 102, computing devices 120A-120N (hereinafter, "computing devices 120"), and a computing system 150 that communicates over network 130 to determine a predicted performance of gaming application 112 when executed in a virtualized environment of computing device 102.

[0011] Computing system 150 may be any suitable remote computing system, such as one or more desktop computers, laptop computers, mainframes, servers, cloud computing systems, virtual machines, etc., that can transmit and receive information over network 130. In some examples, computing system 150 may represent a cloud computing system that provides one or more services over network 130. That is, in some examples, computing system 150 may be a distributed computing system. One or more computing devices, such as computing device 102 and / or computing device 120, may access services provided by the cloud by communicating with computing system 150. Although described herein as being performed at least in part by computing system 150, any or all of the techniques of the present disclosure may be performed by one or more other devices, such as either computing device 102 or computing device 120. That is, in some examples, computing device 102 and / or one or more computing devices 120 may be operable to perform one or more techniques of the present disclosure alone.

[0012] Computing system 150 may include a game performance module 162 and an application distribution module 172. Game performance module 162 and application distribution module 172 may each reside within computing system 150 or one or more other remote computing devices and perform the operations described herein using software, hardware, firmware, or a combination of both hardware, software, and firmware executed thereon. In some examples, game performance module 162 and application distribution module 172 may each be implemented as hardware, software, and / or a combination of hardware and software. Computing system 150 may execute game performance module 162 and application distribution module 172 using one or more processors. Computing system 150 may execute game performance module 162 and / or application distribution module 172 as or within a virtual machine executing on underlying hardware. Game performance module 162 and application distribution module 172 may be implemented in a variety of ways. For example, game performance module 162 and / or application distribution module 172 may be implemented as a downloadable or pre-installed application, i.e., an "app." In another example, game performance module 162 and / or application distribution module 172 may be implemented as part of an operating system of computing system 150. Other examples of computing systems 150 implementing techniques of the present disclosure may include additional components not shown in FIG.

[0013] Network 130 may be any suitable network that enables communication between computing devices 102, 120, and / or 150. Network 130 may include a wide area network such as the Internet, a local area network (LAN), a personal area network (PAN) (e.g., Bluetooth), an enterprise network, a wireless network, a cellular network, a telephone network, a metropolitan area network (e.g., WIFI, WAN, WiMAX, etc.), one or more other types of networks, or a combination of two or more different types of networks (e.g., a combination of a cellular network and the Internet).

[0014] Computing device 102 and each computing device 120 may include portable or mobile devices, such as, but not limited to, mobile phones (including smartphones), laptop computers, tablet computers, wearable computing devices such as smart watches or computerized eyewear, smart television platforms, cameras, computerized appliances, vehicle head units, etc. In some examples, computing device 102 may include fixed computing devices, such as desktop computers, servers, mainframes, etc.

[0015] Each of the computing devices 120 may include a gaming application 112 and an emulator module 114. The gaming application 112 and the emulator module 114 may perform the operations described herein using software, hardware, firmware, or a combination of both hardware, software, and firmware that resides and executes within each of the computing devices 120 or one or more other remote computing devices. In some examples, the gaming application 112 and the emulator module 114 may be implemented as hardware, software, and / or a combination of hardware and software. Each of the computing devices 120 may execute the gaming application 112 and the emulator module 114 using one or more processors. The gaming application 112 and the emulator module 114 may be implemented in a variety of ways. For example, any of the gaming application 112 and / or the emulator module 114 may be implemented as a downloadable or pre-installed application, i.e., an "app." In another example, any of the gaming application 112 and / or the emulator module 114 may be implemented as part of the operating system of each of the computing devices 120. Other examples of computing devices 120 implementing the techniques of this disclosure may include additional components not shown in FIG.

[0016] The computing device 102 may include a user interface component 104 ("UIC 104"), a user interface module 106 ("UI module 106"), an emulator module 114, and an application catalog module 116. The UI module 106, the emulator module 114, and the application catalog module 116 may perform the operations described herein using software, hardware, firmware, or a combination of both hardware, software, and firmware that resides and executes within the computing device 102 or one or more other remote computing devices. In some examples, the UI module 106, the emulator module 114, and the application catalog module 116 may be implemented as hardware, software, and / or a combination of hardware and software. The computing device 102 may execute the UI module 106, the emulator module 114, and the application catalog module 116 using one or more processors. The UI module 106, the emulator module 114, and the application catalog module 116 may be implemented in a variety of ways. For example, any of the modules 106, emulator module 114, and / or application catalog module 116 may be implemented as a downloadable or pre-installed application, or "app." In another example, any of the UI module 106, emulator module 114, and application catalog module 116 may be implemented as part of the operating system of the computing device 102. Other examples of computing devices 102 implementing the techniques of this disclosure may include additional components not shown in FIG.

[0017] The UIC 104 of the computing device 102 may function as an input device and as an output device for the computing device 102. For example, the UIC 104 may function as an input device using a resistive touchscreen, a surface acoustic wave touchscreen, a capacitive touchscreen, a projected capacitive touchscreen, a pressure-sensitive screen, an acoustic pulse recognition touchscreen, or other presence-sensing screen technology. The UIC 104 may function as an output device using any one or more of a liquid crystal display (LCD), a dot matrix display, a light emitting diode (LED) display, a micro LED, an organic light emitting diode (OLED) display, an e-ink, or similar monochrome or color display capable of outputting visible information to a user of the computing device 102.

[0018] In some examples, the UIC 104 may include a presence-sensitive screen that may receive tactile user input from a user of the computing device 102. The UIC 104 may receive tactile user input by detecting one or more taps and / or gestures from a user of the computing device 102 (e.g., a user touching or pointing at one or more locations on the UIC 104 with a finger or a stylus). The presence-sensitive screen of the UIC 104 may present output to the user. The UIC 104 may present output as a user interface that may be related to functionality provided by the computing device 102. For example, the UIC 104 may present various functions and applications running on the computing device 102, such as an electronic message application, a messaging application, a map application, and the like.

[0019] The UI module 106 may interpret input detected on the UIC 104 (e.g., when a user provides one or more gestures at the location of the UIC 104 where a user interface is displayed). The UI module 106 may relay information about the input detected on the UIC 104 to one or more associated platforms, operating systems, applications, and / or services executing on the computing device 102 to cause the computing device 102 to perform functions. The UI module 106 may also receive information and instructions from one or more associated platforms, operating systems, applications, and / or services executing on the computing device 102 (e.g., gaming application 112) to generate a graphical user interface (GUI). Additionally, the UI module 106 may act as an intermediary between one or more associated platforms, operating systems, applications, and / or services executing on the computing device 102 and various output devices (e.g., speakers, LED indicators, vibrators, etc.) of the computing device 102 to generate output (e.g., graphical, auditory, haptic, etc.) on the computing device 102.

[0020] The emulator module 114 may run on each of the computing devices 102 and 120 to provide a virtualization environment that allows applications not designed to run on the computing device 102 or computing device 120 to run in the virtualization environment provided by the emulator module 114 on the computing device 102 or computing device 120. For example, if each of the computing devices 102 and 120 is a laptop or desktop computing device running a desktop operating system, the emulator module 114 may provide a virtualization environment that allows a computing application, such as a gaming application designed to run on a mobile operating system on a smartphone, to run in the virtualization environment provided by the emulator module 114 on each of the computing devices 102 and 120 using the emulator module 114.

[0021] In some examples, emulator module 114 may convert application code (e.g., object code, byte code, etc.) written and / or compiled for a different processing architecture and / or different operating system into code that can be executed on computing device 102 and computing device 120, or may otherwise provide a virtual execution environment distinct from the execution environment of computing device 102 and computing device 120 to enable applications not designed to run on computing device 102 or computing device 120 to be executed on computing device 102 or computing device 120. In some examples, emulator module 114 may implement a virtual machine that emulates underlying hardware components of a physical computer distinct from computing device 102 and computing device 120.

[0022] The gaming applications 112 may execute in a virtualized environment provided by the emulator modules 114 of each of the computing devices 120 to perform the functions of a video game. Examples of the gaming applications 112 may include action games that emphasize hand-eye coordination and reaction time, such as first-person shooter games, simulation games, such as motorsports simulation games or flight simulator games, role-playing games (e.g., massively multiplayer role-playing games), networked multiplayer games, single player games, and the like.

[0023] Because game application 112 executes in a virtualized environment provided by emulator module 114 of a computing device, such as computing device 120A, game application 112 may output image data for display on a display device included on or operably coupled to the computing device. The image data, in some examples, may be frames of graphics that game application 112 outputs for display on the display device during execution of game application 112. For example, the image data may include frames of graphics of an interactive gameplay environment, frames of graphics of a loading screen, frames of graphics of a menu screen, etc.

[0024] The gaming application 112 may render and output image data according to a desired target frame rate, target frames per second (fps), etc. at which the gaming application 112 outputs image data. The frame rate of the image data output by the gaming application 112 may be the rate at which the gaming application 112 outputs frames of graphics. Examples of frame rates at which the gaming application 112 outputs image data may include 30 fps, 60 fps, 120 fps, 144 fps, etc.

[0025] The application catalog module 116 may execute on the computing device 102 and download and install applications on the computing device 102. For example, the application catalog module 116 may download and install applications, such as a gaming application 112 that executes in a virtualized environment provided by an emulator module 114 of the computing device 102. In some examples, the application catalog module 116 may output a graphical user interface (GUI) for display on the UIC 104 presenting a catalog of applications that may be downloaded and installed on the computing device 102, such as a list of recommended or featured applications that may be downloaded and installed on the computing device 102.

[0026] The application catalog module 116 may determine how well a gaming application, such as gaming application 112, is likely to perform when executed in the virtualized environment provided by emulator module 114 of computing device 102, and may determine whether to recommend or feature the gaming application based on the determination of how well the gaming application is likely to perform when executed in the virtualized environment of computing device 102. The application catalog module 116 may determine that a gaming application is likely to perform well when executed in the virtualized environment of computing device 102 if the gaming application is likely to be able to render and output substantially all of the frames of image data at the target frame rate of the gaming application.

[0027] The application catalog module 116 may determine gaming applications that are likely to perform well on the computing device 102 and may recommend or recommend those gaming applications for installation on the computing device 102. Similarly, the application catalog module 116 may not recommend or recommend gaming applications that are likely to not perform well on the computing device 102. The application catalog module 116 may be accessed.

[0028] Different computing devices may differ in a wide variety of ways, such as having different central processing units (CPUs) that have different amounts of processing cores and / or run at different clock speeds, different graphics processing units (GPUs), different amounts of physical memory sizes, and the like. While the application catalog module 116 may have access to performance data for gaming applications that run on some configurations of computing devices, the application catalog module 116 may not have access to performance data for gaming applications that run on a particular configuration of computing device 102.

[0029] According to aspects of the disclosure, the application catalog module 116 may communicate with the computing system 150 to query the computing system 150 for a catalog of gaming applications, such as a list of gaming applications that can be executed in the virtualization environment provided by the emulator module 114 of the computing device 102. The application catalog module 116 may include an indication of one or more device characteristics of the computing device 102 as part of the query.

[0030] The application distribution module 172 may receive a query from the application catalog module 116 and, in response, determine a list of game applications to be included in the catalog of game applications. For example, the application distribution module 172 may determine to list the game application 112 in the catalog of game applications that may be installed on the computing device 102.

[0031] The application distribution module 172 may determine, for each game application listed in the catalog provided to the application catalog module 116, a projected performance of the game application when executed on the computing device 102. In the case of game applications that execute in the virtualized environment provided by the emulator module 114, the application distribution module 172 may determine a projected performance of the game application when executed in the virtualized environment of the computing device 102.

[0032] Thus, the application distribution module 172 may query the game performance module 162 regarding the projected performance of the game application 112 when executed in the virtualized environment of the computing device 102. The query may include one or more device characteristics of the computing device 102 and one or more application characteristics of the game application 112 that the game performance module 162 may use to determine one or more application characteristics of the game application 112.

[0033] The game performance module 162 may be trained via machine learning to determine a predicted performance of the game application 112 when executed in a virtualized environment of the computing device 102 based on one or more device characteristics of the computing device 102 and one or more application characteristics of the game application 112. In some examples, the game performance module 162 may implement one or more neural networks trained by machine learning to determine a predicted performance of the game application 112 when executed on the computing device 102. In general, the one or more neural networks implemented by the game performance module 162 may include multiple interconnected nodes, each node may apply one or more functions to a set of input values ​​corresponding to one or more functions and provide one or more corresponding output values. The one or more characteristics may be one or more device characteristics of the computing device 102, and the one or more corresponding output values ​​of the one or more neural networks may be an indication of a predicted performance of the game application 112 when executed on the computing device 102.

[0034] The game performance module 162 is trained to output a predicted performance of the game application 112 when executed in the virtualized environment of the computing device 102 based on one or more application characteristics of the game application 112 and one or more device characteristics of the computing device 102, in the form of a predicted smoothness score for the game application 112. The predicted smoothness score may be a value, such as between 0 and 1, that corresponds to the percentage of frames rendered by the game application 112 that are predicted to meet a frame time of a target frame rate associated with the game application 112 in a next gameplay session of the game application 112, when executed in the virtualized environment of the computing device 102.

[0035] As discussed above, the gaming application 112 may have an associated target frame rate, such as 60 fps, at which the gaming application 112 is desired to output image data. The target frame rate may correspond to a frame time, which is the amount of time (e.g., in milliseconds) it takes for the gaming application 112 to render a frame of image data in order to render the frames of image data fast enough to meet the target frame rate. In an example where the target frame rate of the gaming application 112 is 60 fps, the frame time to render a frame of image data to meet the target frame rate of 60 fps would be 16.6 milliseconds. Thus, the predicted smoothness score may correspond to the percentage of frames of image data rendered by the gaming application 112 that are predicted to meet the 16.6 millisecond frame time associated with the target frame rate of 60 fps when the gaming application 112 is executed on the computing device 102, for a gaming application 112 having a target frame rate of 60 fps.

[0036] Thus, in some examples, the game performance module 162 may determine a predicted performance of the game application 112 when executed in the virtualized environment of the computing device 102 in the form of a predicted smoothness score. In some examples, the game performance module 162 may classify the predicted performance of the game application 112 when executed in the virtualized environment of the computing device 102 based on the predicted smoothness score. For example, the game performance module 162 may compare the predicted smoothness score to a smoothness threshold, which may be a value between 0 and 1, such as 0.80. If the game performance module 162 determines that the predicted smoothness score is equal to or greater than the smoothness threshold, the game performance module 162 may determine that the game application 112 may perform well when executed in the virtualized environment of the computing device 102. If the game performance module 162 determines that the predicted smoothness score is less than the smoothness threshold, the game performance module 162 may determine that the game application 112 may not perform well when executed in the virtualized environment of the computing device 102.

[0037] Computing system 150 may train one or more neural networks of game performance module 162 using training data generated from monitoring the performance of game application 112 executing in the virtualized environment provided by emulator module 114 on a population of computing devices, such as computing device 120, which may have a variety of different device configurations and / or device characteristics. When game application 112 executes in the virtualized environment provided by emulator module 114 of computing device 120, computing system 150 may receive performance metrics from computing device 120 associated with the execution of game application 112 on computing device 120. Such performance metrics may include, for each frame of image data rendered by game application 112, the amount of time it took game application 112 to render the frame of image data.

[0038] Computing system 150 may use the collected performance metrics to determine, for each gameplay session of game application 112 executing in the virtualized environment provided by emulator module 114 of computing device 120, a smoothness score associated with the gameplay session. The smoothness score associated with the gameplay session may be a percentage of frames of image data rendered by game application 112 during the gameplay session that meet the frame time of a target frame rate associated with game application 112, and the gameplay session may extend from when game application 112 begins executing (e.g., opening game application 112) to when game application 112 stops executing (e.g., quitting game application 112) on computing device 120.

[0039] Thus, computing system 150 may generate training data for training one or more neural networks of game performance module 162, for each game play session of multiple game play sessions on computing device 120, including a corresponding smoothness score associated with one or more device characteristics of the computing device on which the game play session occurs. Thus, computing system 150 may use the training data to train one or more neural networks of game performance module 162 to predict a performance score of game application 112 when executed in the virtualized environment of the computing device.

[0040] In response to determining the predicted performance of the gaming application 112 when executed in the virtualized environment of the computing device 102, the gaming performance module 162 may send an indication of the predicted performance of the gaming application 112 to the application distribution module 172. Accordingly, the application distribution module 172 may send the indication of the predicted performance of the gaming application 112 to the computing device 102. For example, the application distribution module 172 may send to the computing device 102 a catalog listing gaming applications available for download and installation on the computing device 102, including the gaming application 112, where the catalog may include an indication of the predicted performance of the gaming application 112.

[0041] The application catalog module 116 may receive a catalog of gaming applications from the application distribution module 172 of the computing system 150 and may output a GUI displaying the gaming applications listed in the catalog for downloading and installation on the computing device 102, such as for display on the UIC 104. Within the GUI, the application catalog module 116 may present a list of one or more recommended gaming applications and / or one or more featured applications in a visually prominent location within the GUI, such as at the top of the front page of the GUI.

[0042] The application catalog module 116 may determine whether to recommend or recommend the gaming application 112 in the GUI based on the predicted performance of the gaming application 112 when executed in the virtualized environment of the computing device 102. In an example where the predicted performance of the gaming application 112 when executed in the virtualized environment of the computing device 102 is in the form of a predicted smoothness score, the application catalog module 116 may compare the predicted smoothness score to a smoothness threshold, which may be a value between 0 and 1, such as 0.80. The application catalog module 116 may determine that the predicted smoothness score is equal to or greater than the smoothness threshold and determine that the gaming application 112 may perform well when executed in the virtualized environment of the computing device 102. If the application catalog module 116 determines that the predicted smoothness score is less than the smoothness threshold, the application catalog module 116 may determine that the gaming application 112 may not perform well when executed in the virtualized environment of the computing device 102.

[0043] In response to determining that the gaming application 112 is predicted to perform well when executed in the virtualized environment of the computing device 102, the application catalog module 116 may recommend or otherwise recommend the gaming application 112 for installation on the computing device 102, such as by including an indication of the gaming application 112 in a list of featured or recommended applications in the GUI and / or by presenting a visual indication of the gaming application 112 in a visually prominent location within the GUI, such as at the top of the front page of the GUI. Similarly, in response to determining that the gaming application 112 is predicted to not perform well when executed in the virtualized environment of the computing device 102, the application catalog module 116 may refrain from recommending or otherwise recommending the gaming application 112 for installation on the computing device 102.

[0044] In some examples, in response to determining that the gaming application 112 is predicted to perform poorly when executed in the virtualized environment of the computing device 102, the computing device 102 may adjust one or more graphics settings of the gaming application 112 installed on the computing device 102 to improve the performance of the gaming application 112 when executed in the virtualized environment of the computing device 102. For example, the computing device 102 may reduce the resolution of frames of image data output by the computing device 102, reduce the number of textures rendered by the gaming application 112, or reduce the complexity of the graphics rendered by the gaming application 112.

[0045] When the computing device 102 downloads and installs the gaming application 112, the computing system 150 may monitor the performance of the gaming application 112 while it is running in the virtualized environment of the computing device 102. For example, the computing system 150 may monitor performance metrics of the gaming application 112 while it is running in the virtualized environment of the computing device 102 to determine the actual performance of the gaming application 112, such as an actual smoothness score of the gaming application 112 while it is running in the virtualized environment of the computing device 102. The computing system 150 may retrain or fine-tune one or more neural networks of the gaming performance module 162 based on the actual performance of the gaming application 112, such as an actual smoothness score of the gaming application 112 while it is running in the virtualized environment of the computing device 102, so as to be able to more accurately determine the predicted performance of the gaming application 112 when running in the virtualized environment of the computing device.

[0046] The techniques of the present disclosure provide one or more technical advantages. An emulator that provides a virtualized environment for a gaming application may execute the gaming application on a wide variety of computing devices having operating systems and / or processing architectures that differ from the operating systems and / or processing architectures for which the gaming application was designed. Thus, it may be difficult for a computing system to collect performance data for a gaming application when executed on a wide variety of configurations of operating systems and / or processing architectures on which the gaming application is executed. By using one or more neural networks trained using machine learning to determine the predicted performance of a gaming application on a computing device, the techniques of the present disclosure may enable a computing system to more accurately predict the performance of a gaming application when executed on a computing device even if the computing system does not have performance data regarding the performance of the gaming application executed on a particular configuration of the computing device, thereby advancing the art of predicting the performance of a gaming application.

[0047] Furthermore, by using one or more neural networks trained using machine learning to determine the predicted performance of a gaming application of a computing device, the techniques of this disclosure may enable a computing device, in response to determining that the gaming application of the computing device is not predicted to perform well, to adjust one or more parameters of the gaming application, such as the graphics quality of the gaming application, to improve the performance of the gaming application when executed on the computing device, which may improve the user experience of the gaming application executed on the computing device by causing the gaming application to be perceived as smooth and responsive by the user.

[0048] Figure 2 is a block diagram illustrating an exemplary computing system according to one or more aspects of the present disclosure. Figure 2 illustrates only one particular example of a computing system 250, and many other examples of computing system 250 may be used in other examples and may include a subset of the components included in the example computing system 250 or may include additional components not shown in Figure 2. Computing system 250 may be one example of computing system 150 of Figure 1.

[0049] As shown in the example of FIG. 2, the computing system 250 includes one or more processors 240, one or more input devices 242, one or more communication units 244, one or more output devices 246, and one or more storage devices 248. The one or more processors 240 may be an example of the one or more processors 108 of FIG. 1. The one or more input devices 242 and the one or more output devices 246 may be an example of the UIC 104 of FIG. 1. The storage device 248 of the computing system 250 also includes an operating system 226 and a game performance module 262. A communication channel 252 may interconnect each of the components 240, 242, 244, 246, and 248 for inter-component communication (physical, communication, and / or operational). In some examples, the communication channel 252 may include a system bus, a network connection, one or more inter-process communication data structures, or any other component for communicating data between hardware and / or software.

[0050] One or more processors 240 may implement functions and / or execute instructions within computing system 250. For example, processor 240 on computing device 102 may receive and execute instructions stored by storage device 248 that provide the functionality of operating system 226, game performance module 262, metrics recording module 266, metrics processing module 268, training module 270, application distribution module 272, and dashboard module 278. These instructions executed by processor 240 may cause computing system 250 to store and / or modify information within storage device 48 during program execution. Processor 240 may execute instructions of the operating system, game performance module 262, metrics recording module 266, metrics processing module 268, training module 270, application distribution module 272, and dashboard module 278. That is, the operating system 226, the game performance module 262, the metrics recording module 266, the metrics processing module 268, the training module 270, the application distribution module 272, and the dashboard module 278 may operate via the processor 240 to perform the various functions described herein.

[0051] The one or more processors 240 may be or include a digital signal processor (DSP), a general purpose microprocessor, an application specific integrated circuit (ASIC), a field programmable logic array (FPGA), and / or other equivalent integrated circuit or discrete logic circuitry. One or more input devices 242 of the computing system 250 may receive input. Examples of input include, but are not limited to, haptic input, audio input, motion input, optical input, and the like. One or more output devices 246 of the computing device 102 may generate output. Examples of output include haptic output, audio output, and video output.

[0052] One or more communication units 244 of the computing system 250 may communicate with external devices by transmitting and / or receiving data. For example, the computing system 250 may use the communication unit 244 to transmit and / or receive radio signals over a wireless network, such as a cellular wireless network. In some examples, the communication unit 244 may transmit and / or receive satellite signals over a satellite network, such as a Global Positioning System (GPS) network. Examples of the communication unit 244 include a network interface card (e.g., an Ethernet card, etc.), an optical transceiver, a radio frequency transceiver, a GPS receiver, or any other type of device capable of transmitting and / or receiving information. Other examples of the communication unit 44 may include Bluetooth, GPS, 3G, 4G, Wi-Fi radios, as well as a universal serial bus (USB) controller, etc., that are included in a mobile device.

[0053] One or more storage devices 248 in computing system 250 may store information for processing during operation of computing system 250. In some examples, storage device 248 is a temporary storage device, meaning that the primary purpose of storage device 248 is not long-term storage. Storage device 248 on computing system 250 is configured to store information short-term as a volatile memory, such that stored content may not be retained upon deactivation. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art.

[0054] The storage device 248 also includes one or more computer-readable storage media in some examples. The storage device 248 may be configured to store a larger amount of information than volatile memory. The storage device 248 may also be configured to store information long term as a non-volatile memory space, where information may be retained after activation / off cycles. Examples of non-volatile memory include magnetic hard disks, optical disks, floppy disks, flash memory, or forms of electrically programmable memory (EPROM) and electrically erasable and programmable memory (EEPROM). The storage device 248 may store program instructions and / or data associated with the operating system 226, the game performance module 262, the metrics recording module 266, the metrics processing module 268, the training module 270, the application distribution module 272, and the dashboard module 278.

[0055] In accordance with the techniques of this disclosure, the computing system 250 is configured to receive an indication of one or more device characteristics of a computing device from a computing device (e.g., computing device 102 shown in FIG. 1) via one or more communication units 244. For example, the one or more processors 240 are configured to execute the application distribution module 272 to receive a query from the computing device for a catalog of gaming applications that may be installed on the computing device, where the query received from the computing device may include one or more device characteristics of the computing device.

[0056] The indication of the one or more device characteristics of the computing device may include an indication of one or more CPU characteristics of the computing device, which may include any combination of the number of CPUs of the computing device, the number of physical processing cores of the one or more CPUs, the number of logical processing cores of the one or more CPUs, the model of the one or more CPUs, and / or a clock speed of the one or more CPUs, such as an initial clock speed (e.g., a base clock speed or nominal clock speed) of the one or more CPUs and / or a maximum clock speed of the one or more CPUs.

[0057] The indication of the one or more device characteristics of the computing device may also include an indication of one or more GPU characteristics of the computing device, which may include any combination of an indication of a manufacturer (e.g., manufacturer name) and / or model of the one or more GPUs of the computing device, a video RAM (VRAM) size of the one or more GPUs, a memory bandwidth of the one or more GPUs, a current clock speed of the one or more GPUs, a maximum clock speed of the one or more GPUs, a heap size allocated to memory of the one or more GPUs, and / or a version number of a GPU driver installed on the computing device.

[0058] The indication of the one or more device characteristics of the computing device may also include an indication of one or more memory characteristics of the computing device, which may include any combination of the amount of memory of the computing device, such as a total amount of physical RAM installed on the computing device, a type of disk of the computing device, an amount of physical RAM available on the computing device, a number and / or details of physical channels of the RAM of the computing device, and / or a clock speed of the RAM of the computing device.

[0059] The indication of one or more device characteristics of the computing device, in some examples, may also include an indication of the version of the operating system of the computing device and / or the language in which the operating system is localized (e.g., English, French, etc.).

[0060] In some examples, the indication of one or more device characteristics of the computing device may include one or more display characteristics of the computing device. The one or more display characteristics of the computing device may include any combination of a display resolution and / or a display refresh rate of a display of the computing device (e.g., the UIC 104 shown in FIG. 1 ).

[0061] In an example where an emulator (e.g., emulator module 114 shown in FIG. 1 ) is used to execute a gaming application on a computing device in a virtualized environment, the indication of one or more device characteristics of the computing device may include an indication of one or more characteristics of the emulator used to execute the gaming application on the computing device. The one or more characteristics of the emulator may include any combination of a version of the emulator, a total memory size of the emulator, which may be a total amount of memory of the computing device usable and / or allocable by the emulator, and / or a usable memory size of the emulator, which may be an amount of memory of the computing device usable by the emulator.

[0062] The one or more processors 240 are configured to execute the application distribution module 272 to query the game performance module 262, for each of a plurality of game applications included in a catalog of game applications to be transmitted to the computing device, a predicted performance of the game application when executed in a virtualized environment of the computing device (e.g., a virtualized environment provided by the emulator module 114 shown in FIG. 1). For example, the application distribution module 272 may send a query to the game performance module 262 for a predicted performance of the game application 112 shown in FIG. 1 when executed in the virtualized environment of the computing device. The query may include an indication of one or more device characteristics of the computing device received from the computing device.

[0063] In some examples, the query may include an indication of one or more application characteristics of the gaming application. The indication of the one or more application characteristics of the gaming application may include an indication of a name of the gaming application, such as a software package name of the gaming application, and / or a version number of a software package associated with the gaming application. In some examples, the one or more application characteristics of the gaming application may also include any combination of graphics application programming interfaces (APIs) used by the gaming application, such as whether the gaming application uses a cross-platform graphics API (e.g., Vulkan ANGLE) that is translated to a native graphics API of the computing device or whether the gaming application uses a graphics API (e.g., Direct3D) that is native to the computing device, a game engine used by the gaming application, a frame resolution of image data output by the gaming application, and / or an operating system and / or processor architecture on which the gaming application runs.

[0064] One or more processors 240 of computing system 250 are configured to execute game performance module 262 to determine a predicted performance of the game application when executed in a virtualized environment of the computing device based on one or more device characteristics of the computing device and one or more application characteristics of the game application. That is, computing system 250 may use game performance module 262 to predict how well the game application will perform if executed on the computing device.

[0065] The predicted performance of the gaming application when executed in the virtualized environment of the computing device may be a value corresponding to how well the gaming application is likely to perform when executed in the virtualized environment of the computing device. In some examples, the predicted performance may include or correspond to frame rate information. The frame rate information, in some examples, includes a frame time, which is the amount of time (e.g., in milliseconds) that the gaming application is predicted to take to render a frame of image data when executed on the computing device.

[0066] In some examples, the one or more processors 240 are configured to execute the game performance module 262 to determine a predicted performance of the game application when executed on the computing device in the form of a predicted smoothness score. The predicted smoothness score may be a value between 0 and 1 that corresponds to a percentage of frames of image data rendered by the game application when executed on the computing device that are predicted to meet the frame time of a target frame rate associated with the game application in the next game play session. For example, a predicted smoothness score of 0.5 may indicate that 50% of frames of image data rendered by the game application when executed on the computing device are predicted to meet the frame time of a target frame rate associated with the game application in the next game play session.

[0067] As discussed above, a gaming application may have an associated target frame rate, such as 60 fps, at which the gaming application is desired to output image data. The target frame rate may correspond to a frame time, which is the amount of time (e.g., in milliseconds) it takes for the gaming application to render a frame of image data in order to render the frames of image data fast enough to meet the target frame rate. In an example where the target frame rate of a gaming application is 60 fps, the frame time to render a frame of image data to meet the target frame rate of 60 fps would be 16.6 milliseconds. Thus, for a gaming application having a target frame rate of 60 fps, the predicted smoothness score may correspond to the percentage of frames of image data rendered by the gaming application that are predicted to meet the target frame rate of 60 fps when the gaming application is executed on a computing device.

[0068] In some examples, the one or more processors 240 are configured to execute the game performance module 262 to determine a predicted performance of the game application when executed on the computing device in the form of a predicted smoothness score.

[0069] The one or more processors 240 are configured to execute the game performance module 262 and, in response to determining the predicted performance of the game application when executed on the computing device, transmit an indication of the predicted performance of the game application to the application distribution module 272. The one or more processors 240 are configured to execute the application distribution module 272 to generate a catalog of game applications including, for each game application in the catalog, an indication of the associated predicted performance of the game application when executed on the computing device. Thus, the application distribution module 272 may use the one or more communication units 244 to transmit the catalog of game applications to the computing device, thereby transmitting to the computing device an indication of the associated predicted performance of the game application when executed on the computing device.

[0070] In some examples, the game performance module 262 may include a game performance model 264 that implements one or more neural networks trained by machine learning to determine a predicted performance of the game application when executed in the virtualized environment of the computing device. In general, the one or more neural networks implemented by the game performance model 264 may include a plurality of interconnected nodes, where each node may apply one or more functions to a set of input values ​​corresponding to the one or more features and provide one or more corresponding output values. The one or more features may include an indication of one or more application characteristics of the game application and an indication of one or more device characteristics of the computing device, and the one or more corresponding output values ​​of the one or more neural networks may be an indication of a predicted performance of the game application when executed in the virtualized environment of the computing device, such as a predicted smoothness score.

[0071] The game performance model 264 may be trained to determine a predicted performance of a game application (e.g., game application 112 shown in FIG. 1 ) when executed on a computing device (e.g., computing device 102 shown in FIG. 1 ) using training data that includes data collected from a population of computing devices (e.g., computing devices 120 shown in FIG. 1 ) each executing a copy of the same game application (e.g., game application 112 shown in FIG. 1 ). The data collected from the population of computing devices may include, for each of the population of computing devices, one or more device characteristics of the corresponding computing device, such as one or more device characteristics described above.

[0072] The data collected from the population of computing devices may also include, for each of the population of computing devices, performance metrics of the game application executing on the population of computing devices. As copies of the game application execute on the population of computing devices, the one or more processors 240 may be configured to execute a metrics recording module 266 to receive performance metrics of the copies of the game application from the population of computing devices, which may store the received performance metrics in a metrics data store 274, which may be any suitable structured data store, such as a database.

[0073] The performance metrics received from the population of computing devices may include frame time metrics for frames of image data rendered and output by copies of the game application executing on the population of computing devices, such as a frame time for each frame of image data rendered and output by each copy of the game application executing on the population of computing devices. The performance metrics may also include an indication of the start of a game play session and an indication of the end of a game play session by each copy of the game application executing on the population of computing devices.

[0074] The one or more processors 240 are configured to execute metrics processing module 268 to perform processing of performance metrics stored in metrics data store 274. Metrics processing module 268 may use the indication of the start of a gameplay session and the indication of the end of a gameplay session to group the received frame time metrics by gameplay session and determine frame time metrics for each gameplay session. That is, metrics processing module 268 may determine, for each gameplay session of copies of a game application executing on a population of computing devices, frame time metrics for frames rendered by the copies of the game application during that gameplay session.

[0075] Thus, for each gameplay session of a copy of a game application executing on a population of computing devices, the metrics processing module 268 may determine a smoothness score based on the frame time metrics associated with the gameplay session. For example, the metrics processing module 268 may calculate a smoothness score for the gameplay session as a percentage of frames of image data rendered and output by a corresponding copy of the game application during the gameplay session that meet the frame time of a target frame rate associated with the game application.

[0076] The metrics processing module 268 may store, for each gameplay session of copies of the game application executed on the population of computing devices, a smoothness score associated with the gameplay session, one or more device characteristics of the computing devices on which the gameplay session occurred, and a unique gameplay session identifier in a training data store 276, which may be any suitable structured data store, such as a database or table. In this manner, the computing system 250 may generate a set of training data including an association between the device characteristics of the computing devices and the smoothness score for each gameplay session, and the one or more processors 40 may be configured to execute a training module 270 to train the game performance model 264 to predict the performance of the game application when executed on the computing devices via any suitable machine learning techniques.

[0077] In some examples, the computing system 250 may continue to collect data, such as frame time metrics and device characteristics, from a population of computing devices (e.g., computing devices 120 shown in FIG. 1 ), each running a copy of the same game application (e.g., game application 112), and the training module 270 may use the collected data to periodically retrain or fine-tune the game performance model 264 based on the collected data.

[0078] For example, as described above, the metrics processing module 268 may determine from the collected data a smoothness score associated with each of the multiple game play sessions. The game performance model 264 may also determine a predicted performance, such as a predicted smoothness score, for each of the multiple game play sessions based on one or more device characteristics associated with the game play session. Thus, the training module 270 may retrain or fine-tune the game performance model 264 using any suitable machine learning techniques to minimize, for each game session, the difference between the smoothness score associated with the game session and the predicted smoothness score for the game session.

[0079] In some examples, the one or more processors 240 are configured to execute the dashboard module 278 to display on a display device of the one or more output devices 246 or output a model accuracy dashboard to an external computing device. The model accuracy dashboard may present, for each game play session of a plurality of game play sessions of a game application, a predicted smoothness score predicted using the game performance model 264 and an actual predicted smoothness score associated with the game play session, thereby enabling a user, such as a developer of the game performance model 264, to visualize the performance of the game performance model 264.

[0080] 3A-3E are conceptual diagrams illustrating aspects of an example machine learning model, according to an embodiment of the present disclosure. Figures 3A-3E are described below in the context of the game performance model 264 of Figure 2. For example, in some cases, the machine learning model 322 may be an example of the game performance model 264.

[0081] 3A illustrates a conceptual diagram of an example of a machine learning model according to an embodiment of the present disclosure. As shown in FIG. 3A, in some implementations, the machine learning model 322 is trained to receive one or more types of input data and, in response, provide one or more types of output data. Thus, FIG. 3A illustrates the machine learning model 322 performing inference.

[0082] The input data may include one or more features associated with an example or example. In some implementations, the one or more features associated with the example or example may be organized into a feature vector. For example, the game performance model 264 may receive a feature vector including one or more features of a game application and a feature vector including one or more features of a computing device. In some implementations, the output data may include one or more predictions. A prediction may also be referred to as an estimation. Thus, given features associated with a particular example, the machine learning model 322 may output a prediction for such example based on the features. For example, given a feature vector including one or more features of a game application and a feature vector including one or more features of a computing device, the game performance model 264 may output a predicted performance of the game application when executed on the computing device.

[0083] The machine learning model 322 may be or may include one or more of a variety of different types of machine learning models. In particular, in some implementations, the machine learning model 322 may perform classification, regression, clustering, anomaly detection, recommendation generation, and / or other tasks.

[0084] In some implementations, the machine learning model 322 can perform various types of classification based on the input data. For example, the machine learning model 322 can perform two-class classification or multi-class classification. In two-class classification, the output data may include a classification of the input data into one of two different classes. In multi-class classification, the output data may include a classification of the input data into one (or more) of three or more classes. The classification can be single-label or multi-label. The machine learning model 322 can perform discrete category classification, which simply classifies the input data into one or more classes or categories.

[0085] In some implementations, the machine learning model 322 can perform a regression and provide output data in the form of a continuous number. The continuous number can correspond to any number of different metrics or numerical representations, such as, for example, a monetary value, a score (e.g., a performance score), or other numerical representation. By way of example, the machine learning model 322 can perform a linear regression, a polynomial regression, or a nonlinear regression. By way of example, the machine learning model 322 can perform a simple regression or a multiple regression. As discussed above, in some implementations, a softmax function or other function or layer can be used to compress a set of real values ​​associated with three or more possible classes into a set of real values ​​in the range (0,1) that sum to one.

[0086] The machine learning model 322 may perform various types of clustering. For example, the machine learning model 322 may identify one or more previously defined clusters that best correspond to the input data. The machine learning model 322 may identify one or more clusters in the input data. That is, if the input data includes multiple objects, documents, or other entities, the machine learning model 322 may sort the multiple entities included in the input data into multiple clusters. In some implementations in which the machine learning model 322 performs clustering, the machine learning model 322 may be trained using unsupervised learning techniques.

[0087] The machine learning model 322 may perform anomaly or outlier detection. For example, the machine learning model 322 may identify input data that does not conform to expected patterns or other characteristics (e.g., as previously observed from previous input data). By way of example, anomaly detection can be used for fraud detection or system fault detection.

[0088] In some implementations, the machine learning model 322 can provide output data in the form of one or more recommendations. For example, the machine learning model 322 can be incorporated into a recommendation system or engine. As an example, given input data describing previous results of a particular entity (e.g., a score, ranking, rating, etc. indicating a degree of success or enjoyment), the machine learning model 322 can output suggestions or recommendations of one or more additional entities that are predicted to result in a desired outcome (e.g., elicit a score, ranking, rating, etc. indicating success or enjoyment) based on the previous results.

[0089] In some implementations, the machine learning model 322 may be a parametric model, while in other implementations, the machine learning model 322 may be a non-parametric model, in some implementations, the machine learning model 322 may be a linear model, while in other implementations, the machine learning model 322 may be a non-linear model.

[0090] As described above, the machine learning model 322 may be or may include one or more of a variety of different types of machine learning models. Examples of such various types of machine learning models are provided below for illustrative purposes. One or more of the exemplary models described below may be used (e.g., in combination) to provide output data in response to input data. Additional models other than the exemplary models provided below may also be used.

[0091] In some implementations, the machine learning model 322 may be or include one or more classification models, e.g., a linear classification model, a quadratic classification model, etc. The machine learning model 322 may be or include one or more regression models, e.g., a simple linear regression model, a multiple linear regression model, a logistic regression model, a stepwise regression model, a multivariate adaptive regression spline, a locally estimated scatterplot smoothing model, etc.

[0092] In some examples, the machine learning model 322 may be or include one or more decision tree-based models, such as, for example, classification and / or regression trees, iterative binary 3-way decision trees, C4.5 decision trees, chi-squared automated interaction detection decision trees, decision strains, conditional decision trees, etc.

[0093] The machine learning model 322 may be or may include one or more kernel machines. In some implementations, the machine learning model 322 may be or may include one or more support vector machines. The machine learning model 322 may be or may include one or more instance-based learning models, such as, for example, a learning vector quantization model, a self-organizing map model, a local weighted learning model, etc. In some implementations, the machine learning model 322 may be or may include one or more nearest neighbor models, such as, for example, a k-nearest neighbor classification model, a k-nearest neighbor regression model, etc. The machine learning model 322 may be or may include one or more Bayesian models, such as, for example, a naive Bayes model, a Gaussian naive Bayes model, a polynomial naive Bayes model, a mean-one dependent estimator, a Bayesian network, a Bayesian belief network, a hidden Markov model, etc.

[0094] In some implementations, the machine learning model 322 may be or may include one or more artificial neural networks (also referred to simply as neural networks). A neural network may include a group of connected nodes, also referred to as neurons or perceptrons. A neural network may be organized into one or more layers. A neural network that includes multiple layers may be referred to as a "deep" network. A deep network may include an input layer, an output layer, and one or more hidden layers disposed between the input layer and the output layer. The nodes of a neural network may be fully connected or not fully connected.

[0095] The machine learning model 322 may be or may include one or more feedforward neural networks. In a feedforward network, the connections between nodes do not form cycles. For example, each connection may connect a node in a previous layer to a node in a later layer.

[0096] In some cases, the machine learning model 322 may be or may include one or more recurrent neural networks. In some cases, at least some of the nodes of the recurrent neural network may form a cycle. Recurrent neural networks may be particularly useful when processing input data that is continuous in nature. Specifically, in some cases, recurrent neural networks may use recurrent or directed cyclic node connections to pass or retain information from earlier portions of the input data sequence to later portions of the input data sequence.

[0097] In some examples, the sequential input data may include time series data (e.g., sensor data over time or images captured at different times). For example, a recurrent neural network can analyze sensor data over time to detect or predict swipe direction or perform handwriting recognition. Sequential input data may include words in a sentence (e.g., natural language processing, speech detection or processing, etc.), musical notes in a piece of music, sequential actions performed by a user (e.g., detecting or predicting sequential application usage), sequential object states, etc.

[0098] Exemplary recurrent neural networks include long short-term (LSTM) recurrent neural networks, gated recurrent units, bidirectional recurrent neural networks, continuous-time recurrent neural networks, neural history compressors, echo state networks, Elman networks, Jordan networks, recurrent neural networks, Hopfield networks, fully recurrent networks, sequence-to-sequence constructions, and the like.

[0099] In some implementations, the machine learning model 322 may be or may include one or more convolutional neural networks. In some cases, a convolutional neural network may include one or more convolutional layers that perform convolutions on input data with learned filters.

[0100] A filter is also called a kernel. Convolutional neural networks can be particularly useful for vision problems, such as when the input data contains images, either still images or video, but they can also be applied to natural language processing.

[0101] In some examples, the machine learning model 322 may be or may include one or more generative networks, such as, for example, a generative adversarial network. A generative network may be used to generate new data, such as new images or other content.

[0102] The machine learning model 322 may be or may include an autoencoder. In some cases, the goal of an autoencoder is to learn a representation (e.g., a low-dimensional encoding) for a dataset, typically for the purpose of dimensionality reduction. For example, in some cases, an autoencoder encodes input data and attempts to provide output data that reconstructs the input data from the encoding. Recently, the concept of an autoencoder has become more widely used to learn generative models of data. In some cases, an autoencoder may include additional losses beyond the reconstruction of the input data.

[0103] The machine learning model 322 may be or include one or more other forms of artificial neural networks, such as, for example, a deep Boltzmann machine, a deep belief network, a stacked autoencoder, etc. Any of the neural networks described herein can be combined (e.g., stacked) to form more complex networks.

[0104] One or more neural networks can be used to provide embeddings based on the input data. For example, embeddings can be representations of knowledge abstracted from the input data into one or more learned dimensions. In some cases, embeddings can be a useful source for identifying relevant entities. In some cases, embeddings can be extracted from the output of the network, while in other cases, embeddings can be extracted from any hidden node or layer of the network (e.g., near but not the final layer of the network). Embeddings can be useful for performing automatic suggestions of the next video, product suggestions, entity or object recognition, etc. In some cases, embeddings are useful inputs for downstream models. For example, embeddings can be useful for generalizing input data (e.g., search queries) for downstream models or processing systems.

[0105] The machine learning models 322 may include one or more clustering models, such as, for example, a k-means clustering model, a k-median clustering model, an expectation maximization model, a hierarchical clustering model, and the like.

[0106] In some implementations, the machine learning model 322 may perform one or more dimensionality reduction techniques such as, for example, principal component analysis, kernel principal component analysis, graph-based kernel principal component analysis, principal component regression, partial least squares regression, Sammon mapping, multidimensional scaling, projection pursuit, linear discriminant analysis, mixed discriminant analysis, quadratic discriminant analysis, generalized discriminant analysis, flexible discriminant analysis, and auto-encoding.

[0107] In some embodiments, the machine learning model 322 may implement or be subject to one or more reinforcement learning techniques, such as Markov decision processes, dynamic programming, Q-functions or Q-learning, value function techniques, deep Q-networks, differentiable neural computers, asynchronous advantage actacritics, deterministic policy gradients, and the like.

[0108] In some implementations, the machine learning model 322 can be an autoregressive model. In some cases, the autoregressive model can be specified such that the output data depends linearly on its own previous values ​​and on a probability term. In some cases, the autoregressive model can take the form of a stochastic difference equation. One example of an autoregressive model is WaveNet, a model for generating raw speech.

[0109] In some implementations, the machine learning model 322 may include or form part of a multiple model ensemble. As an example, bootstrap aggregation, which may also be referred to as "bagging," may be performed. In bootstrap aggregation, a training dataset is split into several subsets (e.g., via random sampling with replacement) and multiple models are trained on each of the several subsets. At inference time, the outputs of each of the multiple models may be combined (e.g., via averaging, voting, or other techniques) and used as the output of the ensemble.

[0110] One example of an ensemble is a random forest, also known as a random decision forest. A random forest is an ensemble learning method for classification, regression, and other tasks. A random forest is created by generating multiple decision trees at training time. In some cases, at inference time, a class that is the mode of the class (classification) or the average prediction of the individual trees (regression) can be used as the output of the forest. A random decision forest can correct the tendency of decision trees to overfit to the training set.

[0111] Another example of an ensemble technique is stacking, sometimes referred to as stack generalization. Stacking involves training a combiner model to blend or combine predictions of multiple other machine learning models. Thus, multiple machine learning models (e.g., of the same or different types) can be trained based on the training data. Furthermore, the combiner model can be trained to receive predictions from other machine learning models as inputs and, in response, generate a final inference or prediction. In some cases, a single-layer logistic regression model can be used as the combiner model.

[0112] Another example of an ensemble technique is boosting. Boosting may involve building an ensemble step-by-step by repeatedly training weak models and adding them to a final strong model. For example, in some cases, each new model can be trained to emphasize training examples that previous models misinterpreted (e.g., misclassified). For example, the weight associated with each such misinterpreted example can be increased. One common implementation of boosting is AdaBoost, also known as adaptive boosting. Other exemplary boosting techniques include LPBoost, TotalBoost, BrownBoost, xgboost, MadaBoost, LogitBoost, gradient boosting, etc. Additionally, any of the models described above (e.g., regression models and artificial neural networks) may be combined to form an ensemble. As an example, an ensemble may include a top-level machine learning model or heuristic function that combines and / or weights the outputs of the models that form the ensemble.

[0113] In some implementations, multiple machine learning models (e.g., forming an ensemble) can be linked and jointly trained (e.g., via backpropagation through the sequential model ensemble), however, in some implementations, only a subset (e.g., one) of the jointly trained models is used for inference.

[0114] In some implementations, the machine learning model 322 can be used to pre-process input data before it is input to another model. For example, the machine learning model 322 can perform dimensionality reduction techniques and embeddings (e.g., matrix factorization, principal component analysis, singular value decomposition, word2vec / GLOVE, and / or related techniques), clustering, as well as classification and regression for downstream consumption. Many of these techniques are described above and discussed further below.

[0115] As described above, the machine learning model 322 may be trained or otherwise configured to receive input data and, in response thereto, provide output data. The input data may include input data of various types, formats, or variations. By way of example, in various implementations, the input data may include features describing the content (or portion of the content) initially selected by the user, e.g., the content of a document or image selected by the user, links pointing to the user's selection, links within the user selection related to other files available on the device or in the cloud, metadata of the user selection. Additionally, with user permission, the input data may include user usage context obtained from the app itself or other sources. Examples of usage context include breadth of sharing (public, or large group, or private, or shared with specific people), context of sharing, etc. If the user permits, additional input data may include device state, e.g., device location, apps running on the device, etc.

[0116] In some implementations, the machine learning model 322 can receive and use input data in a raw form. In some implementations, the raw input data can be preprocessed. Thus, the machine learning model 322 can receive and use preprocessed input data in addition to or instead of the raw input data.

[0117] In some implementations, pre-processing the input data may include extracting one or more additional features from the raw input data. For example, feature extraction techniques may be applied to the input data to generate one or more new additional features. Exemplary feature extraction techniques include edge detection, corner detection, blob detection, ridge detection, scale invariant feature transforms, motion detection, optical flow, Hough transforms, etc.

[0118] In some implementations, the extracted features may include or be derived from a transformation of the input data into other domains and / or dimensions. As an example, the extracted features may include or be derived from a transformation of the input data into the frequency domain. For example, a wavelet transform and / or a fast Fourier transform may be performed on the input data to generate additional features.

[0119] In some implementations, the extracted features may include statistics calculated from the input data or particular portions or dimensions of the input data. Exemplary statistics include the mode, mean, maximum, minimum, or other metrics of the input data or portions thereof.

[0120] In some implementations, the input data may be sequential in nature, as explained above. In some cases, the sequential input data may be generated by sampling or otherwise segmenting a stream of input data. As an example, frames may be extracted from a video. In some implementations, sequential data may be made non-sequential by summarization.

[0121] Another exemplary pre-processing technique can be to impart portions of the input data, for example by interpolation and / or extrapolation to generate additional synthetic input data.

[0122] As another exemplary preprocessing technique, some or all of the input data may be scaled, standardized, normalized, generalized, and / or regularized. Example regularization techniques include ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least angle regression, cross-validation, L1 regularization, L2 regularization, etc. As an example, some or all of the input data may be normalized by subtracting the mean across feature values ​​in a given dimension from each feature value, then dividing by the standard deviation or other metric.

[0123] As another exemplary pre-processing technique, some or all of the input data can be quantized or discretized. In some cases, qualitative features or variables contained in the input data can be converted into quantitative features or variables. For example, one-hot encoding can be performed.

[0124] In some examples, dimensionality reduction techniques may be applied to the input data before inputting it to the machine learning model 322. Some examples of dimensionality reduction techniques are provided above, including, for example, principal component analysis, kernel principal component analysis, graph-based kernel principal component analysis, principal component regression, partial least squares regression, Sammon mapping, multidimensional scaling, projection pursuit, linear discriminant analysis, mixed discriminant analysis, quadratic discriminant analysis, generalized discriminant analysis, flexible discriminant analysis, auto-encoding, etc.

[0125] In some implementations, during training, the input data may be purposefully distorted in any number of ways to improve the robustness, generalization, or other qualities of the model. Exemplary techniques for distorting the input data include adding noise, changing color, shade, or hue, widening, segmenting, amplifying, etc.

[0126] In response to receiving the input data, the machine learning model 322 can provide output data. The output data can include various types, formats, or variations of output data. By way of example, in various implementations, the output data can include relevant, shareable content along with an initial content selection, stored locally on the user device or stored in the cloud.

[0127] As described above, in some implementations, the output data may include various types of classification data (e.g., two-class classification, multi-class classification, single label, multi-label, discrete classification, regression classification, probabilistic classification, etc.) or may include various types of regression data (e.g., linear regression, polynomial regression, nonlinear regression, simple regression, multiple regression, etc.) In other examples, the output data may include clustering data, anomaly detection data, recommendation data, or other forms of output data as described above.

[0128] In some embodiments, the output data may influence downstream processes or decisions. As an example, in some embodiments, the output data may be interpreted and / or acted upon by rule-based regulators.

[0129] The present disclosure provides systems and methods that include or otherwise utilize one or more machine learning models to output a predicted performance score based on characteristics of a gaming application and a computing device. Any of the different types or forms of input data described above can be combined with any of the different types or forms of machine learning models described above to provide any of the different types or forms of output data described above.

[0130] The systems and methods of the present disclosure may be implemented by or otherwise executed on one or more computing devices, such as computing system 150 and computing system 250. Exemplary computing devices include user computing devices (e.g., mobile computing devices, such as laptops, desktops, tablets, smartphones, wearable computing devices, etc.), embedded computing devices (e.g., devices embedded within vehicles, cameras, image sensors, industrial machinery, satellites, game consoles or controllers, or home appliances, such as refrigerators, thermostats, energy meters, home energy managers, smart home assistants, etc.), server computing devices (e.g., database servers, parameter servers, file servers, mail servers, print servers, web servers, game servers, application servers, etc.), dedicated specialized model processing or training devices, virtual computing devices, other computing devices or computing infrastructure, or combinations thereof.

[0131] 3B illustrates a conceptual diagram of a computing device 302 that is an example of the computing device 102 of FIG. 1. The computing device 302 includes a processing component 340, a memory component 304, and a machine learning model 322. The computing device 302 may store and implement the machine learning model 322 locally (i.e., on-device). Thus, in some implementations, the machine learning model 322 may be stored and / or implemented locally on a user computing device, such as an embedded device or a mobile device. Output data obtained through the local implementation of the machine learning model 322 on the embedded device or user computing device may be used to improve the performance of the embedded device or user computing device (e.g., an application implemented by the embedded device or user computing device).

[0132] 3C illustrates a conceptual diagram of an exemplary client computing device that can communicate over a network with an exemplary server computing system that includes a machine learning model. FIG. 3C includes a client device 302A that communicates over a network 330 with a computing system 350. The client device 302A is an example of the computing device 102 of FIG. 1, the computing system 350 is an example of the computing system 150 of FIG. 1 and the computing system 250 of FIG. 2, and the network 330 is an example of the network 130 of FIG. 1. The computing system 350 stores and implements the machine learning model 322. In some cases, output data obtained through the machine learning model 322 of the computing system 350 may be used to improve other server tasks or may be used by other non-user devices to improve services performed by or for such other non-user devices. For example, the output data may improve other downstream processes performed by the computing system 350 of a user's computing device or an embedded computing device. In other examples, output data obtained through implementation of the machine learning model 322 on the computing system 350 may be transmitted to and used by a user computing device, an embedded computing device, or some other client device, such as the client device 302A. For example, the computing system 350 may be said to be performing machine learning as a service.

[0133] In yet other implementations, different respective portions of the machine learning model 322 may be stored and / or implemented on some combination of a user computing device, an embedded computing device, a server computing device, etc. In other words, portions of the machine learning model 322 may be distributed in whole or in part between the client device 302A and the computing system 350.

[0134] The devices 302 and 350 may execute graph processing techniques or other machine learning techniques using one or more machine learning platforms, frameworks, and / or libraries, such as TensorFlow, Caffe / Caffe2, Theano, Torch / PyTorch, MXnet, CNTK, etc. The devices 302 and 350 may be distributed in different physical locations and connected via one or more networks, including the network 330. When configured as distributed computing devices, the devices 302 and 350 may operate according to a sequential computing architecture, a parallel computing architecture, or a combination thereof. In one example, the distributed computing devices may be controlled or guided through the use of a parameter server.

[0135] In some implementations, multiple instances of the machine learning model 322 may be parallelized to increase processing throughput. For example, multiple instances of the machine learning model 322 may be parallelized on a single processing or computing device or may be parallelized across multiple processing or computing devices.

[0136] Each computing device implementing the machine learning model 322 or other aspects of the disclosure may include several hardware components that enable execution of the techniques described herein. For example, each computing device may include one or more memory devices that store some or all of the machine learning model 322. For example, the machine learning model 322 may be a structured numerical representation stored in memory. The one or more memory devices may also include instructions for implementing the machine learning model 322 or for performing other operations. Examples of memory devices include RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and the like, and combinations thereof.

[0137] Each computing device may also include one or more processing devices that implement some or all of the machine learning model 322 and / or perform other related operations. Exemplary processing devices include one or more of a central processing unit (CPU), a visual processing unit (VPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural processing unit (NPU), a neural processing engine, a core of a CPU, a VPU, a GPU, a TPU, an NPU or other processing device, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a coprocessor, a controller, or a combination of the processing devices described above. The processing devices may be embedded within other hardware components such as, for example, an image sensor, an accelerometer, etc.

[0138] The hardware components (memory devices and / or processing devices) may be distributed across multiple physically distributed computing devices and / or virtually distributed computing systems.

[0139] FIG. 3D illustrates a conceptual diagram of an exemplary computing device in communication with an exemplary training computing system including a model trainer. FIG. 3D includes a computing system 350 in communication with a training device 370 via a network 330. The computing system 350 is an example of the computing system 150 of FIG. 1, and the network 330 is an example of the network 130 of FIG. 1. The machine learning models 322 described herein are trained on a training computing system, such as the training device 370, and then provided for storage and / or implementation on one or more computing devices, such as the computing system 350. For example, the model trainer 372 runs locally on the training device 370. However, in some examples, the training device 370 including the model trainer 372 may be included in or exist separately from the computing system 350 or any other computing device that implements the machine learning models 322.

[0140] In some implementations, the machine learning model 322 may be trained in an offline or online manner. In offline training (also called batch training), the machine learning model 322 is trained on an entire static training data set. In online training, the machine learning model 322 is trained (or retrained) continuously as new training data becomes available (e.g., while performing inference using the model).

[0141] 2 may perform centralized training (e.g., based on a centrally stored dataset) of the machine learning models 322. In other implementations, the machine learning models 322 may be trained, updated, or personalized using distributed training techniques such as distributed training, federated learning, and the like.

[0142] The machine learning model 322 described herein can be trained according to one or more of a variety of training types or techniques. For example, in some implementations, the machine learning model 322 can be trained by the model trainer 372 using supervised learning, where the model is trained on a training dataset that includes examples or examples with labels. The labels can be applied manually by experts, generated through crowdsourcing, or provided by other techniques (e.g., physics-based or complex mathematical models). In some implementations, the training examples can be provided by a user computing device if the user consents. In some implementations, this process can be referred to as personalizing the model.

[0143] 3E illustrates a conceptual diagram of an example training process 390 for training a machine learning model 322 with training data 391 including example input data 392 having labels 393. For example, the input data 392 may be device characteristics of a computing device having labels 393 in the form of smoothness scores associated with game play sessions. The training process 390 is one example of a training process and other training processes may be used.

[0144] The training data 391 used by the training process 390 may include anonymized usage logs of shared flows if the user has permitted such data to be used for training, e.g., content items that have been shared together, bundled content pieces that have already been identified as belonging together from entities in a knowledge graph, etc. In some implementations, the training data 391 may include examples of input data 392 that have been assigned labels 393 that correspond to output data 394.

[0145] In some implementations, the machine learning model 322 can be trained by optimizing an objective function, such as objective function 395. For example, in some implementations, the objective function 395 can be or include a loss function that compares (e.g., determines the difference between) output data generated by the model from the training data and a label associated with the training data (e.g., a ground truth label). For example, the loss function can evaluate the sum or average of squared differences between the output data and the label. In some examples, the objective function 395 can be or include a cost function that represents the cost of a particular outcome or output data. Other examples of objective function 395 can include margin-based techniques, such as, for example, triplet loss or maximum margin training.

[0146] One or more of a variety of optimization techniques can be performed to optimize the objective function 395. For example, the optimization technique(s) can minimize or maximize the objective function 395. Exemplary optimization techniques include Hessian-based techniques and gradient-based techniques, such as, for example, coordinate descent, gradient descent (e.g., stochastic gradient descent), subgradient methods, etc. Other optimization techniques include black-box optimization techniques and heuristics.

[0147] In some implementations, backpropagation can be used in combination with optimization techniques (e.g., gradient-based techniques) to train the machine learning model 322 (e.g., where the machine learning model is a multi-layer model such as an artificial neural network). For example, iterative cycles of propagation and updating model parameters (e.g., weights) can be performed to train the machine learning model 322. Exemplary backpropagation techniques include truncated backpropagation through time, Levenberg-Marquardt backpropagation, and the like.

[0148] In some implementations, the machine learning models 322 described herein can be trained using unsupervised learning techniques. Unsupervised learning can include inferring functions to explain hidden structure from unlabeled data. For example, classifications or categorizations may not be included in the data. Unsupervised learning techniques can be used to create machine learning models that can perform clustering, anomaly detection, learning latent variable models, or other tasks.

[0149] The machine learning model 322 can be trained using semi-supervised techniques that combine aspects of supervised and unsupervised learning. The machine learning model 322 can be trained or otherwise generated through evolutionary techniques or genetic algorithms. In some implementations, the machine learning model 322 described herein can be trained using reinforcement learning. In reinforcement learning, an agent (e.g., a model) can perform actions in an environment and learn to maximize rewards and / or minimize penalties resulting from such actions. Reinforcement learning can differ from supervised learning problems in that correct input / output pairs are not presented and suboptimal actions are not explicitly corrected.

[0150] In some implementations, one or more generalization techniques can be performed during training to improve generalization of the machine learning model 322. Generalization techniques can help reduce overfitting of the machine learning model 322 to the training data. Exemplary generalization techniques include dropout techniques, weight decay techniques, batch normalization, early stopping, subset selection, stepwise selection, etc.

[0151] In some implementations, the machine learning models 322 described herein may include or be influenced by several hyperparameters, such as learning rate, number of layers, number of nodes in each layer, number of leaves in the tree, number of clusters, etc. The hyperparameters may affect the performance of the model. The hyperparameters may be selected manually or automatically by applying techniques such as grid search, black-box optimization techniques (e.g., Bayesian optimization, random search, etc.), gradient-based optimization, etc. Exemplary techniques and / or tools for performing automatic hyperparameter optimization include Hyperopt, Auto-WEKA, Spearmint, Metric Optimization Engine (MOE), etc.

[0152] In some implementations, various techniques can be used to optimize and / or adapt the learning rate when training a model. Exemplary techniques and / or tools for performing learning rate optimization or adaptation include Adagrad, Adaptive Estimation of Moments (ADAM), Adadelta, RMSpropw, etc.

[0153] In some implementations, transfer learning techniques can be used to provide an initial model to begin training the machine learning models 322 described herein.

[0154] In some implementations, the machine learning models 322 described herein may be included in various pieces of computer readable code on a computing device. In one example, the machine learning models 322 may be included in a particular application or program and may be used (e.g., exclusively) by such a particular application or program. Thus, in one example, a computing device may include several applications, one or more of which may each include their own machine learning library and machine learning model(s).

[0155] In another example, the machine learning models 322 described herein may be included in an operating system of a computing device (e.g., a central intelligence layer of the operating system) and may be invoked or otherwise used by one or more applications that interact with the operating system. In some implementations, each application may communicate with the central intelligence layer (and the model(s) stored therein) using an application programming interface (API) (e.g., a common public API across all applications).

[0156] In some implementations, the central intelligence layer can communicate with a central device data layer. The central device data layer can be a central repository of data for the computing device. The central device data layer can communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0157] The technology described herein refers to servers, databases, software applications, and other computer-based systems, as well as actions performed on and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of configurations, combinations, and divisions of tasks and functions between components. For example, the processes discussed herein can be implemented using a single device or component, or multiple devices or components working in combination.

[0158] The database and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0159] Moreover, the machine learning techniques described herein are readily interchangeable and combinable: although specific example techniques have been described, many other techniques exist and can be used in combination with aspects of the present disclosure.

[0160] A brief overview of example machine learning models and related techniques has been provided by this disclosure, for additional details see the following references: Machine Learning A Probabilistic Perspective (Murphy), Rules of Machine Learning: Best Practices for ML Engineering (Zinkevich), Deep Learning (Goodfellow), Reinforcement Learning: An Introduction (Sutton), and Artificial Intelligence: A Modern Approach (Norvig).

[0161] In addition to the above, a user may be provided with controls that allow the user to make choices about both whether and when the systems, programs, or features described herein enable collection of user information (e.g., information about the user's social network, social actions, or activities, occupation, user preferences, or the user's current location), as well as whether content or communications are sent from the server to the user. Additionally, certain data may be processed in one or more ways such that personally identifiable information is removed before it is stored or used. For example, the user's identity may be processed such that the user's personally identifiable information cannot be identified, or if location information is obtained (such as to the city, zip code, or state level), the user's geographic location may be generalized such that the user's specific location cannot be identified. Thus, the user may control what information is collected about the user, how that information is used, and what information is provided to the user.

[0162] 4 is a flowchart illustrating an example of an operational mode of a computing system for determining a predicted performance of a gaming application executed on a computing device in accordance with one or more techniques of this disclosure, and is described below in the context of FIGS.

[0163] 4, one or more processors 240 of computing system 250 may receive from computing device 101 an indication of one or more device characteristics of computing device 102 (402). In some examples, one or more processors 240 may receive from computing device 102 a query for a catalog of gaming applications that may be installed on computing device 102, the query including the indication of the one or more device characteristics of computing device 102. In some examples, the one or more device characteristics of computing device 102 include one or more of a central processing unit (CPU) characteristic of computing device 102, a graphics processing unit (GPU) characteristic of computing device 102, or a memory characteristic of computing device 102.

[0164] The one or more processors 240 may determine (404) a predicted performance of the game application 112 when executed in the virtualized environment of the computing device 102 based at least in part on one or more device characteristics of the computing device 102 and one or more application characteristics of the game application 112 using the game performance model 264 trained using machine learning to determine a predicted performance score for the game application 112. In some examples, the predicted performance of the game application 112 when executed in the virtualized environment of the computing device 102 indicates a smoothness score corresponding to a percentage of frames of image data rendered by the game application 112 when executed in the virtualized environment of the computing device 102 that are predicted to meet a frame time of a target frame rate associated with the game application 112.

[0165] The one or more processors 240 may transmit to the computing device 102 an indication of the predicted performance of the gaming application 112 when executed in the virtualized environment of the computing device 102 (406). In some examples, the one or more processors 240 may transmit to the computing device 102 a catalog of gaming applications that can be installed on the computing device 102, including an indication of the predicted performance of the gaming application 112 when executed in the virtualized environment of the computing device 102.

[0166] In some examples, the game performance model 264 is trained using training data derived from data collected from a population of computing devices 120 running the game application 112, and a predicted performance score for the game application 112 is determined.

[0167] In some examples, the training data includes, for one game play session of a game application 112 in the population of computing devices 120, one or more device characteristics of a corresponding computing device from the population of computing devices 120 associated with a corresponding smoothness score corresponding to a percentage of frames of image data rendered by the game application 112 during a game play session on the corresponding computing device that is predicted to meet a frame time of a target frame rate associated with the game application 112.

[0168] In some examples, the one or more processors 240 may receive data collected from a population of computing devices 120, the data including frame time metrics associated with frames of image data rendered and output by a game application 112 executing on the population of computing devices 120, and may determine a corresponding smoothness score for the game play session based at least in part on the frame time metrics included in the data collected from the population of computing devices 120.

[0169] The present disclosure includes the following examples. Example 1. A method comprising: receiving, by one or more processors of a computing system, an indication of one or more device characteristics of the computing device from a computing device; determining, by the one or more processors, a predicted performance of the gaming application when executing in a virtualized environment of the computing device based at least in part on the one or more device characteristics of the computing device and one or more application characteristics of the gaming application using a game performance model trained using machine learning to determine a predicted performance score of the gaming application; and sending, by the one or more processors, to the computing device an indication of the predicted performance of the gaming application when executing in the virtualized environment of the computing device.

[0170] Example 2. The method described in Example 1, wherein the predicted performance of the game application when executing in the virtualized environment of the computing device is indicated by a smoothness score corresponding to a percentage of frames of image data rendered by the game application when executing in the virtualized environment of the computing device that are predicted to meet the frame time of a target frame rate associated with the game application.

[0171] Example 3. The method of any of Examples 1 and 2, wherein the one or more device characteristics of the computing device include one or more of a central processing unit (CPU) characteristic of the computing device, a graphics processing unit (GPU) characteristic of the computing device, or a memory characteristic of the computing device.

[0172] Example 4. The method of any of Examples 1-3, wherein the game performance model is trained using training data derived from performance metrics collected from a population of computing devices executing the game application to determine the predicted performance score for the game application.

[0173] Example 5. The method of Example 4, wherein the training data includes, for one game play session of a plurality of game play sessions of the game application on the population of computing devices, one or more device characteristics of the corresponding computing devices from the population of computing devices associated with a corresponding smoothness score corresponding to a percentage of frames of image data rendered by the game application during the game play session on the corresponding computing device that is predicted to meet a frame time of a target frame rate associated with the game application.

[0174] Example 6. The method of Example 5, further comprising: receiving, by the one or more processors, the performance metrics collected from the population of computing devices, the performance metrics including frame time metrics associated with frames of image data rendered and output by the game application executing on the population of computing devices; and determining, by the one or more processors, the corresponding smoothness score for the game play session based at least in part on the frame time metrics included in the performance metrics collected from the population of computing devices.

[0175] Example 7. A method according to any of Examples 1 to 6, wherein receiving the indication of the one or more device characteristics of the computing device from the computing device includes receiving, by the one or more processors, a query for a catalog of game applications from the computing device, the query including the indication of the one or more device characteristics of the computing device, and sending to the computing device the indication of the predicted performance of the game application when executing in the virtualized environment of the computing device includes sending, by the one or more processors, to the computing device the catalog of game applications, the query including the indication of the predicted performance of the game application when executing in the virtualized environment of the computing device.

[0176] Example 8. A computing system comprising: a memory; a network interface; and one or more processors operatively connected to the memory and the network interface, the one or more processors configured to: receive an indication of one or more device characteristics of the computing device from a computing device via the network interface; determine a predicted performance of the gaming application when executing in a virtualized environment of the computing device based at least in part on the one or more device characteristics of the computing device and one or more application characteristics of the gaming application using a game performance model trained using machine learning to determine a predicted performance score of the gaming application; and transmit an indication of the predicted performance of the gaming application when executing in the virtualized environment of the computing device to the computing device via the network interface.

[0177] Example 9. The computing system of Example 8, wherein the predicted performance of the game application when executing in the virtualized environment of the computing device indicates a smoothness score corresponding to a percentage of frames of image data rendered by the game application when executing in the virtualized environment of the computing device that are predicted to meet a frame time of a target frame rate associated with the game application.

[0178] Example 10. A computing system as described in any of Examples 8 and 9, wherein the one or more device characteristics of the computing device include one or more of a central processing unit (CPU) characteristic of the computing device, a graphics processing unit (GPU) characteristic of the computing device, or a memory characteristic of the computing device.

[0179] Example 11. A computing system described in any of Examples 8 to 10, wherein the game performance model is trained using training data derived from performance metrics collected from a population of computing devices running the game application to determine the predicted performance score for the game application.

[0180] Example 12. The computing system of Example 11, wherein the training data includes one or more device characteristics of a corresponding computing device from the population of computing devices associated with a corresponding smoothness score corresponding to a percentage of frames of image data rendered by the game application during the game play session on the corresponding computing device that is predicted to meet a frame time of a target frame rate associated with the game application, for one game play session of a plurality of game play sessions of the game application on the population of computing devices.

[0181] Example 13. The computing system of Example 12, wherein the one or more processors are further configured to receive the performance metrics collected from the population of computing devices, the data including frame time metrics associated with frames of image data rendered and output by the game application executing on the population of computing devices, and determine the corresponding smoothness score for the game play session based at least in part on the frame time metrics included in the performance metrics collected from the population of computing devices.

[0182] Example 14. A method according to any of Examples 8 to 13, wherein, to receive from the computing device, the indication of the one or more device characteristics of the computing device, the one or more processors are further configured to receive from the computing device a query for a catalog of game applications including the indication of the one or more device characteristics of the computing device, and, to transmit to the computing device, the indication of the predicted performance of the game application when executing in the virtualized environment of the computing device, the one or more processors are further configured to transmit to the computing device the catalog of game applications including the indication of the predicted performance of the game application when executing in the virtualized environment of the computing device.

[0183] Example 15. A computer-readable storage medium storing instructions which, when executed, cause one or more processors of a computing system to receive from a computing device an indication of one or more device characteristics of the computing device, determine a predicted performance of the gaming application when executing in a virtualized environment of the computing device based at least in part on the one or more device characteristics of the computing device and one or more application characteristics of the gaming application using a game performance model trained using machine learning to determine a predicted performance score of the gaming application, and send to the computing device an indication of the predicted performance of the gaming application when executing in the virtualized environment of the computing device.

[0184] Example 16. A computer-readable storage medium as described in Example 15, wherein the predicted performance of the game application when executing in the virtualized environment of the computing device indicates a smoothness score corresponding to a percentage of frames of image data rendered by the game application when executing in the virtualized environment of the computing device that are predicted to meet a frame time of a target frame rate associated with the game application.

[0185] Example 17. A computer-readable storage medium as described in any of Examples 15 and 16, wherein the one or more device characteristics of the computing device include one or more of a central processing unit (CPU) characteristic of the computing device, a graphics processing unit (GPU) characteristic of the computing device, or a memory characteristic of the computing device.

[0186] Example 18. A computer-readable storage medium described in any of Examples 15 to 17, wherein the game performance model is trained using training data derived from performance metrics collected from a population of computing devices running the game application to determine the predicted performance score for the game application.

[0187] Example 19. A computer-readable storage medium as described in Example 18, wherein the training data includes one or more device characteristics of a corresponding computing device from the population of computing devices associated with a corresponding smoothness score corresponding to a percentage of frames of image data rendered by the game application during the game play session on the corresponding computing device that is predicted to meet a frame time of a target frame rate associated with the game application, for one game play session of a plurality of game play sessions of the game application on the population of computing devices.

[0188] Example 20. A computer-readable storage medium as described in Example 19, wherein the instructions are further configured to cause the one or more processors to receive the performance metrics collected from the population of computing devices, the performance metrics including frame time metrics associated with frames of image data rendered and output by the game application executed on the population of computing devices, and determine the corresponding smoothness score for the game play session based at least in part on the frame time metrics included in the performance metrics collected from the population of computing devices.

[0189] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other storage medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly referred to as a computer-readable storage medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable storage media and media, and data storage media do not include connections, carrier waves, signals, or other transitory media, and instead, such media are directed to non-transitory, tangible storage media. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer readable storage media.

[0190] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Thus, the term "processor" as used herein may refer to any of the foregoing structures, or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided in dedicated hardware and / or software modules. The techniques may also be implemented entirely in one or more circuits or logic elements.

[0191] The techniques of the present disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). This disclosure describes various components, modules, or units to highlight functional aspects of devices configured to perform the disclosed techniques, but does not necessarily require realization by different hardware units. Rather, as described above, the various units may be combined into a hardware unit or provided by a collection of interoperable hardware units, including one or more processors as described above, in combination with appropriate software and / or firmware.

[0192] Various embodiments have been described. These and other embodiments are within the scope of the following claims.

Claims

1. receiving, by one or more processors of a computing system, an indication from a computing device of one or more device characteristics of the computing device; determining, by the one or more processors, a predicted performance of the gaming application when executing in a virtualized environment of the computing device based at least in part on the one or more device characteristics of the computing device and one or more application characteristics of the gaming application using a gaming performance model trained using machine learning to determine a predicted performance score of the gaming application; transmitting, by the one or more processors, to the computing device an indication of the predicted performance of the gaming application when executing in the virtualized environment of the computing device; A method for providing the above.

2. 2. The method of claim 1, wherein the predicted performance of the gaming application when executing in the virtualized environment of the computing device indicates a smoothness score corresponding to a percentage of frames of image data rendered by the gaming application when executing in the virtualized environment of the computing device that are predicted to meet a frame time of a target frame rate associated with the gaming application.

3. 2. The method of claim 1, wherein the one or more device characteristics of the computing device include one or more of a central processing unit (CPU) characteristic of the computing device, a graphics processing unit (GPU) characteristic of the computing device, or a memory characteristic of the computing device.

4. 2. The method of claim 1 , wherein the game performance model is trained using training data derived from performance metrics collected from a population of computing devices executing the game application to determine the predicted performance score for the game application.

5. 5. The method of claim 4, wherein the training data includes, for one game play session of a plurality of game play sessions of the game application on the population of computing devices, one or more device characteristics of a corresponding computing device from the population of computing devices associated with a corresponding smoothness score corresponding to a percentage of frames of image data rendered by the game application during the game play session on the corresponding computing device that is predicted to meet a frame time of a target frame rate associated with the game application.

6. and receiving, by the one or more processors, the performance metrics collected from the population of computing devices, the performance metrics including frame time metrics associated with frames of image data rendered and output by the gaming application executing on the population of computing devices, the method comprising:

6. The method of claim 5, further comprising determining, by the one or more processors, the corresponding smoothness score for the game play session based at least in part on the frame time metric included in the performance metrics collected from the population of computing devices.

7. receiving from the computing device the indication of the one or more device characteristics of the computing device includes receiving, by the one or more processors, from the computing device a query for a catalog of gaming applications that includes the indication of the one or more device characteristics of the computing device; 2. The method of claim 1, wherein transmitting to the computing device the indication of the predicted performance of the game application when executing in the virtualization environment of the computing device includes transmitting, by the one or more processors, to the computing device a catalog of the game application including the indication of the predicted performance of the game application when executing in the virtualization environment of the computing device.

8. 1. A computing system comprising: Memory, A network interface; one or more processors operatively connected to the memory and to the network interface, receiving an indication of one or more device characteristics of the computing device from a computing device via the network interface; determining a predicted performance of the gaming application when executing in a virtualized environment of the computing device based at least in part on the one or more device characteristics of the computing device and one or more application characteristics of the gaming application using a gaming performance model trained using machine learning to determine a predicted performance score of the gaming application; transmitting to the computing device via the network interface an indication of the predicted performance of the gaming application when executing in the virtualized environment of the computing device; the one or more processors configured to: A computing system comprising:

9. 10. The computing system of claim 8, wherein the predicted performance of the gaming application when executing in the virtualized environment of the computing device indicates a smoothness score corresponding to a percentage of frames of image data rendered by the gaming application when executing in the virtualized environment of the computing device that are predicted to meet a frame time of a target frame rate associated with the gaming application.

10. 9. The computing system of claim 8, wherein the one or more device characteristics of the computing device include one or more of a central processing unit (CPU) characteristic of the computing device, a graphics processing unit (GPU) characteristic of the computing device, or a memory characteristic of the computing device.

11. 10. The computing system of claim 8, wherein the game performance model is trained using training data derived from performance metrics collected from a population of computing devices executing the game application to determine the predicted performance score for the game application.

12. 12. The computing system of claim 11, wherein the training data includes, for one game play session of a plurality of game play sessions of the game application on the population of computing devices, one or more device characteristics of the corresponding computing devices from the population of computing devices associated with a corresponding smoothness score corresponding to a percentage of frames of image data rendered by the game application during the game play session on the corresponding computing device that is predicted to meet a frame time of a target frame rate associated with the game application.

13. The one or more processors: and receiving the performance metrics collected from the population of computing devices, the data including frame time metrics associated with frames of image data rendered and output by the gaming application executing on the population of computing devices, the one or more processors being further configured to:

13. The computing system of claim 12, further configured to determine the corresponding smoothness score for the game play session based at least in part on the frame time metric included in the performance metrics collected from the population of computing devices.

14. To receive from the computing device the indication of the one or more device characteristics of the computing device, the one or more processors are further configured to receive from the computing device a query for a catalog of gaming applications that includes the indication of the one or more device characteristics of the computing device; 10. The computing system of claim 8, wherein to transmit to the computing device the indication of the predicted performance of the gaming application when executing in the virtualized environment of the computing device, the one or more processors are further configured to transmit to the computing device a catalog of the gaming application including the indication of the predicted performance of the gaming application when executing in the virtualized environment of the computing device.

15. A computer-readable storage medium storing instructions that, when executed, cause one or more processors of a computing system to: receiving, from a computing device, an indication of one or more device characteristics of the computing device; determining a predicted performance of the gaming application when executing in a virtualized environment of the computing device based at least in part on the one or more device characteristics of the computing device and one or more application characteristics of the gaming application using a gaming performance model trained using machine learning to determine a predicted performance score of the gaming application; transmitting to the computing device an indication of the predicted performance of the gaming application when executing in the virtualization environment of the computing device; A computer-readable storage medium for causing a computer to perform the above steps.

16. 16. The computer-readable storage medium of claim 15, wherein the predicted performance of the gaming application when executing in the virtualized environment of the computing device indicates a smoothness score corresponding to a percentage of frames of image data rendered by the gaming application when executing in the virtualized environment of the computing device that are predicted to meet a frame time of a target frame rate associated with the gaming application.

17. 16. The computer-readable storage medium of claim 15, wherein the one or more device characteristics of the computing device include one or more of a central processing unit (CPU) characteristic of the computing device, a graphics processing unit (GPU) characteristic of the computing device, or a memory characteristic of the computing device.

18. 16. The computer-readable storage medium of claim 15, wherein the game performance model is trained using training data derived from performance metrics collected from a population of computing devices executing the game application to determine the predicted performance score for the game application.

19. 20. The computer-readable storage medium of claim 18, wherein the training data includes, for one game play session of a plurality of game play sessions of the game application on the population of computing devices, one or more device characteristics of the corresponding computing devices from the population of computing devices associated with a corresponding smoothness score corresponding to a percentage of frames of image data rendered by the game application during the game play session on the corresponding computing device that is predicted to meet a frame time of a target frame rate associated with the game application.

20. The instructions further cause the one or more processors to: receiving the performance metrics collected from the population of computing devices, the performance metrics including frame time metrics associated with frames of image data rendered and output by the gaming application executing on the population of computing devices, the instructions further causing the one or more processors to:

20. The computer-readable storage medium of claim 19, further comprising: determining the corresponding smoothness score for the game play session based at least in part on the frame time metric included in the performance metrics collected from the population of computing devices.

Images