Performance prediction for virtualized game applications

Abstract

To provide a method for providing predicted performance of a game application when executed in a virtualized environment of a computing device.SOLUTION: A computing system receives an indication of device characteristics of a computing device to determine a predicted performance score for a game application; uses a game performance model trained using machine learning; determines, on the basis of the device characteristics and application characteristics of the game application, a predicted performance of the game application when executed in a virtualized environment of the computing device; and transmits an indication of the predicted performance to the computing device.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] Computing devices can vary in many respects, such as the performance of a central processing unit (CPU), the performance of a graphics processing unit (GPU), the size and / or resolution of a display, and the performance of memory. Such differences in device configurations can cause game applications executed on various computing devices to behave differently across different computing devices. For example, a game application can output image data at different frame rates while being executed on different computing devices.

Summary of the Invention

[0002] Generally, the techniques of the present disclosure are directed to determining the predicted performance of a game application when executed on a computing device. The predicted performance indicates how well the game application will perform when executed on the computing device. A computing system can use a game performance model trained using machine learning to determine a predicted performance score for the game application, and based at least in part on one or more device characteristics of the computing device and one or more application characteristics of the game application, determine the predicted performance of the game application when executed in a virtualized environment of the computing device, and can transmit an indication of the predicted performance of the game application when executed in the virtualized environment of the computing device to the computing device.

[0003] A computing device may decide whether to recommend and / or suggest a game application for download and installation on the computing device based on its predicted performance when the game application runs in the computing device's virtualized environment. For example, if the predicted performance of a game application when running in the computing device's virtualized environment indicates that the game application may not perform well when running in the computing device's virtualized environment, the computing device may refrain from recommending or suggesting the game application for download and installation on the computing device.

[0004] The computing device may also adjust one or more graphical parameters of a game application based on its predicted performance when running in the computing device's virtualized environment. For example, if the predicted performance of a game application when running in the computing device's virtualized environment indicates that the game application may not perform optimally when running in the computing device's virtualized environment, the computing device may reduce the graphical quality of the image data rendered and output by the game application in order to improve the performance of the game application when running in the computing device's virtualized environment.

[0005] In some embodiments, the techniques described herein include: one or more processors of a computing system receiving an indication of one or more device characteristics of a computing device from a computing device; one or more processors determining the predicted performance of a game application when running in a virtualized environment of the computing device, at least in part on one or more device characteristics of the computing device and one or more application characteristics of the game application, using a game performance model trained using machine learning to determine a predicted performance score of the game application; and one or more processors sending an indication of the predicted performance of the game application when running in a virtualized environment of the computing device to the computing device.

[0006] In some embodiments, the technology described herein relates to a computing system comprising: memory; a network interface; and one or more processors operably connected to the memory and the network interface, the processors being configured to receive, via the network interface, an indication of one or more device characteristics of a computing device from a computing device; to determine the predicted performance of a game application when running in a virtualized environment of the computing device, at least in part on one or more device characteristics of the computing device and one or more application characteristics of the game application, using a game performance model trained using machine learning to determine a predicted performance score of the game application; and to send an indication of the predicted performance of the game application when running in a virtualized environment of the computing device to the computing device via the network interface.

[0007] In some embodiments, the techniques described herein relate to a computer-readable storage medium for storing instructions, wherein when the instructions are executed, one or more processors of a computing system receive an indication of one or more device characteristics of a computing device from a computing device, determine the predicted performance of the game application when running in a virtualized environment of the computing device, at least partially based on one or more device characteristics of the computing device and one or more application characteristics of the game application, using a game performance model trained using machine learning to determine a predicted performance score of the game application, and send an indication of the predicted performance of the game application when running in a virtualized environment of the computing device to the computing device.

[0008] Details of one or more embodiments are described in the accompanying drawings and the following description. Other features, purposes, and advantages of this disclosure will become apparent from the description and drawings, as well as from the claims. [Brief explanation of the drawing]

[0009] [Figure 1] One or more aspects of the present disclosure are conceptual diagrams illustrating an exemplary environment in which an exemplary computing system is configured to determine the predicted performance of a game application when run on an exemplary computing device. [Figure 2] Block diagram showing an exemplary computing system according to one or more aspects of the present disclosure. [Figure 3A] This is a conceptual diagram illustrating an example of a machine learning model according to the embodiments of this disclosure. [Figure 3B] This is a conceptual diagram illustrating an example of a machine learning model according to the embodiments of this disclosure. [Figure 3C] This is a conceptual diagram illustrating an example of a machine learning model according to the embodiments of this disclosure. [Figure 3D] This is a conceptual diagram illustrating an example of a machine learning model according to the embodiments of this disclosure. [Figure 3E] This is a conceptual diagram illustrating an example of a machine learning model according to the embodiments of this disclosure. [Figure 4] A flowchart illustrating example operating modes of a computing system that determine the predicted performance of a game application when run on a computing device, based on one or more of the technologies of this disclosure. [Modes for carrying out the invention]

[0010] Figure 1 is a conceptual diagram showing an exemplary environment in which an exemplary computing system is configured to determine the predicted performance of a game application when it is run on an exemplary computing device, according to one or more embodiments of the present disclosure. In the example of Figure 1, the environment 100 may include a computing device 102, computing devices 120A to 120N (hereinafter, "computing device 120"), and a computing system 150 that communicates over a network 130 to determine the predicted performance of a game application 112 when it is run in a virtualized environment of computing device 102.

[0011] The computing system 150 may be any suitable remote computing system, such as one or more desktop computers, laptop computers, mainframes, servers, cloud computing systems, or virtual machines, that can send and receive information over the network 130. In some examples, the computing system 150 may represent a cloud computing system that provides one or more services over the network 130. That is, in some examples, the computing system 150 may be a distributed computing system. One or more computing devices, such as computing device 102 and / or computing device 120, may access services provided by the cloud by communicating with the computing system 150. Although described herein as being performed at least partially by the computing system 150, any or all of the technologies of this disclosure may be performed by one or more other devices, such as either computing device 102 or computing device 120. That is, in some examples, computing device 102 and / or one or more computing devices 120 may be capable of operating independently to perform one or more of the technologies of this disclosure.

[0012] The computing system 150 may include a game performance module 162 and an application distribution module 172. The game performance module 162 and the application distribution module 172 may each reside within the computing system 150 or on one or more other remote computing devices, and perform the operations described herein using software, hardware, firmware, or a combination of both hardware, software, and firmware running there. In some examples, the game performance module 162 and the application distribution module 172 may each be implemented as hardware, software, and / or a combination of hardware and software. The computing system 150 may run the game performance module 162 and the application distribution module 172 using one or more processors. The computing system 150 may run the game performance module 162 and / or the application distribution module 172 as virtual machines running on underlying hardware, or within virtual machines. The game performance module 162 and the application distribution module 172 can be implemented in various ways. For example, the game performance module 162 and / or the application distribution module 172 may be implemented as downloadable or pre-installed applications, i.e., "apps." In another example, the game performance module 162 and / or the application distribution module 172 may be implemented as part of the operating system of the computing system 150. Other examples of the computing system 150 implementing the technology of this disclosure may include additional components not shown in Figure 1.

[0013] Network 130 may be any suitable network that enables communication between computing device 102, computing device 120, and / or computing system 150. Network 130 may include wide area networks such as the Internet, local area networks (LANs), personal area networks (PANs) (e.g., Bluetooth®), enterprise networks, wireless networks, cellular networks, telephone networks, metropolitan area networks (e.g., WIFI, WAN, WiMAX®, etc.), one or more other types of networks, or combinations of two or more different types of networks (e.g., a combination of a cellular network and the Internet).

[0014] Computing device 102 and each computing device 120 may include, but are not limited to, mobile or portable devices such as mobile phones (including smartphones), laptop computers, tablet computers, wearable computing devices such as smartwatches or computerized eyewear, smart TV platforms, cameras, computerized home appliances, and vehicle head units. In some examples, computing device 102 may include fixed computing devices such as desktop computers, servers, and mainframes.

[0015] Each computing device 120 may include a game application 112 and an emulator module 114. The game application 112 and emulator module 114 may reside in each of the computing devices 120 or in one or more other remote computing devices and perform the operations described herein using software, hardware, firmware, or a combination of both hardware, software, and firmware running therein. In some examples, the game application 112 and emulator module 114 may be implemented as hardware, software, and / or a combination of hardware and software. Each of the computing devices 120 may run the game application 112 and emulator module 114 using one or more processors. The game application 112 and emulator module 114 may be implemented in various ways. For example, either the game application 112 and / or emulator module 114 may be implemented as a downloadable or pre-installed application, i.e., an “app”. In another example, either the game application 112 and / or emulator module 114 may be implemented as part of the operating system of each of the computing devices 120. Other examples of computing devices 120 implementing the technology of this disclosure may include additional components not shown in Figure 1.

[0016] The computing device 102 may include a user interface component 104 ("UIC104"), a user interface module 106 ("UI module 106"), an emulator module 114, and an application catalog module 116. The UI module 106, emulator module 114, and application catalog module 116 may reside within the computing device 102 or on one or more other remote computing devices and perform the operations described herein using software, hardware, firmware, or a combination of both hardware, software, and firmware running therein. In some examples, the UI module 106, emulator module 114, and application catalog module 116 may be implemented as hardware, software, and / or a combination of hardware and software. The computing device 102 may use one or more processors to run the UI module 106, emulator module 114, and application catalog module 116. The UI module 106, emulator module 114, and application catalog module 116 may be implemented in a variety of ways. For example, any of module 106, emulator module 114, and / or application catalog module 116 may be implemented as a downloadable or pre-installed application, i.e., an “app.” In another example, any of UI module 106, emulator module 114, and application catalog module 116 may be implemented as part of the operating system of computing device 102. Other examples of computing device 102 implementing the technology of this disclosure may include additional components not shown in Figure 1.

[0017] The UIC 104 of the computing device 102 can function as both an input and output device for the computing device 102. For example, the UIC 104 can function as an input device using a resistive touchscreen, surface acoustic wave touchscreen, capacitive touchscreen, projected capacitive touchscreen, pressure-sensitive screen, acoustic pulse-recognition touchscreen, or other presence-sensing screen technology. The UIC 104 can function as an output device using one or more of the following: liquid crystal displays (LCDs), dot matrix displays, light-emitting diode (LED) displays, micro-LEDs, organic light-emitting diode (OLED) displays, e-ink, or similar monochrome or color displays capable of outputting visible information to the user of the computing device 102.

[0018] In some examples, the UIC 104 may include a presence-sensing screen capable of receiving haptic user input from a user of the computing device 102. The UIC 104 may receive haptic user input by detecting one or more taps and / or gestures from a user of the computing device 102 (e.g., the user touching or pointing to one or more locations on the UIC 104 with a finger or stylus pen). The presence-sensing screen of the UIC 104 may present outputs to the user. The UIC 104 may present outputs as a user interface that may be related to the functions provided by the computing device 102. For example, the UIC 104 may present various functions and applications that run on the computing device 102, such as electronic messaging applications, messaging applications, and map applications.

[0019] The UI module 106 can interpret input detected by the UIC 104 (for example, when a user provides one or more gestures at the location of the UIC 104 where the user interface is displayed). The UI module 106 can relay information about the input detected by the UIC 104 to one or more related platforms, operating systems, applications, and / or services running on the computing device 102, causing the computing device 102 to perform functions. The UI module 106 can also receive information and instructions from one or more related platforms, operating systems, applications, and / or services running on the computing device 102 (e.g., a game application 112) to generate a graphical user interface (GUI). Furthermore, the UI module 106 can act as an intermediary between one or more related platforms, operating systems, applications, and / or services running on the computing device 102 and various output devices of the computing device 102 (e.g., speakers, LED indicators, vibrators, etc.), causing the computing device 102 to generate outputs (e.g., graphics, auditory, tactile, etc.).

[0020] The emulator module 114 may provide a virtualization environment that allows applications running on computing device 102 and computing device 120, respectively, that are not designed to run on computing device 102 or computing device 120, to run in the virtualization environment provided by the emulator module 114 on computing device 102 or computing device 120. For example, if computing device 102 and computing device 120 are laptop computing devices or desktop computing devices running a desktop operating system, the emulator module 114 may provide a virtualization environment that allows computing applications, such as game applications designed to run on a smartphone's mobile operating system, to run in the virtualization environment provided by the emulator module 114 on computing device 102 and computing device 120, respectively.

[0021] In some examples, the emulator module 114 may convert code of an application (e.g., object code, bytecode, etc.) written and / or compiled for a different processing architecture and / or a different operating system into code executable on computing device 102 and computing device 120, or otherwise provide a virtual execution environment different from the execution environments of computing device 102 and computing device 120 to enable an application not designed to execute on computing device 102 or computing device 120 to execute on computing device 102 or computing device 120. In some examples, the emulator module 114 may implement a virtual machine that emulates hardware components underlying a physical computer different from computing device 102 and computing device 120.

[0022] The game application 112 may be executed in the virtualized environment provided by the respective emulator module 114 of the computing device 120 to perform functions of a video game. Examples of the game application 112 may include action games emphasizing hand-eye coordination and reaction time, such as first-person shooting games, simulation games, such as motor sports simulation games or flight simulator games, role-playing games (e.g., large-scale multiplayer role-playing games), network multiplayer games, single-player games, and the like.

[0023] Game application 112 is executed in a virtualized environment provided by an emulator module 114 of a computing device, such as computing device 120A. Therefore, game application 112 can output image data for display on a display device included in or operatively coupled to the computing device. In some examples, the image data can be frames of graphics that game application 112 outputs for display on the display device during execution of game application 112. For example, the image data can include frames of graphics for an interactive game play environment, frames of graphics for a loading screen, frames of graphics for a menu screen, and the like.

[0024] Game application 112 can render and output image data according to a desired target frame rate, such as a target number of frames per second (fps), at which game application 112 outputs the image data. The frame rate of the image data output by game application 112 can be the rate at which game application 112 outputs frames of graphics. Examples of the frame rate at which game application 112 outputs image data can include 30 fps, 60 fps, 120 fps, 144 fps, and the like.

[0025] The application catalog module 116 runs on the computing device 102 and can download and install applications on the computing device 102. For example, the application catalog module 116 can download and install applications such as a game application 112 that runs in a virtualized environment provided by the emulator module 114 on the computing device 102. In some examples, the application catalog module 116 can output a graphical user interface (GUI) for display on the UIC 104, presenting a catalog of applications that can be downloaded and installed on the computing device 102, such as a list of recommended or suggested applications that can be downloaded and installed on the computing device 102.

[0026] The application catalog module 116 can determine how well a game application, such as game application 112, might perform when run in the virtualized environment provided by the emulator module 114 of the computing device 102, and can determine whether to recommend or suggest the game application based on the determination of how well it might perform when run in the virtualized environment of the computing device 102. The application catalog module 116 can determine that a game application might perform well if, when run in the virtualized environment of the computing device 102, it has the potential to render and output almost all frames of image data at the game application's target frame rate.

[0027] The application catalog module 116 can determine game applications that may perform well on the computing device 102 and may recommend or suggest installing those game applications on the computing device 102. Similarly, the application catalog module 116 may recommend or not recommend game applications that may not perform well on the computing device 102. The application catalog module 116 can be accessed.

[0028] Different computing devices can differ in a wide variety of ways, including having different central processing units (CPUs) with different numbers of processing cores and / or running at different clock speeds, different graphics processing units (GPUs), different amounts of physical memory, and so on. The application catalog module 116 may have access to performance data for game applications running on some configurations of the computing device, but it may not have access to performance data for game applications running on a particular configuration of the computing device 102.

[0029] According to aspects of this disclosure, the application catalog module 116 may communicate with the computing system 150 to query the computing system 150 for a catalog of game applications, such as a list of game applications that can be run in a virtualized environment provided by the emulator module 114 of the computing device 102. The application catalog module 116 may include, as part of the query, an indication of one or more device characteristics of the computing device 102.

[0030] The application distribution module 172 can receive queries from the application catalog module 116 and, in response, determine a list of game applications to be included in the game application catalog. For example, the application distribution module 172 may decide to list game application 112 in the catalog of game applications that can be installed on the computing device 102.

[0031] The application distribution module 172 can determine the predicted performance of each game application listed in the catalog provided to the application catalog module 116 when it is run on the computing device 102. For game applications that run in a virtualized environment provided by the emulator module 114, the application distribution module 172 can determine the predicted performance of the game application when it is run in the virtualized environment of the computing device 102.

[0032] Therefore, the application distribution module 172 may query the game performance module 162 for the predicted performance of the game application 112 when it is run in the virtualized environment of the computing device 102. The query may include one or more device characteristics of the computing device 102, and one or more application characteristics of the game application 112 that the game performance module 162 can use to determine one or more application characteristics of the game application 112.

[0033] The game performance module 162 is trained via machine learning and can determine the predicted performance of the game application 112 when run in a virtualized environment of the computing device 102, based on one or more device characteristics of the computing device 102 and one or more application characteristics of the game application 112. In some examples, the game performance module 162 may implement one or more neural networks trained by machine learning to determine the predicted performance of the game application 112 when run on the computing device 102. Generally, one or more neural networks implemented by the game performance module 162 may include multiple interconnected nodes, each node applying one or more functions to a set of input values ​​corresponding to one or more features and providing one or more corresponding output values. One or more features may be one or more device characteristics of the computing device 102, and one or more corresponding output values ​​of the one or more neural networks may be indicators of the predicted performance of the game application 112 when run on the computing device 102.

[0034] The game performance module 162 is trained to output the predicted performance of the game application 112 when running in the virtualized environment of the computing device 102, in the form of a predicted smoothness score for the game application 112, based on one or more application characteristics of the game application 112 and one or more device characteristics of the computing device 102. The predicted smoothness score may be a value between 0 and 1, corresponding to the percentage of frames rendered by the game application 112 that is predicted to meet the frame time of the target frame rate associated with the game application 112 in the next gameplay session when running in the virtualized environment of the computing device 102.

[0035] As explained above, the game application 112 may have an associated target frame rate, such as 60fps, which is desirable for the game application 112 to output image data. The target frame rate may correspond to the frame time, which is the time (e.g., in milliseconds) it takes for the game application 112 to render frames of image data in order to render them fast enough to meet the target frame rate. In the example where the target frame rate of the game application 112 is 60fps, the frame time to render frames of image data to meet the target frame rate of 60fps would be 16.6 milliseconds. Therefore, the predicted smoothness score may correspond to the percentage of image data frames rendered by the game application 112 that is predicted to meet the frame time of 16.6 milliseconds associated with the target frame rate of 60fps when the game application 112 is running on the computing device 102, in the case of a game application 112 with a target frame rate of 60fps.

[0036] Therefore, in some cases, the game performance module 162 may determine the predicted performance of a game application 112 when run in the virtualized environment of the computing device 102 in the form of a predicted smoothness score. In some cases, the game performance module 162 may classify the predicted performance of a game application 112 when run in the virtualized environment of the computing device 102 based on the predicted smoothness score. For example, the game performance module 162 may compare the predicted smoothness score to a smoothness threshold that can be a value between 0 and 1, such as 0.80. If the game performance module 162 determines that the predicted smoothness score is greater than or equal to the smoothness threshold, the game performance module 162 may determine that the game application 112 can perform well when run in the virtualized environment of the computing device 102. If the game performance module 162 determines that the predicted smoothness score is less than the smoothness threshold, the game performance module 162 may determine that the game application 112 cannot perform well when run in the virtualized environment of the computing device 102.

[0037] The computing system 150 may train one or more neural networks in the game performance module 162 using training data generated from monitoring the performance of a game application 112 running in a virtualized environment provided by the emulator module 114 on a group of computing devices, such as computing device 120, which may have various different device configurations and / or device characteristics. When the game application 112 runs in the virtualized environment provided by the emulator module 114 on computing device 120, the computing system 150 may receive performance metrics from computing device 120 that are associated with the execution of the game application 112 on computing device 120. Such performance metrics may include the time it took the game application 112 to render each frame of image data rendered by the game application 112.

[0038] The computing system 150 may use the collected performance metrics to determine a smoothness score associated with each gameplay session of the game application 112 running in a virtualized environment provided by the emulator module 114 of the computing device 120. The smoothness score associated with a gameplay session may be the percentage of frames of image data rendered by the game application 112 during the gameplay session that meet the frame time of the target frame rate associated with the game application 112, and a gameplay session may extend on the computing device of the computing device 120 from the start of execution of the game application 112 (e.g., opening the game application 112) to the stop of execution of the game application 112 (e.g., until the game application is terminated 112).

[0039] Therefore, the computing system 150 can generate training data for training one or more neural networks of the game performance module 162, for each gameplay session of multiple gameplay sessions on the computing device 120, including a corresponding smoothness score associated with one or more device characteristics of the computing device on which the gameplay session takes place. Thus, the computing system 150 can use the training data to train one or more neural networks of the game performance module 162 and predict the performance score of the game application 112 when run in a virtualized environment of the computing device.

[0040] The game performance module 162 may send an indication of the predicted performance of the game application 112 to the application distribution module 172 in response to determining the predicted performance of the game application 112 when run in the virtualized environment of the computing device 102. Thus, the application distribution module 172 may send an indication of the predicted performance of the game application 112 to the computing device 102. For example, the application distribution module 172 may send the computing device 102 a catalog listing game applications that can be downloaded and installed on the computing device 102, including the game application 112, and the catalog may include an indication of the predicted performance of the game application 112.

[0041] The application catalog module 116 can receive a catalog of game applications from the application distribution module 172 of the computing system 150 and can output a GUI that displays the game applications listed in the catalog for download and installation on the computing device 102, such as for display in the UIC 104. Within the GUI, the application catalog module 116 can present a list of one or more recommended game applications and / or one or more suggested applications in a visually prominent position within the GUI, such as at the top of the GUI's front page.

[0042] The application catalog module 116 may decide whether to recommend or suggest the game application 112 in the GUI based on its predicted performance when run in the virtualized environment of the computing device 102. In an example where the predicted performance of the game application 112 when run in the virtualized environment of the computing device 102 is in the form of a predicted smoothness score, the application catalog module 116 may compare the predicted smoothness score to a smoothness threshold that can be a value between 0 and 1, such as 0.80. If the application catalog module 116 determines that the predicted smoothness score is greater than or equal to the smoothness threshold, it may determine that the game application 112 can deliver excellent performance when run in the virtualized environment of the computing device 102. If the application catalog module 116 determines that the predicted smoothness score is less than the smoothness threshold, it may determine that the game application 112 cannot deliver excellent performance when run in the virtualized environment of the computing device 102.

[0043] The application catalog module 116 may recommend, or otherwise suggest, the game application 112 for installation on the computing device 102 by including an indication of the game application 112 in the GUI's list of recommended or suggested applications, and / or by presenting a visual indication of the game application 112 in a visually prominent location within the GUI, such as at the top of the GUI's front page, in response to determining that the game application 112 is expected to perform well when run in the computing device 102's virtualized environment. Similarly, the application catalog module 116 may refrain from recommending, or otherwise suggesting, the game application 112 for installation on the computing device 102 in response to determining that the game application 112 is not expected to perform well when run in the computing device 102's virtualized environment.

[0044] In some cases, computing device 102 may, in response to determining that a game application 112 is not expected to perform well when run in computing device 102's virtualized environment, adjust one or more graphics settings of the game application 112 installed on computing device 102 to improve the performance of the game application 112 when run in computing device 102's virtualized environment. For example, computing device 102 may reduce the resolution of frames of image data output by computing device 102, reduce the number of textures rendered by the game application 112, or reduce the complexity of graphics rendered by the game application 112.

[0045] When computing device 102 downloads and installs game application 112, computing system 150 may monitor the performance of game application 112 while it is running in the virtualized environment of computing device 102. For example, computing system 150 may monitor performance metrics of game application 112 while it is running in the virtualized environment of computing device 102 to determine the actual performance of game application 112, such as the actual smoothness score of game application 112 while it is running in the virtualized environment of computing device 102. Computing system 150 may retrain or fine-tune one or more neural networks of game performance module 162 based on the actual performance of game application 112, such as the actual smoothness score of game application 112 while it is running in the virtualized environment of computing device 102, so that it can more accurately determine the predicted performance of game application 112 when it is running in the virtualized environment of computing device 102.

[0046] The technology of this disclosure provides one or more technical advantages. Emulators that provide a virtualized environment for game applications can run game applications on a wide variety of computing devices having different operating systems and / or processing architectures than the operating system and / or processing architecture on which the game application was designed. Therefore, when a game application runs on a wide variety of configurations of the operating system and / or processing architecture on which it is run, it can be difficult for a computing system to collect performance data for the game application. By using one or more neural networks trained using machine learning to determine the predictive performance of a game application on a computing device, the technology of this disclosure can enable a computing system to more accurately predict the performance of a game application when it runs on a computing device, even if it does not have performance data on the performance of the game application when it runs on a particular configuration of the computing device, thereby improving the technical field for predicting the performance of game applications.

[0047] Furthermore, by using one or more neural networks trained using machine learning to determine the predicted performance of a game application on a computing device, the technology of this disclosure may enable the computing device to adjust one or more parameters of the game application, such as the graphics quality of the game application, to improve the performance of the game application when it runs on the computing device, in response to the computing device determining that the game application on the computing device is not expected to perform well. This may improve the user experience of the game application running on the computing device by making the game application perceived by the user as smooth and responsive.

[0048] Figure 2 is a block diagram illustrating an exemplary computing system according to one or more embodiments of the present disclosure. Figure 2 shows only one specific example of computing system 250, and many other examples of computing system 250 may be used in other examples and may include a subset of the components included in the exemplary computing system 250, or may include additional components not shown in Figure 2. Computing system 250 may be an example of computing system 150 in Figure 1.

[0049] As shown in the example in Figure 2, the computing system 250 includes one or more processors 240, one or more input devices 242, one or more communication units 244, one or more output devices 246, and one or more storage devices 248. The one or more processors 240 may be an example of the one or more processors 108 in Figure 1. The one or more input devices 242 and one or more output devices 246 may be an example of the UIC 104 in Figure 1. The storage device 248 of the computing system 250 also includes an operating system 226 and a game performance module 262. The communication channel 252 may interconnect each of the components 240, 242, 244, 246, and 248 for inter-component communication (physical, communicative, and / or operational). In some examples, the communication channel 252 may include a system bus, a network connection, one or more inter-process communication data structures, or any other components for communicating data between hardware and / or software.

[0050] One or more processors 240 may implement functions and / or execute instructions within the computing system 250. For example, a processor 240 on computing device 102 may receive and execute instructions stored by storage device 248, which provides functions for the operating system 226, game performance module 262, metrics recording module 266, metrics processing module 268, training module 270, application distribution module 272, and dashboard module 278. These instructions executed by processor 240 may cause the computing system 250 to store and / or modify information in storage device 48 during program execution. The processor 240 may execute instructions for the operating system, game performance module 262, metrics recording module 266, metrics processing module 268, training module 270, application distribution module 272, and dashboard module 278. In other words, the operating system 226, game performance module 262, metrics recording module 266, metrics processing module 268, training module 270, application distribution module 272, and dashboard module 278 are operated by the processor 240 and can perform various functions described herein.

[0051] One or more processors 240 are or may include digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), and / or other equivalent integrated circuits or discrete logic circuits. One or more input devices 242 of the computing system 250 may receive inputs. Examples of inputs include, but are not limited to, haptic inputs, audio inputs, motor inputs, and optical inputs. One or more output devices 246 of the computing device 102 may generate outputs. Examples of outputs include haptic outputs, audio outputs, and video outputs.

[0052] One or more communication units 244 of the computing system 250 may communicate with external devices by transmitting and / or receiving data. For example, the computing system 250 may use the communication units 244 to transmit and / or receive radio signals over a radio network, such as a cellular radio network. In some examples, the communication units 244 may transmit and / or receive satellite signals over a satellite network, such as a Global Positioning System (GPS) network. Examples of communication units 244 include network interface cards (e.g., Ethernet® cards), optical transceivers, radio frequency transceivers, GPS receivers, or any other type of device capable of transmitting and / or receiving information. Other examples of communication units 44 may include Bluetooth®, GPS, 3G, 4G, Wi-Fi® radios, and Universal Serial Bus (USB) controllers found in mobile devices.

[0053] One or more storage devices 248 within the computing system 250 may store information for processing during the operation of the computing system 250. In some examples, the storage device 248 is a temporary storage device, meaning that its primary purpose is not long-term storage. The storage device 248 on the computing system 250 is configured to store information for short periods as volatile memory, and therefore, when deactivated, the stored content may no longer be retained. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art.

[0054] The storage device 248 may, in some examples, also include one or more computer-readable storage media. The storage device 248 may be configured to store larger amounts of information than volatile memory. The storage device 248 may also be configured to store information long-term as a non-volatile memory space, so that the information can be retained after activation / off-cycle. Examples of non-volatile memory include magnetic hard disks, optical disks, floppy disks, flash memory, or electrically programmable memory (EPROM) and electrically erasable and programmable memory (EEPROM). The storage device 248 may store program instructions and / or data associated with the operating system 226, the game performance module 262, the metrics recording module 266, the metrics processing module 268, the training module 270, the application distribution module 272, and the dashboard module 278.

[0055] According to the technology of this disclosure, the computing system 250 is configured to receive indications of one or more device characteristics of a computing device (for example, the computing device 102 shown in Figure 1) from a computing device via one or more communication units 244. For example, one or more processors 240 are configured to run an application distribution module 272 to receive queries from the computing device for a catalog of game applications that can be installed on the computing device, and the queries received from the computing device may include one or more device characteristics of the computing device.

[0056] An indication of one or more device characteristics of a computing device may include an indication of one or more CPU characteristics of the computing device. One or more CPU characteristics of a computing device may include any combination of the number of CPUs in the computing device, the number of physical processing cores of one or more CPUs, the number of logical processing cores of one or more CPUs, the model of one or more CPUs, and / or the initial clock speed of one or more CPUs (e.g., base clock speed or nominal clock speed) and / or the maximum clock speed of one or more CPUs.

[0057] An indication of one or more device characteristics of a computing device may also include an indication of one or more GPU characteristics of the computing device. The GPU characteristics of a computing device may include any combination of the manufacturer (e.g., manufacturer name) and / or model of one or more GPUs of the computing device, the video RAM (VRAM) size of one or more GPUs, the memory bandwidth of one or more GPUs, the current clock speed of one or more GPUs, the maximum clock speed of one or more GPUs, the heap size allocated to the memory of one or more GPUs, and / or the version number of the GPU driver installed on the computing device.

[0058] An indication of one or more device characteristics of a computing device may also include an indication of one or more memory characteristics of the computing device. One or more memory characteristics of a computing device may include any combination of the amount of memory of the computing device, such as the total amount of physical RAM installed in the computing device, the type of disk in the computing device, the amount of physical RAM available in the computing device, the number and / or details of the physical channels of RAM in the computing device, and / or the clock speed of RAM in the computing device.

[0059] Indications of one or more device characteristics of a computing device may, in some examples, also include indications of the version of the computing device's operating system and / or the language in which the operating system is localized (e.g., English, French, etc.).

[0060] In some examples, an indication of one or more device characteristics of a computing device may include one or more display characteristics of the computing device. One or more display characteristics of a computing device may include any combination of the display resolution and / or refresh rate of the display of the computing device's display (e.g., UIC104 shown in Figure 1).

[0061] In an example where a game application is run on a computing device in a virtualized environment using an emulator (for example, emulator module 114 shown in Figure 1), an indication of one or more device characteristics of the computing device may include an indication of one or more characteristics of the emulator used to run the game application on the computing device. One or more characteristics of the emulator may include any combination of the emulator's available memory size, which may be the emulator's version, the total amount of computing device memory available and / or allocable by the emulator, and / or the amount of computing device memory available to the emulator.

[0062] One or more processors 240 are configured to run an application distribution module 272 to query a game performance module 262 for the predicted performance of each of several game applications included in a catalog of game applications sent to the computing device when run in the computing device's virtualization environment (for example, the virtualization environment provided by the emulator module 114 shown in Figure 1). For example, the application distribution module 272 may send a query to the game performance module 262 for the predicted performance of the game application 112 shown in Figure 1 when run in the computing device's virtualization environment. The query may include an indication of one or more device characteristics of the computing device received from the computing device.

[0063] In some examples, a query may include an indication of one or more application characteristics of a game application. This indication may include an indication of the game application's name, such as the game application's software package name, and / or the version number of the software package associated with the game application. In some examples, the application characteristics of a game application may also include any combination of graphics application programming interfaces (APIs) used by the game application, such as whether the game application uses a cross-platform graphics API (e.g., Vulkan ANGLE) that is translated to the computing device's native graphics API, or whether the game application uses a graphics API native to the computing device (e.g., Direct3D), the game engine used by the game application, the resolution of the frames of image data output by the game application, and / or the operating system and / or processor architecture on which the game application runs.

[0064] One or more processors 240 of the computing system 250 are configured to run a game performance module 262 to determine the predicted performance of a game application when it is run in the virtualized environment of the computing device, based on one or more device characteristics of the computing device and one or more application characteristics of the game application. In other words, the computing system 250 can use the game performance module 262 to predict how well a game application will perform when it is run on the computing device.

[0065] The predicted performance of a game application when running in a virtualized computing device environment may be a value that corresponds to how well the game application is likely to perform when running in that environment. In some examples, predicted performance may include or correspond to frame rate information. Frame rate information may include frame time, which in some examples is the time (e.g., in milliseconds) that the game application is expected to take to render a frame of image data while running on the computing device.

[0066] In some examples, one or more processors 240 are configured to run a game performance module 262 to determine the predicted performance of a game application when running on a computing device in the form of a predicted smoothness score. The predicted smoothness score can be a value between 0 and 1, corresponding to the percentage of frames of image data rendered by the game application when running on the computing device that are predicted to meet the frame time of the target frame rate associated with the game application in the next gameplay session. For example, a predicted smoothness score of 0.5 may indicate that 50% of the frames of image data rendered by the game application when running on the computing device are predicted to meet the frame time of the target frame rate associated with the game application in the next gameplay session.

[0067] As explained above, a game application may have an associated target frame rate, such as 60fps, which is desirable for the game application to output image data. The target frame rate can correspond to the frame time, which is the time (e.g., in milliseconds) it takes for the game application to render frames of image data in order to render them fast enough to meet the target frame rate. In the example where the target frame rate of the game application is 60fps, the frame time to render frames of image data to meet the target frame rate of 60fps would be 16.6 milliseconds. Therefore, the predicted smoothness score can correspond to the percentage of image data frames rendered by the game application that is predicted to meet the target frame rate of 60fps when the game application is run on the computing device, in the case of a game application with a target frame rate of 60fps.

[0068] In some examples, one or more processors 240 are configured to run a game performance module 262 to determine the predicted performance of a game application when run on a computing device in the form of a predicted smoothness score.

[0069] One or more processors 240 are configured to run the game performance module 262 and, in response to determining the predicted performance of the game application when run on the computing device, send an indication of the predicted performance of the game application to the application distribution module 272. One or more processors 240 are configured to run the application distribution module 272 to generate a catalog of game applications, for each game application in the catalog, which includes an indication of the associated predicted performance of the game application when run on the computing device. Thus, the application distribution module 272 may use one or more communication units 244 to send the catalog of game applications to the computing device, thereby sending an indication of the associated predicted performance of the game application when run on the computing device to the computing device.

[0070] In some examples, the game performance module 262 may include a game performance model 264 that implements one or more neural networks trained by machine learning to determine the predictive performance of a game application when run in a virtualized environment of a computing device. Generally, the one or more neural networks implemented by the game performance model 264 may include multiple interconnected nodes, each node which may apply one or more functions to a set of input values ​​corresponding to one or more features and provide one or more corresponding output values. The one or more features may include indicators of one or more application characteristics of the game application and indicators of one or more device characteristics of the computing device, and the one or more corresponding output values ​​of the one or more neural networks may be indicators of the predictive performance of the game application, such as a predictive smoothness score when run in a virtualized environment of a computing device.

[0071] The game performance model 264 can be trained to determine the predicted performance of a game application (e.g., game application 112 shown in Figure 1) when run on a computing device (e.g., computing device 102 shown in Figure 1), using training data that includes data collected from a group of computing devices (e.g., computing device 120 shown in Figure 1), each running a copy of the same game application (e.g., game application 112 shown in Figure 1). The data collected from the group of computing devices may include, for each of the group of computing devices, one or more device characteristics of the corresponding computing device, such as one or more device characteristics described above.

[0072] The data collected from a group of computing devices may also include, for each group of computing devices, performance metrics for the game application running on that group. When a copy of the game application runs on a group of computing devices, one or more processors 240 are configured to run a metrics recording module 266 to receive performance metrics for the copy of the game application from the group of computing devices, and the metrics recording module 266 may store the received performance metrics in a metrics data store 274, which may be any suitable structured data store, such as a database.

[0073] Performance metrics received from a group of computing devices may include frame-time metrics relating to frames of image data rendered and output by each copy of the game application running on the group of computing devices, such as the frame time of each frame of image data rendered and output by each copy of the game application running on the group of computing devices. Performance metrics may also include indicators of the start and end of gameplay sessions by each copy of the game application running on the group of computing devices.

[0074] One or more processors 240 are configured to execute a metrics processing module 268 to process performance metrics stored in the metrics data store 274. The metrics processing module 268 may group the received frame-time metrics by gameplay session and determine frame-time metrics for each gameplay session, using indications for the start and end of a gameplay session. That is, for each gameplay session of a copy of a game application running on a group of computing devices, the metrics processing module 268 may determine frame-time metrics for frames rendered by the copy of the game application during that gameplay session.

[0075] Therefore, the metrics processing module 268 can determine a smoothness score for each gameplay session of a copy of the game application running on a group of computing devices, based on the frame-time metrics associated with the gameplay session. For example, the metrics processing module 268 can calculate the smoothness score for a gameplay session as the percentage of frames of image data rendered and output by the corresponding copy of the game application during the gameplay session that satisfy the frame time of the target frame rate associated with the game application.

[0076] The metrics processing module 268 may store, for each gameplay session of a copy of the game application running on a group of computing devices, a smoothness score associated with the gameplay session, one or more device characteristics of the computing device on which the gameplay session occurred, and a unique gameplay session identifier in a training data store 276, which is any appropriate structured data store such as a database or table. In this way, the computing system 250 may generate a set of training data including associations between the device characteristics of the computing devices and the smoothness score for each gameplay session, and one or more processors 40 may be configured to run a training module 270 for training a game performance model 264 to predict the performance of the game application when run on the computing devices via any appropriate machine learning technique.

[0077] In some examples, the computing system 250 may continuously collect data such as frame-time metrics and device characteristics from a group of computing devices (e.g., computing device 120 shown in Figure 1) each running a copy of the same game application (e.g., game application 112), and the training module 270 may use the collected data to periodically retrain or fine-tune the game performance model 264 based on the collected data.

[0078] For example, as described above, the metrics processing module 268 may determine a smoothness score associated with each of several gameplay sessions from the collected data. The game performance model 264 may also determine predictive performance, such as a predicted smoothness score, for each of the several gameplay sessions based on one or more device characteristics associated with the gameplay session. Thus, the training module 270 may retrain or fine-tune the game performance model 264 using any appropriate machine learning technique to minimize the difference between the smoothness score associated with a game session and the predicted smoothness score for that game session for each game session.

[0079] In some examples, one or more processors 240 are configured to run a dashboard module 278 to display on one or more output devices 246 or output a model accuracy dashboard to an external computing device. The model accuracy dashboard may present, for each gameplay session of multiple gameplay sessions of a game application, the predicted smoothness score predicted using the game performance model 264 and the actual predicted smoothness score associated with the gameplay session, thereby enabling users, such as the developers of the game performance model 264, to visualize the performance of the game performance model 264.

[0080] Figures 3A to 3E are conceptual diagrams illustrating an example of a machine learning model according to the embodiments of this disclosure. Figures 3A to 3E are described below in the context of the game performance model 264 in Figure 2. For example, in some cases, machine learning model 322 may be an example of the game performance model 264.

[0081] Figure 3A shows a conceptual diagram of an example of a machine learning model according to an embodiment of the present disclosure. As shown in Figure 3A, in some implementations, the machine learning model 322 is trained to receive one or more types of input data and, in response, provide one or more types of output data. Thus, Figure 3A shows the machine learning model 322 performing inference.

[0082] The input data may include one or more features associated with an example or embodiment. In some embodiments, one or more features associated with an example or embodiment can be organized into a feature vector. For example, the game performance model 264 may receive a feature vector containing one or more features of a game application and a feature vector containing one or more features of a computing device. In some embodiments, the output data may include one or more predictions. Predictions may also be called estimates. Thus, given features associated with a particular example, the machine learning model 322 can output predictions for such an example based on those features. For example, given a feature vector containing one or more features of a game application and a feature vector containing one or more features of a computing device, the game performance model 264 may output a predicted performance of the game application when run on the computing device.

[0083] The machine learning model 322 may be one or more of various different types of machine learning models, or may include them. In particular, in some embodiments, the machine learning model 322 can perform classification, regression, clustering, anomaly detection, recommendation generation, and / or other tasks.

[0084] In some embodiments, the machine learning model 322 can perform various types of classification based on the input data. For example, the machine learning model 322 can perform two-class classification or multi-class classification. In two-class classification, the output data may include the input data classified into one of two different classes. In multi-class classification, the output data may include the input data classified into one (or more) of three or more classes. The classification can be single-label or multi-label. The machine learning model 322 can perform discrete categorical classification, which simply classifies the input data into one or more classes or categories.

[0085] In some embodiments, the machine learning model 322 can perform regressions to provide output data in the form of continuous numerical values. These continuous numerical values ​​can represent any number of different metrics or numerical representations, such as currency values, scores (e.g., performance scores), or other numerical representations. For example, the machine learning model 322 can perform linear regression, polynomial regression, or nonlinear regression. For example, the machine learning model 322 can perform simple regression or multiple regression. As described above, in some embodiments, a softmax function or other function or layer can be used to compress a set of real values, each associated with three or more possible classes, into a set of real values ​​in the range (0,1) where the sum is 1.

[0086] The machine learning model 322 can perform various types of clustering. For example, the machine learning model 322 can identify one or more previously defined clusters that best correspond to the input data. The machine learning model 322 can identify one or more clusters within the input data. That is, if the input data contains multiple objects, documents, or other entities, the machine learning model 322 can sort the multiple entities contained in the input data into multiple clusters. In some embodiments in which the machine learning model 322 performs clustering, the machine learning model 322 can be trained using unsupervised learning techniques.

[0087] The machine learning model 322 can perform anomaly detection or outlier detection. For example, the machine learning model 322 can identify input data that does not conform to expected patterns or other characteristics (e.g., those previously observed from previous input data). As an example, anomaly detection can be used for fraud detection or system failure detection.

[0088] In some embodiments, the machine learning model 322 can provide output data in the form of one or more recommendations. For example, the machine learning model 322 can be incorporated into a recommendation system or engine. As an example, given input data describing previous results for a particular entity (e.g., a score, ranking, or rating indicating the degree of success or enjoyment), the machine learning model 322 can output suggestions or recommendations for one or more additional entities that are expected to produce a desired result (e.g., derive a score, ranking, or rating indicating success or enjoyment) based on the previous results.

[0089] In some embodiments, the machine learning model 322 may be a parametric model, while in other embodiments, the machine learning model 322 may be a nonparametric model. In some embodiments, the machine learning model 322 may be a linear model, while in other embodiments, the machine learning model 322 may be a nonlinear model.

[0090] As explained above, machine learning model 322 can be one or more of various different types of machine learning models, or may include them. Examples of such various types of machine learning models are shown below for illustrative purposes. One or more of the exemplary models described below can be used (for example, in combination) to provide output data in response to input data. Additional models other than the exemplary models provided below can also be used.

[0091] In some embodiments, the machine learning model 322 may be one or more classification models, such as a linear classification model or a quadratic classification model, or include them. The machine learning model 322 may be one or more regression models, such as a simple linear regression model, a multiple linear regression model, a logistic regression model, a stepwise regression model, a multivariate adaptive regression spline, or a locally estimated scatter plot smoothing model, or include them.

[0092] In some examples, the machine learning model 322 may be one or more decision tree-based models, such as classification and / or regression trees, iterative binary 3 decision trees, C4.5 decision trees, chi-squared auto-interaction detection decision trees, decision stocks, and conditional decision trees.

[0093] The machine learning model 322 may be one or more kernel machines, or may include one or more kernel machines. In some embodiments, the machine learning model 322 may be one or more support vector machines, or may include one or more support vector machines. The machine learning model 322 may be one or more instance-based learning models, such as a learning vector quantization model, a self-organizing map model, or a locally weighted learning model, or may include them. In some embodiments, the machine learning model 322 may be one or more nearest neighbor models, such as a k-nearest neighbor classification model or a k-nearest neighbor regression model, or may include them. The machine learning model 322 may be one or more Bayesian models, such as a naive Bayes model, a Gaussian naive Bayes model, a polynomial naive Bayes model, a mean-one dependent estimator, a Bayesian network, a Bayesian belief network, or a hidden Markov model, or may include them.

[0094] In some embodiments, the machine learning model 322 may be one or more artificial neural networks (also simply called neural networks), or may include one or more artificial neural networks. A neural network may include a group of connected nodes, also called neurons or perceptrons. A neural network may be organized into one or more layers. A neural network including multiple layers may be called a “deep” network. A deep network may include an input layer, an output layer, and one or more hidden layers placed between the input and output layers. The nodes of a neural network may be fully connected or not fully connected.

[0095] The machine learning model 322 may be one or more feedforward neural networks, or may include them. In a feedforward network, the connections between nodes do not form cycles. For example, each connection can connect a node in the previous layer to a node in the later layer.

[0096] In some cases, the machine learning model 322 may be or include one or more recurrent neural networks. In some cases, at least some of the nodes of the recurrent neural network may form a cycle. Recurrent neural networks can be particularly useful when processing inherently continuous input data. Specifically, in some cases, a recurrent neural network can use recurrent or directed circular node connections to pass or retain information from earlier to later parts of an input data sequence.

[0097] In some examples, sequential input data can include time-series data (e.g., sensor data over time or images captured at different times). For example, a recurrent neural network can analyze sensor data over time to detect or predict swipe direction or perform handwriting recognition. Sequential input data can also include words in a sentence (e.g., natural language processing, speech detection or processing), musical notes in a song, sequential actions performed by a user (e.g., detecting or predicting usage of a sequential application), and the state of a sequential object.

[0098] Exemplary recurrent neural networks include long-short-term (LSTM) recurrent neural networks, gated recurrent units, bidirectional recurrent neural networks, continuous-time recurrent neural networks, neural history compressors, echo state networks, Elman networks, Jordan networks, recurrent neural networks, Hopfield networks, fully recurrent networks, and inter-sequence configurations.

[0099] In some embodiments, the machine learning model 322 may be or include one or more convolutional neural networks. In some cases, the convolutional neural network may include one or more convolutional layers that perform convolutions on input data using learned filters.

[0100] Filters are also called kernels. Convolutional neural networks can be particularly useful for vision problems, such as when the input data includes images, whether still images or videos. However, convolutional neural networks can also be applied to natural language processing.

[0101] In some examples, the machine learning model 322 may be or include one or more generative networks, such as a generative adversarial network. Generative networks can be used to generate new data, such as new images or other content.

[0102] A machine learning model 322 may be or may include an autoencoder. In some cases, the purpose of an autoencoder is typically to learn a representation (e.g., a low-dimensional encoding) of a dataset, for the purpose of dimensionality reduction. For example, in some cases, an autoencoder attempts to encode the input data and provide output data that reconstructs the input data from the encoding. Recently, the concept of autoencoders has become more widely used to train generative models of data. In some cases, an autoencoder may include additional losses beyond the reconstruction of the input data.

[0103] The machine learning model 322 may be one or more other forms of artificial neural networks, such as a deep Boltzmann machine, a deep belief network, or a stacked autoencoder, or may include them. Any of the neural networks described herein can be combined (e.g., stacked) to form more complex networks.

[0104] One or more neural networks can be used to provide embeddings based on input data. For example, an embedding can be a representation of knowledge abstracted from the input data into one or more learned dimensions. In some cases, embeddings can be a useful source for identifying relevant entities. In some cases, embeddings can be extracted from the output of the network, but in other cases, they can be extracted from any hidden node or layer of the network (e.g., a layer near the end of the network but not the final layer). Embeddings can be useful for performing tasks such as automatic video recommendations, product recommendations, and entity or object recognition. In some cases, embeddings can be useful input to downstream models. For example, embeddings can be useful for generalizing input data (e.g., search queries) to downstream models or processing systems.

[0105] The machine learning model 322 may include one or more clustering models, such as a k-means clustering model, a k-median clustering model, an expected value maximization model, or a hierarchical clustering model.

[0106] In some embodiments, the machine learning model 322 can perform one or more dimensionality reduction techniques, such as principal component analysis, kernel principal component analysis, graph-based kernel principal component analysis, principal component regression, partial least squares regression, summon mapping, multidimensional scaling, projection tracking, linear discriminant analysis, mixture discriminant analysis, quadratic discriminant analysis, generalized discriminant analysis, flexible discriminant analysis, and automatic encoding.

[0107] In some embodiments, the machine learning model 322 may perform or be subject to one or more reinforcement learning techniques, such as Markov decision processes, dynamic programming, Q-function or Q-learning, value function methods, deep Q networks, differentiable neural computers, asynchronous advantage acquaintricritics, and deterministic policy gradients.

[0108] In some embodiments, the machine learning model 322 can be an autoregressive model. In some cases, the autoregressive model can be specified so that the output data depends linearly on its own previous values ​​and probability terms. In some cases, the autoregressive model can take the form of a stochastic difference equation. An example of an autoregressive model is WaveNet, a model that generates raw speech.

[0109] In some embodiments, a machine learning model 322 may be part of, or form part of, a multi-model ensemble. As an example, bootstrap aggregation, also known as "bagging," can be performed. In bootstrap aggregation, the training dataset is divided into several subsets (e.g., via random sampling by substitution), and multiple models are each trained on several subsets. At inference time, the outputs of each of the multiple models can be combined (e.g., via averaging, voting, or other techniques) and used as the output of the ensemble.

[0110] One example of an ensemble is the random forest, also known as a random decision forest. A random forest is an ensemble learning method for classification, regression, and other tasks. A random forest is generated by generating multiple decision trees during training. In some cases, at inference, the forest output can be the mode of a class (classification) or the average prediction of the individual trees (regression). Random decision forests can correct the tendency of decision trees to overfit to the training set.

[0111] Another example of an ensemble technique is stacking, sometimes called stack generalization. Stacking involves training a combiner model to blend or combine predictions from multiple other machine learning models. Thus, multiple machine learning models (e.g., of the same or different types) can be trained based on training data. Furthermore, the combiner model can be trained to take predictions from other machine learning models as input and, in response, generate a final inference or prediction. In some cases, a single-layer logistic regression model can be used as the combiner model.

[0112] Another example of an ensemble technique is boosting. Boosting can involve building an ensemble incrementally by repeatedly training weaker models and adding them to the final stronger model. For example, each new model can be trained to highlight training examples that previous models misinterpreted (e.g., misclassified). For example, the weights associated with each of these misinterpreted examples can be increased. One common embodiment of boosting is AdaBoost, also known as adaptive boosting. Other exemplary boosting techniques include LPBoost, TotalBoost, BrownBoost, xgboost, MadaBoost, LogitBoost, and gradient boosting. Furthermore, ensembles can also be formed by combining any of the models described above (e.g., regression models and artificial neural networks). As an example, an ensemble may include a top-level machine learning model or heuristic function that combines and / or weights the outputs of the models that make up the ensemble.

[0113] In some embodiments, multiple machine learning models (e.g., forming an ensemble) can be linked and trained together (e.g., via backpropagation through a sequential model ensemble). However, in some embodiments, only a subset (e.g., one) of the jointly trained models is used for inference.

[0114] In some embodiments, the machine learning model 322 can be used to preprocess input data and then feed it into another model. For example, the machine learning model 322 can perform dimensionality reduction techniques and embeddings (e.g., matrix decomposition, principal component analysis, singular value decomposition, word2vec / GLOVE, and / or related methods), clustering, and even classification and regression for downstream consumption. Many of these techniques have been described above and will be discussed further below.

[0115] As described above, the machine learning model 322 may be trained or otherwise configured to receive input data and, in response, provide output data. The input data may include input data of various types, formats, or variations. For example, in various embodiments, the input data may include content (or a portion of content) initially selected by the user, e.g., the content of a document or image selected by the user, links pointing to the user's selection, links within the user selection relating to other files available on the device or in the cloud, and features describing the metadata of the user selection. Furthermore, with the user's permission, the input data may include context of the user's usage obtained from the app itself or other sources. Examples of usage context include the scope of sharing (public, or to a large group, or private, or shared with a specific person), the context of sharing, etc. If the user allows it, additional input data may include device state, e.g., the device's location, apps running on the device, etc.

[0116] In some embodiments, the machine learning model 322 can receive and use the input data in its raw form. In some embodiments, the raw input data may be preprocessed. Thus, the machine learning model 322 can receive and use preprocessed input data in addition to, or instead of, the raw input data.

[0117] In some embodiments, preprocessing the input data may include extracting one or more additional features from the raw input data. For example, feature extraction techniques can be applied to the input data to generate one or more new additional features. Exemplary feature extraction techniques include edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transformation, motion detection, optical flow, and Huff transform.

[0118] In some embodiments, the extracted features may include, or be derived from, transformations of the input data to other domains and / or dimensions. For example, the extracted features may include, or be derived from, transformations of the input data to the frequency domain. For instance, wavelet transforms and / or fast Fourier transforms can be performed on the input data to generate additional features.

[0119] In some embodiments, the extracted features may include statistics calculated from the input data, or from specific parts or dimensions of the input data. Exemplary statistics may include the mode, mean, maximum, minimum, or other metrics of the input data or a portion thereof.

[0120] In some embodiments, the input data may be sequential, as described above. In some cases, sequential input data can be generated by sampling or otherwise segmenting a stream of input data. For example, frames can be extracted from a video. In some embodiments, sequential data can be made non-sequential by summarization.

[0121] Another exemplary preprocessing technique involves interpolating a portion of the input data. For example, additional synthetic input data can be generated through interpolation and / or extrapolation.

[0122] As another exemplary preprocessing technique, some or all of the input data can be scaled, standardized, normalized, generalized, and / or regularized. Examples of regularization methods include ridge regression, least absolute contraction and selection operators (LASSO), elastic networks, minimum angle regression, cross-validation, L1 regularization, and L2 regularization. As an example, some or all of the input data can be normalized by subtracting the mean of all feature values ​​of a given dimension from each feature value, and then dividing by the standard deviation or other metric.

[0123] Another exemplary preprocessing technique involves quantizing or discretizing part or all of the input data. In some cases, qualitative features or variables in the input data can be converted into quantitative features or variables. For example, one-hot encoding can be performed.

[0124] In some cases, dimensionality reduction techniques can be applied to the input data before inputting it into the machine learning model 322. For example, several examples of dimensionality reduction methods are provided above, including principal component analysis, kernel principal component analysis, graph-based kernel principal component analysis, principal component regression, partial least squares regression, summon mapping, multidimensional scaling, projection tracking, linear discriminant analysis, mixture discriminant analysis, quadratic discriminant analysis, generalized discriminant analysis, flexible discriminant analysis, and auto-encoding.

[0125] In some embodiments, during training, the input data may be intentionally transformed in any number of ways to improve the robustness, generalization, or other qualities of the model. Exemplary techniques for transforming the input data include adding noise, changing color, shading, or hue, scaling, segmenting, and amplification.

[0126] In response to receiving input data, the machine learning model 322 can provide output data. The output data may include various types, formats, or variations of output data. For example, in various embodiments, the output data may include relevant shareable content, along with initial content selections, that is stored locally on the user's device or in the cloud.

[0127] As described above, in some embodiments, the output data may include various types of classification data (e.g., two-class classification, multi-class classification, single-label, multi-label, discrete classification, regression classification, probability classification, etc.) or various types of regression data (e.g., linear regression, polynomial regression, nonlinear regression, simple regression, multiple regression, etc.). In other examples, the output data may include clustering data, anomaly detection data, recommendation data, or other forms of output data described above.

[0128] In some embodiments, the output data may influence downstream processes or decisions. For example, in some embodiments, the output data may be interpreted by and / or acted upon by rule-based regulators.

[0129] This disclosure provides a system and method that includes, or otherwise utilizes, one or more machine learning models to output a predictive performance score based on the characteristics of a game application and a computing device. Any of the different types or formats of input data described above can be combined with any of the different types or formats of machine learning models described above to provide any of the different types or formats of output data described above.

[0130] The systems and methods of this disclosure may be implemented by one or more computing devices, such as computing system 150 and computing system 250, or otherwise may be executed on such devices. Exemplary computing devices include user computing devices (e.g., mobile computing devices such as laptops, desktop computers, tablets, smartphones, and wearable computing devices), embedded computing devices (e.g., devices embedded in vehicles, cameras, image sensors, industrial machinery, satellites, game consoles or controllers, or household appliances such as refrigerators, thermostats, energy meters, home energy managers, and smart home assistants), server computing devices (e.g., database servers, parameter servers, file servers, mail servers, print servers, web servers, game servers, and application servers), dedicated special model processing or training devices, virtual computing devices, other computing devices or computing infrastructure, or combinations thereof.

[0131] Figure 3B shows a conceptual diagram of a computing device 302, which is an example of the computing device 102 in Figure 1. The computing device 302 includes a processing component 340, a memory component 304, and a machine learning model 322. The computing device 302 may store and implement the machine learning model 322 locally (i.e., on the device). Thus, in some embodiments, the machine learning model 322 may be stored and / or implemented locally on a user computing device such as an embedded device or a mobile device. Output data obtained through the local implementation of the machine learning model 322 on an embedded device or a user computing device may be used to improve the performance of the embedded device or a user computing device (e.g., an application implemented by the embedded device or a user computing device).

[0132] Figure 3C shows a conceptual diagram of an exemplary server computing system containing a machine learning model and an exemplary client computing device that can communicate over a network. Figure 3C includes a client device 302A that communicates with computing system 350 over network 330. Client device 302A is an example of computing device 102 in Figure 1, computing system 350 is an example of computing system 150 in Figure 1 and computing system 250 in Figure 2, and network 330 is an example of network 130 in Figure 1. Computing system 350 stores and implements the machine learning model 322. In some cases, output data obtained through the machine learning model 322 of computing system 350 may be used to improve other server tasks, or to improve services used by and performed by other non-user devices, or services performed for such other non-user devices. For example, the output data may improve other downstream processes performed by computing system 350 on a user's computing device or embedded computing device. In another example, the output data obtained through the implementation of the machine learning model 322 on the computing system 350 may be sent to and used by several other client devices, such as a user computing device, an embedded computing device, or a client device 302A. For example, the computing system 350 could be said to be running machine learning as a service.

[0133] In further embodiments, each different part of the machine learning model 322 may be stored and / or implemented in a combination of user computing devices, embedded computing devices, server computing devices, etc. In other words, parts of the machine learning model 322 may be distributed entirely or partially between the client device 302A and the computing system 350.

[0134] Devices 302 and 350 may perform graph processing or other machine learning techniques using one or more machine learning platforms, frameworks, and / or libraries such as TensorFlow, Caffe / Caffe2, Theano, Torch / PyTorch, MXnet, and CNTK. Devices 302 and 350 may be distributed in different physical locations and connected via one or more networks, including network 330. When configured as distributed computing devices, devices 302 and 350 may operate according to sequential computing architectures, parallel computing architectures, or a combination thereof. For example, distributed computing devices may be controlled or guided through the use of a parameter server.

[0135] In some embodiments, multiple instances of the machine learning model 322 can be parallelized to improve processing throughput. For example, multiple instances of the machine learning model 322 can be parallelized on a single processing device or computing device, or they can be parallelized across multiple processing devices or computing devices.

[0136] Each computing device implementing the machine learning model 322 or other aspects of the present disclosure may include several hardware components that enable the execution of the techniques described herein. For example, each computing device may include one or more memory devices that store some or all of the machine learning model 322. For example, the machine learning model 322 may be a structured numerical representation stored in memory. One or more memory devices may also include instructions for implementing the machine learning model 322 or for performing other operations. Examples of memory devices include RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof.

[0137] Each computing device may also include one or more processing devices that implement some or all of the machine learning model 322 and / or perform other related operations. Exemplary processing devices include a central processing unit (CPU), a visual processing unit (VPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural processing unit (NPU), a neural processing engine, the core of a CPU, VPU, GPU, TPU, NPU or other processing device, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a coprocessor, a controller, or one or more combinations of the processing devices described above. Processing devices may be embedded within other hardware components, such as image sensors or accelerometers.

[0138] Hardware components (memory devices and / or processing devices) can be distributed across multiple physically distributed computing devices and / or virtually distributed computing systems.

[0139] Figure 3D shows a conceptual diagram of an exemplary computing device communicating with an exemplary training computing system, including a model trainer. Figure 3D includes a computing system 350 that communicates with the training device 370 via a network 330. Computing system 350 is an example of computing system 150 in Figure 1, and network 330 is an example of network 130 in Figure 1. The machine learning model 322 described herein is trained on a training computing system, such as the training device 370, and then provided for storage and / or implementation on one or more computing devices, such as computing system 350. For example, the model trainer 372 runs locally on the training device 370. However, in some examples, the training device 370, including the model trainer 372, may be contained within or separately from computing system 350 or any other computing device implementing the machine learning model 322.

[0140] In some embodiments, the machine learning model 322 can be trained offline or online. In offline training (also known as batch training), the machine learning model 322 is trained across a static training dataset. In online training, the machine learning model 322 is continuously trained (or retrained) as new training data becomes available (for example, while performing inference using the model).

[0141] A model trainer 372, which may be an example of the training module 270 in Figure 2, can perform centralized training of the machine learning model 322 (for example, based on a centrally stored dataset). In other embodiments, distributed training techniques such as distributed training and federative learning can be used to train, update, or personalize the machine learning model 322.

[0142] The machine learning model 322 described herein can be trained according to one or more of various training types or training techniques. For example, in some embodiments, the machine learning model 322 may be trained by a model trainer 372 using supervised learning, which is trained on a training dataset containing examples or embodiments with labels. Labels may be applied manually by experts, generated through crowdsourcing, or provided by other techniques (e.g., physics-based or complex mathematical models). In some embodiments, if the user consents, training examples may be provided by a user computing device. In some embodiments, this process may be referred to as model personalization.

[0143] Figure 3E shows a conceptual diagram of a training process 390, which is an exemplary training process for training a machine learning model 322 with training data 391 containing exemplary input data 392 having labels 393. For example, the input data 392 could be device characteristics of a computing device having labels 393 in the form of smoothness scores associated with gameplay sessions. Training process 390 is just one example of a training process, and other training processes may also be used.

[0144] The training data 391 used by the training process 390 may include, if the user allows the use of such data for training, bundled content pieces already identified as belonging together, such as anonymized usage logs of shared flows, content items shared together, entities in the knowledge graph, etc. In some embodiments, the training data 391 may include examples of input data 392 to which labels 393 corresponding to output data 394 have been assigned.

[0145] In some embodiments, the machine learning model 322 can be trained by optimizing an objective function, such as an objective function 395. For example, in some embodiments, the objective function 395 may be, or include, a loss function that compares (e.g., determines the difference between) the output data generated by the model from the training data with the labels associated with the training data (e.g., ground truth labels). For example, the loss function may evaluate the sum or average of the squared differences between the output data and the labels. In some examples, the objective function 395 may be, or include, a cost function that represents the cost of a particular outcome or output data. Other examples of the objective function 395 may include, for example, margin-based techniques such as triplet loss or maximum margin training.

[0146] To optimize the objective function 395, one or more optimization techniques can be performed. For example, the optimization technique(s) can minimize or maximize the objective function 395. Exemplary optimization techniques include Hessian matrix-based techniques and gradient-based techniques such as coordinate descent, gradient descent (e.g., stochastic gradient descent), and subgradient methods. Other optimization techniques include black-box optimization techniques and heuristics.

[0147] In some embodiments, backpropagation can be used in combination with optimization techniques (e.g., gradient-based techniques) to train a machine learning model 322 (for example, if the machine learning model is a multi-layer model such as an artificial neural network). For example, the machine learning model 322 can be trained by performing iterative cycles of propagation and updating model parameters (e.g., weights). Exemplary backpropagation techniques include censored diachronic backpropagation and Levenberg-Marquardt backpropagation.

[0148] In some embodiments, the machine learning models 322 described herein may be trained using unsupervised learning techniques. Unsupervised learning may involve inferring a function to describe hidden structures from unlabeled data, for example, where classification or categorization may not be present in the data. Using unsupervised learning techniques, machine learning models can be created that can perform clustering, anomaly detection, train latent variable models, or other tasks.

[0149] The machine learning model 322 can be trained using semi-supervised techniques that combine aspects of supervised and unsupervised learning. The machine learning model 322 can be trained or otherwise generated through evolutionary techniques or genetic algorithms. In some embodiments, the machine learning model 322 described herein can be trained using reinforcement learning. In reinforcement learning, an agent (e.g., a model) can perform actions in an environment and learn to maximize rewards and / or minimize penalties resulting from such actions. Reinforcement learning may differ from supervised learning in that correct input / output pairs are not presented and suboptimal actions are not explicitly corrected.

[0150] In some embodiments, one or more generalization techniques can be performed during training to improve the generalization of the machine learning model 322. Generalization techniques can help reduce overfitting of the machine learning model 322 to the training data. Exemplary generalization techniques include dropout techniques, weight decay techniques, batch normalization, early stopping, subset selection, and stepwise selection.

[0151] In some embodiments, the machine learning models 322 described herein may include, or be affected by, several hyperparameters, such as the learning rate, the number of layers, the number of nodes in each layer, the number of leaves in the tree, and the number of clusters. Hyperparameters can affect the performance of the model. Hyperparameters can be selected manually or automatically by applying techniques such as grid search, black-box optimization techniques (e.g., Bayesian optimization, random search, etc.), and gradient-based optimization. Exemplary techniques and / or tools for performing automated hyperparameter optimization include Hyperopt, Auto-WEKA, Spearmint, and Metric Optimization Engine (MOE).

[0152] In some embodiments, various techniques can be used to optimize and / or adapt the learning rate during model training. Exemplary techniques and / or tools for performing learning rate optimization or adaptation include Adagrad, Adaptive Moment Estimation (ADAM), Adadelta, and RMSpropw.

[0153] In some embodiments, transfer learning techniques can be used to provide an initial model for initiating training of the machine learning model 322 described herein.

[0154] In some embodiments, the machine learning model 322 described herein can be included in various parts of computer-readable code on a computing device. In one example, the machine learning model 322 may be included in a particular application or program and may be used (e.g., exclusively) by such a particular application or program. Thus, in one example, the computing device may include several applications, one or more of which may include its own machine learning library and machine learning model(s).

[0155] In another example, the machine learning model 322 described herein may be contained within the operating system of a computing device (e.g., the central intelligence layer of the operating system) and may be invoked or otherwise used by one or more applications interacting with the operating system. In some embodiments, each application may communicate with the central intelligence layer (and the model(s) stored therein) using an application programming interface (API) (e.g., a common public API across all applications).

[0156] In some embodiments, the central intelligence layer can communicate with the central device data layer. The central device data layer may be a centralized repository of data from the computing device. The central device data layer can communicate with several other components of the computing device, such as one or more sensors, a context manager, device state components, and / or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0157] The technologies described herein refer to servers, databases, software applications, and other computer-based systems, as well as the actions performed and the information transmitted to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of configurations, combinations, and divisions of tasks and functions between their components. For example, the processes discussed herein can be implemented using a single device or component, or multiple devices or components working together.

[0158] Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0159] Furthermore, the machine learning techniques described herein are readily interchangeable and combinable. While specific exemplary techniques have been described, many other techniques exist and can be used in combination with the embodiments of this disclosure.

[0160] A brief overview of exemplary machine learning models and related techniques is provided in this disclosure. For further details, see the following references: Machine Learning: A Probabilistic Perspective (Murphy), Rules of Machine Learning: Best Practices for ML Engineering (Zinkevich), Deep Learning (Goodfellow), Reinforcement Learning: An Introduction (Sutton), and Artificial Intelligence: A Modern Approach (Norvig).

[0161] In addition to the above description, the user may be provided with controls that allow the user to make choices regarding both whether and when the systems, programs, or functions described herein enable the collection of user information (e.g., information about the user's social networks, social actions, or activities, occupation, user preferences, or user location), and whether content or communications are transmitted from the server to the user. Furthermore, certain data may be processed in one or more ways so that personally identifiable information is removed before it is stored or used. For example, a user's identity may be processed so that personally identifiable information cannot be identified, or if location information is obtained (e.g., at the city, zip code, or state level), the user's geographic location may be generalized so that the user's specific location cannot be identified. Thus, the user may have control over what information is collected about them, how that information is used, and what information is provided to them.

[0162] Figure 4 is a flowchart illustrating an example of operating modes of a computing system that determines the predicted performance of a game application running on a computing device, using one or more of the technologies of this disclosure. Figure 4 is described below in the context of Figures 1 and 2.

[0163] As shown in Figure 4, one or more processors 240 of the computing system 250 may receive indications of one or more device characteristics of computing device 102 from computing device 101 (402). In some examples, one or more processors 240 may receive queries from computing device 102 for a catalog of game applications that can be installed on computing device 102, which include indications of one or more device characteristics of computing device 102. In some examples, one or more device characteristics of computing device 102 include one or more central processing unit (CPU) characteristics of computing device 102, graphics processing unit (GPU) characteristics of computing device 102, or memory characteristics of computing device 102.

[0164] One or more processors 240 may determine the predicted performance of the game application 112 when running in a virtualized environment of the computing device 102, at least in part on one or more device characteristics of the computing device 102 and one or more application characteristics of the game application 112, using a game performance model 264 trained using machine learning to determine the predicted performance score of the game application 112 (404). In some examples, the predicted performance of the game application 112 when running in a virtualized environment of the computing device 102 represents a smoothness score corresponding to the proportion of frames of image data rendered by the game application 112 when running in a virtualized environment of the computing device 102 that are predicted to meet the frame time of a target frame rate associated with the game application 112.

[0165] One or more processors 240 may send an indication of the predicted performance of a game application 112 when run in the virtualized environment of the computing device 102 (406). In some examples, one or more processors 240 may send the computing device 102 a catalog of game applications that can be installed on the computing device 102, which includes an indication of the predicted performance of a game application 112 when run in the virtualized environment of the computing device 102.

[0166] In some examples, the game performance model 264 is trained using training data derived from data collected from a population of computing devices 120 running the game application 112, and a predicted performance score for the game application 112 is determined.

[0167] In some examples, the training data includes one or more device characteristics of corresponding computing devices from a population of computing devices 120, associated with a corresponding smoothness score, which corresponds to the percentage of frames of image data rendered by the game application 112 during the gameplay session on the corresponding computing device that is predicted to meet the frame time of a target frame rate associated with the game application 112, for one of several gameplay sessions of the game application 112 in a population of computing devices 120.

[0168] In some examples, one or more processors 240 may receive data collected from a group of computing devices 120, which may include frame-time metrics associated with frames of image data rendered and output by a game application 112 running on the group of computing devices 120, and may determine a corresponding smoothness score for a gameplay session based at least in part on the frame-time metrics contained in the data collected from the group of computing devices 120.

[0169] This disclosure includes the following embodiments. Example 1. A method comprising: one or more processors of a computing system receiving an indication of one or more device characteristics of a computing device from a computing device; one or more processors determining the predicted performance of the game application when running in a virtualized environment of the computing device, at least partially based on the one or more device characteristics of the computing device and one or more application characteristics of the game application, using a game performance model trained using machine learning to determine a predicted performance score of the game application; and one or more processors transmitting the indication of the predicted performance of the game application when running in the virtualized environment of the computing device to the computing device.

[0170] Example 2. The method according to Example 1, wherein the predicted performance of the game application when running in the virtualization environment of the computing device is indicated by a smoothness score corresponding to the proportion of frames of image data rendered by the game application when running in the virtualization environment of the computing device that are predicted to satisfy the frame time of a target frame rate associated with the game application.

[0171] Example 3. The method according to any one of Examples 1 and 2, wherein the one or more device characteristics of the computing device include one or more of the central processing unit (CPU) characteristics of the computing device, the graphics processing unit (GPU) characteristics of the computing device, or the memory characteristics of the computing device.

[0172] Example 4. The method according to any one of Examples 1 to 3, wherein the game performance model is trained using training data derived from performance metrics collected from a group of computing devices running the game application, and the predicted performance score of the game application is determined.

[0173] Example 5. The method according to Example 4, wherein the training data includes one or more device characteristics of the corresponding computing devices from the group of computing devices associated with a corresponding smoothness score, which corresponds to the percentage of frames of image data rendered by the game application during the gameplay session on the corresponding computing device that is predicted to satisfy the frame time of a target frame rate associated with the game application for one of several gameplay sessions of the game application in the group of computing devices.

[0174] Example 6. The method of Example 5, further comprising: receiving the performance metrics collected from the group of computing devices by one or more processors, wherein the performance metrics include frame-time metrics associated with frames of image data rendered and output by the game application running on the group of computing devices; and determining the corresponding smoothness score for the gameplay session by one or more processors based at least in part on the frame-time metrics included in the performance metrics collected from the group of computing devices.

[0175] Example 7. The method according to any one of Examples 1 to 6, wherein receiving the indication of one or more device characteristics of the computing device from the computing device includes receiving a query from the computing device by one or more processors for a catalog of game applications including the indication of one or more device characteristics of the computing device, and transmitting the indication of the predicted performance of the game applications when running in the virtualization environment of the computing device to the computing device includes transmitting the catalog of game applications including the indication of the predicted performance of the game applications when running in the virtualization environment of the computing device to the computing device by one or more processors.

[0176] Example 8. A computing system comprising: memory; a network interface; and one or more processors operably connected to the memory and the network interface, the one or more processors configured to: receive an indication of one or more device characteristics of the computing device from a computing device via the network interface; determine the predicted performance of the game application when running in a virtualized environment of the computing device, at least partially based on the one or more device characteristics of the computing device and one or more application characteristics of the game application, using a game performance model trained using machine learning to determine a predicted performance score of the game application; and transmit the indication of the predicted performance of the game application when running in the virtualized environment of the computing device to the computing device via the network interface.

[0177] Example 9. The computing system according to Example 8, wherein the predicted performance of the game application when running in the virtualization environment of the computing device indicates a smoothness score corresponding to the proportion of frames of image data rendered by the game application when running in the virtualization environment of the computing device that are predicted to satisfy the frame time of a target frame rate associated with the game application.

[0178] Example 10. The computing system according to any one of Examples 8 and 9, wherein one or more of the device characteristics of the computing device include one or more of the central processing unit (CPU) characteristics of the computing device, the graphics processing unit (GPU) characteristics of the computing device, or the memory characteristics of the computing device.

[0179] Example 11. A computing system according to any one of Examples 8 to 10, wherein the game performance model is trained using training data derived from performance metrics collected from a group of computing devices running the game application, and the predicted performance score of the game application is determined.

[0180] Example 12. The computing system according to Example 11, wherein the training data includes one or more device characteristics of the corresponding computing devices from the group of computing devices associated with a corresponding smoothness score, which corresponds to the percentage of frames of image data rendered by the game application during the game session on the corresponding computing device that is predicted to satisfy the frame time of a target frame rate associated with the game application for one of several gameplay sessions of the game application on the group of computing devices.

[0181] Example 13. The computing system according to Example 12, wherein one or more processors are further configured to receive the performance metrics collected from the group of computing devices, the data including frame-time metrics associated with frames of image data rendered and output by the game application running on the group of computing devices, and to determine the corresponding smoothness score for the gameplay session based at least in part on the frame-time metrics included in the performance metrics collected from the group of computing devices.

[0182] Example 14. The method according to any one of Examples 8 to 13, wherein one or more processors are further configured to receive queries from the computing device for a catalog of game applications, including the indications for one or more device characteristics of the computing device, in order to receive the indications for one or more device characteristics of the computing device from the computing device, and the one or more processors are further configured to send to the computing device the catalog of game applications, including the indications for the predicted performance of the game applications when running in the virtualization environment of the computing device, in order to send to the computing device the indications for the predicted performance of the game applications when running in the virtualization environment of the computing device.

[0183] Example 15. A computer-readable storage medium for storing instructions, wherein, when the instructions are executed, the instructions cause one or more processors of a computing system to: receive an indication of one or more device characteristics of a computing device from a computing device; determine the predicted performance of the game application when running in a virtualized environment of the computing device, at least partially based on the one or more device characteristics of the computing device and one or more application characteristics of the game application, using a game performance model trained using machine learning to determine a predicted performance score of the game application; and transmit the indication of the predicted performance of the game application when running in the virtualized environment of the computing device to the computing device.

[0184] Example 16. The computer-readable storage medium according to Example 15, wherein the predicted performance of the game application when running in the virtualization environment of the computing device indicates a smoothness score corresponding to the proportion of frames of image data rendered by the game application when running in the virtualization environment of the computing device that are predicted to satisfy the frame time of a target frame rate associated with the game application.

[0185] Example 17. A computer-readable storage medium according to any one of Examples 15 and 16, wherein one or more of the device characteristics of the computing device include one or more of the central processing unit (CPU) characteristics of the computing device, the graphics processing unit (GPU) characteristics of the computing device, or the memory characteristics of the computing device.

[0186] Example 18. A computer-readable storage medium according to any one of Examples 15 to 17, wherein the game performance model is trained using training data derived from performance metrics collected from a group of computing devices running the game application, and determines the predicted performance score of the game application.

[0187] Example 19. A computer-readable storage medium according to Example 18, wherein the training data includes one or more device characteristics of the corresponding computing devices from the group of computing devices associated with a corresponding smoothness score, which corresponds to the percentage of frames of image data rendered by the game application during the game session on the corresponding computing device that is predicted to satisfy the frame time of a target frame rate associated with the game application for one of several gameplay sessions of the game application on the group of computing devices.

[0188] Example 20. The computer-readable storage medium according to Example 19, wherein the instruction is further configured to cause one or more processors to receive the performance metrics collected from the group of computing devices, the performance metrics including frame-time metrics associated with frames of image data rendered and output by the game application running on the group of computing devices, and to determine the corresponding smoothness score for the gameplay session based at least in part on the frame-time metrics included in the performance metrics collected from the group of computing devices.

[0189] As an example, and not an limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other storage media that can be used to store desired program code in the form of instructions or data structures and that are accessible by a computer. Also, any connection is properly called a computer-readable storage medium. For example, if instructions are transmitted from a website, server or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals or other temporary media; instead, they refer to non-temporary, tangible storage media. As used herein, the terms "disk" and "disc" include compact discs (CDs), laser discs, optical discs, digital-purpose discs (DVDs), floppy disks, and Blu-ray discs. A disk typically reproduces data magnetically, while a disc reproduces data optically using a laser. Any combination of the above should also be included in the scope of computer-readable storage media.

[0190] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated logic circuits or discrete logic circuits. Therefore, the term “processor” as used herein may refer to any of the aforementioned structures or any other structure suitable for implementing the technology described herein. Furthermore, in some embodiments, the functions described herein may be provided within dedicated hardware and / or software modules. The technology can also be fully implemented in one or more circuits or logic elements.

[0191] The technology of this disclosure can be implemented in a wide variety of devices or apparatus, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). While this disclosure describes various components, modules, or units to highlight the functional aspects of devices configured to perform the disclosed technology, it does not necessarily require implementation by different hardware units. Rather, as described above, the various units may be combined into hardware units, or provided in combination with appropriate software and / or firmware by an assembly of interoperable hardware units, including one or more processors as described above.

[0192] Various embodiments have been described. These embodiments and other embodiments are within the scope of the following claims.

Claims

1. It is a method, One or more processors of a computing system receive data from a computing device that represents one or more device characteristics of the computing device, The method comprises determining the predicted performance of a particular game application when run on one or more processors of the computing device, based at least in part on one or more device characteristics of the computing device and one or more application characteristics of the particular game application, using a game performance model trained using machine learning to determine a predicted performance score of the particular game application, wherein the virtualization environment is hosted by a first operating system running on the computing device, and a second operating system different from the first operating system runs within the virtualization environment, and the method is A method further comprising transmitting data indicating the predicted performance of a particular game application when run in the virtualization environment of the computing device, using one or more processors.

2. The method according to claim 1, wherein the predicted performance of the particular game application when running in the virtualization environment of the computing device represents a smoothness score corresponding to the proportion of frames of image data rendered by the particular game application when running in the virtualization environment of the computing device that are predicted to satisfy the frame time of a target frame rate associated with the particular game application.

3. The method according to claim 1, wherein the one or more device characteristics of the computing device include one or more of the central processing unit (CPU) characteristics of the computing device, the graphics processing unit (GPU) characteristics of the computing device, or the memory characteristics of the computing device.

4. The method according to claim 1, wherein the game performance model is trained using training data derived from performance metrics collected from a group of computing devices running the particular game application, and determines the predicted performance score for the particular game application.

5. The method according to claim 4, wherein the training data includes one or more device characteristics of the corresponding computing devices from the group of computing devices associated with a corresponding smoothness score, which corresponds to the percentage of frames of image data rendered by the specific game application during the gameplay session on the corresponding computing device that is predicted to satisfy the frame time of a target frame rate associated with the specific game application for one of several gameplay sessions of the specific game application in the group of computing devices.

6. The method further includes receiving the performance metrics collected from the group of computing devices by one or more processors, the performance metrics including frame-time metrics associated with frames of image data rendered and output by the particular game application running on the group of computing devices, and the method The method according to claim 5, further comprising using one or more processors to determine the corresponding smoothness score for the gameplay session based at least in part on the frame time metrics included in the performance metrics collected from the group of computing devices.

7. Receiving the data from the computing device that represents one or more device characteristics of the computing device includes receiving a query from the computing device by one or more processors for a catalog of game applications that includes the data representing one or more device characteristics of the computing device, Determining the predicted performance of the particular game application when it runs in the virtualization environment of the computing device is: Each game application listed in the catalog of game applications is determined to be the specific game application, For each of the game applications listed in the catalog of game applications, the predictive performance of the game application when it is run in the virtualization environment of the computing device is determined, The method according to claim 1, wherein transmitting to the computing device the data indicating the predicted performance of a particular game application when run in the virtualization environment of the computing device comprises transmitting to the computing device a catalog of game applications, which includes the data indicating the predicted performance determined for each of the game applications listed in the catalog of game applications, by one or more processors.

8. A computing system, Memory and Network interface and One or more processors operably connected to the memory and the network interface, Receiving data from a computing device via the network interface that indicates one or more device characteristics of the computing device, The system is configured to determine the predicted performance of a particular game application when running in a virtualized environment of the computing device, based at least partially on one or more device characteristics of the computing device and one or more application characteristics of the particular game application, using a game performance model trained using machine learning to determine a predicted performance score of the particular game application, wherein the virtualized environment is hosted by a first operating system running on the computing device, and a second operating system different from the first operating system runs within the virtualized environment. One or more processors are configured to further transmit data indicating the predicted performance of a particular game application when running in the virtualization environment of the computing device to the computing device via the network interface, A computing system equipped with [the following features].

9. The computing system according to claim 8, wherein the predicted performance of the particular game application when running in the virtualization environment of the computing device represents a smoothness score corresponding to the proportion of frames of image data rendered by the particular game application when running in the virtualization environment of the computing device that are predicted to satisfy the frame time of a target frame rate associated with the particular game application.

10. The computing system according to claim 8, wherein the one or more device characteristics of the computing device include one or more of the central processing unit (CPU) characteristics of the computing device, the graphics processing unit (GPU) characteristics of the computing device, or the memory characteristics of the computing device.

11. The computing system according to any one of claims 8 to 10, wherein the game performance model is trained using training data derived from performance metrics collected from a group of computing devices running the particular game application, and determines the predicted performance score for the particular game application.

12. The computing system according to claim 11, wherein the training data includes one or more device characteristics of the corresponding computing devices from the group of computing devices associated with a corresponding smoothness score, which corresponds to the percentage of frames of image data rendered by the specific game application during the gameplay session on the corresponding computing device that is predicted to satisfy the frame time of a target frame rate associated with the specific game application for one of several gameplay sessions of the specific game application in the group of computing devices.

13. The one or more processors described above are The system is further configured to receive the performance metrics collected from the group of computing devices, the performance metrics including frame-time metrics associated with frames of image data rendered and output by the particular game application running on the group of computing devices, and the one or more processors The computing system according to claim 12, further configured to determine the corresponding smoothness score of the gameplay session based at least in part on the frame time metrics included in the performance metrics collected from the group of computing devices.

14. In order to receive the data representing one or more device characteristics of the computing device from the computing device, the one or more processors are further configured to receive queries from the computing device for a catalog of game applications, which includes the data representing one or more device characteristics of the computing device. To determine the predicted performance of the particular game application when it runs in the virtualization environment of the computing device, one or more processors: Each game application listed in the catalog of game applications is determined to be the specific game application, For each of the game applications listed in the catalog of game applications, the system is further configured to determine the predicted performance of the game application when it is run in the virtualization environment of the computing device. The computing system according to any one of claims 8 to 10, wherein one or more processors are further configured to transmit to the computing device a catalog of game applications, which includes the data indicating the predicted performance of each game application listed in the catalog of game applications, in order to transmit to the computing device the data indicating the predicted performance of a particular game application when run in the virtualization environment of the computing device.

15. A program for causing one or more processors to perform the method described in any one of claims 1 to 7.

Images