Memory management in game rendering

Abstract

Some embodiments of the present application are directed to a server system comprising one or more CPUs, a plurality of GPUs, a main dynamic memory storing programs and data for use by the CPUs and / or GPUs during execution of the programs, a static memory pool stored in non-volatile memory, and a memory controller configured to manage the static memory pool. Each GPU includes a local cache and is configured to access a static memory pool via a memory controller. A server system executes a plurality of game sessions of a game title in parallel on one or more CPUs. Each of the plurality of game sessions is associated with a static data item stored in a static memory pool and requires graphical operations executable by a respective GPU using the static data item.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201980006052.0, filed on April 9, 2019. Technical Field

[0003] The present application relates generally to computer technology, including but not limited to methods and systems for managing a server system to support multiple online interactive sessions corresponding to one or more real-time user interactive applications. Background Art

[0004] Internet-connected electronic devices can support a variety of cloud-based media and entertainment applications. These applications include media streaming applications in which a server streams content to a user device, gaming applications in which a user interacts with a game executed on a server from a user device, and various social media and communication applications that allow a large number of users to interact with each other and cloud-hosted content and applications in parallel via their Internet-connected devices. Among cloud-based applications, cloud gaming presents some unique challenges due to the following reasons: widely varying hardware requirements for game titles; a variety of topologies in which cloud-based games can be participated (e.g., by one player, by multiple players in a single location, or by multiple players in multiple locations); the need to reliably and without delay transmit player input to a game server executing a game session and transmit game session output from the game server to the player's device / display; widely varying expectations of players' game speeds and responsiveness; and the desire to provide near real-time game content to viewers in some cases. Other challenges with cloud-based gaming involve providing a consistent gameplay experience for players regardless of where they are located (e.g., close to the server or far away from the server), how they connect to the gaming service (e.g., via a fast or slow Internet connection), and what type of device they use to play the game (e.g., a general-purpose personal device or a dedicated game controller) and view the gameplay output (e.g., a personal device or a media device connected to a media streaming device).

[0005] Specifically, there is a need for a cloud gaming system that supports multiple game sessions for multiple game titles, where games can be executed concurrently with acceptable latency and responsiveness, including for multiple users in the same or different locations playing the same game title, and a variety of input and output devices and network connections. In addition, there is a need for a cloud gaming system that processes user input and outputs high-definition images immediately after receiving player input (e.g., game input typed on a final-use gaming device / controller) in a gaming session, which reflects the results of the player input actions of all game players simultaneously and with acceptable latency. There is also a need for a gaming system that, in some cases, provides a high-definition video stream of gameplay activity to allow viewers to track the gameplay in real time on a corresponding display device. In this way, it would be beneficial to provide a cloud gaming system with an efficient game processing and output mechanism to expand the gaming experience in a wide range of gaming settings, from spontaneous gameplay of users gathered in the same location to online interactive gameplay of multiple users from multiple locations. Summary of the invention

[0006] The embodiments described in this specification are directed to configuring the computing and storage resources of a server system to facilitate concurrent execution of multiple online game sessions with acceptable latency and responsiveness. For example, a static memory pool stored in a non-volatile memory can be provided to store static data items that are usually stored in a main dynamic memory that needs to be constantly refreshed. This static memory pool is shared by multiple processors of the server system and can be accessed at a fairly fast rate without the need for strong power as required by the main dynamic memory. The size of the main dynamic memory can be reduced because part of the main dynamic memory is replaced by a static memory pool to store static data items. In addition, the shader cache stores more than one sequence of compiled shaders to ensure that multiple game sessions are rendered with the desired power and latency performance. When one of the more than one sequence of compiled shaders already present in the shader cache is used to process multiple online game sessions, different sequences of compiled shaders stored in the shader cache are dynamically generated based on real-time usage statistics to meet different performance standards. These different sequences of compiled shaders can be used for subsequent game content rendering of multiple online game sessions.

[0007] In one aspect of the present application, a method for managing storage capacity of a server system includes one or more central processing units (CPUs), multiple graphics processing units (GPUs), main dynamic memory that stores programs and data for use by the CPU and / or GPU during program execution, a static memory pool stored in a non-volatile memory, and a memory controller configured to manage the static memory pool. Each GPU includes a local cache and is configured to access the static memory pool via the memory controller. The method includes executing multiple game sessions of a game title in parallel on one or more CPUs. Each of the multiple game sessions is associated with a static data item and requires a graphics operation executable by the corresponding GPU using the static data item. The static data items are stored in the static memory pool. The method further includes, for a first game session among a plurality of game sessions executed on one or more CPUs, assigning a graphics operation to a first GPU; requesting, by the first GPU, a first copy of a static data item from a memory controller; receiving, from the memory controller, a first copy of the static data item read from a static memory pool; performing, by the first GPU, the graphics operation using the first copy of the static data item; and locally storing (e.g., in a local cache, in particular, in a local cache of the first GPU) a first result of the graphics operation associated with the first game session among the plurality of game sessions. The method further includes, for a second game session among a plurality of game sessions executed on one or more CPUs, assigning a graphics operation to a second GPU; requesting, by the first GPU, a second copy of the static data item from a memory controller; receiving, from the memory controller, a second copy of the static data item read from a static memory pool; performing, by the second GPU, the graphics operation using the second copy of the static data item; and locally storing (e.g., in a local cache, in particular, in a local cache of the second GPU) a second result of the graphics operation associated with the second game session among the plurality of game sessions.

[0008] On the other hand, a method for loading a game title onto a server system includes one or more CPUs, multiple GPUs, a main dynamic memory storing programs and data for use by the CPU and / or GPU during program execution, a static memory pool stored in a non-volatile memory, and a memory controller configured to manage the static memory pool. Each GPU is configured to access the static memory pool via the memory controller. The method includes loading an executable game program corresponding to the game title into the main dynamic memory by the CPU. The CPU is configured to execute multiple online game sessions in parallel. Each game session corresponds to an executable game program and requires graphics operations that can be executed by a corresponding GPU assigned to render the corresponding online game session. The method further includes identifying, by the CPU, static data items used in graphics operations of multiple game sessions; storing the static data items in the static memory pool by the memory controller; and enabling the GPU to execute an online game session to access the static data items while performing graphics operations by the memory controller.

[0009] On the other hand, a method for rendering image frames of an online game session is implemented at a server system, the server system including one or more CPUs, one or more GPUs, a main dynamic memory storing programs and data for use by the CPU and / or GPU during execution of the programs, and a shader cache. The method includes executing multiple online game sessions for a game title in parallel on one or more CPUs, and rendering multiple image frames of each of the multiple online game sessions by the GPU using a first sequence of compiled shaders. The method further includes, while executing the multiple online game sessions, identifying performance criteria for multiple online game sessions associated with the game title, collecting usage statistics characterizing the execution of each online game session, and modifying a first shader in the first sequence of compiled shaders to improve the performance of the first sequence of compiled shaders relative to the performance criteria, thereby creating a modified first sequence of compiled shaders.

[0010] According to some aspects of the present application, a server system includes a memory storing instructions for causing the server system to execute any of the above methods.

[0011] In addition, according to some aspects of the present application, a non-transitory computer-readable storage medium storing one or more applications configured to be executed by a server system includes instructions for causing the server system to execute any of the above methods.

[0012] Other embodiments and advantages may be apparent to those skilled in the art from the description in this specification and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] For a better understanding of the various described embodiments, reference should be made to the following description of the embodiments in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout the figures.

[0014] Figure 1 is an example online interactive gaming environment including a server system and a plurality of client devices according to some implementations.

[0015] Figure 2 is an example online interactive gaming environment in which multiple game servers share a static memory pool in a server system according to some implementations.

[0016] Figure 3A is an example memory system configured to store data in a server system according to some implementations. Figure 3B is an example central processing unit (CPU) with multiple layers of cache in accordance with some embodiments. Figure 3C is an example graphics processing unit (GPU) with multiple layers of cache in accordance with some implementations. Figure 3D is an example memory controller that controls read and write access to a static memory pool shared by multiple processors in a server system in accordance with some implementations.

[0017] Figure 4A is an example data flow during the process of loading a game title onto a server system, according to some implementations. Figure 4B is an example data flow during the process of executing an online gaming session on a server system according to some implementations. Figure 4C is an example data flow during a shader optimization process according to some implementations. Figure 4D is an example data stream of static data times when executing multiple online gaming sessions according to some implementations.

[0018] Figure 5A is a block diagram illustrating an example server system in accordance with some implementations. Figure 5B is a block diagram illustrating data stored in a server system according to some implementations.

[0019] Figure 6 is a block diagram illustrating an example client device (eg, a mobile phone, tablet computer, laptop computer, or game controller) in accordance with some implementations.

[0020] Figure 7 is a block diagram illustrating an example media device in accordance with some implementations.

[0021] Fig. 8A and 8Bis a flow chart illustrating an example method for managing storage capacity of a server system hosting one or more game titles, according to some implementations.

[0022] Fig. 9 is a flow chart illustrating an example method for loading a game title into a server system according to some implementations.

[0023] Fig.10 is an example process for optimizing pipeline shaders concurrently with game rendering in a server system according to some implementations.

[0024] Fig.11A is an example process for optimizing a sequence of compiled shaders according to some implementations. Fig. 11B is a shader hash table that associates multiple sequences of compiled shaders with multiple performance criteria according to some implementations. Fig. 11C is a shader hash table that associates multiple versions of a sequence of compiled shaders with multiple user inputs according to some implementations.

[0025] Fig.12 is a flow chart illustrating an example method of rendering image frames for multiple online game sessions (eg, including optimizing pipeline shaders in the context of image rendering) in accordance with some implementations.

[0026] Like reference numerals designate corresponding parts throughout the drawings. DETAILED DESCRIPTION

[0027] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the detailed description that follows, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to one of ordinary skill in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, processes, components, and networks are not described in detail in order to avoid unnecessarily obscuring aspects of the embodiments.

[0028] The embodiments described in this specification are directed to providing a gaming environment to achieve efficient, portable, low-latency, and interactive gaming experiences for a community of distributed gamers. Some embodiments dynamically allocate cloud gaming hardware resources (e.g., GPUs and encoders) and monitor and utilize the network bandwidth available to individual end users to provide the best online gaming experience to a large number of gamers at the same time. Some embodiments manage cloud gaming storage resources to efficiently provide an online gaming experience. For example, a game server system may add a certain level of static game storage resources for storing static data items used in rendering game content to reduce the need for dynamic memory with relatively large power consumption. Some embodiments dynamically optimize a sequence of compiled shaders to meet one or more performance criteria based on real-time usage statistics for multiple online game sessions, and the compiled shaders are used in the foreground to render game content for these online game sessions.

[0029] Specifically, in some embodiments, the server system includes one or more CPUs, multiple GPUs, and a main dynamic memory that stores programs and data for use by the CPU and / or GPU during program execution. Each GPU includes a local cache. A static memory pool stored in a non-volatile memory is added to the server system. The memory controller of the server system is configured to manage the static memory pool, for example, providing write and / or read access to the static memory pool to each GPU. In the example, the main dynamic memory is a dynamic random access memory (DRAM) that needs to be constantly refreshed, while the static memory pool is a phase change random access memory (PRAM) that does not need to be refreshed as long as it is connected to a power source.

[0030] A server system executes multiple game sessions of a game title in parallel on one or more CPUs. Each of the multiple game sessions is associated with a static data item and requires a graphics operation (e.g., vertex processing, tessellation, and primitive assembly in an Open Graphics Library (OpenGL) rendering pipeline) that can be performed by a corresponding GPU using the static data item. The static data items are stored in a static memory pool. To execute each of the multiple game sessions, the CPU assigns the graphics operation to the corresponding GPU, which requests a copy of the static data item from the memory controller in order to perform the graphics operation. The corresponding GPU receives a copy of the static data item read from the static memory pool from the memory controller and uses the copy of the static data item to perform the graphics operation. The result of the graphics operation is stored locally (e.g., stored in a local cache or in main dynamic memory) in association with the corresponding game session in the multiple game sessions.

[0031] In some embodiments, when CPU loads the executable game program (for example, server-side game module) corresponding to the game title into the main dynamic memory, the static data items stored in the static memory pool are loaded into the static memory pool. Specifically, CPU determines to use static data items in graphics operations, and controls memory controller to store static data items in the static memory pool. Static data items are optionally extracted from the local hard disk drive of the server system, or received from a remote library server. When static data items are stored in the static memory pool, there is no need to store static data items in the main dynamic memory, thereby allowing the main dynamic memory to improve its efficiency by reducing its size and power consumption, without affecting the performance of the CPU and GPU in the server system.

[0032] In some embodiments, the server system includes one or more CPUs, one or more GPUs, a main dynamic memory that stores programs and data for use by the CPU and / or GPU during program execution, and a shader cache. The server system executes multiple online game sessions of a game title in parallel on the CPU, and one or more GPUs render multiple image frames of each of the multiple online game sessions using a first sequence of compiled shaders. When executing multiple online game sessions in the foreground, the server system identifies performance criteria (e.g., related to power usage and game delay) of multiple online game sessions associated with the game title in the background, and collects usage statistics (e.g., user input, power usage, and game delay) that characterize the execution of each online game session. The server system modifies the first shader in the first sequence of compiled shaders to improve the performance of the first sequence of compiled shaders with respect to the performance criteria, thereby creating a modified first sequence of compiled shaders. Optionally, the modified first sequence of compiled shaders is stored as a different sequence of compiled shaders, as a new version of the first sequence of compiled shaders, or instead of the first sequence of compiled shaders. The modified first sequence of compiled shaders can then be used to render image frames for multiple online game sessions and improve performance associated with performance criteria for these online game sessions. By these means, compiled shaders are dynamically optimized based on real-time usage statistics of online game sessions and without pausing execution of online game sessions.

[0033] The game environment hosts multiple game sessions for a game title

[0034] Figure 11 is an example online interactive gaming environment 100 according to some embodiments. The online interactive gaming environment 100 includes one or more client devices (e.g., client devices 102 and 104). Each client device 102 executes one or more game applications. An online gaming session can be initiated and run on a particular game application to allow a user of a client device 102 to play an online interactive game (also referred to as a "game title") hosted by a server system 114. In some embodiments, a client device 102 (e.g., a host client) is configured to invite one or more other client devices 102 to join a gaming scene of an online interactive game. The gaming sessions of these client devices 102 are synchronized to optionally display the same gaming scene of the online interactive game from different perspectives corresponding to their respective users.

[0035] Instead, the server system 114 hosts an online interactive gaming platform to support client devices 102 to participate in one or more gaming applications, including a specific gaming application. Specifically, the server system 114 includes multiple user accounts associated with the client device 102, and authenticates the user of the client device 102 in association with each of the one or more gaming applications. The server system 114 renders and refreshes a scene of a game title on the client device 102, which joins a corresponding gaming session associated with the scene. In some embodiments, the server system 114 can evaluate the capabilities of the client device 102 and / or the quality of the communication connection between the server system 114 and each client device 102, and adaptively generate a synchronous data stream for a gaming session associated with the client device 102. By these means, the server system 114 is configured to facilitate synchronous gaming sessions of online interactive games on two or more client devices 102 simultaneously and with relatively low latency.

[0036] In some embodiments, the client device 102 has a display screen integrated therein for displaying media content associated with the corresponding online game session. In some embodiments, the client device 102 is coupled to the media device 106 and the output device 108. Specifically, the client device 102 can be directly (e.g., via a wire, via a Bluetooth or other wireless communication link), via a local network 110 (e.g., a Wi-Fi network), or via one or more communication networks 112. Communicative coupling to the media device 106. In some embodiments, the client device 102 and the media device 106 are local to each other (e.g., in the same room, in the same house, etc.). The media device 106 is further coupled to one or more output devices 108 (e.g., a television, a display monitor, a sound system, a speaker, etc.) that can output visual and / or audio content. The media device 106 is configured to output content to the output device 108. In some embodiments, the media device 106 is a delivery device (e.g., CHROMECAST by Google Inc.) or a device that otherwise includes delivery functionality. For example, when server system 116 renders game content in response to user input at the client device, client device 102 controls media device 106 to receive the game content from server system 116 via communication network 112 and deliver the received game content to output device 108 .

[0037] In some embodiments, the server system 114 includes a game server 118 and a media stream server 120. The game server 118 is configured to simultaneously provide two or more media streams 130 and 132 for an online interactive game session running on a first client device 102A. The two or more media streams include a low-latency stream 130 and a normal-latency stream 132, which are provided to the first client device 102A and the commentator client device 104 via one or more communication networks 112, respectively. Optionally, the normal-latency stream 132 is provided for educational purposes. When a user on the first client device 102A plays a game session on the first client device 102A, the game session is recorded and broadcast to one or more viewers via the normal-latency stream 132, i.e., the viewers can comment on the game session on the commentator client. The low-latency stream 130 corresponds to the gameplay of the online interactive game session and has a faster response rate and lower transmission delay than the normal-latency stream 132 corresponding to the associated commentary session. Optionally, the reviewer client device 104 is co-located with the client device 102. Optionally, the reviewer device is remote from the client device 102. In some embodiments, the reviewer client device 104 is another client device, such as the client device 102A, which optionally has an integrated or external display. In some embodiments, the reviewer client device 104 is a media device 106 (e.g., a device with casting functionality) coupled to one or more output devices 108 that can output visual and / or audio content. In some embodiments, both the low-latency stream and the normal-latency stream are displayed on the screen of the same output device.

[0038] Each client device 102 is capable of communicating data and sharing information with a central server or cloud computing system (e.g., server system 114) and / or other network-connected devices (e.g., another client device 102, commentator client devices 104, media devices 106, and output devices 108). Data communications may be performed using any of a variety of custom or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART, MiWi, etc.) and / or any of a variety of custom or standard wired protocols (e.g., Ethernet, HomePlug, etc.), or any other suitable communication protocol including communication protocols not yet developed as of the date of filing of this document. In some embodiments, the online interactive gaming environment 100 includes conventional network devices (e.g., ( Figure 1The client devices 102 and their corresponding media and output devices (if any) are communicatively coupled to each other on the local area network via the conventional network device (router not shown). The local area network is communicatively coupled to other parts of the communication network 112 (e.g., wide area network and the Internet). In some embodiments, each of the client devices 102 optionally communicates with one or more other client devices 102 or 104, corresponding media devices 106, or corresponding output devices 108 using one or more radio communication networks (e.g., ZigBee, Z-Wave, Insteon, Bluetooth, Wi-Fi, and / or other radio communication networks).

[0039] In some embodiments, the client devices 102 are remote from each other, i.e., they are not located in the same room or even building. Figure 6 The game application 528 in the game server system 114 starts the game title on the client device 102 to execute on each client device 102. In some embodiments, for each client device 102, the game application independently establishes an online game session 116 with the server system 114. The online game sessions 116 of two or more client devices 102 (e.g., 102A and 102B) are related to each other (e.g., because they are all playing games in the same game domain of the game title), and therefore, the game scene is shared in the game application. The related online game sessions 116 are synchronized with each other, and each online game session 116 optionally shows the same game scene with a unique player perspective corresponding to the corresponding client device 102. Therefore, the user of each client device 102 can play the game on the corresponding client device and affect the output of the online game sessions 116 on other client devices 102. For example, in some embodiments, the game application 528 is configured to maintain a consistent game state among and between users who play the same game at the same time. In some embodiments, the server system 114 and game application 528 maintain a separate session for each user playing the game, which may result in a rendered view of the scene that is specific to that user's separate game state and view, but each of these sessions remains consistent with the sessions of other users playing the same game and interacting with and viewing the same scene.

[0040] Alternatively, in some other embodiments, after the game application of the first client device 102A establishes the online game session 116A, one or more second client devices 102B are invited to join the online game session 116A through an invitation message, and for example, a message with a link (e.g., a URL address) to join the online game session 116A is sent to each of the second client devices 102B. Appropriate controller configurations are provided to each second client device 102B invited to join the online game session 116A. In this application, when the second client 102B joins the online game session 116A, the server system 114 creates a separate game session 116B for each separate second client device 102B. Each separate game session 116B of the corresponding second client device 102B is synchronized with the game session 116A of the first client device 102A and shares the same scene with the game session 116A, but may have a unique player perspective corresponding to the corresponding second client device 102B. After each second client device 102B has received the appropriate controller configuration and joined the online game session 116A (more precisely, started its associated online game session 116B), the user can play the game on the corresponding second client device 102B and affect the output of the online game session 116 running on the other client devices 102.

[0041] The client device 102 is a device that includes and can run one or more different user applications including game applications. In some embodiments, the client device 102 is a smart phone, a tablet device, a laptop computer, a desktop computer, a multimedia device or a remote controller. In some embodiments, the client device 102 includes a dedicated game controller device (e.g., a game controller 102B). In some embodiments, the client device 102 includes one or more user applications configured to operate in conjunction with the media device 106. In some embodiments, the application includes a media device application for pairing the client device 102 with the media device 106 and configuring the media device 106. The application also includes one or more applications that can be associated with the content delivered to the media device 106. In some embodiments, by (e.g., via a local network) directly sending data / content to the media device 106 and / or by directing the media device 106 to a remote location (e.g., a URL or other link to a location at a server system 116 or a game content host) from which the media device 106 can stream or otherwise receive data / content, the application delivers data and / or content to the media device 106. The media device 106 receives data / content from the application and / or remote location and outputs visual and / or audio content corresponding to the received data / content to the output device 108. Thus, an online gaming session 116 is established between the gaming application running on the client device 102, the remote server system 114, and the media device 106.

[0042] In some embodiments, as part of the process of linking related online game sessions 116, the server system 114 evaluates the capabilities of each corresponding client device 102 and / or the quality of the communication connection between the server system 114 and the client device 102. In some embodiments, the server system 114 measures the network latency between the client device 102 and the server system 114. If the measured latency is above a threshold and a lower latency connection is available, the server system 114 can suggest that the client device 102 change to a lower latency connection, or invite the user of the client device 102 to change the client device 102 to a lower latency connection. For example, if the client device 102 is on a cellular wireless connection and a local network is available, the server system 114 can suggest that the client device 102 should connect via the available local network. In some embodiments, the latency threshold requirements vary between games. For example, some games (e.g., action games) are best experienced on lower latency connections, while other games (e.g., online board games or card games) are less demanding on latency. In view of these different requirements associated with different types of games, the server system 114 can make connection recommendations.

[0043] As part of the client device 102 starting or joining a game session 116, the server system 114 communicates with the client device 102 to set up a controller on the client device 102. In some embodiments, this includes the server system 114 evaluating whether the client device 102 has the resources and communication capabilities required for the controller. Depending on the available resources at the client device 102, the quality of the connection, and the game requirements, the controller can be implemented differently at the client device 102. In some embodiments, a web-based controller interface can be used to play the game title. For example, the controller interface for the game can be embedded in a web page, and the web page is rendered in a web browser on the client device 102. Alternatively, in some embodiments, the standardized controller is implemented in a predefined application that is not specific to the game or is directly associated with the game (e.g., a delivery device application, such as Chromecast or Google Cast by Google Inc., or other media device applications), or in the operating system of the client device 102. For example, the device operating system or predefined application on the client device 102 can have a controller submodule. The controller submodule includes one or more standardized controller configurations, templates, etc. Each standardized controller configuration configures the controller submodule to utilize input devices and / or sensors on the client device 102 in some manner to implement a virtual controller. The standardized controller configuration used may vary depending on the type of game and / or client device.

[0044] In addition, in some embodiments, the game has a specific controller configuration that can be implemented on the controller submodule. Such a configuration can be stored at the server system 114 and transmitted to the client device 102 as a part of the process of the client device 102 joining or starting an online game session 116. In some embodiments, the specific controller configuration can be a mixture of a custom controller or a standard controller and a custom controller completely. In addition, in some embodiments, the game requires a specific application associated with the game. For example, the game may require a controller application specifically associated with the game. In some embodiments, the client device 102 can be instructed to download a specific application or a predefined application as a part of starting or joining a session 116. For example, if the client device 102 does not have a predefined application (and controller submodule) or a specific application associated with the game, and such an application is required for playing games, the server system 114 indicates that the client device 102 prompts its user to download and request permission to continue to the user.

[0045] The server system 114 further includes a database 160. The database 160 stores at least one or more game applications hosted on the server system 114 (e.g., Figure 6The database 160 stores session data associated with online game sessions played on the client device 102. Examples of session data for each online game session 116 include, but are not limited to, frame rate, rendering specifications, normal latency requirements, GPU allocation information, encoder allocation information, identification of related sessions, shader pipeline identification, and latest status information. Session data is accessed more frequently than user information 126. In some embodiments, the memory storing user information and session data is composed of flash memory and random access memory (RAM), respectively. In some embodiments, a subset of user information and session data is stored in each game server 118 corresponding to a corresponding game title.

[0046] The server system 114 includes one or more of the following: one or more game servers 118, a media streaming server 120, a game library server 122, a shader library server 124, and a pipeline optimizer server 150. The game server 118 or an alternative front-end server ( Figure 1 The game server 118 may be used to manage user accounts associated with the client devices 102 and 104, for example, through the user account's subscription to membership in one or more online interactive games. After the client device 102 logs into its corresponding user account and starts / joins its online game session 116, the game server 118 not only sets up the game session 116, but also manages each specific game session 116 of the corresponding client device 102 by obtaining game content from a content server (not shown), sending the game content to the game application executing on the client device 102 or the designated media device 104, identifying user requests or actions, rendering gameplay output for the client device 102 or the media device 104 in response to the user request or action, and storing game state data during the corresponding game session 116.

[0047] In some embodiments, the game server 118 dynamically allocates cloud gaming hardware resources (e.g., GPU 140 and encoder 142), and monitors and utilizes the network bandwidth available to individual end users to provide the best cloud gaming experience. In some embodiments, the game server 118 provides multiple performance layers, including a layer that supports high-performance, real-time game sessions with high-definition video / media streaming and another layer that supports commentary media streaming, which has almost no delay or no delay corresponding to one or more actual game streams. Optionally, such commentary media streams are provided to one or more users via a media streaming site such as YouTube, and the media streaming site is optionally managed by the media streaming server 120.

[0048] According to some embodiments, the game server 118 is coupled to a game content host ( Figure 1 ). Examples of third-party gaming content include, but are not limited to, sports games, racing games, role-playing games (RPGs), and first-person shooter (FPS) games. Different instances of these games may have widely varying cloud hardware requirements and networks based on different associated latency requirements and expectations, output video resolutions and game server compute workloads and video encoding / streaming resources, and network bandwidth (e.g., to ensure an optimal user gaming experience - in some cases consistent with different subscription performance tiers).

[0049] As described above, the server system 114 hosts one or more game titles and is configured to execute multiple online game sessions of a single game title to support concurrent gameplay on multiple client devices. Each of the one or more game titles is loaded into the game server 118 before being executed to support any online game session. The game library server 122 is coupled to the game server 118 and is configured to provide executable game programs (e.g., Figure 5A 118) and static data items required to execute the executable game program. After the executable game program is loaded into the game server 118, it is executed by the CPU to instruct the GPU to use one or more image rendering programs (e.g., shader programs) to create multiple image frames for each of the multiple online game sessions. The shader library server 124 is coupled to the game server 118 and is configured to provide standard shader programs according to the needs of different game titles. If the shader program is selected for use by the game title, the shader program is retrieved from the shader library server 124, compiled by the game server 118, and stored locally in the game server 118 for rendering multiple image frames for each online game session. In some embodiments, each image frame of the online game session is rendered by an ordered sequence of compiled shaders (e.g., including vertex shaders, tessellation shaders, geometry shaders, and fragment shaders), wherein each shader implements one or more specific image rendering operations during the process of creating a corresponding image frame.

[0050] Figure 22 is an example online interactive gaming environment 200 in which multiple game servers 118 share a static memory pool 202 in a server system 114 according to some embodiments. The server system 114 includes one or more of the following: multiple game servers 118, a media streaming server 120, a game library server 122, and a shader library server 124, and is configured to host one or more game titles. That is, the server system 114 is communicatively coupled to multiple client devices 102 via one or more communication networks 112, and executes multiple online game sessions corresponding to the game titles in parallel, thereby allowing each of the multiple client devices 102 to participate in the gameplay under the game titles via the corresponding online game sessions. The game server 118 receives session requests from the multiple client devices 102 and initiates, joins, or terminates the online game session accordingly. In response to each session request, one of the game servers 118 is assigned to execute the corresponding online game session. During the process of executing each online game session, the game server 118 assigned to execute the corresponding online game session renders multiple image frames (i.e., session data streams) for the client device that has requested the online game session. In some cases, multiple image frames are rendered in response to receiving user input (e.g., a command to jump, shoot, or turn) from any subset of the online game session sharing the scene of the game title. Optionally, the game server 118 also generates a commentary media stream and provides it directly or indirectly via the media streaming server 120 to viewers who want to comment on the gameplay in real time.

[0051] Each game server 118 includes one or more processing units (e.g., CPU 138, GPU 140, and encoder 142), main dynamic memory 144, and one or more local caches 146. Main dynamic memory 144 is configured to store programs and data for use by CPU 138 and GPU during program execution. Optionally, each game server 118 has its own main dynamic memory 144. Optionally, a subset or all of game servers 118 share main dynamic memory 144. Local cache 146 is configured to temporarily store data, such as media content that has completed vertex post-processing and is waiting for primitive assembly in GPU 140, media content generated by GPU 140 and to be provided to encoder 142 for further encoding.

[0052] In some embodiments, the server system 114 includes a static memory pool 202 in addition to the main memory 144 and the cache 146. The static memory pool 202 is shared among a subset or all of the game servers 118 (e.g., 118-1, 118-2, ..., 118-N) and is configured to store one or more static data items used in multiple online game sessions of the same game title. The server system 114 further includes a memory controller 204 configured to manage the static memory pool 202. In some embodiments, each of the multiple game sessions is associated with a specific static data item and requires a specific graphics operation that can be performed by the corresponding GPU 140 using the specific static data item, and the specific static data item is stored in the static memory pool 202. The corresponding GPU 140 is configured to access the static memory pool 202 via the memory controller 204 to extract the specific static data item.

[0053] Note that in some embodiments, the static data items stored in the static memory pool 202 can be stored in the main dynamic memory 144. The GPU accesses the main dynamic memory 144 at a faster rate than the static memory pool 202. However, the main dynamic memory 144 needs to be refreshed or given a new electronic charge every few milliseconds to compensate for the charge leakage from the memory, and is therefore not efficient in power performance. If the static data items are moved to the static memory pool 202, a portion of the main dynamic memory 144 in the game server 118 can be released and used to store more programs and dynamic data, or a smaller main dynamic memory 144 can be used to reduce power consumption.

[0054] The server system 114 has a multi-level storage scheme including a local cache 146, a main dynamic memory 144, and a static memory pool 202. The local cache 146 includes at least one cache level (e.g., L1, L2). Compared with the main dynamic memory 144 and the static memory pool 202, the local cache 146 has a smaller size, is physically closer to the CPU and GPU, and can be accessed at a faster rate. The local cache 146 stores a copy of the instructions and data retrieved from the main dynamic memory 144 or the static memory pool 202 for direct use by the CPU 138 and the GPU 140. The main dynamic memory 144 is also referred to as the main memory. Each program executed by the CPU 138 and the GPU 140 is loaded into the main dynamic memory 144 and is moved to the cache on a block or instruction basis before execution. The size of the main dynamic memory 144 is crucial because it determines how many programs can be executed at one time and how much data can be easily used for the program. The static memory pool 202 is an addition to the cache 146 of the main dynamic memory 144. In some embodiments, the static memory pool 202 is also referred to as a level 4 cache that can be accessed by both the CPU 138 and the GPU 140, but is shared among multiple game servers 118. The static memory pool 202 has greater latency than the local cache 146 and the main dynamic memory 144, and is therefore used to store static data items that are not frequently used in online game sessions. In the server system 114, this multi-level storage scheme enables efficient execution of multiple online game sessions of a game title, at least in terms of power consumption.

[0055] Data storage in gaming environments

[0056] In some embodiments, the GPU 140 renders game content corresponding to the game title for an online game session according to a graphics pipeline process (e.g., an OpenGL rendering pipeline). The graphics pipeline process includes an ordered sequence of graphics operations, and each graphics operation is implemented based on one or more user-defined shader programs. In each graphics operation, the GPU 140 receives user input or output from the previous graphics operation and generates its own output as an intermediate output of the corresponding graphics operation or the final output of the entire graphics pipeline process. During this process, in addition to the user input and output received from the previous graphics operation, the shader program of the corresponding graphics operation also uses one or more static data items. For example, the static data items used in the graphics operation of the OpenGL rendering pipeline include one or more texture objects, vertex data cache objects, and constant data objects. The graphics pipeline process renders a sequence of image frames for each online game session, and each image frame includes a pixel array (also referred to as a page) with a resolution. In some embodiments, the static data items (e.g., texture objects) corresponding to the graphics pipeline process include one or more images with the same resolution as the image frames of the online game session.

[0057] Static data items are shared across the graphics pipeline processes of multiple online game sessions corresponding to a game title, i.e., do not change across different online game sessions. In some cases, static data items are stored in a portion of random access memory (GPU RAM) associated with each individual GPU 140. This portion of the GPU RAM is explicitly marked as "read-only" by the application program interface (API). In some cases, the GPU RAM is DRAM (e.g., main dynamic memory 144), which spends a significant portion of its power budget on memory refresh. Using GPU RAM to store static data items is a less than ideal solution given that there are memory technologies that do not require refresh cycles.

[0058] In some embodiments, a portion of the GPU's DRAM (e.g., a portion of the main dynamic memory) is replaced with a non-volatile memory that does not require refreshing and uses less power than DRAM. This non-volatile memory is used to store read-only assets (i.e., a subset or all of the static data items) used during the graphics pipeline. Examples of such non-volatile memory include PRAM, flash memory, and solid-state drives. In some embodiments, when PRAM is used, it can handle orders of magnitude fewer write cycles than DRAM. Therefore, in some embodiments, PRAM uses wear leveling and hash allocation. Each page in the PRAM has an associated hash that indicates the stored content, and reads and writes to the PRAM are indexed in this hash (using page table translation). When a game application corresponding to a game title releases its "read-only" assets (e.g., static data items), the memory controller 204 does not immediately release the page corresponding to the read-only asset, but only releases the page corresponding to the read-only asset if the PRAM is full. When released, the page corresponding to the read-only asset is discarded according to the least recently used (LRU) method. This way, the memory cells in the PRAM run approximately the same amount of workload (e.g., providing the same number of accesses to the same game title again) and are protected from excessive writes that could permanently damage the memory cells.

[0059] In some embodiments of cloud-based gaming applications (e.g., in server system 114), after replacing a portion of the GPU's DRAM (e.g., a portion of the main dynamic memory 144) with non-volatile memory, the DRAM of each GPU is reduced to the size required for read / write RAM. The non-volatile memory of each GPU 140 is merged with the non-volatile memory of other GPUs to form a centralized static memory pool 202 (e.g., a PRAM storage device) and shared among these GPUs 140. In some embodiments, access is routed to the static memory pool 202 via an intra-rack fast interconnect, and the intra-rack fast interconnect includes a bus arranged on each server rack or at a similar scale. In addition, a memory lookup table or hash is stored in the static memory pool 202.

[0060] The read-only portion of the GPU asset can be reused in multiple runs, and if many workloads attached to the centralized read-only storage are similar, they can be shared. That is, in some embodiments, if the static data item stored in the static memory pool 202 is extracted, the static data item is reused in two or more online game sessions. When writing to this static memory pool 202, the static data item is hashed according to a memory lookup table or hash to avoid repeated storage of the same static data item. In some embodiments, if a static data item already exists in the static memory pool 202, an enhanced LRU strategy is applied to avoid the static memory pool 202 being completely filled. In some embodiments, an offset is retrieved for an existing data item in the static memory pool 202, and the offset is saved for future memory access to the static data item. When the game application releases the static data item, it is not necessary to erase it until the static memory pool 202 space is used up.

[0061] In some embodiments, the latency for a round trip from each GPU 140 to the static memory pool 202 is measured and one or more GPU latency hiding methods are used to compensate for the latency. The one or more GPU latency hiding methods include, but are not limited to, write-merging, which combines write accesses to the static memory pool 202; tiled / compressed / spatially consistent textures and page-sized jumps from the static memory pool 202. In some cases, since the static memory pool 202 is used (rather than local DRAM) to store static data items, many warps are added to hide memory latency.

[0062] In some embodiments, the centralized static memory pool 202 includes PRAM and PRAM-like memories that typically have limited write cycles. The static memory pool 202 can be swapped out or replaced without any GPUs 140 being affected by the rack.

[0063] In some implementations, the server system 114 supports online gaming sessions for more than one game title. Based on previous workloads of uploading "read-only" assets to the static memory pool 202, the workload is assigned to a rack comprising multiple servers. Game titles are separated by tier, and popular game titles of the same tier are run by game servers 118 on the same rack to allow for a smaller static memory pool.

[0064] In some embodiments, if a large number of GPUs are running the same workload (e.g., an online gaming session corresponding to a gaming title), a shared DRAM pool (rather than a PRAM pool or equivalent) may be used in place of the static memory pool 202, and sufficient reuse may be achieved to amortize DRAM refresh and network power costs.

[0065] Figure 3A 300 is an example storage system 300 having a multi-level storage scheme configured to store data in a server system 114 according to some embodiments. According to the multi-level storage scheme, the storage system 300 of the server system 114 includes a plurality of caches 302 directly arranged in each individual server (e.g., a game server 118). More specifically, each of the plurality of caches 302 is directly associated with a processor of an individual server (e.g., a CPU 138 and a GPU 140 of a game server 118), and is configured to provide both instructions executed by the processor of the individual server and data used during the execution of the instructions. The memory system 300 of the server system 114 also includes a main dynamic memory 144, which is configured to store programs executed by one or more processors of the server (e.g., a game server 118) and data used during program execution. Each program includes a plurality of instructions, and during execution, these instructions are sequentially loaded into the cache 302 of the corresponding processor and wait to be executed by the corresponding processor. In some embodiments, a subset or all of the servers of the server system 114 share the main dynamic memory 144. The data bus couples a subset or all of the servers of the server system 114 to the main dynamic memory 144 to enable write and read access to the virtual machines 144. Optionally, the data bus includes a high-speed data bus via which the main dynamic memory 144 is accessed with a substantially small latency that is less than a threshold bus latency. In some embodiments, each server of the server system 114 has a corresponding main dynamic memory 144 that is accessible only to one or more processors of the corresponding server. For example, the game server 118 includes a main dynamic memory 144 that is accessible to both the CPU 138 and the GPU 140 of the game server 118. The main dynamic memory 144 of the game server 118 stores instructions 304 configured to execute one or more game titles, such as, Figure 5A, and dynamic data 306 associated with the online game session of each game title, such as first game session data 308 and second game session data 310. Specifically, the first game session data 308 optionally includes a texture page 308A, a vertex page 308B, a constant page 308C, and frame data 308D of one or more image frames generated by the GPU 140 for the corresponding online game session of the corresponding game title. Figure 5A and 5B More details about the programs and data stored in the main dynamic memory 144 in the server system 114 are explained.

[0066] According to the multi-level storage scheme, the storage system 300 of the server system 114 further includes a static memory pool 202, which is centralized and shared by multiple game servers 118 in the server system 114. The static memory pool 202 is configured to store one or more static data items, which are used to render image frames of online game sessions of the game title. Examples of the one or more static data items include, but are not limited to, texture objects 312, vertex data cache objects 314, and constant data objects 316. When multiple game servers 118 are assigned to execute multiple online game sessions, each game server 118 obtains one or more static data items from the static memory pool 202 and obtains one or more dynamic data items 306 from its corresponding main dynamic memory 144.

[0067] In addition, in some embodiments, the game server 118 further includes a shader cache 318 coupled to its GPU 140. The shader cache 318 is configured to store at least one sequence of compiled shaders so that the instructions of the compiled shaders can be directly loaded into the GPU 140 to render game content via the graphics pipeline process. In some embodiments, the server system 114 includes a plurality of compiled shader sequences 320, each configured according to performance criteria, and each compiled shader sequence 320 optionally includes a plurality of versions corresponding to different user input conditions. Optionally, a plurality of compiled shader sequences 320 are stored in the shader cache 318. Optionally, a plurality of compiled shader sequences 320 are stored in the static memory pool 202 and are moved to the shader cache 318 or directly to the cache of the GPU 140 if selected for game content rendering.

[0068] In addition, according to the multi-level storage scheme, the storage system 300 of the server system 114 includes a hard disk drive 324, which stores one or more executable game programs corresponding to one or more game titles. During the process of configuring the server system 114 to host game titles, the CPU 138 of the server system sends a program request 326 for the executable game program 328 corresponding to the game title. In response to the program request 326, the hard disk drive 324 returns the executable game program 328 corresponding to the game title to the CPU 138. In some embodiments, the hard disk drive 324 does not store the requested executable game program 328, and the CPU 138 sends a program request 326 to request the executable game program 328 from a remote server (e.g., a game library server 122 or another third-party game provider). The CPU 138 stores the executable game program 328 returned from the hard disk drive 324 or the remote server as a server-side game module 526 in the main dynamic memory 144.

[0069] In addition, in some embodiments, the hard disk drive 324 also stores static data items of one or more game titles. During the process of loading the game title, the CPU 138 of the server system sends a data request 330 for the static data items of the game title. In response to the data request 326, the hard disk drive 324 returns a subset or all of the static data items 332 corresponding to the game title to the CPU 138. In some embodiments, the hard disk drive 324 does not store specific static data items, and the CPU 138 sends a data request 330 to request specific static data items from a remote server (e.g., a game library server 122 or a shader library server 124). One or more static data items 332A returned from the hard disk drive 324 or the remote server are stored in the static memory pool 202. Alternatively, in some embodiments, one or more static data items 332B are stored in the main dynamic memory 144 to be directly accessed by the CPU and GPU of the game server 118.

[0070] In addition, the CPU 138 also sends a shader request to the shader library server to obtain one or more standard shader programs 336 used in association with the game title, and the returned shader programs 336 are stored locally in the shader cache 318. When executing an online game session corresponding to the game title, the CPU 138 and the GPU 140 generate a plurality of dynamic data items 338. These plurality of dynamic data items 338 are stored in the main dynamic memory 144 of the server system 114.

[0071] In some embodiments, the server system 114 includes a dynamic memory controller 204A and a static memory controller 204B, which are configured to manage write and read access to the main dynamic memory 144 and the static memory pool 202, respectively. In some embodiments, the server system 114 includes a single memory controller 204, which is configured to manage write and read access to the main dynamic memory 144 and the static memory pool 202. The main dynamic memory 144 includes a directory 340 that identifies the location of the instructions 304, dynamic data items 306, and static data items 332B (if any) in the main dynamic memory 144. In some embodiments, the static memory pool 202 includes a hash table 342 that identifies the location of the static data item 332A in the static memory pool 202. Optionally, the static data item 332 is identified based on context information. In the example, the static memory pool 202 stores a plurality of compiled shader sequences 320, and their addresses in the static memory pool 202 are identified using performance criteria, user input, and / or usage statistics of the corresponding graphics pipeline process in a hash table. Optionally, this context-based mapping in the hash table 342 is also applied to other static data items. The hash table protects the static memory pool 202 from repeatedly storing static data items associated with different situations, thereby wasting its memory resources.

[0072] Figure 3B is an example CPU 138 with multiple levels of cache according to some embodiments. In some embodiments, each CPU includes multiple CPU cores 350. Each CPU core 350 has a cache 352 that stores instructions executed by the corresponding CPU core and data used during the execution of the instructions. The cache 352 is directly coupled to the corresponding CPU core 350 and can immediately and directly provide instructions and data to the corresponding CPU core 350. In some embodiments, the CPU core 350 also includes a CPU cache 354 shared by multiple CPU cores 350 of the CPU 138. The CPU cache 354 stores data used during the execution of instructions stored in the CPU core 350. The CPU cache 354 feeds its data to the cache 352 directly coupled to the CPU core 350.

[0073] Figure 3Cis an example GPU 140 with multiple layers of cache according to some embodiments. In some embodiments, each GPU includes multiple GPU cores 360. Each GPU core 360 ​​has a cache 360 ​​that stores instructions executed by the corresponding GPU core and data used during the execution of the instructions. The cache 362 is directly coupled to the corresponding GPU core 360 ​​and can immediately and directly provide instructions and data to the corresponding GPU core 362. In some embodiments, the GPU core 360 ​​also includes a GPU cache 364 shared by multiple GPU cores 360 of the GPU 138. The GPU cache 364 stores data used during the execution of instructions stored in the GPU core 360. The GPU cache 364 feeds its data to the cache 362 directly coupled to the GPU core 360. In order to render the game content of the game title, writes and reads in the storage system 300 are managed by pages (i.e., by arrays of image pixels corresponding to image frames of the game content). Page-sized reads are cached in the local GPU memory hierarchy (L3-L1 cache) which may include a small pool of DRAM used as a last level cache, i.e., cached in cache 362 and GPU cache 364.

[0074] Figure 3D is an exemplary memory controller 204 that controls read and write access to a static memory pool 202 shared by multiple processors in a server system 114 according to some embodiments. The memory controller 204 includes one or more of an access control module 372, a page release module 374, a wear leveling module 376, and an LRU module 378. The access control module 372 is configured to access the static memory pool 202 based on a hash table 342 stored therein. The page release module 374 is configured to delete static data items from the static memory pool 202. Optionally, in response to a request to delete a specific data item, the page release module 374 determines whether the static memory pool 202 is full. Based on determining that the static memory pool 202 is not full, the page release module 374 modifies the hash table 342 to disable any access to the specific data item to be deleted. Based on determining that the static memory pool 202 is full, the page release module 374 deletes the specific data item from the static memory pool 202 to release the corresponding storage space.

[0075] The wear leveling module 376 is configured to monitor the rewrite of the memory cells in the static memory pool 202 and evenly distribute the rewrite in the static memory pool 202 for the purpose of extending the service life of the static memory pool 202. The LRU module 378 is configured to remove the least recently used static data items from the static memory pool 202. Specifically, in some embodiments, the LRU module 378 stores the age bit with each stored static data item in the static memory pool 202. When the memory limit has been reached, the static data items that have been less recently accessed will be removed starting from the oldest data item identified according to the age bit. When the LRU module deletes a static data item, the age bit changes for the static data items retained in the static memory pool 202.

[0076] Figure 4A 4 is an example data flow during a process 400 of loading a game title on a server system 114 according to some embodiments. The server system 114 includes a game server 118, a game library server 122, and a shader library server 124 that are remote from each other. The server system 114 is also remote from multiple client devices 102 that participate in gameplay of the game title hosted by the game server 118. The server system 114 (specifically, the game server 118) includes one or more CPUs 138 and multiple GPUs 140, and employs a multi-level storage scheme. According to the multi-level storage scheme, the server system 114 includes a main dynamic memory 144 that stores programs and data for use by the CPU and / or GPU during program execution, a static memory pool 202 stored in a non-volatile memory, and a memory controller 204 configured to manage the static memory pool 202. In response to receiving (401) a user request to load a game title, the CPU 138 sends (402) a program request to the hard disk drive 324 of the game server 118 or the game library server 122, which requests an executable game program (e.g., a server-side game module 526) corresponding to the game title. In some embodiments, the CPU 138 first sends the program request to the hard disk drive 324. If the hard disk drive 324 cannot provide a copy of the executable game program, the CPU 138 sends the program request to the game library server 122. In response to the program request, the hard disk drive 324 or the game library server 122 returns a copy of the executable game program, which is then stored (404) in the primary dynamic memory 144.

[0077] During the process of executing the executable game program, one or more static data items (e.g., texture objects) are required, so the CPU 138 sends (406) a data request to at least one of the hard disk drive 324, the game library server 122, and the shader library server 124, requesting a predetermined set of static data items. If the hard disk drive 324 does not have the corresponding data items, the CPU 138 sends the data request in a sequence (e.g., first to the hard disk drive 324, then to the game library server 122 and / or the shader library server 124). The sequence is the same for the set of static data items, or customized according to each static data item. For each of the first subset of the requested static data items, one of the hard disk drive 324, the game library server 122, and the shader library server 124 returns a copy of the corresponding data item, which is stored (408) in the static memory pool 202. In some embodiments, a second subset of the requested static data items is returned and stored (410) in the main dynamic memory 144. In some embodiments, the requested static data items include a third subset of the requested data items (e.g., one or more shader programs). One of the hard drive 324 and the shader library server 124 returns the third subset of the requested static data items and stores (412) them in the shader cache 318.

[0078] Figure 4B 4 is an example data flow 420 during the process of executing an online game session on the server system 114 (specifically, in the game server 118) according to some embodiments. The CPU receives (422) a plurality of session requests for initiating a plurality of online game sessions under a game title from a plurality of distributed client devices 102. Each client device 102 executes a real-time user interactive game application corresponding to the game title, and sends a corresponding session request from the game application of the corresponding client device 102. The corresponding session request includes user information of the requesting client device 102. After receiving the user information of the corresponding client device 102, the CPU 138 authorizes (424) the corresponding online game session based on the user information, and allocates (426) one of the GPUs 140 to execute the corresponding online game session. In some embodiments, two or more of the plurality of online game sessions are allocated to one of the GPUs 140. In other words, the two or more online game sessions must share the computing and storage resources of one GPU based on time division multiplexing (TDM), spatial multiplexing, or a combination thereof. Further details of GPU allocation in game content rendering are explained in U.S. Provisional Patent Application No. 62 / 646,824, filed on March 22, 2018, entitled “Method and System for Rendering and Encoding Content for an Online Interactive Gaming Session,” which is incorporated herein by reference in its entirety.

[0079] In response to each session request, the CPU 138 copies (428) the instructions of the executable game program of the game title from the main dynamic memory 144 to the local cache of the CPU 138, and executes the instructions one by one. Similarly, if instructed by the CPU 138, the corresponding GPU assigned to execute the corresponding online game session also loads (428) the relevant game rendering instructions in the executable game program to its local cache, and executes the game rendering instructions one by one.

[0080] In some embodiments, each of the plurality of online game sessions is continuously refreshed, for example, at a rate of 60 frames per second (fps). In some embodiments, each of the plurality of online game sessions is refreshed in response to a user action input to the corresponding session itself or a related session. For example, a subset of online game sessions is related when the subset of online game sessions are in the same game scene of a game title, and when any one of the subset of online game sessions inputs a user action, at least the game content delivered to each of the subset of online game sessions is updated.

[0081] Specifically, for an online game session, the CPU 138 receives (430) user input entered via a game application of a first client device participating in the online game session or another different game session sharing the same game scene with the online game session. In response, the CPU 138 executes (432) instructions of an executable game program corresponding to the game title to instruct (438) the GPU 140 assigned to execute the online game session to render a sequence of image frames according to the user input. During the execution of this online game session, dynamic data items associated with this online game session are stored in or extracted from the main dynamic memory 144 (434). Static data items associated with this online game session are extracted from the static memory pool 202 (436). In some embodiments (not shown here), a collection of static data items is extracted from the main dynamic memory 144.

[0082] The assigned GPU 140 then executes (439) game rendering instructions to render image frames for the online game session based on the user input. During execution of the online game session, the GPU 140 obtains (440) compiled shaders from the shader cache 318, extracts (442) static data items from the static memory pool 202, and stores (444) primary dynamic data items in or extracts (444) primary dynamic memory 144. In some embodiments (not shown here), the GPU also extracts one or more static data items from the primary dynamic memory 144. The image frames rendered by the GPU are encoded by the encoder 142 and streamed (446) to the client device 102 executing the online game session. In some embodiments, the image frames rendered by the GPU (e.g., a normal delayed stream intended for audience commentary) are streamed (448) to the client device 102 via the media streaming server 120.

[0083] Figure 4C is an example data flow in the shader optimization process 450 according to some embodiments. In some embodiments, an executable game program (e.g., Figure 5A The server-side game module 526 in the example embodiment includes a game application and a shader pipeline optimizer, which is configured to execute an online game session and a shader pipeline optimization process simultaneously. For example, when the game application executes an online game session in a graphics pipeline process in the foreground (operations 438-448), the shader pipeline optimizer optimizes the compiled shaders used in the graphics pipeline process in the background. In other words, when the graphics pipeline process (operations 438-448) is running, the game server 118 generates (450) one or more sequences of compiled shaders in the background to meet one or more performance criteria.

[0084] Specifically, in some embodiments, the CPU 138 collects (452) usage statistics for a set of online game sessions, optionally sampled from all active online game sessions associated with the game title. The usage statistics may be obtained from the local cache or main dynamic memory 144 of the CPU 138 and the GPU 140. The CPU 138 identifies and obtains (454) one or more performance criteria that need to be met during optimization. In some embodiments, the one or more performance criteria include one or more static data items stored in the static memory pool 202 or the main dynamic memory 144 (optional). Then, the CPU 138 sends (456) to the GPU a request to optimize a sequence of compiled shaders used to execute the active online game session based on the usage statistics and the performance criteria. For each of the one or more performance criteria, the GPU 140 modifies (458) one or more shaders in the compiled shaders of the sequence to improve the performance of the compiled shaders of the sequence relative to the corresponding performance criteria, thereby creating a modified sequence of compiled shaders. In some cases, the modified sequence of compiled shaders is optionally stored to replace the sequence of compiled shaders and used by subsequent online game sessions. In some cases, the compiled shader for the modified sequence is stored as a new version of the compiled shader for the sequence corresponding to the usage statistics and used when a subsequent game state of the online game session is consistent with the usage statistics. In some implementations, each compiled shader for the modified sequence is stored (460) in the shader cache 318 in association with the corresponding performance criterion. Alternatively, Figure 4C In some implementations not shown in FIG. 2 , the compiled shaders and performance criteria for the associated modified sequence are stored in the static memory pool 202 .

[0085] Figure 4D 4 is an example data flow 480 for obtaining static data items when executing multiple online game sessions according to some embodiments. As explained above, when multiple online game sessions are executed in parallel by the game server 118 (482), one or more CPUs assign each online game session to a corresponding GPU. During the execution of each online game session, the corresponding GPU 140 obtains (440) compiled shaders from the shader cache 318, extracts (442) static data items from the static memory pool 202, and stores (444) dynamic data items in or extracts (445) dynamic data items from the main dynamic memory. Figure 4DIn some embodiments, two client devices 102A and 102B respectively execute a first game session and a second game session in a plurality of online game sessions. For example, in response to receiving (484) a user input in a game application corresponding to a game title through client device 102A, both client devices 102A and 102B receive image frames rendered by game server 118. During the process of rendering the corresponding image frames, each of the plurality of game sessions (including the first game session and the second game session) is associated with a static data item and requires a graphics operation executable by the corresponding GPU using the static data item, and the static data item is stored in static memory pool 202.

[0086] For a first game session in the plurality of game sessions, the CPU assigns a graphics operation to a first GPU. The first GPU requests (486) a first copy of a static data item from memory controller 204 through the first GPU, and receives (488) a first copy of the static data item read from static memory pool 202 from memory controller 204. The first GPU then performs the graphics operation using the first copy of the static data item, and locally stores a first result of the graphics operation in association with the first game session in the plurality of game sessions (e.g., by using an identifier). Similarly, for a second game session in the plurality of game sessions, the CPU assigns the graphics operation to a second GPU. The second GPU requests (490) a second copy of the static data item from memory controller 204, and receives (492) a second copy of the static data item read from static memory pool 202 from memory controller 204. The second GPU performs the graphics operation using the second copy of the static data item, and locally stores a second result of the graphics operation in association with the second game session in the plurality of game sessions.

[0087] In some embodiments, for a third game session in the plurality of online game sessions, the CPU assigns the graphics operation to the third GPU. The third GPU determines that the first GPU has requested the first copy of the static data item within a predetermined time period. The third GPU obtains the first copy of the static data item directly from the first GPU. In some embodiments, the first GPU itself has not yet received the first copy from the memory controller, and the third GPU waits to obtain the first copy from the first GPU after the first GPU receives the first copy. In other words, the first GPU and the third GPU merge their requests for the copy of the static data item.

[0088] In some embodiments, each of the plurality of game sessions is associated with a priority, and the first GPU and the second GPU receive copies of static data items read from the static memory pool 202 from the memory controller according to the priorities of the first and second game sessions. For example, the first game session has a higher priority than the second game session. When the first GPU requests the first copy at the same time or within a threshold duration after the second GPU requests the second copy, the second GPU is suspended from receiving the second copy until the first GPU has received the first copy from the memory controller 204.

[0089] Server systems, client devices, game controllers, and media devices

[0090] Figure 5A is a block diagram illustrating a server system 114 according to some implementations. Figure 5B 1 is a block diagram illustrating data stored in a server 500 according to some embodiments. The server system 114 typically includes one or more processing units (e.g., CPU 18, GPU 140, and encoder 142), one or more network interfaces 504, memory 506, and one or more communication buses 508 for interconnecting these components (sometimes referred to as chipsets). The server system 114 may optionally include one or more input devices 510 to facilitate user input, such as a keyboard, mouse, voice command input unit or microphone, touch screen display, touch-sensitive input pad, gesture capture camera, or other input buttons or controls. In addition, the server system 114 may use a microphone and voice recognition or a camera and gesture recognition to supplement or replace the keyboard. In some embodiments, the server system 114 optionally includes one or more cameras, scanners, or light sensor units for capturing images of, for example, a graphic series code printed on an electronic device. The server system 114 may also include one or more output devices 512 that enable the presentation of user interfaces and display content, including one or more speakers and / or one or more visual displays.

[0091] The memory 506 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. The memory 506 optionally includes one or more storage devices located away from one or more processing units. The memory 506 or alternatively the non-volatile memory within the memory 506 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 506 includes a subset or all of the main dynamic memory 144, the static memory pool 202, and the shader cache 318. In some embodiments, the memory 506 or the non-transitory computer-readable storage medium of the memory 506 stores the following programs, modules, and data structures or a subset or superset thereof:

[0092] operating system 516, including processes for handling various basic system services and for performing hardware-related tasks;

[0093] A network communication module 518 for connecting the server 500 (e.g., the game server 118) to other devices (e.g., the server system 114, the client devices 102 and / or 104, and other servers in the media device 106) via one or more network interfaces 504 (wired or wireless) and one or more communication networks 112 (such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.);

[0094] A user interface module 520 for enabling presentation of information at the client devices 102 and / or 104 (e.g., a graphical user interface for presenting applications, widgets, websites and their web pages, and / or games, audio and / or video content, text, etc.);

[0095] a media device module 522 (optional), which is executed to provide server-side functionality for device provisioning, device control, and user account management associated with the media devices 106;

[0096] a location determination module 524 for determining a location of the client device 102 and / or 104 based on location information of any one of the client device ( 102 and / or 104 ) and the media device 106 ;

[0097] a server-side gaming module 526 for providing server-side functionality associated with one or more game titles, including, but not limited to, setting up gaming sessions, storing session state data and other gaming-related data, processing gameplay input from client devices 102, and rendering gaming output (e.g., a sequence of image frames) in response to the gameplay input;

[0098] a media streaming server module 538 for hosting a media streaming site, receiving concurrent auxiliary or supplemental media streams associated with an online gaming session, and providing the concurrent media streams to a client device 104 for simultaneous display with an online gaming session being executed by a gaming application 629 on the same client device 104 or a different client device 102;

[0099] A device / network assessment module 540 for assessing the device and network capabilities of the client device 102 and / or 104, including but not limited to assessing the network bandwidth of the connection to the client device 102 / 104 and assessing whether the client device 102 / 104 has the modules or applications required to play the game;

[0100] Data transfer module 542 for providing data (e.g., game controller configuration 556), software updates, etc. to client devices 102 / 104; and

[0101] Server system data 544, including:

[0102] Client device settings 552 for storing information associated with client devices 102 and 104, including common device settings (e.g., service tier, device model, storage capacity, processing power, communication capabilities, etc.);

[0103] Media device settings 554 (optional) for storing information associated with a user account of the media device module 522, including one or more of account access information and device setting information (e.g., service tier, device model, storage capacity, processing capability, communication capabilities, etc.);

[0104] location / proximity data 556 , including information related to the presence, proximity, or location of any of the client device 102 , the commentator client device 104 , and the media device 106 ;

[0105] Game controller configuration 558, for storing controller configurations for various game titles; and

[0106] User information 558 is used to store information related to one or more game titles hosted on the server system 114 (e.g., Figure 6 Information associated with a user account for each of the gaming applications 629), including, for example, user account information (e.g., identification and password), membership type, preferences, and activity history; and

[0107] Game session data 560, used to store data related to the game session, including, for example, data 562 for a first game session and data 564 for a second game session, wherein the session data 560 for each game session includes, but is not limited to, a frame rate associated with the corresponding game session, rendering specifications, normal latency requirements, GPU allocation information, encoder allocation information, an identifier of the related session, latest state information, a shader pipeline identifier, a texture page 308A, a vertex page 308B, a constant page 308C, and frame data 308D of one or more image frames.

[0108] Static data items 568 for storing data shared by multiple online game sessions, wherein static data items 562 include, but are not limited to, one or more of texture objects, vertex data cache objects, constant data objects, and compiled shaders for sequences optimized according to usage statistics to meet different performance criteria;

[0109] A sequence of compiled shaders 570 for storing shaders that have been compiled and are ready to be executed in a graphics pipeline process, wherein the sequence of compiled shaders includes but is not limited to a default sequence of compiled shaders for rendering game content by default and multiple sequences of compiled shaders optimized based on usage statistics to meet different performance standards; and

[0110] Performance metrics 572 and usage statistics 574 are stored in association with the optimized sequence of compiled shaders, wherein each performance metric 572 corresponds to total power consumption, code size, average latency, total cache usage, and total memory usage for a sequence of compiled shaders during execution of multiple online game sessions, and the usage statistics 574 characterize the execution of each of the multiple online game sessions.

[0111] In some embodiments, the server-side game module 526 includes the following programs, modules, or subsets or supersets thereof:

[0112] a gaming application 528 for executing an online gaming session corresponding to one or more game titles, including authorizing the online gaming session, assigning a GPU to execute a subset of the session, instructing the assigned GPU to execute the corresponding session, and encoding image frames rendered by the assigned GPU;

[0113] a controller device module 530 for interacting with and controlling the dedicated game controller device 102B, e.g., receiving user requests and inputs from the game controller device 102B;

[0114] a shader compiler 532 for compiling shaders for use in an ordered sequence of graphics operations of a graphics pipeline process that renders image frames for an online game session corresponding to a game title;

[0115] a shader pipeline optimizer 534 for modifying one or more shaders in a sequence of compiled shaders used to execute an online game session to improve performance of the sequence of compiled shaders with respect to a performance criterion;

[0116] The game rendering module 536 is used to control the GPU 140 to render image frames using a sequence of default compiled shaders or a sequence of optimized compiled shaders during the execution of an online game session.

[0117] In some embodiments, the server system 114 also includes one or more memory controllers 204 that are interconnected to the CPU, GPU, main dynamic memory 144, and static memory pool 202 using one or more communication buses 508. The one or more memory controllers 204 include a first memory controller 204A configured to manage the main dynamic memory 144 and a second memory controller 204B configured to manage the static memory pool 202. As shown above, elements 516-542 and 552-566 are stored in the main dynamic memory and are accessible via the first memory controller 204A. Static data items 568 are stored in the static memory pool 202 and are accessible via the second memory controller 204B. Compiled shaders 570, performance metrics 572, and usage statistics 574 are optionally stored together in the shader cache 318 or the static memory pool 202. Optionally, usage statistics 574 collected for a particular set of online game sessions are stored as game session data 562 in the main dynamic memory 144.

[0118] Each element identified above can be stored in one or more aforementioned storage devices, and corresponds to an instruction set for performing the above functions. The modules or programs (i.e., instruction sets) identified above do not need to be implemented as separate software programs, processes, modules or data structures, and therefore, each subset of these modules can be combined or otherwise rearranged in various implementations. In some embodiments, memory 506 optionally stores subsets and data structures of the modules identified above. In addition, memory 506 optionally stores additional modules and data structures not described above.

[0119] Figure 61 is a block diagram illustrating an example client device 600 (e.g., client device 102 or 104) of a gaming environment 100 according to some embodiments. Examples of client devices 600 include, but are not limited to, mobile phones, tablet computers, laptop computers, desktop computers, and wearable personal devices. Client device 600 includes one or more processing units (CPUs) 602, one or more network interfaces 604, memory 606, and one or more communication buses 608 for interconnecting these components (sometimes referred to as chipsets). Client device 600 includes one or more input devices 610 that facilitate user input, such as a keyboard, a mouse, a voice command input unit or microphone, a touch screen display, a touch-sensitive input pad, a gesture capture camera, or other input buttons or controls. In addition, some client devices 200 use microphones and voice recognition or cameras and gesture recognition to supplement or replace keyboards. In some embodiments, client device 600 includes one or more cameras, scanners, or photo sensor units for capturing images of, for example, a graphic series code printed on an electronic device. Client device 600 also includes one or more output devices 612 capable of presenting a user interface and displaying content, including one or more speakers and / or one or more visual displays. Optionally, the client device 600 includes a location detection device 614, such as a GPS (global positioning satellite) or other geo-location receiver, for determining the location of the client device 600. The client device 600 may also include a proximity detection device 615, such as an IR sensor, for determining the proximity of the media device 106 and / or the player client 104. The client device 600 may also include one or more sensors 613 (e.g., accelerometers, gyroscopes, etc.) for sensing the motion, orientation, and other parameters of the client device 600, which may be used as input.

[0120] Memory 606 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally, includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Memory 606 optionally includes one or more storage devices remotely located from one or more processing units 602. Memory 606 or, alternatively, non-volatile memory within memory 606, includes non-transitory computer-readable storage media. In some embodiments, memory 606 or non-transitory computer-readable storage media of memory 606 stores the following programs, modules, and data structures, or a subset or superset thereof:

[0121] operating system 616, including processes for handling various basic system services and for performing hardware-related tasks;

[0122] A network communication module 618 for connecting the client device 600 to other devices (e.g., server system 114, media device 106, and other client devices 102 or 104) via one or more network interfaces 604 (wired or wireless) and the local network 110 and one or more networks 112 (such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.);

[0123] a user interface module 620 for enabling presentation of information at the client device 600 via one or more output devices 612 (e.g., a display, speakers, etc.) (e.g., a graphical user interface for presenting applications, widgets, websites and their pages, and / or games, audio and / or video content, text, etc.);

[0124] an input processing module 622 for detecting one or more user inputs or interactions from one of the one or more input devices 610 and interpreting the detected inputs or interactions;

[0125] Web browser module 624 for navigating, requesting (e.g., via HTTP), and displaying websites and their web pages, including a web interface for joining a session 116;

[0126] a media device application 626 for interacting with the media device 106 , including logging into a user account associated with the media device 106 , selectively controlling the media device 106 if associated with the user account, and editing and commenting on settings and data associated with the media device 106 ;

[0127] A client gaming module 628 for providing client functionality associated with one or more game titles, including, but not limited to, setting up a gaming session, locally storing session state data and other gaming-related data, receiving gameplay input from input device 610, and providing gaming output (e.g., a sequence of image frames) in response to the gameplay input, wherein in some embodiments, the client gaming module 628 further includes one or more gaming applications 629 corresponding to the one or more game titles;

[0128] A data download module 630 for downloading data (e.g., game controller configurations 640, client game modules 628 and other applications, updates to modules and applications, and data in memory 606) from the server system 114 and other content hosts and providers; and

[0129] Client data 632 stores at least data associated with the game application 629 and other applications / modules, including:

[0130] Client device settings 634, for storing information associated with the client device 600 itself, including common device settings (e.g., service tier, device model, storage capacity, processing power, communication capabilities, etc.);

[0131] Media device settings 636 for storing information associated with a user account of the media device application 626, including one or more of account access information and device settings (e.g., service tier, device model, storage capacity, processing capability, communication capabilities, etc.);

[0132] Game application settings 638 for storing information associated with a user account for a game application, including one or more of account access information, in-game user preferences, gameplay history data, and other player information;

[0133] Game controller configuration 640 for storing information associated with a configuration of a client game module 628 of a game application 628 (e.g., a configuration received from game controller configuration 456 of FIG. 4 );

[0134] Location data 642 , including information associated with the presence, proximity, or location of any of client devices 102 and 104 and media device 106 ; and

[0135] Local game session data 644 includes information related to online game sessions executed on client device 600 in association with different game titles, such as the latest state information, texture pages, vertex pages, constant pages, and frame data for one or more image frames associated with each online game session.

[0136] exist Figure 6 In some embodiments not shown in the figure, the client game module 628 includes a command module for transmitting messages (e.g., setup commands) between the client device 102 and the media device 106 in menu mode; a streaming module for transmitting gameplay commands (e.g., to the media streaming server 120) between the client device 102 and the server system 114 in gameplay mode; and a controller module for providing a gameplay input interface to the game application. In some embodiments, the client game module 628 is part of the media device application 626 or another application in the memory 606 (e.g., a submodule). In some embodiments, the client game module 628 is part of the operating system 616. In some embodiments, the client game module 628 is a different module or application.

[0137] In some implementations of the client device 102, the media device application 626 (and corresponding media device settings 636) and the game application 629 (and corresponding game application settings 638) are optional. Depending on the particular game that the client device 102 is invited to join, the media device application 626 and the game application 629 may not be required to play. If any of these applications are required in order to play the game (e.g., the game uses the client game module 628 in the media device application 626), and the application is not in the memory 606, the client device 102 may be prompted to download the application.

[0138] In some implementations, the client device 102 includes a game controller device. The game controller device includes one or more input devices 610 that facilitate user input, such as a microphone (or voice command input unit), buttons, directional keys, and joysticks. In some implementations, the game controller 102 also includes gesture recognition features. The game controller 102 also includes one or more output devices that facilitate audio output and / or visual output, including a speaker 320, an LED, and / or optionally a display.

[0139] Each of the elements identified above can be stored in one or more aforementioned memory devices, and corresponds to the instruction set for performing the above functions. The modules or programs (that is, instruction sets) identified above do not need to be implemented as independent software programs, processes, modules or data structures, and therefore the various subsets of these modules can be combined or otherwise rearranged in various implementations. In some embodiments, memory 606 is optionally stored in the subset and the data structure of the modules identified above. In addition, memory 606 optionally stores additional modules and data structures not described above.

[0140] Figure 7 is a block diagram illustrating an example media device 106 of a gaming environment 100 according to some embodiments. The media device 106 typically includes one or more processing units (CPUs) 702, one or more network interfaces 704, memory 706, and one or more communication buses 708 for interconnecting these components (sometimes referred to as chipsets). Optionally, the media device 106 includes a proximity / position detection unit 710, such as an IR sensor, for determining the proximity of a client device 102 or 104.

[0141] Additionally, in some embodiments, the media device 106 includes a radio that implements one or more communication networks and allows the media device 106 to communicate with other devices (e.g., with the client device 102). In some embodiments, the radio is capable of communicating data using any of a variety of custom or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.5A, WirelessHART, MiWi, Whispernet, Bluetooth low energy, etc.), custom or standard wired protocols (e.g., Ethernet, HomePlug, etc.), and / or other suitable communication protocols including those not yet developed as of the date of this application.

[0142] The memory 706 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. The memory 706 optionally includes one or more storage devices remotely located from the one or more processing units 702. The memory 706 or, alternatively, the non-volatile memory within the memory 706 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 706 or the non-transitory computer-readable storage medium of the memory 706 stores the following programs, modules, and data structures, or a subset or superset thereof:

[0143] operating system 716, including processes for handling various basic system services and for performing hardware-related tasks;

[0144] A network communication module 718 for connecting the media device 106 to other computers or systems (e.g., server system 114 and client device 102) via one or more network interfaces 704 (wired or wireless) and one or more networks 112 (such as the Internet, other wide area networks, local area networks, metropolitan area networks, cable TV systems, satellite TV systems, IPTV systems, etc.);

[0145] a content decoding module 720 for decoding content signals received from one or more content sources (e.g., server system 114 output from gaming session 116) and outputting the content in the decoded signals to an output device 108 coupled to media device 106;

[0146] a proximity determination module 722 for determining the proximity of the client devices 102 and / or 104 based on proximity related information detected by the proximity detection unit 710 or provided by the server system 114;

[0147] A media display module 724, used to control media display; and

[0148] a game application 726 for controlling the life cycle of all game components on the media device 106 , receiving messages (e.g., commands) from the game controller 102 , and for relaying game controller input to the server system 114 ;

[0149] a game user interface module 728 for accessing user content (e.g., profile, avatar, purchased games, game catalog, friends, messaging) and optimized graphics for display, and for receiving input from the client device 102 during gameplay;

[0150] an interface control module 730 for interfacing communications between the game application 726 and the game user interface module 728 and for initiating a session pairing request with the server system 114 during game launch;

[0151] The server interface module 732 is used to communicate with the server system 114 and the authentication system (if any), including:

[0152] an authentication submodule 734 for communicating with an authentication system to authenticate a linked user of a game controller;

[0153] A session submodule 736 for receiving game session information (e.g., session id, game server provisioning IP address, port, etc.) from the server system 114 and for invoking a callback on the interface control module 730 that provides this information; and

[0154] a streaming submodule 738 for receiving and displaying in real time during gameplay a game stream transmitted from a media streaming server 120 of the server system 114; and

[0155] The media device data 740 is used to store at least the following data:

[0156] Media device settings 742, for storing information associated with a user account of a media device application, including one or more of account access information and device setting information (e.g., service tier, device model, storage capabilities, processing capabilities, communication capabilities, etc.);

[0157] Location data 744 , including information associated with the presence, proximity, or location of any of the client devices 102 and / or 104 and the media device 106 ; and

[0158] Local game session data 746 includes information associated with online game sessions executed on client device 600 in association with different game titles, such as the latest state information, texture pages, vertex pages, constant pages, and frame data for one or more image frames associated with each online game session.

[0159] Each element identified above can be stored in one or more aforementioned memory devices, and corresponds to the instruction set for performing the above functions. The modules or programs (i.e., instruction sets) identified above do not need to be implemented as separate software programs, processes, modules or data structures, and therefore the various subsets of these modules can be combined or otherwise rearranged in various implementations. In some embodiments, memory 706 optionally stores subsets and data structures of the modules identified above. In addition, memory 706 optionally stores additional modules and data structures not described above.

[0160] Static memory pool

[0161] Fig. 8A and 8B 800 is a flowchart illustrating an example method for managing storage capacity of a server system hosting one or more game titles according to some embodiments. The method 800 is optionally stored in a computer memory or non-transitory computer-readable storage medium (e.g., Figure 5A The instructions in the server-side game module 526 in the method 800 are governed by instructions in the server-side game module 526 in the method 800, and the instructions are executed by one or more processors of the server system 114 (e.g., the CPU and GPU of the game server 118). The computer-readable storage medium may include a disk or optical disk storage device, a solid-state storage device such as flash memory, or other one or more non-volatile storage devices. The instructions stored on the computer-readable storage medium may include one or more of the following: source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Some operations in the method 800 may be combined and / or the order of some operations may be changed.

[0162] The server system 114 includes one or more CPUs 138, multiple GPUs 140, a main dynamic memory 144 storing programs and data for use by the CPU and / or GPU during program execution, a static memory pool 202 stored in a non-volatile memory, and a memory controller 204 configured to manage the static memory pool. Each GPU 140 includes a local cache and is configured to access the static memory pool 202 via the memory controller 204. In some embodiments, the static memory pool 202 is arranged to be close to and shared by the one or more CPUs 138 and multiple GPUs 140 of the server system 114. The CPU 138 and the GPU 140 access the static memory pool 202 via a data bus. In addition, in some embodiments, the main dynamic memory 144 is arranged to be close to and shared by the one or more CPUs and multiple GPUs of the server system. The CPU 138 and the GPU 140 access the main dynamic memory 144 via a data bus. In some embodiments, the static memory pool 202 is at a certain distance from each of the one or more CPUs and multiple GPUs of the server system 114, and the distance is less than a predetermined distance threshold (e.g., <3 meters). That is, the static memory pool 202 is physically located on a server rack, and the one or more CPUs and multiple GPUs of the server system 114 are arranged on the server rack.

[0163] In some embodiments, the memory controller 204 is configured to manage the primary dynamic memory and the static memory pool. Alternatively, in some embodiments, the memory controller 204 includes a first memory controller, and the server system further includes a second memory controller that is different from the first memory controller and is configured to manage the primary dynamic memory.

[0164] The server system 114 executes (802) multiple game sessions of a game title in parallel on one or more CPUs 138. Each of the multiple game sessions is associated with a static data item and requires a graphics operation executable by the corresponding GPU 140 using the static data item and storing the static data item in the static memory pool 202. In some embodiments, the graphics operation is part of the OpenGL graphics rendering pipeline and includes one of vertex processing, tessellation, geometry processing, primitive assembly, rasterization, fragmentation, and presampling. In some embodiments, the static data item is predefined by the developer of the game title during the game development phase. When the game title is initially loaded onto the server system, the predefined static data items are loaded into the static memory pool 144. In some embodiments, the static data item includes one or more standard parameters used in the OpenGL rendering pipeline. Examples of static data items include, but are not limited to, standard textures, vertices, geometries, and constants initially loaded from the OpenGL library.

[0165] For a first game session among multiple game sessions executed (804) on one or more CPUs 138, the CPU 138 assigns (806) a graphics operation to a first GPU. The first GPU requests (808) a first copy of a static data item from the memory controller 204, and receives (810) a first copy of the static data item read from the static memory pool 204 from the memory controller. In some embodiments, after receiving the first copy of the static data item read from the static memory pool from the memory controller, the first GPU temporarily stores the first copy in a local cache or main dynamic memory. The first GPU then performs (812) the graphics operation using the first copy of the static data item, and locally stores (814) a first result of the graphics operation in association with the first game session among the multiple game sessions. In some embodiments, the first GPU has multiple GPU cores and a common cache shared by the multiple GPU cores, and each GPU core includes a corresponding cache memory that stores instructions and data used by the corresponding GPU core.

[0166] In some embodiments, the server system 114 includes a shader cache that stores one or more compiled shaders used by the plurality of GPUs to perform graphics operations. Further, in some embodiments, the server system 114 identifies a performance criterion for a first game session in the plurality of game sessions and selects one of the plurality of compiled shaders based on the performance criterion. The selected one of the plurality of compiled shaders is used by the first GPU to perform graphics operations.

[0167] Similarly, for a second game session (816) of the plurality of game sessions executed on the one or more CPUs, CPU 138 assigns the graphics operation to a second GPU (818). The second GPU requests (820) a second copy of the static data item from the memory controller and receives (822) from memory controller 204 a second copy of the static data item read from static memory pool 202. The second GPU then performs (824) the graphics operation using the second copy of the static data item and locally stores a second result of the graphics operation in association with the second game session of the plurality of game sessions.

[0168] In some embodiments, the main dynamic memory 144 includes a dynamic random access memory, and each of the local cache of the GPU and the local cache of one or more CPUs includes a static random access memory. The static memory pool includes one of a solid-state drive, a flash memory, and a phase-change random access memory. The access rate to the local cache is faster than the access rate to the main dynamic memory 144, which is faster than the access rate to the static memory pool. In some embodiments, the delay caused by access to the static memory pool is compensated during the rendering of the game content in the corresponding GPU.

[0169] In some implementations, a first result of the graphics operation is stored in a local cache of the corresponding GPU in association with a first gaming session in the plurality of gaming sessions (e.g., Figure 3C 364 in the GPU cache). The graphics operation is a first graphics operation and is followed by a second graphics operation. During execution of the second graphics operation, the first GPU 140 obtains a first result of the first graphics operation from the local cache and uses the first result of the first graphics operation to perform the second graphics operation. Optionally, the second graphics operation follows the first graphics operation directly or indirectly. In some embodiments, the first graphics operation and the second graphics operation are both part of a graphics pipeline process (e.g., an OpenGL rendering pipeline) that generates a game content stream in response to a user request or input received from a client device participating in a first game session in a plurality of game sessions. In one example, the first graphics operation and the second graphics operation are both two consecutive operations in the OpenGL rendering pipeline. The first result is an intermediate result generated in the OpenGL rendering pipeline, and the first result can be cleared from the local cache after the second graphics operation uses the first result.

[0170] Alternatively, in some embodiments, the first result of the graphics operation is stored in the main dynamic memory 144 in association with the first game session in the plurality of game sessions. The graphics operation includes a first graphics operation followed by a second graphics operation. During execution of the second graphics operation, the first GPU 140 obtains the first result of the graphics operation from the main dynamic memory 144 and uses the first result of the first graphics operation to execute the second graphics operation following the first graphics operation. Optionally, the first result is stored as part of the dynamic data 306 (e.g., game session data 562).

[0171] In some implementations, the server system 114 further includes an encoder 142. The encoder 142 encodes the first result of the encoder into an image frame associated with a first game session in the plurality of game sessions. The image frame is provided to a client device participating in the first game session in the plurality of game sessions.

[0172] In some implementations, the static data item includes one or more compiled shaders used by the plurality of GPUs 140 to perform graphics operations. Further, in some implementations, when the static data item includes a plurality of compiled shaders, the first GPU identifies a performance criterion for a first gaming session in the plurality of gaming sessions and selects one of the plurality of compiled shaders based on the performance criterion. The selected one of the plurality of compiled shaders is used by the first GPU to perform graphics operations.

[0173] Note that the static data item is shared among multiple game sessions including a first game session and a second game session in the multiple game sessions. In some embodiments, each of the multiple game sessions is associated with a priority, and each of the first GPU and the second GPU receives a corresponding copy of the static data item read from the static memory pool from the memory controller according to the priority of the first game session and the second game session. For example, the first game session has a higher priority than the second game session, and the first GPU requests the first copy at the same time or within a threshold duration after the second GPU requests the second copy. The second GPU is suspended from receiving the second copy until the first GPU has received the first copy from the memory controller.

[0174] In some implementations, the static memory pool 202 includes a hash table, and according to the hash table, a plurality of static data items are stored in association with the game title in the static memory pool 202. In some implementations, the static data item is associated with a tag indicating that the static data item is read-only.

[0175] In some embodiments, the memory controller 204 receives a release request for releasing a static data item. In response to the release request, the memory controller 204 determines whether a static (e.g., cold) memory pool is full. According to the static memory pool being full, the memory controller 204 releases the static data item based on an LRU page replacement method. According to the static memory pool being not full, the memory controller 204 suspends releasing the static data item. The memory controller 204 includes a page release module 374 configured to release data stored in the static memory pool 202.

[0176] In some embodiments, the static memory pool 202 is divided into a plurality of pages, and each of the plurality of pages has a page hash that identifies the location of data stored in the corresponding page. The memory controller 204 reads from and writes to the static memory pool 202 according to the page hash of each page. In an example, this static memory pool 202 includes PRAM.

[0177] It should be understood that the Fig. 8A and 8B The specific order of operations in is exemplary only and is not intended to indicate that the described order is the only order in which the operations may be performed. As described herein, one of ordinary skill in the art will recognize various ways to perform online game sessions using data stored in a static memory pool. In addition, it should be noted that the present invention does not describe any method 900 (e.g., Fig. 9 The details of the other processes described above can also be applied in a similar manner to Figures 8A-8B The method 800 is described. For the sake of brevity, these descriptions are not repeated here.

[0178] Fig. 9 is a flow chart illustrating an example method 900 for loading a game title into a server system 114 according to some implementations. The method 900 is optionally implemented by a computer stored in a computer memory or a non-transitory computer readable storage medium (e.g., Figure 5A The computer-readable storage medium may include a magnetic or optical disk storage device, a solid-state storage device such as flash memory, or other one or more non-volatile storage devices. The instructions stored on the computer-readable storage medium may include one or more of the following: source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Some operations in method 900 may be combined and / or the order of some operations may be changed.

[0179] The server system 114 includes one or more CPUs 138, multiple GPUs 140, a main dynamic memory 144 storing programs and data for use by the CPU and / or GPU during program execution, a static memory pool 202 stored in a non-volatile memory, and a memory controller 204 configured to manage the static memory pool. Each GPU 140 is configured to access the static memory pool 202 via the memory controller 204.

[0180] The CPU 138 loads (902) an executable game program corresponding to the game title (e.g., the server-side game module 526) into the main dynamic memory 144. The CPU 138 is configured to execute (904) multiple online game sessions in parallel, and each game session corresponds to the executable game program. Each of the multiple online game sessions requires (906) graphics operations that can be performed by a corresponding GPU assigned to render the corresponding online game session.

[0181] The CPU 138 identifies (908) static data items used in graphics operations for the plurality of gaming sessions. In some implementations, the static data items include a first static data item, and the first static data item is identified by the CPU 138 in a list of predetermined static data items. The list of predetermined static data items includes a first subset of static data items and a second subset of static data items. The memory controller stores the first subset of static data items including the first static data item in the static memory pool 202, and stores the second subset of static data items in the main dynamic memory 144.

[0182] The memory controller 204 stores (910) the static data items in a static memory pool and enables the GPU executing the online game session to access (912) the static data items while performing graphics operations. In some embodiments, the static data items are standard static data items used when rendering the game title. Before storing the static data items in the static memory pool, the CPU 138 retrieves the static data items from the hard drive 324 of the server system 114 or a remote library server. For example, the static data items include shader programs stored in the shader library server 124. The CPU 138 obtains the shader program from the shader library server 124 during the process of loading the game title and stores it in the static memory pool 202. Alternatively, in some embodiments, the static data items are custom static data items predefined by the developer of the game title and stored in the hard drive 324 of the server system 114. Before storing the static data items in the static memory pool, the CPU 138 retrieves the static data items from the hard drive 324 of the server system 114. Reference above Figure 4A Further details regarding loading game titles onto server system 114 are explained.

[0183] In some embodiments, based on the executable game program, the CPU 138 assigns each of the multiple GPUs to a subset of the multiple online game sessions to be executed thereon. For each of the multiple game sessions, during the execution of graphics operations by the corresponding GPU, the corresponding GPU 140 obtains a copy of the static data item from the static memory pool via the memory controller and uses the copy of the data item to perform graphics operations. In some embodiments, for each of the multiple game sessions, during the execution of graphics operations by the corresponding GPU, the corresponding GPU 140 loads a copy of the static data item into the corresponding local cache of the corresponding GPU. In addition, in some embodiments, for each of the multiple game sessions, during the execution of graphics operations by the corresponding GPU, the corresponding GPU 140 stores dynamic data items and user information in the main dynamic memory 144 in association with the corresponding online game session. In some embodiments, for a subset of the multiple game sessions, the GPU 140 loads a copy of the static data item into the GPU's local cache at one time, and the copy of the static data item is shared by a subset of the GPUs corresponding to the subset of the multiple game sessions. Reference above Figure 4B and 8A -8B explains more details about using static data items to render image frames for an online game session.

[0184] It should be understood that the Fig. 9The specific order of operations in the method 800 is exemplary only and is not intended to indicate that the described order is the only order in which the operations may be performed. As described herein, one of ordinary skill in the art will recognize various ways to load a game title to the server system 114. In addition, it should be noted that the method 800 (e.g., Figures 8A-8B The details of other processes described in the above description can also be similar to Fig. 9 The method 900 is described in the following manner. For the sake of brevity, these details are not repeated here.

[0185] Shader Optimization

[0186] During the process of rendering image frames for an online game session, GPU 140 obtains shaders in source form (GLSL, HLSL) or intermediate form (DXIL, SPIR-V) and other pipeline state data, and compiles shaders for use by the GPU to render game content (e.g., output a sequence of image frames associated with an online game session). Shader compilation can be expensive. For example, some game applications require compiling more than 100,000 shaders, each of which takes several milliseconds. If shader compilation is completed in real time in response to user input, it will take extended loading time and affect the user experience. In some cases, shader compilation is simplified in order to reduce compilation time, and however, due to the time sensitivity of shader compilation, the quality of the resulting compiled shader will be affected. / / This paragraph to the next 8 paragraphs copied from the public content.

[0187] In some embodiments of the present application, GPU 140 is connected to computing resources in server system 114 through a fast path. A full range cache of the compiled pipeline is prepared in advance, indexed by a hash of the compiler input, and stored locally to eliminate or reduce shader compilation time. In addition, in some embodiments, a full range of performance analysis can be performed to identify candidate shaders to be compiled manually or through a slower but more efficient shader pipeline compiler (e.g., Figure 5A In some embodiments, compiled shaders are dynamically replaced in a full-scope cache (e.g., shader cache 318) with higher performance shaders or shaders with some other characteristics. In some embodiments, different collections of compiled shaders are stored for deployment in specific situations, including shaders specifically used in the performance-guided optimization phase. In some embodiments, as improvements are made to the shader pipeline compiler (e.g., compiler 532), shaders used in rendering game content are recompiled, and the game application itself is not affected by these shader optimization operations. In some embodiments, without disrupting the game driver (in Figure 5AIn some embodiments, a dedicated shader is compiled, tested, and selected for a specific game title without disrupting the game driver running on GPU 140 (i.e., server-side game module 526).

[0188] In some embodiments, the game driver (specifically, the shader pipeline optimizer 534 of the server-side game module 526) transparently intercepts the shader pipeline creation / compilation call in the game driver, which runs on each GPU serving the graphics. The game driver evaluates the hash of all inputs of the shader compiler, and sends the request for the matching compiled shader to the shader pipeline server (or shader cache). In the case where the shader pipeline server or cache returns that there is no matching shader (or timeout), the game driver uses a traditional compilation pipeline and stores the result to the shader pipeline server or shader cache, which is also used for the online game session. The shader pipeline server can be implemented in many ways. The shader pipeline server is optionally a central single server, or a distributed / joint server set using standard distributed database technology (including replication for lower latency access).

[0189] In some embodiments, the game driver (specifically, the shader pipeline optimizer 534 of the server-side game module 526) performs full-range analysis and performance guidance optimization. Examples of full-range analysis include timing analysis, shader performance analysis, and power consumption analysis. Specifically, in some embodiments, a sample set of sessions is selected from all online game sessions associated with the game title. These loads (i.e., sampled sessions) are executable by a game driver that specifically collects the time required to run each shader pipeline and reports to a central database. Optionally, the sampled sessions are analyzed in the background using a game server 118 that simultaneously runs an online game session in the foreground. In some cases, such measurements may disrupt the game application (and therefore interfere with the user experience), so the sampled sessions are analyzed on a separate pipeline optimizer server 150. This allows the user experience to maintain high quality while collecting usage statistics. In some embodiments, the shader pipeline optimizer 534 selectively samples within the online game session, for example, sampling a subset of a specific shader or image frame to reduce the impact on the game content rendered in the foreground. We then use these timings to select good candidate shaders for further optimization and invoke manual reviews or automated systems to optimize the shaders. Alternatively, in some embodiments, power consumption is monitored and used to optimize the shader pipeline (i.e., a sequence of compiled shaders). Specifically, the shader pipeline optimizer 534 is instrumented to measure power consumption rather than timing performance.

[0190] In some embodiments involving detailed shader analysis, the shader pipeline optimizer 534 selects modified shaders and drivers that collect fine-grained shader execution information, for example, instrumenting each basic block to emit tags into an auxiliary GPU cache. The auxiliary GPU cache is then analyzed locally, or uploaded in its entirety to our analysis database (e.g., included in database 160). The instrumented shaders are created offline from assets in our full range of shader pipeline caches without disrupting the production driver.

[0191] In some embodiments of shader specialization, frequent optimizations are applied in the shader pipeline optimizer 534 to specialize shaders for popular game titles / applications (typically performed manually by trained engineers), replacing machine-compiled code with manually tuned code. A full-range cache is used to perform this replacement without having to redeploy the driver by replacing the compiled shader pipeline in the full-range cache. In some embodiments, multiple versions of compiled shaders are stored in the full-range cache and deployed to perform different statistical analyses. In some embodiments, power-optimized shaders and performance-optimized shaders are both stored in the shader cache, but are dynamically deployed based on performance criteria, for example, to manage peak power consumption for a game. Note that this extends to many performance management axes, such as power, time, memory usage, and code size. Additionally, in some embodiments,

[0192] In some embodiments involving compiler updates and analysis, the shader pipeline optimizer 534 runs a new version of the shader compiler for all shaders in the shader cache and stores the new version of the shader compiler with the earlier version of the shader compiler. The optimizer 534 then runs the new shader compiler using a sample of online game sessions, measures the performance impact, and determines the applicability of the new shader compiler. By these means, compiled shaders and shader compilers can be dynamically improved in the background without disrupting the game content rendering of the game module.

[0193] In some embodiments, the server system 114 includes an application programming interface (API) to query the distributed shader pipeline cache to internal and external users. For a given set of inputs, the distributed shader pipeline cache is queried to enable further inspection and analysis. For example, map-reduce style queries are run on the distributed shader pipeline cache to perform a full range of analysis on individual shaders and shader pipelines. This API includes only the most commonly used instruction blends for depth-only passes and can be implemented as an export pipeline to Dremel, etc. In addition, the API is configured to provide a compiler browser interface by leveraging the pipeline shader compiler as infrastructure.

[0194] Fig.10 1 is an example process 1000 for optimizing pipeline shaders concurrently with game content rendering in a server system 114 according to some implementations. Figure 5A As shown in , in some embodiments, the server-side game module 526 includes a game application 528, a shader pipeline optimizer 534, a game rendering module 538, and a shader cache 318, which work together to enable concurrent game content rendering and shader pipeline optimization. Specifically, when the game application 528 executes an online game session in a graphics pipeline process in the foreground, the shader pipeline optimizer 534 optimizes the compiled shaders used in this graphics pipeline process in the background.

[0195] The CPU 138 of the server system 114 executes a game application 528 corresponding to a game title to run multiple online game sessions in parallel. The CPU 138 allocates each of the subsets of the GPU 140 to execute a subset of multiple online game sessions (e.g., one or more sessions). The GPU 140 then identifies the first sequence of compiled shaders and uses the identified first sequence of compiled shaders to render multiple image frames of each online game session. In some embodiments, the first sequence of compiled shaders has been stored in the shader cache 318, and once identified, it is loaded directly from the shader cache to the GPU 140 to render image frames of the online game session according to the instructions of the game application 528. In some embodiments, the execution of multiple online game sessions must meet specific performance criteria, and the first sequence of compiled shaders is identified according to the specific performance criteria before it is used to render image frames of each online game session.

[0196] While the GPU is executing multiple online game sessions, the shader pipeline optimizer 534 optimizes the first sequence of compiled shaders in the background. In other words, CPU 138 allocates an unused subset of CPUs and GPUs in server system 114 to identify an alternative sequence of compiled shaders or another version of the first sequence of compiled shaders with respect to one or more performance criteria. Optionally, game server 118 also includes an unused subset of CPUs and GPUs allocated for shader pipeline optimization and an active CPU and GPU executing online game sessions. Optionally, a pipeline optimizer server 150 different from game server 118 includes a subset of CPUs and GPUs allocated for shader pipeline optimization.

[0197] Specifically, while the gaming application 528 executes multiple online gaming sessions, the shader pipeline optimizer 534 identifies performance criteria for multiple online gaming sessions associated with the gaming title and collects usage statistics characterizing the execution of each online gaming session using the first sequence of compiled shaders. The shader pipeline optimizer 534 analyzes the usage statistics and determines that the performance of the first sequence of compiled shaders can be further improved relative to the performance criteria. For example, the shader pipeline optimizer 534 determines that a modification of a first shader in the first sequence of compiled shaders relative to the performance criteria improves the performance of the first sequence of compiled shaders, and therefore, modifies the first shader to create a modified first sequence of compiled shaders. In some embodiments, the shader pipeline optimizer 534 further modifies one or more second shaders in the first sequence of compiled shaders to improve the performance of the first sequence of compiled shaders relative to the performance criteria.

[0198] In some embodiments, the shader pipeline optimizer 534 includes an automatic optimizer 1002, which is configured to optimize the first sequence of compiled shaders relative to performance criteria. That is, the automatic optimizer 1002 obtains the first sequence of compiled shaders from the shader cache, and automatically and without user intervention modifies the first shader in the first sequence of compiled shaders to improve the performance of the first sequence of compiled shaders relative to the performance criteria. Alternatively, in some embodiments, the shader pipeline optimizer 534 includes a manual optimizer 1004, which is configured to provide a user interface that allows a user to manually optimize the first sequence of compiled shaders relative to performance criteria. An administrator user logs in to an administrator account on a game development platform executed on an administrator client device 102. The user interface presents performance criteria, usage statistics, and other related information items to facilitate manual optimization. The server system 114 receives a user instruction to modify the first shader in the first sequence of compiled shaders, and therefore, in response to the user instruction, modifies the first shader in the first sequence of compiled shaders. Furthermore, in some embodiments, the pipeline shader optimizer 534 includes both the automatic optimizer 1002 and the manual optimizer 1004 and supports automatic optimization, manual optimization, or a combination thereof.

[0199] In addition, in some embodiments, a substitute shader is selected from a set of substitute shader options to replace the first shader in the first sequence of compiled shaders. Optionally, the automatic optimizer 1002 automatically identifies the substitute shader option and selects the substitute shader to improve the performance of the first sequence of compiled shaders relative to the performance criteria. Optionally, the automatic optimizer 1002 automatically identifies the substitute shader option, but the manual optimizer 1004 prompts the administrator user to select the substitute shader. Optionally, the manual optimizer 1004 provides relevant information items to facilitate the optimization process that allows the administrator user to identify the substitute shader option and select the substitute shader.

[0200] refer to Fig.10 In some embodiments, the modified first-sequence compiled shader replaces the first-sequence compiled shader in the shader cache after it is created. Alternatively, in some embodiments, the first-sequence compiled shader and the modified first-sequence compiled shader are stored as two versions of the first-sequence compiled shader. Subsequently, the server system 114 collects subsequent usage statistics characterizing the execution of each online game session, and based on the subsequent usage statistics, the GPU can select one of the two versions of the first-sequence compiled shader to render multiple image frames for each of the multiple online game sessions.

[0201] Alternatively, in some embodiments, the performance criteria include a first performance criteria, and the plurality of online game sessions are rendered and executed according to a second performance criteria that is different from the first criteria. The pipeline optimizer 534 enables the shader cache 318 to associate the first performance criteria with the modified first sequence of compiled shaders, and to associate the second performance criteria with the first sequence of compiled shaders.

[0202] In some embodiments, the shader pipeline optimizer 534 obtains an optimized shader pipeline for two or more performance criteria in the background. The performance criteria include a first performance criterion, and the modified first sequence of compiled shaders is stored in the shader cache in association with the first performance criterion. After identifying the modified first sequence of compiled shaders, the shader pipeline optimizer 534 identifies a second performance criterion for multiple online game sessions associated with the game title, and determines the second sequence of compiled shaders to improve the performance of the first sequence of compiled shaders relative to the second performance criterion. The second sequence of compiled shaders is stored in the shader cache 318 in association with the second performance criterion.

[0203] When the shader cache 318 includes multiple sequences of compiled shaders, it builds and stores a shader hash table 1006 that associates each of the multiple performance criteria with the corresponding sequence of compiled shaders ( Fig. 11B). The performance criterion and the first sequence of compiled shaders are associated in a shader hash table. Optionally, the first sequence of compiled shaders may have multiple versions. The usage statistics include information entered by the user, and the multiple versions of the first sequence are associated with different information entered by the user ( Fig. 11C ). For example, when the most popular user input is the first user input (e.g., jumping), the first version of the first sequence of compiled shaders is used to meet the corresponding performance criteria, and when the most popular user input is the second user input (e.g., shooting), the second version of the first sequence of compiled shaders is used to meet the corresponding performance criteria. In addition, in some embodiments, the shader hash table is dynamically updated based on usage statistics.

[0204] Note that the performance criterion corresponds to one or more of total power consumption, code size, average latency, total cache usage, and total memory usage of a first sequence of compiled shaders during the course of executing multiple online game sessions. For example, the performance criterion requires that the code size be below a threshold size. In another example, the performance criterion requires that power consumption be reduced to a threshold power level in view of usage statistics.

[0205] In some embodiments, the usage statistics include user input, power consumption, one or more game delays, code size, memory usage, and cache usage of a first sequence of compiled shaders corresponding to each of a plurality of online game sessions. In a specific example, the usage statistics include total power consumption of the plurality of online game sessions. The first performance criterion and the second performance criterion correspond to response time and power usage of the plurality of online game sessions, respectively. Based on a determination that the total power consumption of the plurality of online game sessions exceeds a predefined power threshold and needs to be reduced, the game application 528 selects a second sequence of compiled shaders and renders a plurality of image frames of each of the plurality of online game sessions using the second sequence of compiled shaders, thereby improving the total power consumption of the plurality of online game sessions based on the second performance criterion.

[0206] Fig.11A is an example process for optimizing pipeline shaders according to some embodiments. Multiple online game sessions associated with a game title are executed using an OpenGL rendering pipeline. The OpenGL rendering pipeline includes at least a subset of pipeline operations, the operations including vertex processing (e.g., converting vertices to screen space), primitive processing (e.g., organizing vertices into primitives that are clipped and culled), rasterization (e.g., rasterizing primitives into pixel fragments), fragment processing (e.g., shading fragments to calculate the color at each pixel), and pixel operations (e.g., blending fragments into a frame buffer at their pixel locations). In a specific example, the OpenGL rendering pipeline includes nine consecutive pipeline operations or stages for rendering an image frame for each online game session. The nine consecutive pipeline stages include a subset or all of the following:

[0207] Vertex specification 1102, used to prepare vertex array data;

[0208] Vertex shader 1104, used to act on each vertex;

[0209] Subdivision surfaces 1106 for optionally subdividing patches of vertex data into smaller primitives;

[0210] A geometry shader 1108 for optionally converting smaller primitives into output primitives;

[0211] Vertex post-processing 1110 for adjusting the output primitives of the final stage and shipping them to different locations, including but not limited to transform feedback, primitive clipping, perspective segmentation, and viewport transformation;

[0212] A primitive component 1112 for dividing the output primitive into a sequence of individual basic primitives;

[0213] Rasterization 1114 for decomposing each individual primitive into discrete elements (also called fragments) based on the primitive's sample coverage;

[0214] A fragment shader 1116 for processing each fragment generated by rasterization into a set of colors and a single depth value; and

[0215] Pre-sampling operations 1118 for processing a set of colors and a single depth value into gameplay multimedia content that can be processed by the encoder 142, for example, for normalization, speed and / or compression, including but not limited to pixel ownership testing, scissor testing, multi-sample operations, stencil testing, depth testing, occluded query updates, blending, sRGB conversion, dithering, logical operations, and write mask and logical operations.

[0216] Each GPU 140 assigned to execute a subset of online game sessions executes nine sequential pipeline stages for each of the subset of online game sessions. In some embodiments, each of the vertex shader 1104, the subdivision surface 1106, the geometry shader 1108, and the fragment shader 1116 includes a respective set of programmable shaders. For any shader in the respective set of programmable shaders, the shader pipeline optimizer 534 may have an option to modify the respective shader, thereby creating a modified sequential shader pipeline.

[0217] More specifically, in the example, the game application 528 executes multiple game sessions and renders multiple image frames for each of the multiple online game sessions using a first sequence of compiled shaders. The first sequence of compiled shaders includes a vertex shader 1104A, a tessellation shader 1106A, a geometry shader 1108, and a fragment shader 1116A as programmable shaders. When the multiple online game sessions are executed in the foreground, the shader pipeline optimizer 534 identifies multiple online game session performance criteria associated with the game title and collects usage statistics characterizing the execution of each online game session. As a result, the shader pipeline optimizer 534 determines to modify the vertex shader 1104A and the tessellation shader 1106A in the first sequence of compiled shaders to improve the performance of the first sequence of compiled shaders relative to the performance criteria, thereby creating a modified first sequence of compiled shaders.

[0218] In particular, the shader pipeline optimizer 534 replaces the vertex shader 1104A and the tessellation shader 1106A with the vertex shader 1104N and the tessellation shader 1106B, while keeping the geometry shader 1108A and the fragment shader 1116A in the modified first sequence of compiled shaders. The modified first sequence of compiled shaders is then stored in the shader cache in association with the performance criteria.

[0219] Fig. 11B is a shader hash table 1006 that associates multiple sequences of compiled shaders with multiple performance criteria according to some embodiments. Fig. 11C It is a shader hash table 1180 that associates multiple versions of a compiled shader sequence with multiple user inputs according to some embodiments. When the shader cache 318 includes multiple compiled shader sequences, it establishes and stores a shader hash table 1006, which associates each of multiple performance criteria with the corresponding sequence of compiled shaders. For example, the performance criterion (performance criterion 1) and the compiled shaders of the first sequence are associated in the shader hash table. Optionally, the compiled shaders of the first sequence can have multiple versions corresponding to different usage statistics. In some embodiments, the usage statistics include information input by the user, and the multiple versions of the first sequence are associated with different information input by the user. For example, when the most popular user input is the first user input (e.g., jumping), the first version of the compiled shader of the first sequence is used to meet the corresponding performance criterion, and when the most popular user input is the second user input (e.g., shooting), the second version of the compiled shader of the first sequence is used to meet the corresponding performance criterion. In addition, in some embodiments, the shader hash table 1050 is dynamically updated according to the usage statistics.

[0220] Fig.121 is a flow chart illustrating an example method 1200 for rendering image frames for multiple online game sessions (e.g., including optimizing pipeline shaders in the background of image rendering) according to some embodiments. The method 1300 is optionally performed by a computer stored in a computer memory or a non-transitory computer readable storage medium (e.g., Figure 5A The method 1200 may be controlled by instructions in the shader pipeline optimizer 534 in the server system 114 and executed by one or more processors of the server system 114. The computer-readable storage medium may include a magnetic or optical disk storage device, a solid-state storage device such as flash memory, or other one or more non-volatile storage devices. The instructions stored on the computer-readable storage medium may include one or more of the following: source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Some operations in the method 1200 may be combined and / or the order of some operations may be changed.

[0221] The method 1200 is implemented at a server system 114. The server system 114 includes one or more CPUs 138, one or more GPUs 140, a main dynamic memory 144 storing programs and data for use by the CPUs and / or GPUs during program execution, and a shader cache 318. The one or more CPUs execute (1202) multiple online game sessions in parallel for a game title. The one or more CPUs, in response to a user request or input, instruct the GPU 140 to render consecutive image frames for each of the multiple online game sessions. Then, the one or more GPUs render (1204) multiple image frames for each of the multiple online game sessions using a first sequence of compiled shaders. In some embodiments, the multiple online game sessions are sampled from all online game sessions processed by the CPU and GPU for the game title.

[0222] While executing the plurality of online game sessions, the server system 114 identifies (1206) a performance criterion for the plurality of online game sessions associated with the game title, and collects (1208) usage statistics characterizing the execution of each of the online game sessions. In some implementations, the performance criterion corresponds to one or more of total power consumption, code size, average latency, total cache usage, and total memory usage of a first sequence of compiled shaders during the course of executing the plurality of online game sessions. In some implementations, the usage statistics include user input, power consumption, one or more game latency, code size, memory usage, and cache usage of the first sequence of compiled shaders corresponding to each of the plurality of online game sessions.

[0223] The server system 114 modifies (1210) a first shader in the first sequence of compiled shaders to improve performance of the first sequence of compiled shaders relative to the performance criterion, thereby creating a modified first sequence of compiled shaders. In some embodiments, in addition to the first shader, a second shader in the first sequence of compiled shaders is modified to improve performance of the first sequence of compiled shaders relative to the performance criterion.

[0224] In some embodiments, the modified first sequence of compiled shaders replaces the first sequence of compiled shaders in the shader cache. Alternatively, in some embodiments, the first sequence of compiled shaders used in the execution of the online game session is stored in the shader cache as a first version of the first sequence of compiled shaders, and the modified first sequence of compiled shaders is stored in the shader cache as a second version of the first sequence of compiled shaders. During subsequent game content rendering, for example, in response to subsequent user input, the server system 114 identifies specific performance criteria for subsequent executions of each online game session, and selects one of the first version and the second version of the first sequence of compiled shaders to render multiple image frames for each of the multiple online game sessions based on the specific performance criteria.

[0225] In some embodiments, to modify a first shader in the first sequence of compiled shaders, the server system 114 selects a substitute shader from a set of substitute shader options to replace the first shader in the first sequence of compiled shaders so that the performance of multiple online game sessions meets the performance criteria. As a result, the modified first sequence of compiled shaders includes the substitute shader. Note that in some embodiments, the set of substitute shader options is provided from the shader library server 124. In some cases, the set of substitute shader options is presented to an administrator user who manages a game title executed on the server system 114, and the server system 114 receives a user selection of the substitute shader from the administrator user via a client device 102 to which the administrator user has logged in.

[0226] In some embodiments, an administrator user logs into an administrator account on a game application executed on an administrator client device 102 and manually optimizes the first sequence of compiled shaders based on the performance criteria and usage statistics. During this optimization process, the server system 114 receives a user instruction to modify a first shader in the first sequence of compiled shaders, and thus, in response to the user instruction, modifies the first shader in the first sequence of compiled shaders. Conversely, in some embodiments, the first shader in the first sequence of compiled shaders is modified automatically and without user intervention to improve the performance of the first sequence of compiled shaders relative to the performance criteria.

[0227] In some embodiments, before the GPU uses the first sequence of compiled shaders to render image frames for the plurality of online game sessions, the server system 114 determines that the plurality of online game sessions need to be rendered according to a performance criterion. In view of the performance criterion, the first sequence of compiled shaders is determined to be used for rendering the plurality of image frames for each of the plurality of online game sessions.

[0228] In some implementations, the performance criteria include a first performance criteria, and the plurality of online game sessions are rendered and executed according to a second performance criteria that is different than the first criteria. The server system 114 associates the first performance criteria with the modified first sequence of compiled shaders, and associates the second performance criteria with the first sequence of compiled shaders.

[0229] In some embodiments, the performance criteria include a first performance criterion, and the modified first sequence of compiled shaders is stored in the shader cache in association with the first performance criterion. In addition, in some embodiments, when executing multiple online game sessions, the server system 114 identifies a second performance criterion for multiple online game sessions associated with the game title, and determines a second sequence of compiled shaders to improve the performance of the first sequence of compiled shaders relative to the second performance criterion. The second sequence of compiled shaders is stored in the shader cache in association with the second performance criterion. In addition, in some embodiments, the usage statistics include a total power consumption of the multiple online game sessions, and the first performance criterion and the second performance criterion respectively optimize the response time and power usage of the multiple online game sessions. Based on determining that the total power consumption of the multiple online game sessions exceeds a predetermined power threshold, the server system 114 selects a second sequence of compiled shaders and uses the second sequence of compiled shaders to render multiple image frames for each of the multiple online game sessions.

[0230] In some embodiments, the server system 114 establishes a shader hash table that associates each of a plurality of performance criteria with a corresponding sequence of compiled shaders. Based on the shader hash, the performance criteria and the compiled shaders of the first sequence are associated in the shader hash table. The shader hash table is stored in a shader cache. In addition, in some embodiments, usage statistics include information entered by a user. Multiple versions of the compiled shaders of the first sequence are associated with the information entered by the user in the shader hash table. In addition, in some embodiments, the server system 114 dynamically updates the shader hash table based on usage statistics, for example, by replacing the compiled shaders of the first sequence in the shader cache with the modified compiled shaders of the first sequence.

[0231] Note that shader optimization is applicable not only to a single server with a CPU and a GPU, but also to a server system including multiple servers (i.e., having multiple CPUs and GPUs). Fig.10 and 11A -11C discusses in more detail optimizing a sequence of compiled shaders used in rendering game content.

[0232] It should be understood that the Fig.12 The specific order of operations in is exemplary only and is not intended to indicate that the described order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize various ways to optimize the series of compiled shaders described herein. In addition, it should be noted that the present invention with respect to methods 800 and 900 (e.g., Figures 8A-8B The details of other processes described in 900) can also be similar to those described above with respect to Fig.12 The method 1200 is described and applied. For the sake of brevity, these details are not repeated here.

[0233] Some embodiments of the present application are directed to a server system, which includes one or more CPUs, multiple GPUs, a main dynamic memory storing programs and data for use by the CPUs and / or GPUs during program execution, a static memory pool stored in a non-volatile memory, and a memory controller configured to manage the static memory pool. Each GPU includes a local cache and is configured to access the static memory pool via the memory controller. The server system executes multiple game sessions of a game title in parallel on one or more CPUs. Each of the multiple game sessions is associated with a static data item and requires a graphics operation executable by the corresponding GPU using the static data item, and the static data item is stored in the static memory pool. For each of the subsets of game sessions executed on the one or more CPUs, the server system assigns the graphics operation to the corresponding GPU, requests a copy of the static data item from the memory controller through the corresponding GPU, and receives from the memory controller a first static data item read from the static memory pool, performs the graphics operation using the copy of the static data item by the corresponding GPU, and locally stores the result of the graphics operation in association with the corresponding game session.

[0234] In summary, some embodiments of the present application are directed to loading a game title on a server system, which includes one or more CPUs, multiple GPUs, a main dynamic memory that stores programs and data for use by the CPU and / or GPU during program execution, a static memory pool stored in a non-volatile memory, and a memory controller configured to manage the static memory pool. The CPU loads an executable game program corresponding to the game title into the main dynamic memory. The CPU is configured to execute multiple online game sessions in parallel. Each game session corresponds to an executable game program and requires graphics operations that can be executed by a corresponding GPU assigned to render the corresponding online game session. The CPU identifies static data items used in graphics operations of multiple game sessions. The memory controller stores the static data items in the static memory pool and enables the GPU executing the online game session to access the static data items while performing graphics operations.

[0235] Some embodiments of the present application are directed to concurrent image rendering and shader optimization for game titles. A server system includes one or more CPUs, multiple GPUs, a main dynamic memory that stores programs and data for use by the CPU and / or GPU during program execution, and a shader cache. The CPU executes multiple online game sessions in parallel for the game title. The GPU renders multiple image frames for each of the multiple online game sessions using a first sequence of compiled shaders. While executing multiple online game sessions, the server system identifies performance criteria for multiple online game sessions associated with the game title, collects usage statistics characterizing the execution of each online game session, and modifies a first shader in the first sequence of shaders to improve the performance of the first sequence of compiled shaders relative to the performance criteria, thereby creating a modified first sequence of compiled shaders.

[0236] Reference has been made in detail to various embodiments, examples of which are illustrated in the accompanying drawings. In the above detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention and the described embodiments. However, the present invention may be practiced without these specific details. In other cases, well-known methods, processes, components, and circuits have not been described in detail in order to avoid unnecessarily obscuring aspects of the embodiments.

[0237] It will be understood that although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first device may be referred to as a second device, and similarly, a second device may be referred to as a first device without changing the meaning of the description, as long as all occurrences of the first device are consistently renamed and all occurrences of the second device are consistently renamed. The first device and the second device are both devices, but they are not the same device.

[0238] The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the claims. As used in the description of the embodiments and the appended claims, the singular forms "one", "an" and "the" are also intended to include plural forms unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more associated listed items. It will also be understood that when used in this specification, the terms "include" and / or "comprising" specify the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0239] As used herein, the term “if” may be interpreted to mean “when,” “upon,” or “in response to determining,” or “upon determining,” or “in response to detecting,” a stated conditional precedent is true, depending on the context. Similarly, the phrase “if it is determined that [a stated conditional precedent is true],” or “if [a stated conditional precedent is true],” or “when [a stated conditional precedent is true],” may be interpreted to mean, “after determining that a stated conditional precedent is true,” or “in response to determining that a stated conditional precedent is true,” or “upon determining that a stated conditional precedent is true,” or “after detecting that a stated conditional precedent is true,” or “in response to detecting that a stated conditional precedent is true,” depending on the context.

[0240] For the purpose of explanation, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise form disclosed. In view of the above teachings, many modifications and variations are possible. The embodiments are selected and described in order to best explain the principles of the invention and its practical application, so that others skilled in the art can best utilize the invention and various embodiments with various modifications to suit the specific purpose intended.

Claims

1. A method comprising: At a server system, wherein the server system includes one or more CPUs, a plurality of GPUs, a main dynamic memory storing programs and data for use by the CPUs and / or the GPUs during program execution, a static memory pool stored in a non-volatile memory, and a memory controller configured to manage the static memory pool, wherein each of the GPUs includes a local cache and is configured to access the static memory pool via the memory controller: executing a plurality of game sessions of a game title in parallel on the one or more CPUs, wherein each of the plurality of game sessions is associated with a static data item and requires a graphics operation executable by a corresponding GPU using the static data item, and the static data item is stored in the static memory pool; For a first game session in the plurality of game sessions executing on the one or more CPUs: Assigning the graphics operation to a first GPU; requesting, by the first GPU, from the memory controller a first copy of the static data item; receiving, from the memory controller, a first copy of the static data item read from the static memory pool; performing, by the first GPU, the graphics operation using the first copy of the static data item; and locally storing a first result of the graphics operation in association with a first gaming session of the plurality of gaming sessions; and For a second game session in the plurality of game sessions executing on the one or more CPUs: assigning the graphics operation to a second GPU; requesting, by the second GPU, a second copy of the static data item from the memory controller; receiving from the memory controller a second copy of the static data item read from the static memory pool; performing, by the second GPU, the graphics operation using the second copy of the static data item; and A second result of the graphics operation is stored locally in association with a second gaming session in the plurality of gaming sessions.

2. The method according to claim 1, wherein: The static memory pool is arranged close to and / or shared by the one or more CPUs and the plurality of GPUs of the server system.

3. The method according to claim 2, wherein: The main dynamic memory is arranged close to the one or more CPUs and the plurality of GPUs of the server system and is shared by the one or more CPUs and the plurality of GPUs of the server system.

4. The method according to claim 2, wherein: The static memory pool has a distance from each of the one or more CPUs and the plurality of GPUs of the server system, and the distance is less than a predetermined distance threshold.

5. The method according to claim 4, wherein: The static memory pool is physically located on a server rack, and the one or more CPUs and the plurality of GPUs of the server system are arranged on the server rack.

6. A method according to any one of the preceding claims, wherein: The main dynamic memory includes a dynamic random access memory; Each of the local cache of the GPU and the local cache of the one or more CPUs comprises static random access memory; and The static memory pool includes one of a solid state drive, a flash memory, and a phase change random access memory.

7. A method according to any one of the preceding claims, wherein: The first GPU has a plurality of GPU cores and a common cache shared by the plurality of GPU cores, and each GPU core includes a corresponding cache memory storing instructions and data used by the corresponding GPU core.

8. A method according to any one of the preceding claims, wherein: The server system includes a shader cache that stores one or more compiled shaders used by the plurality of GPUs to perform the graphics operations.

9. A method according to any one of the preceding claims, wherein: The server system includes a shader cache storing a plurality of compiled shaders, further comprising: identifying performance criteria for a first gaming session of the plurality of gaming sessions; Based on the performance criteria, one of the plurality of compiled shaders is selected, wherein the first GPU performs the graphics operation using the selected one of the plurality of compiled shaders.

10. A server system, comprising: One or more CPUs; Multiple GPUs; as well as A memory storing one or more programs for execution by the CPU and / or the GPU, wherein the memory includes at least a static memory pool and a main dynamic memory managed by a memory controller, and the one or more programs include instructions for implementing any method of claims 1 to 9.

Images