Method for executing a game by a computing system

Abstract

A method of executing a game by a computing system that generates a plurality of video frames using a central processing unit and a graphics processing unit. The method includes: writing asset awareness data (AAD) to a command buffer at a binding time, the AAD being associated with an asset to be used by a drawing call to draw a video frame; loading one or more level of detail (LOD) data of an asset to be used by the drawing call from the asset store to the system memory at a binding time, where the AAD uses the lowest LOD data of the asset to enable execution of the drawing call; the GPU executes the condition command in the command buffer to send a query to the asset store controller for the amount of one or more LOD data that has been loaded into the system memory; and receiving, at the GPU from the asset store controller, an amount of one or more LOD data that has been loaded into the system memory, where the rendering call is processed based on the amount of one or more LOD data that has been loaded into the system memory.

Description

[0001] This patent application is a divisional application of the following invention patent applications:

[0002] Application number: 202010606547.1

[0003] Application date: June 29, 2020

[0004] Invention Title: Asset-aware computing architecture for graphics processing Technical Field

[0005] The present disclosure relates to computing system architectures, and more particularly, to a computing system architecture that includes an asset-aware component including a central processing unit and a graphics processing unit when generating video frames for a game or other interactive application. Background Art

[0006] Video games provide a game play process (gameplay) in a virtualized game world. Different scenes of the game world can be created by executing the corresponding game. Various types of assets (such as textures and models) can be used to generate scenes. These assets are used by the graphics processor to generate scenes. These assets are too large to be placed in the system memory at the same time, so the assets are usually stored on optical media, hard drives, or solid-state devices and loaded into the system memory when needed. Therefore, users may experience long delays when loading assets, and managing the loading of assets to the system memory and releasing the system memory to receive additional assets may form a huge burden on game developers. Therefore, game developers may reduce the complexity of the scene to reduce user delays and required asset management, but the result is that the player's immersive experience is less.

[0007] In this context, embodiments of the present disclosure emerge. Summary of the invention

[0008] Embodiments of the present disclosure relate to a computing system architecture and implementation methods of the computing system architecture, the computing system architecture including a central processing unit and a graphics processing unit configured for asset awareness when processing textures, models or other assets via a graphics pipeline. Asset awareness provides identification of assets required for processing and is used to track the loading of those assets for use in the graphics pipeline. Moreover, asset awareness can know when the minimum assets (e.g., lower LODs) have been loaded by the graphics pipeline for processing, thereby reducing the impact of any stalls in the graphics pipeline, especially when switching between scenes of a virtualized game world.

[0009] Embodiments of the present disclosure disclose an asset-aware computing architecture that is configured to manage the loading of assets into system memory when assets are needed by a graphics pipeline and to release system memory when assets are no longer needed. The computing architecture includes an asset storage area that is configured to have a memory device that provides fast access, thereby allowing assets for scenes drawn in video frames to be loaded into system memory within milliseconds or tens of milliseconds. In particular, assets are loaded from the asset storage area into system memory using a bind time trigger. In this way, when an asset is bound to a corresponding draw call (e.g., at a bind time) and when a central processing unit (CPU) executing a game uses a graphics application programming interface (API) to construct one or more command buffers for a draw call executed by a graphics processing unit (GPU) implementing a graphics pipeline, the loading of assets for one or more draw calls for rendering a video frame is triggered. Because the computing architecture is asset-aware, the GPU is able to execute one or more draw calls for a video frame using a minimum of required assets without waiting until the required assets are fully loaded. The asset store controller is configured to control the management of assets in the asset store and the distribution of assets throughout the asset-aware computing architecture. In one embodiment, the asset store controller can be software running on a CPU with the help of other on-chip units. The asset store controller can alternatively be a chip complex containing a coprocessor (e.g., an additional CPU). Both the CPU and the GPU can be modified to integrate and / or interface with the asset store controller. The asset store includes textures, models, and other data for the game. The asset store is much larger than the system memory. The asset store is aware of its contents, including assets identified and / or tagged using asset identifiers (asset IDs), and details of the content and / or assets (e.g., the level of detail of the corresponding assets). The asset store is updated in an asset-aware manner in various ways, such as when an asset is generated locally, or when an asset is accessed from a mass storage device within a rack, or when an asset is accessed from within a data center, or when an asset is distributed via a network, etc. Conventionally, games executing on the CPU use the GPU API to bind one or more assets (e.g., textures, levels of detail - LODs, models, etc.) to a draw call so that the assets can be used in objects rendered by the draw call. In the asset-aware computing architecture, the GPU API receives additional information from the game executing on the CPU. For example, for each draw call, additional asset aware data (AAD) includes the asset ID of the asset for the draw call, the minimum LOD that must be loaded for the asset before the draw call can be executed in the graphics pipeline, and the loading priority of the asset relative to other asset loads used by the draw call.The GPU API is configured to place at least some AADs into a command buffer (e.g., asset ID, lowest LOD, etc.) for the corresponding draw call and pass at least some (if not all) AADs to the asset store controller. In addition, the GPU API can also be used to notify the asset store controller when rendering each video frame is completed, for example by placing commands into a command buffer. Therefore, in an asset-aware architecture, the asset store controller is receiving an AAD stream (e.g., at least all assets required for a video frame with priority information) and notification of the end of rendering using those assets, etc. After receiving the AAD, the asset store controller begins loading the assets from the asset store into the system memory. The loading process uses a loading priority. For example, the "lowest LOD" can be loaded for all textures and / or models and / or other assets, and then the loading of higher LODs can be performed based on the load priority information of the asset. After the loading is completed, the GPU is notified (e.g., using the asset ID) that the asset has been loaded or is being loaded and the extent of the asset loading (e.g., which LODs have been loaded for the asset). In some embodiments, if rendering (e.g., execution of a corresponding draw call in a graphics pipeline) has begun and the asset required by the draw call has not yet been loaded, there may be an "urgent load" of the asset. In other embodiments, conditional commands in the command buffer can be used to query the loading status of the LOD data and change the command flow, for example, if the lowest LOD data has not yet been loaded, the draw call is skipped completely (i.e., to avoid stalls in the GPU), rendering is performed using proxy assets, or custom rendering is allowed based on the loaded LOD. In still other embodiments, the GPU can use AAD to make these same determinations, for example, if the lowest LOD data has not yet been loaded, the draw call is skipped completely (i.e., to avoid stalls in the GPU). The asset storage area controller is also configured to release unneeded memory, such as when an asset has not been used during the rendering frame and therefore the memory space storing the asset can be released. The GPU can be modified to be asset-aware. For example, when rendering an object, the GPU identifies whether the required LOD of the asset has been loaded. The GPU can issue an "urgent load" request for the required LOD and then pause until the required LOD is loaded. Additionally, the GPU identifies the highest LOD that has been loaded into memory for a particular asset, and renders the corresponding video frame using up to but no more than the highest LOD. As part of the loading and releasing of system memory, page tables, CPU caches, and GPU caches may need to be manipulated, and therefore, the asset store controller, CPU, and GPU may be configured to accelerate these operations. In this way, from the perspective of the game executing on the CPU, the assets required for the corresponding draw call "magically" appear in system memory in a timely manner for use.That is, the game only needs to make draw calls using the GPU API, and the required assets (e.g., minimum LOD) are already ready for use in the graphics pipeline.

[0010] In one embodiment, a method for executing a game by a computing system is disclosed, the computing system using a central processing unit and a graphics processing unit to generate video frames. The method includes generating, by the CPU, a draw call for one of the video frames. The method includes writing one or more commands for the draw call to a command buffer at a bind time. The method includes writing asset-aware data (AAD) for the draw call to the command buffer using a GPU API at the bind time, and starting at the bind time, loading one or more level of detail (LOD) data from an asset store to a system memory used by the computing system. The method includes executing, by the GPU, the draw call for the frame using the LOD data written to the system memory, the GPU using at least the lowest LOD data based on the AAD.

[0011] In another embodiment, a computing system for executing a game to generate video frames is disclosed. The computing system includes: a central processing unit, the central processing unit configured to execute the game. The CPU generates a draw call for one of the video frames, wherein the draw call includes a command. The computing system includes a command buffer, the command buffer is configured to store asset-aware data and a command for the draw call, the AAD and the command are written to the command buffer by the CPU using a GPU API at bind time. The computing system includes an asset store, the asset store is configured to store a plurality of assets of an application, the plurality of assets including one or more levels of detail of the asset used by the draw call. The computing system includes a system memory, the system memory is configured to store one or more level of detail data used by the draw call, the loading of the LOD data from the asset store is triggered by the CPU at the bind time, wherein the LOD data begins to be loaded into the system memory at the same time as the AAD is written to the command buffer. The computing system includes a graphics processing unit configured to execute commands of the draw call for the frame using LOD data written to the system memory, the GPU using at least a lowest LOD data based on the AAD.

[0012] In yet another embodiment, a non-transitory computer-readable medium storing a computer program is disclosed, the computer program being used to execute a game by a computing system that uses a central processing unit and a graphics processing unit to generate video frames. The non-transitory computer-readable medium includes program instructions for generating a draw call by the CPU for one of the video frames. The non-transitory computer-readable medium includes program instructions for writing one or more commands for the draw call to a command buffer at a bind time. The non-transitory computer-readable medium includes program instructions for writing asset-aware data (AAD) for the draw call to the command buffer using a GPU API at the bind time and starting to load one or more level of detail (LOD) data from an asset store to a system memory used by the computing system at the bind time. The non-transitory computer-readable medium includes program instructions for executing the draw call for the frame by the GPU using the LOD data written to the system memory, the GPU using at least the lowest LOD data based on the AAD.

[0013] In yet another embodiment, a method for executing a game by a computing system that uses a central processing unit (CPU) and a graphics processing unit (GPU) for generating a plurality of video frames, the method comprising: processing, by the CPU, a draw call for a video frame of the plurality of video frames; writing one or more commands for the draw call to a command buffer at a bind time; writing asset-aware data (AAD) associated with the draw call to the command buffer at the bind time, the AAD being associated with an asset to be used by the draw call in order to draw the video frame; loading one or more level of detail (LOD) data of the asset to be used by the draw call from an asset store to a system memory at the bind time, wherein the AAD uses the lowest LOD data of the asset to enable execution of the draw call; executing, by the GPU, a conditional command in the command buffer to send a query to an asset store controller for a quantity of the one or more LOD data that has been loaded into the system memory; and receiving, at the GPU, from the asset store controller the quantity of the one or more LOD data that has been loaded into the system memory, wherein the draw call is processed based on the quantity of the one or more LOD data that has been loaded into the system memory.

[0014] In yet another embodiment, a method includes: executing a video game on a central processing unit (CPU) to generate a draw call for rendering an image frame; binding a plurality of assets to the draw call at a bind time, wherein the plurality of assets include asset-aware data; passing the asset-aware data to an asset store controller, wherein the asset-aware data includes at least one of a corresponding asset identifier (ID), a corresponding lowest level of detail (LOD), and a corresponding load priority for each of the plurality of assets; constructing, by a graphics processing unit (GPU), a command buffer for executing the draw call; when binding the plurality of assets to the draw call at the bind time, starting loading, at the bind time, a plurality of lowest LODs of the plurality of assets from an asset store to a system memory based on a plurality of load priorities of the plurality of assets; and executing the command buffer by the GPU using at least the plurality of lowest LODs of the plurality of assets.

[0015] Other aspects of the disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present disclosure may be best understood by referring to the following description in conjunction with the accompanying drawings, in which:

[0017] FIG. 1A is a diagram of a system for providing a game according to one embodiment of the present disclosure.

[0018] 1B is a diagram of a system for providing a game with a backend cloud gaming network providing supporting services according to one embodiment of the present disclosure.

[0019] Figure 1C is a diagram of a system for providing games via a cloud gaming network according to one embodiment of the present disclosure.

[0020] 2 illustrates a computing system including a central processing unit and a graphics processing unit that perceive assets when generating video frames while the CPU is executing a game, according to one embodiment of the present disclosure.

[0021] 3 illustrates a rendering pipeline of a computing system including a central processing unit and a graphics processing unit that perceive assets when generating video frames while the CPU is executing a game, according to one embodiment of the present disclosure.

[0022] 4 illustrates the use of asset awareness data by a computing system including a central processing unit and a graphics processing unit when generating video frames while the CPU is executing a game, according to one embodiment of the present disclosure.

[0023] 5 is an illustration of level of detail data of corresponding assets used in generating video frames for a game executed by a central processing unit and implementing a load clamp during loading of LOD data according to one embodiment of the present disclosure.

[0024] 6A is a flow chart illustrating a method for executing a game by a computing system that uses a central processing unit and a graphics processing unit to generate video frames, according to one embodiment of the present disclosure.

[0025] 6B is a flow chart illustrating consideration of the loading status of an asset's LOD data when executing corresponding draw calls to generate a video frame, according to one embodiment of the present disclosure.

[0026] FIG. 7 illustrates components of an example apparatus that may be used to perform aspects of various embodiments of the present disclosure. DETAILED DESCRIPTION

[0027] Although the following detailed description contains many specific details for illustrative purposes, those skilled in the art will appreciate that many changes and modifications to the following details are within the scope of the present disclosure. Therefore, the aspects of the present disclosure described below are set forth without losing the generality of the claims attached to this specification and without limiting the claims attached to this specification.

[0028] In general, various embodiments of the present disclosure describe a computing system architecture including asset-aware components, such as central processing units and graphics processing units that generate video frames via a graphics pipeline, and implementation methods thereof. Asset awareness by the CPU and GPU provides identification and tracking of assets used in the graphics pipeline so that the minimum assets (e.g., the lowest level of detail - LOD) can be used for processing via the graphics pipeline when generating corresponding video frames. In particular, game code executed on the CPU can anticipate when assets are needed so as to load those assets into the system memory in a timely manner. For example, an asset can be a texture and / or model that is processed via a shader program of the graphics pipeline to generate a video frame displayed via pixels of a display. For example, a forest scene can take as input a tree texture processed via the graphics pipeline. In the case where there may not be enough time to load a large amount of LOD data for textures and / or models into the system memory for processing, in order to avoid stalling the GPU, embodiments of the present disclosure provide asset awareness so that the GPU is configured to track the loading of identified assets and use at least the lowest LOD of the asset to generate the scene. Initially, the scene can be rendered at a low resolution, but in subsequent frames, the scene can be rendered using more complete LOD data. In this way, large and complex scenes of virtualized game worlds can be efficiently generated via the graphics pipeline while reducing the possibility of stalling the GPU to wait for assets to load. This is particularly beneficial when encountering a scene-cut in the game, where the game transitions from one scene to another between two video frames (for example, a scene change in a movie sequence, or starting an interactive game play process after a series of menus). Instead of waiting for the entire LOD data to load to generate a new scene, the minimum LOD data can be loaded for each required asset to be processed via the graphics pipeline to generate the corresponding video frame. For example, by reducing GPU stalls even when rendering complex scenes, the user's immersive experience in playing games can be greatly improved.

[0029] With the above general understanding of various embodiments in mind, example details of the embodiments will now be described with reference to the various figures.

[0030] Throughout the specification, references to "games" or "video games" or "gaming applications" are intended to mean any type of interactive application directed via execution of input commands. For purposes of illustration only, interactive applications include applications for gaming, word processing, video processing, video game processing, etc. Furthermore, the terms introduced above are interchangeable.

[0031] Fig. 1A is a diagram of a system 100A for providing a game according to an embodiment of the present disclosure. As shown, the game is being executed locally on a client device 110 (e.g., a game console) of a corresponding user playing the game. In particular, the instance of the game is executed by a game title processing engine 111. The client device 110 can operate in a single player mode for the user. The game logic 115 (e.g., executable code) that implements the game is stored on the client device 110 and is used to execute the game. For illustrative purposes, the game logic 115 can be distributed to the client device 110 via a portable medium (e.g., an optical medium) or via a network (e.g., downloaded from a game provider via the Internet).

[0032] In one embodiment, the game title processing engine 111 of the client device 110 includes basic processor-based functions for executing games and services associated with game applications. For example, the processor-based functions include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadow processing, culling, transformation, artificial intelligence, etc. In addition, services for game applications include memory management, multi-threading management, quality of service (QoS), bandwidth testing, social networking, social friend management, communication with friends social network, communication channels, text processing, instant messaging, chat support, etc. In particular, the client device 110 is configured as a computing system architecture including asset-aware components, such as a central processing unit and a graphics processing unit that generate video frames via a graphics pipeline. In this way, the computing system architecture is configured to identify, track and use at least the lowest LOD data of assets used in the graphics pipeline (e.g., performing draw calls) when generating corresponding video frames.

[0033] The client device 110 may receive input from various types of input devices, such as a game controller, a tablet computer, a keyboard, a gesture captured by a camera, a mouse, a touchpad, etc. The client device 110 may be any type of computing device having at least a memory and a processor module, and is configured to generate a rendered image executed by the game title processing engine 111, and to display the rendered image on a display (e.g., display 11, or display 11 including a head mounted display-HMD, etc.). For example, the rendered image may be associated with an instance of a game executed locally on the client device 110 to enable a corresponding user's game play process, for example, via input commands for driving the game play process. Some examples of the client device 110 include a personal computer (PC), a game console, a home theater device, a general purpose computer, a mobile computing device, a tablet computer, a phone, or any other type of computing device that can execute an instance of a game.

[0034] FIG. 1B is a diagram of a system 100B for providing a game with a back-end cloud gaming network providing support services according to an embodiment of the present disclosure. As shown, the game is being executed locally on a client device 110 (e.g., a game console) of a corresponding user playing the game, wherein the client device 110 is previously introduced in FIG. 1A. In particular, as previously described, an instance of the game is executed by a game title processing engine 111. That is, the game title processing engine 111 of the client device 110 includes basic processor-based functions for executing the game and services associated with the game application. The game logic 115 (e.g., executable code) that implements the game is stored on the client device 110 and is used to execute the game. As previously described, the client device 110 includes an asset-aware component, such as a central processing unit and a graphics processing unit that generate video frames via a graphics pipeline. In this way, a computing system architecture including a CPU and a GPU is configured to identify, track, and use at least the lowest LOD data of an asset used in a graphics pipeline (e.g., performing a draw call) when generating a corresponding video frame.

[0035] The game server 160 can be configured to support one or more local computing devices supporting multiple users, wherein each local computing device can, for example, execute an instance of a corresponding game in a single player or multi-player mode. For example, in multi-player mode, although the game can be executed locally, the cloud gaming network 190 simultaneously receives information (such as game state data) from each local computing device via the network 150, and distributes the information in one or more of the local computing devices accordingly, so that each user can interact with other users in the virtualized game world of the multi-player game (for example, via the corresponding character in the game). In this way, the cloud gaming network 190 coordinates and combines the game play process of each user in the multi-player game environment by transmitting information via the network 150. The network 150 may include one or more communication technologies. In some embodiments, the network 150 may include a 5th generation (5G) network technology with an advanced wireless communication system.

[0036] Furthermore, in single-player or multi-player mode, the cloud gaming network may provide additional services to the client device 110. For example, the cloud gaming network 190 may store the game state of one or more points in the user's game play process for the purpose of backing up the game play process, thereby providing highlights of the game play process, generating mini-games based on the recorded game play process, providing artificial intelligence (AI)-based information related to the game play process (e.g., support for games), etc.

[0037] Figure 1C 1 is a diagram of a system 100C for providing a game via a cloud gaming network 190 according to one embodiment of the present disclosure, wherein the game is being executed away from a client device 110' (e.g., a thin client) of a corresponding user playing the game. System 100C can provide game control to one or more users, and the one or more users play one or more games via a cloud gaming network 190 via a network 150 in a single player or multi-player mode. In some embodiments, the cloud gaming network can be a cloud gaming network 190 including multiple virtual machines (virtual machines; VMs) running on a hypervisor of a host, wherein one or more virtual machines are configured to utilize the hardware resources available to the hypervisor of the host to execute a game processor module. Network 150 may include one or more communication technologies. In some embodiments, network 150 may include a 5th generation (5G) network technology with an advanced wireless communication system.

[0038] As shown, the cloud gaming network 190 includes a game server 160 that provides access to multiple games. The game server 160 can be any type of server computing device available in the cloud, and can be configured to be one or more virtual machines executed on one or more hosts. For example, the game server 160 can manage the virtual machine of the game processor that supports instantiating the instance of the game for the user. Therefore, the multiple game processors of the game server 160 associated with multiple virtual machines are configured to execute multiple instances of one or more games associated with the game playing process of multiple users. In this way, the back-end server supports the streaming of the media (e.g., video, audio, etc.) of the game playing process of multiple game applications provided to multiple corresponding users. That is, the game server 160 is configured to stream data (e.g., the rendered image and / or frame of the corresponding game playing process) back to the corresponding client device 110' via the network 150. In this way, the computationally complex game application can be executed at the back-end server in response to the controller input received and forwarded by the client device 110'. Each server is capable of rendering images and / or frames, which are then encoded (eg, compressed) and streamed to a corresponding client device for display.

[0039] For example, multiple users can use corresponding client devices 110' to access the cloud gaming network 190 via the network 150, where the client device 110' can be configured similarly to the client device 110 of Figures 1A to 1B (e.g., including a game execution engine 111, etc.), or can be configured to provide a thin client that interfaces with a back-end server, which is configured to provide computing functionality (e.g., including a game title processing engine 111').

[0040] For ease of explanation, Figure 1C 1 , a single client device 110' is shown supporting a corresponding user. In particular, the client device 110' of the corresponding user (not shown) is configured to request access to a game via a network 150, such as the Internet, and to render an instance of the game executed by a game server 160 and delivered to a display device associated with the corresponding user. For example, the user may be interacting with an instance of a game executed on a game processor of the game server 160 via the client device 110'. More particularly, the instance of the game is executed by a game title processing engine 111'. The corresponding game logic (e.g., executable code) 115' that implements the game application is stored and can be accessed via a data storage area (not shown), and is used to execute the game. The game title processing engine 111' is capable of supporting multiple games using multiple game logics, each of which can be selected by the user.

[0041] For example, the client device 110' is configured to interact with the game title processing engine 111' in association with the game play process of the corresponding user, for example, via input commands for driving the game play process. In particular, the client device 110' can receive input from various types of input devices, such as game controllers, tablet computers, keyboards, gestures captured by cameras, mice, touchpads, etc. The client device 110' can be any type of computing device having at least a memory and a processor module, and the computing device can be connected to the game server 160 via a network 150. The back-end game title processing engine 111' is configured to generate a rendered image, which will be distributed via the network 150 to be displayed on the corresponding display in association with the client device 110'. For example, via a cloud-based service, the rendered image can be distributed by an instance of the corresponding game executed on the game execution engine 111' of the game server 160. That is, the client device 110' is configured to receive the rendered image, and to display the rendered image on the display 11. In one embodiment, the display 11 includes an HMD (e.g., displaying VR content). In some embodiments, the rendered image may be streamed wirelessly or by wire directly from a cloud-based service or via a client device 110' (e.g., PlayStation® Remote Play) to a smartphone or tablet computer.

[0042] In one embodiment, the game server 160 and / or the game title processing engine 111' includes basic processor-based functions for executing games and services associated with game applications. For example, the processor-based functions include 2D or 3D rendering, physics, physics simulation, script processing, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadow processing, culling, transformation, artificial intelligence, etc. In addition, the services of the game application include memory management, multi-threading management, quality of service (QoS), bandwidth testing, social networking, social friend management, communication with friends social network, communication channels, text processing, instant messaging, chat support, etc. In particular, the game server 160 and / or the game title processing engine 111' is configured as a computing system architecture including asset-aware components, such as a central processing unit and a graphics processing unit that generate video frames via a graphics pipeline. In this way, the computing system architecture is configured to identify, track and use at least the lowest LOD data of assets used in the graphics pipeline (e.g., performing draw calls) when generating corresponding video frames.

[0043] In one embodiment, the cloud game network 190 is a distributed game server system and / or architecture. In particular, the distributed game engine that executes the game logic is configured to the corresponding instance of the corresponding game. Typically, the distributed game engine uses each of the functions of the game engine, and distributes those functions to be executed by multiple processing entities. Individual functions can be further distributed across one or more processing entities. The processing entity can be configured with different configurations, including physical hardware, and / or configured as a virtual component or a virtual machine, and / or configured as a virtual container, wherein the container is different from the virtual machine because the virtual machine virtualizes the instance of the game application running on the virtualized operating system. The processing entity can utilize and / or rely on the server on one or more servers (computing nodes) of the cloud game network 190 and the underlying hardware of the server, wherein the server can be located on one or more racks. The coordination, assignment and management of the execution of those functions to various processing entities are performed by the distribution synchronization layer. In this way, the execution of those functions is controlled by the distribution synchronization layer to enable the generation of media (e.g., video frames, audio, etc.) for the game application in response to the player's controller input. The distributed synchronization layer enables those functions to be efficiently performed (eg, via load balancing) across distributed processing entities, such that key game engine components / functions are distributed and reorganized for more efficient processing.

[0044] In one embodiment, FIG. 1A to FIG. Figure 1C The game title processing engines 111 and 111' include a CPU and GPU group configured to perform multi-tenant GPU functionality, where one CPU and GPU group can implement the graphics and / or rendering pipelines of multiple games. That is, the CPU and GPU group are shared between multiple games being executed. The CPU and GPU group can be configured as one or more processing devices. In another embodiment, multiple GPU devices are combined to perform graphics processing for a single application executing on the corresponding CPU.

[0045] 2 illustrates a computing architecture 200 including asset-aware components, such as a central processing unit 702 and a graphics processing unit 716, that are asset-aware when generating video frames while the CPU is executing a game, according to one embodiment of the present disclosure. In particular, the video frames are generated using a graphics pipeline as implemented by a GPU 716 and controlled by the CPU 702. The asset awareness performed by the CPU 702 and the GPU 716 provides identification and tracking of assets used in the graphics pipeline so that the minimum assets (e.g., the lowest level of detail - LOD) can be used for processing via the graphics pipeline when generating the corresponding video frames. The computing architecture 200 can be used in FIGS. 1A to 1B. Figure 1CThe game server 160 and / or the game title processing engines 111 and 111' are implemented in the client device 110 and / or the cloud gaming network 190 (for example, in the game server 160 and / or the game title processing engines 111 and 111').

[0046] The computing architecture 200 includes a CPU 702, which is configured to execute a game. In particular, the CPU 702 generates a draw call for a frame in the video frame, and the draw call includes an instruction. In addition, during execution in the graphics pipeline, the instructions of the draw call may require assets. For a particular video frame, there may be multiple draw calls generated by the CPU 702 and executed by the GPU 716. As processing power increases, games are designed to render more complex scenes during the game play process. As a result, the game pushes more and more draw calls for each video frame to generate scenes (e.g., forest scenes, ocean scenes, etc.) used during the game play process of the game. The game code executed on the CPU 702 predicts when assets are needed (e.g., through one or more draw calls for a video frame), and coordinates the loading of assets into the system memory 220 before actual use, so that the assets are available to the GPU 716 for use in rendering scenes via the graphics pipeline at the appropriate time.

[0047] The GPU API 250 may be used to communicate between components or between applications running on components of the computing architecture 200. For example, the GPU API 250 may be run on or called by the CPU 702 to communicate with another component, such as a graphics processing unit, system memory, command buffers, asset store, etc. The GPU API 250 may be running on the asset store controller 230 or a processor within the GPU 716 to communicate with other components; in some embodiments, the GPU API 250 may be partially implemented as fixed-function hardware (rather than a general-purpose CPU) for the purpose of such communication.

[0048] To be asset-aware, the game executing on the CPU 702 generates additional information about the assets used for each draw call. For example, this asset-aware data (AAD) may include an identifier of the corresponding asset (e.g., an asset ID), a minimum LOD that must be loaded into system memory before the corresponding draw call can be executed in the graphics pipeline, and a priority for loading of the asset compared to loading of other assets used in other draw calls or within this draw call. At least some of the AADs for a draw call (e.g., the asset ID) are distributed from the game executing on the CPU 702 using the GPU API 250 to the corresponding command buffer that holds the commands for the draw call. In addition, at least some, if not all, of the AADs for a draw call may be distributed from the game executing on the CPU 702 to the asset store controller 230 using the GPU API 250 to coordinate the loading of assets used in the graphics pipeline for the draw call. In some embodiments, all AADs are distributed to the asset store controller 230 in their entirety.

[0049] The computing architecture 200 includes a GPU 716, which is configured to implement a graphics pipeline. For example, the graphics pipeline can be implemented by the GPU 716 to execute a shader program on the vertices of objects in the scene to generate texture values ​​for the pixels of the display, wherein for efficiency, operations are performed in parallel via the GPU 716. Typically, the graphics pipeline receives input geometry (e.g., vertices of objects in the game world). The vertex shader constructs polygons or primitives that constitute the objects in the scene. The vertex shader or other shader program can perform lighting, shading, shading, and other operations for the polygons. The vertex shader or other shader program can perform depth or z buffering to determine which objects are visible in the scene rendered from the corresponding viewpoint. Rasterization is performed to project objects in the three-dimensional world onto a two-dimensional plane defined by the viewpoint. Pixel-sized fragments are generated for the object, wherein one or more fragments may contribute to the color of the corresponding pixel when displaying the image. The fragments are merged and / or mixed to determine the combined color of each of the pixels, and the fragments are stored in the frame buffer for display.

[0050] In addition, the game may use different types of assets for graphics processing, as controlled by the CPU 702 executing the game. These assets may be used by the GPU 716 in the graphics pipeline to generate scenes in the game, for example, a forest scene may use tree textures and tree models processed by the graphics pipeline as input. As the game develops, the virtualized game world created by executing the corresponding game will become more complex, and may require the operation of the shader program to execute the increasing number of vertices and / or textures of the objects or models in the scene.

[0051] More specifically, the GPU 716 of the computing architecture 200 is configured to execute draw call commands to generate video frames using LOD data. More specifically, the GPU 716 uses at least the lowest LOD data based on the asset-aware data known to the components of the asset-aware computing architecture 200. As previously described, a video frame may require the execution of one or more draw calls, each of which may require one or more assets (e.g., textures, models, etc.). If these assets are to be loaded into the system memory 220, but these assets are too large to fit into the system memory 220 at the same time, and therefore the assets are stored outside the system memory, such as on a hard drive or solid state device (SSD), then the assets can only be used by the GPU 716. During the execution of the game by the CPU 702, the game code executed on the CPU 702 predicts when the assets are needed (e.g., through one or more draw calls for a video frame), and coordinates the loading of the assets into the system memory 220 using the GPU API before they are actually used, so that the assets can be used by the GPU 716 for rendering the scene via the graphics pipeline at the appropriate time. As scenes become more complex, requiring larger and larger texture data, the time to fully load the required assets can increase. Therefore, the graphics pipeline may attempt to use an asset before it is fully loaded.

[0052] In an embodiment of the present disclosure, the asset-aware computing architecture 200 allows assets to become available for processing in the graphics pipeline without stalling the GPU 716 waiting for the assets to be fully loaded into the system memory 220. Stalls are particularly likely to occur in traditional computing architectures when too many inputs (e.g., textures) are provided to render a scene. For example, texture or model asset data for a video frame may not be fully loaded before the corresponding draw call is executed by the graphics pipeline. This is especially true when a scene cut is encountered in a game, where the game transitions from one scene to another between two video frames (e.g., a scene change in a movie sequence, or an interactive game starts after a series of menus). In a traditional computing architecture, the graphics pipeline will stall to wait for the assets to be fully loaded before continuing to execute. On the other hand, the asset-aware computing architecture 200 of the present disclosure embodiment is configured to start executing the graphics pipeline even if the assets (e.g., textures and / or models) have not yet been fully loaded. In particular, all the game (as executed by the CPU) must do is call the GPU API 250 (basically as usual) for the draw call. Because the asset-aware computing architecture 200 is able to load available assets in a timely manner, game developers do not need to focus on customizing the game's scenes (e.g., making the scenes less complex) in order to execute on a more traditional computing architecture, which may be slow or poorly functional, particularly when rendering complex scenes or switching scenes during scene transitions.

[0053] The asset-aware computing architecture 200 includes an asset store 210 configured to store a plurality of assets for a game. In some embodiments, the asset store 210 may be configured as a hard drive, or use an SSD. As memory technology advances, other memory devices may be used to provide faster access to the asset store 210. In this way, all assets used to render a scene may be loaded from the asset store 210 in a few milliseconds or tens of milliseconds, or even faster.

[0054] In an embodiment, various communication and / or bus interfaces may be used to communicate with the asset storage area 210. For example, Peripheral Component Interconnect Express (PCI Express; PCIe) may be used to access the asset storage area 210. In one embodiment, non-volatile memory express or NVM express (NVMe) defines an open logic device interface specification for accessing non-volatile storage media attached via a PCIe bus, wherein access to the asset storage area 210 is implemented via NVMe to improve the performance of accessing memory devices in the asset storage area 210.

[0055] Assets can be loaded into the asset store 210 when the game is loaded for execution. In particular, the asset store 210 stores LOD data 290 for the corresponding draw call, and stores all LOD data for all draw calls. Each asset includes one or more LODs that can be used by the corresponding draw call. For example, a texture asset can be represented by a mip map or a texture map chain with one or more LODs, where each texture map (LOD) or mip map (LOD) provides a different resolution or representation of the texture. For example, a texture map or mip map can be an image mapped to the surface of an object or polygon. Texture mapping can be used to reduce the number of polygons for objects that need to be rendered. In addition, a model asset can represent an object. An asset can include one or more models (e.g., LODs) of different resolutions to represent the object. In this way, a desired model with an appropriate resolution for the asset can be selected for rendering via the graphics pipeline.

[0056] In addition, the asset store 210 includes asset aware data (AAD) for each asset, all or some of which will also reside in the system memory 290. For example, each asset can be tagged by a corresponding identifier (e.g., an asset ID), which can be stored in the asset store 210 in association with the corresponding asset. Moreover, each asset can include other data that provides details of the asset. For example, the AAD for each asset can include information describing the LOD set of the asset. This representative information can be stored in the asset store in association with the corresponding asset. The AAD data can be updated in an asset-aware manner in various ways, such as when the asset is generated locally by the CPU 702 or GPU 716 or accessed from a mass storage device within the rack or from within the data center or via a network. That is, the asset store 210 can be updated in an asset-aware manner from elsewhere in the rack or data center or via a network or from a game.

[0057] The asset-aware computing architecture 200 includes a system memory 220 configured to store one or more LOD data 290 used by a corresponding draw call. In addition, for each draw call used to generate a corresponding video frame, the system memory 220 is loaded with the LOD data. In particular, in one embodiment, when a game executed on the CPU 702 binds an asset (e.g., texture, LOD data, minimum LOD data, etc.) to a corresponding draw call using the GPU API 250 at bind time, the CPU 702 also requests the asset store controller 230 to load the LOD data 290 from the asset store 210 to the system memory 220. This request to load the LOD data into the system memory 220 can be implemented as an extension of the GPU API 250, or implemented through a separate API.

[0058] The system memory 220 may include a plurality of command buffers 240. Each command buffer is configured to store an AAD 260 and a corresponding draw call command, wherein the AAD and the command are stored at a bind time. When accessing the command buffer to execute a corresponding command, the GPU 716 may be made aware of the AAD 260. In particular, the CPU 702 writes the AAD and the command of the draw call to the corresponding command buffer using the GPU API 250. As shown in FIG. 2 , the command buffer 240 a is used to generate a video frame F0 (e.g., frame 0). The command buffer 240 a includes a draw call command 241 (e.g., commands 241 a to 241 n) and an AAD for the corresponding draw call. As previously described, the asset-aware data for the corresponding asset may include one or more asset IDs 261 for the required assets (e.g., textures, models, etc.), and may also include a pointer 262 to the asset ID 261 in the asset storage area 210 and / or the system memory 220, the lowest LOD 263 of the required asset ID (and the corresponding asset), the loading state 264 of the asset ID, and the loading priority 265 of each of the asset IDs (e.g., the loading priority of the asset relative to other assets required for the corresponding draw call or other draw calls). For example, assets used in objects close to the camera may have a high or highest priority. Subsequent video frames are rendered using similarly configured command buffers. For example, command buffer 240b is used to generate video frame F1 (e.g., frame 1)..., and command buffer 240n is used to generate video frame Fn (e.g., frame n).

[0059] Additionally, commands may be generated by the CPU 702 and included in the command buffer to provide notification when a corresponding frame has been rendered. In one embodiment, notification is provided when a corresponding asset of a video frame has been rendered. Thus, notification is generated by the GPU 716 executing a draw call command in the graphics pipeline and dispatched to the asset store controller 230 using the GPU API 250.

[0060] The asset-aware computing architecture 200 includes an asset store controller 230 that is configured to manage storage and access of assets in the asset store 210, and more particularly to manage the flow of asset-aware information throughout the computing architecture 200. In one embodiment, the asset store controller 230 is configured as an "on-chip complex" or unit that includes one or more coprocessors. In another embodiment, the asset store controller 230 is configured as an application running on the CPU 702 with the help of other on-chip units.

[0061] As previously described, the asset store controller 230 is configured to receive an AAD information stream from the CPU 702 using the GPU API 250 or using a separate API for one or more draw calls used to generate a video frame. For example, the AAD includes identification of all assets (e.g., asset IDs) required to draw and / or render the corresponding video frame, and a loading priority for each of those assets. For example, the loading of an asset is given priority compared to the loading of other assets required for the draw call. When executing a draw call in the graphics pipeline, the asset store controller can additionally prioritize the lowest LOD required for the asset.

[0062] Upon receiving the asset information, the asset store controller 230 begins loading one or more assets (e.g., LOD data) for the draw call from the asset store 210 into the system memory 220 based on the AAD (e.g., the loading priority of the asset). In particular, in one embodiment, the request to load the LOD data of the asset for the draw call occurs substantially at the same time as the command for the draw call and the AAD for the draw call are written to the command buffer. That is, the request to load the LOD data of the asset for the draw call occurs at the bind time. By triggering the loading of the LOD data of the one or more assets for the draw call when the one or more assets are bound, at least the lowest LOD can be loaded so that the draw call can be executed in a timely manner by the GPU 716 in the graphics pipeline. In addition, the bind time triggering of the loading of at least the lowest LOD of the asset required for the draw call reduces the occurrence of GPU stalls, when the GPU 716 waits for the required assets to load. The request to load LOD data does not necessarily occur at bind time; in another embodiment, the AAD is written to the command buffer together with the draw call command, and separately, at another time, the AAD is sent to the asset store controller 230 to request the loading of the asset.

[0063] In one embodiment, the loading will be performed according to priority. For example, based on the priority information, the "lowest LOD" of all textures is loaded first, and then the higher LODs are loaded. Moreover, some assets have a higher priority than other assets in terms of loading, so that higher priority assets (e.g., lowest LOD or higher LOD) are loaded before lower priority assets (e.g., lowest LOD or higher LOD of the asset). In one embodiment, when the loading of one or more assets is completed, the GPU 716 is notified (using the asset ID) that the corresponding asset has been loaded and to what extent the asset has been loaded (e.g., the loading status of the corresponding LOD). In another embodiment, the GPU 716 is configured to access the loading status, for example, when a drawing command is executed by the GPU 716.

[0064] The loading status may be determined by the asset store controller 230, which is configured to request and track the loading of LODs for the respective asset. In this manner, the loading status of the respective asset may be communicated from the asset store controller 230 to the GPU 916, for example, directly using the GPU API 250, via a command buffer, or via extraction of the loading status by the GPU 716. Because the AAD includes information about the lowest LOD and possible LOD sets for a particular asset, as well as the loading status of the asset into the system memory 220, the GPU 716 is configured to begin executing a draw call using at least the lowest LOD for a given asset. More specifically, the GPU 716 may determine the highest LOD that has been loaded for the particular asset for the respective draw call based on the AAD (e.g., the loading status). Therefore, the GPU 716 may execute the draw call using the highest LOD that has been loaded instead of any higher LOD (e.g., an LOD that has not yet been loaded).

[0065] In addition, because asset awareness includes the loading status of the corresponding asset for the draw call, the GPU 716 is configured to determine whether the required LOD (e.g., the lowest LOD) of the corresponding asset has been loaded into the system memory. If the lowest LOD has not been loaded, the GPU may issue an "urgent load" request, such as loading the lowest LOD or proxy asset (if the proxy asset is not already in the system memory). The GPU may stall until the lowest LOD or proxy asset has been loaded. The urgent determination may be made before or while executing the draw call. For example, the asset store controller 230 and / or the GPU 716 may make the determination while a previous draw call is being executed in the graphics pipeline or before any draw call has been executed by the GPU 716 for a video frame that includes the use of the asset. If the lowest LOD has not been loaded, the asset store controller 230 or the GPU 716 may issue an "urgent load" request at that time.

[0066] The asset storage area controller 230 is also configured to reallocate system memory. In particular, as previously described, a notification can be generated when the rendering of a video frame has been completed. In addition, the AAD may include a notification from a command buffer about the end of rendering of a specific asset. The notification can be distributed to the asset storage area controller 230 from the GPU that executes the drawing call command in the command buffer using the GPU API 250, wherein the command buffer may include commands for generating and distributing the notification. In this way, the assets used for the video frame can be targeted so that the memory space storing those assets can be released. In addition, the asset storage area controller is configured to track the usage of assets loaded into the system memory (e.g., tracking the usage of LOD data, including, for example, the number of times a texture is read). Therefore, assets that have not yet been used to render a video frame can be targeted so that the memory space storing those assets can be released. Assets can be fully released (all LODs), or only a subset of LODs can be released. To help make these determinations, in some embodiments, the GPU 716 is configured to track asset usage, such as for textures, the lowest and highest LOD used and the total number of accesses or the number of accesses per LOD, for draw calls, the number of pixels written due to the call and the execution or lack of draw calls (some calls may be predictable), and for models, the LOD used and the vertex count generated. In some embodiments, the usage of the loaded LODs of partially loaded assets can be used to prioritize loading the highest LODs of those assets, for example, if LOD 0 has not been loaded for two assets, and it can be seen that substantial use is made of LOD 1 of one asset but not the other, then LOD 0 of the asset with substantial LOD 1 usage should be prioritized.

[0067] As part of the loading and releasing of system memory, page tables, CPU caches, and GPU caches may need to be manipulated, and therefore, the asset store controller, CPU, and GPU may be configured to accelerate these operations. For example, the GPU 716 may be configured to accelerate the loading and releasing of system memory 220 by the asset store controller 230. The loading and releasing of memory by the asset store controller 230 requires the manipulation of page tables and graphics memory 718, and therefore the GPU 716 may be modified to accelerate these processes. Additionally, the CPU 702 may be configured to accelerate the loading and releasing of system memory 220 by the asset store controller 230. The loading and releasing of memory by the asset store controller 230 requires the manipulation of page tables and CPU caches (not shown), and therefore, the CPU 702 may be modified to accelerate these processes.

[0068] In one embodiment, CPU 702 is configured to query asset store controller 230 regarding which assets are currently in system memory 220 or which assets are used for a particular frame of graphics. CPU 702 is also configured to query asset store controller 230 regarding the contents of the asset store.

[0069] In another embodiment, the CPU 702 is configured to provide a hint to the asset store controller 230 regarding loading of the corresponding asset. The hint is not necessarily related to any particular draw call being generated. For example, the hint can be configured as a request to load LOD data for an asset with a particular priority, where the LOD data can be used for a predicted draw call.

[0070] FIG3 illustrates a rendering pipeline of a computing system including a CPU 702 and a GPU 716 according to one embodiment of the present disclosure, wherein the CPU and the GPU perceive assets when generating video frames while the CPU 702 is executing a game. Frame updates will be performed in a timely manner for a predetermined frame rendering rate. For example, embodiments of the present disclosure may perform frame rendering at 60 frames per second. Other embodiments are suitable for performing frame rendering at a slower or faster rate. As previously described, the rendering pipeline 301 may be implemented in the client device 110 and the cloud gaming network 190, either individually or in combination.

[0071] In particular, the rendering pipeline includes the CPU 702, the GPU 716, and memory accessible to both (e.g., the asset store 210, the system memory 220, the shared memory, the vertex buffer, the index buffer, the depth or Z buffer, the frame buffer for storing the rendered video frames to be distributed to the display, etc.). The rendering pipeline (or graphics pipeline) describes the general process for rendering an image, such as when a 3D (three-dimensional) polygon rendering process is used. For example, the rendering pipeline 301 for the rendered image outputs the corresponding color information of each of the pixels in the display, where the color information can represent texture and shading (e.g., color, shading, etc.).

[0072] CPU 702 can be configured to perform scene generation and / or object animation for video frames of games executed on CPU 702. CPU 702 receives input geometry corresponding to objects in 3D virtual environment. The input geometry can be represented as vertices in 3D virtual environment, and information corresponding to each of the vertices. For example, objects in 3D virtual environment can be represented as polygons (e.g., triangles) defined by vertices, wherein the surface of the corresponding polygon is then processed via rendering pipeline 301 to achieve final effect (e.g., color, texture, etc.). Moreover, texture and model information can be provided as input in graphics pipeline. The operation of CPU 702 is well known and is generally described herein. Generally, CPU 702 realizes object animation frame by frame, and the object animation can be hand-drawn animation, captured from human actors and / or simulated according to the force applied to the object and / or the force applied by the object (e.g., external force such as gravity, and internal force of the object causing motion). For example, CPU 702 performs physics simulation of objects and / or other functions in 3D virtual environment. The CPU 702 then issues draw calls for the polygon vertices, which are executed by the GPU 716 .

[0073] In particular, the animation results generated by CPU 702 can be stored in a vertex buffer accessed by GPU 716, which is configured to perform projection of polygon vertices onto the display or mosaic of the projected polygons for the purpose of rendering polygon vertices. That is, GPU 716 can be configured to further establish polygons and / or primitives that constitute objects in the 3D virtual environment, including performing lighting, shading and shading calculations for polygons, depending on the lighting of the scene. Additional operations can be performed, such as clipping to identify and ignore primitives outside the viewing frustum, and rasterization for projecting objects in the scene onto a display (e.g., projecting objects onto an image plane associated with the user's viewpoint). At a simplified level, rasterization includes viewing each primitive, and determining which pixels are affected by the primitive. The fragmentation of primitives can be used to decompose primitives into pixel-sized fragments, wherein each fragment corresponds to a pixel in the display and / or a reference plane associated with a rendering viewpoint. When rendering a frame on a display, one or more fragments of one or more primitives may contribute to the color of a pixel. For example, for a given pixel, the fragments of all primitives in the 3D virtual environment are combined into a pixel for display. That is, the overall texture and shading information of the corresponding pixel is combined to output the final color value of the pixel. These color values ​​can be stored in a frame buffer and scanned to the corresponding pixel when the corresponding image of the scene is displayed frame by frame.

[0074] As shown in FIG3 , the rendering pipeline is shown to include operations sequentially performed by the CPU 702, the GPU 716, and the raster engine for scanning and outputting the rendered frames to the display device 710. For illustration, the rendering pipeline sequences 391 to 395 are shown for rendering frames F0 to F4. Due to space limitations, other pipeline sequences, such as sequences for frames other than F4, are not shown. In the example shown in FIG3 , each of the components of the rendering pipeline 301 can operate at the same frame generation frequency. For example, the CPU 702 and the GPU 716 can operate at the same frame generation frequency to render frames at a predefined frame rate via the rendering pipeline.

[0075] For example, in a first frame cycle marked as time t-0, the CPU 702 performs physics and other simulations on objects and distributes polygon primitives to the GPU 716 along with one or more draw calls for a video frame (e.g., F0). In a second frame cycle marked as time t-1, the GPU 716 performs primitive assembly and other tasks to generate a rendered frame (F0). In a third frame cycle marked by time t-2, frame F0 is scanned out. Similarly, the pipeline sequence 392 renders frame F1 in the second and third frame cycles, and scans out frame F1 in the fourth frame cycle marked by time t-3. Moreover, the pipeline sequence 393 renders frame F2 in the third and fourth frame cycles, and scans out frame F2 in the fifth frame cycle marked by time t-4. Similarly, the pipeline sequence 394 renders frame F3 in the fourth and fifth frame cycles, and scans out frame F3 in the sixth frame cycle marked by time t-5. Furthermore, pipeline sequence 395 renders frame F4 in the fifth and sixth frame cycles, and scans out frame F4 in the seventh frame cycle marked by time t-6.

[0076] 4 illustrates the use of asset-aware data by a computing system including a CPU and a GPU when generating video frames while the CPU is executing a game according to one embodiment of the present disclosure. In particular, a portion of a rendering and / or graphics pipeline is shown for rendering a video frame and includes operations performed sequentially by the CPU 702 and the GPU 716. As shown, the rendering pipeline references a timeline 490. A draw time 402 is shown; the draw time spans the time period from when the GPU 716 begins executing a draw call to when the GPU 716 completes processing of the draw call. The draw time 402 separates frame period 1 from frame period 2.

[0077] Starting within and / or before frame cycle 1, at operation 410, the CPU 702 processes video frame 1 via executing a game. In particular, one or more draw calls are generated for the video frame, wherein the draw calls are executed by the GPU 716. At bind time 401, the CPU 702 binds assets required for one or more draw calls for rendering a portion of the video frame. More specifically, at bind time 401, the CPU 702 writes the command 241 (white frame) of each draw call to a corresponding command buffer (e.g., 240). As shown, at operation 415, the CPU 702 uses the GPU API 250 to write the command 241 and AAD 260 (black frame) for the draw call to the command buffer 240 starting from the bind time 401. The command 241 and the AAD 260 can be written in an ordered manner (e.g., sequentially) or in an unordered manner (e.g., interleaved) with each other.

[0078] Additionally, assets 417 required for one or more draw calls required to render a video frame are loaded into system memory 220 beginning at bind time 401. Assets 417 include one or more LOD data 290, which include the lowest LOD that can be used when the corresponding asset is used to render a video frame. Loading of LOD data 290 into system memory 220 can begin without waiting for CPU 702 to complete writing of all commands required to generate a video frame to command buffer 240, but if asset store controller 230 is busy performing other load operations, such as higher priority LOD data, there may be a delay before the LOD data for asset 417 is loaded into system memory 220. Reallocation 411 of system memory may need to be performed before loading LOD data 290 for asset 417.

[0079] As previously described, in an asset-aware computing architecture including a CPU 702 and a GPU 716, the loading of LOD data can be tracked. In this way, the GPU 716 and / or the asset controller 230 can be configured to determine whether the lowest LOD has been loaded for the corresponding asset. If loaded, the GPU 716 can continue to use the lowest LOD of the corresponding asset to render the video frame. If not loaded, the GPU 716 and / or the asset controller can issue a request for an urgent load of the lowest LOD data. The determination of the loading status and the request for the urgent load can be performed at draw time 402, during the execution of a previous draw call, at the beginning of the GPU's processing of frame 1 shown in operation 420, or at another time. As previously described, the asset store controller can notify the GPU of the loading status of the asset for the corresponding draw call, or the GPU can be configured to query and / or extract the loading status from the asset store controller. In some embodiments, conditional commands in the command buffer can be used to query the loading status of LOD data for asset 417 and change the command flow, such as skipping the draw call entirely (i.e., to avoid stalls in the GPU) if the lowest LOD data has not yet been loaded, to use proxy assets for rendering, or to allow custom rendering based on the loaded LODs. In still other embodiments, the GPU can make these same determinations using AAD, such as skipping the draw call entirely (i.e., to avoid stalls in the GPU) if the lowest LOD data has not yet been loaded.

[0080] The use of LOD data may also be controlled using LOD clamp 403. In order for the GPU 716 not to attempt to use LOD data that has not yet been loaded into system memory, at draw time 402 or earlier, the GPU caches the highest number of LODs that have been loaded (e.g., LOD clamp) and only uses LOD data that is determined to be in system memory before the draw call begins, i.e., clamps access to the LOD data at LOD clamp 403. For example, the LOD clamp may be cached at draw time 402, during execution of one or more previous draw calls, or at the beginning of an entire frame of rendering.

[0081] Beginning in frame cycle 2 and / or before frame cycle 2, the GPU 716 renders a video frame (e.g., frame 1) at operation 420. As previously described, there may be optional early processing by the GPU 716 at operation 420a to determine whether a minimum LOD has been loaded for a particular asset for a corresponding draw call based on the AAD and tracking of loading of that asset before even issuing the draw call.

[0082] 5 is a diagram of level of detail data of corresponding assets used when generating video frames for a game executed by a central processing unit and implementing a load clamp during loading LOD data according to one embodiment of the present disclosure. The terms LOD data and LOD can be used interchangeably, such as LOD data and LOD corresponding to an asset and / or multiple assets. In particular, LOD data of corresponding assets is shown. As previously described, LOD data may include LOD 0, LOD 1, LOD 2, and LOD 3 of different resolutions, wherein LOD 0 has the highest detail by convention, while LOD 1, 2, and 3 have gradually decreasing details. For example, for texture assets, LOD data may be a mip map chain of textures. For assets, the lowest LOD is defined as LOD 2, wherein the lowest LOD can be used for assets during draw calls executed in the graphics pipeline, i.e., draw calls can continue without higher LOD 1 or LOD 0, but cannot continue without LOD 3 and LOD 2 in system memory.

[0083] The LOD clamp may be implemented to ensure that the GPU 716 does not attempt to use LOD data that has not yet been loaded into system memory. At draw time 402 or earlier, the GPU caches a number representing the highest LOD that has been loaded. The cached value of the LOD clamp is used by the GPU when executing a draw call so that the GPU clamps its access to the LOD data (e.g., textures and / or models) of the corresponding asset to this cached value of the LOD clamp so as not to exceed the cached value and not attempt to access content that is not in memory. That is, the LOD clamp is used to ensure that when executing a draw call, the GPU does not attempt to use an LOD that has not yet been loaded (e.g., at draw time 402). In addition, the LOD clamp is used to ensure that the same set of LOD data is used in the execution of a draw call so that LOD data that arrives after draw time 402 (e.g., during the execution of a draw call) and is loaded into system memory is not used to execute the draw call.

[0084] In Figure 5, an example of a LOD clamp is shown by line 403a. In this example, at draw time 402, LOD 3 and LOD 2 have been fully loaded, but LOD 1 has only been partially loaded, so the LOD clamp is set to LOD value 2 (representing LOD 2). The GPU 716 is configured so that the GPU will not use LOD values ​​higher than the LOD clamp. For example, even if the primitive using the texture is close to the camera and is normally rendered using LOD 1 or LOD 0, if the LOD clamp is set to 2, these higher LODs will not be used.

[0085] Still in Figure 5, another example of a LOD clamp is shown by line 403b. In this example, at draw time 402, LOD1 has been fully loaded, and therefore the LOD clamp is set to LOD value 1, and LOD 1 will be used by the GPU 716 when executing a draw call. In some embodiments, the LOD clamp is set at draw time; in other embodiments, the LOD clamp is set earlier within the GPU's processing of the frame 420, or at another earlier time (e.g., at the beginning of the GPU's processing of the frame 420).

[0086] Using the data from Figure 1A to Figure 1C 2 and / or a detailed description of the various client devices 110 and / or cloud gaming network 190 (e.g., in the game server 160 and / or the game title processing engines 111 and 111'), flowchart 600A of FIG6A illustrates a method for executing a game by an asset-aware computing system according to one embodiment of the present disclosure, wherein the asset-aware computing system uses a central processing unit and a graphics processing unit to generate video frames.

[0087] At 610, the method includes generating, by a CPU, a draw call for one of the video frames. The draw call is generated by the CPU during game execution. As previously described, one or more draw calls may be generated in association with rendering a video frame.

[0088] For example, draw calls are generated during a cut-scene that defines the transition between the previous scene and the new scene. Drawing a new scene may require loading many assets into system memory. However, an asset-aware computing architecture is able to render the new scene using the lowest LOD data, reducing the possibility of GPU stalls.

[0089] At 620, the method includes writing commands for the draw call to a command buffer at bind time. In particular, the game executed on the CPU uses the GPU API to bind one or more assets required for the draw call at bind time when processed through the graphics pipeline, that is, the game uses the GPU API to write commands related to the assets to the command buffer.

[0090] At 630, the method includes writing asset awareness data (AAD) for the draw call to the command buffer, and also starting to load one or more LOD data from the asset store to the system memory. In particular, the AAD can be generated or accessed and / or determined when the game is executed on the CPU for each draw call. For example, the AAD can include an identifier (e.g., an asset ID) of each asset required for the corresponding draw call, a minimum LOD of each asset that can be used to render the draw call, and a priority of each asset to be loaded into the system memory relative to other assets required for the corresponding draw call.

[0091] When the CPU (using the GPU API) writes the draw call command and the AAD to the command buffer, the CPU also requests the asset store controller to load one or more LOD data for the draw call into the system memory used by the computing system. When the request is issued to the asset store controller, the LOD data begins to be loaded, but the actual loading of the one or more LOD data may not necessarily have occurred. For example, the first plurality of LOD data of an asset begins to be loaded into the system memory, where the asset will be used when processing the draw call, and where the first plurality of LOD data includes the lowest LOD. In particular, the asset store controller is configured to receive an information stream that includes the AAD of one or more assets required for the draw call. The AAD includes an identifier of the asset required to draw a video frame, an asset loading priority, and a notification that the rendering of the video frame using those assets has ended. The LOD data is loaded from the asset store to the system memory by the asset store controller. As previously described, the CPU may issue a request to the asset store controller using the GPU API at bind time to load the LOD data, and in one embodiment, the request is sent with the AAD, for example. Furthermore, in another embodiment, when the AAD is received by the asset store controller, the AAD may be used to request LOD data such that receipt of the AAD triggers loading of one or more LOD data from the asset store to the system memory.

[0092] The asset store controller is configured to coordinate the loading of assets into system memory and also inform the GPU of the loading status by some means, such as, in one embodiment, writing the loading status to the AAD in the command buffer. Furthermore, conditional commands can be placed into the command buffer for execution, causing the GPU to query and / or extract the loading status of assets (e.g., LOD data) used in a draw call from the asset store controller.

[0093] In addition, the asset store controller is configured to manage reallocation of system memory. In particular, the CPU and / or the asset store controller are notified by the GPU that a draw call has been executed. For example, a command can be placed in a command buffer of a draw call to generate a notification indicating that the draw call has completed rendering a video frame, and the notification is distributed to the CPU and / or the asset store controller. In this way, memory space storing one or more assets used by the draw call can be freed. In another embodiment, the asset store controller can track the use of assets stored in system memory. When an asset has not been used to render a current video frame in a graphics pipeline, the asset store controller can determine that the asset is no longer needed to process the draw call and can free up the corresponding memory space.

[0094] At 640, the method includes executing, by the GPU, a draw call for a frame using the LOD data written to the system memory. The CPU issues a draw call to the GPU for execution at the next draw time (e.g., in the current frame cycle), wherein the draw call is generated by the CPU in the previous frame cycle. In this way, the video frame can be rendered in time to be scanned out in the next frame cycle. The GPU executes the draw call using at least the lowest LOD data based on the AAD. For example, each asset has a lowest LOD that can be used to process a corresponding draw call requiring the corresponding asset. Because the asset-aware computing architecture (e.g., a CPU and / or an asset storage area controller running in conjunction with the GPU) can track the loading of assets (e.g., determine the loading status of each asset), the GPU can determine which LOD is used for the corresponding asset when executing a draw call requiring the asset. For example, for the assets introduced above, the loading of the first plurality of LOD data into the system memory is tracked by the CPU and / or the asset storage area controller. In this way, the CPU and / or asset store controller doing the tracking can call a GPU API or otherwise inform the GPU of the loading status of that asset (e.g., the extent to which the first plurality of LOD data has been loaded). Moreover, the GPU is aware of the first plurality of LOD data associated with the asset. In one embodiment, a cached value of the maximum or highest LOD data loaded into system memory is used to ensure that higher LODs are only used when loaded into system memory, as described more fully below with respect to FIG. 6B .

[0095] 6B is a flowchart 600B according to one embodiment of the present disclosure, which illustrates consideration of the loading status of LOD data of an asset when executing a corresponding draw call to generate a video frame. As previously described, the LOD data for the draw call has been loaded into the system memory. In particular, at 650, the LOD data is loaded according to the priority based on the AAD. For example, the LOD data of the corresponding asset can be prioritized to load the LOD data in a specific order, such as first loading the lowest LOD data from multiple LOD data of all assets in an order based on the priority in the AAD, and then, again loading the remaining LOD data from the lowest detail to the highest detail in an order based on the priority in the AAD.

[0096] As described below, the GPU and / or command buffer may perform one of different actions depending on the loading state. That is, the GPU and / or command buffer may have predictions that allow the GPU and / or command buffer to perform different actions based on the LOD loading state. In particular, some calls and / or commands within the corresponding draw call may be predicted based on the LOD loading state. For example, the command buffer may have branch instructions or equivalent instructions of branch instructions that, when executed, perform different actions based on the LOD loading state; for example, these branch instructions may be used to check the LOD loading state, and if some LODs are loaded, a set of commands are executed; however, if some LODs are not loaded, another set of commands are executed.

[0097] At decision step 655, a comparison is made between the highest LOD data that has been loaded into system memory and the lowest LOD data required for the corresponding asset based on the AAD. As previously described, this comparison can be made at draw time or earlier. For example, the comparison can be made during the execution of one or more previous draw calls or at the beginning of a rendered full frame, etc.

[0098] The GPU is configured to use at least the lowest LOD of the asset, and can use the highest LOD that has been loaded when processing a draw call to render a corresponding video frame, which is also higher than the lowest LOD. For the use of LODs higher than the lowest LOD, the GPU utilizes a "LOD clamp", which is a cached value of the highest LOD that has been fully loaded into the system memory at the time of executing the draw call (e.g., at draw time) or a previous time. In this case, the GPU is configured to ensure that the lowest LOD data is in place and stored to the system memory. Moreover, the LOD clamp is used to ensure that higher LODs (e.g., higher than the lowest LOD data) are only used when loaded into the system memory, that is, the GPU uses the LOD clamp to ensure that higher LOD data that is not loaded when the LOD clamp is determined is not accessed during the execution of the corresponding draw call. For example, at 656, the LOD clamp is set to a value representing the highest LOD data that has been loaded, that is, the LOD clamp is a captured LOD loading state. In this way, the GPU avoids attempting to access LOD data that has not yet been loaded into the system memory. Thus, at 657 , draw calls for the frame are executed by the GPU using the LOD data clamped to the LOD clamp (eg, using the highest LOD data loaded).

[0099] On the other hand, if at decision step 655, the highest LOD data loaded is lower than the lowest LOD of the corresponding asset, various operations may be performed at 660. If the GPU determines that the lowest LOD has not yet been loaded into system memory, the GPU and / or asset store controller may request that a default (i.e., proxy) asset be loaded into system memory at 661 for use when processing the corresponding draw call, which asset should not already be in system memory. The default asset is used by the GPU when executing the draw call at 662. Alternatively, an urgent load of the lowest LOD data may be performed at 663, wherein the GPU stalls until the lowest LOD data has been successfully loaded. In some embodiments, the urgent load may be triggered earlier, such as during a previous draw call, at the start of frame rendering, etc. The lowest LOD data is used by the GPU when executing the draw call at 664. Alternatively, as previously described, the GPU may determine to skip execution of the draw call at 665 (i.e., to avoid a stall in the GPU).

[0100] FIG. 7 illustrates components of an example device 700 that can be used to perform aspects of various embodiments of the present disclosure. For example, FIG. 7 illustrates an exemplary hardware system suitable for executing a game by an asset-aware computing system according to one embodiment of the present disclosure, the asset-aware computing system uses a central processing unit and a graphics processing unit to generate video frames so that the video frames can be rendered using the lowest LOD data of one or more assets required for a draw call. This block diagram illustrates a device 700 that can be incorporated with or can be a personal computer, a server computer, a game console, a mobile device, or other digital device, each of which is suitable for practicing embodiments of the present disclosure. The device 700 includes a central processing unit (CPU) 702, which is used to run software applications and optionally run an operating system. The CPU 702 can be composed of one or more homogeneous or heterogeneous processing cores.

[0101] According to various embodiments, CPU 702 is one or more general purpose microprocessors having one or more processing cores. Other embodiments may be implemented using one or more CPUs having a microprocessor architecture that is particularly suited for highly parallel and computationally intensive applications such as media and interactive entertainment applications configured for graphics processing during game execution. Asset store controller 230 may exist as a separate unit, or the functions of the controller may be performed by CPU 702.

[0102] The memory 704 stores applications and data for use by the CPU 702, the asset storage area controller 230, and the GPU 716. The storage device 706 provides non-volatile storage and other computer-readable media for applications and data, and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD or other optical storage devices, as well as signal transmission and storage media. The user input device 708 communicates user input from one or more users to the device 700, and examples of user input devices may include keyboards, mice, joysticks, touch pads, touch screens, still or video recorders / cameras, and / or microphones. The network interface allows the device 700 to communicate with other computer systems via an electronic communication network, and may include wired or wireless communications via local area networks and wide area networks such as the Internet. The audio processor 712 is adapted to generate analog or digital audio outputs based on instructions and / or data provided by the CPU 702, the memory 704, and / or the storage device 706. Components of device 700 are connected via one or more data buses 722, and include: CPU 702; graphics subsystem including GPU 716 and graphics memory 718; asset store controller 230; asset store 210; memory 704; system memory 220; data storage device 706; user input device 708; display device 710; and audio processor 712.

[0103] For example, an asset store 210 may also be included to store assets needed by the game when rendering scenes. The asset store may be part of the memory 704 and / or storage 706. The system memory 220 stores assets used in draw calls when rendering scenes during game execution, for example.

[0104] The graphics subsystem 714 is further connected to the data bus 722 and the components of the device 700. The graphics subsystem 714 includes a graphics processing unit (GPU) 716 and a graphics memory 718. The graphics memory 718 includes a display memory (e.g., a frame buffer) for storing pixel data for each pixel of an output image. The graphics memory 718 can be integrated with the GPU 716 in the same device, connected to the GPU 716 as a separate device, and / or implemented in the memory 704. The pixel data can be provided directly from the CPU 702 to the graphics memory 718. Alternatively, the CPU 702 provides the GPU 716 with data and / or instructions defining the desired output image, and the GPU 716 generates pixel data for one or more output images according to the data and / or instructions. The data and / or instructions defining the desired output image can be stored in the memory 704 and / or the graphics memory 718. In one embodiment, the GPU 716 includes 3D rendering functionality for generating pixel data for an output image based on instructions and data defining the geometry, lighting, shading, texturing, motion, and / or camera parameters of a scene. The GPU 716 may also include one or more programmable execution units capable of executing shader programs.

[0105] Graphics subsystem 714 periodically outputs pixel data of an image from graphics memory 718 for display on display device 710 or for projection by projection system 740. Display device 710 may be any device capable of displaying visual information in response to a signal from device 700, including CRT, LCD, plasma, and OLED displays. For example, device 700 may provide an analog or digital signal to display device 710.

[0106] Other embodiments for optimizing graphics subsystem 714 may include multi-tenant GPU operation where GPU instances are shared among multiple applications, and distributed GPUs supporting a single game.Graphics subsystem 714 may be configured as one or more processing devices.

[0107] For example, in one embodiment, graphics subsystem 714 can be configured to perform multi-tenant GPU functionality, where one graphics subsystem can implement graphics and / or rendering pipelines for multiple games. That is, graphics subsystem 714 is shared between multiple games being executed.

[0108] In other embodiments, the graphics subsystem 714 includes multiple GPU devices that are combined to perform graphics processing for a single application being executed on a corresponding CPU. For example, multiple GPUs can perform an alternating form of frame rendering, where in a sequential frame cycle, GPU 1 renders the first frame, and GPU 2 renders the second frame, and so on, until the last GPU is reached, and then the initial GPU renders the next video frame (for example, if there are only two GPUs, GPU 1 renders the third frame). That is, the GPUs take turns in rendering frames. Rendering operations can overlap, where GPU 2 can start rendering the second frame before GPU 1 finishes rendering the first frame. In another implementation, multiple GPU devices can be assigned different shader operations in the rendering and / or graphics pipeline. The main GPU is performing primary rendering and compositing. For example, in a group including three GPUs, master GPU 1 may perform primary rendering (e.g., a first shader operation) and compositing of outputs from slave GPU 2 and slave GPU 3, where slave GPU 2 may perform a second shader (e.g., a fluid effect such as a river) operation, and slave GPU 3 may perform a third shader (e.g., a particle smoke) operation, where master GPU 1 composites the results from each of GPU 1, GPU 2, and GPU 3. In this way, different GPUs may be assigned to perform different shader operations (e.g., waving flags, wind, smoke, fire, etc.) to render a video frame. In yet another embodiment, each of the three GPUs may be assigned to a different object and / or portion of a scene corresponding to a video frame. In the above embodiments and implementations, these operations may be performed in the same frame cycle (simultaneously in parallel), or in different frame cycles (sequentially in parallel).

[0109] Thus, in various embodiments, the present disclosure describes a computing system architecture and implementation methods that include asset-aware components, such as central processing units and graphics processing units that generate video frames via a graphics pipeline. Asset awareness by the CPU and GPU provides identification and tracking of assets used in the graphics pipeline, so that the assets can be processed via the graphics pipeline using the minimum assets (e.g., the lowest level of detail - LOD) when generating corresponding video frames.

[0110] It should be understood that the various embodiments defined herein can be combined or assembled into specific implementations using the various features disclosed herein. Therefore, the examples provided are only some possible examples, and are not limited to various implementations that are possible by combining various elements to define more implementations. In some instances, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0111] Embodiments of the present disclosure may be practiced using a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices linked over wired or wireless networks.

[0112] In view of the above embodiments, it should be understood that the embodiments of the present disclosure can use various computer-implemented operations involving data stored in a computer system. These operations are operations that require physical manipulation of physical quantities. Any operation described herein that forms a part of the embodiments of the present disclosure is a useful machine operation. The embodiments of the present disclosure also relate to devices or equipment for performing these operations. The device can be specially constructed for the desired purpose, or the device can be a general-purpose computer selectively activated or configured by a computer program stored in a computer. In particular, various general-purpose machines can be used together with computer programs written according to the teachings herein, or it may be more convenient to construct more specialized devices to perform the desired operations.

[0113] The present disclosure may also be embodied as computer readable code on a computer readable medium. A computer readable medium is any data storage device capable of storing data that can subsequently be read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), read-only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical and non-optical data storage devices. A computer readable medium may include a computer readable tangible medium distributed on a network coupled computer system such that the computer readable code is stored and executed in a distributed manner.

[0114] Although the method operations are described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be adjusted so that the operations occur at slightly different times, or may be distributed in the system to allow the processing operations to occur at various intervals associated with the processing, so long as the processing of overlapping operations is performed in the desired manner.

[0115] Although the foregoing disclosure has been described in detail for the purpose of clear understanding, it will be apparent that certain changes and modifications may be made within the scope of the appended claims. Therefore, the disclosed embodiments should be considered illustrative rather than restrictive, and the disclosed embodiments are not limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.

Claims

1. A method for executing a game by a computing system, the computing system using a central processing unit (CPU) and a graphics processing unit (GPU) for generating a plurality of video frames, the method comprising: processing, by the CPU, a draw call for a video frame from among the plurality of video frames; writing one or more commands for the draw call to a command buffer at bind time; writing asset awareness data (AAD) associated with the draw call to the command buffer at the bind time, the AAD being associated with an asset to be used by the draw call to draw the video frame; loading, at the bind time, one or more level of detail (LOD) data for an asset to be used by the draw call from an asset store to a system memory, wherein the AAD uses the lowest LOD data for the asset to enable execution of the draw call; executing, by the GPU, a conditional command in the command buffer to send a query to an asset store controller regarding an amount of the one or more LOD data that has been loaded into the system memory; and receiving, at the GPU from the asset store controller, a quantity of the one or more LOD data that has been loaded into the system memory, The draw call is processed based on an amount of the one or more LOD data that has been loaded into the system memory.

2. The method according to claim 1, further comprising: sending an urgent load request from the CPU to the asset storage area controller to load the lowest LOD data of the asset into the system memory; and The lowest LOD data is used by the GPU to execute a draw call for the video frame.

3. The method according to claim 1, further comprising: When the proxy asset is not stored in the system memory, loading the proxy asset into the system memory; and A draw call for the video frame is executed by the GPU using the proxy assets stored in the system memory.

4. The method according to claim 1, further comprising: Execution of draw calls for the video frame is skipped to avoid stalls in the GPU.

5. The method according to claim 1, further comprising: Notification is received of an amount of the one or more LOD data that has been loaded into the system memory.

6. The method according to claim 1, further comprising: sending, from the GPU to the asset store controller, a query for an amount of the one or more LOD data that has been loaded into the system memory; and A quantity of the one or more LOD data that has been loaded into the system memory is received at the GPU from the asset store controller.

7. The method according to claim 1, further comprising: An amount of the one or more LOD data that has been loaded into the system memory at a draw time for the video frame is determined by the GPU.

8. The method according to claim 1, further comprising: determining, by the GPU, an amount of the one or more LOD data that has been loaded into the system memory prior to a draw time for the video frame; and The draw call is executed by the GPU prior to a draw time for the video frame.

9. A method comprising: The video game is executed on a central processing unit (CPU) to generate draw calls for rendering image frames; binding a plurality of assets to the draw call at a bind time, wherein the plurality of assets include asset-aware data; communicating the asset-aware data to an asset store controller, wherein the asset-aware data includes at least one of a corresponding asset identifier (ID), a corresponding lowest level of detail (LOD), and a corresponding loading priority for each of the plurality of assets; constructing, by a graphics processing unit (GPU), a command buffer for executing the draw call; When binding the plurality of assets to the draw call at the bind time, starting loading, at the bind time, a plurality of lowest LODs of the plurality of assets from an asset store to a system memory based on a plurality of loading priorities of the plurality of assets; and The command buffer is executed by the GPU using at least the plurality of lowest LODs of the plurality of assets.

10. The method according to claim 9, further comprising: The asset awareness data is loaded into the command buffer.

11. The method according to claim 9, further comprising: After loading the plurality of lowest LODs of the plurality of assets has been completed based on the plurality of loading priorities, loading a next highest level LOD for each of the plurality of assets, Therein, a first next-highest level LOD for a higher priority first asset is loaded before a second next-highest level LOD for a lower priority second asset.

12. The method according to claim 9, further comprising: Dynamically tracking, by the asset storage area controller, a plurality of loading states of the plurality of assets; and transmitting the plurality of loading states to the GPU, Wherein when the plurality of loading statuses indicate that at least the plurality of lowest LODs have been loaded into the system memory, the GPU starts executing the command buffer.

13. The method according to claim 12, Wherein the GPU begins executing the command buffer using a corresponding highest LOD for each of the plurality of assets loaded into the system memory.

14. The method according to claim 12, in, The GPU begins executing the command buffer before a corresponding complete load has been loaded for each of the plurality of assets.

15. The method according to claim 9, further comprising: Determining that a lowest LOD of an asset has not been loaded into the system memory; and An urgent load request is issued by the GPU to load the lowest LOD of the asset by the asset store controller while executing a previous draw call used to render a previous image frame.

Images