Graphics processing unit (GPU) continuous command transmission

Description

1 / 50 Continuous transmission of commands from a graphics processing unit (GPU) CROSS-REFERENCE TO RELATED REQUESTS

[001] This application claims the benefit of U.S. Provisional Patent Application No. 61 / 797,369, filed March 23, 2023, which is incorporated by reference herein in its entirety. BACKGROUND

[002] Cloud-based gaming has steadily increased in popularity over time. In these scenarios, video game graphics and other associated data are typically rendered on remote cloud-based servers and then streamed to game clients, where the data is decoded and displayed on a client device. When processing this video game data, the graphics processing units (GPUs) on the remote servers implement several different hardware components, including rendering and encoding components. In traditional cloud gaming systems, images are rendered on the remote server, encoded, compressed, and streamed to the client device. This process results in the generation and transfer of a large amount of data between the cloud gaming servers and the client device.Therefore, cloud gaming providers require large amounts of energy, a large number of servers, and higher resource costs associated with generating these images. This can lead cloud gaming providers to a model that is not economically viable for some games. This relatively large amount of data can also result in slow gameplay and less-than-ideal response times. This, in turn, can lead to less interest in playing cloud-based games. SUMMARY

[003] As will be described in more detail below, the Petition 870250085825, dated 09 / 23 / 2025, page 19 / 87 2 / 50 of this disclosure provides methods and systems for continuously transmitting or relaying graphics processing unit (GPU) commands in a highly efficient manner. In some cases, GPU commands can be transmitted instead of fully rendered video frames, which can lead to a large reduction in the amount of data transmitted over the network between a remote server and a client device.

[004] In one embodiment, a computer-implemented method is provided, which includes: accessing media frame generation input events produced as part of a multimedia application on a media server, selecting at least one media frame that should be rendered according to the media frame generation input events, determining the graphics processing capabilities of a client device on which the selected media frame should be rendered, generating a rendering command for the selected media frame based on the determined graphics processing capabilities of the client device,where the rendering command includes contextual graphics information and graphics processing unit (GPU) threading information for use in rendering the selected media frame on the client device, and transmits the generated rendering command to the client device to initiate rendering of the selected media frame using the contextual graphics information and GPU threading information.

[005] In some modes, the selected frame is a video frame. In other modes, the selected frame is an audio frame. In some cases, the multimedia application is a video game. In some examples, the generated rendering command is sent to a video game engine running on the client device.

[006] In some cases, contextual graphical information includes vector data for meshes, vector data Petition 870250085825, dated 09 / 23 / 2025, page 20 / 87 3 / 50 specifying positions (also known as temporary vertex stores) or 2D / 3D texture information or other non-vector 2D / 3D information. In some embodiments, GPU thread information includes vertex shader, fragment shader, or any other shader or shader program information to be used. In some embodiments, GPU thread information includes texture sampler configuration information, viewport size, scissor rectangle, blending mode, or stencil information. In some cases, the method further includes compressing contextual graphics information and GPU thread information before transmission to the client device.In some examples, client devices that have increased processing capabilities receive at least an additional portion of contextual graphics information or GPU threading information (such as increased viewport dimensions) for use in rendering the selected media frame.

[007] In some embodiments, contextual graphic information includes 3D depth information, providing support for client devices with 3D display capabilities. In some cases, contextual graphic information includes different graphic information for each of a user's eyes, as the graphic information describes a complete 3D scene and two associated viewports. In some examples, rendering commands are generated on a distribution server that is at a specified physical distance from the client device. In some embodiments, the media server controls the multimedia application's runtime.

[008] In addition, a corresponding system includes at least one physical processor and physical memory comprising computer executable instructions which, when executed by the physical processor, cause the physical processor to: access Petition 870250085825, dated 09 / 23 / 2025, page 21 / 87 4 / 50 media frame generation input events produced as part of a multimedia application on a media server, select at least one media frame that should be rendered according to the media frame generation input events, determine the graphics processing capabilities of a client device on which the selected media frame should be rendered, generate a render command for the selected media frame based on the determined graphics processing capabilities of the client device, wherein the render command includes contextual graphics information and graphics processing unit (GPU) threading information for use in rendering the selected media frame on the client device, and transmit the generated render command to the client device to initiate rendering of the selected media frame using the contextual graphics information and GPU threading information.

[009] In some modalities, the transmitted rendering commands allow the client device to start the game without an installation phase and substantially without delay. In some cases, the method further includes adapting contextual graphical information based on the graphical processing capabilities of the client device. In some examples, adapting contextual graphical information based on the graphical processing capabilities of the client device includes translating shader information into a format that will be understood by the client device. In some cases, adapting contextual graphical information based on the graphical processing capabilities of the client device includes reducing a set of features in the contextual graphical information to provide a reduction in the scope of processing on the client device.In some cases, the multimedia application includes a video game, and the generated rendering command is sent to a video game engine running on the client device. Petition 870250085825, dated 09 / 23 / 2025, page 22 / 87 5 / 50

[010] In some examples, the method described above is encoded as computer-readable instructions in a computer-readable medium. For example, the computer-readable medium may include executable instructions that, when executed by at least one processor of a computing device, cause the computing device to access the media frame generation input events produced as part of a multimedia application on a media server, select at least one media frame to be rendered according to the media frame generation input events, determine one or more graphics processing capabilities of a client device on which the selected media frame should be rendered, generate a rendering command for the selected media frame based on the determined graphics processing capabilities of the client device,where the rendering command includes contextual graphics information and graphics processing unit (GPU) threading information for use in rendering the selected media frame on the client device, and transmits the generated rendering command to the client device to initiate rendering of the selected media frame using the contextual graphics information and GPU threading information.

[011] The features of any of the embodiments described in this document may be used in combination with each other in accordance with the general principles described in this document. These and other embodiments, features and advantages will be better understood after reading the following detailed description together with the attached drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[012] The attached drawings illustrate a series of exemplary embodiments and form part of the descriptive report. Together with the description that follows, these drawings demonstrate and explain various Petition 870250085825, dated 09 / 23 / 2025, page 23 / 87 6 / 50 principles of this disclosure.

[013] FIG. 1 illustrates a computing environment in which the modalities described in this document are designed to operate.

[014] FIG. 2 is a flow diagram of an exemplary method for continuously transmitting or relaying graphics processing unit (GPU) commands in a highly efficient manner.

[015] FIGS. 3A and 3B illustrate alternative computing environments in which the modalities described in this document are designed to operate.

[016] FIG. 4 illustrates a mode in which frame rendering can be transferred to a client device.

[017] FIG. 5 illustrates a mode in which a media server or a distribution server can adjust or translate rendering command parameters for a specific client device.

[018] FIG. 6 is a block diagram of an exemplary content distribution ecosystem.

[019] FIG. 7 is a block diagram of an exemplary distribution infrastructure in the content distribution ecosystem shown in FIG. 6.

[020] FIG. 8 is a block diagram of an exemplary content player in the content distribution ecosystem shown in FIG. 6.

[021] Throughout the drawings, identical reference characters and descriptions indicate similar, but not necessarily identical, elements. Although the exemplary embodiments described in this document are susceptible to various modifications and alternative forms, specific embodiments have been shown as examples in the drawings and will be described in detail herein. However, the exemplary embodiments described in this document are not intended to be limited to the particular forms disclosed. Instead, the Petition 870250085825, dated 09 / 23 / 2025, p. 24 / 87 7 / 50 This disclosure covers all modifications, equivalents and alternatives that fall within the scope of the appended claims. DETAILED DESCRIPTION OF EXEMPLARY MODALITIES

[022] This disclosure is generally directed to methods and systems for transmitting graphics processing unit (GPU) commands in a highly efficient manner. As noted above, applications, such as video games, are often hosted remotely on cloud computing systems. These cloud computing systems are generally designed to render and encode graphics for game clients (e.g., smartphones or laptops, etc.). As part of generating these game graphics, cloud computing systems often implement multiple simultaneous GPUs to render and encode video frames. This process requires large amounts of computing power and equally large amounts of network bandwidth to transmit the rendered and encoded frames to the client device.Furthermore, these large data transmissions make cloud gaming vulnerable to intermittent or low-speed data connections. These connections can result in slow gameplay and slow response times to user input. This, in turn, can reduce the amount of time users spend playing cloud games. Additionally, in some cases, the concentrated computing power requirements on the server side can make cloud gaming economically unviable for reaching a very high number of simultaneous players.

[023] In contrast to traditional cloud gaming schemes, in which rendering occurs on remote cloud-based servers, the modalities presented here generate GPU rendering commands that are transmitted to the client device. These GPU rendering commands Petition 870250085825, dated 09 / 23 / 2025, page 25 / 87 8 / 50 includes information that specifies how a video frame should be rendered and allows the client device itself to perform the rendering. So, instead of transmitting a fully rendered and encoded video frame over the network, cloud-based servers can generate only the information needed to render a video frame on a client device and transfer that data over the network to the client. This results in a substantial reduction in the amount of computing power required on the server side and also a substantial reduction in the amount of data transmitted over the network to the client device. These modalities offer the benefits of a cloud-hosted game (e.g., little to no client-side installation or updates, centralized hosting that reduces cheating and hacking, etc.) while reducing the amount of computing and data transfer performed by the cloud-based system.

[024] Furthermore, as the systems described in this document are designed to take advantage of the increased processing capabilities of client devices, graphics rendered using GPU rendering commands can have superior quality to compressed and pre-rendered video frames. In at least some cases, the systems described in this document determine which client device is playing, determine what the device's graphics processing capabilities are, and then continuously transmit GPU rendering commands to that device that are specific to that device's capabilities. This optimizes cloud-side and client-side calculations as well as network transmissions, providing a remote gaming platform with less latency and better graphics than traditional systems. These embodiments will be described in more detail below in relation to FIGS. 18.

[025] FIG. 1 illustrates a computing environment 100 that includes Petition 870250085825, dated 09 / 23 / 2025, page 26 / 87 9 / 50 a computer system 101. A computer system 101 includes software modules, embedded hardware components such as processors, or includes a combination of hardware and software. A computer system 101 substantially includes any type of computing system, including a local computing system or a distributed computing system (e.g., cloud). In some cases, a computer system 101 includes at least one processor 102 and at least some system memory 103. A computer system 101 includes program modules to perform a variety of different functions. Program modules are hardware-based, software-based, or include a combination of hardware and software. Each program module uses computing hardware and / or software to perform specific functions, including those described below.

[026] Computer system 101 includes a communications module 104 that is configured to communicate with other computer systems. The communications module 104 includes any wired or wireless communication device that can receive data from other computer systems and / or transmit data to them. These communication devices include hardware interfaces including Ethernet adapters, WIFI adapters, hardware radios including, for example, a hardware-based receiver 105, a hardware-based transmitter 106, or a combined hardware-based transceiver capable of both receiving and transmitting data. The radios are cellular radios, Bluetooth radios, global positioning system (GPS) radios, or other types of radios.The 104 communications module is configured to interact with databases, mobile computing devices (such as mobile phones or tablets), embedded systems, or other types of computing systems.

[027] Computer system 101 also includes an access module 107. Access module 107 is configured to access Petition 870250085825, dated 09 / 23 / 2025, page 27 / 87 10 / 50 media frame generation input events 109 that are produced as part of a multimedia application 108 running on a media server. In some embodiments, the computer system 101 is a media server, while in other cases, the computer system 101 communicates with other media servers. In some embodiments, the multimedia application 108 is a video game (e.g., a cloud-based video game or a remote video game), while in other cases, the multimedia application 108 is a streaming video application, a streaming audio application, an image viewing application, or another media-based software application.

[028] Access module 107 is configured to generate media frame generation input events 109 as part of the multimedia application 108. Media frame generation input events 109 include instances in which a media frame is generated. This process involves generating the information necessary to render a frame on a client device. In some cases, the media frame generation input event 109 indicates the start of a frame generation event, indicating that a frame for a video game (or for a video or audio file) should be generated. After this initiation process, the frame selection module 110 selects a frame 112 that should be rendered based on the media frame generation input event 109.

[029] After the specific frame 112 has been selected by the frame selection module 110, the determination module 111 of the computer system 101 determines the graphics processing capabilities 121 of a client device. It should be noted here that any of these steps can be performed in the order described or in a different order. Thus, for example, in some cases, the determination module 111 determines the graphics processing capabilities of the client device 121 Petition 870250085825, dated 09 / 23 / 2025, page 28 / 87 11 / 50 before the media frame generation input event 109. Each client device (e.g., 120, used by user 119) has its own general processor(s) and graphics processor(s). In addition, each client device has its own memory (e.g., RAM), data storage, wireless radios (or network cards), cellular service or internet service provider, and other hardware, software, or network limitations. Consequently, the determination module 111 of computer system 101 is configured to identify the capabilities of client device 121, including general processing capabilities, graphics processing capabilities, memory capabilities, storage capabilities, network capabilities, and other capabilities of client devices to receive, transfer, and process data. These capabilities can then be noted in the database 122.

[030] Using the determined capabilities of device 121, the rendering command generation module 113 of computer system 101 generates a rendering command 114 for the selected media frame 112. The rendering command generation module 113 generates the rendering command 114 based on the determined graphics processing capabilities of the client device 120. In some cases, this includes increasing or decreasing the quality and / or dimensions of the graphic asset, such as reducing or increasing texture sizes, or compressing graphic assets with more or less visual loss, all based on the client device's ability to handle the increase or decrease in quality and the transmission medium's ability to handle more or less data.The render command 114 includes contextual graphics information 115 and graphics processing unit (GPU) threading information 116 needed to render the selected media frame 112 on the client device 120, based on its determined device capabilities 121. The command transmission module of. Petition 870250085825, dated 09 / 23 / 2025, page 29 / 87 12 / 50 rendering 117 then transmits the rendering command 114 to the client device 120 for rendering, potentially after the rendering command 114 is compressed for transmission by the compression module 118. The content of the contextual graphics information 115 and the GPU thread information 116, along with the other elements and modalities explained above, will be described further in relation to method 200 of FIG. 2.

[031] FIG. 2 is a flow diagram of an exemplary computer-implemented method for continuously transmitting or relaying graphics processing unit (GPU) commands in a highly efficient manner. The steps shown in FIG. 2 can be performed by any suitable computer executable code and / or computing system, including the system illustrated in FIG. 1. In one example, each of the steps shown in FIG. 2 may represent an algorithm whose structure includes and / or is represented by several substeps, examples of which will be provided in more detail below.

[032] As illustrated in FIG. 2, a computer-implemented method 200 can be provided that includes: accessing, in step 210, one or more media frame generation input events 109 produced as part of a multimedia application 108 on a media server. The method 200 further includes, in step 220, selecting at least one media frame 112 that is to be rendered according to one or more media frame generation input events 109 and determining, in step 230, one or more graphics processing capabilities 121 of a client device 120 on which the selected media frame 112 is to be rendered.

[033] Method 200 also includes, in step 240, generating a rendering command 114 for the selected media frame based on the determined graphics processing capabilities 121 of the client device 120, where the rendering command Petition 870250085825, dated 09 / 23 / 2025, page 30 / 87 13 / 50 includes contextual graphics information 115 and GPU threading information 116 for use in rendering the selected media frame 112 on the client device 120. Method 200 further includes, in step 250, the transmission of the generated rendering command to the client device to initiate rendering of the selected media frame using the contextual graphics information and GPU threading information.

[034] The term “continuous command transmission,” as used in this document, generally refers to the concept of repeatedly sending GPU rendering or computation commands to a client device that would traditionally be sent to a server’s GPU. In this way, the media server may not use or require a GPU device and may instead leverage the client device’s GPU (and / or CPU). The modalities described in this document are designed to identify and send the contextual information necessary to perform a drawing (or GPU computation task) to another process on the client device and enable that process (and / or the client device’s GPU / CPU) to perform the identified tasks.

[035] In at least some embodiments, continuous command transmission functions intercept graphics API calls (e.g., Vulkan calls) from an application process (e.g., a video game process or video game application), potentially adapt them when necessary (e.g., serialize, translate, or otherwise adapt the API calls), and transmit the API calls (e.g., using a specified protocol that may be suitable for such data transmissions) to a receiving process that will decode and issue new or corresponding graphics API calls, in turn, to the local GPU on the client device. Notably, the embodiments described in this document Petition 870250085825, dated 09 / 23 / 2025, page 31 / 87 14 / 50 encompasses much more than simply performing remote procedure calls (RPCs). Instead, the systems in this document implement and manage logic and state on both sides of the continuous data transmission from server to client. The identified logic and state transmitted to the client device pertain to GPU memory and resource management, resource caching, resource compression, graphics API adaptations (e.g., depending on server / client implementation mismatches, such as shader translation), and high-level logic associated with serializing relatively complex data structures at high frequency.

[036] GPUs are typically asynchronous processing units. A system sends rendering or computation jobs to queues, and these jobs are performed when possible (i.e., when a GPU graphics or computation engine is available to perform the work). The system will then be notified (via fence or semaphore objects or via explicit or implicit commands or resource dependencies) when the job is complete. Continuous command transmission, as presented in this document, is an extension of this queuing and asynchronicity and an extension of this dependency management. Sending a command from one process to another is done asynchronously and typically does not require a response. Or that response or completion signal may come later (much later if the commands are sent over the network).

[037] The term “cloud gaming” or “cloud-based gaming” can refer to any type of video game delivered over a local network and / or the internet, where a back-end server provides at least some of the data used to play the game. In some cases, cloud-based servers can render and encode the video frames that are sent to the client device. 120 Petition 870250085825, dated 09 / 23 / 2025, page 32 / 87 15 / 50 for user display. In other cases, including the modes described in this document, cloud-based servers may receive inputs and calculate changes to the game while offloading some or all of the video rendering to the client device.

[038] Cloud games or other multimedia applications (e.g., streaming video, streaming audio, interactive content, etc.) can be processed by the computer system 101. When the computer system starts the frame generation process as part of a multimedia application 108, the frame selection module 110 selects certain frames 112 (or all input frames) that should be rendered by the client device 120. These frames can be video or audio frames and can be part of a video game, a movie, a song, or other content.

[039] FIG. 3A describes an embodiment in which an emitting process 301 (e.g., a video game, such as a multimedia application 108) generates graphics API calls. These traditional graphics API calls generated by the emitting process 301 are then sent to the local GPU 302 on the media server and processed by a local controller on the GPU. This is the process that would normally occur in a traditional cloud gaming platform. In contrast to this scenario, the embodiment shown in FIG. 3B illustrates the use of a continuous command transmission 312 and an adapter 311. Instead of the graphics API calls being sent to a local GPU 302 on the server, the emitting process (e.g., a video game) 310 sends the graphics API calls to an adapter 311 on the server.Adapter 311 serializes, translates, or otherwise modifies graphics API calls for transfer, via a continuous command transmission 312, to a rendering process 313 on the client device. The process of... Petition 870250085825, dated 09 / 23 / 2025, page 33 / 87 16 / 50 rendering 313 then sends the modified graphics API calls to the client device's local GPU 314 for rendering on the client device.

[040] A rendering command, or “drawing call,” specifies what tasks a GPU thread should perform and how the GPU thread is configured. At least in some cases, the drawing call is an order from the issuing process (e.g., a video game) 310 to render and rasterize a specified geometry using a given context (similarly, for computation commands, to compute various data using a computation kernel and specified inputs and outputs). In the modalities described in this document, rendering commands will carry an extensive description or context with them.In some examples, the rendering commands indicate which geometry or input vectors to use for drawing, which shaders to use, which constant (uniform) shader input values ​​to use, which resources (e.g., textures, temporary stores) to use or link to shaders, which texture samplers to link textures to, which texture sampler parameters to use, which depth test setting to use, which blending mode, viewport size, or other threading settings to use, etc., along with which target frame temporary store references (render target) to render.

[041] These rendering commands are generally relatively small in size, especially when compared to a fully pre-rendered bitmap or encoded video frame. This is due, at least in part, to the referencing and reuse of pre-existing and previously configured shaders, resources, objects, and / or temporary storage. In fact, once these shaders, resources, objects, temporary storage, or other objects have Petition 870250085825, dated 09 / 23 / 2025, page 34 / 87 17 / 50 have been loaded onto a client device, the continuous transmission instructions of command 312 may be sufficient to cause the generation of high-quality images on a client device, while transmitting substantially less data than would be transmitted in a fully rendered video frame.

[042] At least in some embodiments, continuous client command transmission will implement an appropriate protocol for transmitting this type of data. In some cases, the protocol is a compact transport protocol, designed for low latency with some potential to tolerate losses (e.g., web real-time communication (WebRTC), Continuous Transmission Control Protocol (SCTP), Real-time Transport Protocol (RTP), UDP fast internet connections (QUIC), etc.). In some cases, the transmitted data is compressed before transmission. Furthermore, in continuous client command transmission, at least some data may be adapted before transmission. Adaptations include adaptations or changes to the graphics API that align with the specifications of the client device or client GPU.

[043] Continuous Client Command Streaming (CCS) allows for a reduction in the media server GPU load and the associated costs for clients accessing the server. Client devices with higher GPU capabilities (personal computers, laptops, latest generation smartphones, etc.) receive an enhanced experience without incurring additional costs for the media server provider. In these cases, the user experience on the client device can be scaled according to the CPU and / or GPU resources of the client device. In some embodiments, Continuous Client Command Streaming supports three-dimensional (3D) displays, such as virtual reality displays, TVs Petition 870250085825, dated 09 / 23 / 2025, page 35 / 87 18 / 50 3D or other 3D displays.

[044] Furthermore, continuous client command streaming allows a media server CPU to run the underlying game logic (as opposed to a set of GPUs). In this way, cloud games can be run on distribution servers physically located closer to client devices, reducing connection latency. Additionally, by not taking time to render and encode a video frame before streaming that frame, the media server can experience a reduction in latency (e.g., ~10 ms) by avoiding this processing. Continuous client command streaming also allows cloud-based games to be run on Internet Service Provider (ISP) sites, resulting in an even greater reduction in latency. This, in turn, leads to higher Quality of Experience (QoE) and potentially greater user satisfaction with the media provisioning service.

[045] The modalities in this document provide a lossless way to efficiently encode an entire 2D or 3D scene, which bitmaps do not allow. For example, the systems described in this document can render a large amount of high-fidelity text using only pointers to distance fields and / or texture atlases. These hints take only a small fraction of what this text would take to be transmitted via video encoding. Continuous client command transmission also provides enhanced control over video quality. The system can selectively decide to send some textures with reduced size or with some loss of compression. This decision, at least in some modalities, is made dynamically, texture by texture.

[046] When compared to the complete installation and execution of games on the client device (e.g., mobile games or games installed on a so-called TV) Petition 870250085825, dated 09 / 23 / 2025, p. 36 / 87 (19 / 50 intelligent), continuous client command transmission allows the game to start substantially immediately, as soon as any primary assets are transferred, since there is no heavy installation phase. Furthermore, similar to streaming video on demand (SVOD) or video-based cloud gaming, CCS allows for the transmission of a minimum amount of assets and then only transmits what is needed for the current scene or level. CCS also offers near-infinite scalability in terms of game size. CCS games are not limited or restricted by device storage capacity or other similar limitations. Updates for CCS games can be easily released to consumers. In fact, CCS games can be updated whenever needed, virtually without requiring updates to the client device. CCS also allows the game provider to control or own the execution of the game.This allows the provider to reduce the ability of a malicious user to create cheats for the game and provides a greater ability to detect such cheats. The CCS game engine can audit player behavior, track all entries, etc. The value of the game is better preserved because users cannot redistribute the games and cannot pirate or hack CCS games.

[047] Within the scope of continuous client command streaming (CCS), several other continuous command streaming topologies will be defined, each with a different set of requirements and associated benefits. For example, disaggregated continuous command streaming (DCS) makes it possible to run any process that requires a GPU (e.g., game graphics processes) on a different node, server, or environment (e.g., in a different container or virtual machine (VM)) than where the target GPU is located (e.g., the client device). As DCS Petition 870250085825, dated 09 / 23 / 2025, page 37 / 87 20 / 50 must be used on the same machine, chassis, or infrastructure; continuous transmission of disaggregated commands requires less effort (e.g., CPU resources) to make continuous transmission more compact, more space-efficient, or more bandwidth-efficient. At least in some modes, a reliable transport protocol is implemented in DCS (e.g., IPC or TCP). Furthermore, at least in some cases, DCS transmissions are Vulkan-to-Vulkan and therefore will not implement graphical API adaptations.

[048] DCS can allow for more flexibility when designing cloud gaming infrastructure (in terms of hardware and software). For example: if the systems described in this document are to run cloud games on a given device, these systems could perform rendering on another nearby device, perform game logic on an x86 CPU, perform rendering on an advanced RISC (ARM) machine-on-a-chip (SoC) system in the same chassis (for cross-compatibility or backward compatibility), and run games in a virtual machine instead of a container. These DCS modalities can also improve infrastructure utilization by separating CPU usage maximization from GPU usage and / or being able to scale resources differently.This can facilitate content targeting, as separating or splitting targeting dimensions is safer, more portable, and more fault-tolerant, since the systems described in this document do not expose the GPU device node and controllers within the container. Furthermore, the systems in this document can audit or filter each API call (Vulkan).

[049] In some embodiments, hybrid client continuous command transmission (HCCS) can be implemented to generate and transfer rendering commands to a client device. In the HCCS topology, the underlying system can generate Petition 870250085825, dated 09 / 23 / 2025, p. 38 / 87 21 / 50 Some rendering commands are executed on a distribution server (different from the media server (see FIG. 5)) and some rendering commands are generated on the client. For example, text information or heads-up display (HUD) screen are a layer rendered on the client device. In cases involving this type of text or HUD information, lower-fidelity elements are rendered on the distribution server and sent as video (using a traditional cloud gaming chain) and displayed on the client in a video layer or plane below the graphics plane. This layered or hybrid model may be suitable for specific cloud games and may alleviate some load on the distribution server's GPU. HCCS can also be used to ensure that text, user interface, or menu elements remain readable within a user interface.Furthermore, within HCCS, synchronization is carefully managed, as video information may arrive after the continuous command transmission. Therefore, HCCS will synchronize continuous text and video transmissions across the network.

[050] In some embodiments, continuous tracing command transmission (TCS) can be implemented to generate and transfer render commands, either alone or in combination with CCS and HCCS. Continuous tracing command transmission provides the ability to dump a continuous command transmission into persistent storage so it can be replayed later, as many times as desired, in a deterministic manner. TCS allows a game session to be recorded to automate testing. Furthermore, TCS accurately tracks stack performance regressions, facilitates performance optimizations, and improves debugging. TCS also allows the system to reliably capture a performance profile of the same game or the same gameplay on different hardware.

[051] In these cases, TCS corrects the workload and the Petition 870250085825, dated 09 / 23 / 2025, page 39 / 87 22 / 50 makes it replayable, allowing the system to precisely check for hardware differences and automate the collection of large datasets to train an ML model. TCS is also more reliable and / or more deterministic than recording input events. TCS offers the ability to share trace files and collaborate on trace files. In this way, systems no longer need large instruction sets to reprogram a problematic or failed game sequence. TCS also offers the ability to save and replay game sequences for clients or create features around game saves or replays.

[052] In at least some modes, continuous combination command streaming (XCS) is provided and implemented. This continuous combination streaming command combines DCS, CCS, and TCS to provide a complete cloud gaming experience without rendering and encoding video frames on the media server. XCS provides the ability to decouple the game process runtime environment from the processes that perform rendering, provides the ability to design more and perform continuous client command streaming if the client and / or game combination allows such continuous streaming, and provides the ability to capture and / or play back an existing game sequence.

[053] In addition to the benefits listed above regarding DCS, CCS, and TCS, XCS adds the following: being able to dynamically switch or simultaneously support video-based cloud games and CCS-based cloud games during development or at runtime (based on the outcome of transport protocol negotiation, for example), creating a unified vehicle for development and innovation (e.g., a single modular rendering and forwarding process that can support multiple cloud gaming use cases, Petition 870250085825, dated 09 / 23 / 2025, page 40 / 87 23 / 50 multiple hardware configurations, multiple container configurations, etc.), including CCS-based cloud gaming. At least in some cases, having a forwarding daemon or having two-stage continuous command passing means that the systems in this document can develop, refine, and debug the first-stage continuous command passing (e.g., Vulkan to Vulkan), ensure the robustness of the system's virtual resources or the virtual random access memory (VRAM) manager, etc., before adapting, scaling down, or designing for the client device or implementing client continuous command passing.

[054] Video-based cloud games can implement a central rendering daemon to implement various projects, including temporary frame storage grouping, render context sharing, viewport sharing, enhanced command dispatch scaling, vendor-independent VRAM resource deduplication, etc. This can enable and ensure long-term convergence and alignment in terms of product features between video-based cloud games and CCS-based cloud games.

[055] Returning now to FIG. 4, a computing architecture 400 is provided that allows rendering commands to be generated on a server and transmitted to a client for rendering on the client device. For example, the server 401 can implement various hardware or software modules to generate a rendering command stream and transfer these rendering commands (e.g., from 407 to 408 via a transport protocol such as SCTP or RTP) to the client device 402. On the server 401, a game container 403 or virtual machine is configured to run a game process (or multiple instances of a game process). The game process includes a game support layer (GSL) 404 and a Petition 870250085825, dated 09 / 23 / 2025, page 41 / 87 24 / 50 pseudo-Vulkan controller (e.g., a virtual installable client controller (ICD)) 405. The pseudo-Vulkan controller 405 is designed to implement some (or all) of the Vulkan APIs generated as part of the cloud gaming. The pseudo-Vulkan controller 405 will relay or send these API calls out of the container or VM to the server adapter process 406, where graphics and shader API remapping occurs, along with compression and memory management.

[056] At least in some embodiments, the 406 server adapter process or cloud game thread manager is configured to translate and adapt Vulkan continuous command transmission to continuous command transmission of a specific protocol (e.g., webGPU continuous command transmission or other client or target graphics APIs), record or emit trace files, track graphics resources and residence status, maintain a virtual VRAM model, track the interdependence of graphics resources and commands, compress GPU resources according to the best available or most appropriate scheme, perform or model VRAM or GPU resource deduplication, handle color space conversion and video encoding using a local hardware encoder, handle audio encoding, and perform other tasks.The 407 transport process receives the compressed audio stream and the continuous transmission of protocol-specific rendering commands via inter-process communication or some other means. The 407 transport process is configured to reuse the game service manager's transport implementation, receive command blobs prepared for direct sending from the server to the 406 adapter process, and / or encode audio or video bitstreams.

[057] Transport handler 408 receives these rendering commands and passes the audio to an audio API 410. The continuous rendering command transmission is passed to Petition 870250085825, dated 09 / 23 / 2025, page 42 / 87 25 / 50 the 409 client adapter. The 409 client adapter or client-side renderer can be configured to run over RDP and in a web browser. The 409 client adapter can unpack the received data, allocate computing resources to process the rendering commands, and then execute the rendering commands to render video frames on the 402 client device (e.g., using the 411 webGPU renderer).

[058] In some embodiments, the 401 server and / or the 402 client device are configured to collect performance data per game and per device while generating and processing rendering commands. The 401 server can then use machine learning to project how satisfactorily a game will run on a given client device (and whether it is possible to run the game at all). An associated ML model will be trained to recognize, based on previous client devices and previous game sessions, which games will run acceptably well on which devices. In some cases, various thresholds or performance analyses are established to ensure that a client device has the minimum GPU and / or CPU resources to acceptably process a game at a minimum number of frames per second, for example.The ML model can be continuously refined to update the system's understanding of the client device's performance, learning from real-world field implementations.

[059] In addition to raw GPU performance, other dimensions may be considered, such as available VRAM or similar. Similarly, the systems described in this document may perform characterization or profiling tests to build a functional understanding of what each cloud-based game requires in terms of computing resource capabilities (e.g., GPU, compute, resources, VRAM, ratio of streamed resources to immutable resources, etc.). In some Petition 870250085825, dated 09 / 23 / 2025, page 43 / 87 In 26 / 50 cases, this involves leveraging telemetry infrastructure, including game session statistics, with multiple GPU profiles and performance counters enabled.

[060] As noted above, the continuous command transmission described in this document can be used with video content, audio content, video games, still images, or other media items. As noted above, the contextual graphics information sent as part of the render command to the client device may include vector data for meshes, vector data specifying object positions, temporary vertex stores, 2D / 3D bitmap information, 2D / 3D non-vector information, or other information used to render the video frame or audio frame on the client device. In some cases, GPU thread information, which may be sent in addition to the contextual graphics information, includes texture sampler configuration information, viewport size information, blending mode, scissor rectangle, stencil information, or other data used by the GPU when rendering a frame.In some cases, contextual graphics information and / or GPU thread information is compressed by the media server or distribution server before transmission to the client device.

[061] For example, as shown in computing environment 500 of FIG. 5, a media server 501 is configured to generate rendering commands 505 and send these commands to a client device 507. In some cases, the media server 501 generates and sends the rendering commands 505 directly to the client device, while in other cases, a distribution server 503 that is physically closer (in some cases, within a specified distance (e.g., 1 mile, 5 miles, 10 miles, 50 miles, etc.)) to the client device generates and sends the rendering command (by Petition 870250085825, dated 09 / 23 / 2025, page 44 / 87 27 / 50 example, continuous hybrid client command transmission described above). In some embodiments, the render command includes contextual graphics information and / or GPU threading information that needs to be adapted or translated to be understood by the client device 507 or by a video game engine running on the client device. Thus, in such cases, the render command 505 is sent to an adaptation or translation process 506, where the information is appropriately translated or adapted into a format that will be recognized by the client device and / or the video game engine running on the client device 507.

[062] Additionally or alternatively, the contextual graphic information sent with the 505 render command includes 3D depth information. This 3D depth information, along with other potential 3D information, provides support for client devices that have 3D display capabilities. Thus, for example, if the client device 507 is an artificial reality or virtual reality device or a smart television or other device that has 3D capabilities, the 3D depth information sent with the 505 render command will allow the client device to play the video game or scene in three dimensions. In these cases, the contextual graphic information may also include additional viewport information for the user's second eye. The different information presented to each eye allows the user to see three-dimensional images.The depth information and the different signals for each eye may still be smaller in size than the traditionally rendered and encoded video frames transferred over the network.

[063] As noted above, client device 507 Petition 870250085825, dated 09 / 23 / 2025, page 45 / 87 28 / 50 can be any of a variety of different electronic devices, including smartphones, smart TVs, PCs, laptops, AR / VR devices, smartwatches, game consoles, or other electronic devices. In some embodiments, client devices that have enhanced processing capabilities may receive additional portions of contextual graphics information and / or GPU threading information for use in rendering the selected media frame. This may include larger viewport size (e.g., game resolution), more complex shader programs, higher quality texture sampler configuration information, larger vertex temporary stores, higher density meshes, higher resolution, etc. Because the client device has a better GPU, a better CPU, more memory, or other enhanced processing capabilities, the client device can handle and process the increased information.Media server 501 can send additional contextual information or GPU thread information when generating render commands 505 for media files 502. Similarly, distribution server 503, media server 501 can send additional contextual information or GPU thread information when generating render commands 505 for media files 504.

[064] In some modes, the transmitted rendering commands (e.g., 505) allow the client device 507 to start the video game without an installation phase and substantially without lag. Because the client device 507 renders the video frames on demand, based on the rendering command 505, the client device 507 can start the video game without a lengthy initial installation and can start the game without a long loading phase. At least in some cases, the video game can be run without installation and substantially without lag (e.g., instantly after the video game has Petition 870250085825, dated 09 / 23 / 2025, page 46 / 87 29 / 50 (been initiated) due to the media server controlling the video game's runtime. In such examples, since media server 501 (or distribution server 503) controls the video game's runtime, the media server can initialize and run the video game in an efficient manner that allows for a very fast start. In such scenarios, since media server 501 controls the video game's runtime, the media server can also implement changes to the video game in real time while the user plays.

[065] During the process of generating and transmitting rendering commands 505, the adaptation or translation process 506 may adapt or translate contextual graphical information in some way. In some embodiments, the amount of adaptation may be increased or decreased based on the graphical processing capabilities of the client device 507. For example, in some cases, the adaptation of contextual graphical information based on the graphical processing capabilities of the client device 507 includes translating shader information into a format that will be understood by the client device. In these cases, more or less shader information may be provided to the client device 507 based on its processing capabilities.

[066] In other cases, where the processing capabilities of the client device 507 are reduced, adapting contextual graphics information based on the client device's graphics processing capabilities may involve reducing a set of features in the contextual graphics information to provide a reduction in the scope of processing on the client device. This allows the client device to render graphics at a level of detail appropriate for the client device 507 and allows the device to operate in a way that presents a smooth stream of video frames (e.g., 30 fps, 60 fps, 120 fps, etc.) to the user. Petition 870250085825, dated 09 / 23 / 2025, page 47 / 87 30 / 50

[067] In at least some modes, a safety mechanism can be implemented. This safety mechanism ensures that if a given video game or video or audio file does not work on a specific client device, the media server 501 or the distribution server 503 will begin rendering and encoding video frames for transfer to the client device. Thus, in cases where continuous command transmission is not working correctly or is not an option, the safety mechanism will be triggered on the media server 501, and the client device 507 will receive fully rendered and encoded video frames. Furthermore, in such cases, the media server 501 can communicate to the user, through the client device 507, that the game or game engine uses standards that make it incompatible or inefficient with continuous command transmission.In these cases, the user will be informed that fully rendered and encoded video frames will be transferred over the network and that the corresponding increase in network traffic may affect gameplay.

[068] In addition to the method and corresponding modalities described above, a corresponding system includes at least one physical processor and physical memory comprising computer executable instructions that, when executed by the physical processor, cause the physical processor to: access media frame generation input events produced as part of a multimedia application on a media server, select at least one media frame to be rendered according to the media frame generation input events, determine the graphics processing capabilities of a client device on which the selected media frame is to be rendered, generate a rendering command for the selected media frame based on the determined graphics processing capabilities of the client device, wherein the rendering command includes contextual graphics information and Petition 870250085825, dated 09 / 23 / 2025, page 48 / 87 31 / 50 graphics processing unit (GPU) threading information for use in rendering the selected media frame on the client device and transmit the generated rendering command to the client device to start rendering the selected media frame using contextual graphics information and GPU threading information.

[069] In some examples, the method described above is encoded as computer-readable instructions in a computer-readable medium. For example, the computer-readable medium may include executable instructions that, when executed by at least one processor of a computing device, cause the computing device to access the media frame generation input events produced as part of a multimedia application on a media server, select at least one media frame to be rendered according to the media frame generation input events, determine one or more graphics processing capabilities of a client device on which the selected media frame should be rendered, generate a rendering command for the selected media frame based on the determined graphics processing capabilities of the client device,where the rendering command includes contextual graphics information and graphics processing unit (GPU) threading information for use in rendering the selected media frame on the client device, and transmits the generated rendering command to the client device to initiate rendering of the selected media frame using the contextual graphics information and GPU threading information.

[070] FIGS. 6 to 8 illustrate cloud gaming systems, infrastructure, and clients that can be implemented with the modalities described in this document. For example, the following will provide, with reference to FIG. 6, detailed descriptions of Petition 870250085825, dated 09 / 23 / 2025, page 49 / 87 32 / 50 exemplary ecosystems in which content is delivered to end nodes and in which content requests are directed to specific end nodes. The discussion corresponding to FIGS. 7 and 8 presents an overview of an exemplary delivery infrastructure and an exemplary content player used during playback sessions, respectively.

[071] FIG. 6 is a block diagram of a content distribution ecosystem 600 that includes a distribution infrastructure 610 communicating with a game client 620, a content player, or other software application designed to present rendered graphics to a user. In some embodiments, the distribution infrastructure 610 is configured to encode data at a specific data rate and to transfer the encoded data to the game client 620. The game client 620 is configured to receive the encoded data through the distribution infrastructure 610 and to decode the data for playback to a user. The data provided by the distribution infrastructure 610 includes, for example, audio, video, text, images, animations, interactive content, tactile data, virtual or augmented reality data, location data, game data, or any other type of data provided through continuous transmission.

[072] 610 distribution infrastructure generally represents any services, hardware, software, or other infrastructure components configured to deliver content to end users. For example, 610 distribution infrastructure includes content aggregation systems, media transcoding and packaging services, network components, and / or a variety of other types of hardware and software. In some cases, 610 distribution infrastructure is implemented as a highly complex distribution system, a single server, or a media device. Petition 870250085825, dated 09 / 23 / 2025, page 50 / 87 33 / 50 or anything in between. In some instances, regardless of size or complexity, the 610 distribution infrastructure includes at least one 612 physical processor and at least one 614 memory device. One or more 616 modules are stored or loaded into 614 memory to enable adaptive continuous streaming, as discussed in this document.

[073] The game client 620 generally represents any type or form of device or system capable of playing audio, video, or other game content that has been provided by the distribution infrastructure 610. Examples of game clients 620 include, without limitation, mobile phones, tablets, laptop computers, desktop computers, televisions, signal converters, digital media players, virtual reality headsets, augmented reality glasses, and / or any other type or form of device capable of rendering digital content. Like the distribution infrastructure 610, the game client 620 includes a physical processor 622, memory 624, and one or more modules 62 6.All or some of the adaptive streaming processes described in this document are performed or enabled by modules 626, and in some examples modules 616 of the distribution infrastructure 610 coordinate with modules 626 of the game client 620 to provide adaptive streaming of multimedia content.

[074] In certain embodiments, one or more of modules 616 and / or 626 in FIG. 6 represent one or more software applications or programs that, when executed by a computing device, cause the computing device to perform one or more tasks. For example, and as will be described in more detail below, one or more of modules 616 and 626 represent modules stored and configured to run on one or more general-purpose computing devices. One or more of modules 616 and 626 in Petition 870250085825, dated 09 / 23 / 2025, page 51 / 87 34 / 50 FIG. 6 also represents all or parts of one or more special-purpose computers configured to perform one or more tasks.

[075] In addition, one or more of the modules, processes, algorithms, or steps described in this document transform data, physical devices, and / or representations of physical devices from one form to another. For example, one or more of the modules mentioned in this document receive audio data to be encoded, transform the audio data by encoding it, output an encoding result for use in an adaptive audio bitrate system, transmit the transformation result to a content player, and render the transformed data to an end user for consumption. Additionally or alternatively, one or more of the modules mentioned in this document transform a processor, volatile memory, non-volatile memory, and / or any other part of a physical computing device from one form to another by running on the computing device, storing data on the computing device, and / or otherwise interacting with the computing device.

[076] Physical processors 612 and 622 generally represent any type or form of processing unit implemented in hardware capable of interpreting and / or executing computer-readable instructions. In one example, physical processors 612 and 622 access and / or modify one or more of modules 616 and 626, respectively. Additionally or alternatively, physical processors 612 and 622 execute one or more of modules 616 and 626 to facilitate adaptive continuous streaming of multimedia content. Examples of physical processors 612 and 622 include, without limitation, microprocessors, microcontrollers, central processing units (CPUs), field-programmable gate arrays (FPGAs) implementing softcore processors, and application integrated circuits. Petition 870250085825, dated 09 / 23 / 2025, page 52 / 87 35 / 50 specific (ASICs), portions of one or more thereof, variations or combinations of one or more thereof and / or any other suitable physical processor.

[077] 614 and 624 memories generally represent any type or form of volatile or non-volatile storage device or medium capable of storing computer-readable data and / or instructions. In one example, 614 and / or 624 memory stores, loads, and / or holds one or more of the 616 and 626 modules. Examples of 614 and / or 624 memory include, without limitation, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drives (HDDs), solid-state drives (SSDs), optical disk drives, caches, variations or combinations thereof, and / or any other suitable memory device or system.

[078] FIG. 7 is a block diagram of exemplary components of the content delivery infrastructure 610 according to certain embodiments. The delivery infrastructure 610 includes storage 710, services 720, and a network 730. Storage 710 generally represents any device, set of devices, and / or systems capable of storing content for delivery to end users. Storage 710 includes a central repository with devices capable of storing terabytes or petabytes of data and / or includes distributed storage systems (e.g., devices that mirror or cache content at Internet interconnection locations to provide faster access to mirrored content in certain regions). Storage 710 is also configured in any other suitable manner.

[079] As shown, storage 710 can store a variety of different items, including content 712, user data 714 and / or log data 716. Content 712 includes television programs, movies, video games, user-generated content and / or any other suitable type or form of content. Petition 870250085825, dated 09 / 23 / 2025, page 53 / 87 36 / 50 User data 714 includes personally identifiable information (PII), payment information, preference settings, language and accessibility settings, and / or any other information associated with a particular user or content player. Log data 716 includes viewing history information, network throughput information, and / or any other metrics associated with a user's connection or interactions with the distribution infrastructure 610.

[080] 720 services include 722 personalization services, 724 transcoding services, and / or 726 packaging services. 722 personalization services customize recommendations, content streams, and / or other aspects of a user experience with 610 distribution infrastructure. 724 encoding services compress media at different bitrates, which, as described in more detail below, allow for real-time switching between different encodings. 726 packaging services package the encoded video before deploying it to a delivery network, such as the 730 network, for seamless transmission.

[081] A 730 network generally represents any medium or architecture capable of facilitating communication or data transfer. A 730 network facilitates communication or data transfer using wireless and / or wired connections. Examples of 730 networks include, without limitation, an intranet, a wide area network (WAN), a local area network (LAN), a personal area network (PAN), the Internet, power line communications (PLC), a cellular network (e.g., a global system for mobile communications networks (GSM)), parts of one or more thereof, variations or combinations of one or more thereof, and / or any other suitable network. For example, as shown in FIG. 7, a 730 network includes an Internet backbone 732, an Internet service provider 734, and / or a local area network 736. As Petition 870250085825, dated 09 / 23 / 2025, page 54 / 87 37 / 50, discussed in more detail below, bandwidth limitations and bottlenecks within one or more of these network segments trigger video and / or audio bitrate adjustments.

[082] FIG. 8 is a block diagram of an exemplary implementation of the 620 game client of FIG. 6. The 620 game client generally represents any type or form of computing device capable of reading computer-executable instructions. The 620 game client includes, without limitation, laptops, tablets, desktops, servers, mobile phones, multimedia players, embedded systems, wearable devices (e.g., smartwatches, smart glasses, etc.), smart vehicles, game consoles, Internet of Things (IoT) devices such as smart appliances, variations or combinations of one or more of these, and / or any other suitable computing device.

[083] As shown in FIG. 8, in addition to the processor 622 and memory 624, the game client 620 includes a communication infrastructure 802 and a communication interface 822 coupled to a network connection 824. The game client 620 also includes a graphics interface 826 coupled to a graphics device 828, an input interface 834 coupled to an input device 836, and a storage interface 838 coupled to a storage device 840.

[084] 802 communication infrastructure generally represents any type or form of infrastructure capable of facilitating communication between one or more components of a computing device. Examples of 802 communication infrastructure include, without limitation, any type or form of communication bus (e.g., a Peripheral Component Interconnect (PCI) bus, PCI Express (PCIe) bus, a memory bus, a front-side bus, an Integrated Drive Electronics (IDE) bus, a control or register bus, a host bus, Petition 870250085825, dated 09 / 23 / 2025, page 55 / 87 38 / 50 etc.)

[085] As noted, 624 memory generally represents any type or form of volatile or non-volatile storage device or medium capable of storing computer-readable data and / or other instructions. In some examples, 624 memory stores and / or loads an 808 operating system for execution by the 622 processor. In one example, the 808 operating system includes and / or represents software that manages computer hardware and software resources and / or provides services common to computer programs and / or applications in the 620 game client.

[086] The operating system 808 performs various system management functions, such as managing hardware components (e.g., graphical interface 826, audio interface 830, input interface 834, and / or storage interface 838). The operating system 808 also provides process and memory management models for the playback application 810. The modules of the playback application 810 include, for example, a temporary content storage 812, an audio decoder 818, and a video decoder 820.

[087] Playback application 810 is configured to retrieve digital content via communication interface 822 and to play back the digital content via graphical interface 826. Graphical interface 826 is configured to transmit a rendered video signal to graphics device 828. In normal operation, playback application 810 receives a request from a user to play a specific title or specific content. Playback application 810 then identifies one or more encoded video and audio streams associated with the requested title. After playback application 810 has located the encoded streams associated with the requested title, playback application 810 downloads sequence header indices associated with each stream. Petition 870250085825, dated 09 / 23 / 2025, pp. 56 / 87 39 / 50 encoded associated with the requested title from the 610 distribution infrastructure. A sequence header index associated with the encoded content includes information related to the encoded sequence of data included in the encoded content.

[088] In one embodiment, the playback application 810 begins transferring the content associated with the requested title by transferring sequence data encoded to the lowest audio and / or video playback bitrates to minimize startup time for playback. The requested digital content file is then transferred to the temporary content storage 812, which is configured to serve as a first-in, first-out queue. In one embodiment, each downloaded data unit includes either a video data unit or an audio data unit. As the video data units associated with the requested digital content file are downloaded to the game client 620, the video data units are sent to the temporary content storage 812.Similarly, as the audio data units associated with the requested digital content file are downloaded to the game client 620, the audio data units are sent to the content temporary storage 812. In one embodiment, the video data units are stored in the video temporary storage 816 within the content temporary storage 812, and the audio data units are stored in the audio temporary storage 814 of the content temporary storage 812.

[089] A video decoder 820 reads video data units from temporary video storage 816 and outputs the video data units in a sequence of video frames corresponding in duration to the fixed playback time interval. Reading a video data unit from Petition 870250085825, dated 09 / 23 / 2025, page 57 / 87 40 / 50 of the temporary video storage 816 effectively removes the video data unit from the temporary video storage 816. The sequence of video frames is then rendered by the graphical interface 826 and transmitted to the graphics device 828 to be displayed to a user.

[090] An audio decoder 818 reads audio data units from the temporary audio storage 814 and outputs the audio data units as a sequence of audio samples, usually synchronized in time with a sequence of decoded video frames. In one embodiment, the sequence of audio samples is transmitted to the audio interface 830, which converts the sequence of audio samples into an electrical audio signal. The electrical audio signal is then transmitted to a loudspeaker of the audio device 832, which, in response, generates an acoustic output.

[091] In situations where the bandwidth of the distribution infrastructure 610 is limited and / or variable, the playback application 810 downloads and stores in temporary storage consecutive portions of video data and / or audio data from video encodings with different bitrates based on a variety of factors (e.g., scene complexity, audio complexity, network bandwidth, device capabilities, etc.). In some modes, video playback quality is prioritized over audio playback quality. Audio playback quality and video playback quality are also balanced against each other, and in some modes, audio playback quality is prioritized over video playback quality.

[092] The 826 graphics interface is configured to generate video data frames and transmit the video data frames to the 828 graphics device. In one embodiment, the 826 graphics interface is included as part of an integrated circuit, along with the 622 processor. Alternatively, Petition 870250085825, dated 09 / 23 / 2025, pp. 58 / 87 41 / 50 the 826 graphics interface is configured as a hardware accelerator that is distinct (i.e., not integrated within) a chipset that includes the 622 processor.

[093] The graphical interface 826 generally represents any type or form of device configured to forward images for display on the graphical device 828. For example, the graphical device 828 is manufactured using liquid crystal display (LCD) technology, cathode ray tube technology, and light-emitting diode (LED) display technology (organic or inorganic). In some embodiments, the graphical device 828 also includes a virtual reality display and / or an augmented reality display. The graphical device 828 includes any technically feasible means of generating an image for display. In other words, the graphical device 828 generally represents any type or form of device capable of visually displaying information forwarded by the graphical interface 826.

[094] As illustrated in FIG. 8, the game client 620 also includes at least one input device 836 coupled to the communication infrastructure 802 via the input interface 834. The input device 836 generally represents any type or form of computing device capable of providing computer-generated or human-generated input to the game client 620. Examples of input devices 836 include, without limitation, a keyboard, a pointing device, a speech recognition device, a touch screen, a wearable device (e.g., a glove, a watch, etc.), a controller, variations or combinations of one or more thereof, and / or any other type or form of electronic input mechanism.

[095] The 620 game client also includes a storage device 840 coupled to the communication infrastructure 802 via a storage interface 838. The device of Petition 870250085825, dated 09 / 23 / 2025, page 59 / 87 42 / 50 storage 840 generally represents any type or form of storage device or medium capable of storing data and / or other computer-readable instructions. For example, the storage device 840 could be a magnetic disk drive, a solid-state drive, an optical disk drive, a flash drive, or the like. The storage interface 838 generally represents any type or form of interface or device for transferring data between the storage device 840 and other components of the game client 620.

[096] Many other devices or subsystems are included in or connected to the 620 game client. On the other hand, one or more of the components and devices illustrated in FIG. 8 do not need to be present to practice the modes described and / or illustrated in this document. The devices and subsystems referenced above are also interconnected in modes different from that shown in FIG. 8. The 620 game client is also employed in any number of software, firmware, and / or hardware configurations. For example, one or more of the example modes disclosed in this document are coded as a computer program (also called computer software, software applications, computer-readable instructions, or computer control logic) in a computer-readable medium.The term “computer-readable media,” as used in this document, refers to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media include, without limitation, transmission-type media such as carrier waves, and non-transient-type media such as magnetic storage media (e.g., hard disk drives, tape drives, etc.), optical storage media (e.g., Compact Discs (CDs), Digital Video Discs (DVDs), and Blu-ray discs), and electronic storage media (e.g., solid-state drives). Petition 870250085825, dated 09 / 23 / 2025, pp. 60 / 87 43 / 50 and flash media), and other digital storage systems.

[097] A computer-readable medium containing a computer program is loaded into the game client 620. All or part of the computer program stored on the computer-readable medium is then stored in memory 624 and / or storage device 840. When executed by processor 622, a computer program loaded into memory 624 causes processor 622 to perform and / or be a device for performing the functions of one or more of the example embodiments described and / or illustrated in this document. Additionally or alternatively, one or more of the example embodiments described and / or illustrated in this document are implemented in firmware and / or hardware. For example, the game client 620 is configured as an Application-Specific Integrated Circuit (ASIC) adapted to implement one or more of the example embodiments disclosed in this document. Example Modalities:

[098] Example 1: A computer-implemented method comprising: accessing one or more media frame generation input events produced as part of a multimedia application on a media server, selecting at least one media frame to be rendered based on one or more media frame generation input events, determining one or more graphics processing capabilities of a client device on which the selected media frame is to be rendered, generating a render command for the selected media frame based on the determined graphics processing capabilities of the client device, wherein the render command includes contextual graphics information and graphics processing unit (GPU) threading information for use in rendering the selected media frame on the client device, and transmitting the generated render command to the client device to initiate rendering of the selected media frame. Petition 870250085825, dated 09 / 23 / 2025, pp. 61 / 87 44 / 50 media selected using contextual graphics information and GPU threading information.

[099] Example 2: The computer-implemented method of Example 1, where the selected frame comprises a video frame.

[100] Example 3: The computer-implemented method of Example 1 or Example 2, where the selected frame comprises an audio frame.

[101] Example 4: The computer-implemented method of any of Examples 1-3, wherein the multimedia application comprises a video game.

[102] Example 5: The computer-implemented method of any of Examples 1-4, where the generated rendering command is sent to a video game engine running on the client device.

[103] Example 6: The computer-implemented method of any of Examples 1-5, wherein the contextual graphical information comprises at least one of: vector data for meshes, vector data specifying positions, temporary vertex stores, 2D / 3D bitmap information, or 2D / 3D non-vector information.

[104] Example 7: The computer-implemented method of any of Examples 1-6, wherein the GPU thread information includes at least one of: texture sampler configuration information, viewport size, or stencil information.

[105] Example 8: The computer-implemented method of any of Examples 1-7, further comprising the compression of contextual graphics information and GPU thread information before transmission to the client device.

[106] Example 9: The computer-implemented method of any of Examples 1 to 8, wherein client devices that have increased processing capabilities receive at least Petition 870250085825, dated 09 / 23 / 2025, pp. 62 / 87 45 / 50 an additional portion of contextual graphics information or GPU threading information for use in rendering the selected media frame.

[107] Example 10: The computer-implemented method of any of Examples 1 to 9, wherein the contextual graphical information includes 3D depth information, providing support for client devices with 3D display capabilities.

[108] Example 11: The computer-implemented method of any of Examples 1-10, wherein the contextual graphical information includes different graphical information for each of a user's eyes.

[109] Example 12: The computer-implemented method of any of Examples 1-11, wherein the rendering commands are generated on a distribution server that is within a specified physical distance of the client device.

[110] Example 13: The computer-implemented method of any of Examples 1-12, where the media server controls the runtime of the multimedia application.

[111] Example 14: A system comprising: at least one physical processor and physical memory comprising computer executable instructions that, when executed by the physical processor, cause the physical processor to: access one or more media frame generation input events produced as part of a multimedia application on a media server, select at least one media frame to be rendered based on one or more media frame generation input events, determine one or more graphics processing capabilities of a client device on which the selected media frame is to be rendered, generate a rendering command for the selected media frame based on the determined graphics processing capabilities of the client device, wherein the rendering command includes contextual graphics information and unit threading information. Petition 870250085825, dated 09 / 23 / 2025, pp. 63 / 87 46 / 50 graphics processing (GPU) for use in rendering the selected media frame on the client device and transmit the generated rendering command to the client device to initiate rendering of the selected media frame using contextual graphics information and GPU threading information.

[112] Example 15: The system of Example 14 or Example 15, in which the media frame is part of a video game and in which the transmitted rendering commands allow the client device to start the video game without a setup phase and substantially without delay.

[113] Example 16: The system of any of the Examples 1415, further comprising the adaptation of contextual graphical information based on the graphical processing capabilities of the client device.

[114] Example 17: The system of any of the Examples 1416, in which the adaptation of contextual graphic information based on the graphic processing capabilities of the client device comprises the translation of shader information into a format that will be understood by the client device.

[115] Example 18: The system of any of Examples 14 to 17, in which the adaptation of contextual graphical information based on the graphical processing capabilities of the client device comprises the reduction of a set of resources in the contextual graphical information to provide a reduction in the scope of processing on the client device.

[116] Example 19: The system of any of the Examples 1418, wherein the multimedia application comprises a video game, and wherein the generated rendering command is sent to a video game engine running on the client device.

[117] Example 20: A non-transient computer-readable medium comprising one or more computer-executable instructions which, when executed by at least one Petition 870250085825, dated 09 / 23 / 2025, pp. 64 / 87 47 / 50 processor of a computing device, cause the computing device to: access one or more media frame generation input events produced as part of a multimedia application on a media server, select at least one media frame to be rendered according to one or more media frame generation input events, determine one or more graphics processing capabilities of a client device on which the selected media frame should be rendered, generate a rendering command for the selected media frame based on the determined graphics processing capabilities of the client device,where the rendering command includes contextual graphics information and graphics processing unit (GPU) threading information for use in rendering the selected media frame on the client device, and transmits the generated rendering command to the client device to initiate rendering of the selected media frame using the contextual graphics information and GPU threading information.

[118] As detailed above, the computing devices and systems described and / or illustrated in this document generally represent any type or form of computing device or system capable of executing computer-readable instructions, such as those contained in the modules described in this document. In their most basic configuration, each of these computing devices may include at least one memory device and at least one physical processor.

[119] In some instances, the term memory device generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing computer-readable data and / or instructions. In one example, a memory device may store, load, and / or hold one or more of the modules described in this document. Examples of Petition 870250085825, dated 09 / 23 / 2025, pp. 65 / 87 48 / 50 memory devices include, without limitation, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drives (HDDs), solid-state drives (SSDs), optical disk drives, caches, variations or combinations thereof, or any other suitable storage memory.

[120] In some instances, the term physical processor generally refers to any type or form of processing unit implemented in hardware capable of interpreting and / or executing computer-readable instructions. In one example, a physical processor may access and / or modify one or more modules stored in the memory device described above. Examples of physical processors include, without limitation, microprocessors, microcontrollers, Central Processing Units (CPUs), Field Programmable Gate Arrays (FPGAs) implementing softcore processors, Application-Specific Integrated Circuits (ASICs), parts of one or more thereof, variations or combinations thereof, or any other suitable physical processor.

[121] Although illustrated as separate elements, the modules described and / or illustrated in this document may represent portions of a single module or application. Furthermore, in certain embodiments, one or more of these modules may represent one or more software applications or programs that, when executed by a computing device, may cause the computing device to perform one or more tasks. For example, one or more of the modules described and / or illustrated in this document may represent modules stored and configured to run on one or more of the computing devices or systems described and / or illustrated in this document. One or more of these modules may also represent all or parts of one or more special-purpose computers configured to perform one or more tasks. Petition 870250085825, dated 09 / 23 / 2025, pp. 66 / 87 49 / 50

[122] Furthermore, one or more of the modules described in this document may transform data, physical devices and / or representations of physical devices from one form to another. Additionally or alternatively, one or more of the modules mentioned in this document may transform a processor, volatile memory, non-volatile memory and / or any other portion of a physical computing device from one form to another by running on the computing device, storing data on the computing device and / or otherwise interacting with the computing device.

[123] In some embodiments, the term “computer-readable media” generally refers to any form of device, carrier or medium capable of storing or transporting computer-readable instructions. Examples of computer-readable media include, without limitation, transmission-type media such as carrier waves, and non-transient-type media such as magnetic storage media (e.g., hard disk drives, tape drives and floppy disks), optical storage media (e.g., compact discs (CDs), digital video discs (DVDs) and BLU-RAY discs), electronic storage media (e.g., solid-state drives and flash media) and other distribution systems.

[124] The process parameters and sequence of steps described and / or illustrated in this document are given by way of example only and may be varied as desired. For example, although the steps illustrated and / or described in this document may be shown or discussed in a specific order, these steps do not necessarily have to be performed in the order illustrated or discussed. The various exemplary methods described and / or illustrated in this document may also omit one or more of the steps described or illustrated in this document or include additional steps beyond those disclosed.

[125] The previous description was provided to allow that Petition 870250085825, dated 09 / 23 / 2025, pp. 67 / 87 50 / 50 other skilled individuals in the field may make better use of various aspects of the exemplary embodiments disclosed in this document. This exemplary description is not intended to be exhaustive or limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of this disclosure. The embodiments disclosed in this document should be considered in all respects as illustrative and not restrictive. Reference should be made to the appended claims and their equivalents in determining the scope of this disclosure.

[126] Unless otherwise indicated, the terms connected and coupled to (and their derivatives), as used in the descriptive report and claims, should be interpreted as allowing direct and indirect connections (i.e., through other elements or components). Furthermore, the terms a or an, as used in the descriptive report and claims, should be interpreted as meaning at least one of. Finally, for ease of use, the terms including and having (and their derivatives), as used in the descriptive report and claims, are interchangeable and have the same meaning as the word comprising. Petition 870250085825, dated 09 / 23 / 2025, pp. 68 / 87

Claims

1 / 5 CLAIMS 1. A computer-implemented method, characterized in that it comprises: accessing one or more media frame generation input events produced as part of a multimedia application on a media server; selecting at least one media frame to be rendered based on one or more media frame generation input events; determining one or more graphics processing capabilities of a client device on which the selected media frame is to be rendered; generating a rendering command for the selected media frame based on the determined graphics processing capabilities of the client device, wherein the rendering command includes contextual graphics information and graphics processing unit (GPU) threading information for use in rendering the selected media frame on the client device;and transmit the generated rendering command to the client device to initiate rendering of the selected media frame using contextual graphics information and GPU threading information.

2. A computer-implemented method according to claim 1, characterized in that the selected media frame comprises a video frame.

3. A computer-implemented method according to claim 1, characterized in that the selected media frame comprises an audio frame.

4. A computer-implemented method according to claim 1, characterized in that the multimedia application comprises a video game.

5. A computer-implemented method, according to Petition 870250085825, dated 09 / 23 / 2025, page 69 / 87 2 / 5, claim 4, characterized in that the generated rendering command is sent to a video game engine running on the client device.

6. A computer-implemented method according to claim 1, characterized in that the contextual graphic information comprises at least one of: vector data for meshes, vector data specifying positions, temporary vertex stores, 2D / 3D bitmap information, or 2D / 3D non-vector information.

7. A computer-implemented method according to claim 1, characterized in that the GPU thread information includes at least one of: texture sampler configuration information, viewport size, or stencil information.

8. A computer-implemented method according to claim 1, characterized in that it further comprises the compression of contextual graphics information and GPU thread information prior to transmission to the client device.

9. A computer-implemented method according to claim 1, characterized in that client devices possessing enhanced processing capabilities receive at least an additional portion of contextual graphics information or GPU threading information for use in rendering the selected media frame.

10. A computer-implemented method according to claim 1, characterized in that the contextual graphic information includes 3D depth information, providing support for client devices with 3D display capabilities.

11. Computer-implemented method according to claim 10, characterized in that the contextual graphic information includes different graphic information for each eye of a user. Petition 870250085825, dated 09 / 23 / 2025, page 70 / 87 3 / 5 12. A computer-implemented method according to claim 1, characterized in that the rendering commands are generated on a distribution server that is within a specified physical distance from the client device.

13. A computer-implemented method according to claim 12, characterized in that the media server controls the execution time of the multimedia application.

14. A system characterized in that it comprises: at least one physical processor; and physical memory comprising computer executable instructions which, when executed by the physical processor, cause the physical processor to: access one or more media frame generation input events produced as part of a multimedia application on a media server; select at least one media frame to be rendered in accordance with one or more media frame generation input events; determine one or more graphics processing capabilities of a client device on which the selected media frame is to be rendered;generate a rendering command for the selected media frame based on the determined graphics processing capabilities of the client device, wherein the rendering command includes contextual graphics information and graphics processing unit (GPU) threading information for use in rendering the selected media frame on the client device; and transmit the generated rendering command to the client device to initiate rendering of the selected media frame using the contextual graphics information and the GPU threading information.

15. System, according to claim 14, characterized in that the media frame is part of a video game and in that the transmitted rendering commands allow the client device to start the video game without an installation phase and substantially without delay.

16. System according to claim 14, characterized in that it further comprises the adaptation of contextual graphic information based on the graphic processing capabilities of the client device.

17. System according to claim 16, characterized in that the adaptation of contextual graphic information based on the graphic processing capabilities of the client device comprises the translation of shader information into a format that will be understood by the client device.

18. System according to claim 16, characterized in that the adaptation of contextual graphic information based on the graphic processing capabilities of the client device comprises reducing a set of features in the contextual graphic information to provide a reduction in the scope of processing on the client device.

19. System according to claim 18, characterized in that the multimedia application comprises a video game, and in that the generated rendering command is sent to a video game engine running on the client device.

20. Non-transient computer-readable medium, characterized in that it comprises one or more computer-executable instructions which, when executed by at least one processor of a computing device, cause the computing device to: access one or more media frame generation input events produced as part of a multimedia application on a media server; Petition 870250085825, dated 09 / 23 / 2025, p.72 / 87 5 / 5 select at least one media frame that should be rendered according to one or more media frame generation input events; determine one or more graphics processing capabilities of a client device on which the selected media frame should be rendered; generate a render command for the selected media frame based on the determined graphics processing capabilities of the client device, wherein the render command includes contextual graphics information and graphics processing unit (GPU) threading information for use in rendering the selected media frame on the client device; and transmit the generated render command to the client device to initiate rendering of the selected media frame using the contextual graphics information and GPU threading information. Petition 870250085825, dated 09 / 23 / 2025, pp. 73 / 87.

Images